MEM: Multi-Scale Embodied Memory for Vision Language Action Models

In plain words

Robot policies that act from the camera image alone forget what happened a moment ago, and they forget whatever happened earlier in a task. This paper gives such a policy two kinds of memory. For the last few seconds it keeps a compact, video-like view of what the cameras saw, which helps when an arm blocks the object it wants to grab, and helps it try a different grip after a miss. For the longer story, such as which steps of a recipe are finished, it keeps a short written summary that it rewrites as the task goes on.

The two memories are combined in one model that is built on an existing robot policy. The authors report that this lets the robot carry out tasks that take up to about fifteen minutes, such as tidying a kitchen or making a grilled cheese sandwich, and that the robot adjusts its approach when a first attempt fails.

TL;DR

• MEM adds two memories to a vision-language-action (VLA) policy: short-horizon image memory and long-horizon language memory §3.1.
• Short-horizon memory is a video encoder: a ViT whose every 4th layer also attends across time, with no new learnable parameters §3.3.
• Long-horizon memory is a text summary that the high-level policy updates from its previous summary; training labels come from an LLM that compresses subtask histories §3.2.
• Integrated into pi0.6 (Gemma3-4B VLM), the authors report up to 15-minute tasks and matching pi0.6 on dexterous tasks that need no memory §4.1, §4.3. All results are author-reported.

Problem

• A policy that conditions on a dense sequence of past observations becomes infeasible for tasks that span tens of minutes §3.1. Feeding each frame through a VLA backbone makes latency grow beyond real-time limits as the context grows fig. 2 §3.3.
• Image-only memory cannot cheaply hold a multi-minute history, and a text-only memory loses fine spatial detail such as grasp height §2.
• The authors assume the robot must handle partial observability (occlusion, objects put out of view) and must work within real-time inference constraints §1 §3.3.

Method

Factorization. The full policy is written as a low-level policy that outputs an action chunk and a high-level policy that outputs the next subtask and an updated language memory §3.1, unnumbered display:

\pi(a_{t:t+H}, l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\; \pi_{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g)

• Here g is the task goal in text, l_{t+1} the subtask instruction, m_t the language memory, K \ll T the number of dense frames the low-level policy sees §3.1.
• Novelty relative to prior subtask-based hierarchies: \pi_{HL} also predicts the updated language memory from its own previous prediction §3.1.

Language memory (long-horizon). §3.2

1. The high-level policy is trained to predict m_{t+1} from m_t, the current observation and the task, so it chooses when and how to update memory §3.2.
2. Training labels are built per episode. The subtask instructions, each with a success-or-failure flag, are passed to an off-the-shelf pre-trained LLM, which is asked to summarize only information still relevant for the future §3.2.
3. The LLM is instructed to remove or compress information where possible. For example, individual object attributes are replaced by a count and a location §3.2.
4. Failed attempts are not carried into the memory: the memory does not change until the subtask succeeds §4.1.

# Build language-memory training labels for one episode  [§3.2]
# Input:  subtask instructions (T steps); success flag per subtask
# Output: memory text after each step

def label_episode(subtasks, succeeded, llm):
    # Initial memory value is not stated  [§3.2]
    memory = ""
    labels = []
    for step in range(len(subtasks)):
        # Ask a pre-trained LLM to keep only what still matters  [§3.2]
        prompt = summary_prompt(memory, subtasks[step], succeeded[step])
        memory = llm(prompt)
        labels.append(memory)
    return labels

Video encoder (short-horizon). §3.3 app. "Video encoder with Space-Time separable attention"

1. Each frame is patchified separately and enters a standard ViT §3.3.
2. A fixed sinusoidal temporal position encoding e(t) is added, with e(0)=0 for the current frame. The encoder then reuses the ViT's query, key and value projections app..
3. Every 4th layer adds causal temporal attention, in which each patch attends to the same patch at earlier frames. All other layers use spatial attention within each frame §3.3.
4. In the upper layers, tokens of past frames are dropped, so only current-frame tokens reach the VLA backbone. The number of tokens therefore matches a single-frame VLA §3.3 fig. 3.
5. Initialization: with K=1 the encoder equals the pre-trained single-image ViT, so it can start from any pre-trained VLM §3.3.

# Space-time separable video encoder over one camera  [§3.3]
# Input:  patches (T, N, d): T frames, N patches per frame  [app.]
# Output: tokens of the current frame only  # (N, d)

def video_encoder(patches, offsets, num_layers):
    # Sinusoidal time encoding; zero at offset 0  [app.]
    tokens = patches + time_encoding(offsets)
    for layer in range(num_layers):
        # Every 4th layer: causal attention across time  [§3.3]
        # Which layers keep the temporal pass is not stated beyond "every 4th"
        if layer % 4 == 0:
            tokens = temporal_causal_attention(tokens)
        # Spatial attention inside each frame  [§3.3]
        tokens = spatial_attention(tokens)
        # Drop past-frame tokens in upper layers  [§3.3] [fig. 3]
        # The exact layer where dropping starts is not stated
        tokens = keep_current_frame(tokens)
    return tokens

Cost. Naive joint attention over time and space costs \mathcal{O}(n^2K^2) per layer; the factorized version costs \mathcal{O}(Kn^2 + nK^2) §3.3, where n is patches per frame and K the number of frames.

Integration into pi0.6. §3.4

• Base model: pi0.6 VLA, initialized from a Gemma3-4B VLM. Actions are trained with discrete FAST tokens and a flow-matching action expert of 860M parameters. Gradients do not flow from the action expert into the VLM §3.4.
• Input: up to four camera streams at 448x448 px each §3.4.
• Past proprioceptive states are projected linearly into the backbone embedding, one token per state §3.4.
• Pre-training uses 6 observations (5 past plus current) with 1 s stride §3.4. Post-training extends this to up to 18 frames, described as 54 s §3.4.
• Inference uses real-time chunking (RTC), either at inference time or in training §3.4.
• Pre-training mixes teleoperated demonstrations, policy rollouts, human corrections, vision-language data and video-language data §3.4.

# Two-level control loop with mixed-modal memory  [§3.1]
# Input:  task goal (text), robot environment
# Output: executed action chunks

def control_loop(goal, env, frames_k):
    # Empty-memory start is not stated; this is a placeholder  [§3.2]
    memory = initial_memory
    frames = []
    while not env.done():
        # Keep the last K observations, K value not stated here  [§3.1]
        frames = append_last_k(frames, env.observe(), frames_k)
        # High level: next subtask and updated memory  [§3.1] [§3.2]
        subtask, memory = policy_hl(frames[-1], memory, goal)
        # Low level: action chunk from frames and subtask  [§3.1]
        actions = policy_ll(frames, subtask, goal)
        env.execute(actions)

Hyperparameters

Results

Author-reported. Setup: real robots, pi0.6 with MEM, baselines with the same backbone, 10 rollouts per policy and task §4.1 §4.3.

Numbers in the text:

Other results are shown only in figures, and their values are not stated in the text:

Claims in the text: MEM is the only model strong across all tested memory capabilities §4.3; MEM matches pi0.6 on dexterous tasks that need no memory §4.3; naive language memory (concatenated instructions) works significantly worse than learned summaries §4.1.

Paper vs code

No official code.

Searched: the paper and abstract name no code link. The project page (pi.website/research/memory) links the paper and no repository, weights or data. A GitHub repository search for "multi-scale embodied memory" returned one result, wingAGI/open-pi-mem, an unofficial reimplementation, which is excluded. The authors' organisation (Physical-Intelligence) has public repositories (openpi, aloha, others), none named for MEM; a code search inside openpi needed authentication and could not be run, so it is unverified whether openpi contains MEM code. None of the results or hyperparameters in this card were checked against code.

Pitfalls

• Memory horizon conflict. The post-training horizon is "up to 18 frames and 54 seconds" §3.4, but "up to 1 minute" §4.3. The 54 s figure implies 3 s spacing, which is not stated.
• Which observations feed the high-level policy. §3.1 conditions \pi_{HL} on o_t alone, but §3.2 writes o_{t-K:t}. The implementer must choose.
• Labels depend on an unnamed LLM. The LLM, its prompt and the compression rule are not stated. The text only says current LLMs do this "effectively" §3.2. Expect the label style, and hence the text-memory distribution, to depend on these choices.
• Train-inference shift in memory. Naive concatenated instructions degrade when a subtask is retried, because training data almost never repeats a subtask §4.1. The compression rule that drops failed attempts is the key mitigation; implementers who skip it should expect this failure.
• Temporal-attention schedule. Which layer gets the temporal pass (layer 0, 3 or other) and which layer starts dropping past-frame tokens are not stated §3.3 fig. 3. The appendix composes spatial and temporal attention in a nested form whose order is not spelled out in prose app..
• Unusual key term in the appendix. The attention formula in app. adds a key k_{0,0} to every query's softmax set. Its role is not explained. It may be a global token, but this is inferred, not stated.
• Cost claim is asymptotic. The \mathcal{O} expressions §3.3 give no constants. Latency numbers exist only as a figure fig. 2.
• Figure numbering. The main text places a teaser figure in the title block, and then decrements the figure counter in the source [main.tex]. Figures in this card are numbered 1 to 9 in order of appearance after the teaser. A compiled PDF may number them differently, so cite by section if in doubt.
• Small evaluation counts. Each task uses 10 rollouts §4.1. Standard errors on 10 trials are large, and differences between MEM and ablations should be read with that in mind.
• Author-reported only. Results come from the authors' robots and tasks. No independent evaluation or open weights are referenced.
• Pre-training data mix matters. The authors attribute robustness to pre-training on diverse robot and non-robot video, and show that memory added only in post-training is worse §4.3 fig. 8. Reproducing the method without that data mix may not reproduce the results.
• Compute and latency. Real-time latency is claimed, but the on-robot latency budget and hardware are only partly stated: timings were measured on one H100 with four camera streams fig. 2.

Lineage

• Builds on: the pi0.6 VLA, used as the base model and backbone §3.4. Also builds on the FAST tokenizer §3.4 and real-time chunking §3.4.
• Compared against: pool memory, in the style of contextVLA (average-pooled frame encodings) §4.3; proprio memory, in the style of TA (proprioceptive history only) §4.3; post-training-only video memory, in the style of CronusVLA §4.3.
• Superseded by: not known from content already in this wiki.