EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies

In plain words

Robot policies that only look at the current camera image forget things that go out of view. If a robot lifts a cover to see a block's colour and then puts the cover back, the colour is gone from the picture. Many earlier fixes keep a running log of past images, but the log fills up with near-identical frames, or they squeeze the history into a short summary that loses detail.

EventVLA keeps two kinds of memory. The first is a fixed set of reference frames: the very first image, which shows the layout, and a few recent frames, which show motion. The second is a learned "notebook" that the policy writes to on purpose. Alongside its action prediction, the policy guesses which of the next steps will show something worth remembering, and when a guess is confident enough, it saves that frame. Only a handful of such frames are kept, and the oldest is dropped when the notebook is full.

The idea is that the policy decides when to remember, instead of storing everything or nothing. The authors also built a simulation benchmark where the answer to a task depends on something seen only briefly in the middle of the episode.

TL;DR

EventVLA is an end-to-end VLA policy built on a Qwen3-VL-4B backbone in simulation (π0.5 on the real robot). Besides fixed visual anchors, a small head predicts keyframe probabilities over the 50-step action chunk, and frames above a threshold are written into a bounded FIFO buffer. The authors report 67.8% on RMBench, 75.2% on their RoboTwin-MeM benchmark, and 78.8% on four real bimanual tasks, all author-reported. Use it when a task hinges on a briefly visible cue, and check the Paper vs code section before reproducing the numbers, because the released code differs in several hyperparameters.

Problem

Standard VLA policies map the current observation and instruction to an action and assume the task-relevant state stays visible §1. When a cue appears only briefly, for example under an opaque cover, a fixed history window misses it, and dense memory buffers fill with redundant frames §1 §2. Recurrent or compressed memories lose fine visual detail §1. The paper asks when and what visual evidence a VLA should keep, without exhausting compute §1.

The authors also argue that existing memory benchmarks can be solved with static anchors alone (RMBench) or test broader memory skills (RoboMME) §2. The new benchmark is meant to isolate transient, mid-episode evidence §4.

Method

Memory structure. The policy is a_t = π(o_t, M_{t-1}, l), where the memory is M_t = A_t ∪ E_t eq. 1 §3.1. The anchors are the initial frame o_0 plus a sliding window of recent frames o_{t-K} … o_{t-1} eq. 2 §3.1. The sim configuration uses the frames at offsets −30 and −15 in addition to o_0 app. B.3 tab. 6 tab. 7. The real-robot configuration uses o_0 with offsets −60, −40 and −20 tab. 8. The paper calls the anchors rule-based, and the code keeps them as fixed frame offsets [code: EventVLA/examples/RoboTwin-Mem/train_files/eventvla_robotwin_mem.yaml#L138-L143].

Keyframe Evidence Memory (KEM). A lightweight head reads the final hidden states of the action-query tokens for the H-step chunk and outputs one sigmoid probability per future step §3.2 eq. 3. The default foresight horizon is H_k = H = 50 §3.2. In the released code the head is a LayerNorm, a linear layer, GELU and a linear layer to one logit [code: EventVLA/eventvla/model/framework/EventVLA.py#L299-L304].

Writing events. At step t, a predicted step i ≥ 1 whose probability reaches τ_commit = 0.55 causes the raw image at t+i to be committed to the event buffer E_t §3.2 app. B.3. The buffer has capacity N_max = 5 and drops its oldest entry when full §3.2 app. B.3. At each step, the model input is the concatenation of anchors, buffer entries and the current frame, in that order eq. 4 §3.2.

Training. The total loss is an action loss plus λ times a keyframe loss eq. 5 §3.3. The keyframe loss is a sequence-averaged binary cross-entropy against soft targets app. A.1 eq. 6. Each target is a raised cosine of the step distance to the nearest ground-truth keyframe: y = 0.5 (1 + cos(π d / R)) for d ≤ R, and 0 otherwise app. A.1. The radius R is not given in the text. During training, each sample writes ground-truth keyframes with probability α and its own thresholded predictions otherwise. α decays linearly from 1 to 0 over training app. A.1 §3.3.

Keyframe labels. The paper labels training demonstrations offline with Qwen3-VL-235B-A22B-Instruct-FP8 served by vLLM, using 2 to 3 hand-annotated demonstrations per task as in-context examples and 128 uniformly sampled frames per episode, for example app. A.3. The labeling prompt is given in the appendix app. A.3.

Inference. The paper converts the probability vector into write events by local-peak selection (non-maximum suppression, radius w) followed by a cooldown of C steps app. A.2 eq. 8. Pseudocode below follows that description. The released code differs, see Paper vs code.

# One training step of the KEM policy  [§3.2] [§3.3] [eq. 1]-[eq. 5]
# Input:  frames o_0 .. o_t (main camera), instruction l,
#         ground-truth keyframe steps of the episode, event buffer
# Output: scalar training loss

def training_step(frames, t, instruction, gt_keyframes, buffer):
    # Visual anchors: first frame plus two earlier frames (sim)  [eq. 2]
    # [app. B.3]; the code pads or clips when t < 30 (not checked)
    anchors = [frames[0], frames[t - 30], frames[t - 15]]
    # Event buffer: newest N_max = 5 stored keyframe images, FIFO  [§3.2]
    memory = buffer[-5:]
    # Hidden states of the action-query tokens, one per future step  [§3.2]
    hidden = vla_backbone(anchors, frames[t], memory, instruction)
    # Keyframe logits for the H = 50 future steps  [eq. 3]
    logits = keyframe_head(hidden)            # (H,)
    # Soft target: raised cosine around each ground-truth keyframe  [app. A.1]
    targets = soft_targets(t, gt_keyframes, radius=R)   # (H,)
    # Sequence-averaged BCE; the released code weights positives by 7.0
    # [app. A.1] [code: EventVLA/eventvla/model/framework/EventVLA.py#L1543-L1552]
    keyframe_loss = weighted_bce(logits, targets, pos_weight=7.0)
    # Action loss; the released code uses L1 regression  [eq. 5]
    # [code: EventVLA/eventvla/model/framework/EventVLA.py#L1247]
    action_loss = l1(predict_actions(hidden), actions_gt)
    # Joint objective with balancing weight lambda  [eq. 5]
    return action_loss + lam * keyframe_loss

# Write-event selection at inference, as the paper describes it
# [app. A.2] [eq. 8]
# Input:  probs with probs[i] = p_t^i for i in 1..H (probs[0] unused)
#         t: current step; last_write: step of the last stored keyframe
#         (None before the first write)
# Output: list of steps at which to store the current image

def select_writes(probs, t, last_write, tau, nms_w, cooldown):
    # Candidates: at or above tau and maximal within +-nms_w  [eq. 8]
    candidates = []
    for i in range(1, H + 1):
        # NMS neighbourhood, clipped to the chunk  [eq. 8]
        low = max(1, i - nms_w)
        high = min(H, i + nms_w)
        local_max = max(probs[low:high + 1])
        if probs[i] >= tau and probs[i] == local_max:
            candidates.append(i)
    # Accept in order; a write needs a gap larger than cooldown  [app. A.2]
    writes = []
    for i in candidates:
        if last_write is None or t + i - last_write > cooldown:
            writes.append(t + i)
            last_write = t + i
    return writes

Hyperparameters

Sim values come from the RoboTwin-MeM and RMBench tables. The real-robot column comes from the real-robot table. "Not stated" means the paper does not give a value. Values the released code uses differently are flagged in Paper vs code.

Results

Author-reported throughout. Averages are over the tasks in each table.

RMBench, 9 tasks tab. 1 tab. 9. The streamlined EventVLA variant uses visual anchors only. The table averages are as printed.

RoboTwin-MeM, 8 tasks tab. 2. The n-value is the number of intermediate keyframes each task requires §4 tab. 5. Columns are grouped by n.

RoboTwin 2.0, Markovian tasks tab. 3. Success rates, Easy and Hard splits. The paper says the memory mechanism does not hurt reactive control.

Controlled comparison of memory strategies tab. 4. Same backbone and memory budget. Real results are averaged over task configurations and seeds.

Foresight horizon and annotation source tab. 10.

Real-world bimanual robot, ARX ACONE, 20 trials per task §5.3 fig. 4. Per-task numbers are in the text; the π0.5 range is also from the text.

Inference cost, A100, batch size 1 tab. 11 §5.3:

Paper vs code

Official code: https://github.com/InternRobotics/EventVLA, cloned at commit a4e02f741f78ab01eb76f6a0bc6685f72f60ed24. The abstract links it. The organisation is the Shanghai AI Laboratory's research group, and the paper's affiliations include Shanghai AI Laboratory. The repository holds two code trees, EventVLA/ (training and serving) and RoboTwin-Mem/ (benchmark and data tools). The root has no LICENSE file. EventVLA/LICENSE is MIT (copyright StarVLA Team), and RoboTwin-Mem/LICENSE is MIT.

Discrepancies, each with both sides:

1. Keyframe loss weight. The sim table lists λ = 0.1 tab. 7, and the real-robot text sets λ = 0.1 app. B.3. The released sim config and launch script use keyframe_loss_weight: 1.0 [code: EventVLA/examples/RoboTwin-Mem/train_files/eventvla_robotwin_mem.yaml#L56] run_eventvla_train.sh.
2. BCE positive weight. The paper's keyframe loss is an unweighted sequence-averaged BCE eq. 6. The code uses pos_weight = 7.0 [code: EventVLA/eventvla/model/framework/EventVLA.py#L1543-L1552] [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L68].
3. Inference write rule. The paper specifies non-maximum suppression with radius w = 8 and cooldown C = 10 app. A.2 eq. 8 app. B.3. The released YAML sets both to 20 [code: EventVLA/examples/RoboTwin-Mem/train_files/eventvla_robotwin_mem.yaml#L64-L65]. The released eval client turns the model-side filter off [code: EventVLA/examples/RoboTwin-Mem/eval_files/model2robotwin_mem_interface.py#L216] and uses a single candidate per chunk, the argmax over future steps, if it reaches the threshold 0.55 [code: EventVLA/examples/RoboTwin-Mem/eval_files/model2robotwin_mem_interface.py#L675-L700]. It rejects a candidate within a 20-step window of the last committed keyframe [code: EventVLA/examples/RoboTwin-Mem/eval_files/model2robotwin_mem_interface.py#L212-L215] EventVLA.py. The NMS code path runs only when enable_keyframe_inference_event_filter is set EventVLA.py.
4. Teacher-forcing schedule. The paper says α decays linearly "over the training duration" app. A.1 §3.3. The released run holds α = 1 for 10,000 steps and then decays linearly over 30,000 steps [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L77-L78] EventVLA.py.
5. Training steps. The sim table says 80,000 steps tab. 6 tab. 7. The launch script passes 100,000 [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L103]. The YAML says 80,000 [code: EventVLA/examples/RoboTwin-Mem/train_files/eventvla_robotwin_mem.yaml#L147].
6. Buffer size. The YAML default is 4 [code: EventVLA/examples/RoboTwin-Mem/train_files/eventvla_robotwin_mem.yaml#L54], but the launch script sets 5 [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L19], which matches the paper app. B.3.
7. Action loss. The paper writes a generic "regression or flow-matching" action loss eq. 5 app. A.1. The code uses L1 regression on predicted actions EventVLA.py.
8. Stored camera. The paper does not say which camera is stored. The released code stores only the main camera (cam_high, head or main), and excludes the wrist cameras [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L96-L97].
9. Real-robot code is not released. The repository's to-do list has "Real-world inference and evaluation code" and "Real-world fine-tuned model" unchecked README.md. A Pi05MEM class exists [code: EventVLA/eventvla/model/framework/Pi05MEM.py#L32-L33], but no real-robot training or evaluation config was found. The real-robot numbers cannot be reproduced from this repository.
10. Keyframe labeling is not released. I found no script that calls Qwen3-VL to label keyframes. extract_task_keyframes.py reads a keyframe_steps field from episode metadata rather than computing it [code: RoboTwin-Mem/script/extract_task_keyframes.py#L132]. Where that metadata comes from was not checked.

Checked and consistent: the sim anchor offsets (−30 and −15 frames) and the commit threshold (0.55) match between paper and code.

Pitfalls

• Citation numbering. Equation numbers here follow LaTeX source order, because the compiled PDF was not available. Equations eq. 6 and eq. 8 are the appendix keyframe loss and NMS rule, and the joint loss appears again in the appendix. Table numbers follow the same order. Check against the compiled PDF before quoting a number.
• The RMBench result does not test KEM. The 67.8% headline on RMBench is the anchors-only variant tab. 1 tab. 9. With KEM added, RMBench drops to 66.7% tab. 4. The KEM gains are on RoboTwin-MeM and the real robot.
• Headline gain is not defined. The abstract's "+40%" average improvement does not appear in any table. Computed here from the printed tables, not stated by the authors: RMBench 67.8 vs Mem-0 42.0 (+25.8 points); RoboTwin-MeM 75.2 vs MemoryVLA (QwenOFT) 10.8 (+64.4); real 78.8 vs π_MEM 37.5 (+41.3).
• Anchors at the start of an episode. The sim config reads frames at t−30 and t−15 eventvla_robotwin_mem.yaml. The paper does not say what fills these slots before step 30. I did not verify the code's padding behaviour.
• Write delay. The eval client commits a planned write only when its step is reached model2robotwin_mem_interface.py and replans after the commit. The paper does not describe this delay. An implementation that writes at prediction time will behave differently.
• Event-count ranges. The benchmark table gives Press Button Keyframe a range of [2, 5] and Find Seal and Seal Stamp a range of [1, 4] tab. 5. The results table files them under n = 5 and n = 4. The paper does not explain the mapping.
• Buffer overflow. Long episodes (over 1,500 steps for some tasks app. A.3) fill the 5-slot FIFO quickly. The Limitations section says early evidence is evicted §6. Plan a larger buffer if the task has many events.
• Labeling accuracy. The paper reports an average temporal error under 10 steps in simulation and within 50 steps on real data app. A.3. These are the authors' figures. The labeler's code and prompt-level reproducibility are not released.
• Cross-task transfer is untested. The authors list unseen objects, distractors and cross-task transfer as unexplored §6.
• Compute. Training uses 8 GPUs [code: EventVLA/examples/RoboTwin-Mem/train_files/run_eventvla_train.sh#L58]. Inference is reported at about 7 GB on an A100 tab. 11.

Lineage

• Builds on: the QwenOFT backbone from the StarVLA code base [code: EventVLA/eventvla/model/framework/EventVLA.py] (the paper cites [starvla]; no arXiv id is available in this wiki's sources). Also builds on the Qwen3-VL-4B backbone app. B.3.
• Compared against: Diffusion Policy, ACT, π0.5, X-VLA, QwenOFT tab. 1; MemER, Mem-0, MemoryVLA (OpenVLA and QwenOFT variants) tab. 1; π_MEM §5.3. The paper also reproduces Periodic, Random, Visual-change and Reactive frame-selection baselines tab. 4.
• Differs from MemER: the paper says its contribution is foresight-based, end-to-end keyframe prediction rather than keyframe retrieval §2.
• Superseded by: not known from wiki content.