VideoVLA: Video Generators Can Be Generalizable Robot Manipulators

In plain words

Most robot policies that read a camera image and a sentence, then output motor commands, start from a model trained to understand pictures and text. VideoVLA starts from a different kind of model: one trained to generate video, which has already learned how scenes change over time.

The model is trained to predict two things at once. One is the next few robot actions. The other is a short video of what the scene should look like after those actions. Both come out of the same step-by-step denoising process, so each prediction can inform the other. The authors report that imagined futures that look right tend to go with successful actions, and that the model handles objects and skills it saw little of in training.

The main cost is speed. A large video model is slow to run, and the authors report about 3 Hz control on their real-robot setup.

TL;DR

• Adapts a pretrained video diffusion transformer (CogVideoX-5B) into a vision-language-action policy §3.2 §4.1.
• One denoising model jointly outputs future video latents and a 6-step action chunk; the first 3 actions are executed §3.4 §4.1.
• Author-reported gains on SIMPLER (Google and WidowX), on novel objects and on cross-embodiment skills, with small trial counts per task tab. 1 tab. 2 tab. 3 app. A.
• The official MIT-licensed repository holds only the inference script and the model/loss code; training code and optimizer settings are not in it.

Problem

Generalisation in robot manipulation to new tasks, objects and settings is limited when the policy is built on large pretrained image-and-text understanding models [abstract]. The authors argue that pretrained video generators, which model how the world evolves, are a better starting point §1. The method needs a video backbone that can be conditioned on the current image and on text §3.2.

Method

Problem formulation §3.1

Inputs are a text instruction and the current observation. Outputs are:

1. an action chunk of K actions, each a 7-D vector: three wrist-rotation dimensions, three wrist-translation dimensions, and a binary gripper value (0 closed, 1 open) §3.1;
2. a video clip of N future frames, predicted in latent form §3.1.

The action chunk and the video may run at different frequencies, so one action can cover several frames. After the chunk runs, the model is queried again on the new observation until the task ends §3.1.

Encoders §3.3

• Text. A T5 encoder maps the instruction to a fixed 226-token sequence §3.3.
• Video. The 3D causal VAE from CogVideoX encodes the clip. Because of the causal design, the first latent encodes only the first frame, which is the current observation. At inference only the observation is encoded; in training the whole clip is encoded §3.3.

Joint denoising backbone §3.2 §3.4

• A DiT-style network (Diffusion Transformer) takes one token sequence built by concatenating the text tokens, the flattened current-observation latent, the flattened future latents and the action tokens. Each latent is flattened in raster order §3.4.
• The backbone is initialised from pretrained CogVideoX §3.4.
• Future latents and actions get Gaussian noise under the DDPM schedule. The model is trained to denoise both with the DDPM diffusion loss §3.4.
• The diffusion timestep enters through adaptive LayerNorm, as in DiT §3.4.
• Attention is bidirectional by default: action tokens and video tokens can attend to each other app. B.

The method section contains no numbered equations and no algorithm box. The objective is stated in words §3.4. The paper's DDPM and DiT references are cited there; the exact loss form is taken from the code (see Paper vs code).

# VideoVLA training step  [§3.3] [§3.4] [code: loss.py#L136-L212]
# Input:  clip frames (B, N, 3, H, W), action chunk (B, K, 7), instruction
# Output: scalar loss

def training_step(frames, actions, instruction):
    # Encode the clip with the causal 3D VAE  [§3.3]
    latents = vae_encode(frames)                 # (B, n, C, h, w), C = 16
    # Encode the instruction with T5, 226 tokens  [§3.3]
    text_tokens = t5_encode(instruction)         # (B, 226, 4096)
    # Draw one diffusion step per sample, shared by video and actions  [§4.3]
    step = sample_timestep(batch_size)           # 1000 discrete steps
    # Noise both the video latents and the actions  [§3.4]
    video_noise = randn_like(latents)
    action_noise = randn_like(actions)
    noisy_video = add_noise(latents, video_noise, step)
    noisy_actions = add_noise(actions, action_noise, step)
    # Code keeps latent 1 as a clean conditioning input (channel concat)
    # and denoises all n latents; the paper says j >= 2 only  [§3.4]
    video_pred, action_pred = dit(
        text_tokens, noisy_video, noisy_actions, latents[:, :1], step
    )
    # Denoising loss on both modalities, each with weight 1
    # [code: config_use/action_config/videovla_config.yaml#L144-L145]
    video_loss = denoise_loss(video_pred, latents)
    action_loss = denoise_loss(action_pred, actions)
    return video_loss + action_loss

# VideoVLA inference: one action chunk per call  [§3.1] [§3.3] [§4.1]
# Input:  current observation image, instruction
# Output: the first 3 of 6 predicted actions

def act(observation, instruction):
    # Encode only the current observation  [§3.3]
    first_latent = vae_encode(observation)[:1]
    text_tokens = t5_encode(instruction)
    # Start video latents and actions from Gaussian noise
    # [code: sample_video_action.py#L165-L167]
    video_noise = randn(n, C, h, w)
    action_noise = randn(6, 7)
    # Denoise video and actions jointly; the repo config uses 50 steps
    # [code: config_use/action_config/videovla_config.yaml#L160]
    video, actions = joint_sampler(
        text_tokens, first_latent, video_noise, action_noise
    )
    # The repo script saves only the video; the action output is unused
    # [code: sample_video_action.py#L251-L255]
    # Only the first 3 actions are executed, then the model is queried again
    # [§4.1]
    return actions[:3]

Hyperparameters

Results

Author-reported throughout. Averages are over the tasks listed in each table. Each cell is a success rate in %. Where a cell has two values, it is "Visual Matching / Variant Aggregation" (VM / VA) tab. 1.

Table 1. SIMPLER in-domain evaluation tab. 1. Setup: trained on OXE only; WidowX (VM, 4 tasks, 24 trials each) and Google robot (VM and VA, 4 tasks) app. A. Bold and underline from the paper are omitted.

The paper reports the highest overall average, 63.0 vs 62.6 for CogACT. On WidowX Put Carrot, VideoVLA scores 20.8 against 62.5 for π0 and 50.8 for CogACT tab. 1.

Table 2. Novel objects, SIMPLER, Google robot (Pick Up, 10 objects) tab. 2 §4.1. Trials: 25 per object app. A.

Table 3. New skills, SIMPLER, Google robot tab. 3 §4.1. Skills come from the WidowX training data and are absent from the Google robot data. Trials: 20 per skill app. A.

The text reports 28.2 points over CogACT on average §4.1.

Table 4. Real-world in-domain, Realman robot tab. 4. All models are pretrained on OXE and fine-tuned on the authors' 5,824-sample dataset. Place reports both stages tab. 4 caption. Trials: Pick Up 24, Stack 48, Place 24 app. A.

Table 5. Real-world novel objects, Realman robot tab. 5. Task: pick up the novel object and place it on a coloured plate; colours White, Blue and Yellow. Trials: 12 per object app. A.

Table 6. Real-world cross-embodiment skills, Realman robot tab. 6. Skills seen only on the WidowX robot. Trials: 16 per skill app. A.

Ablations (SIMPLER, Google robot, Visual Matching; 3 tasks unless noted).

The generalisation columns of the dual-prediction ablation are 65.2 / 48.6 for the default, 12.7 / 4.4 for no video loss, and 11.3 / 2.1 for action only tab. 9.

Imagination vs execution tab. 10 §4.4. Visual imagination is judged by humans. Success requires following the instruction and having no salient geometric distortion. Novel objects: 84.0 imagined, 65.2 executed. New skills: 63.4 imagined, 48.6 executed.

Motion similarity §4.4 fig. 2. The authors compare SIFT and SAM-PT trajectories of the imagined and executed videos with Hungarian matching and cosine similarity. The paper reports that higher similarity goes with higher success. Plot values are not tabulated in the text.

Trials per task app. A, tables 11 and 12. SIMPLER Google VM: 300, 240, 216, 108 trials; VA: 825, 600, 378, 189. WidowX: 24 per task. Real robot: Pick Up 24, Stack 48, Place 24. At 24 trials, one success is 4.2 points, so the small gaps in the tables (for example 63.0 vs 62.6) are within noise.

Paper vs code

Checked at commit 05b5cca7d4ec32e786b7e10dcbfbf5926637b293 of VideoVLA-Project/VideoVLA (MIT), linked from the project page [https://videovla-nips2025.github.io/]. The repo has no training script, so training settings can only be checked against the config and loss.

• Sampler and steps. Paper: "DDIM sampling with 50 denoising steps" §4.1. Code: the sampler class is VPSDEDPMPP2MSampler_withact, a DPM-Solver++ (2M) style multistep update, with 50 steps sampling.py videovla_config.yaml. The limitations section says 10 DDIM steps give ~1.1 s per chunk app. D. The repo ships no 10-step setting.
• Guidance. The paper does not mention classifier-free guidance. The config uses DynamicCFG with scale 6 and a 10% text-dropout rate videovla_config.yaml videovla_config.yaml guiders.py.
• First latent. Paper: the current-observation latent V1 is a clean conditioning token and only future latents j ≥ 2 are denoised §3.4. Code: the first-frame image is noised with a log-normal sigma before encoding diffusion_video.py diffusion_video.py, channel-concatenated to the noisy video (in_channels 32 = 16 + 16) dit_video_concat_withact.py, dropped to zeros with 5% probability diffusion_video.py, and the loss covers all latents because fixed_frames: 0 and the loss has no masking loss.py videovla_config.yaml.
• Loss form. Paper: "DDPM's diffusion loss" §3.4. Code: the sgm denoiser with zero-terminal-SNR DDPM discretisation; the loss compares denoiser output to the clean target with weight 1/(1 − ᾱ), for video and actions alike loss.py videovla_config.yaml.
• Robot-ID embedding. Not in the paper. The DiT adds a learned robot-ID embedding to the action tokens dit_video_concat_withact.py dit_video_concat_withact.py. The inference script sets robot_ids to action_dim, i.e. 7 sample_video_action.py. The robot_ids: 1 value in the inference config is not read by the code inference.yaml.
• Action output. The paper says the model predicts the 6-step chunk and that the first 3 are executed §4.1. The inference script computes the action samples but saves only the video sample_video_action.py sample_video_action.py. No action decoding, de-normalisation, gripper thresholding or execution loop is in the repo.
• Optimizer and schedule. Paper: AdamW, lr 1e-5, weight decay 1e-4, 100K pretraining and 15K fine-tuning iterations, batch 256 §4.1. Code: no training script; the learning rate and weight decay are CLI arguments arguments_withact.py. Not verifiable from the repo.
• Consistent. Action chunk length 6 and action dimension 7 videovla_config.yaml videovla_config.yaml; T5 with 226 tokens videovla_config.yaml; 13 frames for real-robot inference inference.yaml; bidirectional, non-decoder transformer videovla_config.yaml (attention masks not traced line by line); synchronous shared timestep for video and actions loss.py.

Pitfalls

• Sampler default. Using the repo's 50-step default is not the 10-step setting behind the reported 1.1 s per chunk app. D. Expect higher latency with the default (not measured here).
• Latent 1 handling. The paper treats latent 1 as a clean observation token §3.4. The code denoises it and conditions through channel concatenation of a noised first frame. An implementation that follows the paper will not match the code's behaviour.
• Undocumented noise augmentation. The first-frame noise (log-normal sigma, mean −3, std 0.5) and the 5% conditioning dropout are not in the paper diffusion_video.py videovla_config.yaml.
• Action outputs. No gripper binarisation, normalisation or decoding is in the repo, so the binary gripper dimension §3.1 has to be implemented by you.
• Robot-ID. The inference script's robot_ids value differs from the config value, and the config field is ignored sample_video_action.py.
• Trial counts. The real-world novel-object table has 13 columns (10 named objects plus B-1 to B-3) while the text says 12 objects §4.2 tab. 5. Treat the table headers and text as ambiguous. Small trial counts (12 to 48) make differences of a few points unreliable app. A.
• Human-judged imagination. Visual imagination success is human-rated, not automated §4.4.
• Speed. The authors report ~3 Hz effective control on an H100 app. D. Training used 32 AMD MI300X GPUs with batch 256 §4.1. Inference memory for a 5B backbone is not stated.
• Backbone quality matters. Training from scratch drops to 12.6 average tab. 7. The pretrained video prior is load-bearing.
• Frame vs action frequency. Frames and actions may run at different rates §3.1. The paper gives no mapping between them beyond the frame counts §4.1.
• Table and figure numbers follow source order in the LaTeX. Floats may shift them in the rendered PDF.

Lineage

• Builds on: CogVideoX-5B video generator and its 3D causal VAE and T5 text encoder §3.3 §4.1. The DiT architecture §3.4. DDPM §3.4. Open X-Embodiment pretraining data, card at [/papers/2310.08864] §4. SIMPLER evaluation environment §4.1.
• Compared against: OpenVLA [/papers/2406.09246], SpatialVLA, π0, CogACT, Octo, RT-1-X and RT-2-X tab. 1.
• Superseded by: not known from the paper or the repo.