DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control

In plain words

Most robot "brains" built on large vision-language models are scaled up on the language and vision side, while the part that produces motions gets less attention. DexVLA keeps a vision-language model that reads the camera images and the instruction, and adds a separate, much larger action generator with about one billion parameters. That generator makes motions with the same kind of step-by-step denoising that image diffusion models use.

Training happens in three stages, from simple to hard, much as a learner would progress. First the action generator learns general motor skills from data gathered on many different robot bodies. Then it is aligned with one specific robot. Finally it is adapted to a particular task. The model also writes short sub-step descriptions of what it is doing, such as smoothing the wrinkles, then aligning the sleeves, then securing the folds. The authors found that this lets it finish long tasks without a separate high-level planner.

The authors report that the system folds a shirt without task-specific training and handles a long laundry task better than pi0 when both are given the task as a plain instruction. These are the authors' own numbers, measured on their own robots.

TL;DR

• DexVLA pairs a Qwen2-VL-2B backbone with a 1B-parameter diffusion-transformer action expert (ScaleDP-H), joined by a projection and FiLM-style conditioning §3.1 app. C.3.
• Training has three stages: cross-embodiment expert pre-training, embodiment alignment, and task post-training §3.2.
• Author-reported: 0.92 shirt folding without task-specific training §4.1; laundry folding 0.4 versus pi0 0.2 under direct prompting §4.3; LIBERO average 97.3 app. A.2.
• The official code is non-commercial, has no stage-1 training code, and departs from the paper in several places: optimizer beta, freezing in stage 2, the embodiment heads, and the projector and FiLM design (see Paper vs code).

Problem

The authors name two bottlenecks in generalist robot policies §1. First, data scarcity: OpenVLA and Octo rely on the Open X-Embodiment dataset (about 4,000 hours), and pi0 and pi0.5 on a 10,000-hour dataset, and human demonstrations are costly to collect §1. Second, architectural imbalance: scaling goes mostly into the VLM (OpenVLA uses a 7B VLM, pi0 a 3B VLM), so the VLM stays disconnected from sensorimotor context §1.

For long, contact-rich tasks, the paper says pi0 needs an external high-level policy, SayCan, that updates the instruction every two seconds §1 §3.2. DexVLA targets tasks such as laundry folding, where one episode takes more than two minutes, using only direct language prompting and no external planner §4.3 §1.

Method

Architecture §3.1 app. C.3. The backbone is Qwen2-VL-2B. Images from up to three camera views are encoded and their visual tokens concatenated. The VLM produces two kinds of output: reasoning tokens (the sub-step text) and action tokens. According to the paper, the action tokens go through a projection of two linear layers with LayerNorm, and the reasoning tokens modulate the expert through FiLM layers §3.1. The action expert is ScaleDP, a transformer diffusion policy with 32 blocks, width 1280 and 16 heads, about 1B parameters app. C.3. The paper states the expert has one output head per robot configuration, similar to Octo §3.1. The released code implements the projection and FiLM differently; see Paper vs code.

Losses §3.1. The total loss is L = L_diff + alpha * L_ntp, where L_diff is the diffusion loss on actions and L_ntp is the next-token loss on the language output. The paper sets alpha = 1 in all experiments, because L_ntp converges early in training §3.1. The paper does not define L_diff further (noise schedule, prediction target and loss form are not stated in the paper).

Stage 1: cross-embodiment pre-training §3.2 app. C.3 app. C.4. Only the action expert is trained, decoupled from the VLM. Each camera view goes through a randomly initialised ResNet-50 and the instruction through a pretrained DistilBERT, and the language embedding enters through FiLM layers. The data is all cross-embodiment data, about 100 hours across 91 tasks and four embodiments. Sub-step annotations are provided about every five seconds of demonstration §3.2 app. C.2.

Stage 2: embodiment-specific alignment §3.2. The data is filtered to a single embodiment. The VLM, the projection and the expert are trained jointly. The paper says the VLM's visual encoder is frozen in this stage §3.2. The text says this alignment lets the expert ground the VLM's understanding in the target robot's motor space §3.2.

Stage 3: task-specific adaptation §3.2. Expert demonstrations for a specific task are used to refine fluent execution. The paper uses this stage for dexterous, long tasks. It skips the stage for simpler tasks that the model already handles well, such as shirt folding and picking known objects from a bin §3.2.

Sub-step reasoning §3.2 app. C.2. In stages 2 and 3, sub-step annotations are used as intermediate language output that the model learns to generate, not as input instructions §3.2. Annotation used Grounding-DINO and DINOv2 for object tasks (grasp success from box IoU between gripper and object), and Google Gemini 2.0 to segment long single-object tasks against a hand-written sub-step list. Only the stage-3 labels were checked by hand app. C.2.

Action expert: conditioning and sampling. The paper says the reasoning drives the expert through FiLM but does not state how many condition vectors the expert receives. The released code passes one pooled vector per example (plus the robot state), not a token sequence. The expert predicts noise (epsilon) and samples with DDIM; the code uses 10 sampling steps at inference. The paper does not state these details §3.1. The code values are listed under Paper vs code.

# One training step: VLM next-token loss plus action-expert loss  [§3.1]
# Input:  images (B, V, 3, 240, 320); prompt; sub-step labels; actions
# Output: scalar loss

def training_step(batch):
    # Encode each camera view and concatenate the visual tokens  [§3.1]
    visual_tokens = concat_views(batch.images)                  # (B, N, d)
    # Run the VLM over visual and language tokens  [§3.1]
    vlm_hidden = vlm(visual_tokens, batch.prompt)               # (B, T, d)
    # Next-token loss on the sub-step text, as the VLM's own output [§3.2]
    ntp_loss = cross_entropy(lm_head(vlm_hidden), batch.substep_labels)
    # Turn the VLM output into one condition vector for the expert [§3.1]
    condition = make_condition(vlm_hidden, batch.substep_labels)  # (B, d)
    # Diffusion loss on the action chunk for the given robot state [§3.1]
    diff_loss = action_expert_loss(batch.actions, condition, batch.state)
    # Weighted sum; alpha = 1 in all experiments  [§3.1]
    return diff_loss + alpha * ntp_loss

# Diffusion loss for the action expert (epsilon prediction)  [§3.1]
# Input:  actions (B, H, A) scaled to [-1, 1]; is_pad (B, H); condition (B, d)
# Output: scalar error averaged over the chunk

def action_expert_loss(actions, is_pad, condition, state):
    # Draw one noise level per example  (100 levels in code; not stated)
    timesteps = random_int(0, 100, size=batch_size)
    noise = random_normal(shape=actions.shape)
    # Forward process: noise the clean action chunk to that level
    noisy_actions = add_noise(actions, noise, timesteps)
    # Predict the noise from noisy actions, time, condition and state
    noise_pred = action_expert(noisy_actions, timesteps, condition, state)
    # Squared error with padded future steps zeroed out  (code detail)
    squared_error = (noise_pred - noise) ** 2
    valid = ~is_pad                                             # (B, H, 1)
    return (squared_error * valid).mean()

# Inference: plan a chunk, execute it from a queue, re-plan periodically
# Input:  policy, robot, instruction, stats (action min and max)
# Output: none (runs for a fixed step budget)

def run_episode(policy, robot, instruction, stats):
    # Actions planned but not yet executed
    queued_actions = []
    for step in range(max_steps):
        # Every 16th control step, plan a new chunk and reset the queue
        if step % 16 == 0:
            observation = robot.observe()
            # Predicted chunk, shape (H, A); H = 50 in the training scripts
            chunk = policy.predict_chunk(observation, instruction)
            # Keep only the first 16 actions of the new chunk
            queued_actions = chunk[:16]
        # Pop one action, map it back from [-1, 1], and execute it
        action = denormalise(queued_actions.pop(0), stats)
        robot.execute(action)

Hyperparameters

Values from the paper's hyperparameter table and text are cited to the paper. Values the paper does not give are "not stated"; values taken from the official code are marked as code and cited to the file.

Results

Author-reported, on the authors' own robots. Each task was scored over 10 trials unless noted §4.1 §4.2 §4.3. Values the text does not give are marked "not stated"; the paper's bar charts are not transcribed here.

Without task-specific adaptation (Stage 2 model, one set of parameters) §4.1:

New embodiment, 100 demonstrations each (fine-tuned from Stage 2) §4.2:

Long-horizon, direct prompting (no task-specific adaptation claim beyond Stage 3) §4.3 app. A.4 app. D.2:

Ablation: training stages (average score; shirt folding and laundry folding) §4.4:

Ablation: action-expert size (shirt folding, average score) §4.4: UNet 93M: 0.17; diffusion expert 410M: 0.63; diffusion expert 1B: 0.92.

Ablation: sub-step reasoning (shirt folding, average score) app. D.1: 0.07 and 0.0 for the two single-stage settings, 0.92 with sub-steps in both stages. The table rows do not map cleanly onto the text (see Pitfalls).

Sub-step reasoning versus SayCan (bussing table, hard) app. D.2: SayCan 0.58; DexVLA's implicit sub-steps 0.70.

LIBERO simulation (Stage 1 weights only, not Stage 2 or 3) app. A.2:

The paper does not say whether the baseline numbers were re-run or copied.

Visual generalisation, 3 trials per setting app. A.1:

Zero-shot transfer, gripper to dexterous hand app. A.5: 60% success over 30 novel objects, against 67% with the original gripper.

Training cost of stage 1 app. A.3: 0.89 epochs per hour when training only the expert, 0.32 when training the whole VLA; the paper reports 2.78 times faster. Hardware is not stated.

Paper vs code

Checked against the official repository https://github.com/midea-ai/DexVLA at commit e0810a18986fa3a9b62437feebeaf019194fc902. Repository choice: the owner is midea-ai, the organisation of the authors' company (Midea Group, affiliation 1 of the author list). The same organisation hosts other papers' code by these authors (for example TinyVLA, arXiv:2409.12514, whose author list includes Junjie Wen and Yichen Zhu). Its README names the authors and links the project page. The README also points to juruobenruo/DexVLA, a repository owned by a student account that is not among the authors. I did not use that one. I read the training scripts, train_vla.py, the Qwen2-VL model and trainer code, the ScaleDP policy head, the data normalisation and the evaluation script.

• Stage 1 code is missing. No stage-1 training script or dataset loader for cross-embodiment pretraining is in the repository. The README links ScaleDP-H weights it describes as the stage-1 output README.md. The stage-1 learning rate, the ResNet-50 and DistilBERT path, and the cross-embodiment data pipeline cannot be checked against the code.
• Embodiment heads. The paper describes one head per robot configuration §3.1. The policy head has one output layer [code: policy_heads/models/transformer_diffusion/modeling_scaledp.py#L246] and no per-embodiment indexing that I could find.
• Adam beta2. The paper gives beta2 = 0.95 §3.2. The training path uses the trainer's optimizer built from TrainingArguments, where beta2 defaults to 0.98 and the scripts do not override it train_vla.py qwen2_vla_trainer.py. The 0.95 in configure_optimizers [code: policy_heads/models/transformer_diffusion/modeling_scaledp.py#L370] is not reached from the trainer path that I traced.
• Stage 2 freezing. The paper freezes the VLM's visual encoder in stage 2 §3.2. The stage-2 script sets freeze_vision_tower False [code: scripts/train_dexvla_stage2.sh#L32]. The model loader then sets the vision tower trainable model_load_utils.py.
• Training length. The paper gives 5 epochs per stage §3.2. The code trains for a step count: 100000 steps in stage 2 [code: scripts/train_dexvla_stage2.sh#L39] and 80000 in stage 3 [code: scripts/train_dexvla_stage3.sh#L41]. The dataset size in steps is not reported, so the epochs cannot be recovered.
• Projection. The paper says two linear layers with LayerNorm §3.1. The code pools the token sequence to one vector, then applies Linear, GELU, Linear, and an output dropout of 0. The LayerNorm line is commented out [code: qwen2_vla/utils/fusion_modules.py#L3-L20].
• FiLM. The paper says FiLM scales and shifts the projection layers' parameters §3.1. The code applies FiLM to the projected activations, with scale and shift both zero-initialised. It then adds the mean hidden state of the prompt segment fusion_modules.py [code: qwen2_vla/models/modeling_qwen2_vla.py#L1867-L1894]. The code uses the prompt segment (labels equal to -100) as the action input and the supervised reasoning segment as the FiLM condition.
• Expert conditioning. The expert receives one condition vector per example plus the robot state, passed through a three-layer MLP [code: policy_heads/models/transformer_diffusion/modeling_scaledp.py#L222-L228] modeling_scaledp.py. It does not attend to VLM tokens directly.
• Loss combination. The code's total loss is the diffusion loss plus llm_loss_weight times the cross-entropy, with the default weight 1.0 modeling_qwen2_vla.py [code: train_vla.py#L88]. This matches the paper's alpha = 1.
• Diffusion details. The code uses a DDIM scheduler with 100 training steps, squared-cosine betas, epsilon prediction and clipped samples [code: policy_heads/models/transformer_diffusion/modeling_scaledp.py#L262-L269]. It samples one noise level per example and computes a squared-error loss masked by padding modeling_scaledp.py. The mean divides by all elements, not by the number of unpadded ones. The paper does not state any of these.
• Action chunk. The stage-2 and stage-3 scripts set chunk_size 50 [code: scripts/train_dexvla_stage2.sh#L21] train_dexvla_stage3.sh. The expert's own default is 16 [code: policy_heads/models/transformer_diffusion/configuration_scaledp.py#L26], and the DiVLA script uses 16 train_divla.sh.
• Stage 3 LoRA flag. The stage-3 script sets lora_module "vit llm" but lora_enable False, so no LoRA adapters are created [code: scripts/train_dexvla_stage3.sh#L16] train_dexvla_stage3.sh. Stage 3 is full fine-tuning.
• Optimizer epsilon and warmup. Adam epsilon 1e-7 train_vla.py. The paper does not state warmup. The scripts use 1% in stage 2 and 0.1% in stage 3 train_dexvla_stage2.sh [code: scripts/train_dexvla_stage3.sh#L49].
• Action and state normalisation. Actions are min-max scaled to [-1, 1] with a small margin; robot state is z-scored, with the standard deviation clipped at 1e-2 utils.py [code: data_utils/utils.py#L216-L230]. The paper does not state these choices.
• Evaluation. The eval loop re-plans every 16 control steps and keeps the first 16 actions of each chunk smart_eval_agilex.py smart_eval_agilex.py. The paper does not describe the deployment loop.

Pitfalls

• Code licence. The repository's LICENSE is a CC BY-NC style licence (the summary names Attribution and NonCommercial, with no version). Using the code commercially is not permitted by that licence. This card cites the code by path and line for facts only and quotes none of it. Anyone implementing from this card should work from the paper and from this card's description, not copy the repository.
• Stage 1 is unverifiable from code. The ScaleDP-H checkpoint is the only released stage-1 artefact. Its training recipe (data, ResNet-50, DistilBERT, stage-1 learning rate) is documented in the paper alone.
• Abstract versions differ. The arXiv metadata abstract (version 3) names Octo, OpenVLA and Diffusion Policy as baselines. The LaTeX source fetched for this card names OpenVLA and pi0. This card follows the LaTeX source.
• Sub-step labels are not released. Stage-2 and stage-3 sub-step labels came from Gemini 2.0 segmentation and from a hand-written sub-step list app. C.2. The labels are not in the repository, and the paper gives no sensitivity analysis for annotation quality. Expect reproduction cost here.
• Sub-step ablation table is ambiguous. The text says direct prompting for the expert gives 0.07 and direct prompting in both stages gives 0. The table has checkmark rows for stage 2 only (0.07) and stage 1 only (0), and no row for "both direct". Read these three numbers with caution app. D.1.
• Small trial counts. Results use 10 trials per task (3 for visual generalisation). The paper gives no variance, and per-task differences of a few trials are within noise.
• Baseline comparability. The paper says baselines were fine-tuned on the same data and for the same number of epochs as Stage 2 §4.1. It does not say the same for LIBERO baselines, and pi0 is not in the Stage 2 comparison.
• Speed claim. The introduction says the model runs at 60 Hz on one A6000 §1. The paper does not say what was timed, and the eval loop re-plans every 16 control steps.
• Stage 3 is needed for laundry folding. Without stage 3 the laundry score drops from 0.4 to 0 §4.4. The paper's text on stage 3 reducing cross-task generalisation sits in a commented-out section of the source, so this card does not rely on it.
• Unexplained no-stage-1 failure. Training without stage 1 gives zero success. The paper's explanation (a large expert is hard to optimise, and stage 1 "warms up" its parameters) is a hypothesis, not a tested result §4.4.
• Zero-shot transfer is limited. Gripper-to-hand transfer loses 7 points (60% against 67%), and the authors attribute the gap to appearance, camera pose and grasp height app. A.5.

Lineage

• Builds on: Qwen2-VL, arXiv:2409.12191, as the VLM backbone §3.1; ScaleDP, arXiv:2409.14411, as the transformer diffusion policy head §3.1; Diffusion Policy §2; Octo, for multi-head output §3.1.
• Related: TinyVLA, arXiv:2409.12514, a diffusion VLA with a smaller action head; DiffusionVLA and pi0 §2.
• Compared against: OpenVLA, Octo, Diffusion Policy §4.1; pi0 with direct prompting §4.3; pi0-FAST on LIBERO app. A.2. SayCan is compared as an alternative sub-step source app. D.2.
• Superseded by: not stated.