OpenVLA: An Open-Source Vision-Language-Action Model

In plain words

A robot arm can be taught a task by copying examples, but each new task or robot usually needs its own model. OpenVLA is one large model meant to be a starting point for many robots. You give it a camera image and a sentence such as "put the carrot in the bowl", and it outputs the motor commands for the next moment.

It is built by taking a model that already understands pictures and text and training it further on about 970k recorded robot demonstrations from many different robots. The robot commands are written as word-like tokens, so the model predicts them the same way it predicts words. The authors release the weights and the training code, and they show that the model can be adapted to a new robot with a small, cheap fine-tuning step that runs on a single GPU.

TL;DR

OpenVLA is a 7B-parameter open vision-language-action model trained on 970k Open X-Embodiment episodes §1 §3.3. The authors report that it beats the closed 55B RT-2-X on two robot platforms §5.1 and that LoRA fine-tuning on a new robot matches full fine-tuning §5.3. Use the card when you need to implement the model, its action tokenizer, or its fine-tuning recipe. Check the Paper vs code section before copying any number: the released mixture weights differ from the paper's table.

Problem

Robot policies trained on task-specific data generalise poorly to new distractor objects and to new instructions §1. Large vision-language models carry broad knowledge, but the authors note that VLAs (such as RT-2) were closed, and that no best-practice guidance existed for adapting them to new robots or tasks, especially on consumer hardware §1.

Method

Architecture. The model has three parts: a vision encoder, a projector and an LLM backbone §3.1 fig. 2. The vision encoder concatenates DINOv2 and SigLIP features channel-wise, so each image patch gets both feature sets. The encoder is about 600M parameters, the projector is a two-layer MLP, and the backbone is Llama 2 7B. The model starts from the Prismatic-7B VLM §3.1.

Action tokens. Each of the 7 action dimensions is discretized separately into one of 256 bins. The bin width divides the interval between the 1st and 99th percentile of that dimension in the training data, which ignores outliers §3.2. The 256 action tokens overwrite the 256 least-used tokens of the Llama tokenizer, which are the last 256 vocabulary entries §3.2.

Training objective. The model is trained with next-token prediction, and the cross-entropy loss is computed only on the action tokens §3.2. The authors fine-tune all weights, including the vision encoder, because freezing it hurt VLA performance in their runs §3.4.

Data. The authors keep only manipulation datasets with at least one third-person camera and single-arm end-effector control, then reuse the Octo mixture weights §3.3. DROID is added at a 10% weight, but it is removed for the final third of training because its action-token accuracy stayed low §3.3 app. A. The full list of datasets and weights is in app. A.

Training recipe. Pretraining uses a fixed learning rate of 2e-5 without warmup, a global batch of 2048, and 27 epochs over the mixture §3.4 §3.5. Training ran on 64 A100 GPUs for 14 days, or 21,500 A100-hours §3.5.

Inference. The model produces 7 action tokens, decodes them to continuous actions, and un-normalises them. Single-image input only; no proprioception and no observation history §6. Inference in bfloat16 needs about 15 GB of GPU memory and runs at about 6 Hz on one RTX 4090 without compilation or speculative decoding §3.5. A remote inference server is provided §3.5.

Fine-tuning. Full fine-tuning on 10 to 150 demonstrations per task §5.2. LoRA with rank 32 on all linear layers matches full fine-tuning on the Franka-Tabletop tasks while training 1.4% of the parameters §5.3. With LoRA, a new task takes 10 to 15 hours on a single A100 §5.3.

Pseudocode: action tokenization (paper §3.2; code action_tokenizer.py).

# Normalise one action dimension to [-1, 1] with its 1st and 99th percentiles
# [§3.2] [code: prismatic/vla/datasets/rlds/utils/data_utils.py#L76-L88]
def normalise(action, low, high):
    scaled = 2 * (action - low) / (high - low + 1e-8) - 1
    return clip(scaled, -1, 1)                  # (7,)

# Digitize into 256 uniform edges over [-1, 1]; ids run from 1 to 256
# [§3.2] [code: prismatic/vla/action_tokenizer.py#L31-L41]
def action_to_token_ids(normalised, edges, vocab_size):
    bin_index = digitize(normalised, edges)     # (7,), values in 1..256
    # Use the last 256 vocabulary ids, which the tokenizer overwrites [§3.2]
    return vocab_size - bin_index               # (7,)

Pseudocode: decoding at inference (code action_tokenizer.py and openvla.py).

# Map predicted token ids back to bin centres (one per dimension)
# [code: prismatic/vla/action_tokenizer.py#L32, #L66]
def token_ids_to_normalised(token_ids, vocab_size, centres):
    index = clip(vocab_size - token_ids - 1, 0, len(centres) - 1)
    return centres[index]                       # (7,)

# Un-normalise with the dataset statistics for the chosen embodiment
# [code: prismatic/models/vlas/openvla.py#L94-L101]
def decode_action(normalised, low, high):
    return 0.5 * (normalised + 1) * (high - low) + low   # (7,)

Pseudocode: training loss (paper §3.2; code datasets.py).

# Next-token cross-entropy over the sequence; non-action positions are masked
# [§3.2] [code: prismatic/vla/datasets/datasets.py#L63]
def training_loss(model, images, prompt_ids, action_token_ids):
    logits = model(images, prompt_ids, action_token_ids)   # (B, T, vocab)
    # Only action-token positions contribute to the loss [§3.2]
    return cross_entropy(logits, targets=action_token_ids, mask=is_action)

Hyperparameters

Results

Author-reported. Setups: BridgeData V2 WidowX, 17 tasks with 10 rollouts each (170 rollouts) §5.1 app. B.1; Google robot, 12 tasks with 5 rollouts each (60 rollouts) §5.1 app. B.2. Values are mean success rate ± standard error.

The paper's headline gain is 16.5 points over RT-2-X on 29 tasks §1. Our derivation from the appendix tables reproduces that figure only when rollouts are pooled. See Pitfalls.

Parameter-efficient fine-tuning (two Franka-Tabletop tasks, 33 rollouts per approach in the summary table) §5.3 app. B.3.2:

Quantized inference (eight BridgeData V2 tasks, 80 rollouts per precision) §5.4:

With blocking control, the same precisions score 70.0%, 74.4% and 68.8% (bfloat16, int8, int4), which the authors read as showing that the int8 drop came from speed, not from precision app. D.4.

LIBERO simulation (fine-tuned via LoRA r=32; SR and rank across four suites, three seeds, 500 trials each) app. E.2:

Ablations on the 8-task BridgeData V2 subset app. D.1 app. D.2: OpenVLA 76.3%; OpenVLA trained only on BridgeData V2 45.6%; the same without the DINOv2 encoder 40.6%. The authors report that OpenX pretraining matters more than the fused encoder.

Frozen vs fine-tuned vision encoder (two Prismatic backbones, small task set, different evaluation configuration) app. D.3: fine-tuning gave 95% vs 85% (SigLIP backbone) and 90% vs 25% (LLaVA backbone) on the two first tasks. The authors report 80.0% vs 46.7% averaged over the evaluations where both were run.

Paper vs code

The official repository is github.com/openvla/openvla at the commit above. The project page linked from the title page points to it LICENSE. The repository carries an MIT license with copyright held by three of the authors.

• Data mixture. The paper's table gives percentage shares app. A. The released final mixture oxe_magic_soup_plus_minus uses unnormalised sampling weights that differ from those shares, for example BC-Z 7.5% in the paper and weight 0.2 in code, NYU Franka 0.8% vs 3.0, UCSD Kitchen under 0.1% vs 2.0, and Furniture Bench 2.4% vs 0.1 mixtures.py. Language Table and DROID are commented out of this mixture mixtures.py mixtures.py. The config comment describes two stages: a mix that includes DROID for the first 70% of training, followed by resampling vla.py.
• Bin decoding. The paper describes 256 bins §3.2. The tokenizer builds 256 edges, which give 255 bin centres, and decodes to those centres action_tokenizer.py action_tokenizer.py. The paper does not say whether decoding uses centres or edges.
• Normalisation. The paper uses 1st and 99th quantiles §3.2. The OpenVLA RLDS path uses the BOUNDS_Q99 scheme datasets.py, which clips to [-1, 1] data_utils.py. Clipping is code-only; the paper does not state it.
• Fine-tuning learning rate. The paper states 2e-5 for OpenVLA pretraining §3.4 and gives no fine-tuning learning rate. The fine-tuning script defaults to 5e-4 finetune.py.
• LoRA alpha. Not stated in the paper. The code sets alpha to min(rank, 16) finetune.py.
• Epoch count. The paper says 27 epochs §3.4. The base config sets epochs=1000 with max_steps unset vla.py. The final run's step count is not stated. The TeX source defines a "150k" iteration macro that the text does not use.
• Quantized training. The paper evaluates quantized inference only. The code has a 4-bit NF4 fine-tuning option, marked "CAUTION: Reduces memory but hurts performance" finetune.py finetune.py.
• Image augmentation. The paper does not mention it. The fine-tuning default is on finetune.py and pretraining default is off train.py.
• Action-token range. The tokenizer assumes the action tokens are the last 256 vocabulary entries, and computes action_token_begin_idx from that assumption action_tokenizer.py. It must be checked for any other tokenizer.

Pitfalls

• Headline arithmetic. The 16.5-point gain is reproduced only with pooled rollouts (74.3% vs 57.8% over 230 rollouts). Averaging the per-task success rates from the same tables gives 76.6% vs 62.1%, a 14.5-point gap (our derivation). Report which one you use.
• Diffusion Policy gap. The 20.4-point figure in the abstract §1 could not be reproduced by simple averages of the appendix values: 18.7 points on Franka-Tabletop and 23.3 on Franka-DROID app. B.3.2. The paper does not say how it was computed.
• Small trial counts. BridgeData tasks have 10 rollouts, Google tasks 5, and Franka tasks 5 to 12 §5.1 app. B.3.1. Most per-task differences are one to three successes. Standard errors are given, but many comparisons overlap.
• RT-2-X is a special case. RT-2-X is closed and was queried with its second-most-likely action to avoid a freezing problem, so the comparison depends on that workaround app. C. Dynamic re-querying performed worse in the authors' tests app. C.
• Zero actions. Many BridgeData demonstrations start with an all-zero action, and a policy trained on them can freeze. The authors drop the first transition of each demonstration app. C. For LIBERO they drop no-op actions, and they rotate third-person images 180 degrees for their hardware app. E.1.
• Blocking control matters for quantization. int8 inference is 1.2 Hz on the A5000, which changes the controller dynamics, so int8 looks worse in non-blocking runs §5.4 app. D.4. Measure control frequency before judging a precision.
• Different checkpoint for PEFT and quantization. The PEFT and quantization experiments use a SigLIP-only checkpoint trained on the Octo mixture [§5.3 footnote]. Their numbers are not directly comparable to the main results.
• Single image, no proprioception. The model takes one image and outputs one action per step, with no history and no action chunking §6. The authors report under 90% success on their tasks §6. They say faster inference is needed before ALOHA-rate (50 Hz) control and more dexterous bimanual tasks can be tested §6.
• Licences beyond this card. The card is CC BY 4.0, which covers the paper. The code is MIT. The authors state that the training data and code of the three base components are not open, while the weights are §1 footnote. Check the dataset and Llama 2 terms before commercial use.
• Figure and table numbers. Figure numbers here follow source order and were not checked against the compiled PDF. Where a number disagrees, use the section.

Lineage

• Builds on: the Prismatic-7B VLM and its fused DINOv2+SigLIP encoder §3.1; RT-2's approach of predicting actions as tokens §3.2; the Open X-Embodiment dataset §3.3; the Octo mixture weights §3.3; LoRA for fine-tuning §5.3.
• Compared against: RT-1-X, RT-2-X and Octo on BridgeData V2 and Google robot tasks §5.1; Diffusion Policy, with and without matched inputs, and Octo for fine-tuning §5.2; LIBERO baselines app. E.2.
• Superseded by: not stated on this card.