OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

In plain words

A robot that follows a spoken or written instruction has to find the right object in its camera image and then move its arm. Most recent approaches fine-tune a large vision-language model on robot data, which tends to blur the general image-and-text understanding that made the model useful in the first place. The robot then does well on the scenes it was trained on and poorly on new objects.

OTTER keeps the vision-language model frozen. Its trick is to look at the image through the instruction: for each word in the instruction, it asks which parts of the image match that word, and it keeps only those parts. The selection needs no extra trained weights. The result is a short list of instruction-relevant visual tokens, which a small trained transformer turns into arm motions.

The point is generalization. In the authors' tests, the same frozen model handled objects and scenes it had never seen in robot training better than fine-tuned alternatives.

TL;DR

OTTER is a vision-language-action policy built on a frozen CLIP ViT-L/14. The last attention layer's patch features are softmax-weighted by their similarity to each instruction token, with a learnable temperature. The selected features, a pooled instruction vector and a proprioception token go into a 4-to-8-layer causal transformer that predicts chunks of 10-D delta end-effector actions. Author-reported gains are largest on unseen objects and scenes. The released code is a PyTorch re-implementation, and its defaults differ from the paper in several places (see Pitfalls).

Problem

Fine-tuning a pretrained vision-language model on robot data can overfit the robot data and erode the model's semantic alignment between images and text. Models such as CLIP are known to lose zero-shot and long-tail accuracy under this kind of fine-tuning §1. Direct feature passing, where all image tokens and text tokens go into the policy separately, also depends on learning the image-text connection from a small robot dataset §1 §5.1. The paper asks whether the existing alignment can be used directly, without weakening it §1. The experiments use a Franka arm on real tasks and the LIBERO simulator §4.1.

Method

Overview §3. OTTER has two stages. A frozen CLIP encoder gives text word features and image patch features at each time step. A learned head then combines them into one token per time step for a policy transformer. Only the pooling layers, proprioception encoder, transformer and action head are trained; the CLIP weights stay frozen §3.1 §3.2.

Visual features from the last attention layer §3.1 eq. 1 eq. 2. The paper uses CLIP's last vision attention output X_attn, not the block output X_out. ClearCLIP reports that X_attn carries cleaner text-aligned semantics than X_out, and the appendix argues that the residual, the FFN and the final layer norm make X_out noisier app. C fig. atten.

Text-aware selection §3.1 eq. 4. After normalization, the paper takes the cosine-style similarity between every text token and every patch, applies a softmax over patches with temperature τ, adds a 2D sin-cos position embedding to the patches, and takes the weighted sum. Each text token therefore gets one visual vector. The softmax acts as the selection step: a patch that matches a word gets most of that word's weight §3.1 fig. fusion.

Pooling and token assembly §3.2. Each camera's text-aware tokens are pooled to a single vector with a learned cross-attention block that has 4 query readouts, 8 heads and 2 blocks. The instruction words get their own learned pooling, which gives one text vector that is constant over the time steps. The robot's 10-D proprioception goes through an MLP. The three parts are concatenated into one token per time step §3.2 fig. model.

Policy and action head §3.2 tab. model-arch. A causal transformer runs over the sequence of per-step tokens. For each output token, an MLP predicts the next H actions at once, where H is the action horizon. The paper uses a context length of 12 steps for real robots and 4 for simulation [tab. model-arch].

Action parameterization §3.2. Actions are delta end-effector poses. For each future absolute transform T_i, the model predicts the relative transform from the current end-effector pose, with the rotation written as the first two rows of the rotation matrix (6 numbers). The absolute gripper value is appended, giving 10 numbers per step. The paper says it uses temporal ensembling together with receding-horizon control at execution time, and that an action horizon of 8 steps gave the best results §3.2.

Training objective. The paper does not name a loss function §3.2 §4.1. See Pitfalls for what the code does.

Pseudocode 1 — text-aware tokens for one camera. This is the core operation, following eq. 4.

# Text-aware visual tokens for one camera  [§3.1] [eq. 4]
# Input:  patches (B, n, d_v): projected last-layer attention, n = h * w
# Input:  words (B, m, d_t): projected CLIP text features, m = first_k_tokens
# Input:  tau: learnable temperature (scalar)
# Input:  pos_emb (n, d_v): fixed 2D sin-cos position embedding
# Output: aware (B, m, d_v)

def text_aware_tokens(patches, words, tau, pos_emb):
    # L2-normalize both sides, as in CLIP  [§3.1]
    words = l2_normalize(words)                       # (B, m, d_t)
    patches = l2_normalize(patches)                   # (B, n, d_v)
    # Similarity of each word with each patch  [§3.1] [eq. 4]
    scores = words @ transpose(patches)               # (B, m, n)
    # Softmax over patches: the selection step  [§3.1] [eq. 4]
    weights = softmax(scores / clamp(tau, 0, 100), axis=-1)
    # Add the position embedding to every patch  [eq. 4]
    patches = patches + pos_emb                       # (B, n, d_v)
    # Each word takes a weighted average of the patches  [eq. 4]
    return weights @ patches                          # (B, m, d_v)

Pseudocode 2 — one time-step token. This assembles the inputs of the transformer, following §3.2.

# One token per time step from images, text and proprioception  [§3.2]
# Input:  images: one image per camera (cameras = 2 in code)
# Input:  text: instruction token ids
# Input:  proprio (B, 10): end-effector translation, 6-D rotation, gripper
# Output: token (B, 128 + 512 + 64), before the input projection

def observation_token(images, text, proprio):
    # Frozen CLIP gives word features and patch features  [§3.1]
    words, patches = frozen_clip(images, text)
    # Per camera: select visual tokens, then pool to one vector  [§3.2]
    visual = []
    for camera in cameras:
        aware = text_aware_tokens(patches[camera], words,
                                  tau[camera], pos_emb)
        visual.append(attention_pool(aware, readouts=4, heads=8, blocks=2))
    # Pool the instruction words to one vector  [§3.2]
    text_vector = attention_pool(words, readouts=4, heads=8, blocks=2)
    # Encode proprioception with an MLP  [§3.2]
    embodiment = proprio_mlp(proprio)
    # Concatenate into one token per time step  [§3.2]
    return concat(text_vector, *visual, embodiment)

Pseudocode 3 — policy and action head. The transformer and head follow §3.2 and tab. model-arch. The position embedding is in the code only; the paper does not describe it.

# Causal policy over a context window  [§3.2] [tab. model-arch]
# Input:  tokens (B, T, d): one token per time step, T = context length
# Output: actions (B, T, H, 10): H = action horizon

def policy(tokens):
    # Sinusoidal position embedding per step (code only)  [code: otter.py#L281]
    x = input_projection(tokens) + sinusoidal_pe(T)   # (B, T, d_model)
    # Causal transformer; the table gives 4 blocks, 512 wide, 8 heads
    for block in causal_blocks:
        x = block(x, causal=True)                     # (B, T, d_model)
    # An MLP maps each step to the whole chunk of H actions  [§3.2]
    actions = action_mlp(x)                           # (B, T, H * 10)
    return reshape(actions, (B, T, H, 10))

Pseudocode 4 — delta action targets. This follows §3.2 and matches the data code for the DROID-style layout dataset.py.

# Delta action targets for one chunk  [§3.2]
# Input:  pose_now (4, 4): current end-effector transform
# Input:  poses_future (H, 4, 4): absolute future transforms T_1 .. T_H
# Input:  gripper_future (H,): absolute gripper value per step
# Output: actions (H, 10)

def delta_action_chunk(pose_now, poses_future, gripper_future):
    # Relative transforms from the current end effector  [§3.2]
    relative = inverse(pose_now) @ poses_future       # (H, 4, 4)
    # Translation, then the first two rows of R flattened (6-D)  [§3.2]
    translation = relative[:, :3, 3]                  # (H, 3)
    rotation = flatten(relative[:, :2, :3])           # (H, 6)
    # Append the absolute gripper value  [§3.2]
    return concat(translation, rotation, gripper_future[:, None])

Inference-time execution §3.2 droid_inference.py. The paper names temporal ensembling plus receding-horizon control, and it gives no ensembling weights. The released DROID script uses receding-horizon control by default. It executes the first action_exec_horizon steps of each chunk, defaulting to 12 droid_inference.py. The temporal ensemble in the same script averages overlapping predictions with weights that grow as exp(0.05 · i) droid_inference.py.

Training regime app. B.2 tab. training_details. All models use a batch of 64, AdamW at 3e-4 with 2000 warmup steps and cosine decay, and images at 224 × 224. The table says these values are shared by real and simulation runs. Augmentation uses brightness, contrast, saturation, hue and a random resize ratio. Models are trained on 4 A100 80GB GPUs app. B.2.

Hyperparameters

Values from the paper unless the row says "code" or "code only". "Not stated" means the paper does not give the value.

Architecture tab. model-arch, with the larger real-robot values in parentheses where the paper gives them.

Training tab. training_details, shared by real and simulation.

Baselines §4.2 app. B.1. Octo uses the public small 1.5 checkpoint app. B.1. OpenVLA predicts 16 actions per step (action chunking added by the authors) app. B.1. Octo's context length is raised to 10 for the multi-primitive test §5.1. The simulation baseline numbers for Octo and OpenVLA are copied from the OpenVLA paper, not re-run tab. sim_results.

Results

Author-reported. Real-robot setup: a Franka arm, 19 in-distribution and 15 unseen training tasks across four primitives, 10 trials per task §4.1. Simulation: LIBERO-Spatial, -Object and -Goal, plus 10 unseen tasks built from LIBERO-90, 50 trials per task §4.1 tab. sim_results. Errors are standard errors across trials §4.1. The number of training seeds is not stated.

Single primitive, real robot §5.1 tab. real_exp. Pick and place only. In-distribution: 100 trials. Unseen: 70 trials.

Source: tab. real_exp. The authors attribute π0-Fast-Droid's 61% to its Droid training, which is mostly pick-and-place §5.1.

Multiple primitives, real robot, unseen tasks §5.1 tab. real_all_exp. 150 trials over 15 unseen tasks.

The paper reports these means as given; see Pitfalls on how they relate to the columns.

Simulation, LIBERO §5.2 tab. sim_results. 30 in-distribution tasks and 10 unseen tasks.

* Numbers copied from the OpenVLA paper tab. sim_results.

Ablations, real robot §5.3 tab. real_exp app. D tab. real_abl_exp. Relative to full OTTER (68% training, 62% unseen): removing f_e costs 28 and 33 points; removing f'_l costs about 10 points on both sets §5.3. Removing the CLIP vision encoder costs about 51 points on both sets (17% and 11%) §5.3 tab. real_exp. Fine-tuning CLIP lowers unseen success from 62% to 15% §5.3 app. C.

On 70 unseen pick-and-place trials tab. real_abl_exp: DFP-OTTER (CLS) 6%, OTTER with cross-attention instead of text-aware selection 2%, OTTER 62%.

Ablations, simulation §5.3 tab. sim_results_ablation.

Pretraining on human videos §5.4.1 tab. real_exp_human. DexYCB and HOI4D give 15.6k human hand trajectories, relabelled with instructions app. C.1. Pretraining on them and then fine-tuning on robot data:

Scaling §5.4 fig. clip_flops. The paper scales the CLIP encoder across ViT-B/32, ViT-B/16 and ViT-L/14 and plots task success against inference FLOPs. The text gives +27.5 points on training and +39.3 points on unseen pick-and-place tasks from ViT-B/32 to ViT-L/14 §5.4. These numbers are only in the text, not in a table.

Limitations §6. The method assumes the robot can be described by SE(3) transforms, so multi-finger hands are excluded. Long-horizon tasks and complex scenes are not studied §6.

Paper vs code

Code: github.com/Max-Fu/otter at 4be49c4d289dec5fe1350c976c212dddf3470596, Apache-2.0 (LICENSE.txt). The project page (ottervla.github.io) links this repository as the PyTorch code. The owner is Letian Fu, a listed author, per the README's author link readme.md.

The project page also links a JAX repository, github.com/FangchenLiu/otter_jax, owned by author Fangchen Liu. I did not clone it. The PyTorch README says the paper's experiments were run in that JAX repository and that this code is a re-implementation readme.md. The paper's numbers may therefore not reproduce from this code.

I read the model, the data transforms, the training script and the DROID inference script. Findings:

• Matches the paper. The last-block attention output is hooked as the visual feature otter.py. The CLS token is removed, then ln_post, the projection and an L2 normalization are applied otter.py. The text side applies ln_final, the projection and an L2 normalization otter.py. The selection is a softmax over patches with a clamped temperature, and the position embedding is added to the patches before the weighted sum models.py.
• Matches the paper. The pooling uses 4 readouts, 8 heads and 2 blocks args.py transformer.py. The delta actions are relative transforms from the current end-effector pose, with the 6-D rotation taken from the first two rows dataset.py.
• Differs: the released default is the larger model. The defaults are 8 blocks and width 768 args.py, the parenthesized values in the paper's table. The paper's main 4-block, 512-wide setting is not the default.
• Differs: default action horizon and execution. The default horizon is 12 args.py. The DROID script executes 12 steps with receding-horizon control on by default droid_inference.py. The paper's text says 8 steps is best §3.2.
• Differs: training steps. The paper's table gives 40000 steps, or 60000 in parentheses app. B.2. The default is 80000 args.py. The README example loads checkpoint 60000 readme.md.
• Differs: no gradient clipping. The paper lists a clip threshold of 1 app. B.2. The training loop calls the scaler with no clip value, so no clipping happens engine.py misc.py.
• Differs: CLIP caching. The paper says CLIP features are cached at inference, which allows 50 Hz on one GPU app. B.1. The code runs both CLIP encoders on every call otter.py. The cache it keeps holds transformer inputs, not CLIP features otter.py.
• Differs: loss. The paper does not name a loss. The forward pass uses an L1 loss unless use_mse is set otter.py. The constructor reads shared_config.mse_loss otter.py, but SharedConfig has no such field args.py. Reading the code, constructing the model with this config should raise AttributeError. I did not run it.
• Differs: text-token masking is off by default. pool_true_text defaults to False args.py, so the pooling sees all 15 text tokens, including start, end and padding tokens. When it is on, those tokens are masked otter.py.
• Differs: temperature initialization. The paper gives no start value. The code starts τ at 0.07 models.py. Each camera has its own τ otter.py.
• Not in the code. The random resized crop in the paper's augmentation table was not found. The repo does a centre square crop and a bicubic resize obs_transforms.py.

Pitfalls

• The code as cloned does not construct the model. See the mse_loss finding above. Treat the loss as unknown until the field is fixed or the code is run. The paper does not name a loss §3.2.
• Action horizon is inconsistent. The table says 12 for real robots and 1 for simulation tab. model-arch. The text says an action horizon of 8 steps was best §3.2. The default is 12 args.py. The execution horizon is 12 in the DROID script droid_inference.py. Pick the value your experiment needs and state it.
• Model size is ambiguous. The architecture table lists two configurations in one row. The code uses the larger one. Check the parameter count of the model you build against the paper's 11.8M or 25.5M figure tab. model-arch.
• The 50 Hz claim needs CLIP caching. The code does not cache CLIP features otter.py. Expect a lower control rate unless you add that caching yourself.
• Gradient clipping is listed but not applied. Adding a clip at 1 will change optimization compared with the released loop.
• Only τ is learnable §3.1. That sentence refers to the CLIP part. The pooling, proprioception MLP, transformer and action head are all trained.
• Text-token mask. Whether start, end and padding tokens enter the pooling is not stated. The code default includes them args.py.
• Temperature. Only the clamp to [0, 100] is stated §3.1. The start value and whether τ is shared across cameras are not. The code uses a per-camera τ starting at 0.07 models.py.
• Mean column in the multi-primitive table. The mean is not the average of the four columns. For OTTER-OXE-L the columns average 80% and the mean reads 77%. For OTTER-L they average 69.75% and the mean reads 71% tab. real_all_exp. The paper does not say how the mean is computed.
• Partial credit in pick-and-place. Each trial scores 0.5 for picking the right object and 1 for placing it, with 0 otherwise app. A.2. Timeouts are 30 s for most models and 60 s for OpenVLA app. A.2. The tables do not say whether partial credit is included in the reported success rates.
• "More than 50% worse" is a relative figure. Without the CLIP vision encoder, unseen success falls from 61% to 29% in simulation, which is 32 points or about 52% relative §5.3 tab. sim_results_ablation.
• Trial counts disagree. The simulation table caption says 50 trials per unseen task tab. sim_results. The simulation ablation table says 100 trials on the unseen tasks tab. sim_results_ablation. The 61% unseen figure appears in both.
• Encoder size is inconsistent. The real-robot model is ViT-L/14 tab. model-arch. The scaling figure caption says ViT-L/16 fig. clip_flops. The text says ViT-L/14 §5.4. The simulation model is ViT-L/32 tab. model-arch.
• Sample sizes are small. Unseen real-robot results use 70 trials tab. real_exp. The error bars are standard errors across trials, not across training seeds §4.1.
• Baseline setup. Octo and OpenVLA in simulation come from the OpenVLA paper tab. sim_results. The OpenVLA baseline predicts 16 actions per step, a chunking the authors added app. B.1.
• Human-video pretraining has no wrist camera. The human data has a different camera angle, no wrist view and different proprioception frames §5.4.1. The gain is on unseen tasks only, and training-task success does not improve.
• Appendix caption copy error. The caption of the simulation task table refers to the real-world setting app. A.1.

Lineage

• Builds on: CLIP, for the frozen encoder and the text-image similarity §3.1 (no card here); ClearCLIP, for using the last attention output §3.1 (no card here).
• Compared against: Octo and OpenVLA, both fine-tuned on the same robot data §5.1. π0-Fast-Droid is compared as released §5.1; it has no card here. The FAST action tokenizer it uses is FAST §4.2.
• Superseded by: not stated in this source.