Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

In plain words

A robot policy that maps a camera image and an instruction straight to motor commands tends to react without planning. It does not say which step of the task it is on or where the gripper should go next. Emma-X is trained to write a short plan before it acts. For each part of a demonstration it writes the subtask being done, the reasoning behind it, where the gripper should be at the start of the next part, and a movement plan to get there. Only then does it output the motor commands.

The training labels are built from existing robot demonstrations. Each demonstration is cut into pieces where the arm's motion changes or the gripper opens or closes. A vision-language model then labels each piece while looking at its images, not only at a text description. The base model is OpenVLA, which is fine-tuned on these labelled examples.

The authors report better success on real-robot tasks that need spatial reasoning. The price is that the model writes far more text per step than OpenVLA, so it is slower.

TL;DR

Emma-X is a 7B vision-language-action model fine-tuned from OpenVLA on BridgeData V2, with demonstrations relabelled by Gemini 1.5 Pro §3.2. Before each action it writes a subtask, reasoning, a 2D gripper target and a 3D movement plan §3.3. Author-reported on 12 real WidowX tasks: 57.5% success and 71.66% half-success, against 33.33% and 45.41% for OpenVLA tab. 1. The official code is usable as a reference, but its plan generation, dataset builder and Bridge evaluation script differ from the paper in several places (see Paper vs code).

Problem

The authors argue that OpenVLA-style policies show "muscle memory" responses and lack long-horizon spatial reasoning about how the robot should move next §1. ECoT adds chain-of-thought reasoning to a VLA, but its subtask labels come from text-only prompts to Gemini, which the authors say hallucinate because the model cannot see the scene §1 §3.2. Per-frame labelling with Gemini was noisy in their experiments, so they label segments instead of single frames §3.1.

The policy is modelled as π(a | s, T): a state image s and an instruction T give a 7-dimensional action, which is the end-effector velocity in x, y, z, the orientation (roll, pitch, yaw) and the gripper open/close command §2.1. Chain- of-thought training needs every state labelled with its subtask §2.2.

Method

The method has a data-construction stage and a model stage. The paper describes the stages as segmentation, data generation and the model §3.

1. Segmentation §3.1. The end-effector trajectory is clustered with HDBSCAN using a custom distance that adds position, orientation and time eq. 1. A segment break happens where the cluster label changes or the gripper state changes between consecutive frames §3.1. A segment is a run of consecutive states, and a trajectory is a list of segments §2.2.
2. Segment labels §3.2. Gemini receives the images of each segment in order, numbered, together with the task instruction. For each segment it returns a subtask and a justification §3.2 app. B.
3. Gripper target §3.2. For frame t, the target gripper position g_t is the 2D gripper position at the first frame of the next segment. The 2D position is found with OWLv2 and SAM §3.2.
4. Movement plan §3.2 app. E. For frame t, the movement plan m_t is the displacement from frame t to the first frame of the next segment. It is written as text in a fixed template with a step count for each direction and a gripper word. The template follows the motion-language idea of RT-H §3.2.
5. Training sample §3.2. Each frame gives one sample. The input is the instruction, the current image and the current gripper position. The target contains the segment's reasoning and subtask, the target gripper position, the movement plan and the actions §3.2 §3.3.
6. Model §3.3. The model is a 7B VLA fine-tuned from OpenVLA on this data. It first outputs the subtask and a scene description, then the target 2D position and the 3D movement, and finally the 7-dimensional action §3.3. Actions are discretised into tokens as in OpenVLA §4.1.
7. Look-ahead §3.2. The model predicts the gripper position at the start of the next segment before it chooses the action for the current frame. The paper argues this lets the policy plan toward a future state and still react to the present one §3.2.

Algorithm 1 in the appendix gives the training and inference loops in the same order app. F. The pseudocode below follows the paper's text. Where the released code differs, see Paper vs code.

# Distance between frames i and j: position, plus lambda times orientation,
# plus beta times time gap [eq. 1] [§3.1 fn.: lambda = 1, beta = 0.03]
def distance(i, j):
    return (
        norm(position[i] - position[j])
        + lam * norm(orientation[i] - orientation[j])
        + beta * abs(time[i] - time[j])
    )

# Cluster the frames with HDBSCAN on that distance [§3.1]
clusters = hdbscan(frames, metric=distance)

# A segment break is a change of cluster label or of gripper state between
# consecutive frames [§3.1]
def is_breakpoint(t):
    changed_cluster = clusters[t] != clusters[t - 1]
    changed_gripper = gripper_open[t] != gripper_open[t - 1]
    return changed_cluster or changed_gripper

segments = split_where(is_breakpoint, frames)

# For each segment k, Gemini labels its subtask and reasons from the segment
# images and the instruction [§3.2] [app. B]
subtask[k], reasoning[k] = gemini(segment_images[k], instruction)

# For each frame t: 2D gripper position at the first frame of the next segment
# [§3.2]
next_gripper[t] = gripper_2d(first_frame(next_segment(t)))

# For each frame t: movement plan from frame t to that frame, in template
# text [§3.2] [app. E]
movement[t] = describe_motion(pose[next_start(t)] - pose[t])

# Target text for frame t, in the order given in [§3.3] and [app. F]
def training_target(t):
    label = segment_label[segment_of(t)]
    text = label.reasoning + label.subtask
    text += next_gripper[t]
    text += movement[t]
    # 7-D action, each dimension tokenised into one of 256 bins [§4.1]
    text += tokenize(action[t])
    return text

# Inference loop, one action per model call [app. F]
def control_loop(instruction, env):
    while not task_done(env):
        image = env.observe()
        # 2D position of the gripper in the current image [§3.3]
        gripper = detect_gripper(image)
        # Reasoning, subtask, target position, movement, then action [§3.3]
        text = model.generate(instruction, image, gripper)
        action = parse_action(text)
        env.step(action)

Hyperparameters

Values marked "code" are not stated in the paper. They come from the released repository and are included because an implementation may need them. Check the code notes before relying on them.

Results

Author-reported. Setup: 12 real-world WidowX-250 manipulation tasks in four groups (spatial relation, OOD object, OOD instruction, in domain) §4.2 tab. 1. Baselines are OpenVLA and ECoT §4.3. Each task was run 10 times and scored as success or half-success §4.2. Averages are over the 12 tasks.

Table 1: main real-robot results tab. 1

The paper reports the gains as 24.17 points in success and 26.25 points in half-success over OpenVLA §4.4. Both differences match the averages in the table tab. 1. See Paper vs code for the ECoT average.

Figure 4: category averages of h_succ fig. 4 (values from the plot source). Spatial relation: ECoT 28, OpenVLA 36, Emma-X 71. OOD object: ECoT 38, OpenVLA 77, Emma-X 87. In domain: ECoT 15, OpenVLA 65, Emma-X 83. OOD instruction: ECoT 18, OpenVLA 13, Emma-X 50. The text says the largest gain is on spatial-relation tasks, 35% over OpenVLA and 29% over ECoT in h_succ §4.4.

Table 2: ablations on six prompts, 10 rollouts each tab. 2 §4.5. Values in brackets are the drops the paper prints in red.

In the table, "w/o m_t" drops the 3D movement plan, "w/o g_t" drops the next gripper position, "w/o GR_t" drops the grounded reasoning, and "w/o HDBSCAN" segments by gripper open and close only §4.5 tab. 2. "OpenVLA w/ FT" is OpenVLA fine-tuned on BridgeData V2 for the same number of epochs §4.3.

Table 2 uses six sampled prompts §4.5, not the 12 tasks of Table 1, so its full-model row is not the same evaluation as Table 1. Do not compare the two directly.

Paper vs code

The official repository is github.com/declare-lab/Emma-X at code_commit above. Its README links the arXiv paper and the project page README.md. The repository has no LICENSE file, so no licence is recorded and code is cited by path and line only. The repository also holds the Bridge evaluation script and the plan-generation pipeline.

• Segmentation method used for the release. The paper combines HDBSCAN with gripper breakpoints §3.1. The plan-generation script imports get_soft_segment, which clusters poses only and never reads the gripper generate_plans.py utils.py. The combined function exists but is not used utils.py. The paper does not say which segmentation produced the released dataset.
• Distance. The paper uses ||p|| + lambda ||r|| + beta |t| eq. 1. The code takes one Euclidean norm over the six pose values (position and three orientation terms together), and multiplies time by a weight of 1 applied to frame index divided by 30 utils.py. This is not the same distance, and the effective time weight is 1/30 per frame rather than 0.03.
• Gripper signal. The paper uses the gripper state §3.1. The gripper-aware code uses the commanded gripper action utils.py.
• Noise points. The paper does not say how HDBSCAN noise is handled. The code carries the previous frame's label forward utils.py.
• Movement plan. The paper describes a displacement for position and orientation §3.2. The code takes the position displacement, but takes the orientation at the next segment start as an absolute angle, not a change create_dataset.py. The paper does not state its units; the code multiplies position by 1000 and rounds to integer steps, and converts orientation to degrees utils.py.
• Gripper target at inference. The paper says OWLv2 and SAM §3.2 §3.3. The inference code uses OWL-ViT base (patch16), not OWLv2, then SAM. It takes the mask pixel that maximises a coordinate sum, with fixed offsets, not a centroid gripper_position.py gripper_position.py. The training gripper labels are not computed by this code. They are read from the ECoT embodied-features file create_dataset.py.
• Gripper in the eval prompt. The paper says the 2D gripper is an input at test time §3.3. The Bridge script detects it, but the prompt line that includes it is commented out, so the prompt contains only the instruction run_bridgev2_eval.py. The training template includes a "CURRENT GRIPPER" line create_dataset.py.
• Action chunk. The paper's Algorithm 1 takes one action per model call app. F. The training code makes the target all actions from frame t to the next segment start create_dataset.py, and the eval script executes that whole chunk before it queries the model again run_bridgev2_eval.py.
• Training mix. The README's training command uses the config moveonly-moveguided-single-policy README.md. That config's data mix, movement_policy, trains on the movement plan and actions only, with no reasoning, subtask or target gripper datasets.py vla.py. The full output order described in the paper matches the pred_all mix datasets.py vla.py. The code does not say which mix produced the released weights.
• Epochs. The paper says 3 epochs §4.1. The movement_policy config sets 200 vla.py.
• Gemini sampling. The paper names the model only §3.2 fn.. The code sets temperature 1.5, top-p 0.99 and top-k 0 generate_plans.py.
• Dataset size. The paper says about 60,000 trajectories §4.1. The code's dataset log lists 43,566 valid trajectories after plan validation create_dataset.py.
• Appendix segment statistics. The paper lists 6.9 segments per trajectory, 5.5 frames per segment and 32.8 frames per trajectory app. D. The code's log lists 4.7 samples per trajectory and 6.94 frames per sample for the multiple-policy build, 5.457 frames per sample for the augmented build, and 32.8 frames per trajectory create_dataset.py. The "6.9" in the paper matches the code's frames-per-sample figure, not its segments-per-trajectory figure. This is an inference from the labels, not a confirmed mislabel.
• Motion-plan decoder. The encoder multiplies position by 1000 and orientation by 180/pi utils.py. The decoder multiplies position by pi/180 and orientation by 1e-3 solver.py. The two do not invert each other, so a parsed movement plan is not in the training units. The decoder runs only for type == "pos" prismatic.py. The Bridge eval uses the action path, so the mismatch does not affect the robot run.

Pitfalls

• The symbol g_t means two things. In Section 3.2 it is the gripper position at the start of the next segment, and in Section 3.3 and Algorithm 1 it is the current gripper position §3.2 §3.3 app. F. The code takes the current position as the input and the next segment start as the target create_dataset.py create_dataset.py. Use the code's convention and name the two quantities differently in your own code.
• Labels are not reproducible. The subtask labels come from Gemini 1.5 Pro with temperature 1.5 and no seed generate_plans.py. The reasoning text is therefore not reproducible. The code's log shows 35,963 valid plans out of 38,660 training trajectories, so about 7% were dropped generate_plans.py.
• Actions per step. The training target is a variable-length chunk of actions, one chunk per segment or per frame depending on the build [code: GCOT/create_dataset.py#L163-L165]. Decide how many actions to run before you re-query, because the paper's loop re-queries each step and the eval re-queries per chunk.
• Gripper detector mismatch. Training gripper labels come from the ECoT file. Inference uses OWL-ViT plus SAM with a fixed-offset point. The paper reports no agreement between the two. The Limitations say SAM can fail when the gripper is occluded or out of frame [Limitations].
• Numerical sensitivity of the action decoding. The model's bins span [-1, 1] with 256 edges, so adjacent bins are 2/255 ≈ 0.0078 apart in normalised units action_tokenizer.py. Actions are mapped back with the 1st and 99th percentile statistics stored with the model prismatic.py. The paper does not state the percentile method.
• Units in the distance. Position and orientation share one norm in the code utils.py, so the scale of orientation relative to position changes the clusters. The paper does not state units for either.
• Possible segment merging in the dual variant. The dual code builds its key as cluster label × 100 plus the frame index of the last gripper change utils.py. Once that index reaches 100, two different (label, index) pairs can give the same key, so adjacent segments may merge. This does not affect the shipped pose-only path, which has no gripper term.
• Hidden cost. The model writes about ten times as many tokens per step as OpenVLA, which raises latency [Limitations]. The data pipeline needs a Gemini call per trajectory and a long run; the code notes about five hours for the plan step generate_plans.py. The download is about 124 GB for the BridgeData V2 RLDS set and 30 GB for the base checkpoint README.md.
• Ablations are small. Each ablation prompt has 10 rollouts, so one prompt moves in steps of 10 points, and the table averages six prompts §4.5. No confidence intervals are reported.
• Half-success is hand-written. The half-success criteria are a short description per task, not a measured quantity app. C.
• The ECoT baseline is not specified. The paper does not say whether the ECoT numbers come from the released checkpoint or a retrained model §4.3.
• Results are not fully consistent. The text says the spatial-relation gain over ECoT is 29%, but the Figure 4 value of 28 for ECoT and the Table 1 averages give a gap of about 43 points §4.4 fig. 4 tab. 1. The ECoT h_succ average printed in Table 1 is 28.75, while the mean of its 12 rows is 29.17 tab. 1. The success averages match.
• Attribution of gains is confounded. Emma-X differs from ECoT in the reasoning labels, the segmentation and the training data. The ablations in Table 2 do not include a text-only Gemini variant, so the effect of image-based labelling on its own is not isolated tab. 2.
• Output format is brittle. Parsing depends on the literal keys "NEXT GRIPPER:", "MOVEMENT:" and "POLICIES:" solver.py. A movement plan that fails to parse becomes a vector of -100 values solver.py. Handle the failure case.

Lineage

• Builds on: OpenVLA, which Emma-X fine-tunes §3.3 §4.1; the OpenVLA card. HDBSCAN for segmentation §3.1; Gemini 1.5 Pro for segment labels §3.2; OWLv2 and SAM for the gripper position, following ECoT §3.2; the motion-language idea of RT-H (arXiv:2403.01823) for the movement template §3.2.
• Compared against: ECoT (arXiv:2407.08693), reasoning VLA trained on BridgeData V2 §4.3; OpenVLA, with and without fine-tuning on BridgeData V2 §4.3.
• Superseded by: not stated on this card.