Open X-Embodiment: Robotic Learning Datasets and RT-X Models

In plain words

Robot learning has a data problem. Most robot policies are trained on one robot, in one lab, on one narrow set of tasks, so each new robot starts from scratch. Language models got better by training on large, varied collections of text. This paper asks whether robots can get the same benefit.

The authors gathered robot demonstration datasets from many labs and 22 robot types, and converted them into one shared format. They then trained two general policies on the combined data. Each policy looks at camera images and a written instruction, and outputs the next robot action. Because the action values are shared, one model can be used across several robots. The main finding is that a policy trained on the mixed data often does better on each robot's own tasks than a policy trained only on that robot's data. This is the "positive transfer" the paper reports.

TL;DR

The paper releases the Open X-Embodiment dataset: 60 existing robot datasets from 22 embodiments, converted to RLDS and described in §3. It also trains two policies, RT-1-X and RT-2-X, on a mixture of robot data §4.3. Author-reported results show positive transfer: RT-1-X beats each dataset's own baseline on most small-data domains, and RT-2-X gains on emergent skills §5. Read this card for the data format, the discrete action tokens and the places where the released code differs from the text.

Problem

Robot policies are usually trained for one robot, one environment and one task family, and the data is too narrow to pool across labs §1. The paper asks whether a policy trained on data from many embodiments can improve on policies trained only on one embodiment's data (positive transfer), and whether organising such data into one repository makes future research easier §1. The two goals are stated as (1) evaluate positive transfer and (2) organise the datasets §1. The paper does not claim to invent new architectures; it applies two existing Transformer policies with minimal change §1.

Method

Dataset (§3). The repository stores episodes in the RLDS format, which keeps the sensor and action layout of each robot §3.1. It pools 60 datasets from 34 research labs and covers 22 embodiments §3.1; the paper reports over one million trajectories §3. The abstract reports 527 skills and 160,266 tasks [abstract]; §3.2 uses PaLM to extract objects and behaviours from the instructions §3.2.

Data format consolidation (§4.1).

1. The policy input is a short history of images and a language instruction.
2. Each dataset contributes one canonical camera view, resized to a common resolution. The paper does not state that resolution §4.1.
3. Each dataset's actions are converted to a 7-DoF end-effector action (x, y, z, roll, pitch, yaw, gripper), normalised per dataset before discretisation §4.1.
4. Coordinate frames are not aligned. Actions may be absolute, relative or velocity commands, so one output vector can move different robots differently §4.1.

Two policies (§4.2).

• RT-1-X is RT-1 trained on the robotics mixture only. RT-1 is a 35M parameter network: a per-image EfficientNet with FiLM conditioning on a language embedding, followed by a decoder-only Transformer §4.2 fig. 2. The visual and language representations are fused into 81 vision-language tokens, and the history is 15 images §4.2.
• RT-2-X is RT-2 with a vision-language backbone, here RT-2-PaLI-X §4.2. Actions are written as text tokens of the language model §4.2.

Both models use a categorical cross-entropy loss over their output tokens §4.3.

Action tokenisation (§4.2). Each action is split into 8 dimensions: one discrete termination value and seven end-effector values. Each dimension is binned into 256 uniform bins §4.2. The released code uses 11 action tokens instead, which adds base-displacement fields; see Paper vs code.

# Action tokenization: continuous action -> 11 integer tokens
# Input:  action dict with the keys below, for one timestep
# Output: int tokens, shape (11,), values in [0, vocab_size - 1]

def tokenize_action(action, vocab_size, world_range):
    # Terminate flag is a 3-way one-hot; keep its argmax (0, 1 or 2)
    # [code: models/rt1.py#L287-L291]
    tokens = [argmax(action["terminate_episode"])]
    # Fixed bounds per field; the world-vector bound depends on the variant
    # [code: models/rt1.py#L293-L299]
    bounds = {
        "world_vector": world_range,
        "gripper_closedness_action": (-1.0, 1.0),
        "base_displacement_vector": (-1.0, 1.0),
        "rotation_delta": (-pi / 2, pi / 2),
        "base_displacement_vertical_rotation": (-pi, pi),
    }
    for name, (low, high) in bounds.items():
        # Clip to the bounds, then map linearly to [0, 1]
        # [code: models/rt1.py#L300-L302]
        value = clip(action[name], low, high)
        value = (value - low) / (high - low)
        # Scale to the vocabulary and truncate to an integer bin
        # [code: models/rt1.py#L303-L304]
        tokens.append(int(value * (vocab_size - 1)))
    return concatenate(tokens)

Inference for one control step (§4.2, §4.3). The released inference code takes 15 frames, resizes them, runs the tokenizer and decoder, and reads 11 action tokens from the last step.

# RT-1-X inference for one control step
# Input:  frames (15, H, W, 3) uint8; instruction_embedding (15, 512)
# Output: action dict for the current step

def act(frames, instruction_embedding):
    # Resize each frame to 300x300 and scale to [0, 1]
    # [code: models/rt1_inference_example.py#L126-L131]
    images = scale(resize(frames, 300, 300))
    # Tokenize each frame into 81 tokens, conditioned on the instruction
    # [§4.2] [code: models/rt1.py#L114-L189]
    image_tokens = image_tokenizer(images, instruction_embedding)
    # No action history: action slots are zero-filled
    # [code: models/rt1.py#L449-L456]
    action_tokens = zeros(15, 11, 512)
    # Decoder-only Transformer over the interleaved sequence, causal mask
    # [§4.2] [code: models/rt1.py#L468-L480] [code: models/rt1.py#L511-L551]
    logits = transformer(image_tokens, action_tokens)
    # Logits at the 11 action positions of the last step; each logit is
    # read before the token it predicts
    # [code: models/rt1_inference_example.py#L98-L102]
    action_logits = logits[-1, 80:91]
    # Greedy choice, one token per dimension
    # [code: models/rt1_inference_example.py#L104-L105]
    predicted = argmax(action_logits, axis=-1)
    # Map bins back to continuous values
    # [code: models/rt1.py#L311-L355]
    return detokenize_action(predicted)

Training mixture and co-fine-tuning (§4.3). The mixture uses data from 9 manipulators and 12 named datasets: RT-1, QT-Opt, Bridge, Task Agnostic Robot Play, Jaco Play, Cable Routing, RoboTurk, NYU VINN, Austin VIOLA, Berkeley Autolab UR5, TOTO and Language Table §4.3. RT-1-X is trained on this mixture alone. RT-2-X is co-fine-tuned with web vision-language data in an approximately one-to-one split §4.3.

Evaluation (§5). The study runs 3,600 trials across 6 robots §5. Baselines are each dataset's original method and an RT-1 trained on that dataset alone §5.1.

Hyperparameters

Values not listed by the paper are marked not stated. Values from the released code are labelled as such.

Results

Author-reported. Setup: the in-distribution Bridge and RT-1 evaluations on WidowX and Google Robot, the co-trained models described in §4.3 §5.1.

Large-data domains, success rate tab. 1.

The paper states that RT-1-X underfits these large domains and does not beat RT-1; RT-2-X does §5.1 tab. 1.

Small-data domains. RT-1-X beats the Original Method on 4 of the 5 datasets (Kitchen Manipulation, Cable Routing, NYU Door Opening, AUTOLab UR5, Robot Play) §5.1. The per-dataset numbers are only in fig. 3, a bar plot, and the text does not give them. The caption says RT-1-X is "50% higher" than the Original Method or RT-1 on mean success rate fig. 3; the paper does not say whether this is relative or in percentage points.

Emergent skills and generalisation, RT-2-X ablations tab. 2.

Source: tab. 2, §5.2, §5.3. The text says RT-2-X outperforms RT-2 by about 3x on emergent skills §5.2; the table gives 75.8 / 27.3 ≈ 2.8.

Paper vs code

The official repository is google-deepmind/open_x_embodiment at code_commit README.md. It holds the dataset colabs and a JAX RT-1 / RT-1-X model with an inference example. It does not contain RT-2-X, the 55B or 5B models, or the training data mixture code. Differences:

• Action tokens. The paper uses 8 dimensions §4.2. The code emits 11 tokens: 1 terminate, 3 world vector, 3 rotation, 1 gripper, 1 base rotation, 2 base displacement rt1.py rt1_inference_example.py.
• Bins. The paper says 256 bins §4.2. The inference script builds the model with vocab_size = 512 rt1_inference_example.py. The class default is 256 rt1.py.
• Layer size. The paper gives no layer size §4.2. The class default is 128 rt1.py; the inference script uses 256 rt1_inference_example.py.
• Action ranges. The paper normalises each dataset's actions before discretising §4.1. The code instead clips to fixed bounds, with world vector (-2, 2) for RT-1-X rt1_inference_example.py and (-1, 1) as the class default rt1.py. The normalisation statistics are not in this repository.
• Input resolution. The paper says "a common resolution" without a number §4.1. The code resizes to 300x300 rt1_inference_example.py.
• Language input. The paper uses a USE embedding §4.2. The code takes a precomputed 512-d embedding and drops any string rt1_inference_example.py. The encoder that produced it is not in the repository.
• Gripper encoding. The training notebook maps the Bridge gripper signal to gripper_closedness_action with -1 for open and 1 for closed Minimal_Training_Example.ipynb. The paper does not describe this mapping §4.1.
• Feed-forward block. The paper does not name one §4.2. The code defaults to SwiGLU rt1.py.
• Control rate. The paper gives 3–10 Hz §4.3. The README example uses 3 Hz [README.md#L29].

Checked and consistent: 15-frame history §4.2 vs seqlen=15 rt1_inference_example.py; 81 image tokens §4.2 vs num_image_tokens=81 rt1_inference_example.py; the decoder-only Transformer with a causal mask over action tokens §4.2 vs rt1.py; EfficientNet with FiLM conditioning §4.2 vs rt1.py.

Pitfalls

• Action dimension count. Implementers following the text will build an 8-dimensional head. The released model emits 11 tokens, so a checkpoint does not match the paper's layout. Check the checkpoint's output size before you train against it.
• Base fields default to zero. The training notebook starts every action with zeros, then lets each dataset's mapping overwrite them Minimal_Training_Example.ipynb. How a dataset without base data is handled is not stated in the paper; check its mapping before you read a zero base command as a measured "no motion".
• Binning is biased. Encoding truncates value * (vocab - 1) to an integer rt1.py, and decoding returns the lower edge of the bin rt1.py. Decoded values are biased low by up to one bin width, so the error is not zero even on training data.
• Greedy, per-dimension decoding. The inference script takes the argmax of each token separately rt1_inference_example.py. The paper does not say how actions are decoded §4.2. Per-token argmax can combine bins that never occur together in the data (inference, not stated in the paper).
• Termination fallback. If the model predicts no termination flag, the script sets the last one to the base-pose mode rt1_inference_example.py. This changes the emitted action and is not in the paper.
• Coordinate frames differ per robot. The same action vector can cause different motions on different robots §4.1. The policy input is images and an instruction only, with no embodiment field §4.1.
• Ablation confound. The text attributes the emergent-skill gap between 55B and 5B to model size, by comparing row (2) with row (4) §5.3 tab. 2. Row (2) has no history and row (4) has history 2, so the comparison changes two factors at once.
• Co-training ratio. The 1:1 split is approximate and the mixture weights are not stated §4.3.
• Trial counts. The paper reports 3,600 trials in total §5 but not the number per table cell. Small differences in Table 1 cannot be judged without it.
• Training recipe missing. Optimiser, learning rate, step count and batch size are not stated in the paper. The example notebook values are a minimal example, not the recipe behind the released checkpoints Minimal_Training_Example.ipynb.
• RT-2-X is not reproducible from this repository. No RT-2-X code or checkpoint is in the repository. The README links only an RT-1-X JAX checkpoint [README.md#L39-L43].
• Dataset licences. This card covers the paper and its code only. The licences of the 60 constituent datasets are not recorded here. Check the dataset spreadsheet linked from the project page before reusing any data.

Lineage

• Builds on: RT-1 (architecture; the released code cites this arXiv id rt1.py) and RT-2, with the RT-2-PaLI-X backbone §4.2. The arXiv id for RT-2 is not in the source tree.
• Compared against: each dataset's original method, for example LCBC on Bridge tab. 1, and RT-1 trained on the dataset alone §5.1.