DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

In plain words

Robots that learn to manipulate objects usually learn by copying recorded demonstrations. Most such recordings come from a few rooms with the same table and the same camera positions, so a policy trained on them often fails when the lighting, the objects or the scene change.

DROID is a large set of demonstrations recorded in many different places. Many people used the same robot arm and the same recording setup, so the data stayed consistent while the scenes and tasks varied a lot. The authors also released camera calibration for much of the data, which tells you where each camera sits relative to the robot.

The authors tested whether the data helps. They mixed it with a small set of task-specific demonstrations while training a policy, and that policy did better than one trained on the small set alone, and better than one mixed with a different large robot dataset. The gains held both when the test setting matched the training tasks and when the scene or objects changed.

TL;DR

• DROID: 76k successful teleoperated demonstrations, 350 hours, 564 scenes, collected on one shared Franka hardware stack across 13 institutions §1 §3.
• Each episode has three stereo camera streams, robot state and crowdsourced language instructions §3.1 §3.2 app. B.
• Co-training a diffusion policy 50/50 with DROID beats no co-training and co-training with Open X-Embodiment on six lab, office and kitchen tasks §5.2.
• Author-reported margins: 22% absolute in-distribution and 17% out-of-distribution over the next-best method, averaged over tasks §5.2.
• Released under CC BY 4.0. Code is MIT-licensed (robomimic fork). Calibration is post-hoc and partly automatic, so treat it as an estimate app. G.

Problem

Policies generalize poorly when the training data covers few scenes, tasks and viewpoints. Earlier large robot datasets were collected mostly in controlled, lab-like settings with fixed cameras. Collecting diverse data at scale is hard because of logistics, safety and the cost of hardware and supervision §1.

The paper measures diversity along five axes: tasks (verbs), objects, scenes, camera viewpoints and interaction locations §4. It also argues that a dataset is only useful if the same hardware and protocol can be reproduced across labs §3.

Method

Data collection. Each collector moves a Franka Panda arm with a Robotiq gripper onto a height-adjustable desk, sets up two exterior Zed 2 stereo cameras and a wrist-mounted Zed Mini, and teleoperates with a Meta Quest 2 headset §3.1. Actions are recorded in joint space and end-effector space at 15 Hz §3.1 app. B. The collector calibrates the exterior cameras with a checkerboard and OpenCV, then enters the tasks feasible in the current scene. A GUI samples tasks at random so that easy tasks are not over-represented §3.2 app. A. The GUI also prompts random scene changes such as moving the base or the cameras §3.2. A collector stays in one scene for up to about 100 trajectories or 20 minutes before moving on §3.2.

Dataset content. Each trajectory holds three 1280×720 stereo RGB streams, joint and end-effector state, gripper state, extrinsic calibration for both exterior cameras, building name, collector ID, a GPT-4V scene type and 1–3 crowdsourced instructions app. B §3.2. Unsuccessful trajectories (about 16k) are released but not counted in the 76k §3.2.

Analysis. Verbs are extracted from instructions with a semantic parser and de-duplicated with GPT-4 §4. Scene types are labelled automatically with GPT-4V, using the prompt in app. C. A unique scene is defined as a substantial change to the robot's workspace, not a change of objects or external camera pose; duplicate scene IDs are merged within each robot-lab-building group app. D.

Policy learning. The policy is a diffusion policy, following the cited prior work §5.1. Two external views are downsampled to 128×128 and passed through an ImageNet-pretrained ResNet-50. A frozen DistilBERT embedding of the instruction and the robot state are concatenated with the visual features and passed through an MLP, then a U-Net diffusion head produces 16 action steps. Eight of them run open loop before the policy is queried again §5.1 [app. F.1]. The training batch is 50% in-domain demonstrations and 50% DROID trajectories, excluding those marked unsuccessful §5.1.

# Co-training batch  [§5.1] [app. F.2]
# Input: in_domain (this task's demos), droid (successful DROID episodes)
# Output: one batch of 128 samples, half from each source

def sample_batch(in_domain, droid, batch_size=128):
    # split the batch evenly between the two datasets  [§5.1]
    half = batch_size // 2
    # in-domain half comes from this task's demonstrations  [§5.1]
    own = random_samples(in_domain, half)
    # other half comes from successful DROID episodes  [§5.1] [app. F.2]
    shared = random_samples(droid, batch_size - half)
    return concat(own, shared)

# Policy forward pass  [§5.1] [app. F.1]
# Input: two external RGB views, instruction, robot state
# Output: 16 action steps, of which the first 8 run open loop

def predict_actions(views, instruction, robot_state):
    # downsample each view to 128 x 128  [§5.1]
    small = [resize(view, 128) for view in views]
    # colour jitter, then random 116 x 116 crop in training  [app. F.1]
    small = [random_crop(jitter(img), 116) for img in small]
    # ImageNet-pretrained ResNet-50 encoder per view  [§5.1]
    visual = [resnet50(img) for img in small]
    # frozen DistilBERT embedding of the instruction  [§5.1]
    language = distilbert(instruction)
    # concatenate features and run the observation MLP  [app. F.1]
    features = mlp(concat(visual, language, robot_state))
    # U-Net head denoises the action sequence with DDIM  [app. F.1]
    actions = ddim_denoise(unet, features, horizon=16)
    return actions

Camera calibration (post-hoc). The released calibration has three sets §3.2 app. G.5:

• Camera-to-base for about 36k scenes, combining two methods. The first projects forward-kinematics keypoints with the existing extrinsics, runs SAM seeded by them, and compares the mask with a rendered robot silhouette (PyTorch3D). It keeps SAM masks with confidence above 0.65 and an IoU of at least 0.7 app. G.1. This gave about 30k scenes. The second runs a tuned CtRNet-X on all data with confidence thresholds 0.08 / 0.1 / 0.05, rejects outliers with a median-absolute-deviation rule at 2.5 MAD, and keeps scenes with mean reprojection error at most 20 app. G.2. This gave about 12k scenes.
• Camera-to-camera for all scenes, from a modified DUSt3R. Depth and intrinsics are fixed, and camera poses are refined to minimise 3D alignment error. Quality is the count of reciprocal nearest-neighbour 3D matches app. G.3.
• A superset of about 24k scenes where both cameras are calibrated to the base, using an IoU threshold of 0.6, reprojection error of 20 and the top 30% of match counts app. G.5.

Hyperparameters

Values for the policy and training come from the appendix table and the text. "Code" entries are read from the official repository at code_commit and are not stated in the paper.

Evaluation tasks, with in-domain demonstrations and the out-of-distribution (OOD) variant §5.1 app. E:

Results

Author-reported. Setup: six tasks at four locations (lab, office, household), diffusion policy, 10 A/B rollouts per task and method, in-distribution (ID) and OOD evaluation §5.1 §5.2. The comparison baselines are no co-training and co-training with a curated subset of Open X-Embodiment (OXE) §5.2.

Per-task numbers are only in the co-training figure, not in the text. The full DROID co-training matches or outperforms the subsampled one on all three tasks compared in §5.3 §5.3.

Paper vs code

Official code, at code_commit: the policy-learning repository (github.com/droid-dataset/droid_policy_learning, linked from the project page), a robomimic fork that imports a DROID helper package. The hardware repository (github.com/droid-dataset/droid, also linked) was not checked. The calibration pipeline is not in the policy repository: a search of it found only a commented-out success check in convert_droid.py.

• Diffusion sampler. Paper: DDIM app. F.1. Default config: DDPM on, DDIM off diffusion_policy_config.py. The DROID generator turns DDIM on droid_runs_language_conditioned.py. The paper matches the generator, not the default config.
• Learning-rate schedule. Paper: "Linear" app. F.1. Code: no decay unless an epoch schedule is set, and the default schedule is empty diffusion_policy_config.py torch_utils.py. The DROID generator does not set one. Paper and code disagree; the code is what was run.
• Training length. Paper: 25,000 steps, 50,000 for Cook Lentils OOD app. F.1. Code: epochs, train.num_epochs = 100000 in the generator droid_runs_language_conditioned.py, and 600 in the helper default helper.py. Which one applies depends on call order (not traced). The step count cannot be checked from the code.
• Camera views. Paper: RGB from "the two external cameras" §5.1. Code uses only the left image of each external camera; the right stereo images are commented out droid_runs_language_conditioned.py droid_runs_language_conditioned.py.
• Action space. Paper: absolute end-effector translation, rotation and gripper §5.1. Code: absolute position, a 6D rotation representation and gripper position droid_runs_language_conditioned.py. The paper does not name the rotation representation.
• Noise samples. Not stated in paper. Code: 8 in the DROID generator, 1 in the base config (see table).
• Success filter. Paper excludes unsuccessful trajectories from the co-training data §5.1. The HDF5 converter's success filter is commented out convert_droid.py; the HDF5 loader does not filter by success. The RLDS path does: filter_success in rlds_utils.py, used in train.py.
• Co-training mix. Paper: 50/50 §5.1. Code: a sample is weighted by its dataset weight divided by dataset length dataset.py, and dataset weights default to 1.0 train_utils.py. The DROID generator loads only one dataset; the two-dataset weights are commented out droid_runs_language_conditioned.py. A 50/50 mix must be set by the user.
• Segmentation. Not in paper. The DROID dataset class treats each HDF5 file as one demo and splits it at the first gripper close and open dataset.py dataset.py. If the gripper signal is absent, it uses the one-third and two-thirds points.
• Calibration. The paper's exact thresholds for the three calibration sets are in app. G, but the code is not in this repository, so they are unverified here.

Pitfalls

• Dataset size is inconsistent. The abstract says 84 tasks; the text macro used in the body gives 86 §1 [abstract]. The paper uses 76k trajectories for the dataset and 40K for the policy experiments app. F.2. Report whichever you use.
• Scene-type count is inconsistent. §4 says 10 scene types. The caption in app. C says 9. The prompt in app. C lists 11 labels plus "Unknown". Use the prompt labels if you reproduce the annotation.
• Scene count depends on dedup. The paper reports 2,080 raw scene IDs before merging app. D. The 564 figure comes after its heuristic merge, which is conservative and does not catch the same scene recorded by two robots app. D.
• "Clean up desk" eraser position differs. §5.1 says fixed; app. E says varied on a fixed schedule of positions and orientations.
• Failed trajectories count separately. About 16k unsuccessful episodes are included in the release but not counted in 76k §3.2. Filter explicitly when you train.
• Co-training uses a time-dependent subset. The 40K-episode subset is the "first 40K successful trajectories" for which language annotations were available at training time app. F.2. Rebuilding it later gives a different set.
• Joint policy for two tasks. Toasting and Close Waffle Maker share one in-domain policy, and their no-co-training baseline is trained on both app. F.2.
• Calibration numbers differ. The superset uses IoU 0.6 app. G.5, while the first camera-to-base filter uses IoU 0.7 app. G.1. Reprojection-error units are not stated. The CtRNet-X "tuned" hyperparameters are described as tuned for DROID, but the paper does not say how they were tuned app. G.2.
• Calibration generalises only partly. CtRNet-X was trained on Franka robots and its transfer to other robots is untested. DUSt3R fails on cluttered scenes with low overlap app. G.4.
• Evaluation is small. Ten A/B rollouts per task and method §5.2. The paper reports standard error only in the figure caption. Treat the per-task differences as noisy.
• Figure numbers are unreliable. The title block adjusts the figure counter after the teaser, so the source may number more than one figure "1". Cite by section here, not by figure number.
• Cost. The camera-to-base filter took about 1 day, and the CtRNet-X pass about 5 days, on 8 A100 GPUs app. G.1 app. G.2. Training cost is not stated.

Lineage

• Builds on: Open X-Embodiment (OXE) as a comparison dataset and the curated split used by Octo §5.2 app. F.2; Diffusion Policy for the policy §5.1; robomimic for the training code app. F.1; Polymetis for control §3.1; SAM, PyTorch3D, CtRNet-X and DUSt3R for calibration app. G.
• Compared against: no co-training (in-domain only) and OXE co-training §5.2.
• Superseded by: not stated.