GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

In plain words

Robots that work in homes and factories need two things: a body that can reach and grasp, and a "brain" that can see a scene, understand an instruction such as "put the apple in the basket", and decide how to move. GR00T N1 is a single neural network built for humanoid robots that does both. It is trained once on many kinds of data and then adapted to a specific robot with a small amount of extra demonstration data.

The model is split into two parts that work like fast and slow thinking. The slow part looks at the camera image and the instruction and works out what the task is. The fast part takes that summary and generates the motor commands for the next short stretch of movement, refining a rough guess into a smooth motion in a few steps.

The hard part is data. Robots have no equivalent of the huge text and image collections that language models learn from. The authors therefore mix real robot recordings, simulated demonstrations, videos generated by video models, and videos of people doing tasks. The videos have no motor commands, so the authors infer pseudo-commands for them. Training on this mixture is what the authors credit for the model's ability to work with little task-specific data.

TL;DR

• Vision-language-action (VLA) model for humanoid robots: an Eagle-2 vision-language backbone §2.1 plus a diffusion transformer that outputs action chunks §2.1.
• Trained with flow matching on a "data pyramid" of real robot, simulated, neural-generated and human-video data; actions for action-less video come from latent actions and an inverse dynamics model §2.2, §3.
• Author-reported: beats Diffusion Policy and BC-Transformer on three simulation suites and on GR-1 real-robot tasks, most clearly with 10% of the post-training data §4.4, tab. 2, tab. 3.
• Weights and code are public, but the linked code is a later release (N1.7) than the paper; see Paper vs code.

Problem

Robot foundation models lack a web-scale dataset. Data for one humanoid platform is small, and pooling datasets across robots leaves many incompatible "data islands" with different sensors, action spaces and control modes §1. Real humanoid teleoperation is slow and costly to scale, because controlling two arms and dexterous hands at once is hard to do with a human operator §2.2, §3.2. The paper targets this gap with a model that accepts varying state and action dimensions and that can learn from data with no action labels §1, §2.1.

Method

Architecture §2.1, fig. 2, fig. 3. The model has two modules trained together:

1. System 2 (vision-language). Images are resized to 224 x 224 and pixel shuffled to 64 tokens per frame. These tokens and the text instruction go through the Eagle-2 language model (initialised from SmolLM2, with a SigLIP-2 image encoder). For the released 2.2B model, the representation taken from layer 12 of the language model is used, because the authors found middle layers faster and more successful than the final layer §2.1. The VLM has 1.34B parameters in total §2.1.
2. System 1 (action). A diffusion transformer (DiT) with alternating cross-attention to the VLM tokens and self-attention over the noised action tokens and the state embedding. Each embodiment has its own MLP encoder for state and actions, and its own MLP decoder for actions, so the shared transformer sees a common width §2.1. The action encoder also takes the diffusion timestep. The action chunk length is H = 16 §2.1.

Training objective §2.1, eq. 1. Flow matching regresses a velocity field on noised action chunks. Flow time is drawn from a Beta distribution rescaled by s = 0.999, following the pi-0 recipe the authors cite §2.1:

# Flow-matching training step  [§2.1] [eq. 1]
# Input:  actions (B, H, d_a) ground-truth chunks; context from the VLM
# Output: scalar loss
# Note: the sign convention here is the paper's text; the N1.7 code
#       uses the opposite one. See Pitfalls.

def training_step(actions, context, state):
    # Sample one Gaussian noise tensor per chunk  [§2.1]
    noise = sample_standard_normal(actions.shape)  # (B, H, d_a)
    # Sample flow time tau in [0, 1] from the rescaled Beta  [§2.1]
    tau = sample_flow_time(batch_size=actions.shape[0])  # (B,)
    # Interpolate between the chunk and noise, as written  [§2.1]
    noised = tau * actions + (1 - tau) * noise
    # Predict the paper's denoising target from the noised chunk  [§2.1]
    prediction = model(context, noised, state, tau)
    target = noise - actions
    return mean_squared_error(prediction, target)

Sampling §2.1. At inference the chunk starts from Gaussian noise and is integrated with K = 4 forward-Euler steps §2.1:

# Action sampling by Euler integration  [§2.1]
# Input:  state (d_s,), context from the VLM
# Output: action chunk (H, d_a)

def sample_actions(state, context):
    # Start the chunk at flow time 0 from pure noise  [§2.1]
    actions = sample_standard_normal((H, d_a))  # (H, d_a)
    for step in range(K):
        # Predict the velocity at the current flow time  [§2.1]
        tau = step / K
        velocity = model(context, actions, state, tau)
        # Euler update with step size 1/K, as written  [§2.1]
        actions = actions + velocity / K
    return actions

Latent actions §2.2, fig. 4. Human and neural videos have no motor commands. A VQ-VAE is trained on frame pairs (current frame x_t, frame x_{t+H} a fixed window later). Its encoder then serves as an inverse dynamics model that yields a continuous latent action per pair. These latents are used as flow-matching targets under a separate "LAPA" embodiment tag. Training on all data together is meant to give one shared latent action space §2.2.

Neural trajectories §2.2, §3.2, fig. 5, fig. 13. Image-to-video models were fine-tuned on the in-house teleoperation data (88 hours) and used to generate 827 hours of video from existing first frames and new language prompts. Generated videos are filtered by a multimodal LLM judge that sees 8 frames and are re-captioned when they fail §2.2, app. F. Their actions come from latent actions and/or an inverse dynamics model trained per embodiment §2.3, app. F.

Simulation trajectories §2.2, §3.2. DexMimicGen expands a few dozen human demonstrations collected by teleoperation (Leap Motion) into new demonstrations by segmenting them into object-centred subtasks, transforming the segments, replaying them in simulation and keeping only successes §3.2.

Data pyramid and co-training §1, §2.2, §3, tab. 1. Real robot data is at the top, simulated and neural data in the middle, and human video at the base. Training batches are sampled across the whole mixture. In post-training, real and neural trajectories are co-trained at a 1:1 ratio §2.3.

Auxiliary loss app. F. A linear head on the final vision-language tokens predicts the normalised image coordinates of the target object, and is trained with a squared loss. The total loss is the flow-matching loss plus this term.

Hyperparameters

Results

Author-reported. Simulation numbers are average success rates over 100 trials per task with 100 demonstrations per task unless noted. Real-robot numbers use the GR-1 humanoid and partial credit (see Pitfalls). The baselines are BC-Transformer (RoboMimic) and Diffusion Policy §4.3.

Simulation (tab. 2; per-task numbers in tab. 4; see Pitfalls on the averages):

Real GR-1 (tab. 3; per-task numbers in tab. 5):

The four real task groups are pick-and-place, articulated objects, industrial and multi-agent coordination §4.2.

Pre-training only (no post-training) on two real GR-1 tasks: 76.6% (11.5 of 15 trials) for a bimanual handover and 73.3% (11 of 15) for placing unseen objects into an unseen container §4.4. Half-credit trials count as 0.5 §4.4.

Neural trajectories as added co-training data §4.4, fig. 9: gains of +4.2, +8.8 and +6.8 points on RoboCasa at 30, 100 and 300 demos (averaged); +5.8 points averaged over 8 GR-1 tasks. Latent and IDM labels were compared in simulation; the paper reports IDM labels as better at larger data sizes §4.4.

Speed. Sampling 16 actions takes 63.9 ms on an L40 GPU in bf16 §2.1.

Paper vs code

The repository linked in the paper is NVIDIA/Isaac-GR00T §1. Its owner is NVIDIA, which is listed as an author. We cloned it at commit 944fe70c4fd54e3fcb80a7d7258a328f9c061176 (Apache-2.0).

The code at this commit is not the model in this paper. It contains GR00T N1.7 (gr00t/model/gr00t_n1d7), a later release that uses a different backbone (Cosmos-Reason2-2B, a Qwen3-VL architecture) [code: README.md#L101]. Its defaults also differ from the paper: action_horizon is 40, not 16 gr00t_n1d7.py, and hidden_size is 1024 gr00t_n1d7.py. The N1 model code was not present in this checkout, so the N1 rows below are unverified. The following comparisons use the N1.7 code only as a later reference:

• Flow-time sampling matches the paper: Beta(1.5, 1.0) with noise scale s = 0.999 gr00t_n1d7.py, gr00t_n1d7.py, gr00t_n1d7.py. This is the same distribution as the paper's §2.1, with t = (1 - b) s.
• The number of denoising steps is 4 gr00t_n1d7.py, matching §2.1.
• The interpolation and target differ in sign from the paper's text. N1.7 uses noisy = (1 - t) * noise + t * actions and velocity = actions - noise gr00t_n1d7.py, and the Euler update is actions += dt * velocity with dt = 1/K gr00t_n1d7.py gr00t_n1d7.py. This is the only self-consistent pairing. The paper writes the target as noise minus actions and the update with a plus sign, which is inconsistent with its own interpolation (see Pitfalls).
• The vision layer choice is 12 in N1.7 gr00t_n1d7.py, the same number as the paper's, but the backbone differs.

No code was checked for the N1 action-expert architecture, the latent-action VQ-VAE, the DexMimicGen pipeline or the auxiliary detection loss. Treat these parts as paper-only.

Pitfalls

• Wrong release in the linked repository. Anyone who follows the link to reproduce N1 will get N1.7 (different backbone, action horizon 40, different checkpoints). Pin the N1 release explicitly, or implement from the paper.
• Sign inconsistency in the flow equations. The paper defines the interpolation tau * A + (1 - tau) * noise §2.1, then the regression target noise - A and the Euler update A + V / K §2.1, eq. 1. Taken literally, integrating from noise to data needs the opposite sign. The N1.7 code uses velocity = actions - noise with noisy = (1 - t) noise + t actions and actions += dt * velocity, which is self-consistent. Implement that convention and verify with a toy example before training.
• Pre-training batch size only in the table. The pre-training batch size (16,384) and step count (200,000) appear only in tab. 6. The text gives a global batch of 1024 and 60k steps for the post-training experiments §4.3. Use the table for pre-training and do not transfer the post-training values.
• Simulation averages do not follow from the columns. In tab. 2 the "Average" for GR00T (45.0%) is not the mean of its three suite values (32.1, 66.5, 50.0 gives 49.5). Diffusion Policy's average (33.4%) is not the mean of 25.6, 56.1 and 32.7 (38.1). The DexMG column also differs from the per-task table at the same 100-demo setting (66.5% in tab. 2, 58.5% when averaged in tab. 4; Diffusion Policy 56.1% vs 46.9%). Do not use the headline 45.0% as an exact figure; check per-task numbers in tab. 4.
• Partial credit in real-robot results. A trial where the object is grasped but not placed scores 0.5 §4.4. Real-robot averages therefore mix half and full successes, and per-cell trial counts are small (10 per task, 5 for Pack Machinery, 15 for the pre-training tasks) §4.2, §4.4.
• Checkpoint selection. Simulation scores are the maximum over the last five checkpoints, saved every 500 steps §4.3. This favours the best of several evaluations and is not a single held-out number.
• Control rate vs chunk time. The paper gives a 120 Hz action rate §1 and 63.9 ms to sample a 16-action chunk on an L40 §2.1. Sixteen steps at 120 Hz span about 133 ms, so the sampling time fits inside one chunk's duration. The paper does not say how often a new chunk is sampled or how chunks are scheduled, so check that before setting a control loop.
• Auxiliary loss weight. The total loss is L = L_fm + L_det app. F with no extra weight, and the detection head's input and the layer used are not stated.
• Neural-trajectory count. The related-work section mentions about 300k neural trajectories §5, while §2.2 and §3.2 report 827 hours of video. Hours are the measure used in the tables, so use that.
• Simulation trajectory count. §3.2 gives 780,000 simulated trajectories for pre- and post-training and 540,000 for pre-training alone §3.2. The split between the two is not given.
• Frozen parts. The post-training text says the language component of the VL backbone is frozen §2.3, while tab. 6 lists the vision encoder as unfrozen in both phases. Whether the vision encoder is trained after pre-training is not stated.
• Data licences. The pre-training corpus combines Open X-Embodiment, AgiBot, Ego4D, Ego-Exo4D and other sets, each with its own terms §3.1, §3.3. The paper says the checkpoint and data are public §1, but the terms for each source are not summarised here; check them before reuse.
• Compute is large. Pre-training used about 50,000 H100 GPU hours §2.3. Neural-trajectory generation used about 105k L40 GPU hours §3.2. For compute-constrained fine-tuning on a single A6000, the authors give a batch size of up to 200 when only the adapter layers and DiT are trained, and up to 16 when the vision encoder is tuned §2.3.

Lineage

• Builds on: the Eagle-2 vision-language model and SigLIP-2 §2.1; the flow-matching action objective and timestep distribution of pi-0 §2.1; DiT §2.1; latent action learning with a VQ-VAE (LAPA) §2.2; DexMimicGen for simulated data §3.2; RoboCasa for simulation tasks §4.1.
• Compared against: BC-Transformer (RoboMimic) and Diffusion Policy §4.3.
• Later release: the repository README lists GR00T N1.5, N1.6 and N1.7 releases, with N1.7 changing the backbone to Cosmos-Reason2-2B README.md. N1.7 is therefore not a drop-in replacement for N1.