Real-World Humanoid Locomotion with Reinforcement Learning

In plain words

The paper teaches a human-sized robot (the Digit humanoid) to walk by practising in a computer simulation first and then running the learned controller on the real robot without any extra tuning on hardware.

The controller is a transformer, the same kind of network behind large language models. Instead of looking only at its current sensor readings, it looks at a short memory of what it sensed and what it commanded a moment earlier, and it uses that history to choose the next move. The authors' hunch is that this history reveals things about the ground, such as slope or a snagged foot, so the robot can adjust without being explicitly told what the terrain is.

To make the simulated practice transfer to the real world, the simulator is randomised: robot dynamics, control parameters, physics, sensor noise and delay vary between runs, so the robot cannot rely on one exact model. The policy is first trained with access to the full simulator state as a teacher, and then a student that sees only sensor data is trained to copy the teacher while also learning from reward.

TL;DR

• Fully learned walking controller for the full-size Digit humanoid, trained in simulation with PPO and deployed zero-shot.
• A causal transformer over observation-action history acts as the policy; the authors argue it adapts in context without weight updates.
• Reported: outdoor walking without falls, robustness to pushes and payloads, emergent arm swing, emergent gait change on slopes and recovery from foot-trapping. All results are author-reported.

Problem

Classical and optimisation-based humanoid controllers can be very effective, but they are hard to generalise and adapt to new environments § Introduction. The paper proposes a fully learning-based controller for real-world humanoid locomotion § Introduction.

The robot is treated as partially observable: the policy sees noisy proprioceptive observations, not the true state, so the problem is formalised as a POMDP § Materials and Methods: Policy learning.

Direct RL in observation space is described as slow and resource-intensive because of limited sample efficiency § Teacher state-policy supervision. The two-step teacher-student setup is the authors' answer to that cost.

The robot is blind: it has no cameras or other exteroceptive sensors, so it cannot see steps or obstacles before contact § Outdoor deployment [§ Emergent recovery from foot-trapping].

Method

Robot and setting. Digit is about 1.6 m tall, weighs about 45 kg and has 30 degrees of freedom in its floating-base model; the shin and tarsus are passive and the toe is driven through rods § Digit humanoid robot. The policy outputs PD setpoints for 16 actuated joints and predicted PD gains for 8 actuated leg joints; the four toe motors are not learned and keep default positions with fixed PD gains § Policy learning.

Step 1, teacher. A teacher policy \pi_s(a_t|s_t) is trained in simulation on the full state, assuming full observability. Its reward is tuned by trial and error, inspired by biomechanics of human walking, with no pre-computed gait library § Teacher state-policy supervision § Policy learning. Teacher architecture: an MLP with hidden sizes [512, 512, 256, 128] [§ Model architecture].

Step 2, student. A causal transformer \pi_o receives a window of observation-action pairs, o_t, a_{t-1}, o_{t-1}, \dots, o_{t-l+1}, a_{t-l}, and outputs the next action a_t § Model architecture. Each pair is one token. Each token is embedded with an MLP and given a sinusoidal positional encoding. Self-attention is restricted to preceding tokens (causal) [§ Model architecture]. Attention is the standard scaled dot-product form eq. 1.

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V eq. 1

The student is trained with a joint objective that adds an RL loss to a KL term against the teacher eq. 2:

L(\pi_o) = L_{RL}(\pi_o) + \lambda\, D_{KL}(\pi_o \parallel \pi_s) eq. 2

• \lambda is annealed to zero during training, typically reaching zero at the midpoint of the training horizon, so the student benefits from the teacher early and can surpass it later [§ Joint optimization with reinforcement learning].
• No pre-computed trajectories or offline datasets are used; both terms are optimised on-policy § Joint optimization with reinforcement learning.
• The RL algorithm is PPO with an actor-critic setup; actor and critic do not share weights § Policy learning.

Commands and reward. The walking command is three numbers: x and y linear velocity and yaw rate § Natural walking. Commands are resampled at a fixed interval, which the Natural walking section gives as every 10 seconds; commands below a small cut-off are set to zero § Domain randomization. The reward includes energy-minimisation terms § Dynamic arm swing.

# Student observation policy pi_o  [§ Model architecture]
# Input:  observation-action history, window l = 16 pairs
# Output: next action (16 PD setpoints + 8 leg PD gains)

def student_policy(observations, actions):
    # Embed each observation-action pair as one token  [§ Model architecture]
    tokens = embed_pairs(observations, actions)  # (B, l, d_in), MLP [512, 512]
    # Add sinusoidal positional encodings  [§ Model architecture]
    tokens = add_sinusoidal_positions(tokens)    # (B, l, 192)
    # Four causal transformer blocks: width 192, 4 heads, MLP ratio 2.0
    # Each token attends only to itself and earlier tokens  [§ Model architecture]
    features = causal_transformer(tokens)        # (B, l, 192)
    # Action head: MLP hidden [256, 128]; per-position readout is not stated
    return action_head(features)                 # (B, l, action_dim)

# Joint training of the student  [§ Joint optimization] [eq. 2]
# Teacher pi_s is already trained on the full state

def train_student(teacher, student, env, total_steps):
    # On-policy rollouts only; no offline dataset  [§ Joint optimization]
    for step in range(total_steps):
        # lambda decays to zero, reaching zero near mid-training  [§ Joint]
        weight = kl_weight(step, total_steps)
        rollouts = collect_rollouts(env, student)
        # RL term: PPO loss on the student's policy  [§ Policy learning]
        rl_loss = ppo_loss(student, rollouts)
        # Teacher term: KL divergence in the direction written in eq. 2
        kl_loss = kl_divergence(student, teacher, rollouts)  # [eq. 2]
        update(student, rl_loss + weight * kl_loss)

Simulation. Training uses Isaac Gym for rigid-body and contact dynamics § Closed kinematic chain. Because Isaac Gym cannot model the closed kinematic chains of Digit's legs, the rods are replaced by a "virtual spring" with high stiffness, driven by forces from its deviation from nominal length, and an alternating sub-step scheme corrects spring length § Closed kinematic chain. Domain randomisation covers robot dynamics, control parameters, environment physics, observation noise and observation delay; terrains are smooth planes, rough planes and smooth slopes § Domain randomization. The randomisation table and command ranges are in the supplement, which is not in the arXiv source [§ Domain randomization].

Sim-to-real. The policy is first checked in the manufacturer's high-fidelity Agility simulator, which models closed chains and sensor noise. This step does not change network weights and only filters out unsafe policies [§ Sim-to-real transfer]. On hardware the network runs at 50 Hz and the joint PD controller at 1 kHz; joint encoders and IMU data come through the manufacturer's API [§ Sim-to-real transfer]. The authors state that combined dynamics, terrain and delay randomisation gave high-quality transfer § Sim-to-real transfer.

Training compute. Isaac Gym with thousands of randomised environments, parallelised across four A100 GPUs fig. 7.

Hyperparameters

Results

Author-reported. Setup: Digit humanoid; training in Isaac Gym; the manufacturer's Agility simulator for pre-hardware checks. Baselines are the company controller (simulation) and MLP, TCN and LSTM student architectures (ablations). The paper reports no results table; the numbers below are the ones given in text, and the rest are in figures only.

Paper vs code

No official code. A search of the authors' project page (no code links), the GitHub repository search (a third-party transformer reimplementation exists and was not used), and the first author's GitHub account (no matching repository) found nothing official.

Pitfalls

• The supplement is missing from the source. The paper defers PPO hyperparameters, reward terms, observation and action space composition, domain-randomisation ranges and command ranges to a supplement that is not in the arXiv LaTeX source § Policy learning § Domain randomization. An implementer must obtain it separately or treat these as not stated.
• KL direction. Equation 2 writes D_{KL}(\pi_o \parallel \pi_s), while the text describes distilling the teacher "through" KL divergence [§ Teacher state-policy supervision]. Forward and reverse KL differ; implement the direction in eq. 2 as written, and check against the supplement.
• Lambda schedule. Only "gradually annealed to zero, typically reaching zero at the midpoint" is given § Joint optimization. Linear, cosine or step decay, and the starting value, are not stated.
• Per-position readout. The text says the model is trained by autoregressive prediction of the next action fig. 7, but it does not say whether every position's output is supervised or only the last one.
• Positional encoding. The paper says sinusoidal, but not the frequencies or whether encodings are added to pairs or to separate observation and action tokens § Model architecture.
• Token layout. The history is written as one token per observation-action pair, and the window is 16 pairs; the exact token dimension depends on the observation space, which is not stated.
• Virtual spring for closed chains. Isaac Gym cannot simulate the closed kinematic chains, so the rods are approximated by a stiff virtual spring with sub-step correction § Closed kinematic chain. Stiffness and sub-step count are not stated, and the authors say simulation approximations remain a challenge for transfer § Sim-to-real transfer.
• Evaluation depends on a proprietary simulator. The Agility simulator and the Digit robot are not publicly available; the simulator checks cannot be reproduced from the paper.
• Partial success scoring. Simulated terrain trials give partial credit for crossing part of a terrain, and results are mean success with 95% CI over 10 runs § Comparison to the state of the art. Both controllers were run in the same simulator, but the paper does not describe how partial credit is assigned beyond this.
• Blind policy. No exteroception: the policy reacts to obstacles only after contact, and steps were not in training. Foot-trapping recovery is reported as emergent § Emergent recovery from foot-trapping; the authors describe it as not pre-programmed.
• Known limitations. The policy is slightly asymmetric (better to the left), imperfect at velocity tracking, and can fall under a very strong pull on a cable § Limitations.
• Arm swing. Arm swing emerges without reference motions or an explicit constraint, but the reward has energy terms; the authors suggest but do not show that swing saves energy § Dynamic arm swing.
• Real-world hardware risk. The unstable-planks comparison was not run on hardware because of possible damage § Comparison to the state of the art.

Lineage

• Builds on: PPO [Schulman2017 as cited in § Policy learning]; the transformer architecture § Model architecture; Isaac Gym simulation [§ Closed kinematic chain]; the causal transformer used for language § Model architecture.
• Compared against: MLP, TCN (Bai et al. 2018), LSTM (Hochreiter & Schmidhuber 1997) student architectures § Neural network comparisons; the company controller of the robot § Comparison to the state of the art.
• Superseded by: not stated in this card's source.