Learning Sim-to-Real Humanoid Locomotion in 15 Minutes

In plain words

Teaching a humanoid robot to walk in a simulator is now fast, but moving the learned controller onto a real robot is slow. Each time the robot misbehaves on hardware, the simulator has to be fixed and the whole training run repeated, and the tricks that make the transfer robust (random terrain, random pushes, random physics) make training slower again.

The paper's answer is to use a different kind of learning algorithm. Instead of the common on-policy method, it uses off-policy methods (FastSAC and FastTD3), which learn from old experience stored in a replay buffer, and it runs them on thousands of simulated robots at once. A few settings matter a lot at this scale, such as how far the actions may move each joint, how the critic is averaged, and how much the robot is allowed to explore. The reward function is kept to a handful of terms. With these choices the authors report a walking policy in about 15 minutes on one consumer GPU.

TL;DR

The paper gives a training recipe, not a new algorithm: FastSAC and FastTD3 (off-policy RL) on massively parallel simulation, with specific settings for actions, normalization, critics, exploration and reward. The authors report velocity-tracking locomotion for Unitree G1 and Booster T1 trained in 15 minutes on a single RTX 4090 under strong domain randomization, and whole-body motion tracking for G1. Results are author-reported and mostly shown only in figures. The official code (Apache-2.0) is the Holosoma repository, and it differs from the paper in several settings listed under Paper vs code.

Problem

Sim-to-real iteration is slow for humanoids. A policy is trained in simulation, tested on hardware, and fails because of unmodelled dynamics or sensing errors. The fix is usually more domain randomization, terrain randomization or a curriculum, and each of these makes exploration harder and pushes humanoid training back into the multi-hour range §1. The paper states that PPO is the de facto algorithm for sim-to-real RL, and that FastTD3 was previously shown to reach the real robot only on humanoids with a subset of joints §1 §2.1. The assumptions are a GPU-based parallel simulator with thousands of environments and a reward that can be kept simple §2.2 §4.

Method

The recipe has two parts: an off-policy learner tuned for large-scale parallel simulation §2.1, and a reward with fewer than ten terms §2.2.

Learner and scale §2.1. Off-policy RL re-uses past transitions instead of discarding them, which the paper argues makes it attractive when simulation is the bottleneck §2.1. The authors use thousands of parallel environments §2.1 fig. 2f. Batch size up to 8K gives consistent improvement, and more gradient steps per simulation step usually train faster §2.1 fig. 2b. The paper does not give exact batch or update counts for each experiment (see Hyperparameters).

Action bounds §2.1. The tanh policy needs action bounds. The authors set each joint's bound from the joint limit minus its default position, for PD controllers. A footnote says that, once training is stable, unbounded actions also work, and the authors still recommend this scheme for other robots §2.1 (footnote 1).

Normalization §2.1 fig. 2c. Observation normalization helps. Unlike FastTD3, the authors add layer normalization, which they find stabilizes high-dimensional tasks.

Critics §2.1 fig. 2a. The target uses the average of the two Q-values, not the minimum used by clipped double Q-learning, which the authors report as harmful with layer normalization. The critic is distributional (C51), because quantile regression was too costly at large batch sizes. Discount is gamma = 0.97 for velocity tracking and gamma = 0.99 for whole-body tracking §2.1 fig. 2d fig. 2e.

FastSAC exploration §2.1. The maximum standard deviation of the pre-tanh action is 1.0, instead of the e^2 bound of a common SAC implementation. The initial temperature alpha is 0.001, and alpha is tuned automatically. The target entropy is 0.0 for locomotion and -|A|/2 for tracking, where |A| is the action dimension §2.1.

FastTD3 exploration §2.1. Gaussian noise standard deviation is sampled from [0.01, 0.05] per environment, which the authors report as best §2.1.

Optimizer §2.1. Adam, learning rate 3e-4, weight decay 0.001 (lower than the 0.1 of the FastTD3 paper), and beta_2 = 0.95 §2.1.

Reward §2.2. The locomotion reward has essential terms only: linear and angular velocity tracking, foot-height tracking for swing, a default-pose penalty, feet penalties (parallel orientation, no crossing), an alive reward, torso uprightness penalties, and an action-rate penalty §2.2. Episodes end on ground contact by the torso or by non-foot body parts §2.2. Symmetry augmentation is used §2.2. Penalty weights follow a curriculum that ramps them up as episode length grows §2.2. The tracking reward follows BeyondMimic with DeepMimic-style termination, plus velocity pushes §2.2.

Learning update alg. 1. Algorithm 1 gives the FastSAC loop. The algorithm caption says the distributional critic is omitted for simplicity.

# FastSAC: one environment step, then several gradient steps  [alg. 1]
# Scalar critics here: the distributional critic is omitted in [alg. 1]
# Requires: actor, two critics, two target critics, log_alpha, buffer

def fastsac_env_step(env, actor, critics, target_critics, log_alpha, buffer):
    # Act with the stochastic policy and store the transition  [alg. 1]
    actions = actor.sample(observations)                 # (N, act_dim)
    next_observations, rewards = env.step(actions)       # (N, obs_dim)
    buffer.add(observations, actions, rewards, next_observations)

    # Several gradient steps per environment step  [§2.1] [alg. 1]
    for update in range(num_updates):
        # Draw a mini-batch from the replay buffer  [alg. 1]
        batch = buffer.sample(batch_size)

        # Target: average over the two target critics, minus entropy  [alg. 1]
        # The paper's gamma/2 * sum over critics equals gamma * mean  [alg. 1]
        next_actions, next_log_probs = actor.sample(batch.next_observations)
        next_values = mean(
            target_critic(batch.next_observations, next_actions)
            for target_critic in target_critics
        )
        targets = batch.rewards + gamma * (
            next_values - alpha * next_log_probs
        )

        # Critic update: regress each critic to the shared target  [alg. 1]
        for critic in critics:
            critic_loss = mean((critic(batch.observations, batch.actions)
                                - targets) ** 2)
            critic.update(critic_loss)

        # Actor update with the reparameterised sample  [alg. 1]
        new_actions, log_probs = actor.sample(batch.observations)
        q_values = mean(
            critic(batch.observations, new_actions) for critic in critics
        )
        actor_loss = mean(alpha * log_probs - q_values)
        actor.update(actor_loss)

        # Entropy temperature update towards the target entropy  [alg. 1] [§2.1]
        # target_entropy: 0.0 for locomotion, -|A|/2 for tracking  [§2.1]
        alpha_loss = mean(-log_alpha.exp() * (log_probs + target_entropy))
        log_alpha.update(alpha_loss)

        # Move target critics towards the critics  [alg. 1]
        # rho is not stated in the paper
        for target, critic in zip(target_critics, critics):
            target = rho * target + (1 - rho) * critic

Locomotion setup §3.1 fig. 3. Target velocities are resampled every 10 s, and set to zero with 20% probability so the robot learns to stand. Training mixes flat and rough terrain. Domain randomization covers push perturbations, action delay, PD-gain, mass and friction randomization, and center-of-mass randomization for G1 only §3.1. The caption gives push intervals of 5 to 10 s, and 1 to 3 s for the Push-Strong variant, with a maximum episode length of 20 s fig. 3.

Whole-body tracking setup §3.2 fig. 5. The authors follow BeyondMimic. Each episode samples a random motion segment. The randomized quantities are the ground friction, the center of mass, a bias on joint positions, the body mass, the PD gains, plus external pushes §3.2.

Hyperparameters

Values in the paper are cited to the paper. Values marked "code" are not stated in the paper and come from the official repository at code_commit.

Results

Author-reported. Setup: a single RTX 4090 for all locomotion runs fig. 3 fig. 2; 4 x L40s GPUs and 16384 parallel environments for whole-body tracking fig. 1 fig. 5. Baselines are PPO and FastTD3 §3.1 §3.2. The text gives no numeric results beyond the 15-minute training time, so the numbers are in the figures, which this card does not transcribe.

Analysis figures: clipped double Q (CDQ) fig. 2a, number of updates fig. 2b, normalization fig. 2c, gamma for locomotion fig. 2d, gamma for tracking fig. 2e, number of environments for tracking fig. 2f. The text reports no numbers for these; read the figures for values.

Paper vs code

Repository: https://github.com/amazon-far/holosoma, cloned at c0f88ef (the commit recorded above). The paper links it §1 §5 and cites it as Holosoma [bib: Holosoma]. The repository holds a FastSAC agent. No FastTD3 agent or FastTD3 source was found in it (no file name matching *td3*), so the FastTD3 settings in this card are not checked against code.

• Critic target aggregation. Algorithm 1 and §2.1 give a target that averages the two target Q-values alg. 1 §2.1. The code builds one projected target distribution per critic from its own target network and does not average or take a minimum across critics in the update [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L445-L459] fast_sac.py. The .min() at fast_sac_agent.py is only logged (qf_min, fast_sac_agent.py).
• Optimizer. The paper says Adam §2.1. The code uses torch.optim.AdamW with weight decay 0.001 fast_sac_agent.py.
• Action bounds. The paper uses the joint limit minus the default position §2.1. The code takes, per joint, the larger of the two distances from the default to a limit, and divides it by the PD action scale [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L138-L148].
• Actor update frequency. Algorithm 1 updates the actor in every inner update alg. 1. The code updates it only on some inner updates: with the default num_updates = 8 and policy_frequency = 4, it runs on indices 1 and 5 fast_sac_agent.py algo.py.
• Batch sampling. The code draws one batch of batch_size * num_updates transitions per environment step and splits it across the updates [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L560-L568]. The paper does not describe sampling.
• Target update. Algorithm 1 uses an unstated rho. The code uses target = (1 - tau) * target + tau * critic with tau = 0.125 [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L815-L816] algo.py.
• Reward term count. The paper says "less than 10 terms" §2.2. The G1 FastSAC reward block defines 10 terms [code: src/holosoma/holosoma/config_values/loco/g1/reward.py#L102-L185].
• Tracking environment count. The Figure 1 caption says 16384 environments fig. 1. The code's WBT config sets 4096 [code: src/holosoma/holosoma/config_values/wbt/g1/experiment.py#L89]. The paper does not say which count produced Figure 5.
• Tracking settings not in the paper. The WBT config uses 4 updates per step, policy_frequency 2, tau 0.05, 501 atoms, gamma 0.99 and target-entropy ratio 0.5 experiment.py. Only gamma and the target entropy are in the paper §2.1.
• Matches. Learning rate 3e-4, max sigma 1.0, initial alpha 0.001, beta_2 0.95, gamma 0.97 (locomotion), locomotion target entropy 0.0, layer normalization and symmetry augmentation for locomotion agree between paper and code.

Pitfalls

• The paper and code disagree on the critic target. Following Algorithm 1 gives averaged targets. Following the code gives per-critic targets. The paper does not say which one produced Figures 3 to 5. Test both on a small task.
• Action bounds. The paper's formula does not define the symmetric max-of-two-distances rule the code uses. The paper also says unbounded actions can work after training is stable §2.1 (footnote 1).
• Distributional support. The code clamps the projected target to the value support fast_sac.py. Locomotion and tracking both use [-20, 20] in the code, but the paper does not give the range. If returns leave the range, the targets are silently clipped.
• Mixed-precision and compiled training. The config enables AMP with bf16 algo.py and torch.compile algo.py. The paper does not mention either.
• Reward terms. Each reward term needs its weight. The paper gives none, so the weights above come only from the code and may need retuning for other robots.
• Exploration with the entropy temperature. The paper notes that a large initial alpha with the e^2 std bound caused instability §2.1. The code's auto-tuned alpha starts at 0.001 algo.py.
• Environment count is not comparable. Locomotion environment count is not stated. The code default is 4096, and tracking uses 4096 in the code but 16384 in the figure caption.
• Results are figure-only. The paper gives no numeric results in the text besides the 15-minute training time. Numbers must be read from the figures, which are not transcribed here.
• Timing claims are hardware-specific. The 15-minute claim is for one RTX 4090 [abstract] fig. 3, and the tracking runs use 4 x L40s fig. 1.
• Joint-limit action scale depends on the PD action scale. The code divides by the robot's control action_scale [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L148]. Changing that scale changes the effective range.

Lineage

• Builds on: FastTD3, arXiv:2505.22642 (no card yet). The paper builds on it for the off-policy scaling recipe and for the first sim-to-real FastTD3 deployment on partial-joint humanoids §1 §2.1. Changes: layer normalization, averaged critic, CDQ removed, FastSAC tuning, full-body control §2.1.
• Also builds on: SAC, TD3, PPO, BeyondMimic for the tracking reward and setup §2.1 §2.2 §3.2. These are named in the paper and have no cards yet.
• Compared against: PPO and FastTD3 §3.1 §3.2.
• Superseded by: none known.