The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control

In plain words

A humanoid robot that learns to walk from recorded examples needs good examples. This paper asks how much the examples matter. The authors first train a walking controller with reinforcement learning in a simulator, and they make that simulator deliberately messy: bumpy ground and obstacles, pushes to the body, and changed joint stiffness and friction. The walking controller then produces the example walks.

A diffusion policy, a model trained to clean up noisy sequences of actions, learns from those examples. The authors build many example sets, each with one kind of mess turned on, in three sizes. They then check how well each trained model walks on a plain floor and on a rougher one.

Their main finding is that walking needs much more data than the robot-arm tasks where diffusion policies are known to work. The variety of the mess matters too. Bumpy terrain and pushes give the most usable data. Simply collecting more calm walking does not fix the problem.

TL;DR

• Question: how dataset diversity and size affect diffusion policies (DPs) for whole-body humanoid control §1 §5.
• A source RL policy, trained with AMP on a simulated Unitree H1 in IsaacGym, generates 24 datasets: 8 randomization setups × 3 sizes (500K, 2M, 8M transitions) §3 §5.
• Author-reported: terrain and perturbation randomization give the most robust DPs. The 8M "All" DP matches the source RL policy on a non-randomized target (success 1.0, tracking 0.13 vs 0.12) §5.1 tab. 3.
• The authors conclude that locomotion needs substantially more data than manipulation §1 §6.
• No official code found. The DP's own hyperparameters are not stated and the paper defers them to DiffuseLoco §4 (see Paper vs code).

Problem

Distilling a policy from demonstrations is still hard, and most diffusion-policy work on robots uses small, sparse real-world datasets §1. Reinforcement-learning locomotion policies trained in randomized simulators are robust, but the effect of the source data on diffusion policies for whole-body control was not studied §1.

The paper assumes that a reinforcement-learning policy trained with AMP can serve as a data generator for the DP §3 §4. It also assumes the datasets cover the target conditions well enough for the DP to generalise, which the results test in two target environments §5.

Method

Overview. The pipeline has four stages fig. 1:

1. Train a walking RL policy with AMP under extensive domain randomization §3.
2. Use that policy to collect datasets, one randomization setup at a time §3.2.
3. Train one diffusion policy on each dataset §4.
4. Evaluate each DP in a non-randomized and in a randomized target environment §5.

Source RL policy (AMP). The policy is trained with Adversarial Motion Prior (AMP) §3 §3.1. The task reward asks the robot to track a commanded base velocity §3.1 eq. 1:

r_{\text{g}} = w_v \exp\left(-\|\hat{v}_{xy} - v_{xy}\|\right) + w_\omega \exp\left(-|\hat{\omega} - \omega|\right) eq. 1

• Commands are sampled uniformly: forward and lateral velocities in (−1, 1) m/s and (−0.7, 0.7) m/s, and yaw rate in (−1.57, 1.57) rad/s §3.1.
• A discriminator provides a style reward learned from motion data. The task reward, the regularization rewards in tab. 1 and the style reward are combined §3.1.
• The imitation data is 10 AMASS motion-capture sequences: forward, backward and sideways walking, in-place rotation, circles and figure-8 walks §3.
• The AMASS hand and foot keypoints are converted to humanoid motion with MuJoCo inverse kinematics, which accounts for the humanoid's morphology and possible collisions §4 fig. 1.
• The action is a 19-dimensional vector of desired positional changes for each actuated joint, processed by a PD controller §3.1.
• The critic gets privileged observations of 69 dimensions: base linear and angular velocities, gravity orientation, joint positions and velocities, the command, and the previous action. The policy gets the same observation without the velocities §3.1.
• Robustness comes from the domain randomization in tab. 2. The paper describes it as "extensive" and gives the ranges there §3.

Randomization setups for data collection. Each setup switches on one randomization cluster during data collection §3.2. The source policy is the same for all setups §3 §4.

• No randomization. The baseline setup, labelled "No rand." in fig. 2 and fig. 3.
• Dynamics randomization. Simulator parameters are randomized as in tab. 2, following the approach in §3.2 (a prior work on sim-to-real domain randomization, cited there).
• Perturbations. Force pushes on the torso of random amplitude, capped at 0.6 m/s, applied every 3 seconds §3.2.
• Terrain. IsaacGym terrains with obstacles and bumpy surfaces. Terrain parameters are not stated §3.2.
• Initial state. Each episode starts from a reference state sampled from the expert (mocap) data. With 50% probability, random joint positions and velocities in a limited range are added. The range is not stated §3.2.
• Humanoid scale. Scales 0.8, 1.0 and 1.2 are used during collection. Mass and inertia scale with k³ and k⁵, and PD gains and torque limits scale with k⁴ §3.2. The authors first trained three per-scale policies, then found that the single-scale policy was robust across scales and used it for all collection §3.2.
• Terrain and perturbations combined. Labelled "Terrain + Perturb." in the results fig. 2 fig. 3.
• All. Labelled "All" in the results. Its exact composition is ambiguous, see Pitfalls.

Each setup is collected at three dataset sizes, 500K, 2M and 8M transitions of observations and actions. That gives 24 datasets §3 §5.

Diffusion policy. One DP is trained per dataset §4. The architecture follows DiffuseLoco §2 §4:

• Two 2-layer MLP encoders embed the previous state-action transitions and the goal information §4.
• The diffusion timestep embedding is a second conditioning input §4.
• The noisy ground-truth actions pass through six Transformer decoder layers. Each layer has 8-head cross-attention to the conditioning §4.
• The training loss is the mean squared error between predicted and sampled noise §4.
• Receding-horizon control: the policy predicts n future steps and executes only the first one §4.

# Dataset collection for one randomization setup  [§3] [§3.2]
# Input:  amp_policy, setup (one of 8), num_transitions
# Output: list of (observation, action) pairs

def collect_dataset(amp_policy, setup, num_transitions):
    # Only the randomization of this setup is switched on  [§3.2]
    dataset = []
    while len(dataset) < num_transitions:
        # Each episode starts from a reference state from the mocap data  [§3.2]
        state = sample_reference_state()
        # With 50% probability, add random joint positions and velocities  [§3.2]
        state = maybe_randomize_initial_state(state, setup)
        # Build terrain, humanoid scale and dynamics for this setup  [§3.2]
        env = make_environment(setup)
        obs = env.reset(state)
        # Episode length is not stated; the eval episode is 500 steps  [§5]
        for step in range(episode_steps):
            # The policy acts on the observation without base velocities  [§3.1]
            action = amp_policy(obs)
            dataset.append((obs, action))
            obs = env.step(action)
    return dataset

# One diffusion policy training step  [§4]
# Input:  history (previous transitions), goal, future actions (n, 19)
# Output: scalar loss

def diffusion_training_step(history, goal, actions, timestep):
    # Two 2-layer MLP encoders embed the history and the goal  [§4]
    condition = encode_history_and_goal(history, goal)
    # The diffusion timestep embedding is a conditional input  [§4]
    time_embedding = embed_timestep(timestep)
    # Add sampled noise to the ground-truth actions  [§4]
    noise = sample_gaussian(actions.shape)
    noisy_actions = add_noise(actions, noise, timestep)  # schedule not stated
    # Six Transformer decoder layers, each with 8-head cross-attention  [§4]
    predicted_noise = transformer_decoder(
        noisy_actions,
        condition,
        time_embedding,
    )
    # Mean squared error between predicted and sampled noise  [§4]
    return mean_squared_error(predicted_noise, noise)

# Receding-horizon execution at test time  [§4]
# Input:  policy, history, goal
# Output: one action for the simulator

def act(policy, history, goal):
    # Predict actions for n future steps; n is not stated  [§4]
    action_chunk = policy.sample(history, goal)
    # Execute only the first predicted action  [§4]
    return action_chunk[0]

Evaluation. Each DP is trained with 3 random seeds §5. There are two target environments §3 §5:

• Non-randomized target: flat ground, no randomization §3 §5.1.
• Randomized target: milder terrain and dynamics randomization, with the ranges in tab. 4. External perturbations are not applied §5.2.

Each evaluation lasts 10 seconds, which is 500 simulation steps. The commanded velocity is 1 m/s forward §5. Metrics §5:

• Success rate: the fraction of environments that do not terminate before the episode ends. Reported values lie in [0, 1].
• Tracking: the Euclidean distance between the current and the commanded velocity. Lower is better.
• Smoothness: the sum of squared L2 norms of consecutive actions. Lower is better. The unit is not stated.

Hyperparameters

Values as stated in the paper. "Not stated" means the paper gives no value.

Domain randomization during AMP training tab. 2:

Regularization rewards tab. 1:

Dynamics randomization during evaluation (randomized target) tab. 4:

Results

Author-reported. Setup: IsaacGym, simulated Unitree H1, three dataset sizes, eight randomization setups, three seeds per DP, and the source RL policy as baseline §5 §3.

Headline comparison, using the 8M "All" DP that the authors call their best §5.1 §5.2:

The full results follow. Each cell is mean ± the spread printed in the paper. The paper does not define the spread. Rows are dataset size and metric, columns are randomization setups.

Non-randomized target fig. 2, flat ground without randomization §5.1:

Source RL policy on this target, 100 runs tab. 3: success 1.0; tracking 0.12 ± 0.0; smoothness 2.89 ± 0.02.

Randomized target fig. 3, milder terrain and dynamics randomization §5.2:

Source RL policy on this target, 100 runs tab. 5: success 1.0; tracking 0.13 ± 0.02; smoothness 3.45 ± 0.36.

Text claims §5.1 §5.2 §6:

• Non-randomized target, 500K: no DP walks stably, even with randomization.
• Non-randomized target, 2M: some setups walk robustly. Perturbations, terrain and terrain + perturbations reach success near 1.0. Scales and initial state do not.
• Non-randomized target, 8M: most setups are strong. Dynamics randomization reaches 0.95 success.
• Randomized target, 2M: the text says perturbations give strong results and the other setups perform poorly. See Pitfalls for the mismatch with the table.
• Randomized target, 8M: perturbation and terrain randomization are strongest. Dynamics randomization reaches 0.49.
• Scale randomization loses performance on the randomized target at 8M, from 0.82 and 0.79 success for scales and initial state fig. 3.

Paper vs code

No official code. The abstract links a project page, https://sites.google.com/view/dps-for-humanoid-control, which the paper's text does not mention. The project page contains no code link. The GitHub repository search for the title words, for "dps-for-humanoid-control" and for "diffusion policy humanoid isaacgym amp" returned nothing. A GitHub user search for "kaidanov" returned accounts that could not be tied to the authors, and none was used.

Pitfalls

• The DP is under-specified. The paper gives the encoder and decoder sizes, the conditioning and the loss. It gives no diffusion-step count, noise schedule, prediction horizon n, optimizer, learning rate, batch size, epoch count or action normalization. It defers these to DiffuseLoco §4. An implementer must either take them from DiffuseLoco or mark them as unknown.
• The RL policy is under-specified. The reward weights w_v and w_ω, the discount γ, the style-reward weight, the PD gains, the control frequency and the RL algorithm settings are not stated §3.1. The 500 steps in a 10-second evaluation suggest a 50 Hz control rate, but the paper does not say so.
• The "All" column is ambiguous. The active results tables label the final column "All" fig. 2 fig. 3. Commented-out versions of the same tables label it "All but scales" (LaTeX source, main.tex lines 461 and 487). The text does not say whether "All" includes humanoid-scale randomization. The reported gap between "All" and "Terrain + pert." is small at 8M, so the question matters for the headline numbers.
• Text and table disagree on the randomized 2M results. The text says that at 2M only perturbations give strong results, and that the other configurations perform poorly §5.2. The table shows success 0.96 for terrain + perturbations and 0.99 for "All", while terrain alone gives 0.57 fig. 3. Use the table values.
• The randomized-target results have a caption error. The caption of the randomized-target figure contains two sentences, one about the randomized and one about the non-randomized target fig. 3. The table itself is for the randomized target.
• Training and evaluation randomization ranges differ. Compare tab. 2 with tab. 4. The added base mass is [−2, 2] kg in training and [−1, 1] kg in evaluation. The evaluation also drops external perturbations, and it reduces the dynamics ranges §5.2. The results on the randomized target are therefore not a direct test of the training distribution.
• Scale randomization was not trained as a separate policy. The paper first trained three per-scale RL policies. It then used a single policy trained at one scale for all data collection, because it stayed robust across scales §3.2. A reader expecting per-scale experts will find a different setup.
• Terrain and initial-state parameters are missing. The terrain obstacle and bump parameters, and the initial-state joint range, are not stated §3.2. The results cannot be reproduced exactly from the paper.
• Std spread is undefined. The "±" values are printed without the statistic they describe. The paper evaluates each DP with 3 seeds §5, and the number of evaluation runs per DP is not stated. The baseline is reported over 100 runs tab. 3 tab. 5. The two sample sizes are not directly comparable.
• The smoothness unit is not stated. Smoothness is the sum of squared L2 norms of consecutive actions §5, and values range from about 2.9 to 277. Compare only within the same table.
• Tracking and training reward differ. Training rewards yaw rate and lateral velocity §3.1. The reported tracking metric is the Euclidean distance between current and commanded velocity, and the evaluation command is forward only at 1 m/s §5. Read the tracking metric as a forward-speed error.
• Success-rate definition. The paper describes success as a count of evaluation environments that survive to the end of the episode §5. The table values are between 0 and 1, so this is presumably a fraction. The denominator is not stated.
• Collective training is mentioned but not reported. The paper says the datasets are used to train DPs "both individually and collectively" §3.2. No collective-training result appears in the results section §5.
• Dataset size is in transitions, not episodes. The episode count per dataset is not stated §3 §5.
• Costs. Training takes about 23 hours 17 minutes on one V100-16GB for the 8M dataset. About 5 hours are needed for 2M §5.
• Figure and table numbers come from source order. This card numbers figures and tables by the order of their captions in the LaTeX source. The compiled PDF was not checked, so check fig. 1–fig. 3 and tab. 1–tab. 5 against the published version.

Lineage

• Builds on: AMP, the adversarial motion-prior method used to train the source RL policy §3. Its domain-randomization approach for dynamics follows an earlier sim-to-real work cited in §3.2.
• Builds on: DiffuseLoco, which the paper follows for the diffusion-policy architecture and the receding-horizon control §2 §4.
• Related: diffusion policies for manipulation, which the paper contrasts with its locomotion setting §1 §6.
• Compared against: only the source RL policy, as a baseline on both targets §5 tab. 3 tab. 5. No other DP method is compared.
• Superseded by: not known from this wiki.