Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

In plain words

Teaching a simulated legged robot to walk usually means writing a long list of reward terms by hand: reward the forward speed, but also punish shaking, jerky joints, foot slips and wasted energy. Each term needs tuning, and the result often looks unnatural and fails on real hardware.

This paper swaps most of that list for one learned "style" score. The score comes from a few seconds of motion capture of a dog. A small neural network, the discriminator, learns to tell dog motion apart from the robot's motion, and the robot is rewarded for looking like the dog. The style score is then mixed with a simple goal reward, such as "move at this speed".

The learned gaits include pacing at low speed and trotting and cantering at higher speeds, and the policies use less energy than the hand-tuned baselines in simulation. The same approach was run on a real quadruped robot.

TL;DR

• Learns a style reward with an adversarial discriminator (AMP) from about 4.5 s of dog motion capture §3.3.
• Trains with PPO on velocity-command tracking plus the learned style reward, weighted 0.35 task and 0.65 style §3.6.
• Author-reported: lower mechanical cost of transport in simulation (0.93 to 1.12) than a 13-term hand-designed style reward (1.37 to 1.54) tab. 2 §4.1.
• Gait transitions (pace to canter) emerge with speed §4.3 fig. 2. Real-robot evidence is qualitative only.
• Code: official repository, BSD-3-Clause, built on legged_gym and rsl_rl. Several hyperparameters differ from the paper (see Paper vs code).

Problem

Under-specified reward functions lead simulated legged agents to learn violent or physically implausible behaviour, which transfers badly to hardware §1. The usual fix is a hand-designed reward with many style terms, which needs domain knowledge and tuning per platform and task §1 §2.1. Motion imitation by tracking clips is an alternative, but it constrains the policy to the reference clips and needs motion planning and annotation when the data is diverse §2.2. Adversarial imitation learning matches the state distribution of a dataset rather than tracking clips §2.3, and the paper asks whether an adversarial motion prior can give real-robot policies without complex rewards §1.

Method

The task is a Markov decision process over legged-robot states. The policy tracks a commanded base velocity (forward, lateral and yaw) through a velocity-tracking reward §3.1 eq. 1. The total reward is a weighted sum of that task reward and a style reward eq. 2 §3.2. The style reward is produced by a discriminator trained on (state, next state) pairs: reference pairs from motion capture against pairs produced by the policy §3.2 eq. 3. The discriminator output is mapped to a reward in [0, 1] eq. 4.

Reference data is a set of German Shepherd clips (pacing, trotting, cantering, turning in place; 4.5 s in total), retargeted to the A1 robot with inverse kinematics and forward kinematics §3.3. Discriminator states are joint angles, end-effector positions, joint velocities and base velocities, computed from finite differences §3.3. The policy is trained with PPO and the discriminator with its least-squares objective, in one loop §3.2 §3.6.

# Style reward and combined reward  [§3.2] [eq. 2] [eq. 4]
# disc: discriminator; state, next_state: (N, 43) tensors (see [§3.3])
# Output: per-environment reward (N,)

def style_reward(disc, state, next_state):
    # Scalar discriminator score for each transition  [§3.2]
    score = disc(state, next_state)
    # Bound to [0, 1]: close to 1 when the score is near +1  [eq. 4]
    return max(0.0, 1.0 - 0.25 * (score - 1.0) ** 2)

def combined_reward(disc, task_reward, state, next_state):
    # Weighted sum of task and style rewards  [eq. 2] [§3.6]
    style = style_reward(disc, state, next_state)
    return 0.35 * task_reward + 0.65 * style

# Discriminator objective, minimised  [§3.2] [eq. 3]
# ref_pairs: transitions from motion capture
# policy_pairs: transitions produced by the policy

def discriminator_loss(disc, ref_pairs, policy_pairs, w_gp=10.0):
    # Reference transitions should score +1  [eq. 3]
    ref_term = mean((disc(ref_pairs) - 1.0) ** 2)
    # Policy transitions should score -1  [eq. 3]
    policy_term = mean((disc(policy_pairs) + 1.0) ** 2)
    # Penalise the gradient norm on reference pairs  [eq. 3] [§3.2]
    grad = gradient_of(disc(ref_pairs), ref_pairs)
    penalty = (w_gp / 2.0) * mean(norm(grad) ** 2)
    return ref_term + policy_term + penalty

# One PPO iteration with the discriminator in the loop  [§3.6]
# Paper: 5280 environments, 126,720 transitions per iteration

def train_iteration(policy, disc, env, ref_data):
    # Collect transitions with the current policy  [§3.4]
    for step in range(steps_per_env):
        # Target joint angles; PD controllers turn them into torques  [§3.4]
        action = policy.sample(obs)
        next_obs, next_state, task_reward = env.step(action)
        # Style and task rewards for the transition  [§3.2] [eq. 2]
        reward = combined_reward(disc, task_reward, state, next_state)
        buffer.add(obs, action, reward, next_obs, state, next_state)
        obs, state = next_obs, next_state
    # Advantages and returns with PPO  [§3.6]
    compute_advantages(buffer)
    # Five epochs over minibatches; policy and discriminator  [§3.6]
    for epoch in range(5):
        for batch in minibatches(buffer, ref_data):
            policy_loss = ppo_loss(policy, batch)
            disc_loss = discriminator_loss(disc, batch.ref, batch.policy)
            optimise(policy_loss + disc_loss)

steps_per_env = 24 is implied, not stated: 126,720 transitions divided by 5,280 environments §3.6. The paper does not give the number of rollout steps.

The paper optimises the policy for return and the discriminator with its own objective §3.2. The code sums the PPO loss and the discriminator loss into one loss with one backward pass amp_ppo.py.

Hyperparameters

Values are as stated in the paper unless marked "code". Code values are from the shipped A1 configuration (a1_amp) and are not in the paper.

Results

Author-reported, in simulation, on the A1 robot with commanded forward velocities of 0.4 to 1.6 m/s. Baselines: a 13-term hand-designed style reward from Rudin et al. §4.1 tab. 3, and no style reward. The ± values are as printed; the paper does not say what they measure.

• Cost of transport is the paper's mechanical definition: positive joint power over weight times speed §4.1.
• Qualitative: with a command rising from 1 to 2 m/s the policy switches from pacing to a canter §4.3 fig. 2. A 0.8 m/s pacing and 1.7 m/s trotting example is shown fig. 3.
• Sinusoidal linear and yaw commands are tracked by the AMP policy; the no-style policy was only run in simulation §4.2 fig. 6.
• Real-robot results are described in words and figures only. No real-robot numbers appear in the text §4.3.

Paper vs code

Repository searched: the paper's footnote link (bit.ly) redirects to the project page, whose "Code" link points to Alescontrela/AMP_for_hardware. That URL returns 404 now. A GitHub search for AMP_for_hardware found escontra/AMP_for_hardware (owner login matches the first author's surname; its description repeats this paper's title). I cloned that repository and treated it as the authors' code, which is an inference to verify. The repository is at commit bfb0dbdcf32bdf83a916790bddf193fffc7e79b8.

Discrepancies found, each with both citations:

• Style and task weights. Paper weights are 0.65 (style) and 0.35 (task) §3.6. The code mixes 0.7 style with 0.3 task, and multiplies the style term by 2.0 a1_amp_config.py amp_discriminator.py. So the style reward range is [0, 2], not [0, 1] eq. 4.
• Discriminator loss scale. Paper eq. 3 has no factor on the squared least-squares terms eq. 3. The code multiplies the sum of both by 0.5 amp_ppo.py. Relative to the adversarial terms, the code's gradient-penalty weight is 20, against 5 implied by eq. 3 (w_gp / 2 = 5).
• Gradient-penalty target. Eq. 3 writes the gradient with respect to the discriminator parameters φ eq. 3. The code takes the gradient of the score with respect to the input pair amp_discriminator.py.
• Discriminator activation. Paper: ELU §3.4. Code: ReLU amp_discriminator.py.
• Policy std. Paper: initial std 0.25, and the policy outputs the std §3.4. Code: the std is a learned parameter that does not depend on the state, with initial value 1.0 actor_critic.py legged_robot_config.py.
• Control rate. Paper: 30 Hz §3.4. Code: 0.03 s per policy step (33.3 Hz) a1_amp_config.py legged_robot.py.
• Environment count and minibatch. Paper: 5280 environments and 21,120-transition minibatches §3.6. Code: 5480 environments a1_amp_config.py and 4 minibatches, which gives about 32,880 transitions a1_amp_config.py.
• Domain randomisation. Friction range, push speed and gain range all differ from tab. 1. See the Hyperparameters table. The code's push is a fixed schedule (every 15 s, first at step 0) with a random velocity legged_robot.py legged_robot.py. The paper says "random intervals" §3.5. Kp and Kd are drawn independently per actuator and per environment legged_robot.py.
• Policy observation. The paper lists joint angles, joint velocities, orientation and previous actions §3.4. The code's policy observation also includes the velocity command and gravity direction, and drops base linear and angular velocity legged_robot.py a1_amp_config.py.
• Discriminator state. The paper's state lists joint angles, end-effector positions, joint and base velocities §3.3. The code adds the base height, giving 43 values per state legged_robot.py motion_loader.py.
• Discriminator optimiser and input. The code uses separate weight decay for the trunk (1e-3) and head (1e-1), inputs normalised with running statistics, and a replay buffer of 1,000,000 policy transitions. None are mentioned in the paper amp_ppo.py amp_ppo.py amp_ppo.py.
• Reward in the rollout. The style reward is computed in the runner and then mixed with the task reward inside the discriminator's predict_amp_reward amp_on_policy_runner.py amp_discriminator.py.
• Task reward form. Paper eq. 1 uses an unsquared norm for linear error and an absolute value for yaw error eq. 1. The code uses squared errors for both, inside exp with width 0.25 legged_robot.py legged_robot_config.py.
• Reference-state probability. The config sets reference_state_initialization_prob = 0.85, but nothing reads it. Every reset uses reference frames legged_robot.py.
• Motion data. The paper describes one retargeted German Shepherd dataset of 4.5 s §3.3. The repository ships six clips with frame durations and weights pace0.txt trot0.txt. No retargeting script is in the repository.
• Non-AMP baseline. Table 3's 13-term reward is not shipped as a config. The closest config, A1RoughCfg, uses torque penalty −2e-4 and joint-limit penalty −10, decimation 4, and Kp 20 / Kd 0.5 a1_config.py (compare tab. 3).
• Cost of transport. No script in the repository computes cost of transport. The formula appears only in the paper §4.1.
• Table versus text. The text gives the complex-reward cost of transport as 1.37 to 1.65 §4.1, but the table's values run from 1.37 to 1.54 tab. 2.
• Code licence. The repository LICENSE is BSD-3-Clause and keeps the ETH Zurich and NVIDIA copyright lines from legged_gym LICENSE.

Pitfalls

• Discriminator weights. Implementing eq. 2 and eq. 4 literally gives a different balance from the code. Use the code's 0.7 / 0.3 with a 2.0 scale if you want to reproduce the released behaviour, and say which you use.
• Gradient-penalty target. Use the gradient with respect to the input pair, not the parameters, as in the code. Eq. 3's notation suggests the parameters.
• Replay buffer. The discriminator sees policy transitions from a buffer of up to 1,000,000 past transitions, not only the current rollout. This is a stabilising choice the paper does not describe a1_amp_config.py.
• Terminal states. On a reset, the reward uses the terminal state before the reset, not the first state of the new episode amp_on_policy_runner.py.
• Time-outs. The value bootstrap at time-outs is added to the reward amp_ppo.py. The paper does not mention it.
• Shared optimiser. The discriminator and policy use one Adam optimiser and one learning rate. The learning rate is adjusted by the policy's KL alone amp_ppo.py.
• Expert sampling. Reference transitions are sampled from 2,000,000 preloaded pairs, weighted by each clip's MotionWeight, with the time step set to the environment's dt a1_amp_config.py motion_loader.py amp_on_policy_runner.py.
• Unstated values. PD gains, action scale, default posture, discount, clip ratio, entropy coefficient, learning-rate start and gradient clipping are not in the paper. The code values above are one set that works, not something the paper validates.
• Sim-only evidence for the baseline. The no-style policy shows violent vibration and was only evaluated in simulation §4.1 fig. 5. The comparison with real-robot behaviour is qualitative.
• Sim-to-real claims. The sections labelled "real" describe policies run on the robot, but no numeric real-robot metric is reported in the text §4.2 §4.3. Treat transfer as demonstrated qualitatively, not measured.
• Cost of transport. The formula keeps only the positive part of joint power §4.1, so the numbers depend on that choice. Compare them only within this setup.
• Compute. About 4 billion simulation steps on 5280 environments, about 16 hours on one V100 §3.6.

Lineage

• Builds on: Adversarial Motion Priors (Peng et al., cited as [peng21] in the paper, ACM TOG 2021). No arXiv id is given in the source and no card exists here. It supplies the discriminator objective and style reward §3.2.
• Builds on: DeepMimic (Peng et al., [peng18]) for motion retargeting and reference-state initialisation §3.3 §3.5. No card exists here.
• Compared against: the 13-term complex style reward of Rudin et al. [arXiv:2109.11978] and no style reward §4.1 tab. 3.
• Superseded by: not known from content already in this wiki.