Adversarial imitation learning

Overview

The policy is trained with a task reward and a second, learned reward that says how much a transition looks like the reference data. A discriminator is trained to tell reference transitions from transitions the policy produced, and the policy is rewarded for making its transitions look like the reference 2203.15103 §3.2. Because the style term comes from data, the designer does not have to write each style term by hand 2203.15103 §1. The paper frames this as matching the state distribution of a dataset instead of tracking individual reference clips 2203.15103 §2.3.

Only one paper card currently lists this concept, so the variants and trade-offs below are those of that single paper.

Variants

Adversarial motion priors (AMP)

The style reward is a bounded function of the discriminator's score on a (state, next state) pair. The discriminator is trained with a least-squares objective: reference pairs toward +1, policy pairs toward −1, plus a gradient penalty on the reference pairs 2203.15103 §3.2 2203.15103 §3.6. The reference data is about 4.5 s of dog motion capture retargeted to a quadruped 2203.15103 §3.3. The style reward is mixed with a velocity-tracking task reward, weighted 0.35 task and 0.65 style, and the policy is trained with PPO 2203.15103 §3.6 2203.15103 §3.2. Card: Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions.

When to use which

Implementation sketch

The loop follows the paper's description: the policy collects transitions, the combined reward is computed for each transition, and the discriminator is trained on reference transitions against recent policy transitions. Notes mark where the paper's choices sit.

# Style reward from the discriminator  [arxiv:2203.15103 §3.2] [eq. 4]
# disc: discriminator; s, s_next: state features of one transition

def style_reward(disc, s, s_next):
    # Raw discriminator score for the transition  [§3.2]
    score = disc(s, s_next)
    # Bounded to [0, 1]; highest near a score of +1  [eq. 4]
    return max(0.0, 1.0 - 0.25 * (score - 1.0) ** 2)

def total_reward(task_r, style_r):
    # Paper weights: 0.35 task, 0.65 style  [§3.6] [eq. 2]
    return 0.35 * task_r + 0.65 * style_r

# Least-squares discriminator loss with gradient penalty  [eq. 3] [§3.6]
# ref_pairs: transitions from reference motion
# policy_pairs: transitions from the current policy

def disc_loss(disc, ref_pairs, policy_pairs, w_gp=10.0):
    # Reference transitions are pushed toward +1  [eq. 3]
    ref_term = mean((disc(ref_pairs) - 1.0) ** 2)
    # Policy transitions are pushed toward -1  [eq. 3]
    pol_term = mean((disc(policy_pairs) + 1.0) ** 2)
    # Penalise the gradient norm on reference pairs, weight 10  [§3.6]
    penalty = (w_gp / 2.0) * mean(grad_norm_sq(disc, ref_pairs))
    return ref_term + pol_term + penalty

Policy and discriminator are updated in one loop: PPO on the combined reward, then the discriminator update on reference and policy batches 2203.15103 §3.2 2203.15103 §3.6. The paper does not state the rollout length; the number of transitions per iteration is given in 2203.15103 §3.6.

Contradictions

Within the paper, the text and the results table disagree on one number. The text gives the complex-reward cost of transport as 1.37 to 1.65 2203.15103 §4.1. The table gives 1.37 to 1.54 2203.15103 tab. 2. No second paper in the wiki reports on this method, so no disagreement between papers is known. Compare the simulated numbers only within the paper's own setup, since the cost of transport keeps only positive joint power 2203.15103 §4.1.

Papers

• Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions: a learned discriminator style reward, mixed with a velocity-tracking reward, for quadruped locomotion, compared with a 13-term hand-designed style reward in simulation.