---
kind: concept
slug: adversarial-imitation-learning
title: Adversarial imitation learning
summary: >-
  Training a policy to match reference behaviour by rewarding it for fooling a discriminator that separates reference transitions from policy transitions. The learned style reward replaces many hand-designed terms, and the trade-offs are a reference dataset, adversarial training and simulation-only comparisons. Currently sourced from one paper card (AMP).
tags: [reinforcement-learning, imitation-learning, legged-locomotion]
papers: ["2203.15103"]
related: [imitation-learning, inverse-reinforcement-learning, reward-design, learning-from-demonstration, sim-to-real-transfer, robot-simulation, proximal-policy-optimization, domain-randomization]
status: draft
updated_at: 2026-11-08
---

## Overview

The policy is trained with a task reward and a second, learned reward that
says how much a transition looks like the reference data. A discriminator is
trained to tell reference transitions from transitions the policy produced,
and the policy is rewarded for making its transitions look like the
reference [arxiv:2203.15103 §3.2]. Because the style term comes from data, the
designer does not have to write each style term by hand [arxiv:2203.15103 §1].
The paper frames this as matching the state distribution of a dataset instead
of tracking individual reference clips [arxiv:2203.15103 §2.3].

Only one paper card currently lists this concept, so the variants and
trade-offs below are those of that single paper.

## Variants

### Adversarial motion priors (AMP)

The style reward is a bounded function of the discriminator's score on a
(state, next state) pair. The discriminator is trained with a least-squares
objective: reference pairs toward +1, policy pairs toward −1, plus a gradient
penalty on the reference pairs [arxiv:2203.15103 §3.2] [arxiv:2203.15103 §3.6].
The reference data is about 4.5 s of dog motion capture retargeted to a
quadruped [arxiv:2203.15103 §3.3]. The style reward is mixed with a
velocity-tracking task reward, weighted 0.35 task and 0.65 style, and the
policy is trained with PPO [arxiv:2203.15103 §3.6] [arxiv:2203.15103 §3.2].
Card: [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](/papers/2203.15103).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Legged locomotion, a reference motion is available, and a hand-tuned multi-term style reward is hard to design | AMP-style adversarial style reward mixed with the task reward | The paper replaces a 13-term hand-designed style reward with one learned term and reports lower simulated cost of transport across 0.4 to 1.6 m/s [arxiv:2203.15103 §4.1] [arxiv:2203.15103 tab. 2] |
| No reference motion or demonstration data | Not this method | The discriminator needs reference transitions to learn from [arxiv:2203.15103 §3.2]. The paper does not cover a setting without them |
| Need a measured real-robot success rate | Plan a separate evaluation | The paper's real-robot results are qualitative. It reports no numeric real-robot metric [arxiv:2203.15103 §4.2] [arxiv:2203.15103 §4.3] |
| Need a comparison against hand-designed rewards on hardware | Treat the paper's comparison as simulation-only | The no-style-reward baseline was only evaluated in simulation [arxiv:2203.15103 §4.1] |

## Implementation sketch

The loop follows the paper's description: the policy collects transitions, the
combined reward is computed for each transition, and the discriminator is
trained on reference transitions against recent policy transitions. Notes
mark where the paper's choices sit.

```pseudocode
# Style reward from the discriminator  [arxiv:2203.15103 §3.2] [eq. 4]
# disc: discriminator; s, s_next: state features of one transition

def style_reward(disc, s, s_next):
    # Raw discriminator score for the transition  [§3.2]
    score = disc(s, s_next)
    # Bounded to [0, 1]; highest near a score of +1  [eq. 4]
    return max(0.0, 1.0 - 0.25 * (score - 1.0) ** 2)

def total_reward(task_r, style_r):
    # Paper weights: 0.35 task, 0.65 style  [§3.6] [eq. 2]
    return 0.35 * task_r + 0.65 * style_r
```

```pseudocode
# Least-squares discriminator loss with gradient penalty  [eq. 3] [§3.6]
# ref_pairs: transitions from reference motion
# policy_pairs: transitions from the current policy

def disc_loss(disc, ref_pairs, policy_pairs, w_gp=10.0):
    # Reference transitions are pushed toward +1  [eq. 3]
    ref_term = mean((disc(ref_pairs) - 1.0) ** 2)
    # Policy transitions are pushed toward -1  [eq. 3]
    pol_term = mean((disc(policy_pairs) + 1.0) ** 2)
    # Penalise the gradient norm on reference pairs, weight 10  [§3.6]
    penalty = (w_gp / 2.0) * mean(grad_norm_sq(disc, ref_pairs))
    return ref_term + pol_term + penalty
```

Policy and discriminator are updated in one loop: PPO on the combined reward,
then the discriminator update on reference and policy batches
[arxiv:2203.15103 §3.2] [arxiv:2203.15103 §3.6]. The paper does not state
the rollout length; the number of transitions per iteration is given in
[arxiv:2203.15103 §3.6].

## Contradictions

Within the paper, the text and the results table disagree on one number. The
text gives the complex-reward cost of transport as 1.37 to 1.65 [arxiv:2203.15103 §4.1].
The table gives 1.37 to 1.54 [arxiv:2203.15103 tab. 2]. No second paper in the
wiki reports on this method, so no disagreement between papers is known.
Compare the simulated numbers only within the paper's own setup, since the
cost of transport keeps only positive joint power [arxiv:2203.15103 §4.1].

## Papers

- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](/papers/2203.15103): a learned discriminator style reward, mixed with a velocity-tracking reward, for quadruped locomotion, compared with a 13-term hand-designed style reward in simulation.