---
kind: concept
slug: inverse-reinforcement-learning
title: Inverse reinforcement learning
summary: >-
  Inferring a reward function from demonstrations. Covered: a maximum-likelihood observer that assumes the demonstrator is doing the task, a pedagogical variant that models a teacher who shows it, cooperative IRL, and adversarial (discriminator-based) IRL. Behaviour alone under-determines the reward. Results are author-reported, mostly on small or lab-collected data.
tags: [reward-learning, teaching, human-behaviour, alignment, adversarial-learning]
papers: ["1811.07871", "2309.02473", "2411.03817", "2412.11360", "2610.14004"]
related: [imitation-learning, learning-from-demonstration, preference-optimization, rlhf, reward-modeling, motion-retargeting, llm-agent, generative-adversarial-network]
status: draft
updated_at: 2026-11-01
---

## Overview

Inverse reinforcement learning (IRL) takes demonstrations and returns a reward
function that explains them. The idea is the inverse of planning: an RL
algorithm maps a reward to behaviour, and IRL maps behaviour back to a reward
[arxiv:1811.07871 §7.2]. Two problems follow. The reward is under-determined
by behaviour alone: R = 0 is always a solution, and recovering a unique answer
needs extra assumptions about the demonstrator [arxiv:1811.07871 §7.2]. The
survey puts it as ambiguity: a policy can be optimal for many rewards, so an
IRL method must add structure to choose one [arxiv:2309.02473 §III].

A second issue is that IRL usually assumes the demonstrator is doing the task
well, not that they are trying to make the reward easy to learn
[arxiv:2610.14004 §1]. The observer then needs a model of the demonstrator.
The standard choice is a softmax planner: the demonstrator picks actions with
probability that grows with their value, controlled by a temperature
[arxiv:2610.14004 §2.1.1] [eq. 2]. The observer scores each candidate reward
by the probability of the trajectory under that model, with a uniform prior
over rewards [arxiv:2610.14004 §2.1.2] [eq. 3].

Five sources cover this page. The survey
[A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](/papers/2309.02473)
places IRL inside imitation learning. It also lists repeated environment
interaction and a known transition model as practical limits of IRL
[arxiv:2309.02473 §III]. The position paper
[Scalable agent alignment via reward modeling](/papers/1811.07871) treats IRL
as one way to learn a reward from behaviour, and lists its weaknesses as
motivation for other approaches [arxiv:1811.07871 §7.2]. The study
[Showing versus Doing: Teaching by Demonstration](/papers/2610.14004) asks
what changes when the demonstrator is told to show the task instead of doing
it. Its answer is that the trajectories differ, and that a standard reward
learner recovers the reward better from the showing ones
[arxiv:2610.14004 §4.2.2] [arxiv:2610.14004 §4.3]. The study's own learner is
still standard IRL. The new part is a model of the teacher. The last two
sources use a discriminator as the reward signal: one for robot manipulation
[Visual IRL for Human-Like Robotic Manipulation](/papers/2412.11360) and one
for LLM agents [From Novice to Expert: LLM Agent Policy Optimization via
Step-wise Reinforcement Learning](/papers/2411.03817).

## Variants

The survey sorts IRL methods by how they choose one reward out of the many
that fit: maximum margin, maximum entropy, and Bayesian [arxiv:2309.02473 §III].
The variants below are cut differently, by demonstrator model and reward
signal. The Bayesian family is the closest match to the study's observer,
which keeps a uniform prior over rewards and scores each by trajectory
likelihood [arxiv:2610.14004 eq. 3]. That match is this page's reading, not
the survey's.

### Maximum-likelihood IRL with a doing model

The observer assumes the demonstrator is doing the task under a softmax
policy, and learns reward weights by maximum likelihood. The paper calls this
MLIRL. For its feature-reward task, the non-goal weights are constrained to
be non-positive [arxiv:2610.14004 §4.3]. The paper uses this learner on both
human conditions, Do and Show, and on model-generated trajectories
[arxiv:2610.14004 §4.3] [fig. 2]. Card: [Showing versus Doing: Teaching by Demonstration](/papers/2610.14004).

### Pedagogical IRL: the observer models a teacher who shows

The observer does not assume the trajectory came from doing. It assumes a
teacher chose the trajectory to make the true reward easiest to infer. The
teacher picks trajectories with a softmax over the observer's posterior,
raised to a teacher parameter alpha [arxiv:2610.14004 §2.3] [eq. 5]. The
candidate trajectories are built by enumeration: they are kept if their length
is at most l_max and their probability under some candidate reward's doing
policy is at least p_min [arxiv:2610.14004 alg. 1]. The paper uses this model
to predict human Show behaviour and compares it with the standard planner
[arxiv:2610.14004 §4.2.1] [fig. 3]. Card: [Showing versus Doing: Teaching by Demonstration](/papers/2610.14004).

This variant answers the position paper's objection that IRL assumes the
human optimises reward directly, even when a demonstration would be a more
efficient way to communicate it [arxiv:1811.07871 §7.2]. The study does not
cite that objection; the link is this page's reading.

### Cooperative IRL: a two-player game with a shared prior

Cooperative IRL (CIRL) is a formal model of reward learning as a game between
a user and an agent. Both share a prior over the user's reward function. The
user observes the reward; then both act to optimise it. The optimal play
trades off communicating the reward against maximising expected reward
[arxiv:1811.07871 §7.3]. The position paper uses CIRL as a comparison point
and does not evaluate it [arxiv:1811.07871 §7.3]. Card: [Scalable agent alignment via reward modeling](/papers/1811.07871).

### Adversarial IRL: a discriminator supplies the reward

A discriminator is trained to tell expert (state, action) pairs from pairs
produced by the current policy. The reward is computed from the discriminator
output, and the policy is optimised against that reward. The survey describes
the reward as coming from the discriminator's confusion, and says the approach
avoids solving a full RL problem at every iteration. It also notes that the
min-max training can suffer from vanishing gradients and failure to converge
[arxiv:2309.02473 §IV]. Visual IRL uses this form, called AIRL. Its reward is
the log-odds of the discriminator output, and its policy is optimised with
TRPO [arxiv:2412.11360 §IV.A] [eq. 2] [eq. 4]. The paper applies it to
per-frame 3D keypoints and object positions from video of human manipulation
[arxiv:2412.11360 §IV.A]. Card: [Visual IRL for Human-Like Robotic Manipulation](/papers/2412.11360).

StepAgent applies a GAIL-style discriminator to the steps of an LLM agent. It
trains the discriminator to separate agent steps from expert steps, then uses
log D as a per-step reward for PPO [arxiv:2411.03817 §4.2.2] [eq. 10]
[alg. 1]. Expert steps come from recorded trajectories, and each agent step is
sampled from the same prefix [arxiv:2411.03817 §4.1]. Card: [From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning](/papers/2411.03817).

Reported results are the paper's own. Visual IRL reports a learned-behaviour
accuracy of 90.4% against 83.3% for MAP-BIRL fed with discrete state-action
trajectories. The text gives this comparison, with no table [arxiv:2412.11360 §V.D].
StepAgent's inverse-RL variant reaches the same range as its implicit-reward
variant, with differences of 0.1 to 0.7 points in every cell, from one run per
setting [arxiv:2411.03817 tab. 2]. The step-wise discriminator is one of two
ways that paper uses expert steps. The other is a DPO-style implicit reward,
which uses no discriminator [arxiv:2411.03817 §4.2.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Demonstrations are ordinary task performance | Maximum-likelihood IRL with a doing model | In the study's Experiment 2 the standard planner fits the Do condition better than the pedagogical model does [arxiv:2610.14004 §4.2.1] [fig. 3] |
| Demonstrator was asked to show, so the goal should be easy to read | Expect different paths. Consider a teacher-aware model | Show-instructed people took more varied, looping paths [arxiv:2610.14004 §4.2.2]. Pedagogical fits favour Show in the plot only [fig. 3] |
| Behaviour alone may not pin down the reward | Add assumptions about the demonstrator, or a model where the teacher is informative | Without extra assumptions the reward is not unique [arxiv:1811.07871 §7.2]. The teacher model is one such assumption [arxiv:2610.14004 §2.3] |
| The agent and user can act together and the reward is the goal | Cooperative IRL | The game gives the user a reason to communicate the reward [arxiv:1811.07871 §7.3] |
| Need to simulate a teacher's demonstrations | Pedagogical teacher model | The study's model, but the teacher code is not released, so only its equations are available [arxiv:2610.14004 alg. 1] |
| Demonstrator may be suboptimal | Use a method that weights demonstrations | The survey lists weighting methods that use confidence scores or discriminator-based weights. One reweighted BC scheme converges only when optimal demonstrations are the majority [arxiv:2309.02473 §VI-A] |
| Environment interaction is expensive or unsafe | Consider the survey's behavioural-cloning families instead of IRL | IRL needs repeated environment interaction to estimate the reward [arxiv:2309.02473 §III]. BC is supervised learning and suffers covariate shift [arxiv:2309.02473 §II] |
| The transition model is unknown | Check the IRL method's assumptions before choosing it | The survey lists a known transition model as an IRL assumption [arxiv:2309.02473 §III] |
| Demonstrations are many and the trajectory bound is large | Plan for enumeration cost | The trajectory set grows with l_max. The study gives no runtime or memory figure [arxiv:2610.14004 alg. 1] [app. 2] |
| Demonstrations are video or observations, and a discriminator-based reward is wanted | Adversarial IRL with TRPO | Visual IRL applies it to keypoints and object positions and optimises its policy with TRPO [arxiv:2412.11360 §IV.A]. The survey warns that training can be unstable [arxiv:2309.02473 §IV] |
| LLM agent with expert trajectories, no step labels, and the base model can be sampled at any prefix | Step-wise discriminator or implicit DPO reward | Both need no human step labels; the two variants score within 0.1 to 0.7 points of each other [arxiv:2411.03817 tab. 2] [§4.1] |

Within the study, the showing demonstrations did not earn more reward for the
participant: total reward won showed no difference between conditions
[arxiv:2610.14004 §4.2.2]. Use that before assuming showing is a cost or a
gain to the demonstrator.

## Implementation sketch

Observer posterior over candidate rewards, from the doing model [eq. 3].
Pseudocode in our own words; the teacher step is written from the equations.

```pseudocode
# Observer posterior over candidate rewards  [arxiv:2610.14004 §2.1.2] [eq. 3]
# Input:  trajectories j, candidate rewards R_1..R_N, uniform prior
# Output: P(R_i | j) for each trajectory j
def observer_posterior(trajectories, rewards, doing_likelihood):
    # likelihood[i][j] = P_Doing(j | R_i) under a softmax planner  [eq. 2]
    likelihood = doing_likelihood(rewards, trajectories)
    # Uniform prior: normalise over rewards for each trajectory
    return likelihood / likelihood.sum(axis=0)

# Pedagogical teacher over trajectories for each reward  [eq. 5]
# Input:  posterior P(R_i | j), teacher parameter alpha
# Output: P_Showing(j | R_i), normalised over trajectories
def showing_distribution(posterior, alpha):
    raised = posterior ** alpha
    return raised / raised.sum(axis=1, keepdims=True)
```

Keep the candidate trajectory set finite. Algorithm 1 uses the length bound
l_max and the probability floor p_min to do this [arxiv:2610.14004 alg. 1].

Adversarial variant: the reward is computed from a discriminator, not from a
planner. This is the AIRL form that Visual IRL uses [arxiv:2412.11360 eq. 2]
[eq. 4]. The discriminator update is not written out in the paper.

```pseudocode
# AIRL reward from a discriminator  [arxiv:2412.11360 eq. 4]
# Input:  discriminator D, state-action pairs (s, a)
# Output: reward for each pair
def airl_reward(discriminator, states, actions):
    # Probability that each pair is from the expert  [eq. 2]
    probs = discriminator(states, actions)
    # Reward as the log-odds of that probability  [eq. 4]
    return log(probs) - log(1 - probs)
```

## Hyperparameters

The study reports these values; the Experiment 1 and Experiment 2 settings
are not the same. The position paper trains no models and states no IRL
settings [arxiv:1811.07871 §7.2]. The survey proposes and runs no method, so
it gives no IRL settings either [arxiv:2309.02473 §III].

| Setting | Parameter | Values reported | Source |
|---|---|---|---|
| Experiment 1 (goal grid) | lambda, alpha, p_min, l_max | 2, 1.0, 1e-6, 4 | [fig. 1] |
| Experiment 2, Do condition | standard planner lambda | 0.02 to 0.04 | [app. 2] |
| Experiment 2, Show condition | standard planner lambda | 0.02 to 0.30 | [app. 2] |
| Experiment 2, both conditions | pedagogical alpha | 1 to 20 | [app. 2] |
| Experiment 2, both conditions | pedagogical l_max | 7 to 11 | [app. 2] |
| Experiment 2, both conditions | pedagogical lambda, p_min | 0.05 to 0.20; 1e-10 to 1e-5 | [app. 2] |
| Visual IRL (AIRL), discriminator, reward and TRPO | architecture, learning rate, discount, iterations | not stated | [arxiv:2412.11360 §IV.A] |
| StepAgent, reflection stage | learning rate, batch size, epochs | 5e-7, 16, 1 | [arxiv:2411.03817 §6.4] |
| StepAgent, implicit-RL beta and inverse-RL entropy weight | beta, lambda | not stated | [arxiv:2411.03817 §4.2.1] [§4.2.2] |
| StepAgent, training iterations | WebShop peak | 3 (implicit), 7 (inverse) | [arxiv:2411.03817 §7.4] |

Experiment 2 values were fit by maximum median likelihood, one set per reward
and condition, not per participant [arxiv:2610.14004 §4.2.1]. The study does
not state the discount factor, the planner horizon or the MLIRL step settings
in the text [arxiv:2610.14004 §4.2.1] [arxiv:2610.14004 §4.3].

StepAgent's main runs do not state an iteration count. The sweep shows the
peak differs by method, and performance falls after it
[arxiv:2411.03817 §7.4]. Choose the count on a held-out split.

## Contradictions

Three points differ in framing, convention or internal consistency:

- Identifiability. The position paper says IRL is not uniquely identifiable
  from behaviour alone, gives R = 0 as a solution, and says identification
  needs assumptions about the human, such as full rationality plus a designed
  task sequence. In full generality it calls the problem impossible
  [arxiv:1811.07871 §7.2]. The study fixes one softmax model of the demonstrator
  and reports recovery under it [arxiv:2610.14004 §2.1.1] [arxiv:2610.14004 §4.3].
  The study does not state a general identifiability result. Neither source
  resolves the general question.
- Discriminator reward sign. Visual IRL sets the reward to the log-odds that a
  pair came from the expert [arxiv:2412.11360 eq. 4], so expert-like pairs
  score high. StepAgent trains its discriminator to score agent pairs high and
  expert pairs low, and its Algorithm 1 uses log D as the reward
  [arxiv:2411.03817 eq. 10] [alg. 1], so agent-like pairs score high. The two
  targets are opposite. The StepAgent card reads its sign as possibly inverted
  relative to GAIL, and the paper does not discuss this. Its own Section 5 uses
  D itself as the reward [arxiv:2411.03817 §5]. Check the sign against the code
  before reusing either formula.
- Within the study: the goal reward in Experiment 2 is 10 points in the tile
  description [arxiv:2610.14004 §4] and 5 in the participant design
  [arxiv:2610.14004 §4.1]. The released code returns 5 at the goal
  [code: src/main/java/irl/PGridRF.java#L85-L87].

## Papers

- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](/papers/2309.02473): a 2023 survey that places IRL among imitation methods, sorts IRL and adversarial IL by their assumptions, and lists 66 works in one table. It reports no experiments.
- [Scalable agent alignment via reward modeling: a research direction](/papers/1811.07871): a position paper that places IRL among alternatives to reward modeling, lists its two shortcomings, and summarises cooperative IRL. It reports no experiments.
- [Showing versus Doing: Teaching by Demonstration](/papers/2610.14004): models teaching by demonstration as Bayesian pedagogy, compares showing and doing in two human studies, and shows standard IRL recovers the reward better from showing trajectories (author-reported).
- [Visual IRL for Human-Like Robotic Manipulation](/papers/2412.11360): learns a reward from RGB-D human keypoints and object positions with AIRL, then maps the human arm to a cobot (author-reported on two lab tasks). The retargeting half is covered by [motion-retargeting](/concepts/motion-retargeting).
- [From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning](/papers/2411.03817): trains LLM agents with a per-step reward from expert and agent steps, either by a DPO-style implicit reward or a discriminator (author-reported on web and science agent benchmarks).