Inverse reinforcement learning

Overview

Inverse reinforcement learning (IRL) takes demonstrations and returns a reward function that explains them. The idea is the inverse of planning: an RL algorithm maps a reward to behaviour, and IRL maps behaviour back to a reward 1811.07871 §7.2. Two problems follow. The reward is under-determined by behaviour alone: R = 0 is always a solution, and recovering a unique answer needs extra assumptions about the demonstrator 1811.07871 §7.2. The survey puts it as ambiguity: a policy can be optimal for many rewards, so an IRL method must add structure to choose one 2309.02473 §III.

A second issue is that IRL usually assumes the demonstrator is doing the task well, not that they are trying to make the reward easy to learn 2610.14004 §1. The observer then needs a model of the demonstrator. The standard choice is a softmax planner: the demonstrator picks actions with probability that grows with their value, controlled by a temperature 2610.14004 §2.1.1 eq. 2. The observer scores each candidate reward by the probability of the trajectory under that model, with a uniform prior over rewards 2610.14004 §2.1.2 eq. 3.

Five sources cover this page. The survey A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges places IRL inside imitation learning. It also lists repeated environment interaction and a known transition model as practical limits of IRL 2309.02473 §III. The position paper Scalable agent alignment via reward modeling treats IRL as one way to learn a reward from behaviour, and lists its weaknesses as motivation for other approaches 1811.07871 §7.2. The study Showing versus Doing: Teaching by Demonstration asks what changes when the demonstrator is told to show the task instead of doing it. Its answer is that the trajectories differ, and that a standard reward learner recovers the reward better from the showing ones 2610.14004 §4.2.2 2610.14004 §4.3. The study's own learner is still standard IRL. The new part is a model of the teacher. The last two sources use a discriminator as the reward signal: one for robot manipulation Visual IRL for Human-Like Robotic Manipulation and one for LLM agents From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning.

Variants

The survey sorts IRL methods by how they choose one reward out of the many that fit: maximum margin, maximum entropy, and Bayesian 2309.02473 §III. The variants below are cut differently, by demonstrator model and reward signal. The Bayesian family is the closest match to the study's observer, which keeps a uniform prior over rewards and scores each by trajectory likelihood 2610.14004 eq. 3. That match is this page's reading, not the survey's.

Maximum-likelihood IRL with a doing model

The observer assumes the demonstrator is doing the task under a softmax policy, and learns reward weights by maximum likelihood. The paper calls this MLIRL. For its feature-reward task, the non-goal weights are constrained to be non-positive 2610.14004 §4.3. The paper uses this learner on both human conditions, Do and Show, and on model-generated trajectories 2610.14004 §4.3 fig. 2. Card: Showing versus Doing: Teaching by Demonstration.

Pedagogical IRL: the observer models a teacher who shows

The observer does not assume the trajectory came from doing. It assumes a teacher chose the trajectory to make the true reward easiest to infer. The teacher picks trajectories with a softmax over the observer's posterior, raised to a teacher parameter alpha 2610.14004 §2.3 eq. 5. The candidate trajectories are built by enumeration: they are kept if their length is at most l_max and their probability under some candidate reward's doing policy is at least p_min 2610.14004 alg. 1. The paper uses this model to predict human Show behaviour and compares it with the standard planner 2610.14004 §4.2.1 fig. 3. Card: Showing versus Doing: Teaching by Demonstration.

This variant answers the position paper's objection that IRL assumes the human optimises reward directly, even when a demonstration would be a more efficient way to communicate it 1811.07871 §7.2. The study does not cite that objection; the link is this page's reading.

Cooperative IRL: a two-player game with a shared prior

Cooperative IRL (CIRL) is a formal model of reward learning as a game between a user and an agent. Both share a prior over the user's reward function. The user observes the reward; then both act to optimise it. The optimal play trades off communicating the reward against maximising expected reward 1811.07871 §7.3. The position paper uses CIRL as a comparison point and does not evaluate it 1811.07871 §7.3. Card: Scalable agent alignment via reward modeling.

Adversarial IRL: a discriminator supplies the reward

A discriminator is trained to tell expert (state, action) pairs from pairs produced by the current policy. The reward is computed from the discriminator output, and the policy is optimised against that reward. The survey describes the reward as coming from the discriminator's confusion, and says the approach avoids solving a full RL problem at every iteration. It also notes that the min-max training can suffer from vanishing gradients and failure to converge 2309.02473 §IV. Visual IRL uses this form, called AIRL. Its reward is the log-odds of the discriminator output, and its policy is optimised with TRPO 2412.11360 §IV.A eq. 2 eq. 4. The paper applies it to per-frame 3D keypoints and object positions from video of human manipulation 2412.11360 §IV.A. Card: Visual IRL for Human-Like Robotic Manipulation.

StepAgent applies a GAIL-style discriminator to the steps of an LLM agent. It trains the discriminator to separate agent steps from expert steps, then uses log D as a per-step reward for PPO 2411.03817 §4.2.2 eq. 10 alg. 1. Expert steps come from recorded trajectories, and each agent step is sampled from the same prefix 2411.03817 §4.1. Card: From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning.

Reported results are the paper's own. Visual IRL reports a learned-behaviour accuracy of 90.4% against 83.3% for MAP-BIRL fed with discrete state-action trajectories. The text gives this comparison, with no table 2412.11360 §V.D. StepAgent's inverse-RL variant reaches the same range as its implicit-reward variant, with differences of 0.1 to 0.7 points in every cell, from one run per setting 2411.03817 tab. 2. The step-wise discriminator is one of two ways that paper uses expert steps. The other is a DPO-style implicit reward, which uses no discriminator 2411.03817 §4.2.1.

When to use which

Within the study, the showing demonstrations did not earn more reward for the participant: total reward won showed no difference between conditions 2610.14004 §4.2.2. Use that before assuming showing is a cost or a gain to the demonstrator.

Implementation sketch

Observer posterior over candidate rewards, from the doing model eq. 3. Pseudocode in our own words; the teacher step is written from the equations.

# Observer posterior over candidate rewards  [arxiv:2610.14004 §2.1.2] [eq. 3]
# Input:  trajectories j, candidate rewards R_1..R_N, uniform prior
# Output: P(R_i | j) for each trajectory j
def observer_posterior(trajectories, rewards, doing_likelihood):
    # likelihood[i][j] = P_Doing(j | R_i) under a softmax planner  [eq. 2]
    likelihood = doing_likelihood(rewards, trajectories)
    # Uniform prior: normalise over rewards for each trajectory
    return likelihood / likelihood.sum(axis=0)

# Pedagogical teacher over trajectories for each reward  [eq. 5]
# Input:  posterior P(R_i | j), teacher parameter alpha
# Output: P_Showing(j | R_i), normalised over trajectories
def showing_distribution(posterior, alpha):
    raised = posterior ** alpha
    return raised / raised.sum(axis=1, keepdims=True)

Keep the candidate trajectory set finite. Algorithm 1 uses the length bound l_max and the probability floor p_min to do this 2610.14004 alg. 1.

Adversarial variant: the reward is computed from a discriminator, not from a planner. This is the AIRL form that Visual IRL uses 2412.11360 eq. 2 eq. 4. The discriminator update is not written out in the paper.

# AIRL reward from a discriminator  [arxiv:2412.11360 eq. 4]
# Input:  discriminator D, state-action pairs (s, a)
# Output: reward for each pair
def airl_reward(discriminator, states, actions):
    # Probability that each pair is from the expert  [eq. 2]
    probs = discriminator(states, actions)
    # Reward as the log-odds of that probability  [eq. 4]
    return log(probs) - log(1 - probs)

Hyperparameters

The study reports these values; the Experiment 1 and Experiment 2 settings are not the same. The position paper trains no models and states no IRL settings 1811.07871 §7.2. The survey proposes and runs no method, so it gives no IRL settings either 2309.02473 §III.

Experiment 2 values were fit by maximum median likelihood, one set per reward and condition, not per participant 2610.14004 §4.2.1. The study does not state the discount factor, the planner horizon or the MLIRL step settings in the text 2610.14004 §4.2.1 2610.14004 §4.3.

StepAgent's main runs do not state an iteration count. The sweep shows the peak differs by method, and performance falls after it 2411.03817 §7.4. Choose the count on a held-out split.

Contradictions

Three points differ in framing, convention or internal consistency:

• Identifiability. The position paper says IRL is not uniquely identifiable from behaviour alone, gives R = 0 as a solution, and says identification needs assumptions about the human, such as full rationality plus a designed task sequence. In full generality it calls the problem impossible 1811.07871 §7.2. The study fixes one softmax model of the demonstrator and reports recovery under it 2610.14004 §2.1.1 2610.14004 §4.3. The study does not state a general identifiability result. Neither source resolves the general question.
• Discriminator reward sign. Visual IRL sets the reward to the log-odds that a pair came from the expert 2412.11360 eq. 4, so expert-like pairs score high. StepAgent trains its discriminator to score agent pairs high and expert pairs low, and its Algorithm 1 uses log D as the reward 2411.03817 eq. 10 alg. 1, so agent-like pairs score high. The two targets are opposite. The StepAgent card reads its sign as possibly inverted relative to GAIL, and the paper does not discuss this. Its own Section 5 uses D itself as the reward 2411.03817 §5. Check the sign against the code before reusing either formula.
• Within the study: the goal reward in Experiment 2 is 10 points in the tile description 2610.14004 §4 and 5 in the participant design 2610.14004 §4.1. The released code returns 5 at the goal PGridRF.java.

Papers

• A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges: a 2023 survey that places IRL among imitation methods, sorts IRL and adversarial IL by their assumptions, and lists 66 works in one table. It reports no experiments.
• Scalable agent alignment via reward modeling: a research direction: a position paper that places IRL among alternatives to reward modeling, lists its two shortcomings, and summarises cooperative IRL. It reports no experiments.
• Showing versus Doing: Teaching by Demonstration: models teaching by demonstration as Bayesian pedagogy, compares showing and doing in two human studies, and shows standard IRL recovers the reward better from showing trajectories (author-reported).
• Visual IRL for Human-Like Robotic Manipulation: learns a reward from RGB-D human keypoints and object positions with AIRL, then maps the human arm to a cobot (author-reported on two lab tasks). The retargeting half is covered by motion-retargeting.
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning: trains LLM agents with a per-step reward from expert and agent steps, either by a DPO-style implicit reward or a discriminator (author-reported on web and science agent benchmarks).