Showing versus Doing: Teaching by Demonstration

In plain words

When you show someone how to do a task, you do not just do it efficiently. You may take a longer route that makes your goal obvious, or pause where a choice tells the watcher something about the rules. Learning systems usually assume the person they watch is simply doing the job well, and they try to guess the hidden reward from that.

This paper treats a demonstration as a message. The model imagines a teacher who knows the reward and wants a watching learner to work it out. The teacher therefore picks the path that best separates the true goal from the alternatives. The authors also compare that teacher with a plain planner on two small grid tasks, one with a goal and one with coloured tiles that are safe or dangerous. In both, people who were asked to show the task behaved differently from people who were asked to do it.

The authors then train a standard reward-learning method on these demonstrations. It learns the reward better from the showing trajectories than from the doing ones. The point is that the way a demonstration is made affects what a learner can take from it.

TL;DR

• Teaching by demonstration is modelled as Bayesian pedagogy: a teacher chooses trajectories that make the true reward easiest for an observer to infer §2.2 eq. 4.
• The "showing" teacher is defined on top of an observer who assumes the demonstrator is doing the task with a softmax planner §2.1 eq. 3 eq. 5.
• Exp. 1 (goal grid, 59 participants): show-instructed people choose disambiguating paths more often than do-instructed people §3.2 fig. 1.
• Exp. 2 (feature rewards, 58 participants): show-instructed people take more varied, looping paths that signal tile types §4.2.2 fig. 2.
• Standard maximum-likelihood IRL recovers the reward better from showing trajectories than from doing trajectories, both human and model-generated §4.3 fig. 2.

Problem

Inverse reinforcement learning (IRL) infers a reward from demonstrations and usually assumes the expert is doing the task rather than intentionally showing it §1. The paper asks how a person who shows a task differs from one who just does it, and whether standard IRL algorithms gain from showing demonstrations §1. Prior work on user-assisted teaching, legible motion and reward coordination is cited, but the authors state that none had modelled how people naturally teach reward functions by demonstration §1.

Method

Overview. The paper builds a "Pedagogical Inverse Reinforcement Learning" model. It combines two pieces: an observer model from IRL, in which the observer infers a reward from a trajectory assuming the demonstrator maximises it with a softmax (Boltzmann) policy, and a teacher model from Bayesian pedagogy, in which the teacher picks the datum that maximises the learner's posterior on the target concept §2 §2.1 §2.2 §2.3. The teacher's choice is applied to whole trajectories, so the teacher reasons about how the observer will read the trajectory under the doing model §2.3.

Steps.

1. Doing model. The expert chooses action a_t in state s_t with a Boltzmann policy over the optimal action values. The temperature parameter is lambda §2.1.1 eq. 2.

P_{\text{Doing}}(a_t \mid s_t, R) = \frac{\exp\{Q^*(s_t,a_t)/\lambda\}}{\sum_{a'}\exp\{Q^*(s_t,a')/\lambda\}} \qquad \text{[eq. 2]}

2. Observer. For a trajectory j the observer computes a posterior over reward functions R, with uniform prior P(R) §2.1.2 eq. 3.

P_{\text{Observing}}(R \mid j) = \frac{P_{\text{Doing}}(j \mid R)\,P(R)}{\sum_{R'} P_{\text{Doing}}(j \mid R')\,P(R')} \qquad \text{[eq. 3]}

3. Teacher, example level. In Bayesian pedagogy the teacher chooses datum d to maximise the learner's probability of the target concept h, using a softmax with parameter alpha §2.2 eq. 4.

P_{\text{Teacher}}(d \mid h) = \frac{P_{\text{Learner}}(h \mid d)^{\alpha}}{\sum_{d'} P_{\text{Learner}}(h \mid d')^{\alpha}} \qquad \text{[eq. 4]}

4. Teacher, trajectory level (showing). The same form is applied to trajectories j for a given reward R §2.3 eq. 5.

P_{\text{Showing}}(j \mid R) = \frac{P_{\text{Observing}}(R \mid j)^{\alpha}}{\sum_{j'} P_{\text{Observing}}(R \mid j')^{\alpha}} \qquad \text{[eq. 5]}

5. Trajectory set. Algorithm 1 builds the candidate trajectories alg. 1. For each candidate reward R_i it computes Q-values, takes a softmax with lambda, and keeps trajectories of length at most l_max that start in the given start states and have a product of action probabilities above p_min under some reward's doing policy alg. 1 lines 1-6.

# Showing distribution over trajectories  [alg. 1]
# Input:  start states, candidate rewards R_1..R_N, transition T, gamma,
#         lambda (doing softmax), alpha (teacher), l_max, p_min
# Output: P_Showing(j | R) for each reward R (one row per reward)

def showing_distribution(starts, rewards, transition, params):
    # 1. Hypothetical doing policy for each candidate reward  [alg. 1 l. 1-5]
    policies = []
    for reward in rewards:
        q_values = action_values(starts, reward, transition, params.gamma)
        policies.append(softmax(q_values, params.lambda_))
    # 2. Keep trajectories up to l_max whose probability under some policy
    #    exceeds p_min  [alg. 1 l. 6]
    trajectories = candidate_trajectories(starts, policies, params)
    # 3. Hypothetical doing probabilities, shape (N, M)  [alg. 1 l. 7]
    p_doing = doing_probabilities(policies, trajectories)
    # 4. Observer posterior over rewards for each trajectory, uniform P(R)
    #    [alg. 1 l. 8] [eq. 3]
    p_observing = normalise_over_rewards(p_doing)
    # 5. Teacher: raise the posterior to alpha and normalise over
    #    trajectories, for each reward  [alg. 1 l. 9] [eq. 5]
    return normalise_over_trajectories(p_observing ** params.alpha)

6. Predicted state-action probabilities. For fitting, the paper turns a distribution over trajectories into a policy: P(a | s, R) is the sum over trajectories of P(a | s, j) P(j | R), where P(a | s, j) is the fraction of visits to s in j that took action a §4.2.1.

7. Standard planner, for comparison. The doing model is the Boltzmann policy for a given reward and lambda §4.2.1.

8. Experiment 1 (goal grid). Participants find the goal location, then are assigned to a Do or Show condition. In Show they are told a partner will use their moves to find the goal §3.1. The analysis counts, for goals & and %, how often the final approach enters from the disambiguating side (from the left for &, from above for %) §3.2.

9. Experiment 2 (feature rewards). Participants learn which of three tile colours are safe, then either do or show the task on a new grid, and their trajectories are used both for behavioural analysis and for IRL §4.1 §4.2 §4.3. The tile grid is shown in fig. 2.

10. Teaching MLIRL. Human and model trajectories are fed to maximum-likelihood IRL (MLIRL) with non-goal feature weights constrained to be non-positive. Recovery is scored by the distance between learned and true weights §4.3 fig. 2.

Reward used for the Experiment 2 IRL fits, in the released code. The reward is 5 at the goal and minus the sum of squared weights on visited coloured tiles otherwise PGridCostRF.java PGridRF.java.

# Experiment 2 reward during MLIRL fitting  [code: PGridCostRF.java#L21-L37]
# Input:  next state, goal cell, three feature weights w (cyan, orange, purple)
# Output: scalar reward for entering the next state

def reward(next_state, weights):
    # The goal cell pays a fixed 5  [code: PGridRF.java#L85-L87]
    if next_state.position == GOAL_CELL:
        return 5.0
    features = tile_features(next_state)  # (3,) 0/1, one tile type at most
    # Squaring the weights keeps every coloured-tile term non-positive, which
    # is how the released code enforces the constraint  [§4.3]
    return -sum(features[i] * weights[i] ** 2 for i in range(3))

Hyperparameters

Experiment 1 model fit (caption of fig. 1). The paper states these values for the model trajectories shown:

Experiment 2 maximum median likelihood parameters, copied from the appendix table app. 2. Columns are reward codes, read as safe (o) or dangerous (x) for the tile order orange, purple, cyan.

Doing condition:

Showing condition:

Other Experiment 2 settings:

Results

Author-reported. Human studies on Amazon Mechanical Turk. Experiment 1: 59 analysed participants in a 2x2 design (Show vs Do between subjects, goal repeated) §3.1 §3.2. Experiment 2: 58 analysed participants on the feature-reward task §4.1. The baseline for behaviour is the Do condition, and for the model comparison, the standard planner §4.2.1. Figures show model predictions and participant trajectories fig. 1 fig. 2.

The paper does not give numeric values for the MLIRL distances or the model-fit medians in the text; they appear only in fig. 2 and fig. 3.

Paper vs code

Repository: jmacglashan/pedagogical_irl, owned by author James MacGlashan. It has no LICENSE file and no license in its pom.xml, so the code is unlicensed and not quoted here. Pseudocode above is written from the code, with citations only.

• Teacher model not released. Algorithm 1 and the equations for P_Showing and the trajectory set are not in the repository. The repo holds Java for MLIRL fitting, feature and grid definitions and the parser, plus JSON data files [repo tree]. The pedagogical trajectories (model_pedagogy_trials.json) are provided as data, with no generator code. The values of lambda, alpha, l_max and p_min in app. 2 therefore cannot be checked against code.
• Goal reward. The paper says the yellow terminal tile is worth 10 points §4 but its design says the goal is worth 5 §4.1. The code returns 5 at the goal cell PGridRF.java PGridCostRF.java.
• Non-positive constraint. The paper says non-goal weights are constrained to be non-positive §4.3. The code does not bound the weights; it squares them and uses minus the square as the weight PGridCostRF.java. The post-processing step then multiplies each parameter by minus itself, so the returned weight is minus the square Driver.java.
• Feature order. The paper's caption orders the codes as orange, purple, cyan fig. 2. The code's feature vector is cyan (index 0), orange (index 1), purple (index 2) IRLFeatures.java with tile numbering cyan = 1, orange = 2, purple = 3 AMTGridGenerator.java. Reorder outputs before comparing with the paper.
• Initialisation. The call randomInit(-1, 0) computes roll * range - lower, which with lower = -1 gives values in [1, 2), not in [-1, 0) PGridRF.java Driver.java Driver.java. After squaring, initial weights are in [-4, -1]. The paper does not state an initialisation.
• Restarts. Each reported number is an average over 30 MLIRL runs Driver.java. The paper does not mention restarts. The initialiser draws from a generator seeded with 0 (RandomFactory.getMapped(0)) PGridRF.java. Whether each restart gets different values depends on BURLAP's internals, which are not in this repo.
• Released data. The main method reads the new_set human showing trajectories and writes irl_human_showing_0.01.json Driver.java. Other runs are commented out. Which data version the paper's numbers used is not stated.
• Goal cell and grid. The goal is hard-coded at (5, 2) on a 6 by 5 grid Driver.java AMTGridGenerator.java. The Experiment 1 grid is not in the repository.
• Fitting the teacher. The parameter search over lambda, alpha, l_max and p_min described in §4.2.1 is not in the repository. Only the MLIRL reward fits are.

Pitfalls

• Lambda is called an "inverse temperature" §2.1.1 but enters the softmax as Q / lambda eq. 2. So small lambda means nearly greedy and large lambda means nearly uniform. The text's description of the limits is correct for this formula; the name is not.
• Algorithm 1 lists alpha and gamma in its Require block but uses lambda in the softmax without listing it there alg. 1.
• Algorithm 1 uses N x M and M is not defined in the paper. Define M as the size of the trajectory set after the p_min and l_max filters.
• The trajectory set grows with l_max. The paper uses l_max 4 in Experiment 1 and 7 to 11 in Experiment 2 fig. 1 app. 2, but reports no runtime or memory cost. Plan for enumeration cost; this is not stated.
• Eq. 2 in the paper writes the normaliser with s_i and a_i while the numerator uses s_t and a_t. Read them as the same state and action.
• Algorithm 1 thresholds trajectories by p_min under any reward's doing policy alg. 1 line 6. A trajectory that is rare under every candidate reward is dropped, which can change the showing distribution without changing the formula.
• The appendix proof covers a 3 by 2 gridworld with two goals, trajectories of length 2 and gamma = 0.99 app. 1. It shows a single inequality for that case. It is not a general theorem about showing.
• Experiment 1 excludes "swerving" from the model's prediction; the paper calls it unpredicted §3.2. It is a separate, post hoc result.
• Model fits are one set of parameters per reward function and condition, chosen to maximise the median likelihood across participants, not fit per participant §4.2.1.
• The paper reports participants' fits and the MLIRL results in figures only. Numbers that a reimplementation needs for checking, such as the MLIRL distances, are not in the text.
• The source file uses the NIPS 2016 style, and the arXiv metadata gives v1 as 2026-09-27 p. 1. The arXiv abstract page also lists a NeurIPS 2016 venue. The card uses the arXiv date.
• The Experiment 2 learning and applying phases were played for 8 reward assignments, and the card does not report any per-assignment analysis §4.1.

Lineage

• Builds on: Bayesian models of pedagogy and concept teaching by example, cited as Shafto et al. (2014) §2.2. The observer side builds on IRL §2.1.2 and on maximum-likelihood IRL, cited as Babes et al. (2011) and MacGlashan et al. (2015) §4.3.
• Compared against: a standard planner that assumes doing §4.2.1 fig. 3. The paper also cites user-assisted teaching, legible motion and cooperative reward coordination as related work §1.
• Superseded by: not known from content already in this wiki.