Reward learning from human preferences and demonstrations in Atari

In plain words

Teaching a game-playing program needs a score to chase. Writing that score by hand is hard when the goal is vague, so this paper asks people for help in two ways. They can show a few complete games, which the program copies. They can also watch two short clips of the program playing and pick the better one.

The paper learns a stand-in for the score from those picks: a second network that guesses how good each moment is. The game-playing network then learns from that guessed score instead of the real one. The authors test this on nine Atari games. Showing games and picking clips together does better than copying the demonstrations alone in most of them, and sometimes beats the people who showed the games. The picks have to keep coming while the agent learns, because a stand-in score that never gets corrected can be gamed.

TL;DR

• Two feedback channels: expert demonstrations at the start, and pairwise preferences over 25-step clips during training §2.1.
• A convolutional reward model is fitted to the preferences with a Bradley-Terry style cross-entropy loss §2.4.
• The policy is DQfD (dueling, double, prioritized DQN with 1- and 3-step returns and a large-margin demo loss), and it reads the learned reward when it samples batches §2.3 app. A.3.
• Author-reported: beats imitation in 7 of 9 Atari games in the abstract, with superhuman scores on Pong and Enduro §4 §3.1.

Problem

Reinforcement learning needs a reward function, and for many tasks one cannot be written down §1. Imitation learning from demonstrations copies the demonstrator, so it cannot exceed them and needs demonstrations that are hard to produce §1. Learning from preferences avoids that, but random exploration rarely covers the states worth judging, and preferences carry only a few hundred bits per hour of human time §1. The agent never sees the game reward. Synthetic labels are generated from it, and the game score is used for evaluation §3 §3.2.

Method

The method has four parts: an expert who gives demonstrations, an annotator who gives preferences (possibly the same person), a reward model fitted to the preferences and optionally to the demonstrations, and a policy trained from demonstrations plus the reward model's output §2.2.

Training protocol alg. 1. The policy is pretrained on the demonstrations first, which also produces the first set of trajectories. Clip pairs are drawn from those trajectories and labelled; optionally, demonstration clips are paired with agent clips and labelled automatically. The reward model is trained on the buffer. The policy is pretrained again on the demonstrations, now with learned rewards. Then the main loop repeats M = 500 iterations: the policy trains for N = 10^5 agent steps on the reward model, new pairs are sampled and labelled, and the reward model is trained on k batches from the buffer alg. 1 app. A.4.

Policy. The policy is DQfD: a dueling, double Q-network with prioritized replay and 1- and 3-step returns. Its loss is a Q loss plus a large-margin supervised loss that applies only to demonstration transitions, plus L2 regularization §2.3. The replay buffer keeps demonstrations permanently, and transitions store no reward; the reward is recomputed from the observation by the reward model each time a batch is sampled §2.3 app. A.3.

Reward model. A convolutional network takes a single observation (four stacked frames) and outputs one scalar; actions are not an input §2.4 app. A.3. The preference probability for a clip is a softmax over the summed predicted rewards of the two clips §2.4, unnumbered display equation. Training minimizes the cross-entropy against the annotator's label, where a label is (1, 0), (0, 1) or (0.5, 0.5) for "equal" §2.4. Clip pairs the annotator marks as incomparable are discarded §2.4.

Regularization of the reward model. Because the training set is only a few thousand pairs, the paper adds: validation holdout, adaptive L2 weight, a zero- mean Gaussian prior on predicted rewards, Gaussian input noise, batch normalization and per-channel dropout, and a 10% label-noise model app. A.3. The reward is re-normalized every 100,000 agent steps to zero mean and standard deviation 0.05 over the annotation buffer §2.4 app. A.3.

Clip selection and labels. Clips are chosen uniformly at random; the paper tried ensemble uncertainty and it did not help §2.5. In the "autolabel" setting, each clip from the agent's trajectories is paired with a random demonstration clip and labelled to prefer the demonstration; this gives 2k extra labels from k requested labels §2.5.

# Reward learning from demos and preferences  [alg. 1] [§2.2]
# Input:  demos (expert trajectories), annotator
# Output: policy, reward_model

def train(demos, annotator):
    # Behavioral-cloning pretraining from demos, 20000 batches  [app. A.4]
    policy = pretrain_on_demos(demos, batches=20000)
    # Roll out the policy to get the initial trajectories  [alg. 1] [app. A.4]
    initial = run_policy(policy)
    # Sample clip pairs and label them; 500 pairs in the full schedule  [§2.5]
    buffer = label(sample_pairs(initial, count=500), annotator)
    if use_autolabels:
        # Pair each agent clip with a random demo clip, demo preferred  [§2.5]
        buffer += autolabel(initial, demos)
    # Fit the reward model on the buffer, 50000 batches  [app. A.4]
    reward_model = fit_reward(buffer, batches=50000)
    # Second demo-only pretraining, now with the learned reward  [alg. 1]
    policy = pretrain_on_demos(demos, reward_model, batches=60000)
    for iteration in range(500):  # M = 500  [app. A.4]
        # Reward model is frozen during the iteration  [app. A.4]
        policy = train_rl(policy, reward_model, steps=10**5)
        # Sample new clip pairs from fresh trajectories  [§2.5]
        new_pairs = sample_pairs(trajectories(policy), count=schedule(iteration))
        buffer += label(new_pairs, annotator)
        # Train the reward model on k batches from the buffer  [alg. 1] [app. A.4]
        reward_model = fit_reward(buffer, batches=6250)
        # Re-normalize reward to mean 0, std 0.05  [§2.4] [app. A.3]
        reward_model = normalize(reward_model, buffer, std=0.05)
    return policy, reward_model

Preference loss for one labelled pair. Written out for implementers:

# Cross-entropy loss for one clip pair  [§2.4] [app. A.3]
def pair_loss(reward_model, clip_1, clip_2, label):
    # Sum of predicted per-step rewards over each clip  [§2.4]
    score_1 = sum(reward_model(obs) for obs in clip_1)
    score_2 = sum(reward_model(obs) for obs in clip_2)
    # Probability that clip 1 is preferred  [§2.4]
    p_1 = exp(score_1) / (exp(score_1) + exp(score_2))
    # Model a 10% chance of a uniformly random label  [app. A.3]
    p_1 = 0.9 * p_1 + 0.05
    # Label is (1, 0), (0, 1) or (0.5, 0.5)  [§2.4]
    return -(label[0] * log(p_1) + label[1] * log(1 - p_1))

Settings. The reward model uses Adam with learning rate 3e-4; the policy uses Adam with learning rate 6.25e-5 app. A.3. Full details are in the Hyperparameters table.

Hyperparameters

Sources are the appendix unless noted. Tables in the appendix have no captions, so they are cited by section.

Demonstrations app. A.2:

Results

Author-reported. Setup: nine Atari games, 50 million agent steps, three seeds per experiment, synthetic oracle labels unless stated §3 fig. 1. The main results are in fig. 1 and fig. 2; the paper gives few numbers in text, so the rows below are the ones it states.

DQN + preferences vs A3C + preferences (no demonstrations) app. F:

The A3C columns are copied from Christiano et al. as the paper reports them.

DQfD + feedback (ours) vs DQfD + true reward app. G:

The authors say this is not a like-for-like comparison, because the synthetic oracle makes the true reward indirectly available to the agent app. G.

Paper vs code

No official code.

Search: the paper and the arXiv record contain no code, project or repository link. Searched GitHub for "preferences demonstrations atari", "dqfd atari", "human preferences atari reward model" and for "reward learning human preferences demonstrations atari" within the google-deepmind organisation. The hits were unrelated DQfD reimplementations (felix-kerkhoff/DQfD, MIT) and unrelated repositories; none is by the authors.

Pitfalls

• Abstract and body disagree on how many games beat imitation. The abstract says 7 of 9 games [abstract]; §3.1 says the method beats imitation in every game except Private Eye, which is 8 of 9 §3.1. The text says the "7 games" result holds across conditions. Treat 7 as the conservative figure.
• Reward scale is arbitrary and matters. The reward model is only trained on comparisons, so the paper re-normalizes it every 100,000 steps to standard deviation 0.05. The authors say this value was chosen empirically and is important for stability §2.4. Changing it changes the effective learning rate of the policy.
• The label-noise model is an assumption, not a measurement. The paper assumes a 10% uniform-random annotator and trains against P_e = 0.9 P + 0.05 app. A.3. The text does not say whether the second class probability is 1 - P_e; the card's pseudocode assumes it.
• Reward is recomputed on every sampled batch. Stored transitions have no reward; the reward model is evaluated when a batch is drawn §2.3 app. A.3. The paper does not discuss the compute cost of this; the reward model must be evaluated on each sampled batch.
• Demonstrations are pretrained twice. The first pass is needed so the policy produces trajectories to label; the second uses learned rewards for the Q loss alg. 1 §2.2. The paper says the first pass's Q loss is "noisy and acts as a regularizer" because the reward is random at that point app. A.4.
• Episode boundaries are removed. Life loss and episode end are hidden from the agent, and a penalty is added at episode end, except for Pong app. A.1. The penalty value is not stated. The paper attributes poor Seaquest performance to this setup app. B.
• Score is blanked. The game score is replaced with a black background, so the agent cannot read reward from it app. A.1. Demonstration frames must match the agent's blanking exactly; the paper reports that a small mismatch dropped scores to zero in demo-heavy games app. H.
• Autolabels pair each agent clip with a demo clip. The labelled pairs always prefer the demonstration. The paper adds this to the buffer with no noise model §2.5 app. A.4.
• Clip pairs marked incomparable are dropped, not treated as a 0.5 label. Indifferent labels (0.5, 0.5) are a separate case §2.4.
• Uncertainty-based clip selection was tried and did not help §2.5. Only uniform sampling is in the main method.
• Frozen reward models are exploited. With a fixed reward model, performance dropped in Hero, Montezuma's Revenge and Private Eye as the agent found loopholes §3.2 fig. 4. The paper's claim is that online preference feedback prevents this, but it was measured only on Atari with synthetic labels and in a few human runs.
• Human labels are weaker. The paper reports 5 to 10% mislabelling and says mislabelling significantly hurts Montezuma's Revenge §3.1 app. E. Human shaping can produce reward pits §4. The human experiments used one run for the reward-alignment figure fig. 3 fig. 8.
• Sparse rewards give high loss floors. Indifferent labels have minimum cross-entropy log 2 = 0.693, so the loss floor depends on the share of indifferent pairs in each game app. C.
• Hyperparameter names are not fully defined. The dropout "alpha" and the strength of the reward prior are not stated.
• Results are single-setup. The main comparisons use three runs per experiment and show standard deviation in fig. 1 only.

Lineage

• Builds on: DQfD (Hester et al., 2018) for the policy and demonstration pretraining §2.3; preference learning from Christiano et al. (2017) for the reward model and clip setup §2.4.
• Compared against: imitation learning (DQfD pretraining only), no-demo preference learning with DQN and with A3C §3 app. F; DQfD trained on the true reward app. G.
• Superseded by: not stated in this card.

Figures

Figures are not reproduced. Each is described and cited here.

• fig. 1: mean episode returns for nine games across four setups and four label schedules after 50 million steps, with standard deviation over three runs; it includes the human-preference run.
• fig. 2: normalized per-game performance versus total labels, shown as box plots over all nine games.
• fig. 3: model reward against true reward over 25 and 1000 steps for Enduro, Montezuma's Revenge and Q*bert, synthetic and human labels.
• fig. 4: frozen reward model experiments; left shows the full-schedule results, right shows true versus model return for three games.
• fig. 5 and fig. 6: performance against human effort, and best performance per effort, with and without demonstrations app. B.
• fig. 7: reward model cross-entropy against the label-entropy floor app. C.
• fig. 8: model versus true reward over 25 and 1000 steps for the other six games app. D.
• fig. 9: performance against mislabelling rate in nine games app. E.