Improving alignment of dialogue agents via targeted human judgements

In plain words

Chatbots trained only to sound helpful can still give wrong answers, say harmful things, or slip into bad habits. This paper builds a question-answering chat agent called Sparrow and tries to make it better in three ways.

First, instead of asking people a vague "was this good?" question, the authors write a list of short, specific rules (for example, do not make threatening statements, do not give financial advice). Human raters judge each rule on its own, which gives cleaner feedback on what went wrong.

Second, when Sparrow makes a factual claim, it searches the web and shows the excerpt it relied on next to the answer. That lets a rater check the claim quickly, and it gives the model a reason to stay close to its sources.

Third, the feedback is turned into reward models that score replies, and those scores train the chat model with reinforcement learning. Sparrow also samples several candidate replies at answer time and keeps the best-scoring one. The authors report that this makes Sparrow preferred over simpler baselines and harder to push into breaking the rules, but it still shows uneven behaviour across groups of people.

TL;DR

• Sparrow is a dialogue agent built on Chinchilla 70B and trained with RLHF §2.8.
• Human raters judge short natural-language rules separately, and those labels train a rule reward model §2.1 §2.5.
• The agent issues Google searches, shows the evidence, and is scored by a preference model that rewards answers the evidence supports §2.4.
• Reranking at test time (model@8) and RL together give the best results in the author evaluation §3.1 §3.2.
• The authors report lower targeted-rule violations under probing, but bias in aggregate can grow after RL §3.1 §3.6.1.

Problem

Helpful dialogue has no programmatic reward, so the training signal comes from human judgements, which are noisy and depend on what raters can check §1 §2.3. Broad labels such as "toxic" or "harmful" are ambiguous and hard for raters to apply consistently §2.1. Factual answers also cannot be verified from the dialogue alone, so raters need evidence to judge them §2.4. The paper assumes the goal is information-seeking dialogue, where success is easier to define than in open chit-chat §1.

Method

The pipeline has five parts, run in a loop: prompted base model, human data collection, reward models, reranking, and RL §2.

Base agent. Dialogue-Prompted Chinchilla 70B (DPC) is an unmodified Chinchilla with a hand-written prompt that shows a user and an agent taking turns §2.2 app. A.1. For evidence, the prompt adds two extra roles, a search query and a search result §2.2. Turns are sampled with nucleus sampling app. A.2.

Rules. The authors wrote 23 rules covering helpfulness, correctness and harm, each short enough to judge alone §2.1 app. F. The rules were drawn from harm taxonomies in the literature, and an in-house legal expert helped with advice rules §2.1.

Human data. Two tasks §2.3:

• Per-turn response preference. Raters pick the best of several candidate continuations. The chosen reply is added to the dialogue.
• Adversarial probing. Raters are given one rule and try to make the agent break it, then say whether it did.

For evidence replies, raters also answer whether the reply is plausible and whether the evidence supports it. They answer these questions before choosing the best reply §2.4.

Evidence. The model writes a search query, Google Search returns results, and the page is scraped and cut to a short excerpt around the search snippet. The excerpt is added to the dialogue as a search-result turn and shown to raters as quoted evidence §2.4 app. C.

Reward models. Both are fine-tuned from Chinchilla 70B §2.5.

• Preference reward model. Scores each reply on an Elo-style scale, trained with a softmax over the candidate set. Auxiliary losses predict whether an evidence-backed claim is supported and plausible. When raters mark all options as bad, a phantom option at zero is added §2.5 app. D.1.
• Rule reward model. A conditional classifier that takes a dialogue and one rule and estimates the probability the rule was broken. Trained by instruction tuning to output Yes or No §2.5 app. D.3.

Reranking. The agent samples eight candidates per turn: four without evidence and four with evidence, the latter from two search queries with two results each. Each candidate is scored with eq. 1 and the highest is shown §2.6.

# Score one candidate reply for reranking  [eq. 1] [§2.6]
# pref_score: scalar from the preference reward model
# avg_pref_score: mean preference score over a validation set
# rule_probs: one probability per rule, each in [0, 1]
def rerank_score(pref_score, avg_pref_score, rule_probs):
    # Preference term: softmax-style ratio, equal to sigmoid(pref - avg)
    pref_term = exp(pref_score) / (
        exp(pref_score) + exp(avg_pref_score)
    )
    # Rule term: geometric mean of the rule-following probabilities  [eq. 1]
    rule_term = product(rule_probs) ** (1 / len(rule_probs))
    return pref_term * rule_term

Supervised fine-tuning. The SFT baseline trains the base model directly on replies that were rated preferred and that broke no rule §2.7.

Reinforcement learning. Advantage actor-critic (A2C) with a value head §2.8 app. E. The policy is initialised from the SFT model (or the base model). Training episodes are single statements sampled from a mixed buffer of dialogue contexts: public questions, human conversations, LM-generated red team questions, and the agent's own self-play. The reward for an agent turn is the normalised preference score, plus the mean of the normalised rule scores, minus length and format penalties. A KL penalty to the initial model keeps the policy close to where it started §2.8 app. E.2 app. E.3.

# One batched A2C update  [§2.8] [app. E]
# buffer: dialogue contexts from questions, humans, red team and self-play
# reward models: preference, one rule model per rule
def rl_step(policy, value_head, buffer, reward_models, teacher):
    # Sample a context and prefix the role-specific prompt  [app. E.1]
    context = buffer.sample()
    statement = policy.generate(role_prompt(context))
    # Score only agent turns on preference and rule compliance  [app. E.3]
    pref = reward_models.preference(context, statement)
    rules = reward_models.rules(context, statement)
    # Length and format penalties are subtracted at episode end  [app. E.3]
    reward = whiten(pref) + mean(whiten(rules)) - length_penalty(statement)
    # A2C loss plus a KL term to the initial model, weight 0.2  [app. E]
    loss = a2c_loss(policy, value_head, reward) + 0.2 * kl(policy, teacher)
    return loss

Targeting. Adversarial red-team questions come from a prompted Chinchilla that lists questions for a chosen adjective (for example medical or legal) and a target such as someone or an AI app. E.2.

Freezing. For reward models and RL the bottom 64 of 80 transformer layers are shared and frozen, and only the top layers are trained. This lets the policy, reward models, value head and teacher share memory §2.5 app. E app. E.

Hyperparameters

Results

Author-reported. Setup: 70B-scale agents, paid raters on ELI5 questions and free annotator dialogues, Chinchilla baselines §3.1 §3.2. The three-model preference rate and per-rule violation rates are shown in figures; their exact values are not stated in the text.

Headline numbers from the text:

Evidence per model, supported and plausible on evidence-backed turns §3.2:

Paper vs code

No official code. The paper links to a samples page for transcripts and ratings, not to code §3.3. When fetched, that link redirected to the DeepMind homepage rather than a samples page. A GitHub repository search for "sparrow dialogue" returned only an unrelated personal project. A search restricted to the google-deepmind organisation for "sparrow" returned no results. No code link appears in the LaTeX source.

Pitfalls

• Many training values are not stated. The preference and rule reward model learning rates, the sweep ranges, the auxiliary loss weights (α, β), the RL length-penalty and invalid-format weights (β, γ), and the warmup and decay steps are missing app. D app. E.3. A reimplementation has to sweep these itself.
• Eq. 1 is ambiguous. The average preference score is defined over "the valid set" with no size or source stated eq. 1. The rule term is a geometric mean over rules, so one near-zero rule probability drives the whole score toward zero eq. 1.
• Normalisation is described two ways. Section 2.8 gives a running mean and standard deviation, and the reward formula in appendix E.3 uses whitening. The batch or running statistics used for whitening are not stated. Confirm the choice before implementing.
• Off-policy data. Nucleus sampling makes the RL training data off-policy, and the authors do not correct for it §2.8.
• The preference reward is myopic for search queries. Search-query turns are rewarded only by the preference model, not by the downstream answer app. E.4.
• Evidence depends on a proprietary search API. Results change over time, and the search configuration (SafeSearch, Reddit exclusion, snippet matching threshold) is part of the method app. C. Results from a different engine will differ.
• Context budget limits evidence. Only one search fragment fits in a turn, and fragments are cut to 500 characters §4.1 app. C.
• Supported and plausible is not correct. Raters judge whether evidence supports a reply, not whether the claim is true §3.2 §4.1. The separate correctness check is small and done by the authors §3.3.
• Rule coverage is uneven. Only the first 18 of the 23 rules were used for probing §3.1. Several rules had little data. The paper says the interventions did not reduce violations for stereotypes, medical advice, legal advice, microaggressions and insults §3.1.
• Costs the abstract hides. Reranking runs eight samples and two searches per turn at inference time; RL adds no inference cost but is expensive to train §2.8 §2.6. Five models share memory during RL training app. E.
• Distributional bias can grow. RL increased bias scores on Winogender and BBQ, and the mitigations address only single-example harms §3.6.1 §3.6.
• Table and figure numbers are not checked. The compiled PDF was not available, so this card cites by section rather than by table or figure number.
• Rule-conditioned classifier cost. Predictions for all rules share the dialogue prefix, so cost grows only weakly with rule count app. D.3.

Lineage

• Builds on: Dialogue-Prompted Chinchilla (Hoffmann et al. 2022; Rae et al. 2021 prompt method) §2.2; A2C from Mnih et al. 2016 §2.8; learning to summarise with human feedback (Stiennon et al. 2020) for the iterative data loop §2; GopherCite (Menick et al. 2022) for evidence and reward models §2.4 §2.5.
• Compared against: DPC (never search, always search, choose search), SFT, GopherCite §3.1 §3.2. Related systems discussed: LaMDA, Anthropic assistant, SeeKeR, BlenderBot 3, WebGPT §1 §3.2.
• Superseded by: not stated in this card's wiki content.