Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

In plain words

Chatbots are often more useful when they have been tuned on what people actually prefer. This paper does that for an assistant that should be both helpful and harmless. People chat with two candidate replies and pick the better one, or, when they are trying to provoke bad behaviour, the worse one. A separate judge model learns to predict those picks.

The chatbot is then trained with reinforcement learning to get high marks from that judge. A penalty keeps it from drifting too far from its starting version. The team repeats the loop: the new chatbot produces replies that are collected and judged again, and the judge and the chatbot are retrained on the larger pile.

The two goals pull against each other. A model pushed hard to avoid harm tends to refuse too often, and a model pushed hard to be helpful can become less careful. The paper reports that large models handle this trade-off better than small ones, and that the tuning does not hurt their general test scores.

TL;DR

• Crowdworkers chat with a language model and pick the more helpful reply, or the more harmful one for red-teaming. A preference model (PM) learns from those picks §2.1 §3.
• RLHF then trains the chatbot with PPO, using the PM score minus a KL penalty as reward §4.1 eq. 2.
• An iterated "online" loop retrains PMs and policies with fresh crowd data, and crowdworkers prefer the online model §4.5 fig. 1.
• Helpfulness and harmlessness conflict: training on one alone makes the PM worse than chance on the other §5.1 fig. 19.
• Author-reported: 86% PM accuracy on an HHH benchmark (52B static PM) §3.4 fig. 5.

Problem

Helpful, honest and harmless assistants are hard to specify by rule. The authors use human preference data in dialogue, where people can judge many kinds of text tasks, and turn that data into a training signal §1 §2. They also note that alignment work done on small models can give wrong conclusions for large ones, and that small models lose general ability after this tuning ("alignment tax") §1.1 §7.

Assumptions: crowd judgments are the target, and the crowd is not screened for agreement with researchers §2.1. Honesty is not the main target; the authors say other methods may do better for it §1, footnote.

Method

Data. Crowdworkers write a message, then see two model replies and pick one, then continue the chat §2.1. Two datasets are collected. The helpfulness set asks for the more helpful reply. The red-teaming set asks for the more harmful reply §2.2. Comparisons are kept only when the worker expressed a preference stronger than the weakest option. Ties are dropped and all kept comparisons have equal weight §2.2.

Replies come from three model types: a context-distilled 52B model (base data), a rejection-sampling variant that picks the best of k samples under an early PM (usually k = 16), and RLHF models deployed in later stages (online data) §2.3. Table-style counts are in the Hyperparameters section.

Preference model. Each PM is a language model with a scalar output read at a special end-of-context token after the final reply. The full dialogue is in context; only the last reply is scored app. A.2. Training has three stages: language-model pretraining, preference-model pretraining (PMP) on comparison data built from StackExchange, Reddit and Wikipedia, and fine-tuning on the human comparisons. Each stage runs for one epoch app. A.2. The PM is trained so that the better item in a pair scores higher §4.1. The loss function is not written out in this paper.

Calibration matters because the PM score is the reward. The paper models the probability of preferring A over B as a logistic function of the PM score difference, and checks how well this holds §3.3 §4.1 eq. 3.

Context distillation (before RL). The policy starts from a model trained to imitate a model that sees a helpfulness prompt. The data are generated by prepending that prompt to text, taking the teacher's top-50 log-probabilities at each following token, and training the student with a KL loss over those 50 tokens plus one "other" bucket app. A.1. The HHH prompt text itself is not given in this paper.

RL (RLHF).

# RLHF with PPO against a fixed preference model  [§4.1] [eq. 2] [app. B.1]
# Input:  policy (starts as the context-distilled model), reference policy,
#         pm (preference model), prompts (human-side turns only)
# Output: updated policy

def rlhf_epoch(policy, reference, pm, prompts):
    # The policy writes one final reply per prompt, up to 128 tokens
    # (32 in the robustness studies)  [app. B.1]
    responses = policy.sample(prompts, max_tokens=128)
    # Read the PM score at the end-of-context token of each reply  [app. A.2]
    pm_scores = pm(prompts, responses)                 # (B,)
    # Empirical KL between policy and its initial snapshot, measured on the
    # sampled outputs; the estimator is not stated  [§4.3]
    kl = kl_estimate(policy, reference, responses)     # (B,)
    # Reward is the PM score minus lambda times KL, lambda = 0.001  [eq. 2]
    rewards = pm_scores - 0.001 * kl                   # (B,)
    # PPO with clip 0.2, discount 1, no entropy bonus; each batch of
    # samples is reused K times (K = 1 scan, 2 robustness, 4 online)  [app. B.1]
    return ppo_update(policy, responses, rewards, clip=0.2, reuse=K)

Prompts are the human turns from the PM training split, with the old replies removed. They are supplemented by model-generated prompts made by few-shot sampling from about ten existing human queries. The RLHF run uses 137k of the first kind and 369k of the second §4.1 app. B.1. Earlier turns of each prompt were written by other models, not by the policy under training app. B.3.

Robustness studies. The static data are split 50:50 into two halves. One PM is trained on each half. A policy is optimised against the "train PM" and scored by the "test PM" §4.2. Two scans are run: a fixed 52B train PM across all policy sizes, and a train PM matched to each policy size §4.2.

KL relation. Across training, the test-PM score grows roughly linearly with the square root of the policy's KL from its start. The authors give a heuristic argument and call it a conjecture §4.3 fig. 13.

Iterated online RLHF.

# Iterated online RLHF  [§4.5] [app. B.3]
# Each round retrains from scratch; no model is updated in place  [§7.1]

def online_round(comparisons, crowd_policies):
    # Crowdworkers chat with the latest RLHF snapshots and judge them
    # (roughly weekly over about five weeks)  [§2.3]
    new_comparisons = collect_comparisons(crowd_policies)
    comparisons = comparisons + new_comparisons
    # Train a new PM scan on all data, with a 2048-token context  [app. A.2]
    pm = train_preference_model(comparisons, context=2048)
    # Then a new RLHF scan against that PM  [app. B.1]
    policy = rlhf_train(context_distilled_model, pm, prompts)
    return policy, pm

Online RLHF in the final run uses K = 4 and a learning rate that halves every 100,000 samples app. B.1 app. B.3. The online PM is trained with a 2048-token context, not 1024, which is one of two differences from the earlier static run §4.5.

Helpful-versus-harmless mixing. Mixtures from 100% helpful to 100% harmless (in steps of 10%) are compared, with the total fixed at the size of the red-teaming set §5.1 fig. 19. A loss weight lambda on the harmless term is also tried, L = L_help + lambda L_harm, for lambda in {1, 2, 3, 4, 10} §5.1 fig. 27.

Other uses. Summarisation comparisons can be mixed into PM training with no loss in accuracy on either task §5.2 fig. 20. Code-finetuned models can be RLHF-trained on the same helpful and harmless data §5.3 fig. 21. Out-of-distribution scores based on a Mahalanobis-style distance to helpfulness data can flag most harmful prompts, and a few harmful examples improve this further §5.4 fig. 22 fig. 23.

Hyperparameters

Not stated in this paper: PM and PPO batch sizes, number of RL steps, optimiser (beyond LR schedules), GAE or advantage estimator, value-function details, sampling temperature used during RL, hardware and compute.

Results

Author-reported. Setup: 7 model sizes (13M to 52B); RLHF on the authors' own crowdworker data; evaluation on HHH, NLP benchmarks, and crowd Elo. Crowd comparisons use the authors' chat interface §2.1 §2.4.

Other reported numbers: crowdworker-researcher label agreement about 63% §2.1; PM calibration is close for helpful-only PMs and slightly under-confident for the mixed PM §3.3 fig. 9; in the robustness plot, train and test PM scores agree until roughly 150k training samples and then diverge §4.2 fig. 4; in a conflict test, a PM trained only on one of helpful or harmless data scores below chance on the other §5.1 fig. 19.

Paper vs code

No official code.

The paper links anthropics/hh-rlhf for the helpfulness dataset §1, footnote. That repository (MIT, archived) holds preference data and red-teaming transcripts, not training code. Its README points to a Hugging Face dataset. It is not used as code.

Searched: the GitHub organisation search for rlhf inside anthropics and for helpful harmless assistant rlhf found only that dataset repository. No training code, configs or evaluation code for the paper were found. Third-party reimplementations were not used.

Paper-internal note: the paper says helpfulness data is public and harmlessness data will be released later §1, footnote. This card does not depend on either release.

Pitfalls

• Reward scale. PM scores are only meaningful relative to each other. The paper's Elo and win-rate maps assume that. The paper does not say whether RL rewards are mean-shifted before PPO; mean removal is described for comparing PMs app. B.2. Check this before copying the reward.
• KL term. The KL penalty uses an empirical estimate over sampled outputs §4.3. The estimator, and how the per-sequence KL is spread over tokens, are not stated. The authors say lambda = 0.001 may be unnecessary §4.1.
• Over-optimisation. PM calibration and robustness fall as PM score rises §3.3 §4.2. The authors expect diminishing returns from RL once a policy reaches a high PM score §3.3, and attribute the over-refusal seen in early policies to over-optimising harmlessness §4.4 fig. 14.
• Harmless-data direction. In the red-teaming set, workers picked the more harmful reply, so the data show what to avoid and not what a good refusal looks like §2.2. The authors link this to the under-optimised helpfulness and the weak refusal tuning §4.4.
• Distribution shift. The RLHF policy writes only the final reply after prior turns generated by other models. Crowd comparisons compare full dialogues, so naive PM scores overstate Elo app. B.3 fig. 35.
• Crowd drift. Crowdworker quality was not held fixed and may have changed over the project. Workers were not filtered on agreement; the authors found only about 63% agreement with researchers §2.1. The online-vs-static comparison is therefore not a clean experiment, though a controlled run was added §4.5 fig. 16.
• Overfitting. RLHF on the static PM over-optimises the train PM; the test PM diverges after about 150k training samples in the robustness plot fig. 4. Smaller PMs and policies are less stable, and some runs were stopped early fig. 13.
• Model-scale transfer. Most data came from 52B models, so smaller models are often out-of-distribution to the PM. Smaller RLHF models lost general performance (the "alignment tax") §1.1 §7.
• Sampling-dependent evaluations. RLHF narrows output distributions, so log-likelihood or forced-choice tests can mislead. The forced-choice NLP format also produces sudden jumps that the authors call misleading §4.6.1.
• Unstated details. Context-distillation prompt text, the PMP labelling pipeline, the architecture and the training batch sizes are referred to earlier work or not given app. A.1 app. A.2 §3.1. An implementer must find those elsewhere.
• Scaling formula. The fitted accuracy formula app. A.3 is for the helpfulness-only case, the log base is not stated, and the authors say other distributions do not fit it.
• Crowd ceiling. The preference-based online result is compared to writers under a non-adversarial test, and the authors caution against reading it as general performance §1.2.
• Honesty. The model is more truthful on TruthfulQA, but the authors say RLHF closes only a bit less than half of the zero-shot to few-shot gap §7.1 fig. 5. Do not treat the RLHF result as an honesty fix.
• Bias. Gender bias in RLHF models tracks that of the base LM, and sentiment improved for all groups without removing bias §7.1 fig. 40 fig. 17.

Lineage

• Builds on: arXiv:2112.00861 (LM specifications, PMP, HHH evaluation, context distillation) §3.1 app. A; arXiv:2009.01325 (learning-to-summarise RLHF recipe, KL penalty) §4.1; arXiv:1707.06347 (PPO) app. B.1.
• Compared against: plain and context-distilled LMs, and RLHF on the static dataset without online data fig. 1 fig. 16.
• Related card in this wiki: DPO is a later method that drops the separate reward model and RL loop. This card does not claim that DPO supersedes this work.