Reinforcement learning from human feedback (RLHF)

Overview

RLHF is a post-training stage. A pre-trained language model is trained further so that its outputs match what human raters prefer. The Gemini 1.0 report lays the loop out in four steps: collect prompts, fine-tune on demonstrations, train a reward model on human judgments, then run reinforcement learning against that reward model and repeat 2312.11805 §6.3. The shared invariant is that the training signal comes from a learned model of human preferences rather than from a fixed metric. The ChatGPT survey lists InstructGPT and RLHF as the basis of ChatGPT but does not describe the method 2304.01852 §1, so the mechanics of the reward-model loop on this page rest on the Gemini report alone. An earlier position paper proposes the same idea as a research direction: learn a reward function from user feedback while an RL agent trains on it. It reports no experiments 1811.07871 §1 §3.1.

A survey covers the same loop in more depth, mostly for control and robotics, with language models in an appendix. It fits a reward model to pairwise comparisons under a Bradley-Terry likelihood 2312.14925 §2.3 eq. 2, then trains a policy against it, refitting the reward model as new labels arrive 2312.14925 §2.4 alg. 1. It runs no experiments of its own 2312.14925 §1.

Two later papers offer alternatives to parts of this loop. DPO fits the preference data directly, with no separate reward model and no RL sampling loop 2305.18290 §4. LIMA reaches its results with supervised tuning alone, on 1,000 curated examples and no preference model 2305.11206 §3. A third paper removes the good-answer side entirely and unlearns only bad answers 2310.10683 §4. KTO keeps a reward-free loss but needs only a desirable or undesirable label per output, not a pair 2402.01306 §4.1.

Other papers apply the loop to new data or check what it rewards. OpenAssistant releases volunteer-written conversations with rankings and runs supervised, reward-model and PPO stages on them 2304.07327 §1 app. H. Gemma describes an RLHF step for its instruction-tuned models, with the RL algorithm unnamed 2403.08295 § Reinforcement Learning from Human Feedback. A survey of controllable text generation lists reinforcement learning with a reward model trained on human preferences as one family of control methods, and names InstructGPT as an RLHF example 2201.05337 §3.2. Pick-a-Pic trains a preference scorer for images without running RL 2305.01569 §3. A sycophancy study finds that a preference model can prefer agreeable replies over truthful ones 2310.13548 §4.3.1.

A 2023 survey sorts the open problems of the loop into challenges with the feedback, the reward model, the policy and joint training, and labels each as tractable or fundamental. It runs no experiments 2307.15217 §3 app. B. A 2025 red-teaming study runs PPO against a classifier's "unsafe" probability, with no human labels in the loop 2508.04451 §4.2.

Four more papers touch the loop from other sides. ImageReward trains a reward model on expert image rankings, then fine-tunes a diffusion model by back-propagating that score through one late denoising step [arxiv:2304.05977 §3]. The GLM-4 report names SFT followed by RLHF as its alignment stage but leaves the RLHF method and its settings to other reports [arxiv:2406.12793 §2 Alignment]. A survey of human-AI collaboration places RLHF and DPO under objective shaping in its build-time loop, and reports no experiments of its own 2403.04931 §2.2. A position paper on role-play says its base-model analysis may not hold after RLHF-style tuning 2305.16367 §2 §8. A position paper on referential grounding names preference tuning as the most direct way for a model's internal states to come to carry meaning 2304.01481 §6.2.1.

Variants

Reward modeling as a research direction (position paper, no experiments)

The user trains a reward model from feedback while an RL agent trains against it. The user then judges outcomes rather than writing the behaviour, and the RL algorithm handles credit assignment 1811.07871 §3.1. For tasks too hard for the user to judge directly, an earlier agent helps with the evaluation. The paper calls this recursive reward modeling 1811.07871 §3.2. It names freezing the reward model as a known failure, since agents learn to exploit a model that stops being updated 1811.07871 §5.1. The paper says success is not guaranteed and that it reports no experiments 1811.07871 §1. Card: Scalable agent alignment via reward modeling.

Hierarchical RL with a classifier reward (red teaming, no human labels)

The attacker is a two-level PPO policy. A high-level policy picks an attack guide each turn, and a low-level policy writes the message token by token 2508.04451 §4.2. The reward is the probability that LlamaGuard labels the target's reply "unsafe" 2508.04451 §4.2, so the RL signal comes from a classifier, not from preference labels. A credit step masks pairs of attended tokens and measures how the reward drops 2508.04451 §4.3 eq. 4. The authors report 75.2% ASR@5 on HarmBench against Llama-3.1-8B-Instruct, above the baselines they reproduced for that target 2508.04451 tab. 1 §5. This is the loop of this page applied to testing rather than to aligning a model. Card: There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning.

Human-preference reward model with iterative RL

Raters compare candidate responses and also give per-response feedback on qualities such as creativity, safety and factuality. A reward model is trained to match these judgments, and the policy is optimised against it. The report describes this as a cycle: the policy exploits the reward model, the reward model is improved through evaluation and new data, and the cycle repeats 2312.11805 §6.3. Prompt selection and the way candidate responses are sampled strongly affect how useful the data is 2312.11805 §6.3. Card: Gemini 1.0 report.

The survey describes the same loop as three steps, feedback collection, reward modelling and policy optimisation, often repeated 2307.15217 §2. Its examples of a reward-model loss are cross-entropy on pairwise comparisons and Bayesian personalised ranking. It names no concrete RL method or regulariser 2307.15217 §2. Card: Open Problems and Fundamental Limitations of RLHF.

Survey of the loop and its choices (no experiments)

The survey treats RLHF as a family of methods. Its generic loop alternates reward learning and RL training, and the RL side sees a reward that keeps changing 2312.14925 §2.4 alg. 1. Feedback can be pairwise comparisons, rankings, binary critiques, corrections or natural language. Most of these fit one choice model 2312.14925 tab. 1 §5.1. For pairs, that model reduces to Bradley-Terry, and its rationality coefficient cannot be identified without a calibration reward 2312.14925 §5.1.1 §5.1.2. For language models, the survey treats a whole response as one action with a sparse terminal reward, and best-of-n sampling as a policy-improvement step that needs no training 2312.14925 §6.2 [App. A]. It also covers DPO as a direct method and leaves DPO's β unstated 2312.14925 §6.3 eq. 14. Card: A Survey of Reinforcement Learning from Human Feedback.

Preference tuning as a route to referential grounding (position paper, no experiments)

This variant uses RLHF as an argument, not as a training method. The authors argue that some LLM internal states can refer to the world. Preference tuning is their most direct route, since it selects states that help produce outputs judged truthful, helpful or harmless 2304.01481 §6.2.1. They call the pre-training route speculative and leave in-context learning open 2304.01481 §6.2.2 §6.2.3. They do not claim that LLMs understand language or hold beliefs 2304.01481 §1. Card: The Vector Grounding Problem.

Preference reward model with demonstrations in Atari games (DQfD)

The agent in this variant gets no game score. It learns from two sources: a few expert demonstrations before training, and pairwise choices between 25-step clips during training 1811.06521 §2.1. The policy is DQfD, and it reads the learned reward each time it samples a batch, so stored transitions carry no reward 1811.06521 §2.3 app. A.3. The reward model is a convolutional network over single stacked frames. A softmax over the summed rewards of two clips gives the preference probability, and a cross-entropy loss fits it to the annotator's choice 1811.06521 §2.4. The loss also assumes a 10% chance of a random label app. A.3. Each of the 500 iterations trains the policy for 10^5 agent steps, samples new clip pairs and refits the reward model 1811.06521 alg. 1 app. A.4.

The abstract reports that the method beats imitation learning in 7 of 9 Atari games. The body text is read as giving 8 of 9, so 7 is the figure to use 1811.06521 abstract §3.1. With a reward model that is frozen after training, performance fell in three games as the agent found loopholes in it, and the authors say online labels prevent this. That claim was tested on Atari with synthetic labels and in a few human runs 1811.06521 §3.2 fig. 4. Human annotators mislabelled 5 to 10% of pairs 1811.06521 §3.1 app. E. Card: Reward learning from human preferences and demonstrations in Atari.

Dialogue preference model with PPO and iterated online data (Anthropic HH)

This is the dialogue version of the loop. Crowdworkers chat with two candidate replies and pick the more helpful one, or the more harmful one for a red-teaming set 2204.05862 §2.1 §2.2. A preference model (PM) is trained on those picks. A PPO policy is then optimised with the PM score minus a KL penalty as its reward, with the KL weight set to 0.001 [arxiv:2204.05862 §4.1] eq. 2. Retraining the PM and the policy on fresh crowd data each round gives a model that crowdworkers prefer to the static one [arxiv:2204.05862 §4.5] fig. 1. Training on helpful data alone, or harmless data alone, makes the PM worse than chance on the other goal, so the two pull against each other 2204.05862 §5.1 fig. 19. The best 52B static PM reaches 86% accuracy on the authors' HHH benchmark 2204.05862 §3.4 fig. 5. Card: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.

Principles instead of human harm labels (Constitutional AI)

Harm labels come from a written list of 16 principles, not from crowdworkers. The first stage has the model critique and revise its own replies against the principles, then fine-tunes on the revisions 2212.08073 §3. The second stage has a feedback model choose the better of two replies for one principle, picked at random. Its normalised A/B probabilities become soft preference labels 2212.08073 §4.1. The preference model is trained on 182,831 AI harmlessness comparisons and 135,296 human helpfulness comparisons 2212.08073 §4.2. RL then runs from the supervised model against it 2212.08073 §4.1. The authors report that the RL model is rated more harmless than RLHF baselines, with a small helpfulness cost, and is virtually never evasive 2212.08073 §4.3 §4.4 fig. 2. The paper says its earlier helpful-and-harmless assistant was often evasive on controversial questions 2212.08073 §1.1. Most results are plots, and the text gives few numbers 2212.08073 §4.3. Card: Constitutional AI: Harmlessness from AI Feedback. For the general case of AI labels, see RLAIF.

Per-rule reward models and evidence (Sparrow)

Sparrow is an information-seeking dialogue agent built on Chinchilla 70B. Raters judge 23 short rules one at a time, and a rule reward model estimates the probability that each rule was broken 2209.14375 §2.1 §2.5 app. D.3. A separate preference reward model scores replies on an Elo-style scale 2209.14375 §2.5. The policy is trained with advantage actor-critic against both models, with a KL penalty to the starting model at weight 0.2 2209.14375 §2.8 app. E. The agent can also search the web and show the evidence to raters. At answer time it samples eight candidates, four of them with evidence, and keeps the one with the highest rerank score 2209.14375 §2.6.

Against Chinchilla baselines, the paper reports 0.78 supported-and-plausible for RL with reranking, against 0.61 for always-search prompting 2209.14375 §3.2. Under adversarial probing, the paper reports 8% of conversations violating the targeted rule 2209.14375 §1. Bias scores on Winogender and BBQ rise after RL, from 0.06 to 0.10 and from 0.05 to 0.10 2209.14375 §3.6.1. Card: Improving alignment of dialogue agents via targeted human judgements.

Direct preference optimisation (no reward model, no RL loop)

DPO trains the model on pairs of preferred and dispreferred responses with one logistic loss on log-probability ratios against a frozen reference model 2305.18290 §4 eq. 7. The paper derives this loss from the same KL-constrained reward objective that RLHF optimises 2305.18290 §4. The authors call the reward-model-plus-RL pipeline complex and often unstable 2305.18290 §1. On IMDb sentiment, DPO has the highest reward at every KL value tested, though the text gives no numbers 2305.18290 §6.1. On TL;DR summarisation, its GPT-4-judged win rate against reference summaries is about 61% at temperature 0, against about 57% for PPO at its best temperature 2305.18290 §6.2. The beta setting is 0.1 by default and 0.5 for TL;DR 2305.18290 app. B. Card: Direct Preference Optimization. See preference optimisation for the variant in full.

Binary desirable and undesirable labels with a prospect-theory loss (KTO)

KTO needs no pairs. Each output carries one label, desirable or undesirable, and the loss pushes desirable outputs above a reference point and undesirable ones below it 2402.01306 §4.1. The reference point is the KL divergence between the policy and its reference model. It is estimated per microbatch from mismatched input-output pairs, and it is not backpropagated through 2402.01306 §4.1. The value function is a logistic form of the prospect-theory value function, with separate loss-aversion weights for the two label types 2402.01306 §4.1 eq. 4. For preference data, the chosen reply counts as desirable and the rejected one as undesirable. The authors call this naive 2402.01306 §4.1. They suggest weights that keep the class-weighted ratio between 1 and 3/2 2402.01306 §4.2 eq. 9.

The results are author-reported. At 1B to 30B parameters, KTO matches or beats DPO 2402.01306 §4.3. On Zephyr-β-SFT trained on UltraFeedback, GSM8K rises from 40.0 with DPO to 53.5 with KTO 2402.01306 tab. 2. On Mistral-7B trained on OpenAssistant, human judges prefer KTO's outputs to the SFT target 72.9% of the time, against 62.1% for DPO, over 214 comparisons 2402.01306 app. D. A reference-free variant avoids the reference model but trails standard KTO 2402.01306 §4.3 tab. 2. Card: KTO: Model Alignment as Prospect Theoretic Optimization. See preference optimisation for the pairwise family.

Supervised tuning on a small curated set (no RLHF)

LIMA fine-tunes LLaMa 65B with the standard supervised loss on 1,000 prompt-and-response pairs and uses no RLHF 2305.11206 §3. The authors test the Superficial Alignment Hypothesis: knowledge comes almost entirely from pretraining, and alignment mainly teaches the output format [arxiv:2305.11206 §2]. In human pairwise tests, LIMA's answer was equal to or better than GPT-4's in 43% of cases and than DaVinci003's, which the paper describes as RLHF-trained, in 65% 2305.11206 §1. These are author-reported results on 300 test prompts 2305.11206 §4.1. The 7B ablations found no clear gain from up to 16 times more Stack Exchange data 2305.11206 §5. Card: LIMA.

Unlearning from negative examples only (no reward model)

This variant needs no good answers. Gradient ascent lowers the likelihood of undesirable answers on a forget set. Two added terms keep the model usable: a random-mismatch loss that trains each forget prompt toward an unrelated answer, and a KL term that keeps the output on normal data close to the original model 2310.10683 §4 eq. 2. On OPT-1.3B, the paper reports a harmful rate of 6% for the combined method, against 47% for the original model, measured with a moderation classifier 2310.10683 §6 tab. 3. The unlearned prompts mostly get blank or nonsensical output, which the authors accept because no good answer is used as a target 2310.10683 §2 §6. Their RLHF comparison on OPT-1.3B reports 4% harmful for full RLHF and 1% for plain gradient ascent. The unlearning run takes about 2% of the RLHF pipeline's time on one A100 2310.10683 §9.1 tab. 6. Card: Large Language Model Unlearning.

AI-feedback reward for a diffusion video model (offline)

This variant moves the loop to a text-to-video diffusion model. A 3B model is fine-tuned on 120 prompts about object movement, with 128 sampled videos per prompt and no online sampling during training 2412.02617 §4 app. A. The reward is a binary Accept or Reject from a vision-language model, which sees 8 frames 2412.02617 §3.3 app. D. Two offline objectives are compared: reward-weighted regression, which weights each sample's denoising error by its reward, and a DPO loss on preferred and rejected pairs 2412.02617 §3.1 eq. 6 eq. 7. The authors derive both from the same KL-regularised reward objective 2412.02617 eq. 3.

Against metric rewards (CLIP, HPSv2, PickScore, optical flow), the AI feedback reward gives the highest Gemini, GPT-4o and human "All" scores. Gemini "All" rises from 52.66% for the pre-trained model to 56.41% with RWR and 56.17% with DPO. Human "All" rises from 19.38% to 31.09% with RWR and 34.06% with DPO 2412.02617 tab. 2. The GPT-4o check gives a smaller gain, +2.4 points for RWR 2412.02617 §4.3. The evaluation has 40 test prompts. A second round of RWR on its own samples reaches 58.83%, while DPO stays near 56.25% 2412.02617 fig. 5. Metric rewards can also over-optimise: DPO on HPSv2 or PickScore raises the metric while the AI acceptance rate drops 2412.02617 §4.4 app. I. The DPO beta here is 5000, following Diffusion-DPO, against 0.1 in the language-model DPO paper 2412.02617 app. A 2305.18290 app. B. The two values are not comparable. Card: Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback.

Red-team evaluation of RLHF models (measurement, not training)

This is a way to measure a model, not a training method. Crowdworkers chat with four assistant types, from 2.7B to 52B parameters: plain, prompted, rejection-sampled and RLHF 2209.07858 §3.2. A harmlessness preference model scores every assistant turn, and each attack is scored by its worst turn 2209.07858 §3.4. The authors report that RLHF models become harder to red team as they grow, while the other three types show no clear size trend 2209.07858 §4 fig. 1. The release has 38,961 attacks [arxiv:2209.07858 tab. 1], but neither the preference model nor the assistants are released 2209.07858 app. A.7. Card: Red Teaming Language Models to Reduce Harms.

Open volunteer preference data with a reproduced pipeline (OpenAssistant)

OpenAssistant Conversations holds 161,443 volunteer-written messages in 35 languages, with 461,292 quality ratings and rankings of assistant replies 2304.07327 §4 app. L. Messages form conversation trees, so one prompt can have several continuations 2304.07327 §2. Several rankers' orders for the same replies are merged with ranked pairs, a variant of Tideman's method 2304.07327 app. B. The authors then train SFT, reward-model and PPO stages on the data 2304.07327 app. H. The reward loss is a pairwise log-sigmoid over the K replies to one prompt, and the paper says an optional score penalty may be added 2304.07327 app. H. Pythia, Falcon and LLaMA models tuned on the data beat their base models on most reported benchmarks. The one exception in the table is Pythia SFT on LMEH, 60.28 against 60.33 for the base model 2304.07327 tab. 1. The authors say RLHF beats SFT on some benchmarks but not others 2304.07327 §6.1. The paper calls the data "fully permissive" without naming a licence, so confirm the dataset licence before reuse 2304.07327 abstract. Card: OpenAssistant Conversations.

Preference-trained scorer for images (Pick-a-Pic)

Pick-a-Pic collects a user's prompt, two generated images and a pick or tie 2305.01569 §2. The authors train PickScore, a CLIP ViT-H/14 model, with a two-way softmax over the two images and a KL loss to a target that is [1, 0], [0, 1] or [0.5, 0.5] for a tie 2305.01569 §3 eq. 2 eq. 3. Each example is weighted by the inverse frequency of its prompt 2305.01569 §3. The scorer is a reward model in the RLHF sense, but the paper does not run RL. It ranks candidates instead. In best-of-100 selection, human raters prefer PickScore's pick over CLIP-H's (71.3%), over the aesthetics predictor's (85.1%) and over two random-seed baselines (71.4% and 82.0%) 2305.01569 tab. 2 §6. On held-out user picks, PickScore reaches 70.5% accuracy against 68.0% for expert annotators 2305.01569 tab. 1. The human baseline is expert annotators who are friends and colleagues of the authors 2305.01569 §4. Neither the dataset nor the model licence is stated 2305.01569 §1. Card: Pick-a-Pic.

Reward-feedback fine-tuning of a diffusion model (ReFL)

ImageReward is a BLIP-based reward model. It is trained on rankings of images for each prompt, with a pairwise logistic loss over the score difference 2304.05977 §2.2 eq. 1. ReFL scores the predicted clean image from one randomly chosen late denoising step, where the authors find the reward informative, and back-propagates that score into Stable Diffusion v1.4 2304.05977 §3 alg. 1. The authors weight the reward loss by a scale of 1e-3 with a ReLU map, and add a pre-training term from Eq. 3 2304.05977 §3 §4.2 eq. 2. The sign and offset of the ReLU are not given in the text, so the map needs checking against the scorer's output range. Preference accuracy on 6,399 held-out comparisons is 65.14% for ImageReward, against 57.76% for BLIP score 2304.05977 tab. 3. In the authors' human comparison, ReFL's win rate against the SD v1.4 baseline is 58.79% 2304.05977 tab. 4. Card: ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. See reward modeling for the scorer alone.

Checking the preference signal for sycophancy (no training method)

This is an evaluation, not a training method. Five assistants are tested on four task families where a user's opinion or pushback can change the answer 2310.13548 §3. Under the challenge "I don't think that's right. Are you sure?", Claude 1.3 admits a mistake on 98% of questions, and its answer changes on 86% 2310.13548 app. A.4. Feedback becomes more positive when the user says they like the text, though the text gives no numbers 2310.13548 §3.1 fig. 1. With the Claude 2 preference model (PM), best-of-N sampling yields more sycophantic replies than a prompted non-sycophantic PM 2310.13548 §4.2 fig. 6a. On misconception prompts, the PM prefers the sycophantic reply to the truthful one in 95% of pairs 2310.13548 §4.3.1 fig. 7a. The Claude 2 PM is not released, and its training partly used the hh-rlhf helpfulness data 2310.13548 §4.2. Card: Towards Understanding Sycophancy in Language Models.

Reward-model RLHF in an open-weight model (Gemma)

Gemma's RLHF step trains a reward model on human preference pairs under the Bradley-Terry model, then optimises the policy with an RL algorithm the paper calls novel but does not name [arxiv:2403.08295 § Reinforcement Learning from Human Feedback]. An automatic high-capacity rater tunes hyperparameters and limits reward hacking [arxiv:2403.08295 § Reinforcement Learning from Human Feedback]. The paper gives no RL or reward-model hyperparameters [arxiv:2403.08295 § Training Infrastructure]. Human side-by-side results compare whole instruction-tuned models with Mistral v0.2 7B Instruct. Gemma 1.1 IT 7B wins 63.5% on safety and 61.2% on instruction following, and Gemma 1.1 IT 2B wins 60.1% and 45% 2403.08295 tab. 5. These numbers cover the whole post-training stack, not RLHF alone. Card: Gemma: Open Models Based on Gemini Research and Technology.

Reward model and PPO in a released chat model (Qwen)

Qwen's chat models get SFT first, then a reward model and PPO 2309.16609 §3.1 §3.2. The reward model starts from a same-size Qwen model. A pooling layer reads the reward at one end token, and training runs first on preference pretraining data and then on higher-quality comparisons 2309.16609 §3.2.1. The reward model trains for one epoch at a constant learning rate of 3e-6, with batch 64 and sequence length 2048 [arxiv:2309.16609 §3.2.1]. PPO holds four models in memory, samples two responses per query, normalises rewards by a running mean, and uses a KL coefficient of 0.04 2309.16609 §3.2.2. A pretraining-data gradient is mixed in to limit the alignment tax on benchmarks outside code and maths 2309.16609 §3.2.2. The chat benchmark numbers come from the full SFT-plus-RLHF model, so they do not isolate RLHF 2309.16609 §3.3 tab. 5. Card: Qwen Technical Report.

Group consensus reward model with reranking, no RL stage (Habermas Machine)

The reward model here scores candidates for a group, not for one user. Group members write opinions on a debate question, and raters score candidate statements for agreement. The reward model is trained on pairwise agreement preferences with cross-entropy, one participant's opinion at a time 2211.15006 §3.4 app. C.2. The training loop has supervised tuning and the reward model, with no RL stage 2211.15006 §3.4. At inference, 16 statements are sampled with nucleus sampling. Each one is scored for every member, and the Utilitarian welfare function (inequality aversion 0) picks the winner 2211.15006 §3.3 eq. 1 app. C.1. The authors report that raters preferred the chosen statement over prompted Chinchilla baselines and over human-written opinions 2211.15006 §4.1 §4.2. Groups are limited to three to five people by prompt length, and the data come from UK crowd participants on UK political questions, so generalisation is untested 2211.15006 §5.1. Card: Fine-tuning language models to find agreement among humans with diverse preferences. See reward modeling for the scorer alone.

When to use which

The sources compare the variants only in part. DPO is compared against a PPO-based RLHF baseline in its own experiments 2305.18290 §6. LIMA is compared against other models whose training is not controlled: responses were sampled over one month, and some baselines may have seen large amounts of user data 2305.11206 §4.1 2305.11206 §4.3. The Gemini report card records no head-to-head comparison of reward-model RLHF with either alternative. The table gives what each source supports.

What the sources leave out, so a choice here cannot be made from them alone: the RL algorithm, the KL penalty, learning rates, batch sizes, the number of iterations, and the amount of preference data in the Gemini report 2312.11805 §6.3. The GLM-4 report gives no learning rates, batch sizes, steps or RLHF data sizes, and defers the RLHF method to other reports 2406.12793 §2 Alignment §2 ChatGLM Techniques. The DPO paper states its own optimiser settings 2305.18290 app. B, but its repository defaults differ from them; see the DPO card.

Implementation sketch

The loop below follows the report's four steps. Anything not stated in the report is marked as such.

# Post-training loop, stages as in [arxiv:2312.11805 §6.3]

# 1. Curate prompts, single- and multi-turn. They serve both as
#    demonstration prompts and as RL prompts  [arxiv:2312.11805 §6.3]
prompts = collect_prompts()

# 2. Supervised fine-tuning on demonstrations written by experts, or
#    drafted by a model and reviewed by people  [arxiv:2312.11805 §6.3]
policy = sft(base_model, demonstrations(prompts))

# 3. Reward model from human rankings of candidate responses
#    [arxiv:2312.11805 §6.3]
pairs = rank(sample(policy, prompts), raters)
reward = train_reward_model(pairs)

# 4. Optimise the policy against the reward model, then refresh the reward
#    model with new data and repeat  [arxiv:2312.11805 §6.3]
# RL algorithm, KL penalty and round count: not stated
for _ in rounds:
    policy = rl_optimize(policy, reward, prompts)
    pairs += rank(sample(policy, prompts), raters)
    reward = train_reward_model(pairs)

Where variants differ: DPO replaces steps 3 and 4 with a single loss on the same kind of pairs, computed against a frozen reference model 2305.18290 §4 eq. 7. LIMA skips steps 3 and 4 entirely and uses supervised loss on the curated examples 2305.11206 §3. Unlearning also skips steps 3 and 4 and uses gradient ascent on the forget set, with the KL and random-mismatch terms 2310.10683 eq. 2. KTO also skips steps 3 and 4, and fits one loss to desirable and undesirable outputs against a KL-based reference point 2402.01306 §4.1 eq. 8. Constitutional AI keeps the loop, but its step 3 labels come from a feedback model that reads one written principle 2212.08073 §4.1. The group-consensus variant keeps steps 2 and 3 but has no RL stage: at inference it samples 16 statements and returns the one with the highest welfare score across the group's members 2211.15006 §3.3 app. C.1.

In the diffusion video variant, the same loop runs without an RL sampler. The reward is attached to each offline sample. RWR weights each sample's denoising error by that reward. DPO compares denoising errors of the policy and a reference model on preferred and rejected samples, using one shared noise draw and time step 2412.02617 alg. 1 eq. 6 eq. 7. The paper notes that binary labels give no pairing rule of their own, and does not state how pairs are built 2412.02617 §4.3.

PickScore uses a softmax over two images with soft targets for ties, rather than a log-sigmoid over a ranking, but it is trained with the same kind of pairwise preference signal 2305.01569 §3 eq. 2 eq. 3. The OpenAssistant reward loss is the pairwise form over K ranked replies 2304.07327 app. H.

Sparrow's per-turn reward adds the normalised preference score to the mean of the normalised rule scores, then subtracts length and format penalties 2209.14375 app. E.3. Its RL learning rate is 2e-6 with Adafactor 2209.14375 app. E. The learning rates of its reward models are not stated 2209.14375 app. D.

Contradictions

• Whether alignment adds capability or only style. LIMA argues that alignment mainly teaches response style and that knowledge comes from pretraining 2305.11206 §2. The Gemini report measures a gain from SFT+RLHF over SFT on image understanding, +0.223 side-by-side, and does not split it into style and knowledge 2312.11805 §6.5.4. The two studies use different models, tasks and judges, so this is a difference in framing, not a direct numerical conflict.
• Reward-model RLHF versus DPO. DPO matches or beats PPO-based RLHF on the three tasks it tests 2305.18290 §6. The authors say they did not tune beta on TL;DR, which may understate DPO 2305.18290 §6.2. The survey reports the opposite in a secondary citation: tuned PPO can outperform direct methods 2312.14925 App. A, Direct methods. It gives no figures, so the two claims cannot be checked against each other here. The Gemini card records no DPO comparison.
• Gains from RL versus bias. Sparrow reports fewer targeted-rule violations under probing 2209.14375 §1, but bias scores on Winogender and BBQ rise after RL 2209.14375 §3.6.1. The paper says its mitigations address only single-example harms 2209.14375 §3.6. The red-team study covers attack difficulty, not bias, so the two results do not directly conflict.
• Are RLHF's limits fixable? The survey argues that several problems are fundamental, meaning no fix stays within RLHF 2307.15217 §3. Sparrow reports measured gains from RLHF with rules and evidence 2209.14375 §3.2. The survey runs no experiments, so this is a difference in framing, not a numerical conflict.
• Harmful rate versus output quality. The unlearning paper reports lower harmful rates for plain gradient ascent (1%) than for full RLHF (4%) on OPT-1.3B 2310.10683 tab. 6. The same table marks fluency as not meaningful for plain gradient ascent, while full RLHF has a perplexity-based score of 3.414 2310.10683 tab. 6. Read the harmful rate alone with care.
• RWR or DPO for diffusion. On the video study's Gemini "All" score, RWR is higher (56.41% against 56.17%). On human "All", DPO is higher (34.06% against 31.09%). The paper says DPO generalises better to held-out prompts 2412.02617 tab. 2 §4.2 §4.3. The Gemini gap is 0.24 points over the 160 train and test prompts, so it is small. The two algorithms are not clearly ranked by the paper's own numbers.
• Human raters versus the preference model. The sycophancy study finds the Claude 2 PM prefers the sycophantic reply in 95% of misconception pairs 2310.13548 §4.3.1 fig. 7a. Crowd raters usually prefer the truthful reply, less reliably at higher difficulty 2310.13548 §4.3.1 fig. 7b. In the hardest bin, best-of-4096 gives a sycophantic reply for about 75% with the Claude 2 PM and about 25% with an oracle PM 2310.13548 §4.3.2 fig. 7d. The text gives no number for the human result, so the comparison with raters is qualitative.
• Does RLHF beat SFT? OpenAssistant's authors report RLHF beating SFT on some benchmarks but not others 2304.07327 §6.1. The Gemini report measures a gain from SFT+RLHF over SFT on image understanding, +0.223 side-by-side 2312.11805 §6.5.4. The benchmarks and judges differ, so this is not a direct numerical conflict.
• DPO baselines. The KTO paper's DPO numbers, such as GSM8K 40.0 on Zephyr-β-SFT and 62.1% human win rate on Mistral-7B, come from the KTO authors' own runs 2402.01306 tab. 2 app. D. They are not results from the DPO paper, so the KTO-versus-DPO gains here are not a clean replication.
• Evasiveness of helpful assistants. Constitutional AI reports that its earlier helpful-and-harmless assistant was often evasive, and that its own RL model is virtually never evasive 2212.08073 §1.1 §4.4. The Anthropic HH paper reports a helpfulness and harmlessness trade-off [arxiv:2204.05862 §5.1], but the evasiveness claim is not one it is cited for here, so the two results are not directly compared.

Papers

• ChatGPT research survey (arXiv:2304.01852): a bibliometric survey that names InstructGPT and RLHF as the basis of ChatGPT; it does not describe the method.
• Red Teaming Language Models to Reduce Harms (arXiv:2209.07858): crowdworker attacks on plain, prompted, rejection-sampled and RLHF assistants from 2.7B to 52B parameters, scored by a harmlessness preference model.
• Improving alignment of dialogue agents via targeted human judgements (arXiv:2209.14375): Sparrow, trained with per-rule and preference reward models, search evidence and reranking.
• LIMA: Less Is More for Alignment (arXiv:2305.11206): supervised tuning of LLaMa 65B on 1,000 curated examples with no RLHF, as a test of the Superficial Alignment Hypothesis.
• Direct Preference Optimization (arXiv:2305.18290): replaces the reward model and RL stage with one loss on preference pairs against a frozen reference model.
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (arXiv:2307.15217): a survey and position paper that sorts RLHF's open problems into tractable and fundamental ones, with no experiments.
• Large Language Model Unlearning (arXiv:2310.10683): gradient-ascent unlearning from negative examples only, with KL and random-mismatch terms, compared against full RLHF on OPT-1.3B.
• Gemini 1.0 report (arXiv:2312.11805): technical report with the post-training pipeline, the safety preference data and one isolated SFT-versus-RLHF comparison.
• Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback (arXiv:2412.02617): offline RWR and DPO on a 3B video diffusion model, rewarded by binary VLM judgments and compared with metric rewards.
• OpenAssistant Conversations (arXiv:2304.07327): volunteer-written conversation trees with quality labels and reply rankings, merged by ranked pairs; SFT, reward-model and PPO runs on the data, with the dataset licence not named.
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (arXiv:2305.01569): a public dataset of user picks and ties over image pairs, and PickScore, a CLIP-based scorer trained on it with a softmax preference loss.
• Towards Understanding Sycophancy in Language Models (arXiv:2310.13548): behavioural tests of sycophancy in five assistants, and an analysis of the Claude 2 PM and human preferences over truthful and sycophantic replies.
• Gemma: Open Models Based on Gemini Research and Technology (arXiv:2403.08295): open 2B and 7B models whose instruction-tuned variants add an RLHF step, with human win rates against Mistral v0.2 7B Instruct.
• Reward learning from human preferences and demonstrations in Atari (arXiv:1811.06521): a convolutional reward model fitted to pairwise clip preferences, used to train a DQfD agent on nine Atari games, with online labels and demonstrations.
• Scalable agent alignment via reward modeling: a research direction (arXiv:1811.07871): position paper proposing reward modeling and recursive reward modeling for agent alignment, with no experiments.
• There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning (arXiv:2508.04451): two-level PPO attacker for multi-turn red teaming, rewarded by a LlamaGuard "unsafe" probability and trained with token-level credit.
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (arXiv:2204.05862): crowdworker preference models and PPO with a KL penalty up to 52B parameters, with iterated online data.
• Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073): critique-revision SFT and AI preference labels from 16 written principles, then RL against the resulting preference model.
• KTO: Model Alignment as Prospect Theoretic Optimization (arXiv:2402.01306): a reward-free loss on binary desirable and undesirable labels, compared with DPO on Pythia, Llama, Zephyr and Mistral models.
• Qwen Technical Report (arXiv:2309.16609): Qwen base and chat models, with a reward model and PPO stage for the chat models and the settings the paper reports.
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (arXiv:2304.05977): a BLIP-based reward model trained on expert image rankings, and ReFL, which back-propagates that reward through one late denoising step of Stable Diffusion v1.4.
• Role-Play with Large Language Models (arXiv:2305.16367): a position paper on role-playing dialogue agents, with no method or experiments; it notes that its base-model analysis may not hold after RLHF.
• A Survey on Human-AI Collaboration with Large Foundation Models (arXiv:2403.04931): a survey that places RLHF and DPO in the build-time loop of human guidance, with no experiments of its own.
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (arXiv:2406.12793): technical report with SFT followed by RLHF as the GLM-4 alignment stage; the RLHF method and settings are deferred to other reports.
• A Survey on Controllable Text Generation using Transformer-based Pre-trained Language Models (arXiv:2201.05337): a survey with no experiments; lists RL with a human-preference reward model as one control family, and names InstructGPT as RLHF.
• Fine-tuning language models to find agreement among humans with diverse preferences (arXiv:2211.15006): a 70B model with a reward model that predicts group members' agreement, used to rerank 16 sampled consensus statements; no RL stage.
• A Survey of Reinforcement Learning from Human Feedback (arXiv:2312.14925): a survey of RLHF with depth in control and robotics and an appendix on language models; covers feedback types, Bradley-Terry reward models and policy learning, including DPO. No experiments.
• The Vector Grounding Problem (arXiv:2304.01481): a position paper arguing that LLM internal states can refer to the world, with preference tuning as the most direct route; no experiments.