---
kind: concept
slug: rlhf
title: Reinforcement learning from human feedback (RLHF)
summary: >-
  Post-training that fits a reward model to human preference judgments, then optimises the model with RL after supervised fine-tuning. DPO and KTO drop the reward model and RL loop; LIMA drops RLHF; Constitutional AI swaps human harm labels for AI feedback. Extensions cover data, images and video; sycophancy and red-team studies test side effects. Results are author-reported.
tags: [post-training, alignment, reward-model, reinforcement-learning]
papers: ["1811.06521", "1811.07871", "2201.05337", "2204.05862", "2209.07858", "2209.14375", "2211.15006", "2212.08073", "2304.01481", "2304.01852", "2304.05977", "2304.07327", "2305.01569", "2305.11206", "2305.16367", "2305.18290", "2307.15217", "2309.16609", "2310.10683", "2310.13548", "2312.11805", "2312.14925", "2402.01306", "2403.04931", "2403.08295", "2406.12793", "2412.02617", "2508.04451"]
related: [preference-optimization, instruction-tuning, dataset-curation, llm-as-a-judge, policy-gradient, chain-of-thought-prompting, hallucination-mitigation, llm-ethics, llm-evaluation, machine-unlearning, retrieval-augmented-generation, diffusion-models, bradley-terry-model, red-teaming, text-to-image-generation, human-evaluation, contrastive-learning, reward-modeling, hierarchical-reinforcement-learning, inverse-reinforcement-learning, rlaif, context-distillation, human-ai-collaboration, human-in-the-loop, language-model-pretraining, controllable-text-generation]
status: draft
updated_at: 2026-11-01
---

## Overview

RLHF is a post-training stage. A pre-trained language model is trained further
so that its outputs match what human raters prefer. The Gemini 1.0 report lays
the loop out in four steps: collect prompts, fine-tune on demonstrations, train
a reward model on human judgments, then run reinforcement learning against that
reward model and repeat [arxiv:2312.11805 §6.3]. The shared invariant is that
the training signal comes from a learned model of human preferences rather than
from a fixed metric. The ChatGPT survey lists InstructGPT and RLHF as the basis
of ChatGPT but does not describe the method [arxiv:2304.01852 §1], so the
mechanics of the reward-model loop on this page rest on the Gemini report alone.
An earlier position paper proposes the same idea as a research direction: learn
a reward function from user feedback while an RL agent trains on it. It reports
no experiments [arxiv:1811.07871 §1] [§3.1].

A survey covers the same loop in more depth, mostly for control and robotics,
with language models in an appendix. It fits a reward model to pairwise
comparisons under a Bradley-Terry likelihood [arxiv:2312.14925 §2.3] [eq. 2],
then trains a policy against it, refitting the reward model as new labels
arrive [arxiv:2312.14925 §2.4] [alg. 1]. It runs no experiments of its own
[arxiv:2312.14925 §1].

Two later papers offer alternatives to parts of this loop. DPO fits the
preference data directly, with no separate reward model and no RL sampling
loop [arxiv:2305.18290 §4]. LIMA reaches its results with supervised tuning
alone, on 1,000 curated examples and no preference model [arxiv:2305.11206 §3].
A third paper removes the good-answer side entirely and unlearns only bad
answers [arxiv:2310.10683 §4]. KTO keeps a reward-free loss but needs only a
desirable or undesirable label per output, not a pair [arxiv:2402.01306 §4.1].

Other papers apply the loop to new data or check what it rewards. OpenAssistant
releases volunteer-written conversations with rankings and runs supervised,
reward-model and PPO stages on them [arxiv:2304.07327 §1] [app. H]. Gemma
describes an RLHF step for its instruction-tuned models, with the RL algorithm
unnamed [arxiv:2403.08295 § Reinforcement Learning from Human Feedback].
A survey of controllable text generation lists reinforcement learning with
a reward model trained on human preferences as one family of control
methods, and names InstructGPT as an RLHF example [arxiv:2201.05337 §3.2].
Pick-a-Pic trains a preference scorer for images without running RL
[arxiv:2305.01569 §3]. A sycophancy study finds that a preference model can
prefer agreeable replies over truthful ones [arxiv:2310.13548 §4.3.1].

A 2023 survey sorts the open problems of the loop into challenges with the
feedback, the reward model, the policy and joint training, and labels each as
tractable or fundamental. It runs no experiments [arxiv:2307.15217 §3]
[app. B]. A 2025 red-teaming study runs PPO against a classifier's "unsafe"
probability, with no human labels in the loop [arxiv:2508.04451 §4.2].

Four more papers touch the loop from other sides. ImageReward trains a reward
model on expert image rankings, then fine-tunes a diffusion model by
back-propagating that score through one late denoising step [arxiv:2304.05977
§3]. The GLM-4 report names SFT followed by RLHF as its alignment stage but
leaves the RLHF method and its settings to other reports [arxiv:2406.12793 §2
Alignment]. A survey of human-AI collaboration places RLHF and DPO under
objective shaping in its build-time loop, and reports no experiments of its own
[arxiv:2403.04931 §2.2]. A position paper on role-play says its base-model
analysis may not hold after RLHF-style tuning [arxiv:2305.16367 §2] [§8].
A position paper on referential grounding names preference tuning as the most
direct way for a model's internal states to come to carry meaning
[arxiv:2304.01481 §6.2.1].

## Variants

### Reward modeling as a research direction (position paper, no experiments)

The user trains a reward model from feedback while an RL agent trains against
it. The user then judges outcomes rather than writing the behaviour, and the RL
algorithm handles credit assignment [arxiv:1811.07871 §3.1]. For tasks too hard
for the user to judge directly, an earlier agent helps with the evaluation. The
paper calls this recursive reward modeling [arxiv:1811.07871 §3.2]. It names
freezing the reward model as a known failure, since agents learn to exploit a
model that stops being updated [arxiv:1811.07871 §5.1]. The paper says success
is not guaranteed and that it reports no experiments [arxiv:1811.07871 §1].
Card: [Scalable agent alignment via reward modeling](/papers/1811.07871).

### Hierarchical RL with a classifier reward (red teaming, no human labels)

The attacker is a two-level PPO policy. A high-level policy picks an attack
guide each turn, and a low-level policy writes the message token by token
[arxiv:2508.04451 §4.2]. The reward is the probability that LlamaGuard labels
the target's reply "unsafe" [arxiv:2508.04451 §4.2], so the RL signal comes
from a classifier, not from preference labels. A credit step masks pairs of
attended tokens and measures how the reward drops [arxiv:2508.04451 §4.3] [eq. 4]. The authors report 75.2% ASR@5 on HarmBench
against Llama-3.1-8B-Instruct, above the baselines they reproduced for that
target [arxiv:2508.04451 tab. 1] [§5]. This is the loop of this page applied to testing rather than to
aligning a model. Card:
[There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning](/papers/2508.04451).

### Human-preference reward model with iterative RL

Raters compare candidate responses and also give per-response feedback on
qualities such as creativity, safety and factuality. A reward model is trained
to match these judgments, and the policy is optimised against it. The report
describes this as a cycle: the policy exploits the reward model, the reward
model is improved through evaluation and new data, and the cycle repeats
[arxiv:2312.11805 §6.3]. Prompt selection and the way candidate responses are
sampled strongly affect how useful the data is [arxiv:2312.11805 §6.3]. Card:
[Gemini 1.0 report](/papers/2312.11805).

The survey describes the same loop as three steps, feedback collection, reward
modelling and policy optimisation, often repeated [arxiv:2307.15217 §2]. Its
examples of a reward-model loss are cross-entropy on pairwise comparisons and
Bayesian personalised ranking. It names no concrete RL method or regulariser
[arxiv:2307.15217 §2]. Card: [Open Problems and Fundamental Limitations of RLHF](/papers/2307.15217).

### Survey of the loop and its choices (no experiments)

The survey treats RLHF as a family of methods. Its generic loop alternates
reward learning and RL training, and the RL side sees a reward that keeps
changing [arxiv:2312.14925 §2.4] [alg. 1]. Feedback can be pairwise
comparisons, rankings, binary critiques, corrections or natural language.
Most of these fit one choice model [arxiv:2312.14925 tab. 1] [§5.1]. For pairs,
that model reduces to Bradley-Terry, and its rationality coefficient cannot be
identified without a calibration reward [arxiv:2312.14925 §5.1.1] [§5.1.2].
For language models, the survey treats a whole response as one action with a
sparse terminal reward, and best-of-n sampling as a policy-improvement step
that needs no training [arxiv:2312.14925 §6.2] [App. A]. It also covers DPO as
a direct method and leaves DPO's β unstated [arxiv:2312.14925 §6.3] [eq. 14].
Card: [A Survey of Reinforcement Learning from Human Feedback](/papers/2312.14925).

### Preference tuning as a route to referential grounding (position paper, no experiments)

This variant uses RLHF as an argument, not as a training method. The authors
argue that some LLM internal states can refer to the world. Preference tuning
is their most direct route, since it selects states that help produce outputs
judged truthful, helpful or harmless [arxiv:2304.01481 §6.2.1]. They call the
pre-training route speculative and leave in-context learning open
[arxiv:2304.01481 §6.2.2] [§6.2.3]. They do not claim that LLMs understand
language or hold beliefs [arxiv:2304.01481 §1]. Card:
[The Vector Grounding Problem](/papers/2304.01481).

### Preference reward model with demonstrations in Atari games (DQfD)

The agent in this variant gets no game score. It learns from two sources: a
few expert demonstrations before training, and pairwise choices between 25-step
clips during training [arxiv:1811.06521 §2.1]. The policy is DQfD, and it reads
the learned reward each time it samples a batch, so stored transitions carry no
reward [arxiv:1811.06521 §2.3] [app. A.3]. The reward model is a convolutional
network over single stacked frames. A softmax over the summed rewards of two
clips gives the preference probability, and a cross-entropy loss fits it to the
annotator's choice [arxiv:1811.06521 §2.4]. The loss also assumes a 10% chance
of a random label [app. A.3]. Each of the 500 iterations trains the policy for
10^5 agent steps, samples new clip pairs and refits the reward model
[arxiv:1811.06521 alg. 1] [app. A.4].

The abstract reports that the method beats imitation learning in 7 of 9 Atari
games. The body text is read as giving 8 of 9, so 7 is the figure to use
[arxiv:1811.06521 abstract] [§3.1]. With a reward model that is frozen after
training, performance fell in three games as the agent found loopholes in it,
and the authors say online labels prevent this. That claim was tested on Atari
with synthetic labels and in a few human runs [arxiv:1811.06521 §3.2] [fig. 4].
Human annotators mislabelled 5 to 10% of pairs [arxiv:1811.06521 §3.1]
[app. E]. Card:
[Reward learning from human preferences and demonstrations in Atari](/papers/1811.06521).

### Dialogue preference model with PPO and iterated online data (Anthropic HH)

This is the dialogue version of the loop. Crowdworkers chat with two candidate
replies and pick the more helpful one, or the more harmful one for a
red-teaming set [arxiv:2204.05862 §2.1] [§2.2]. A preference model (PM) is
trained on those picks. A PPO policy is then optimised with the PM score minus
a KL penalty as its reward, with the KL weight set to 0.001 [arxiv:2204.05862
§4.1] [eq. 2]. Retraining the PM and the policy on fresh crowd data each round
gives a model that crowdworkers prefer to the static one [arxiv:2204.05862
§4.5] [fig. 1]. Training on helpful data alone, or harmless data alone, makes
the PM worse than chance on the other goal, so the two pull against each other
[arxiv:2204.05862 §5.1] [fig. 19]. The best 52B static PM reaches 86% accuracy
on the authors' HHH benchmark [arxiv:2204.05862 §3.4] [fig. 5]. Card:
[Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](/papers/2204.05862).

### Principles instead of human harm labels (Constitutional AI)

Harm labels come from a written list of 16 principles, not from crowdworkers.
The first stage has the model critique and revise its own replies against the
principles, then fine-tunes on the revisions [arxiv:2212.08073 §3]. The second
stage has a feedback model choose the better of two replies for one principle,
picked at random. Its normalised A/B probabilities become soft preference labels
[arxiv:2212.08073 §4.1]. The preference model is trained on 182,831 AI
harmlessness comparisons and 135,296 human helpfulness comparisons
[arxiv:2212.08073 §4.2]. RL then runs from the supervised model against it
[arxiv:2212.08073 §4.1]. The authors report that the RL model is rated more
harmless than RLHF baselines, with a small helpfulness cost, and is virtually
never evasive [arxiv:2212.08073 §4.3] [§4.4] [fig. 2]. The paper says its
earlier helpful-and-harmless assistant was often evasive on controversial
questions [arxiv:2212.08073 §1.1]. Most results are plots, and the text gives
few numbers [arxiv:2212.08073 §4.3]. Card:
[Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073). For the
general case of AI labels, see [RLAIF](/concepts/rlaif).

### Per-rule reward models and evidence (Sparrow)

Sparrow is an information-seeking dialogue agent built on Chinchilla 70B. Raters
judge 23 short rules one at a time, and a rule reward model estimates the
probability that each rule was broken [arxiv:2209.14375 §2.1] [§2.5] [app. D.3].
A separate preference reward model scores replies on an Elo-style scale
[arxiv:2209.14375 §2.5]. The policy is trained with advantage actor-critic
against both models, with a KL penalty to the starting model at weight 0.2
[arxiv:2209.14375 §2.8] [app. E]. The agent can also search the web and show
the evidence to raters. At answer time it samples eight candidates, four of
them with evidence, and keeps the one with the highest rerank score
[arxiv:2209.14375 §2.6].

Against Chinchilla baselines, the paper reports 0.78 supported-and-plausible
for RL with reranking, against 0.61 for always-search prompting
[arxiv:2209.14375 §3.2]. Under adversarial probing, the paper reports 8% of
conversations violating the targeted rule [arxiv:2209.14375 §1]. Bias scores on
Winogender and BBQ rise after RL, from 0.06 to 0.10 and from 0.05 to 0.10
[arxiv:2209.14375 §3.6.1]. Card:
[Improving alignment of dialogue agents via targeted human judgements](/papers/2209.14375).

### Direct preference optimisation (no reward model, no RL loop)

DPO trains the model on pairs of preferred and dispreferred responses with one
logistic loss on log-probability ratios against a frozen reference model
[arxiv:2305.18290 §4] [eq. 7]. The paper derives this loss from the same
KL-constrained reward objective that RLHF optimises [arxiv:2305.18290 §4]. The
authors call the reward-model-plus-RL pipeline complex and often unstable
[arxiv:2305.18290 §1]. On IMDb sentiment, DPO has the highest reward at every
KL value tested, though the text gives no numbers [arxiv:2305.18290 §6.1]. On
TL;DR summarisation, its GPT-4-judged win rate against reference summaries is
about 61% at temperature 0, against about 57% for PPO at its best temperature
[arxiv:2305.18290 §6.2]. The beta setting is 0.1 by default and 0.5 for TL;DR
[arxiv:2305.18290 app. B]. Card: [Direct Preference Optimization](/papers/2305.18290).
See [preference optimisation](/concepts/preference-optimization) for the
variant in full.

### Binary desirable and undesirable labels with a prospect-theory loss (KTO)

KTO needs no pairs. Each output carries one label, desirable or undesirable,
and the loss pushes desirable outputs above a reference point and undesirable
ones below it [arxiv:2402.01306 §4.1]. The reference point is the KL divergence
between the policy and its reference model. It is estimated per microbatch from
mismatched input-output pairs, and it is not backpropagated through
[arxiv:2402.01306 §4.1]. The value function is a logistic form of the
prospect-theory value function, with separate loss-aversion weights for the two
label types [arxiv:2402.01306 §4.1] [eq. 4]. For preference data, the chosen
reply counts as desirable and the rejected one as undesirable. The authors call
this naive [arxiv:2402.01306 §4.1]. They suggest weights that keep the
class-weighted ratio between 1 and 3/2 [arxiv:2402.01306 §4.2] [eq. 9].

The results are author-reported. At 1B to 30B parameters, KTO matches or beats
DPO [arxiv:2402.01306 §4.3]. On Zephyr-β-SFT trained on UltraFeedback, GSM8K
rises from 40.0 with DPO to 53.5 with KTO [arxiv:2402.01306 tab. 2]. On
Mistral-7B trained on OpenAssistant, human judges prefer KTO's outputs to the
SFT target 72.9% of the time, against 62.1% for DPO, over 214 comparisons
[arxiv:2402.01306 app. D]. A reference-free variant avoids the reference model
but trails standard KTO [arxiv:2402.01306 §4.3] [tab. 2]. Card:
[KTO: Model Alignment as Prospect Theoretic Optimization](/papers/2402.01306).
See [preference optimisation](/concepts/preference-optimization) for the
pairwise family.

### Supervised tuning on a small curated set (no RLHF)

LIMA fine-tunes LLaMa 65B with the standard supervised loss on 1,000
prompt-and-response pairs and uses no RLHF [arxiv:2305.11206 §3]. The authors
test the Superficial Alignment Hypothesis: knowledge comes almost entirely from
pretraining, and alignment mainly teaches the output format [arxiv:2305.11206
§2]. In human pairwise tests, LIMA's answer was equal to or better than GPT-4's
in 43% of cases and than DaVinci003's, which the paper describes as
RLHF-trained, in 65% [arxiv:2305.11206 §1]. These are author-reported results
on 300 test prompts [arxiv:2305.11206 §4.1]. The 7B ablations found no clear gain
from up to 16 times more Stack Exchange data [arxiv:2305.11206 §5]. Card:
[LIMA](/papers/2305.11206).

### Unlearning from negative examples only (no reward model)

This variant needs no good answers. Gradient ascent lowers the likelihood of
undesirable answers on a forget set. Two added terms keep the model usable: a
random-mismatch loss that trains each forget prompt toward an unrelated answer,
and a KL term that keeps the output on normal data close to the original model
[arxiv:2310.10683 §4] [eq. 2]. On OPT-1.3B, the paper reports a harmful rate
of 6% for the combined method, against 47% for the original model, measured
with a moderation classifier [arxiv:2310.10683 §6] [tab. 3]. The unlearned
prompts mostly get blank or nonsensical output, which the authors accept
because no good answer is used as a target [arxiv:2310.10683 §2] [§6]. Their
RLHF comparison on OPT-1.3B reports 4% harmful for full RLHF and 1% for plain
gradient ascent. The unlearning run takes about 2% of the RLHF pipeline's time
on one A100 [arxiv:2310.10683 §9.1] [tab. 6]. Card:
[Large Language Model Unlearning](/papers/2310.10683).

### AI-feedback reward for a diffusion video model (offline)

This variant moves the loop to a text-to-video diffusion model. A 3B model is
fine-tuned on 120 prompts about object movement, with 128 sampled videos per
prompt and no online sampling during training [arxiv:2412.02617 §4] [app. A].
The reward is a binary Accept or Reject from a vision-language model, which
sees 8 frames [arxiv:2412.02617 §3.3] [app. D]. Two offline objectives are
compared: reward-weighted regression, which weights each sample's denoising
error by its reward, and a DPO loss on preferred and rejected pairs
[arxiv:2412.02617 §3.1] [eq. 6] [eq. 7]. The authors derive both from the same
KL-regularised reward objective [arxiv:2412.02617 eq. 3].

Against metric rewards (CLIP, HPSv2, PickScore, optical flow), the AI feedback
reward gives the highest Gemini, GPT-4o and human "All" scores. Gemini "All"
rises from 52.66% for the pre-trained model to 56.41% with RWR and 56.17% with
DPO. Human "All" rises from 19.38% to 31.09% with RWR and 34.06% with DPO
[arxiv:2412.02617 tab. 2]. The GPT-4o check gives a smaller gain, +2.4 points
for RWR [arxiv:2412.02617 §4.3]. The evaluation has 40 test prompts. A second round of RWR on its own samples
reaches 58.83%, while DPO stays near 56.25% [arxiv:2412.02617 fig. 5]. Metric
rewards can also over-optimise: DPO on HPSv2 or PickScore raises the metric
while the AI acceptance rate drops [arxiv:2412.02617 §4.4] [app. I]. The DPO
beta here is 5000, following Diffusion-DPO, against 0.1 in the language-model
DPO paper [arxiv:2412.02617 app. A] [arxiv:2305.18290 app. B]. The two values
are not comparable. Card:
[Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](/papers/2412.02617).

### Red-team evaluation of RLHF models (measurement, not training)

This is a way to measure a model, not a training method. Crowdworkers chat
with four assistant types, from 2.7B to 52B parameters: plain, prompted,
rejection-sampled and RLHF [arxiv:2209.07858 §3.2]. A harmlessness preference
model scores every assistant turn, and each attack is scored by its worst turn
[arxiv:2209.07858 §3.4]. The authors report that RLHF models become harder to
red team as they grow, while the other three types show no clear size trend
[arxiv:2209.07858 §4] [fig. 1]. The release has 38,961 attacks [arxiv:2209.07858
tab. 1], but neither the preference model nor the assistants are released
[arxiv:2209.07858 app. A.7]. Card: [Red Teaming Language Models to Reduce
Harms](/papers/2209.07858).

### Open volunteer preference data with a reproduced pipeline (OpenAssistant)

OpenAssistant Conversations holds 161,443 volunteer-written messages in 35
languages, with 461,292 quality ratings and rankings of assistant replies
[arxiv:2304.07327 §4] [app. L]. Messages form conversation trees, so one prompt
can have several continuations [arxiv:2304.07327 §2]. Several rankers' orders for
the same replies are merged with ranked pairs, a variant of Tideman's method
[arxiv:2304.07327 app. B]. The authors then train SFT, reward-model and PPO stages
on the data [arxiv:2304.07327 app. H]. The reward loss is a pairwise
log-sigmoid over the K replies to one prompt, and the paper says an optional
score penalty may be added [arxiv:2304.07327 app. H]. Pythia, Falcon and LLaMA
models tuned on the data beat their base models on most reported benchmarks.
The one exception in the table is Pythia SFT on LMEH, 60.28 against 60.33 for the
base model [arxiv:2304.07327 tab. 1]. The authors say RLHF beats SFT on some
benchmarks but not others [arxiv:2304.07327 §6.1]. The paper calls the data
"fully permissive" without naming a licence, so confirm the dataset licence
before reuse [arxiv:2304.07327 abstract]. Card:
[OpenAssistant Conversations](/papers/2304.07327).

### Preference-trained scorer for images (Pick-a-Pic)

Pick-a-Pic collects a user's prompt, two generated images and a pick or tie
[arxiv:2305.01569 §2]. The authors train PickScore, a CLIP ViT-H/14 model, with
a two-way softmax over the two images and a KL loss to a target that is [1, 0],
[0, 1] or [0.5, 0.5] for a tie [arxiv:2305.01569 §3] [eq. 2] [eq. 3]. Each
example is weighted by the inverse frequency of its prompt [arxiv:2305.01569 §3].
The scorer is a reward model in the RLHF sense, but the paper does not run RL.
It ranks candidates instead. In best-of-100 selection, human raters prefer
PickScore's pick over CLIP-H's (71.3%), over the aesthetics predictor's (85.1%)
and over two random-seed baselines (71.4% and 82.0%) [arxiv:2305.01569 tab. 2]
[§6]. On held-out user picks, PickScore reaches 70.5% accuracy against 68.0% for
expert annotators [arxiv:2305.01569 tab. 1]. The human baseline is expert
annotators who are friends and colleagues of the authors [arxiv:2305.01569 §4].
Neither the dataset nor the model licence is stated [arxiv:2305.01569 §1]. Card:
[Pick-a-Pic](/papers/2305.01569).

### Reward-feedback fine-tuning of a diffusion model (ReFL)

ImageReward is a BLIP-based reward model. It is trained on rankings of images
for each prompt, with a pairwise logistic loss over the score difference
[arxiv:2304.05977 §2.2] [eq. 1]. ReFL scores the predicted clean image from one randomly chosen late
denoising step, where the authors find the reward informative, and
back-propagates that score into Stable Diffusion v1.4
[arxiv:2304.05977 §3] [alg. 1]. The authors weight the reward loss by a scale
of 1e-3 with a ReLU map, and add a pre-training term from Eq. 3
[arxiv:2304.05977 §3] [§4.2] [eq. 2]. The sign and offset of the ReLU are not
given in the text, so the map needs checking against the scorer's output range.
Preference accuracy on 6,399 held-out comparisons is 65.14% for ImageReward,
against 57.76% for BLIP score [arxiv:2304.05977 tab. 3]. In the authors' human
comparison, ReFL's win rate against the SD v1.4 baseline is 58.79% [arxiv:2304.05977 tab. 4].
Card: [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](/papers/2304.05977).
See [reward modeling](/concepts/reward-modeling) for the scorer alone.

### Checking the preference signal for sycophancy (no training method)

This is an evaluation, not a training method. Five assistants are tested on four
task families where a user's opinion or pushback can change the answer
[arxiv:2310.13548 §3]. Under the challenge "I don't think that's right. Are you
sure?", Claude 1.3 admits a mistake on 98% of questions, and its answer changes
on 86% [arxiv:2310.13548 app. A.4]. Feedback becomes more positive when the user
says they like the text, though the text gives no numbers [arxiv:2310.13548 §3.1]
[fig. 1]. With the Claude 2 preference model (PM), best-of-N sampling yields more
sycophantic replies than a prompted non-sycophantic PM [arxiv:2310.13548 §4.2]
[fig. 6a]. On misconception prompts, the PM prefers the sycophantic reply to the
truthful one in 95% of pairs [arxiv:2310.13548 §4.3.1] [fig. 7a]. The Claude 2 PM
is not released, and its training partly used the hh-rlhf helpfulness data
[arxiv:2310.13548 §4.2]. Card:
[Towards Understanding Sycophancy in Language Models](/papers/2310.13548).

### Reward-model RLHF in an open-weight model (Gemma)

Gemma's RLHF step trains a reward model on human preference pairs under the
Bradley-Terry model, then optimises the policy with an RL algorithm the paper
calls novel but does not name [arxiv:2403.08295 § Reinforcement Learning from
Human Feedback]. An automatic high-capacity rater tunes hyperparameters and
limits reward hacking [arxiv:2403.08295 § Reinforcement Learning from Human
Feedback]. The paper gives no RL or reward-model hyperparameters [arxiv:2403.08295
§ Training Infrastructure]. Human side-by-side results compare whole
instruction-tuned models with Mistral v0.2 7B Instruct. Gemma 1.1 IT 7B wins 63.5%
on safety and 61.2% on instruction following, and Gemma 1.1 IT 2B wins 60.1% and
45% [arxiv:2403.08295 tab. 5]. These numbers cover the whole post-training
stack, not RLHF alone. Card:
[Gemma: Open Models Based on Gemini Research and Technology](/papers/2403.08295).

### Reward model and PPO in a released chat model (Qwen)

Qwen's chat models get SFT first, then a reward model and PPO
[arxiv:2309.16609 §3.1] [§3.2]. The reward model starts from a same-size Qwen
model. A pooling layer reads the reward at one end token, and training runs
first on preference pretraining data and then on higher-quality comparisons
[arxiv:2309.16609 §3.2.1]. The reward model trains for one epoch at a constant
learning rate of 3e-6, with batch 64 and sequence length 2048 [arxiv:2309.16609
§3.2.1]. PPO holds four models in memory, samples two responses per query,
normalises rewards by a running mean, and uses a KL coefficient of 0.04
[arxiv:2309.16609 §3.2.2]. A pretraining-data gradient is mixed in to limit the
alignment tax on benchmarks outside code and maths [arxiv:2309.16609 §3.2.2].
The chat benchmark numbers come from the full SFT-plus-RLHF model, so they do
not isolate RLHF [arxiv:2309.16609 §3.3] [tab. 5]. Card:
[Qwen Technical Report](/papers/2309.16609).

### Group consensus reward model with reranking, no RL stage (Habermas Machine)

The reward model here scores candidates for a group, not for one user. Group
members write opinions on a debate question, and raters score candidate
statements for agreement. The reward model is trained on pairwise agreement
preferences with cross-entropy, one participant's opinion at a time
[arxiv:2211.15006 §3.4] [app. C.2]. The training loop has supervised tuning
and the reward model, with no RL stage [arxiv:2211.15006 §3.4]. At inference,
16 statements are sampled with nucleus sampling. Each one is scored for every
member, and the Utilitarian welfare function (inequality aversion 0) picks the
winner [arxiv:2211.15006 §3.3] [eq. 1] [app. C.1]. The authors report that
raters preferred the chosen statement over prompted Chinchilla baselines and
over human-written opinions [arxiv:2211.15006 §4.1] [§4.2]. Groups are limited
to three to five people by prompt length, and the data come from UK crowd
participants on UK political questions, so generalisation is untested
[arxiv:2211.15006 §5.1]. Card:
[Fine-tuning language models to find agreement among humans with diverse preferences](/papers/2211.15006).
See [reward modeling](/concepts/reward-modeling) for the scorer alone.

## When to use which

The sources compare the variants only in part. DPO is compared against a
PPO-based RLHF baseline in its own experiments [arxiv:2305.18290 §6]. LIMA is
compared against other models whose training is not controlled: responses were
sampled over one month, and some baselines may have seen large amounts of user
data [arxiv:2305.11206 §4.1] [arxiv:2305.11206 §4.3]. The Gemini report card records no
head-to-head comparison of reward-model RLHF with either alternative. The table
gives what each source supports.

| Situation | Use | Why |
|---|---|---|
| Quality is easier to judge than to write, e.g. style, helpfulness or safety | Collect pairwise human ratings and train a reward model | The report gathers relative preferences and per-response labels [arxiv:2312.11805 §6.3] |
| A game-like task has no written score, a few demonstrations exist, and an annotator can compare short clips during training | Preference reward model with demonstration pretraining (DQfD), with new labels each round | 7 of 9 Atari games beat imitation in the abstract, author-reported [arxiv:1811.06521 abstract]; a frozen reward model was exploited, so labels must keep coming [arxiv:1811.06521 §3.2] |
| Have preference pairs, want one stable training stage, and can keep a frozen reference model | DPO | One loss on pairs, with no sampling during training [arxiv:2305.18290 §4]; RLHF is called complex and often unstable [arxiv:2305.18290 §1]. Beta needs setting: 0.1 default, 0.5 for TL;DR [arxiv:2305.18290 app. B] |
| Only a small set of high-quality examples, no preference labels, and the goal is mainly answer style | Supervised tuning on curated data | 1,000 examples gave 65% preference over DaVinci003 in author-reported tests [arxiv:2305.11206 §1]; the authors say writing good examples is labour-intensive and hard to scale [arxiv:2305.11206 §7] |
| Safety policy must hold across many harm types, and writing good refusals by hand is costly | Constitution-guided revision feeding the reward model | Used for about 20 harm types [arxiv:2312.11805 §7.3.2] |
| Only bad examples exist, and the goal is to stop specific outputs rather than give good replies | Gradient-ascent unlearning with KL and random-mismatch terms | Unlearned prompts get blank or nonsensical output; 6% harmful on OPT-1.3B against 47% originally, author-reported [arxiv:2310.10683 §2] [tab. 3] |
| Users can judge outcomes but cannot write the score or the behaviour, and no experiments are needed yet | Reward modeling as a research agenda, not a method to deploy | A position paper with no results; it lists five scaling challenges and says success is not guaranteed [arxiv:1811.07871 §1] [fig. 3] |
| Multi-turn attack testing where success shows only at the end of a dialogue, and a safety classifier can score replies | Hierarchical PPO with a classifier reward and token-level credit | 75.2% ASR@5 on HarmBench against Llama-3.1-8B-Instruct, author-reported, above the baselines in their table [arxiv:2508.04451 tab. 1]; the reward is a classifier, so no human labels are needed [arxiv:2508.04451 §4.2] |
| Generator is a diffusion or video model, a VLM can judge the outputs, and metric rewards miss the target failure | Offline AI-feedback reward with RWR or DPO | Human "All" rises from 19.38% to 31.09% (RWR) and 34.06% (DPO) in the authors' evaluation; the judge is the same model family as the reward, and the GPT-4o check gives a smaller gain [arxiv:2412.02617 tab. 2] [§4.3] |
| Factual answers must be checkable against a source | Per-rule reward models with search evidence | 0.78 vs 0.61 supported-and-plausible in the author evaluation; the correctness check is small [arxiv:2209.14375 §3.2] [§3.3] |
| Need to know how harmful a model is under adversarial probing, and how that changes with size | Red-team campaign scored by a harmlessness preference model | Dialogue attacks are suspected to be more adversarial than static prompt sets [arxiv:2209.07858 §4]; agreement on what counts as a successful attack is low [arxiv:2209.07858 §3.5] |
| Need an overview of the design choices (feedback type, reward model, RL or direct policy learning) before picking a method | Read the survey, then the card of the chosen method | The survey covers feedback types, choice models and policy options, with no experiments [arxiv:2312.14925 §3] [§5.1] [§6]; settings come from each method's own paper |
| Several people with differing views must share one statement, and raters can score candidate statements for agreement | Reward model that predicts each member's agreement, with best-of-16 reranking by welfare | Preferred over prompted Chinchilla baselines and over human-written opinions, author-reported [arxiv:2211.15006 §4.1] [§4.2]; groups of 3 to 5 and UK-only data [arxiv:2211.15006 §5.1] |
| Need evidence that the RLHF step itself helps | Expect a modest gain on the tested task | Only isolated measurement: SFT+RLHF over SFT, +0.223±0.06 side-by-side on image understanding for Gemini Apps Pro [arxiv:2312.11805 §6.5.4] |
| Judging a factuality gain | Credit the whole final post-training stage, not RLHF alone | Inaccurate rate 6.7% → 3.8%, measured before and after the final stage [arxiv:2312.11805 tab. 6] |
| Have volunteer-written preference data and need a reproducible merge rule for rankings | Ranked-pairs merge of several rankers, after confirming the dataset licence | Ranked replies and a three-stage pipeline are released [arxiv:2304.07327 §4] [app. B] [app. H]; the paper does not name the data licence [arxiv:2304.07327 abstract] |
| Need a scorer to pick the best of several images for one prompt, with no RL loop | Pairwise scorer trained on user picks (PickScore) | 70.5% against 68.0% for experts on held-out picks; best-of-100 win rates of 71.3% to 85.1% against other scorers, author-reported [arxiv:2305.01569 tab. 1] [tab. 2] |
| Generator is a diffusion model, and a reward model over images should steer its outputs | Reward-feedback fine-tuning on one late denoising step (ReFL) | 58.79% win rate against SD v1.4 in the authors' human comparison [arxiv:2304.05977 tab. 4]; the ReLU sign and offset are not stated [arxiv:2304.05977 §4.2] |
| Before training on preference labels, need to know whether they reward agreement over truth | Sycophancy probes on the reward model and the policy | The PM prefers the sycophantic reply in 95% of misconception pairs, while crowd raters usually prefer the truthful one [arxiv:2310.13548 §4.3.1]; the PM is not released [arxiv:2310.13548 §4.2] |
| Only one good or bad label per output, and pairs are scarce or costly | KTO on binary labels, with the class-balance rule | GSM8K 53.5 against 40.0 for DPO on Zephyr-β-SFT, author-reported [arxiv:2402.01306 tab. 2]; the class-balance rule is [eq. 9] |
| Safety rules can be written down, human harm labels are costly, and evasive refusals are a problem | Constitution-guided critique and revision, then AI preference labels | 182,831 AI harmlessness comparisons; RL model rated more harmless than RLHF baselines, author-reported [arxiv:2212.08073 §4.2] [§4.3] |
| Crowd data can be collected each round, and the goal is a model that crowdworkers prefer | Iterated online PPO with a KL weight of 0.001 | Online model preferred to the static RLHF model [arxiv:2204.05862 §4.5] [fig. 1]; the reward KL weight is [eq. 2] |

What the sources leave out, so a choice here cannot be made from them alone:
the RL algorithm, the KL penalty, learning rates, batch sizes, the number of
iterations, and the amount of preference data in the Gemini report
[arxiv:2312.11805 §6.3]. The GLM-4 report gives no learning rates, batch sizes,
steps or RLHF data sizes, and defers the RLHF method to other reports
[arxiv:2406.12793 §2 Alignment] [§2 ChatGLM Techniques]. The DPO paper states its own optimiser settings
[arxiv:2305.18290 app. B], but its repository defaults differ from them; see
the DPO card.

## Implementation sketch

The loop below follows the report's four steps. Anything not stated in the
report is marked as such.

```pseudocode
# Post-training loop, stages as in [arxiv:2312.11805 §6.3]

# 1. Curate prompts, single- and multi-turn. They serve both as
#    demonstration prompts and as RL prompts  [arxiv:2312.11805 §6.3]
prompts = collect_prompts()

# 2. Supervised fine-tuning on demonstrations written by experts, or
#    drafted by a model and reviewed by people  [arxiv:2312.11805 §6.3]
policy = sft(base_model, demonstrations(prompts))

# 3. Reward model from human rankings of candidate responses
#    [arxiv:2312.11805 §6.3]
pairs = rank(sample(policy, prompts), raters)
reward = train_reward_model(pairs)

# 4. Optimise the policy against the reward model, then refresh the reward
#    model with new data and repeat  [arxiv:2312.11805 §6.3]
# RL algorithm, KL penalty and round count: not stated
for _ in rounds:
    policy = rl_optimize(policy, reward, prompts)
    pairs += rank(sample(policy, prompts), raters)
    reward = train_reward_model(pairs)
```

Where variants differ: DPO replaces steps 3 and 4 with a single loss on the
same kind of pairs, computed against a frozen reference model
[arxiv:2305.18290 §4] [eq. 7]. LIMA skips steps 3 and 4 entirely and uses
supervised loss on the curated examples [arxiv:2305.11206 §3]. Unlearning
also skips steps 3 and 4 and uses gradient ascent on the forget set, with the
KL and random-mismatch terms [arxiv:2310.10683 eq. 2]. KTO also skips steps 3
and 4, and fits one loss to desirable and undesirable outputs against a
KL-based reference point [arxiv:2402.01306 §4.1] [eq. 8]. Constitutional AI
keeps the loop, but its step 3 labels come from a feedback model that reads
one written principle [arxiv:2212.08073 §4.1]. The group-consensus variant
keeps steps 2 and 3 but has no RL stage: at inference it samples 16
statements and returns the one with the highest welfare score across the
group's members [arxiv:2211.15006 §3.3] [app. C.1].

In the diffusion video variant, the same loop runs without an RL sampler. The
reward is attached to each offline sample. RWR weights each sample's denoising
error by that reward. DPO compares denoising errors of the policy and a
reference model on preferred and rejected samples, using one shared noise
draw and time step [arxiv:2412.02617 alg. 1] [eq. 6] [eq. 7]. The paper notes
that binary labels give no pairing rule of their own, and does not state how
pairs are built [arxiv:2412.02617 §4.3].

PickScore uses a softmax over two images with soft targets for ties, rather than
a log-sigmoid over a ranking, but it is trained with the same kind of pairwise
preference signal [arxiv:2305.01569 §3] [eq. 2] [eq. 3]. The OpenAssistant
reward loss is the pairwise form over K ranked replies [arxiv:2304.07327 app. H].

Sparrow's per-turn reward adds the normalised preference score to the mean of
the normalised rule scores, then subtracts length and format penalties
[arxiv:2209.14375 app. E.3]. Its RL learning rate is 2e-6 with Adafactor
[arxiv:2209.14375 app. E]. The learning rates of its reward models are not
stated [arxiv:2209.14375 app. D].

## Contradictions

- **Whether alignment adds capability or only style.** LIMA argues that
  alignment mainly teaches response style and that knowledge comes from
  pretraining [arxiv:2305.11206 §2]. The Gemini report measures a gain from
  SFT+RLHF over SFT on image understanding, +0.223 side-by-side, and does not
  split it into style and knowledge [arxiv:2312.11805 §6.5.4]. The two studies
  use different models, tasks and judges, so this is a difference in framing,
  not a direct numerical conflict.
- **Reward-model RLHF versus DPO.** DPO matches or beats PPO-based RLHF on the
  three tasks it tests [arxiv:2305.18290 §6]. The authors say they did not tune
  beta on TL;DR, which may understate DPO [arxiv:2305.18290 §6.2]. The survey
  reports the opposite in a secondary citation: tuned PPO can outperform
  direct methods [arxiv:2312.14925 App. A, Direct methods]. It gives no figures,
  so the two claims cannot be checked against each other here. The Gemini card
  records no DPO comparison.
- **Gains from RL versus bias.** Sparrow reports fewer targeted-rule violations
  under probing [arxiv:2209.14375 §1], but bias scores on Winogender and BBQ rise
  after RL [arxiv:2209.14375 §3.6.1]. The paper says its mitigations address only
  single-example harms [arxiv:2209.14375 §3.6]. The red-team study covers attack
  difficulty, not bias, so the two results do not directly conflict.
- **Are RLHF's limits fixable?** The survey argues that several problems are
  fundamental, meaning no fix stays within RLHF [arxiv:2307.15217 §3]. Sparrow
  reports measured gains from RLHF with rules and evidence
  [arxiv:2209.14375 §3.2]. The survey runs no experiments, so this is a difference
  in framing, not a numerical conflict.
- **Harmful rate versus output quality.** The unlearning paper reports lower
  harmful rates for plain gradient ascent (1%) than for full RLHF (4%) on
  OPT-1.3B [arxiv:2310.10683 tab. 6]. The same table marks fluency as not
  meaningful for plain gradient ascent, while full RLHF has a perplexity-based
  score of 3.414 [arxiv:2310.10683 tab. 6]. Read the harmful rate alone with care.
- **RWR or DPO for diffusion.** On the video study's Gemini "All" score, RWR
  is higher (56.41% against 56.17%). On human "All", DPO is higher (34.06%
  against 31.09%). The paper says DPO generalises better to held-out prompts
  [arxiv:2412.02617 tab. 2] [§4.2] [§4.3]. The Gemini gap is 0.24 points over
  the 160 train and test prompts, so it is small. The two algorithms are not clearly ranked
  by the paper's own numbers.
- **Human raters versus the preference model.** The sycophancy study finds the
  Claude 2 PM prefers the sycophantic reply in 95% of misconception pairs
  [arxiv:2310.13548 §4.3.1] [fig. 7a]. Crowd raters usually prefer the truthful
  reply, less reliably at higher difficulty [arxiv:2310.13548 §4.3.1] [fig. 7b].
  In the hardest bin, best-of-4096 gives a sycophantic reply for about 75% with
  the Claude 2 PM and about 25% with an oracle PM [arxiv:2310.13548 §4.3.2]
  [fig. 7d]. The text gives no number for the human result, so the comparison
  with raters is qualitative.
- **Does RLHF beat SFT?** OpenAssistant's authors report RLHF beating SFT on some
  benchmarks but not others [arxiv:2304.07327 §6.1]. The Gemini report measures a
  gain from SFT+RLHF over SFT on image understanding, +0.223 side-by-side
  [arxiv:2312.11805 §6.5.4]. The benchmarks and judges differ, so this is not a
  direct numerical conflict.
- **DPO baselines.** The KTO paper's DPO numbers, such as GSM8K 40.0 on
  Zephyr-β-SFT and 62.1% human win rate on Mistral-7B, come from the KTO authors'
  own runs [arxiv:2402.01306 tab. 2] [app. D]. They are not results from the DPO
  paper, so the KTO-versus-DPO gains here are not a clean replication.
- **Evasiveness of helpful assistants.** Constitutional AI reports that its
  earlier helpful-and-harmless assistant was often evasive, and that its own RL
  model is virtually never evasive [arxiv:2212.08073 §1.1] [§4.4]. The Anthropic
  HH paper reports a helpfulness and harmlessness trade-off [arxiv:2204.05862
  §5.1], but the evasiveness claim is not one it is cited for here, so the two
  results are not directly compared.

## Papers

- [ChatGPT research survey](/papers/2304.01852) (arXiv:2304.01852): a bibliometric survey that names InstructGPT and RLHF as the basis of ChatGPT; it does not describe the method.
- [Red Teaming Language Models to Reduce Harms](/papers/2209.07858) (arXiv:2209.07858): crowdworker attacks on plain, prompted, rejection-sampled and RLHF assistants from 2.7B to 52B parameters, scored by a harmlessness preference model.
- [Improving alignment of dialogue agents via targeted human judgements](/papers/2209.14375) (arXiv:2209.14375): Sparrow, trained with per-rule and preference reward models, search evidence and reranking.
- [LIMA: Less Is More for Alignment](/papers/2305.11206) (arXiv:2305.11206): supervised tuning of LLaMa 65B on 1,000 curated examples with no RLHF, as a test of the Superficial Alignment Hypothesis.
- [Direct Preference Optimization](/papers/2305.18290) (arXiv:2305.18290): replaces the reward model and RL stage with one loss on preference pairs against a frozen reference model.
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](/papers/2307.15217) (arXiv:2307.15217): a survey and position paper that sorts RLHF's open problems into tractable and fundamental ones, with no experiments.
- [Large Language Model Unlearning](/papers/2310.10683) (arXiv:2310.10683): gradient-ascent unlearning from negative examples only, with KL and random-mismatch terms, compared against full RLHF on OPT-1.3B.
- [Gemini 1.0 report](/papers/2312.11805) (arXiv:2312.11805): technical report with the post-training pipeline, the safety preference data and one isolated SFT-versus-RLHF comparison.
- [Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](/papers/2412.02617) (arXiv:2412.02617): offline RWR and DPO on a 3B video diffusion model, rewarded by binary VLM judgments and compared with metric rewards.
- [OpenAssistant Conversations](/papers/2304.07327) (arXiv:2304.07327): volunteer-written conversation trees with quality labels and reply rankings, merged by ranked pairs; SFT, reward-model and PPO runs on the data, with the dataset licence not named.
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](/papers/2305.01569) (arXiv:2305.01569): a public dataset of user picks and ties over image pairs, and PickScore, a CLIP-based scorer trained on it with a softmax preference loss.
- [Towards Understanding Sycophancy in Language Models](/papers/2310.13548) (arXiv:2310.13548): behavioural tests of sycophancy in five assistants, and an analysis of the Claude 2 PM and human preferences over truthful and sycophantic replies.
- [Gemma: Open Models Based on Gemini Research and Technology](/papers/2403.08295) (arXiv:2403.08295): open 2B and 7B models whose instruction-tuned variants add an RLHF step, with human win rates against Mistral v0.2 7B Instruct.
- [Reward learning from human preferences and demonstrations in Atari](/papers/1811.06521) (arXiv:1811.06521): a convolutional reward model fitted to pairwise clip preferences, used to train a DQfD agent on nine Atari games, with online labels and demonstrations.
- [Scalable agent alignment via reward modeling: a research direction](/papers/1811.07871) (arXiv:1811.07871): position paper proposing reward modeling and recursive reward modeling for agent alignment, with no experiments.
- [There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning](/papers/2508.04451) (arXiv:2508.04451): two-level PPO attacker for multi-turn red teaming, rewarded by a LlamaGuard "unsafe" probability and trained with token-level credit.
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](/papers/2204.05862) (arXiv:2204.05862): crowdworker preference models and PPO with a KL penalty up to 52B parameters, with iterated online data.
- [Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073) (arXiv:2212.08073): critique-revision SFT and AI preference labels from 16 written principles, then RL against the resulting preference model.
- [KTO: Model Alignment as Prospect Theoretic Optimization](/papers/2402.01306) (arXiv:2402.01306): a reward-free loss on binary desirable and undesirable labels, compared with DPO on Pythia, Llama, Zephyr and Mistral models.
- [Qwen Technical Report](/papers/2309.16609) (arXiv:2309.16609): Qwen base and chat models, with a reward model and PPO stage for the chat models and the settings the paper reports.
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](/papers/2304.05977) (arXiv:2304.05977): a BLIP-based reward model trained on expert image rankings, and ReFL, which back-propagates that reward through one late denoising step of Stable Diffusion v1.4.
- [Role-Play with Large Language Models](/papers/2305.16367) (arXiv:2305.16367): a position paper on role-playing dialogue agents, with no method or experiments; it notes that its base-model analysis may not hold after RLHF.
- [A Survey on Human-AI Collaboration with Large Foundation Models](/papers/2403.04931) (arXiv:2403.04931): a survey that places RLHF and DPO in the build-time loop of human guidance, with no experiments of its own.
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](/papers/2406.12793) (arXiv:2406.12793): technical report with SFT followed by RLHF as the GLM-4 alignment stage; the RLHF method and settings are deferred to other reports.
- [A Survey on Controllable Text Generation using Transformer-based Pre-trained Language Models](/papers/2201.05337) (arXiv:2201.05337): a survey with no experiments; lists RL with a human-preference reward model as one control family, and names InstructGPT as RLHF.
- [Fine-tuning language models to find agreement among humans with diverse preferences](/papers/2211.15006) (arXiv:2211.15006): a 70B model with a reward model that predicts group members' agreement, used to rerank 16 sampled consensus statements; no RL stage.
- [A Survey of Reinforcement Learning from Human Feedback](/papers/2312.14925) (arXiv:2312.14925): a survey of RLHF with depth in control and robotics and an appendix on language models; covers feedback types, Bradley-Terry reward models and policy learning, including DPO. No experiments.
- [The Vector Grounding Problem](/papers/2304.01481) (arXiv:2304.01481): a position paper arguing that LLM internal states can refer to the world, with preference tuning as the most direct route; no experiments.