A Survey of Reinforcement Learning from Human Feedback

In plain words

Reinforcement learning normally needs a reward function that scores every outcome. For many tasks, that function is hard to write down. A system that is rewarded for the wrong thing can find a way to score well without doing what people wanted.

The approach in this survey is to ask people instead. A person compares two behaviours and says which is better. A reward model is fitted to those answers, and a learning algorithm then trains a policy against that model. The reward model can be refined as more answers come in, so the goal can be corrected during training.

The survey covers this idea across several domains, with most depth in control and robotics. It explains the feedback types, how to choose which comparisons to ask for, how the reward model is trained, and how the policy is trained. It also has an appendix on language models, where a newer route skips the reward model.

TL;DR

• A survey of reinforcement learning from human feedback (RLHF), covering control and robotics in depth and language models in an appendix §1 app. A.
• Its core setting: a reward model is fitted to pairwise trajectory comparisons under a Bradley-Terry likelihood, then a policy is trained against it §2.3 eq. 2.
• Policy learning covers adapted RL algorithms and direct methods such as DPO §6 eq. 14.
• It reports no experiments and no new results. The few values it mentions are cited from other papers.

Problem

Reward engineering is hard when success is difficult to define, and sparse rewards push designers toward shaping terms that can be gamed §1.1. Excess optimisation of an imperfect reward can produce reward hacking, where the specified reward rises while the intended goal does not §1.1. Learning from demonstrations has its own limits: rewards are hard to identify from demonstrations, demonstrations are not always available, and a demonstrator is rarely the best possible teacher §1.1. RLHF is presented as a way to define and refine the objective with human feedback during training §1 §1.1. The survey notes that whether it actually solves alignment is debated §1.1.

Method

Setting §2.1 §2.2. The environment is a Markov decision process without a usable reward signal. A human oracle answers queries, and the learner gets labels rather than rewards §2.2. A trajectory return is the discounted sum of per-step rewards eq. 1 §2.1. The survey uses the same symbol R for the return of a trajectory and for the reward function, with different arguments §2.1.

R(\tau) = \sum_{h=0}^{H-1} \gamma^{h} R(s_h, a_h) eq. 1

Reward learning from pairwise comparisons §2.3. The probability that the human prefers trajectory 1 to trajectory 2 follows the Bradley-Terry model, written in terms of the utilities of the two trajectories §2.3, unnumbered display. Maximum likelihood over the dataset gives the objective in eq. 2, which is equivalent to minimising a cross-entropy loss §2.3.

\max_{\psi} \prod_{i=1}^{N} \frac{1}{1 + \exp(R_\psi(\tau^i_2) - R_\psi(\tau^i_1))} eq. 2

Generic RLHF loop §2.4 alg. 1. The basic algorithm repeats two phases: reward learning and RL training §2.4. Reward learning generates queries from a replay buffer, collects the oracle's answers into a dataset, and updates the reward model alg. 1. RL training refreshes the buffer with trajectories from the current policy, then updates the actor and the critic using the learned reward alg. 1. The survey notes that the RL part sees a non-stationary reward, because the reward model is updated during training §2.4.

# Generic RLHF loop, actor-critic variant  [alg. 1] [§2.4]
# Input:  policy theta, critic phi, reward model psi, oracle
# Output: trained policy theta

def generic_rlhf(theta, phi, psi, oracle, iterations):
    # Replay buffer starts with random trajectories  [alg. 1]
    buffer = sample_random_trajectories()
    # Human answers accumulate in this dataset  [alg. 1]
    dataset = []
    for step in range(iterations):
        # Reward learning: build queries from buffer segments  [alg. 1]
        queries = generate_queries(buffer)
        # The oracle answers the queries  [§2.4]
        dataset.extend(oracle.answer(queries))
        # Fit the reward model to all answers so far  [alg. 1] [eq. 2]
        psi = fit_reward_model(psi, dataset)
        # RL training: add trajectories from the current policy  [alg. 1]
        buffer.extend(rollout(theta))
        # Actor and critic learn from the learned reward  [alg. 1]
        theta = update_actor(theta, buffer, psi)
        phi = update_critic(phi, buffer, psi)
    return theta

Feedback types §3 tab. 1. Pairwise trajectory comparisons are the most studied case. The survey places pairwise comparisons, rankings, state and action preferences, binary critiques, scalar feedback, corrections, action advice, implicit feedback and natural language under its RLHF umbrella, and it treats PbRL and SSRL as subsets tab. 1 §1.2.

Choice model §5.1. Almost every feedback type is cast as a choice among a set of candidate labels built from the query §5.1. The most used model is the Boltzmann (Gumbel-perturbed utility) model, which gives a probability proportional to the exponential of the label's utility eq. 8 §5.1.1. For pairs it reduces to Bradley-Terry §5.1.1. A rationality coefficient β scales the utilities, so that larger β makes the labeler more reliable eq. 10 §5.1.2. Other choice models mentioned include Thurstone, ridge-noise and link-function models §5.1.4.

P(l \mid m(q)) \propto \exp(\beta \, u_l) eq. 10

Utility learning §5.2. Learning is framed as empirical risk minimisation over the labels eq. 12 §5.2.1. For pairwise comparisons, the loss becomes the Bradley-Terry negative log-likelihood eq. 12 §5.2.1. A regularisation term is added eq. 13 §5.2.1. The reward model class can be linear §5.2.1 or a deep network §5.2.1. A Bayesian variant places a prior on the reward parameters and updates it with the likelihood of the labels §5.2.2.

Policy learning §6.1. Standard RL methods such as PPO, SAC and DQN can be run on the learned reward §6.1. Value-based methods can be made to work by relabelling the replay buffer whenever the reward model changes §6.1. Over-optimisation of an approximate reward is handled by KL regularisation towards a reference policy, or by constrained RL §6.1.

Bandit framing for generative models §6.2. For language models, the survey views a response as a single action with one sparse reward at the end §6.2 [App. A]. This makes exploration simpler, and best-of-n sampling becomes a policy-improvement step with no training §6.2 [App. A].

Direct policy optimisation §6.3 eq. 14. DPO trains the policy straight from preference triplets. The loss is a logistic loss on the difference of log-ratios of the policy and a frozen reference policy, scaled by β eq. 14 §6.3. The survey lists related variants but does not expand them §6.3.

\mathcal{L}_{DPO} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{ref}(y_l \mid x)} \right) \right] eq. 14

# DPO loss on preference triplets  [§6.3] [eq. 14]
# Input:  batch of (prompt x, preferred y_w, rejected y_l), frozen reference
# Output: scalar loss; beta is not stated in the paper

def dpo_loss(policy, reference, batch, beta):
    # Log-ratio of each output against the frozen reference  [eq. 14]
    prompts, preferred, rejected = batch
    ratio_w = policy.logp(preferred, prompts)
    ratio_w = ratio_w - reference.logp(preferred, prompts)
    ratio_l = policy.logp(rejected, prompts)
    ratio_l = ratio_l - reference.logp(rejected, prompts)
    # Logistic loss on the scaled gap between the two log-ratios  [eq. 14]
    margin = beta * (ratio_w - ratio_l)
    return -mean(log_sigmoid(margin))

Theory §7. Most theoretical results assume a bounded reward and that the true reward or the human model lies in the model class §7. A separate line drops the parametric preference model and seeks a Nash equilibrium between policies, which avoids assuming transitive preferences §7.3.

Hyperparameters

The survey runs no experiments. The table lists the quantities in its equations and the values it states. Anything not in the table is not stated.

Results

No experiments. The paper reports no new results, so there are no headline numbers to compare. It describes benchmarks and datasets others have built, such as B-Pref, RewardBench, MineRL BASALT and TruthfulQA, but reports no results on them §8.3 §8.4.

Paper vs code

No official code.

The survey has no code repository or project link of its own. Its LaTeX source links only third-party libraries and datasets, including imitation, trlX and the Anthropic HH-RLHF data §8.2 §8.4. None of them is written by the authors, so none is cited here. A survey has nothing to reproduce, so no further GitHub search was made.

Pitfalls

• Overloaded notation. R denotes both the return of a trajectory and the reward function, with different arguments §2.1. Check which is meant before implementing.
• Likelihood sign. The product in eq. 2 is maximised. A training loop needs the negative log-likelihood, which is a sum of softplus terms of the form log(1 + exp(R(loser) − R(winner))). Implement the stable form instead of computing the exponential directly.
• Per-step versus bandit reward. Reward learning uses a sum of discounted per-step rewards eq. 1. The bandit view of language models uses one sparse terminal reward §6.2. Those two setups give different credit assignment, and the survey does not choose between them for you.
• Non-stationary reward. The learned reward changes during RL. Value-based methods need the replay buffer relabelled after each reward update §6.1.
• Over-optimisation. A learned reward is only an approximation. Optimising it hard can produce reward hacking §6.1 §9.2. KL regularisation to a reference policy is the main remedy for language models, and the survey notes it prevents catastrophic forgetting [App. A, Regularization].
• DPO is underspecified. The paper does not give β or the reference policy's value. It writes π(y | x) without saying whether token log-probabilities are summed or averaged §6.3 eq. 14. Implementers must choose; the sum is the reading that matches the equation.
• DPO lacks online data. Direct methods often have no online data, which can cause distribution shift. The survey reports that tuned PPO can outperform direct methods, citing Xu et al. 2024 [App. A, Direct methods].
• Transitivity. Parametric choice models such as Bradley-Terry impose transitive preferences, which real humans may violate. The Nash-equilibrium line of theory exists partly for this reason §7.3.
• Rationality is not identifiable. The rationality coefficient cannot be estimated without a calibration reward §5.1.2.
• Identifiability of rewards. Using a trajectory's return as the utility can make the reward non-identifiable. The survey cites a proposal to use regret instead §5.1.3.
• Labeler errors. Humans can prefer confident but wrong answers, and reward models can learn spurious correlations in the feedback §9.2.
• Terminology. The survey defines RLHF broadly, to include PbRL and SSRL. Other papers use the terms more narrowly or interchangeably §1.2 tab. 1.
• Equation numbers. Equation numbers on this card were counted from the LaTeX source, where each numbered align row counts once. Check them against the PDF before quoting them elsewhere. Where in doubt, cite the section.

Lineage

• Builds on: the earlier survey of preference-based RL by Wirth et al. (2017), which this survey explicitly continues §1.3. Its arXiv id is not in this wiki, so it is not linked.
• Compared against: the prior and related surveys in app. B, which cover human-in-the-loop RL and the relation between PbRL and RLHF.
• Superseded by: not known from this wiki's content.