---
kind: concept
slug: reward-modeling
title: Reward modeling
summary: >-
  Learning a score from judgements of outputs or from a sparse return, then using it to train an agent or generator, or to pick among candidates. Covers a user-in-the-loop agenda, an Atari agent, two image scorers, a pairwise LLM ranker, a group-consensus reranker and a learned split of team returns. Results are author-reported.
tags: [alignment, reinforcement-learning, reward-modeling, human-feedback, ranking, credit-assignment, social-choice]
papers: ["1811.07871", "1811.06521", "2304.05977", "2303.14420", "2306.02561", "2412.14779", "2312.14925", "2211.15006"]
related: [rlhf, preference-optimization, inverse-reinforcement-learning, learning-from-demonstration, human-in-the-loop, policy-gradient, bradley-terry-model, learning-to-rank, llm-as-a-judge, text-to-image-generation, diffusion-models, generative-model-evaluation, low-rank-adaptation, credit-assignment, reward-design, multi-agent-reinforcement-learning, distributional-alignment, nucleus-sampling]
status: draft
updated_at: 2026-11-01
---

## Overview

Reinforcement learning needs a reward, and generation needs a way to say which
output is better. When nobody has written that score down, reward modeling
fills the gap with a second model. The model is fitted to judgements about
outputs, and then it scores new outputs. The score can train an agent or a
generator, or it can choose among candidates without asking a judge each time.

A 2023 survey of reinforcement learning from human feedback sets out the common
recipe. It fits a reward model to pairwise trajectory comparisons under a
Bradley-Terry likelihood, then trains a policy against that model. It also
classifies the feedback types and choice models, and reports no experiments
[arxiv:2312.14925 §2.3] [§2.4] [alg. 1].

The seven papers in this set, apart from the survey, use a learned reward in
seven different ways:

- **Original proposal.** A user judges an agent's outcomes, a reward model is
  fitted to those judgements, and an RL agent is trained on the learned reward
  [arxiv:1811.07871 §1] [§3.1]. The paper is a research agenda and reports no
  experiments.
- **Preference-trained reward for an Atari agent.** The reward model is fitted
  to pairwise preferences over short clips of play, and a DQfD agent learns from
  its output. Expert demonstrations seed the agent [arxiv:1811.06521 §2.2]
  [alg. 1].
- **Group-consensus reranker for LLM statements.** A reward model predicts how
  much each member of a small group would agree with a candidate statement,
  given that member's written opinion. A welfare function combines those
  predictions, and the best of 16 sampled statements is shown. The reward model
  does not train the generator [arxiv:2211.15006 §3.3] [§3.4].
- **Scalar scorer for images.** ImageReward is trained on expert rankings of
  generated images. Its single score drives both best-of-N selection and the
  fine-tuning of a diffusion model [arxiv:2304.05977 §2.2] [§3].
- **Scorer for choosing a generator's training images.** HPS is a CLIP model
  fine-tuned on human choices among images from one prompt. Its score picks the
  images used to train a LoRA adapter, rather than being back-propagated
  through the generator [arxiv:2303.14420 §5] [§6].
- **Pairwise ranker for LLM outputs.** PairRanker reads an instruction with two
  candidate answers and says which is better. Its labels come from a text
  metric, not from people. It ranks N candidates from several LLMs [arxiv:2306.02561
  §3.2] [§3.3].
- **Learned split of a sparse team return.** Nobody judges outcomes here. A
  network is fitted so that its per-step outputs add up to the episodic return,
  and the return is split over time and agents. Single-agent RL learners then
  train on the split reward [arxiv:2412.14779 §4.1] [§4.2].

The Atari reward model, ImageReward and HPS each output one scalar per input.
The PairRanker outputs only pairwise margins. The team-return model outputs one
value per agent and time step [arxiv:2412.14779 §4.2]. The consensus reward
model outputs one scalar for each opinion and candidate pair [arxiv:2211.15006
§3.4]. Those output shapes, and what the model is fitted to, decide which
variant fits a task. The original proposal does not fix the form of its reward
model [arxiv:1811.07871 §3].

The original proposal rests on two assumptions. The first is that user
intentions can be learned to high enough accuracy. The second is that, for many
tasks, judging an outcome is easier than producing the right behaviour
[arxiv:1811.07871 §1]. The paper does not define "high enough", and the second
assumption fails when outcomes have few possible values, such as yes or no
[arxiv:1811.07871 §1].

The paper names five challenges that could stop reward modeling from scaling:
how much feedback is needed, how that feedback is distributed, reward hacking,
unacceptable outcomes, and the gap between reward and the result the user wants
[arxiv:1811.07871 §5] [fig. 3]. It then lists ten approaches that might address
them, and it describes a recursive variant for harder tasks [arxiv:1811.07871
§3.2]. Success is not guaranteed [arxiv:1811.07871 §7].

For the same loop applied to language models with preference data, see the
[RLHF](/concepts/rlhf) page.

## Variants

### Online reward modeling with the user in the loop

The agent acts, the user judges the resulting trajectories, the reward model is
refitted on that feedback, and the agent is trained on the reward model's
predictions [arxiv:1811.07871 §3.1]. The user gives no per-action labels. The
RL algorithm does the credit assignment that would otherwise fall to the user
[arxiv:1811.07871 §3.1]. The paper lists four advantages: the user need not
judge every interaction, the policy and the reward model can be checked
separately, a stronger RL algorithm can be swapped in, and the user need not
solve credit assignment [arxiv:1811.07871 §3.1].

The reward model is kept online, meaning it keeps learning while the agent
trains. The paper cites preliminary experiments in other work where agents
learned to exploit a reward model that was frozen after training. Keeping the
model online also lets it follow the states the agent visits and lets the user
catch attempts to game it [arxiv:1811.07871 §5.1]. Card: [Scalable agent
alignment via reward modeling](/papers/1811.07871).

### Preference-trained reward for an Atari agent

The agent starts from a few expert game episodes, which it copies by behaviour
cloning [arxiv:1811.06521 §2.2] [app. A.4]. It then plays, and short 25-step
clips of its play are paired and labelled by an annotator [§2.4] [app. A.3].
The reward model is a convolutional network that reads one stacked observation
and outputs one scalar. It is fitted to the labels with a cross-entropy loss
over the summed rewards of each clip pair, which is in the Bradley-Terry style
[arxiv:1811.06521 §2.4]. The policy (DQfD) reads that reward when it samples
replay batches, so transitions store no reward [§2.3] [app. A.3].

The loop repeats: the policy trains on the reward model, new pairs are labelled,
and the reward model is refitted [alg. 1]. The main schedule uses 500
iterations and 6,800 labels in total [app. A.4] [§3]. Labels can be synthetic,
taken from the game score that the agent never sees, or they can come from
people [§3] [§3.1]. The paper's "autolabels" pair each agent clip with a
demonstration clip and count as extra labels [§2.5].

The paper reports that a reward model frozen during training is exploited: with
it fixed, returns fell in three games as the agent found loopholes [§3.2]
[fig. 4]. The authors take this as support for keeping preference feedback
coming during training. They measured it only on Atari, with synthetic labels
and a few human runs [§3.2] [§3.1]. Card: [Reward learning from human
preferences and demonstrations in Atari](/papers/1811.06521).

### Recursive reward modeling

For outcomes too complex for the user to judge, an earlier agent helps. Agent
A_1 is trained by plain reward modeling. Agent A_k is trained while A_{k-1}
assists the user in evaluating outcomes [arxiv:1811.07871 §3.2] [fig. 2]. The
assistance could summarise data, interpret the agent's internals, or solve
sub-problems the user has split off [arxiv:1811.07871 §3.2]. The helper's task
differs from the task the new agent is trained to solve [arxiv:1811.07871
§3.2].

The user has to split the evaluation into sub-evaluations that together cover
the whole outcome. An aspect left out of the split can be optimised in an
arbitrary direction. The paper suggests one agent proposes a split and another
critiques it by naming what was left out [arxiv:1811.07871 §3.2]. The paper
links the recursion depth to a complexity-theory picture: reward modeling
matches one existential question, and recursion depth k matches k alternating
quantifiers [arxiv:1811.07871 §3.2]. It describes the scheme as an instance of
iterated amplification [arxiv:1811.07871 §3.2]. Whether errors of A_{k-1} grow
in A_k is left open, and the paper proposes ensembles or self-correction as
untested options [arxiv:1811.07871 §3.2]. Card: [Scalable agent alignment via
reward modeling](/papers/1811.07871).

### Preference-trained scalar scorer for a generator

ImageReward is a scalar scorer for text-to-image outputs. Annotators place all
images for a prompt into five preference slots, best to worst. Each ranked list
gives pairs, and the scorer is trained with a pairwise logistic loss on the
difference of the two scores [arxiv:2304.05977 §2.1] [§2.2] [eq. 1]. That loss
has the Bradley-Terry form (see [Bradley-Terry model](/concepts/bradley-terry-model)).
The backbone is BLIP, with a small MLP on top of the text encoder's output
[arxiv:2304.05977 §2.2]. The annotation took about two months and needed a
vendor, a written guideline and quality inspectors [arxiv:2304.05977 §2.1].

The scorer has two uses in the paper:

- **Selection and evaluation.** On 6,399 held-out human comparisons, the scorer
  reaches 65.14% preference accuracy, against 57.76% for BLIP score and 54.82%
  for CLIP score [arxiv:2304.05977 tab. 3]. As a best-of-64 selector on the
  paper's human test, it is slightly ahead of PickScore and HPS
  [arxiv:2304.05977 tab. 6]. The margins there are small, and the test is the
  authors' own.
- **Fine-tuning a generator (ReFL).** Instead of scoring finished images only,
  ReFL draws one random late step out of 40, denoises up to it, predicts the
  clean image at that step, decodes it and back-propagates the reward
  [arxiv:2304.05977 §3] [alg. 1]. The authors report that back-propagating only
  through the last step was unstable [arxiv:2304.05977 §3]. In human comparisons
  with Stable Diffusion v1.4, ReFL wins 58.79% of comparisons on real user
  prompts [arxiv:2304.05977 tab. 4].

The paper's objective also adds a pre-training regulariser, whose weight is not
stated [arxiv:2304.05977 §3] [eq. 3]. The released code does not compute that
term [code: ImageReward/ReFL.py#L710-L768]. The paper gives the stopping step
as 30 to 40 in §3 and 1 to 10 in §4.2, and the code draws 30 to 39
[code: ImageReward/ReFL.py#L714]. Treat the paper and the code as two different
recipes, and check the stopping range before you rely on it.

Card: [ImageReward](/papers/2304.05977).

### Scorer for choosing a generator's training images

Human Preference Score (HPS) is a second scalar scorer for text-to-image
outputs, and it is used in a different way. CLIP ViT-L/14 is fine-tuned so that
a prompt's text embedding lands close to the embedding of the image people
preferred. The score is 100 times the cosine similarity between the two
embeddings [arxiv:2303.14420 §5]. The paper gives the training objective in
words only, and the exact loss is not stated [§5]. The human choices come from
users of a public Discord channel [§3].

HPS does not enter a reward loss. For each prompt, the images are ranked by
score, and the top and bottom images are kept if their softmax weight clears
α/n, with α = 2.0 [§6]. A rank-32 LoRA adapter for Stable Diffusion v1.4 is then
trained on the kept images, with the non-preferred captions prefixed by "Weird
image." At inference that phrase is the negative prompt [§6]. The scorer
chooses the training data, and the adapter learns from that data.

Author-reported results. On 5,000 user choices, HPS reaches 43.5% accuracy,
against 32.9% for CLIP ViT-L/14 and 42.0% for human participants [arxiv:2303.14420
tab. 2]. The adapted model moves FID from 19.72 to 19.35 and the HPS value from
0.1898 to 0.1916 [tab. 4]. In a study with 20 participants and 100 prompts, 74%
of the adapted model's images got more than ten positive votes, against 22% for
the original [§7.2] [fig. 8]. The paper gives no variance for the table
figures and no significance test [tab. 4].

The paper's dataset is distributed under a non-commercial licence [arxiv:2303.14420
app. datasheet]. The paper text is CC BY 4.0, but that does not cover the
dataset, so reusing the data needs its own permission. Code caveats, such as a
LoRA rank default that differs from the paper, are recorded on the
[HPS card](/papers/2303.14420).

Card: [Human Preference Score](/papers/2303.14420).

### Pairwise ranker over candidate outputs

PairRanker is not a scalar scorer. It reads the instruction and two candidates
together, and gives the confidence that the first is better [arxiv:2306.02561
§3.3]. The paper argues that scoring each candidate on its own struggles when
all candidates are fluent and differ only subtly [arxiv:2306.02561 §1] [§3.1].
Over N candidates, it fills an N by N matrix. The MaxLogits rule then turns that
matrix into a ranking, and the paper reports it as the best of the three
aggregators it tried [arxiv:2306.02561 §3.3].

Its training labels come from BARTScore, a text metric, computed against a
reference answer. The paper uses BARTScore because its rankings correlate best
with GPT-Rank among the metrics it compares [arxiv:2306.02561 §5.1] [tab. 3].
Training samples 5 pairs per input [arxiv:2306.02561 §3.3]. No human labels are
used.

Results are author-reported, on the MixInstruct benchmark of 11 open LLMs. The
PairRanker alone reaches a GPT-Rank of 3.20, against 3.90 for the best single
model, OpenAssistant. In that table, lower GPT-Rank is better [arxiv:2306.02561
tab. 2]. The GPT-Rank judge is ChatGPT, and the paper does not state the version
[arxiv:2306.02561 §5.1]. Fusing the top three candidates with a seq2seq model
(GenFuser, Flan-T5-XL) gives GPT-Rank 3.01 [arxiv:2306.02561 tab. 2] [§4]. On
conventional tasks the gains over the base model are smaller: about 2 to 6%
on CNN/DM, CommonGen and WMT18 with the MaxLogits rule [arxiv:2306.02561 app.
C.4].

Cost depends on the aggregator. Full comparison needs N(N-1) forward passes per
input, which is 110 for N = 11 [arxiv:2306.02561 §3.3]. The released bubble
mode runs both orders at each step, so it takes 2(N-1) passes
[code: llm_blender/pair_ranker/ranker.py#L632-L716]. Card: [LLM-Blender](/papers/2306.02561).

### Group-consensus reranker over sampled statements

The reward model here scores a statement for one person, and the score is then
aggregated over a group. Training uses a question, one participant's opinion,
and two candidate statements. The pair is kept only if the two agreement
ratings differ by at least two Likert points, and the loss is cross-entropy
[arxiv:2211.15006 §3.4] [app. C.2]. The reward head is a linear layer on a
pretrained 70B model, and the paper's own description of the loss is brief
[app. C.2].

At inference, a supervised fine-tuned model samples 16 statements. The reward
model scores each statement for every opinion in the group, and a welfare
function ranks the statements. The main results use the Utilitarian case, a
plain mean over people [arxiv:2211.15006 §3.3] [eq. 1]. The reranker decides
which statement is shown, but it is not a training signal for the generator.
The supervised step trains only on quality ratings, so that stances stay diverse
[arxiv:2211.15006 §3.4] [app. C.1.2].

Author-reported results. Raters preferred the reranked statement over the mean
human opinion in 78% of comparisons, and over the best human opinion in 65%
[arxiv:2211.15006 §4.2]. The paper says that the choice of welfare function
made no reliable difference in mean or minimum agreement [§5.1] [app. D.5], so
the gain comes from the whole pipeline. The raters were UK-based crowd
participants on UK political questions, and the authors note that the
population may inflate agreement [§5.1]. The paper's own limit on group size is
prompt length: groups of three to five people [§5.1]. The authors' public
repository does not include this reward model's training code; see the card's
Paper vs code section. Card: [Fine-tuning language models to find agreement
among humans with diverse preferences](/papers/2211.15006).

### Direct policy optimisation, without a reward model

Direct methods skip the reward model. The policy is trained straight from
preference triplets, each a prompt with a preferred and a dispreferred response.
DPO is the example the survey gives. Its loss is a logistic loss on the
difference of log-ratios between the policy and a frozen reference policy,
scaled by β [arxiv:2312.14925 §6.3] [eq. 14]. The survey does not give β or
the reference policy's value [§6.3]. Because the reward is never fitted, the
policy can only be as good as the preference data it is trained on, and the
survey notes that direct methods often have no online data, which can cause
distribution shift [App. A, Direct methods]. Card: [A Survey of Reinforcement
Learning from Human Feedback](/papers/2312.14925). For the training objectives
themselves, see the [preference optimization](/concepts/preference-optimization)
page.

### Learned split of a sparse team return

No judgements are collected. The model is fitted so that its per-step outputs
add up to the episodic return of a finished game [arxiv:2412.14779 §4.1]. The
return is split over time with non-negative weights that sum to 1, then over
agents with a second set of such weights [§4.2]. The architecture extends
AREL's attention reward model with alternating temporal and agent attention
blocks [§4.5]. Single-agent policy-gradient learners, IPPO and MAPPO, train on
the split reward in place of the sparse one [§5.1]. The paper states the sum
constraint but no loss for fitting the model [§4.1].

The paper argues that this split is potential-based reward shaping, so the
optimal policy is kept [§4.3]. The proof assumes a discount of 1 [§4.3], while
the paper's own setting uses a discount [§3.1]. Its potential also depends on
the whole episode, whereas standard potential-based shaping takes a function of
the state [§3.3] [§4.3]. The evidence is one scenario, 5m_vs_6m in SMACLite,
shown as a figure without numbers [§5.2] [fig. 1]. The paper says the split
learns faster and beats other redistribution baselines there [§5.1] [§6]. Card: [TAR² (Agent-Temporal Credit Assignment)](/papers/2412.14779).

## When to use which

The paper's options for the first variant are arguments, not measured results,
so each row there states the condition and the paper's suggested response. The
rows for the other variants cite their reported numbers.

| Situation | Use | Why |
|---|---|---|
| The user can judge finished outcomes but cannot write a reward or describe how to reach it | Online reward modeling | The user judges outcomes and the RL algorithm does credit assignment [arxiv:1811.07871 §1] [§3.1] |
| The outcome is too complex for the user to judge directly, and the task splits into sub-evaluations that cover the whole outcome | Recursive reward modeling, with an earlier agent as helper | The helper assists evaluation; a missing sub-evaluation can be exploited [arxiv:1811.07871 §3.2] |
| The outcome has only a few values, such as yes or no | Not settled by this paper | The paper says its second assumption fails here [arxiv:1811.07871 §1] |
| The agent could reach states the reward model never saw, or harmful outcomes must be prevented before they happen | Off-policy feedback, with model-based planning if possible | The reward model has to be accurate off-policy [arxiv:1811.07871 §5.2] [§5.6] |
| The agent exploits the reward model | Keep the reward model online, then add side-constraints or adversarial search; for language models, KL regularisation towards a reference policy | Frozen reward models are exploited [arxiv:1811.07871 §5.1]; the Atari paper saw the same in three games [arxiv:1811.06521 §3.2]; blocking actions beats penalising them [arxiv:1811.07871 §5.7]; the survey names KL regularisation as the main remedy for language models [arxiv:2312.14925 §6.1] [App. A, Regularization] |
| You can score the agent with a game's own score, and want to test preference-based reward against it, not deploy it | Preference-trained reward with synthetic labels from the game score | Reported gains over imitation in 7 of 9 Atari games; the labels are not human judgements [arxiv:1811.06521 abstract] [§4] |
| The preference labels come from people who will make mistakes | Keep labelling online, and budget for label noise | The paper reports 5 to 10% mislabelling in human labels and says it hurts one game [arxiv:1811.06521 §3.1] [app. E] |
| Feedback is scarce or expensive | Existing data, hierarchical or natural-language feedback, and uncertainty to choose which states to label | The paper's cost example is about $50 million for 500,000 hours [arxiv:1811.07871 §5.3]; uncertainty can drive active learning [§5.9] |
| You need to tune a text-to-image generator toward human preference, and you can afford ranked human comparisons | Scalar scorer trained on ranked pairs, then best-of-N or ReFL | The scorer beats BLIP and CLIP on held-out human pairs [arxiv:2304.05977 tab. 3]; the annotation took two months [§2.1] |
| You only need to pick the best of N images, with no training | An off-the-shelf scorer, checked on your own prompts | The margins over other scorers in the paper's test are small [arxiv:2304.05977 tab. 6] |
| You want to shift a text-to-image model toward preferred images, and you prefer not to train through a scorer | Score each prompt's images, keep the top and bottom ones above a softmax threshold, then train a LoRA with a negative-prompt marker | Small changes in FID and HPS [arxiv:2303.14420 tab. 4]; 74% against 22% of images with over ten positive votes, in a 20-person study [§7.2] |
| You need to reuse the preference dataset commercially | Do not reuse it without its own licence | The paper's datasheet gives the dataset a non-commercial licence [arxiv:2303.14420 app. datasheet] |
| You have several candidate outputs per input, from different models, and a reference answer at training time | Pairwise ranker with metric labels, optionally followed by fusion | Reported gain over the best single model on MixInstruct [arxiv:2306.02561 tab. 2] |
| You have a reward model for a language model and want better outputs without any RL training | Best-of-n sampling against the reward model | The survey treats best-of-n as a policy-improvement step that needs no training [arxiv:2312.14925 §6.2] [App. A] |
| You want to tune a language model on preference pairs and do not want to fit a separate reward model | Direct method such as DPO | The survey gives the loss without a reward model, but leaves β and the reference policy unstated, and reports that tuned PPO can beat direct methods, citing Xu et al. 2024 [arxiv:2312.14925 §6.3] [App. A, Direct methods] |
| You need one statement that a small group can accept, and you can collect each member's opinion and ratings of candidate statements | Group-consensus reranker: a per-person agreement model, aggregated by a welfare function over N samples | Win rate of 78% against the mean human opinion, author-reported on UK crowd groups [arxiv:2211.15006 §4.2]; the welfare choice did not matter reliably [§5.1] |
| You need one scalar reward to drive RL | Scalar scorer, not a pairwise ranker | The PairRanker outputs pairwise margins, and the paper does not use it as a training reward [arxiv:2306.02561 §3.3] |
| Several agents share one delayed team return, and you want to keep a single-agent learner such as IPPO or MAPPO | Learned split of the episodic return | The paper reports the split helps learning, but only as a figure on one scenario [arxiv:2412.14779 §5.1] [fig. 1] |
| You need a guarantee that the optimal policy is kept | Check the proof's assumptions before relying on it | The equivalence is proved with a discount of 1 [arxiv:2412.14779 §4.3], and the paper's own setting uses a discount [§3.1] |

## Implementation sketch

The loops below are one reading of the papers' setups. The first two sketch
the online variant and its recursive form. The third is the pairwise loss used
by the ImageReward-style scorer, the fourth is the per-prompt selection rule
that HPS uses to choose training images, the fifth is pairwise ranking over
candidates, the sixth is the direct DPO loss, which has no reward model, and the
seventh is the group reranker. The
Atari paper's own training protocol is closest to the first loop, with the
policy and reward model updated in turns. The papers give no shared algorithm
box, so the names are ours.

```pseudocode
# One reward-modeling iteration, user in the loop  [arxiv:1811.07871 §3.1]
# Input:  env, agent (policy), reward_model, user
# Output: updated agent and reward_model
def reward_modeling_iteration(env, agent, reward_model, user):
    # The agent acts and produces trajectories  [§3.1]
    trajectories = agent.rollout(env)
    # The user judges outcomes; no per-action labels  [§3.1]
    feedback = user.evaluate(trajectories)
    # Refit the reward model, kept online  [§3.1] [§5.1]
    reward_model.update(trajectories, feedback)
    # Score the trajectories with the learned reward and train  [§3.1]
    rewards = reward_model.predict(trajectories)
    agent.update(trajectories, rewards)
    return agent, reward_model
```

```pseudocode
# Recursive variant: each level is judged with the previous agent's help  [§3.2]
# Input:  tasks ordered from simplest to hardest
# Output: one agent per level
def recursive_reward_modeling(tasks, user):
    agents = []
    helper = None
    for task in tasks:
        # Level 1 has no helper; later levels get the previous agent  [§3.2]
        agent = train_with_reward_modeling(task, user, assistant=helper)
        agents.append(agent)
        helper = agent
    return agents
```

```pseudocode
# Pairwise loss for a scalar scorer on ranked images  [arxiv:2304.05977 §2.2] [eq. 1]
# Input:  prompts (B,), preferred and dispreferred images
# Output: scalar loss
def pairwise_reward_loss(prompts, better, worse):
    # Score the preferred and the dispreferred image for each prompt  [§2.2]
    score_better = reward(prompts, better)  # (B,)
    score_worse = reward(prompts, worse)  # (B,)
    # Logistic loss on the score gap  [eq. 1]
    gap = score_better - score_worse
    return -mean(log_sigmoid(gap))
```

```pseudocode
# Keep the best and worst image per prompt by score  [arxiv:2303.14420 §6]
# Input:  groups: prompt -> list of (image, score); alpha = 2.0  [§6]
# Output: preferred and non-preferred (image, prompt) pairs
def select_by_score(groups, alpha):
    preferred, non_preferred = [], []
    for prompt, items in groups.items():
        scores = [score for _, score in items]
        n = len(scores)
        # Softmax weight of the top image in its group  [§6]
        best = argmax(scores)
        if softmax(scores)[best] > alpha / n:
            preferred.append((items[best][0], prompt))
        # Same rule on negated scores, for the lowest-scoring image  [§6]
        worst = argmin(scores)
        if softmax([-s for s in scores])[worst] > alpha / n:
            non_preferred.append((items[worst][0], prompt))
    return preferred, non_preferred
```

```pseudocode
# Full comparison matrix and MaxLogits ranking  [arxiv:2306.02561 §3.3]
# Input:  source text; list of N candidates
# Output: candidate indices, best first
def rank_candidates(source, candidates):
    n = len(candidates)
    matrix = zeros(n, n)
    # Fill M[i][j] for every ordered pair with i != j  [§3.3]
    for i in range(n):
        for j in range(n):
            if i != j:
                matrix[i][j] = pair_logit(source, candidates[i], candidates[j])
    # MaxLogits: total margin of each candidate over all others  [§3.3]
    scores = sum(matrix - transpose(matrix), axis=1)
    return argsort(scores, descending=True)
```

```pseudocode
# DPO loss on preference pairs, no reward model  [arxiv:2312.14925 §6.3] [eq. 14]
# Input:  batch of (prompt, preferred, dispreferred), frozen reference policy
# Output: scalar loss; beta is not stated in the survey
def dpo_loss(policy, reference, prompts, preferred, dispreferred, beta):
    # Log-ratio of each response against the frozen reference  [eq. 14]
    ratio_win = policy.logp(preferred, prompts) - reference.logp(preferred, prompts)
    ratio_lose = policy.logp(dispreferred, prompts) - reference.logp(dispreferred, prompts)
    # Logistic loss on the scaled gap between the two log-ratios  [eq. 14]
    return -mean(log_sigmoid(beta * (ratio_win - ratio_lose)))
```

```pseudocode
# Group reranking by Utilitarian welfare, alpha = 0  [arxiv:2211.15006 §3.3] [eq. 1]
# Input:  question; 3 to 5 opinions; candidate statements from the SFT model
# Output: the candidate with the highest mean predicted agreement
def consensus_rerank(question, opinions, candidates):
    # Predicted agreement of each member with each candidate  [§3.3]
    agreement = [[reward(question, op, c) for c in candidates] for op in opinions]
    # Welfare with alpha = 0 is the mean over people  [eq. 1]
    welfare = [mean(column) for column in zip(*agreement)]
    return candidates[argmax(welfare)]
```

The variants differ in where they put the reward. The online variant trains the
agent on the reward model's output [§3.1]. The Atari variant does the same, and
its policy reads the reward at sampling time [arxiv:1811.06521 §2.3]. Recursive
reward modeling passes an
assistant into the evaluation step [§3.2]. The scalar scorer trains on pairs of
human rankings and can also back-propagate its score into a generator [arxiv:2304.05977
§3]. HPS uses its score only to choose training images, so the score is not a
training signal for the adapter [arxiv:2303.14420 §6]. The pairwise ranker never produces a scalar, so its output can only order
candidates [arxiv:2306.02561 §3.3]. The DPO loss has no reward function at all:
the policy's log-ratio to the reference plays that role [arxiv:2312.14925 §6.3].
The team-return model gives one value per
agent and step, and those values sum to the episodic return rather than to any
judgement [arxiv:2412.14779 §4.2]. The group reranker is the only variant that
scores each candidate per person and then aggregates across people, and its
reward is used only to rank samples at inference [arxiv:2211.15006 §3.3]. The reward model's architecture, loss and
update schedule in the first variant are not specified in the paper, so `update`
and `predict` are left to the implementer [arxiv:1811.07871 §3].

## Contradictions

- **Frozen versus online scorer.** The first paper says that a reward model
  frozen after training can be exploited by an agent, citing preliminary results
  in other work, and it recommends keeping the model online [arxiv:1811.07871
  §5.1]. ImageReward uses a frozen scorer while it fine-tunes the generator, and
  reports human gains over the baseline [arxiv:2304.05977 §3] [tab. 4]. The
  ImageReward paper, as compiled here, reports no check for exploitation of its
  frozen scorer. The Atari paper does report one, for a frozen reward model in
  three games [arxiv:1811.06521 §3.2]. HPS uses its score to select data, and
  its paper reports no test for exploitation [arxiv:2303.14420 §6]. The papers
  do not test the same setup, and this page does not resolve the disagreement.
- **Source of the labels.** The first paper treats the user's own judgement as
  the training signal [arxiv:1811.07871 §3.1]. The Atari paper trains on game
  score labels for its main results, and on human labels in a separate set of
  runs with reported mislabelling [arxiv:1811.06521 §3] [§3.1]. ImageReward
  trains on expert human rankings and argues that automatic metrics miss human
  preference [arxiv:2304.05977 §2.3]. HPS trains on choices made by users of a
  public Discord channel [arxiv:2303.14420 §3]. PairRanker learns from BARTScore,
  and its agreement is measured against GPT-Rank, a ChatGPT judgement whose
  version is not stated [arxiv:2306.02561 §5.1] [app. E]. The sources measure
  different things, so their numbers are not directly comparable.
- **Accuracy of CLIP as a judge.** ImageReward reports CLIP score at 54.82% and
  its own scorer at 65.14%, on 6,399 held-out human comparisons [arxiv:2304.05977
  tab. 3]. HPS reports CLIP ViT-L/14 at 32.9% and its own classifier at 43.5%, on
  5,000 user choices [arxiv:2303.14420 tab. 2]. The HPS table's random baseline
  is 26.1%, not the 50% of a pairwise test, so the two tests probably use
  different choice sets. This page cannot tell which setup is closer to a real
  use, and it does not resolve the gap.

## Papers

- [Scalable agent alignment via reward modeling: a research direction](/papers/1811.07871): sets out reward modeling as a research direction, with five scaling challenges, ten approaches and recursive reward modeling. No experiments.
- [Reward learning from human preferences and demonstrations in Atari](/papers/1811.06521): a reward model fitted to pairwise clip preferences, and a DQfD agent trained on it, with expert demonstrations. Author-reported, nine games.
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](/papers/2304.05977): a scalar preference scorer trained on ranked human comparisons, and ReFL, which fine-tunes Stable Diffusion v1.4 through it.
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](/papers/2303.14420): a CLIP-based scorer trained on user choices, used to select training images for a LoRA adapter of Stable Diffusion v1.4 with a negative-prompt marker.
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](/papers/2306.02561): a pairwise ranker (PairRanker) over candidate LLM outputs, and a fuser (GenFuser) that writes one answer from the top three.
- [TAR² (Agent-Temporal Credit Assignment)](/papers/2412.14779): a learned split of a sparse team return into per-step, per-agent rewards for single-agent learners. One scenario, shown as a figure.
- [A Survey of Reinforcement Learning from Human Feedback](/papers/2312.14925): a survey of RLHF with the Bradley-Terry reward model, choice models, policy learning and direct methods such as DPO. No experiments.
- [Fine-tuning language models to find agreement among humans with diverse preferences](/papers/2211.15006): a per-person agreement reward model that reranks 16 sampled statements by Utilitarian welfare for groups of three to five. Author-reported, UK crowd participants.