Reward modeling

Overview

Reinforcement learning needs a reward, and generation needs a way to say which output is better. When nobody has written that score down, reward modeling fills the gap with a second model. The model is fitted to judgements about outputs, and then it scores new outputs. The score can train an agent or a generator, or it can choose among candidates without asking a judge each time.

A 2023 survey of reinforcement learning from human feedback sets out the common recipe. It fits a reward model to pairwise trajectory comparisons under a Bradley-Terry likelihood, then trains a policy against that model. It also classifies the feedback types and choice models, and reports no experiments 2312.14925 §2.3 §2.4 alg. 1.

The seven papers in this set, apart from the survey, use a learned reward in seven different ways:

• Original proposal. A user judges an agent's outcomes, a reward model is fitted to those judgements, and an RL agent is trained on the learned reward 1811.07871 §1 §3.1. The paper is a research agenda and reports no experiments.
• Preference-trained reward for an Atari agent. The reward model is fitted to pairwise preferences over short clips of play, and a DQfD agent learns from its output. Expert demonstrations seed the agent 1811.06521 §2.2 alg. 1.
• Group-consensus reranker for LLM statements. A reward model predicts how much each member of a small group would agree with a candidate statement, given that member's written opinion. A welfare function combines those predictions, and the best of 16 sampled statements is shown. The reward model does not train the generator 2211.15006 §3.3 §3.4.
• Scalar scorer for images. ImageReward is trained on expert rankings of generated images. Its single score drives both best-of-N selection and the fine-tuning of a diffusion model 2304.05977 §2.2 §3.
• Scorer for choosing a generator's training images. HPS is a CLIP model fine-tuned on human choices among images from one prompt. Its score picks the images used to train a LoRA adapter, rather than being back-propagated through the generator 2303.14420 §5 §6.
• Pairwise ranker for LLM outputs. PairRanker reads an instruction with two candidate answers and says which is better. Its labels come from a text metric, not from people. It ranks N candidates from several LLMs [arxiv:2306.02561 §3.2] §3.3.
• Learned split of a sparse team return. Nobody judges outcomes here. A network is fitted so that its per-step outputs add up to the episodic return, and the return is split over time and agents. Single-agent RL learners then train on the split reward 2412.14779 §4.1 §4.2.

The Atari reward model, ImageReward and HPS each output one scalar per input. The PairRanker outputs only pairwise margins. The team-return model outputs one value per agent and time step 2412.14779 §4.2. The consensus reward model outputs one scalar for each opinion and candidate pair [arxiv:2211.15006 §3.4]. Those output shapes, and what the model is fitted to, decide which variant fits a task. The original proposal does not fix the form of its reward model 1811.07871 §3.

The original proposal rests on two assumptions. The first is that user intentions can be learned to high enough accuracy. The second is that, for many tasks, judging an outcome is easier than producing the right behaviour 1811.07871 §1. The paper does not define "high enough", and the second assumption fails when outcomes have few possible values, such as yes or no 1811.07871 §1.

The paper names five challenges that could stop reward modeling from scaling: how much feedback is needed, how that feedback is distributed, reward hacking, unacceptable outcomes, and the gap between reward and the result the user wants 1811.07871 §5 fig. 3. It then lists ten approaches that might address them, and it describes a recursive variant for harder tasks [arxiv:1811.07871 §3.2]. Success is not guaranteed 1811.07871 §7.

For the same loop applied to language models with preference data, see the RLHF page.

Variants

Online reward modeling with the user in the loop

The agent acts, the user judges the resulting trajectories, the reward model is refitted on that feedback, and the agent is trained on the reward model's predictions 1811.07871 §3.1. The user gives no per-action labels. The RL algorithm does the credit assignment that would otherwise fall to the user 1811.07871 §3.1. The paper lists four advantages: the user need not judge every interaction, the policy and the reward model can be checked separately, a stronger RL algorithm can be swapped in, and the user need not solve credit assignment 1811.07871 §3.1.

The reward model is kept online, meaning it keeps learning while the agent trains. The paper cites preliminary experiments in other work where agents learned to exploit a reward model that was frozen after training. Keeping the model online also lets it follow the states the agent visits and lets the user catch attempts to game it 1811.07871 §5.1. Card: Scalable agent alignment via reward modeling.

Preference-trained reward for an Atari agent

The agent starts from a few expert game episodes, which it copies by behaviour cloning 1811.06521 §2.2 app. A.4. It then plays, and short 25-step clips of its play are paired and labelled by an annotator §2.4 app. A.3. The reward model is a convolutional network that reads one stacked observation and outputs one scalar. It is fitted to the labels with a cross-entropy loss over the summed rewards of each clip pair, which is in the Bradley-Terry style 1811.06521 §2.4. The policy (DQfD) reads that reward when it samples replay batches, so transitions store no reward §2.3 app. A.3.

The loop repeats: the policy trains on the reward model, new pairs are labelled, and the reward model is refitted alg. 1. The main schedule uses 500 iterations and 6,800 labels in total app. A.4 §3. Labels can be synthetic, taken from the game score that the agent never sees, or they can come from people §3 §3.1. The paper's "autolabels" pair each agent clip with a demonstration clip and count as extra labels §2.5.

The paper reports that a reward model frozen during training is exploited: with it fixed, returns fell in three games as the agent found loopholes §3.2 fig. 4. The authors take this as support for keeping preference feedback coming during training. They measured it only on Atari, with synthetic labels and a few human runs §3.2 §3.1. Card: Reward learning from human preferences and demonstrations in Atari.

Recursive reward modeling

For outcomes too complex for the user to judge, an earlier agent helps. Agent A_1 is trained by plain reward modeling. Agent A_k is trained while A_{k-1} assists the user in evaluating outcomes 1811.07871 §3.2 fig. 2. The assistance could summarise data, interpret the agent's internals, or solve sub-problems the user has split off 1811.07871 §3.2. The helper's task differs from the task the new agent is trained to solve [arxiv:1811.07871 §3.2].

The user has to split the evaluation into sub-evaluations that together cover the whole outcome. An aspect left out of the split can be optimised in an arbitrary direction. The paper suggests one agent proposes a split and another critiques it by naming what was left out 1811.07871 §3.2. The paper links the recursion depth to a complexity-theory picture: reward modeling matches one existential question, and recursion depth k matches k alternating quantifiers 1811.07871 §3.2. It describes the scheme as an instance of iterated amplification 1811.07871 §3.2. Whether errors of A_{k-1} grow in A_k is left open, and the paper proposes ensembles or self-correction as untested options 1811.07871 §3.2. Card: Scalable agent alignment via reward modeling.

Preference-trained scalar scorer for a generator

ImageReward is a scalar scorer for text-to-image outputs. Annotators place all images for a prompt into five preference slots, best to worst. Each ranked list gives pairs, and the scorer is trained with a pairwise logistic loss on the difference of the two scores 2304.05977 §2.1 §2.2 eq. 1. That loss has the Bradley-Terry form (see Bradley-Terry model). The backbone is BLIP, with a small MLP on top of the text encoder's output 2304.05977 §2.2. The annotation took about two months and needed a vendor, a written guideline and quality inspectors 2304.05977 §2.1.

The scorer has two uses in the paper:

• Selection and evaluation. On 6,399 held-out human comparisons, the scorer reaches 65.14% preference accuracy, against 57.76% for BLIP score and 54.82% for CLIP score 2304.05977 tab. 3. As a best-of-64 selector on the paper's human test, it is slightly ahead of PickScore and HPS 2304.05977 tab. 6. The margins there are small, and the test is the authors' own.
• Fine-tuning a generator (ReFL). Instead of scoring finished images only, ReFL draws one random late step out of 40, denoises up to it, predicts the clean image at that step, decodes it and back-propagates the reward 2304.05977 §3 alg. 1. The authors report that back-propagating only through the last step was unstable 2304.05977 §3. In human comparisons with Stable Diffusion v1.4, ReFL wins 58.79% of comparisons on real user prompts 2304.05977 tab. 4.

The paper's objective also adds a pre-training regulariser, whose weight is not stated 2304.05977 §3 eq. 3. The released code does not compute that term ReFL.py. The paper gives the stopping step as 30 to 40 in §3 and 1 to 10 in §4.2, and the code draws 30 to 39 ReFL.py. Treat the paper and the code as two different recipes, and check the stopping range before you rely on it.

Card: ImageReward.

Scorer for choosing a generator's training images

Human Preference Score (HPS) is a second scalar scorer for text-to-image outputs, and it is used in a different way. CLIP ViT-L/14 is fine-tuned so that a prompt's text embedding lands close to the embedding of the image people preferred. The score is 100 times the cosine similarity between the two embeddings 2303.14420 §5. The paper gives the training objective in words only, and the exact loss is not stated §5. The human choices come from users of a public Discord channel §3.

HPS does not enter a reward loss. For each prompt, the images are ranked by score, and the top and bottom images are kept if their softmax weight clears α/n, with α = 2.0 §6. A rank-32 LoRA adapter for Stable Diffusion v1.4 is then trained on the kept images, with the non-preferred captions prefixed by "Weird image." At inference that phrase is the negative prompt §6. The scorer chooses the training data, and the adapter learns from that data.

Author-reported results. On 5,000 user choices, HPS reaches 43.5% accuracy, against 32.9% for CLIP ViT-L/14 and 42.0% for human participants [arxiv:2303.14420 tab. 2]. The adapted model moves FID from 19.72 to 19.35 and the HPS value from 0.1898 to 0.1916 tab. 4. In a study with 20 participants and 100 prompts, 74% of the adapted model's images got more than ten positive votes, against 22% for the original §7.2 fig. 8. The paper gives no variance for the table figures and no significance test tab. 4.

The paper's dataset is distributed under a non-commercial licence [arxiv:2303.14420 app. datasheet]. The paper text is CC BY 4.0, but that does not cover the dataset, so reusing the data needs its own permission. Code caveats, such as a LoRA rank default that differs from the paper, are recorded on the HPS card.

Card: Human Preference Score.

Pairwise ranker over candidate outputs

PairRanker is not a scalar scorer. It reads the instruction and two candidates together, and gives the confidence that the first is better [arxiv:2306.02561 §3.3]. The paper argues that scoring each candidate on its own struggles when all candidates are fluent and differ only subtly 2306.02561 §1 §3.1. Over N candidates, it fills an N by N matrix. The MaxLogits rule then turns that matrix into a ranking, and the paper reports it as the best of the three aggregators it tried 2306.02561 §3.3.

Its training labels come from BARTScore, a text metric, computed against a reference answer. The paper uses BARTScore because its rankings correlate best with GPT-Rank among the metrics it compares 2306.02561 §5.1 tab. 3. Training samples 5 pairs per input 2306.02561 §3.3. No human labels are used.

Results are author-reported, on the MixInstruct benchmark of 11 open LLMs. The PairRanker alone reaches a GPT-Rank of 3.20, against 3.90 for the best single model, OpenAssistant. In that table, lower GPT-Rank is better [arxiv:2306.02561 tab. 2]. The GPT-Rank judge is ChatGPT, and the paper does not state the version 2306.02561 §5.1. Fusing the top three candidates with a seq2seq model (GenFuser, Flan-T5-XL) gives GPT-Rank 3.01 2306.02561 tab. 2 §4. On conventional tasks the gains over the base model are smaller: about 2 to 6% on CNN/DM, CommonGen and WMT18 with the MaxLogits rule [arxiv:2306.02561 app. C.4].

Cost depends on the aggregator. Full comparison needs N(N-1) forward passes per input, which is 110 for N = 11 2306.02561 §3.3. The released bubble mode runs both orders at each step, so it takes 2(N-1) passes ranker.py. Card: LLM-Blender.

Group-consensus reranker over sampled statements

The reward model here scores a statement for one person, and the score is then aggregated over a group. Training uses a question, one participant's opinion, and two candidate statements. The pair is kept only if the two agreement ratings differ by at least two Likert points, and the loss is cross-entropy 2211.15006 §3.4 app. C.2. The reward head is a linear layer on a pretrained 70B model, and the paper's own description of the loss is brief app. C.2.

At inference, a supervised fine-tuned model samples 16 statements. The reward model scores each statement for every opinion in the group, and a welfare function ranks the statements. The main results use the Utilitarian case, a plain mean over people 2211.15006 §3.3 eq. 1. The reranker decides which statement is shown, but it is not a training signal for the generator. The supervised step trains only on quality ratings, so that stances stay diverse 2211.15006 §3.4 app. C.1.2.

Author-reported results. Raters preferred the reranked statement over the mean human opinion in 78% of comparisons, and over the best human opinion in 65% 2211.15006 §4.2. The paper says that the choice of welfare function made no reliable difference in mean or minimum agreement §5.1 app. D.5, so the gain comes from the whole pipeline. The raters were UK-based crowd participants on UK political questions, and the authors note that the population may inflate agreement §5.1. The paper's own limit on group size is prompt length: groups of three to five people §5.1. The authors' public repository does not include this reward model's training code; see the card's Paper vs code section. Card: Fine-tuning language models to find agreement among humans with diverse preferences.

Direct policy optimisation, without a reward model

Direct methods skip the reward model. The policy is trained straight from preference triplets, each a prompt with a preferred and a dispreferred response. DPO is the example the survey gives. Its loss is a logistic loss on the difference of log-ratios between the policy and a frozen reference policy, scaled by β 2312.14925 §6.3 eq. 14. The survey does not give β or the reference policy's value §6.3. Because the reward is never fitted, the policy can only be as good as the preference data it is trained on, and the survey notes that direct methods often have no online data, which can cause distribution shift [App. A, Direct methods]. Card: A Survey of Reinforcement Learning from Human Feedback. For the training objectives themselves, see the preference optimization page.

Learned split of a sparse team return

No judgements are collected. The model is fitted so that its per-step outputs add up to the episodic return of a finished game 2412.14779 §4.1. The return is split over time with non-negative weights that sum to 1, then over agents with a second set of such weights §4.2. The architecture extends AREL's attention reward model with alternating temporal and agent attention blocks §4.5. Single-agent policy-gradient learners, IPPO and MAPPO, train on the split reward in place of the sparse one §5.1. The paper states the sum constraint but no loss for fitting the model §4.1.

The paper argues that this split is potential-based reward shaping, so the optimal policy is kept §4.3. The proof assumes a discount of 1 §4.3, while the paper's own setting uses a discount §3.1. Its potential also depends on the whole episode, whereas standard potential-based shaping takes a function of the state §3.3 §4.3. The evidence is one scenario, 5m_vs_6m in SMACLite, shown as a figure without numbers §5.2 fig. 1. The paper says the split learns faster and beats other redistribution baselines there §5.1 §6. Card: TAR² (Agent-Temporal Credit Assignment).

When to use which

The paper's options for the first variant are arguments, not measured results, so each row there states the condition and the paper's suggested response. The rows for the other variants cite their reported numbers.

Implementation sketch

The loops below are one reading of the papers' setups. The first two sketch the online variant and its recursive form. The third is the pairwise loss used by the ImageReward-style scorer, the fourth is the per-prompt selection rule that HPS uses to choose training images, the fifth is pairwise ranking over candidates, the sixth is the direct DPO loss, which has no reward model, and the seventh is the group reranker. The Atari paper's own training protocol is closest to the first loop, with the policy and reward model updated in turns. The papers give no shared algorithm box, so the names are ours.

# One reward-modeling iteration, user in the loop  [arxiv:1811.07871 §3.1]
# Input:  env, agent (policy), reward_model, user
# Output: updated agent and reward_model
def reward_modeling_iteration(env, agent, reward_model, user):
    # The agent acts and produces trajectories  [§3.1]
    trajectories = agent.rollout(env)
    # The user judges outcomes; no per-action labels  [§3.1]
    feedback = user.evaluate(trajectories)
    # Refit the reward model, kept online  [§3.1] [§5.1]
    reward_model.update(trajectories, feedback)
    # Score the trajectories with the learned reward and train  [§3.1]
    rewards = reward_model.predict(trajectories)
    agent.update(trajectories, rewards)
    return agent, reward_model

# Recursive variant: each level is judged with the previous agent's help  [§3.2]
# Input:  tasks ordered from simplest to hardest
# Output: one agent per level
def recursive_reward_modeling(tasks, user):
    agents = []
    helper = None
    for task in tasks:
        # Level 1 has no helper; later levels get the previous agent  [§3.2]
        agent = train_with_reward_modeling(task, user, assistant=helper)
        agents.append(agent)
        helper = agent
    return agents

# Pairwise loss for a scalar scorer on ranked images  [arxiv:2304.05977 §2.2] [eq. 1]
# Input:  prompts (B,), preferred and dispreferred images
# Output: scalar loss
def pairwise_reward_loss(prompts, better, worse):
    # Score the preferred and the dispreferred image for each prompt  [§2.2]
    score_better = reward(prompts, better)  # (B,)
    score_worse = reward(prompts, worse)  # (B,)
    # Logistic loss on the score gap  [eq. 1]
    gap = score_better - score_worse
    return -mean(log_sigmoid(gap))

# Keep the best and worst image per prompt by score  [arxiv:2303.14420 §6]
# Input:  groups: prompt -> list of (image, score); alpha = 2.0  [§6]
# Output: preferred and non-preferred (image, prompt) pairs
def select_by_score(groups, alpha):
    preferred, non_preferred = [], []
    for prompt, items in groups.items():
        scores = [score for _, score in items]
        n = len(scores)
        # Softmax weight of the top image in its group  [§6]
        best = argmax(scores)
        if softmax(scores)[best] > alpha / n:
            preferred.append((items[best][0], prompt))
        # Same rule on negated scores, for the lowest-scoring image  [§6]
        worst = argmin(scores)
        if softmax([-s for s in scores])[worst] > alpha / n:
            non_preferred.append((items[worst][0], prompt))
    return preferred, non_preferred

# Full comparison matrix and MaxLogits ranking  [arxiv:2306.02561 §3.3]
# Input:  source text; list of N candidates
# Output: candidate indices, best first
def rank_candidates(source, candidates):
    n = len(candidates)
    matrix = zeros(n, n)
    # Fill M[i][j] for every ordered pair with i != j  [§3.3]
    for i in range(n):
        for j in range(n):
            if i != j:
                matrix[i][j] = pair_logit(source, candidates[i], candidates[j])
    # MaxLogits: total margin of each candidate over all others  [§3.3]
    scores = sum(matrix - transpose(matrix), axis=1)
    return argsort(scores, descending=True)

# DPO loss on preference pairs, no reward model  [arxiv:2312.14925 §6.3] [eq. 14]
# Input:  batch of (prompt, preferred, dispreferred), frozen reference policy
# Output: scalar loss; beta is not stated in the survey
def dpo_loss(policy, reference, prompts, preferred, dispreferred, beta):
    # Log-ratio of each response against the frozen reference  [eq. 14]
    ratio_win = policy.logp(preferred, prompts) - reference.logp(preferred, prompts)
    ratio_lose = policy.logp(dispreferred, prompts) - reference.logp(dispreferred, prompts)
    # Logistic loss on the scaled gap between the two log-ratios  [eq. 14]
    return -mean(log_sigmoid(beta * (ratio_win - ratio_lose)))

# Group reranking by Utilitarian welfare, alpha = 0  [arxiv:2211.15006 §3.3] [eq. 1]
# Input:  question; 3 to 5 opinions; candidate statements from the SFT model
# Output: the candidate with the highest mean predicted agreement
def consensus_rerank(question, opinions, candidates):
    # Predicted agreement of each member with each candidate  [§3.3]
    agreement = [[reward(question, op, c) for c in candidates] for op in opinions]
    # Welfare with alpha = 0 is the mean over people  [eq. 1]
    welfare = [mean(column) for column in zip(*agreement)]
    return candidates[argmax(welfare)]

The variants differ in where they put the reward. The online variant trains the agent on the reward model's output §3.1. The Atari variant does the same, and its policy reads the reward at sampling time 1811.06521 §2.3. Recursive reward modeling passes an assistant into the evaluation step §3.2. The scalar scorer trains on pairs of human rankings and can also back-propagate its score into a generator [arxiv:2304.05977 §3]. HPS uses its score only to choose training images, so the score is not a training signal for the adapter 2303.14420 §6. The pairwise ranker never produces a scalar, so its output can only order candidates 2306.02561 §3.3. The DPO loss has no reward function at all: the policy's log-ratio to the reference plays that role 2312.14925 §6.3. The team-return model gives one value per agent and step, and those values sum to the episodic return rather than to any judgement 2412.14779 §4.2. The group reranker is the only variant that scores each candidate per person and then aggregates across people, and its reward is used only to rank samples at inference 2211.15006 §3.3. The reward model's architecture, loss and update schedule in the first variant are not specified in the paper, so update and predict are left to the implementer 1811.07871 §3.

Contradictions

• Frozen versus online scorer. The first paper says that a reward model frozen after training can be exploited by an agent, citing preliminary results in other work, and it recommends keeping the model online [arxiv:1811.07871 §5.1]. ImageReward uses a frozen scorer while it fine-tunes the generator, and reports human gains over the baseline 2304.05977 §3 tab. 4. The ImageReward paper, as compiled here, reports no check for exploitation of its frozen scorer. The Atari paper does report one, for a frozen reward model in three games 1811.06521 §3.2. HPS uses its score to select data, and its paper reports no test for exploitation 2303.14420 §6. The papers do not test the same setup, and this page does not resolve the disagreement.
• Source of the labels. The first paper treats the user's own judgement as the training signal 1811.07871 §3.1. The Atari paper trains on game score labels for its main results, and on human labels in a separate set of runs with reported mislabelling 1811.06521 §3 §3.1. ImageReward trains on expert human rankings and argues that automatic metrics miss human preference 2304.05977 §2.3. HPS trains on choices made by users of a public Discord channel 2303.14420 §3. PairRanker learns from BARTScore, and its agreement is measured against GPT-Rank, a ChatGPT judgement whose version is not stated 2306.02561 §5.1 app. E. The sources measure different things, so their numbers are not directly comparable.
• Accuracy of CLIP as a judge. ImageReward reports CLIP score at 54.82% and its own scorer at 65.14%, on 6,399 held-out human comparisons [arxiv:2304.05977 tab. 3]. HPS reports CLIP ViT-L/14 at 32.9% and its own classifier at 43.5%, on 5,000 user choices 2303.14420 tab. 2. The HPS table's random baseline is 26.1%, not the 50% of a pairwise test, so the two tests probably use different choice sets. This page cannot tell which setup is closer to a real use, and it does not resolve the gap.

Papers

• Scalable agent alignment via reward modeling: a research direction: sets out reward modeling as a research direction, with five scaling challenges, ten approaches and recursive reward modeling. No experiments.
• Reward learning from human preferences and demonstrations in Atari: a reward model fitted to pairwise clip preferences, and a DQfD agent trained on it, with expert demonstrations. Author-reported, nine games.
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation: a scalar preference scorer trained on ranked human comparisons, and ReFL, which fine-tunes Stable Diffusion v1.4 through it.
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference: a CLIP-based scorer trained on user choices, used to select training images for a LoRA adapter of Stable Diffusion v1.4 with a negative-prompt marker.
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion: a pairwise ranker (PairRanker) over candidate LLM outputs, and a fuser (GenFuser) that writes one answer from the top three.
• TAR² (Agent-Temporal Credit Assignment): a learned split of a sparse team return into per-step, per-agent rewards for single-agent learners. One scenario, shown as a figure.
• A Survey of Reinforcement Learning from Human Feedback: a survey of RLHF with the Bradley-Terry reward model, choice models, policy learning and direct methods such as DPO. No experiments.
• Fine-tuning language models to find agreement among humans with diverse preferences: a per-person agreement reward model that reranks 16 sampled statements by Utilitarian welfare for groups of three to five. Author-reported, UK crowd participants.