Preference optimization

Overview

Preference optimization trains a model on judgments of which of two outputs is better, and pushes probability toward the preferred output. The classic route for language models has three stages: collect preferences, fit a reward model to them, then optimise the policy against that reward with a penalty for drifting from the base model 2307.15217 §2. The direct variants skip the reward model and the RL loop. DPO keeps the penalty as a frozen reference model 2305.18290 §4, and DPO derives its loss from the same KL-constrained reward objective that RLHF optimises, so the optimum can be written in terms of the policy itself 2305.18290 §4. Not every variant needs pairs: KTO trains on one good or bad label per output 2402.01306 §1. The pairing can also be made per step inside an agent trajectory, comparing an expert action with the agent's own action at the same prefix 2411.03817 §4.1.

The survey in this set treats reward-model RLHF as having open problems in the feedback, the reward model and joint training 2307.15217 §3. It names direct, reward-model-free preference training as an alternative 2307.15217 §4.2.3 but runs no experiments 2307.15217 §1. A second survey gives the DPO loss in the same form and fits the reward model with a Bradley-Terry likelihood 2312.14925 §6.3 2312.14925 eq. 14. It also reports no experiments 2312.14925 §8. For the full pipeline see RLHF.

Variants

Direct Preference Optimization (pairwise, reference-based)

Each training example is a prompt with a preferred and a rejected completion. The loss is a logistic function of the difference between the policy's and the reference's log-probability ratios, scaled by beta. No sampling from the policy is part of the update loop 2305.18290 §4. Card: Direct Preference Optimization.

Author-reported results:

• IMDb sentiment (GPT-2-large): DPO has the highest reward at every KL value in the sweep, though the text gives no numbers 2305.18290 §6.1.
• TL;DR summarisation: a GPT-4-judged win rate of about 61% at temperature 0, against about 57% for PPO at its best temperature 2305.18290 §6.2. DPO's win rate stays stable as sampling temperature rises, where PPO's falls 2305.18290 §6.2.
• Dialogue (Anthropic-HH, Pythia-2.8B): DPO is the only tested method that beats the preferred completions in the paper's GPT-4 evaluation 2305.18290 §6.2.

Beta is 0.1 in most runs and 0.5 for summarisation 2305.18290 app. B. The authors say they did not tune beta for summarisation, which may understate DPO there 2305.18290 §6.2.

KTO (one binary label per output, reference-based)

Each output carries one label, desirable or undesirable, and no pairing is needed. The loss scores each output with a prospect-theory value function of its policy-to-reference log-ratio r. The reference point z0 is estimated from mismatched outputs in the same microbatch and clamped at zero, with no gradient through it. Desirable and undesirable outputs get separate loss-aversion weights, lambda_D and lambda_U 2402.01306 §4.1 2402.01306 eq. 8. Card: KTO: Model Alignment as Prospect Theoretic Optimization.

Settings in the paper's sweeps: AdamW, learning rate 5e-6, lambda_D and lambda_U both 1, effective batch size 32 2402.01306 tab. 1. Beta varies across the table's rows, from 0.05 to 0.50, and 0.1 is the default 2402.01306 tab. 1 2402.01306 §4.2. The authors say the good KTO learning rate is usually 2x to 10x the good DPO rate, and that DPO's optimal beta does not transfer 2402.01306 §4.2. Their rule for the weights is lambda_D n_D / (lambda_U n_U) between 1 and 3/2, where n_D and n_U count the two labels 2402.01306 eq. 9 2402.01306 §4.2.

Author-reported results:

• Zephyr-beta-SFT trained for one epoch on UltraFeedback: GSM8K 40.0 with DPO against 53.5 with KTO, BBH 44.1 against 52.6, and MMLU 58.2 against 58.6 2402.01306 tab. 2.
• Mistral-7B on OpenAssistant, GPT-4-judged win rate against the SFT target: 0.600 with DPO, 0.652 with KTO on all outputs, and 0.631 with one output per input, which cuts the training data by 72% 2402.01306 tab. 3.
• Across 1B to 30B models, KTO alone beats DPO alone on Llama 7B, 13B and 30B. The gap is significant at 7B and 30B, and Pythia shows no significant difference 2402.01306 fig. 3.
• With up to 90% of the desirable examples discarded, KTO on Llama 7B still matches or beats DPO 2402.01306 fig. 5.
• Ablations on Zephyr-beta-SFT: dropping z0 gives GSM8K 49.5, a risk-neutral value function gives BBH 6.1, and the reference-free variant gives GSM8K 47.5 2402.01306 tab. 2.

Diffusion DPO-style denoising loss (pairwise)

The same logistic pairwise loss, adapted to diffusion models. Denoising errors of the policy and the reference on a preferred and a rejected sample replace the log-probabilities. All four error terms share one noise draw and one time step 2412.02617 §3.1 2412.02617 eq. 7. Card: Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback.

Setting: a 3B text-to-video model, 120 training prompts and 40 test prompts, with accept or reject labels from a vision-language model 2412.02617 §3.3 2412.02617 §4.

• Gemini "All" accuracy: 52.66% for the pre-trained model, 56.17% for DPO with AI feedback 2412.02617 tab. 2.
• Human "All": 19.38% pre-trained, 34.06% for DPO with AI feedback 2412.02617 tab. 2.
• On the test split, DPO scores 55.00% against 50.94% for reward-weighted regression with the same feedback 2412.02617 tab. 2.
• Video quality metrics stay closer to the pre-trained model than SFT does, for example FVD 11.98 for DPO against 14.13 for SFT 2412.02617 tab. 3.
• Over two iterative rounds, DPO saturates at about 56.25% on Gemini "All", while reward-weighted regression reaches about 58.83% 2412.02617 §4.3 2412.02617 fig. 5.

Reward-weighted regression (scalar or binary scores)

Not pairwise. Each sample is scored, and a denoising loss weights its error by the reward, so high-reward samples count more 2412.02617 §3.1 2412.02617 eq. 6. The same paper uses it as the second offline algorithm. With the AI feedback reward it reaches 56.41% Gemini "All" and 31.09% human "All" 2412.02617 tab. 2. It needs no pairing rule, because each sample carries its own label.

DPO as one stage of a post-training recipe (pairwise, text)

Not a new loss. The Phi-3 report runs DPO as the last step after supervised fine-tuning 2404.14219 §2. Its rejected responses are unwanted behaviours, and its DPO data covers chat, reasoning and responsible-AI cases 2404.14219 §2. The report does not state the DPO beta or other DPO settings 2404.14219 §2. Its results are for the finished models, so the card records no separate figure for the DPO stage 2404.14219 §3. Card: Phi-3 Technical Report.

Mixtral 8x7B's instruct model is also supervised fine-tuning followed by DPO 2401.04088 §4. The instruction data and the DPO settings are not stated 2401.04088 §4, and the reported results are for the instruct model as a whole, so the card gives no separate DPO effect 2401.04088 §4. Card: Mixtral of Experts.

Metric-scored pairs after SFT (pairwise, text, no judge in the loop at train time)

Not a new loss. The pairs come from a reward that is the average of several metric scores on each sampled candidate. Four candidates are sampled per student turn, and a pair is kept only when the averages differ by more than a threshold epsilon of 0.1 2601.04025 §2.3 2601.04025 app. B.1. The metrics are reference-based, and two of them use proprietary LLM judges 2601.04025 §2.2. The DPO step follows SFT, with beta 0.1 and one epoch on a random 20% of the training dialogues 2601.04025 app. B.1. Card: Simulated Students in Tutoring Dialogues: Substance or Illusion?.

Author-reported results on the Eedi test set, turn level 2601.04025 tab. 1:

• Llama 3.1 8B: DPO against SFT is higher on acts (0.6840 against 0.6671), correctness (0.5761 against 0.5670), knowledge (0.8787 against 0.8766) and cosine similarity (0.7390 against 0.7383). ROUGE-L is 0.3212 for both.
• Errors is the exception. DPO is lower than SFT on it: 0.0529 against 0.0661 for Llama 3.1 8B, and 0.0584 against 0.0815 for Llama 3.2 3B. The paper attributes the weak errors result to a sparse reward signal 2601.04025 §4.4.
• Zero-shot GPT-4.1 scores below the fine-tuned models on acts (0.4998) and on cosine similarity (0.5460) 2601.04025 tab. 1.
• The Oracle prompt, which sees an LLM summary of the whole dialogue including the real student's errors, scores highest on correctness (0.6755) and highest on errors (0.1872) in the table. The authors say this row leaks real information 2601.04025 §4.1.
• In a single-metric ablation on Llama 3.2 3B, DPO with the averaged reward reaches acts 0.6962 against 0.6795 for SFT on a 20% test subset 2601.04025 tab. 4. That subset's SFT row differs from the full-test SFT row, so the two tables are not directly comparable.
• Fine-tuned students write much shorter turns than prompted ones or real students: 2.28 words for DPO 8B, against 4.11 for real students 2601.04025 §4.3.

Human evaluation covered 38 dialogues, three evaluators, and only DPO 8B, zero-shot and Oracle. The paper does not say how its 1 to 5 linguistic scale was converted 2601.04025 tab. 2 2601.04025 app. E.

Step-wise implicit reward for LLM agents (pairwise, per step)

Each expert trajectory is cut after every action. At each prefix, the expert action is the preferred response and the agent's sampled action is the dispreferred one, and the DPO objective is applied to that pair. No human step labels are used 2411.03817 §4.1 2411.03817 §4.2.1 2411.03817 eq. 6. The paper says the reference model starts from the agent policy but does not say whether it is frozen 2411.03817 §4.2.1. Card: From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning.

Author-reported results, one run per cell, on Mistral-7B and Llama-3-8B:

• WebShop: 66.2 for the implicit variant against 64.1 for the best baseline (ETO) on Mistral-7B, and 67.2 against 65.7 on Llama-3-8B 2411.03817 tab. 2.
• Multi-hop QA exact match: 31.0 against 28.2 (ETO) on HotpotQA with Mistral-7B, and 38.1 against 35.2 with Llama-3-8B 2411.03817 tab. 2.
• The best iteration count for WebShop is 3 for this variant and 7 for the inverse-RL variant 2411.03817 §7.4 2411.03817 fig. 4.

Settings reported: reflection learning rate 5e-7, batch size 16, one epoch 2411.03817 §6.4. Beta is not stated 2411.03817 eq. 6. The paper's second route, a discriminator trained with PPO, is inverse RL rather than a preference loss, so see Inverse reinforcement learning 2411.03817 §4.2.2.

Reward-model RLHF (baseline)

Fit a reward model, then optimise the policy with RL. This is the baseline that the direct variants are measured against 2305.18290 §6. See RLHF.

Shared caveats

• Beta controls how far the policy may move from the reference. A larger value keeps it closer and a smaller value lets it move further 2305.18290 §4.
• The DPO loss uses whole-completion log-probabilities. The paper does not discuss length effects 2305.18290 §4.
• Win rates depend on the judge prompt, and the authors say results are sensitive to it 2305.18290 §6.4 2305.18290 §7. KTO's wins are also GPT-4-judged. GPT-4 agreed with individual human judgments on 68.7% of KTO comparisons and 65.9% of DPO comparisons 2402.01306 app. D.
• Splitting a preference pair into one desirable and one undesirable output is called naive by the KTO authors, who leave better decompositions to future work 2402.01306 §4.1.
• The diffusion paper's headline judge is the same model family as its reward. Its GPT-4o check shows a smaller gain, and the test split has only 40 prompts 2412.02617 §4.3.
• Imaging quality drops slightly under every reward, including AI feedback 2412.02617 tab. 2.
• Step-level pairs are built from the agent's own action at each expert prefix, so they test a different thing from response-level pairs. Each agent step is scored against one expert step, and the paper reports one run per cell with no variance 2411.03817 §4.1 2411.03817 tab. 2.
• Metric-scored pairs inherit the judge's errors. The authors warn that training on judge-scored metrics can overfit the judge, and they recommend expert human evaluation when metrics are used in training 2601.04025 §2.3. The paper's text forms all pairs above the threshold, while the released code keeps only the six largest-gap pairs per turn. The card records this difference paper card.

When to use which

Implementation sketch

Most variants share one loss over a batch, with a frozen reference and no sampling from the policy. The pairwise DPO loss below is the reference shape. KTO keeps the batch but drops the pairs, and the diffusion variant keeps the shape and changes the inputs.

# Pairwise DPO loss over a batch  [arxiv:2305.18290 §4]
# Each log-prob is the summed token log-probability of a full completion
def dpo_loss(pol_win, pol_lose, ref_win, ref_lose, beta):
    # Policy log-ratio, preferred minus rejected  [arxiv:2305.18290 §4]
    pol_margin = pol_win - pol_lose        # (B,)
    # Same margin under the frozen reference; no gradient flows here  [§4]
    ref_margin = ref_win - ref_lose        # (B,)
    # Implicit reward gap, scaled by beta  [arxiv:2305.18290 §4]
    logit = beta * (pol_margin - ref_margin)   # (B,)
    # Negative log-sigmoid, averaged over the batch  [arxiv:2305.18290 §4]
    return -log_sigmoid(logit).mean()

Where variants differ:

• KTO. Drop the pair. Each output's log-ratio r is scored against a reference point z0, the clamped mean of mismatched log-ratios in the microbatch, with no gradient through z0. A desirable output contributes lambda_D times sigmoid(beta (r - z0)), and an undesirable one contributes lambda_U times sigmoid(beta (z0 - r)). The loss is lambda minus that value, averaged 2402.01306 §4.1 2402.01306 eq. 8. The card has full pseudocode.
• Diffusion. Replace each log-probability with a denoising error on one shared noise draw and time step, then flip the sign of the logit, because a lower error is the better outcome 2412.02617 §3.1 2412.02617 eq. 7.
• Reward-weighted regression. Drop the pair. Weight each sample's denoising error by its reward 2412.02617 eq. 6.
• Metric-scored pairs. Keep the pairwise loss unchanged. Only the pair builder differs: score each candidate by the mean of its metrics, then keep ordered pairs whose gap exceeds epsilon (0.1 in the paper). The loss sees only the winner and loser 2601.04025 §2.3 2601.04025 app. B.1.

Contradictions

One disagreement concerns PPO against direct methods. The DPO paper reports DPO matching or beating PPO on sentiment and TL;DR summarisation 2305.18290 §6.1 2305.18290 §6.2. The first survey reports, citing Xu et al. (2024), that tuned PPO can outperform direct methods 2312.14925 app. A. The survey runs no experiments of its own, so its statement is a citation, not a measurement. Both are left as stated.

Otherwise, no direct conflict in the core claims. The settings differ, so the results do not directly test each other. The DPO-over-PPO comparison is made only for text 2305.18290 §6, and the dialogue result is not a PPO comparison 2305.18290 §6.2. In the diffusion paper, DPO and reward-weighted regression are close on the headline Gemini metric, 56.17% and 56.41% 2412.02617 tab. 2, and that paper runs no PPO baseline. The survey gives no numbers for either method 2307.15217 §1.

One difference in scope is worth noting. The KTO paper says that up to 7B parameters, alignment gave almost no gain over SFT for the methods it compared 2402.01306 §3.3. The DPO paper reports a GPT-4-judged win rate of about 61% against the reference summaries on TL;DR, with a GPT-J-based 6B SFT model, and reports dialogue results on Pythia-2.8B 2305.18290 §6.2. The tasks, judges and baselines differ, so the two statements are not a direct test of each other. Both are left as the papers state them.

The step-wise agent paper reports gains over the best baseline, ETO in every cell, on WebShop, Mind2Web, ScienceWorld, ALFWorld and three multi-hop QA sets 2411.03817 tab. 2. Those numbers are on agent benchmarks with exact-match or reward metrics, not on the response-level win rates above, so they do not confirm or contradict them.

Papers

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model: the pairwise logistic loss against a frozen reference, with results on sentiment, summarisation and dialogue.
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback: a survey of problems in reward-model RLHF, naming direct preference training as an alternative.
• A Survey of Reinforcement Learning from Human Feedback: a survey of RLHF that gives the DPO loss and the Bradley-Terry reward model; no experiments, and the DPO beta is not stated.
• Simulated Students in Tutoring Dialogues: Substance or Illusion?: DPO after SFT with pairs ranked by an average of reference-based metrics, for simulated tutoring students on Llama models.
• Phi-3 Technical Report: a small language model whose post-training ends with DPO on unwanted-behaviour rejects; the DPO settings are not stated.
• Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback: a diffusion DPO-style loss and reward-weighted regression, trained on accept or reject feedback from a vision-language model.
• KTO: Model Alignment as Prospect Theoretic Optimization: a binary-label loss with a prospect-theory value and a batch-based reference point, compared with DPO from 1B to 30B parameters.
• Mixtral of Experts: a sparse mixture-of-experts model whose instruct version is SFT followed by DPO; the DPO settings are not stated.
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning: a step-wise DPO-style loss on expert versus agent actions at each prefix, plus an inverse-RL variant with PPO.