--- kind: concept slug: preference-optimization title: Preference optimization summary: >- Fine-tuning a model on preference judgments without a separate reward model or an RL sampling loop. DPO uses one logistic loss on pairs against a frozen reference and matches or beats PPO-based RLHF on sentiment and TL;DR summarisation up to 6B. KTO needs only a good or bad label per output. A diffusion variant applies the pairwise idea to video. tags: [alignment, post-training, fine-tuning] papers: ["2305.18290", "2307.15217", "2312.14925", "2401.04088", "2404.14219", "2412.02617", "2402.01306", "2411.03817", "2601.04025"] related: [rlhf, bradley-terry-model, policy-gradient, instruction-tuning, diffusion-models, llm-as-a-judge, reward-modeling, llm-agent, imitation-learning, inverse-reinforcement-learning] status: draft updated_at: 2026-10-31 --- ## Overview Preference optimization trains a model on judgments of which of two outputs is better, and pushes probability toward the preferred output. The classic route for language models has three stages: collect preferences, fit a reward model to them, then optimise the policy against that reward with a penalty for drifting from the base model [arxiv:2307.15217 §2]. The direct variants skip the reward model and the RL loop. DPO keeps the penalty as a frozen reference model [arxiv:2305.18290 §4], and DPO derives its loss from the same KL-constrained reward objective that RLHF optimises, so the optimum can be written in terms of the policy itself [arxiv:2305.18290 §4]. Not every variant needs pairs: KTO trains on one good or bad label per output [arxiv:2402.01306 §1]. The pairing can also be made per step inside an agent trajectory, comparing an expert action with the agent's own action at the same prefix [arxiv:2411.03817 §4.1]. The survey in this set treats reward-model RLHF as having open problems in the feedback, the reward model and joint training [arxiv:2307.15217 §3]. It names direct, reward-model-free preference training as an alternative [arxiv:2307.15217 §4.2.3] but runs no experiments [arxiv:2307.15217 §1]. A second survey gives the DPO loss in the same form and fits the reward model with a Bradley-Terry likelihood [arxiv:2312.14925 §6.3] [arxiv:2312.14925 eq. 14]. It also reports no experiments [arxiv:2312.14925 §8]. For the full pipeline see [RLHF](/concepts/rlhf). ## Variants ### Direct Preference Optimization (pairwise, reference-based) Each training example is a prompt with a preferred and a rejected completion. The loss is a logistic function of the difference between the policy's and the reference's log-probability ratios, scaled by beta. No sampling from the policy is part of the update loop [arxiv:2305.18290 §4]. Card: [Direct Preference Optimization](/papers/2305.18290). Author-reported results: - IMDb sentiment (GPT-2-large): DPO has the highest reward at every KL value in the sweep, though the text gives no numbers [arxiv:2305.18290 §6.1]. - TL;DR summarisation: a GPT-4-judged win rate of about 61% at temperature 0, against about 57% for PPO at its best temperature [arxiv:2305.18290 §6.2]. DPO's win rate stays stable as sampling temperature rises, where PPO's falls [arxiv:2305.18290 §6.2]. - Dialogue (Anthropic-HH, Pythia-2.8B): DPO is the only tested method that beats the preferred completions in the paper's GPT-4 evaluation [arxiv:2305.18290 §6.2]. Beta is 0.1 in most runs and 0.5 for summarisation [arxiv:2305.18290 app. B]. The authors say they did not tune beta for summarisation, which may understate DPO there [arxiv:2305.18290 §6.2]. ### KTO (one binary label per output, reference-based) Each output carries one label, desirable or undesirable, and no pairing is needed. The loss scores each output with a prospect-theory value function of its policy-to-reference log-ratio r. The reference point z0 is estimated from mismatched outputs in the same microbatch and clamped at zero, with no gradient through it. Desirable and undesirable outputs get separate loss-aversion weights, lambda_D and lambda_U [arxiv:2402.01306 §4.1] [arxiv:2402.01306 eq. 8]. Card: [KTO: Model Alignment as Prospect Theoretic Optimization](/papers/2402.01306). Settings in the paper's sweeps: AdamW, learning rate 5e-6, lambda_D and lambda_U both 1, effective batch size 32 [arxiv:2402.01306 tab. 1]. Beta varies across the table's rows, from 0.05 to 0.50, and 0.1 is the default [arxiv:2402.01306 tab. 1] [arxiv:2402.01306 §4.2]. The authors say the good KTO learning rate is usually 2x to 10x the good DPO rate, and that DPO's optimal beta does not transfer [arxiv:2402.01306 §4.2]. Their rule for the weights is lambda_D n_D / (lambda_U n_U) between 1 and 3/2, where n_D and n_U count the two labels [arxiv:2402.01306 eq. 9] [arxiv:2402.01306 §4.2]. Author-reported results: - Zephyr-beta-SFT trained for one epoch on UltraFeedback: GSM8K 40.0 with DPO against 53.5 with KTO, BBH 44.1 against 52.6, and MMLU 58.2 against 58.6 [arxiv:2402.01306 tab. 2]. - Mistral-7B on OpenAssistant, GPT-4-judged win rate against the SFT target: 0.600 with DPO, 0.652 with KTO on all outputs, and 0.631 with one output per input, which cuts the training data by 72% [arxiv:2402.01306 tab. 3]. - Across 1B to 30B models, KTO alone beats DPO alone on Llama 7B, 13B and 30B. The gap is significant at 7B and 30B, and Pythia shows no significant difference [arxiv:2402.01306 fig. 3]. - With up to 90% of the desirable examples discarded, KTO on Llama 7B still matches or beats DPO [arxiv:2402.01306 fig. 5]. - Ablations on Zephyr-beta-SFT: dropping z0 gives GSM8K 49.5, a risk-neutral value function gives BBH 6.1, and the reference-free variant gives GSM8K 47.5 [arxiv:2402.01306 tab. 2]. ### Diffusion DPO-style denoising loss (pairwise) The same logistic pairwise loss, adapted to diffusion models. Denoising errors of the policy and the reference on a preferred and a rejected sample replace the log-probabilities. All four error terms share one noise draw and one time step [arxiv:2412.02617 §3.1] [arxiv:2412.02617 eq. 7]. Card: [Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](/papers/2412.02617). Setting: a 3B text-to-video model, 120 training prompts and 40 test prompts, with accept or reject labels from a vision-language model [arxiv:2412.02617 §3.3] [arxiv:2412.02617 §4]. - Gemini "All" accuracy: 52.66% for the pre-trained model, 56.17% for DPO with AI feedback [arxiv:2412.02617 tab. 2]. - Human "All": 19.38% pre-trained, 34.06% for DPO with AI feedback [arxiv:2412.02617 tab. 2]. - On the test split, DPO scores 55.00% against 50.94% for reward-weighted regression with the same feedback [arxiv:2412.02617 tab. 2]. - Video quality metrics stay closer to the pre-trained model than SFT does, for example FVD 11.98 for DPO against 14.13 for SFT [arxiv:2412.02617 tab. 3]. - Over two iterative rounds, DPO saturates at about 56.25% on Gemini "All", while reward-weighted regression reaches about 58.83% [arxiv:2412.02617 §4.3] [arxiv:2412.02617 fig. 5]. ### Reward-weighted regression (scalar or binary scores) Not pairwise. Each sample is scored, and a denoising loss weights its error by the reward, so high-reward samples count more [arxiv:2412.02617 §3.1] [arxiv:2412.02617 eq. 6]. The same paper uses it as the second offline algorithm. With the AI feedback reward it reaches 56.41% Gemini "All" and 31.09% human "All" [arxiv:2412.02617 tab. 2]. It needs no pairing rule, because each sample carries its own label. ### DPO as one stage of a post-training recipe (pairwise, text) Not a new loss. The Phi-3 report runs DPO as the last step after supervised fine-tuning [arxiv:2404.14219 §2]. Its rejected responses are unwanted behaviours, and its DPO data covers chat, reasoning and responsible-AI cases [arxiv:2404.14219 §2]. The report does not state the DPO beta or other DPO settings [arxiv:2404.14219 §2]. Its results are for the finished models, so the card records no separate figure for the DPO stage [arxiv:2404.14219 §3]. Card: [Phi-3 Technical Report](/papers/2404.14219). Mixtral 8x7B's instruct model is also supervised fine-tuning followed by DPO [arxiv:2401.04088 §4]. The instruction data and the DPO settings are not stated [arxiv:2401.04088 §4], and the reported results are for the instruct model as a whole, so the card gives no separate DPO effect [arxiv:2401.04088 §4]. Card: [Mixtral of Experts](/papers/2401.04088). ### Metric-scored pairs after SFT (pairwise, text, no judge in the loop at train time) Not a new loss. The pairs come from a reward that is the average of several metric scores on each sampled candidate. Four candidates are sampled per student turn, and a pair is kept only when the averages differ by more than a threshold epsilon of 0.1 [arxiv:2601.04025 §2.3] [arxiv:2601.04025 app. B.1]. The metrics are reference-based, and two of them use proprietary LLM judges [arxiv:2601.04025 §2.2]. The DPO step follows SFT, with beta 0.1 and one epoch on a random 20% of the training dialogues [arxiv:2601.04025 app. B.1]. Card: [Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025). Author-reported results on the Eedi test set, turn level [arxiv:2601.04025 tab. 1]: - Llama 3.1 8B: DPO against SFT is higher on acts (0.6840 against 0.6671), correctness (0.5761 against 0.5670), knowledge (0.8787 against 0.8766) and cosine similarity (0.7390 against 0.7383). ROUGE-L is 0.3212 for both. - Errors is the exception. DPO is lower than SFT on it: 0.0529 against 0.0661 for Llama 3.1 8B, and 0.0584 against 0.0815 for Llama 3.2 3B. The paper attributes the weak errors result to a sparse reward signal [arxiv:2601.04025 §4.4]. - Zero-shot GPT-4.1 scores below the fine-tuned models on acts (0.4998) and on cosine similarity (0.5460) [arxiv:2601.04025 tab. 1]. - The Oracle prompt, which sees an LLM summary of the whole dialogue including the real student's errors, scores highest on correctness (0.6755) and highest on errors (0.1872) in the table. The authors say this row leaks real information [arxiv:2601.04025 §4.1]. - In a single-metric ablation on Llama 3.2 3B, DPO with the averaged reward reaches acts 0.6962 against 0.6795 for SFT on a 20% test subset [arxiv:2601.04025 tab. 4]. That subset's SFT row differs from the full-test SFT row, so the two tables are not directly comparable. - Fine-tuned students write much shorter turns than prompted ones or real students: 2.28 words for DPO 8B, against 4.11 for real students [arxiv:2601.04025 §4.3]. Human evaluation covered 38 dialogues, three evaluators, and only DPO 8B, zero-shot and Oracle. The paper does not say how its 1 to 5 linguistic scale was converted [arxiv:2601.04025 tab. 2] [arxiv:2601.04025 app. E]. ### Step-wise implicit reward for LLM agents (pairwise, per step) Each expert trajectory is cut after every action. At each prefix, the expert action is the preferred response and the agent's sampled action is the dispreferred one, and the DPO objective is applied to that pair. No human step labels are used [arxiv:2411.03817 §4.1] [arxiv:2411.03817 §4.2.1] [arxiv:2411.03817 eq. 6]. The paper says the reference model starts from the agent policy but does not say whether it is frozen [arxiv:2411.03817 §4.2.1]. Card: [From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning](/papers/2411.03817). Author-reported results, one run per cell, on Mistral-7B and Llama-3-8B: - WebShop: 66.2 for the implicit variant against 64.1 for the best baseline (ETO) on Mistral-7B, and 67.2 against 65.7 on Llama-3-8B [arxiv:2411.03817 tab. 2]. - Multi-hop QA exact match: 31.0 against 28.2 (ETO) on HotpotQA with Mistral-7B, and 38.1 against 35.2 with Llama-3-8B [arxiv:2411.03817 tab. 2]. - The best iteration count for WebShop is 3 for this variant and 7 for the inverse-RL variant [arxiv:2411.03817 §7.4] [arxiv:2411.03817 fig. 4]. Settings reported: reflection learning rate 5e-7, batch size 16, one epoch [arxiv:2411.03817 §6.4]. Beta is not stated [arxiv:2411.03817 eq. 6]. The paper's second route, a discriminator trained with PPO, is inverse RL rather than a preference loss, so see [Inverse reinforcement learning](/concepts/inverse-reinforcement-learning) [arxiv:2411.03817 §4.2.2]. ### Reward-model RLHF (baseline) Fit a reward model, then optimise the policy with RL. This is the baseline that the direct variants are measured against [arxiv:2305.18290 §6]. See [RLHF](/concepts/rlhf). ### Shared caveats - Beta controls how far the policy may move from the reference. A larger value keeps it closer and a smaller value lets it move further [arxiv:2305.18290 §4]. - The DPO loss uses whole-completion log-probabilities. The paper does not discuss length effects [arxiv:2305.18290 §4]. - Win rates depend on the judge prompt, and the authors say results are sensitive to it [arxiv:2305.18290 §6.4] [arxiv:2305.18290 §7]. KTO's wins are also GPT-4-judged. GPT-4 agreed with individual human judgments on 68.7% of KTO comparisons and 65.9% of DPO comparisons [arxiv:2402.01306 app. D]. - Splitting a preference pair into one desirable and one undesirable output is called naive by the KTO authors, who leave better decompositions to future work [arxiv:2402.01306 §4.1]. - The diffusion paper's headline judge is the same model family as its reward. Its GPT-4o check shows a smaller gain, and the test split has only 40 prompts [arxiv:2412.02617 §4.3]. - Imaging quality drops slightly under every reward, including AI feedback [arxiv:2412.02617 tab. 2]. - Step-level pairs are built from the agent's own action at each expert prefix, so they test a different thing from response-level pairs. Each agent step is scored against one expert step, and the paper reports one run per cell with no variance [arxiv:2411.03817 §4.1] [arxiv:2411.03817 tab. 2]. - Metric-scored pairs inherit the judge's errors. The authors warn that training on judge-scored metrics can overfit the judge, and they recommend expert human evaluation when metrics are used in training [arxiv:2601.04025 §2.3]. The paper's text forms all pairs above the threshold, while the released code keeps only the six largest-gap pairs per turn. The card records this difference [paper card](/papers/2601.04025). ## When to use which | Situation | Use | Why | |---|---|---| | Text model, pairwise preference data, a supervised reference model, and no appetite for an RL loop | DPO [arxiv:2305.18290 §4] | Needs log-probabilities of both completions under the policy and the reference. Reported to match or beat PPO on sentiment and TL;DR summarisation [arxiv:2305.18290 §6.1] [arxiv:2305.18290 §6.2]. For dialogue the card cites comparisons with Best of 128 and Preferred-FT, not PPO [arxiv:2305.18290 §6.2] | | Only a good or bad label per output, and pairs are scarce or costly | KTO [arxiv:2402.01306 §4.1] | Needs no pairs. Reported to beat DPO alone on Llama 7B, 13B and 30B, with no significant difference on Pythia [arxiv:2402.01306 fig. 3], and to raise GSM8K on Zephyr-beta-SFT from 40.0 (DPO) to 53.5 [arxiv:2402.01306 tab. 2]. Both results are author-reported | | Reference model too costly to hold in memory | KTO reference-free variant [arxiv:2402.01306 §4.3] | Avoids a second model, but trails standard KTO, for example GSM8K 47.5 against 53.5 [arxiv:2402.01306 tab. 2] | | Good and bad labels are very unbalanced | KTO with lambda weights set by the ratio rule [arxiv:2402.01306 eq. 9] | The rule is an empirical finding of the authors. They say toxicity-type tasks, where the downside matters more, may favour the reverse weighting [arxiv:2402.01306 §4.2] | | Text model, no supervised reference model | Build the reference from the chosen completions, as the paper does for dialogue [arxiv:2305.18290 §6.2] | The paper reports this for dialogue only; the effect of skipping this step is not measured | | Preference pairs where the rejected side is an unwanted behaviour, inside a larger post-training pipeline | DPO as the final stage after SFT [arxiv:2404.14219 §2] | This is the report's recipe. The card records no measured effect of the DPO stage, and its settings are not stated. The Mixtral instruct model follows the same SFT-then-DPO order [arxiv:2401.04088 §4], also with no stated settings | | Diffusion or video model, feedback can be paired | Diffusion DPO-style loss [arxiv:2412.02617 §3.1] | Paper reports better held-out results than reward-weighted regression, on a small test set [arxiv:2412.02617 tab. 2] | | Per-sample accept or reject labels, and no pairing rule | Reward-weighted regression [arxiv:2412.02617 §3.1] | Needs no pairs. The paper does not state how pairs are built from binary labels for DPO [arxiv:2412.02617 §4.3] | | Multi-step LLM agent, expert trajectories available, no step-level labels, and a reward only at the end of a trajectory | Step-wise implicit reward [arxiv:2411.03817 §4.2.1] | Turns expert-versus-agent actions at the same prefix into pairs, with no human step labels. Reported to beat ETO on all seven benchmarks in the table, though the gaps are small on several [arxiv:2411.03817 tab. 2]. Assumes an SFT base agent and expert data [arxiv:2411.03817 §3.2] | | Reference outputs exist, several automatic metrics score each candidate, and you can afford a judge model at data-build time | Metric-scored pairs after SFT [arxiv:2601.04025 §2.3] | Turns metric averages into pairs with a gap threshold. Reported to beat SFT on most turn-level metrics for Llama 3.1 8B, but not on errors [arxiv:2601.04025 tab. 1]. The judges are proprietary models, and the paper warns that training on judge-scored metrics can overfit the judge [arxiv:2601.04025 §2.3] | | Reusable reward model needed, for example to rank outputs or to run RL on scores | Reward-model RLHF [arxiv:2307.15217 §2] | The survey lists open problems with this route [arxiv:2307.15217 §3] but gives no head-to-head evidence, so this page does not rank it | ## Implementation sketch Most variants share one loss over a batch, with a frozen reference and no sampling from the policy. The pairwise DPO loss below is the reference shape. KTO keeps the batch but drops the pairs, and the diffusion variant keeps the shape and changes the inputs. ```pseudocode # Pairwise DPO loss over a batch [arxiv:2305.18290 §4] # Each log-prob is the summed token log-probability of a full completion def dpo_loss(pol_win, pol_lose, ref_win, ref_lose, beta): # Policy log-ratio, preferred minus rejected [arxiv:2305.18290 §4] pol_margin = pol_win - pol_lose # (B,) # Same margin under the frozen reference; no gradient flows here [§4] ref_margin = ref_win - ref_lose # (B,) # Implicit reward gap, scaled by beta [arxiv:2305.18290 §4] logit = beta * (pol_margin - ref_margin) # (B,) # Negative log-sigmoid, averaged over the batch [arxiv:2305.18290 §4] return -log_sigmoid(logit).mean() ``` Where variants differ: - **KTO.** Drop the pair. Each output's log-ratio r is scored against a reference point z0, the clamped mean of mismatched log-ratios in the microbatch, with no gradient through z0. A desirable output contributes lambda_D times sigmoid(beta (r - z0)), and an undesirable one contributes lambda_U times sigmoid(beta (z0 - r)). The loss is lambda minus that value, averaged [arxiv:2402.01306 §4.1] [arxiv:2402.01306 eq. 8]. The card has full pseudocode. - **Diffusion.** Replace each log-probability with a denoising error on one shared noise draw and time step, then flip the sign of the logit, because a lower error is the better outcome [arxiv:2412.02617 §3.1] [arxiv:2412.02617 eq. 7]. - **Reward-weighted regression.** Drop the pair. Weight each sample's denoising error by its reward [arxiv:2412.02617 eq. 6]. - **Metric-scored pairs.** Keep the pairwise loss unchanged. Only the pair builder differs: score each candidate by the mean of its metrics, then keep ordered pairs whose gap exceeds epsilon (0.1 in the paper). The loss sees only the winner and loser [arxiv:2601.04025 §2.3] [arxiv:2601.04025 app. B.1]. ## Contradictions One disagreement concerns PPO against direct methods. The DPO paper reports DPO matching or beating PPO on sentiment and TL;DR summarisation [arxiv:2305.18290 §6.1] [arxiv:2305.18290 §6.2]. The first survey reports, citing Xu et al. (2024), that tuned PPO can outperform direct methods [arxiv:2312.14925 app. A]. The survey runs no experiments of its own, so its statement is a citation, not a measurement. Both are left as stated. Otherwise, no direct conflict in the core claims. The settings differ, so the results do not directly test each other. The DPO-over-PPO comparison is made only for text [arxiv:2305.18290 §6], and the dialogue result is not a PPO comparison [arxiv:2305.18290 §6.2]. In the diffusion paper, DPO and reward-weighted regression are close on the headline Gemini metric, 56.17% and 56.41% [arxiv:2412.02617 tab. 2], and that paper runs no PPO baseline. The survey gives no numbers for either method [arxiv:2307.15217 §1]. One difference in scope is worth noting. The KTO paper says that up to 7B parameters, alignment gave almost no gain over SFT for the methods it compared [arxiv:2402.01306 §3.3]. The DPO paper reports a GPT-4-judged win rate of about 61% against the reference summaries on TL;DR, with a GPT-J-based 6B SFT model, and reports dialogue results on Pythia-2.8B [arxiv:2305.18290 §6.2]. The tasks, judges and baselines differ, so the two statements are not a direct test of each other. Both are left as the papers state them. The step-wise agent paper reports gains over the best baseline, ETO in every cell, on WebShop, Mind2Web, ScienceWorld, ALFWorld and three multi-hop QA sets [arxiv:2411.03817 tab. 2]. Those numbers are on agent benchmarks with exact-match or reward metrics, not on the response-level win rates above, so they do not confirm or contradict them. ## Papers - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](/papers/2305.18290): the pairwise logistic loss against a frozen reference, with results on sentiment, summarisation and dialogue. - [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](/papers/2307.15217): a survey of problems in reward-model RLHF, naming direct preference training as an alternative. - [A Survey of Reinforcement Learning from Human Feedback](/papers/2312.14925): a survey of RLHF that gives the DPO loss and the Bradley-Terry reward model; no experiments, and the DPO beta is not stated. - [Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025): DPO after SFT with pairs ranked by an average of reference-based metrics, for simulated tutoring students on Llama models. - [Phi-3 Technical Report](/papers/2404.14219): a small language model whose post-training ends with DPO on unwanted-behaviour rejects; the DPO settings are not stated. - [Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](/papers/2412.02617): a diffusion DPO-style loss and reward-weighted regression, trained on accept or reject feedback from a vision-language model. - [KTO: Model Alignment as Prospect Theoretic Optimization](/papers/2402.01306): a binary-label loss with a prospect-theory value and a batch-based reference point, compared with DPO from 1B to 30B parameters. - [Mixtral of Experts](/papers/2401.04088): a sparse mixture-of-experts model whose instruct version is SFT followed by DPO; the DPO settings are not stated. - [From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning](/papers/2411.03817): a step-wise DPO-style loss on expert versus agent actions at each prefix, plus an inverse-RL variant with PPO.