Direct Preference Optimization: Your Language Model is Secretly a Reward Model

In plain words

A language model trained on raw text can finish sentences but does not know which answers people like. The usual fix has three stages: teach it to follow instructions, train a separate judge (a reward model) on people's choices between two answers, then use reinforcement learning to push the model toward answers the judge scores highly. The second and third stages are fiddly and can become unstable.

This paper shows that the same goal can be reached in one stage. You show the model pairs of answers marked "preferred" and "not preferred", and adjust it so the preferred answer becomes relatively more likely than the rejected one. A frozen copy of the starting model keeps the adjusted model from drifting too far from how it already writes. The math behind this gives a simple two-answer classification loss, so the training loop is ordinary supervised training, with no sampling from the model during training.

The idea is useful if you want to align a model with preferences and do not want to run an RL pipeline. The trade-off is that the method depends on the quality of the preference pairs and on a strength setting (beta) that controls how far the model may move from the reference.

TL;DR

DPO replaces RLHF's reward-model-plus-PPO stage with one loss on preference pairs, computed against a frozen reference model. The loss is derived to optimise the same KL-constrained reward objective that RLHF uses §4 eq. 3 eq. 7. The authors report results equal to or better than PPO-based RLHF on the three tasks they test, at up to 6B parameters, with little hyperparameter tuning §6. The code is public and Apache-2.0 licensed.

Problem

Standard RLHF first fits a reward model to pairwise preferences under the Bradley-Terry model §3 eq. 1 eq. 2, then maximises that reward with RL while keeping the policy close to a reference model via a KL penalty §3 eq. 3. The authors state that this pipeline is complex and often unstable, and that it needs several models and sampling from the policy inside the training loop §1 §3. The method assumes preference data for prompts x with a preferred completion y_w and a dispreferred completion y_l, labelled under a Bradley-Terry or Plackett-Luce model §3. It assumes a reference policy pi_ref, normally the SFT model; if no SFT model exists, pi_ref is obtained by maximising likelihood of the preferred completions §4.

Method

Key idea. The KL-constrained reward maximisation problem has the closed-form optimum pi_r(y|x) = (1/Z(x)) pi_ref(y|x) exp(r(x,y)/beta) §4 eq. 4 app. A.1. Rearranging gives the reward as beta times the log-ratio of the policy to the reference, plus a term that depends only on x §4 eq. 5. Substituting this into the Bradley-Terry model makes the intractable partition function Z(x) cancel out §4 eq. 6 app. A.2. The result is a loss on the policy alone §4 eq. 7.

Loss. For each triple (x, y_w, y_l) in the preference data D, with pi_theta the model being trained and pi_ref frozen §4 eq. 7:

# DPO loss for a batch of preference pairs  [eq. 7]  [code: trainers.py#L45-L87]
# Inputs, each shape (B,): summed token log-probs of the chosen (y_w)
# and rejected (y_l) completion under the policy and the reference
def dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta):
    # Log-ratio of chosen to rejected under the policy  [eq. 7]
    policy_logratio = policy_chosen - policy_rejected    # (B,)
    # The same log-ratio under the frozen reference  [eq. 7]
    ref_logratio = ref_chosen - ref_rejected             # (B,)
    # Implicit reward margin, scaled by beta  [eq. 7] [§4]
    margin = beta * (policy_logratio - ref_logratio)     # (B,)
    # Mean of the negative log-sigmoid of the margin  [eq. 7]
    loss = -log_sigmoid(margin).mean()
    return loss

Gradient. The gradient increases the likelihood of y_w and decreases that of y_l, and each pair is weighted by sigmoid(r_hat(x, y_l) - r_hat(x, y_w)), where r_hat = beta times the policy-to-reference log-ratio. A pair the implicit reward already orders correctly gets a small weight; a mis-ordered pair gets a large one §4. The derivation is in app. A.4.

Procedure (DPO outline). §4

1. Sample completions y_1, y_2 from pi_ref for each prompt, and label them with preferences. Public preference datasets may be used instead §4.
2. Set pi_ref to the SFT model when one is available. Otherwise set pi_ref to the likelihood-maximising model described above §4.
3. Minimise the DPO loss over D with the chosen beta. No sampling from the policy is part of the training loop §4 trainers.py.

Theory. Two equivalent reward functions differ by a function of x only and induce the same optimal policy §5 app. A.5. The authors show that every reward class consistent with Plackett-Luce (and so Bradley-Terry) can be written in the DPO form for some policy, and that this representation is unique §5 app. A.6. They also argue that PPO's reward has an extra normalising term that is a soft value of pi_ref. The DPO reparameterisation avoids that term and so needs no baseline §5.2. Plackett-Luce rankings over K answers have a DPO form as well app. A.3; the repository implements only the pairwise case.

Hyperparameters

Results

Author-reported. Each setting is described in §6; numbers are read from the text or tables, and figure values are not exact.

Other reported numbers: 22 runs in the sentiment sweep §6.1. Sample GPT-4 judgments are in app. D.2. The human study used 25 raters and 275 DPO-vs-PPO judgments, with 200 PPO-vs-PPO and 125 SFT judgments, and human-human agreement of 65% to 87% depending on the comparison app. D.3 tab. 2.

Paper vs code

The paper read here is the arXiv v3 LaTeX source. The official code is eric-mitchell/direct-preference-optimization (author Eric Mitchell, Apache-2.0), cloned at f8b8c0f49dc92a430bae41585f9d467d3618fe2f. Its README and the paper's links do not list other official repositories for this paper. The repository does not contain the sentiment or TL;DR experiments. Its loaders cover Anthropic-HH, Stanford Human Preferences and StackExchange README.md preference_datasets.py.

Discrepancies:

• Learning rate. Paper: 1e-6 app. B. Code default: 5e-7 config.yaml. README commands do not override it README.md. A run that follows the README gets 5e-7 unless it passes lr explicitly.
• Batch size. Paper: 64 app. B. Code default: 4 config.yaml. README DPO examples use batch_size=64 or batch_size=32 with gradient_accumulation_steps=2. In the code, batch_size is the number of examples per optimiser step; gradient accumulation splits that batch into micro-batches trainers.py. So the README comment "effective batch size 64" for the batch_size=32 run is wrong under the code's semantics.
• Warmup. Paper: linear from 0 to 1e-6 over 150 steps app. B. Code: multiplier min(1, (step+1)/(warmup+1)), so the first update already uses 1/151 of the learning rate and full rate is reached at step 150 trainers.py.
• Beta for TL;DR. Paper: 0.5 app. B. The repo's loss config leaves beta as a required field dpo.yaml. The per-task value must be set on the command line.
• Sampling. The paper evaluates at temperatures 0 to 1 §6.2. The repo's sampler always calls do_sample=True and sets no temperature trainers.py. The temperature sweep and the greedy temperature-0 setting are not in the repo.
• Sampling during training. The paper says DPO avoids sampling during fine-tuning §1 §4. That holds for the update loop trainers.py. The evaluation step does sample from the policy and reference when sample_during_eval is on, which is the default trainers.py config.yaml.
• Reference model. The code computes reference log-probs under torch.no_grad() with the reference in eval mode trainers.py trainers.py. This matches the frozen pi_ref in the paper.
• Loss. The code's default loss equals the paper's Eq. 7 when label_smoothing=0 trainers.py. The comment in the code points to Eq. 7 of this paper.
• Log-probability aggregation. The DPO path sums token log-probs over each completion, not averages trainers.py trainers.py. This matches the sequence-level log-ratio in the paper. The code has an average_log_prob option but does not use it for DPO.

Features in the code that the paper does not describe: conservative DPO via label_smoothing trainers.py dpo.yaml, a reference_free option that replaces the reference term with 0 trainers.py, and IPO trainers.py. All are off by default.

Not verified in the code: the sentiment setup, the GPT-4 judging and the human study. The repo has no code for them, so their figures and tables rest on the paper's text.

Pitfalls

• Learning-rate and batch-size defaults do not match the paper (see Paper vs code). Set lr=1e-6 and batch_size=64 explicitly to reproduce the paper's optimisation settings. The README's batch-size comment is inconsistent with the code.
• Beta is task-dependent and not tuned by the authors. The paper uses 0.1 for most runs and 0.5 for TL;DR, and says it did not tune beta there §6.2 app. B. The sentiment sweep covered 0.05 to 5 §6.1. Beta scales the implicit reward and so the strength of the KL constraint §4: a larger beta keeps the policy closer to pi_ref, a smaller one lets it move further.
• Summed log-probs make the margin depend on length. The loss uses whole-completion log-probs, so longer completions contribute larger terms. The paper does not discuss length normalisation or length bias.
• Likelihood-minimisation baselines degenerate. The paper reports that the unlikelihood baseline without the DPO weighting produced meaningless text on summarisation and dialogue, so it dropped it from those tasks app. C.3 tab. 3. The DPO weighting is what keeps it stable §4.
• The reference must match the data distribution. The authors say that preference data comes from the SFT model where possible, and for dialogue they train a Preferred-FT model first so that the completions are in-distribution §4 §6.2. The paper does not measure what happens without this step.
• Prompt format is a hidden detail. For the Anthropic-HH and SHP loaders, prompts are wrapped as \n\nHuman: ... \n\nAssistant: preference_datasets.py preference_datasets.py. The paper does not state a prompt template.
• Length limits and clipping. The paper does not state a maximum length, gradient clipping or epoch count. The repo uses 512 total tokens, 256 prompt tokens, clipping at 10.0 and one epoch config.yaml. Replies are cut at the maximum length during sampling trainers.py.
• Evaluation is GPT-4 based. The win rates depend on the judge prompt. The authors note that GPT-4 prefers longer, more repetitive summaries than humans do under the simple prompt, which is why they use the concise prompt §6.4. They also state that win rates are sensitive to the prompt §7 app. C.2. Judge model is gpt-4-0314 app. C.2.
• Human study is small. 25 volunteers, mostly Stanford students and graduates app. D.3. Human-human agreement on the DPO comparison is 65% tab. 2.
• Out-of-distribution result is a single experiment. The CNN/DailyMail test uses the TL;DR policies and temperatures 0 and 0.25 only §6.3 tab. 1. The authors call it initial evidence.
• Compute cost. The paper's comparison is about simplicity: DPO needs no reward model and no RL loop. The authors' runs used 4 A100 80GB GPUs with FSDP for SFT and DPO, and the README reports about 1.5 hours for SFT and 2.75 hours for DPO on Pythia-2.8B README.md. The paper gives no GPU-hours.
• Scale. Experiments reach 6B parameters. Larger models are left to future work §7.

Lineage

• Builds on: the RLHF pipeline (reward modelling, then RL fine-tuning) §3, the Bradley-Terry and Plackett-Luce preference models §3 app. A.3, and the KL-constrained reward objective §3 eq. 3.
• Compared against: PPO-based RLHF, PPO with ground-truth reward (PPO-GT), Preferred-FT, unlikelihood training and Best of N sampling §6.
• Related work named by the authors: contextual dueling bandits and preference-based RL, which learn from preference data §2.