---
kind: concept
slug: bradley-terry-model
title: Bradley-Terry model
summary: >-
  The Bradley-Terry model turns pairwise win, loss or tie outcomes into one latent strength per item. Covers the plain fit, tie handling, a task-aware variant fit by EM for ranking robot policies from real-robot A/B trials, and a reward-model loss trained on preferred clips. Based on three papers so far.
tags: [ranking, pairwise-comparison, evaluation, crowdsourcing, reward-modeling]
papers: ["1811.06521", "2312.14925", "2506.18123"]
related: [expectation-maximization, human-evaluation, vision-language-action-model, reward-modeling, rlhf, learning-from-demonstration, preference-optimization]
status: draft
updated_at: 2026-10-31
---

## Overview

Many items are compared in pairs, and each comparison has a winner, a loser or
a tie. The goal is one number per item that orders them. The shared core is a
latent strength θ for each item, with the probability that A beats B given by
σ(θ_A − θ_B), where σ is the logistic function [arxiv:2506.18123 §3.2]. Only
differences matter, so the fit fixes the overall shift, for example by
centring the strengths at zero.

Three sources use this core in different ways. One fits a reward model to
pairwise preferences over short clips of game play, so the strength becomes a
reward score [arxiv:1811.06521 §2.4]. Another ranks robot policies, and
corrects for the fact that each comparison is on a different task
[arxiv:2506.18123 §3.2]. A survey of reinforcement learning from human feedback
gives the same likelihood for reward learning from trajectory comparisons, and
treats the model as the two-item case of a wider choice model
[arxiv:2312.14925 §2.3] [arxiv:2312.14925 §5.1.1] [arxiv:2312.14925 eq. 2]. The variants also
differ in how they treat ties.

## Variants

### Plain Bradley-Terry fit

The basic model with no task terms. The paper that ranks robot policies uses it
as a comparison point, under the name "standard Bradley-Terry", and it ranks
below its task-aware fit [arxiv:2506.18123 §5.2]. Its appendix fits this
baseline by gradient ascent and counts a tie as half a win for each side
[arxiv:2506.18123 app. B.4]. The public code fits the same model with Davidson
ties and Newton's method instead; see the card's Paper vs code section.
Card: [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](/papers/2506.18123).

### Plain Bradley-Terry with Davidson ties

A tie is given its own probability, scaled by a tie rate, rather than being
counted as half a win. The paper's task-aware model uses this Davidson-style
tie term [arxiv:2506.18123 app. B.1]. The public code for the same paper fits
a plain Bradley-Terry with Davidson ties in place of the task-aware model, as
noted in the card's Paper vs code section (see Contradictions). The paper does
not present this plain form as a separate baseline, so its effect on rankings
is not reported on its own.
Card: [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](/papers/2506.18123).

### Task-aware Bradley-Terry with latent buckets

The paper's proposed variant. Each trial is given a hidden task bucket t out of
T, with a prior weight and a difficulty. Each policy also gets an offset for
each bucket. A policy solves a task with a probability set by its strength,
its offset and the bucket difficulty, and the trial outcome follows from the
two solve probabilities [arxiv:2506.18123 §3.2] [arxiv:2506.18123 eq. 1]
[arxiv:2506.18123 app. B.1]. This lets the same ranking be fitted across trials
on different tasks, without requiring the tasks to match.

The parameters are fit by approximate maximum likelihood with EM. The paper
reports 60 buckets and 60 EM iterations as defaults [arxiv:2506.18123 tab. 1].
Card: [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](/papers/2506.18123).

### Reward-model Bradley-Terry loss on clip pairs

A reward model is fitted to human or synthetic preferences between two short
clips. Each clip's score is the sum of the model's per-step rewards over the
clip. The probability that the first clip is preferred is a softmax over the
two sums, which is the same as the logistic form σ(S_1 − S_2) for two scores.
Training minimises the cross-entropy against the annotator's label. A label is
(1, 0), (0, 1) or (0.5, 0.5) for "equal". Pairs the annotator marks as
incomparable are dropped rather than given the 0.5 label [arxiv:1811.06521 §2.4].
The survey states the same maximum-likelihood objective over pairwise
trajectory comparisons, equivalent to a cross-entropy loss [arxiv:2312.14925 §2.3] [arxiv:2312.14925 eq. 2].

The paper's loss also smooths the predicted probability with a 10% chance of a
uniformly random label [arxiv:1811.06521 app. A.3]. This is an assumption about
the annotators, not a measured rate. The survey describes a similar error
model, a 90% Boltzmann choice and otherwise random, attributed to Christiano et
al. [arxiv:2312.14925 §5.1.2]. The reward model learns only
differences between clips, so the paper re-normalises it to zero mean and a
fixed standard deviation, and says the value was chosen empirically
[arxiv:1811.06521 §2.4].
Card: [Reward learning from human preferences and demonstrations in Atari](/papers/1811.06521).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Comparisons are on different tasks or scenes that cannot be matched | Task-aware fit with latent buckets | The paper reports it ranks closest to an exhaustive oracle, ahead of the plain fit and Elo [arxiv:2506.18123 §5.2] [fig. 6]. Its per-trial task buckets are the mechanism for this case [arxiv:2506.18123 §3.2]. |
| Comparisons are on one shared task set, or tasks are matched | Plain fit | The task model is there to handle task effects. Where tasks are matched, the paper's argument for it does not apply [arxiv:2506.18123 §3.2]. The paper does not test this case directly. |
| Need a quick ordering with little implementation work | Plain fit or Elo | Simpler to implement. The paper's comparisons show both below the task-aware fit on oracle agreement, so expect a less reliable ranking [arxiv:2506.18123 §5.2]. |
| Preference labels and progress scores both exist | Report both | The paper finds progress-based and preference-based rankings complementary, and says both should be reported [arxiv:2506.18123 §5.2]. |
| Need a reward function from pairwise judgements on trajectory clips, with no reward signal | Reward-model loss on clip pairs | The paper fits this to 25-step Atari clip pairs and trains an RL agent on the result [arxiv:1811.06521 §2.4]. Most of its labels are synthetic, taken from the game score [arxiv:1811.06521 §3]. Its reported comparison with imitation learning is 7 of 9 games in the abstract, and the body gives 8 of 9 [arxiv:1811.06521 §3.1]. |
| Labels may be noisy, for example from human annotators | Reward-model loss with a noise term | The paper reports 5 to 10% mislabelling from human labellers [arxiv:1811.06521 §3.1] and trains with a 10% noise term [arxiv:1811.06521 app. A.3]. |
| Labeller reliability is unknown and no calibration reward exists | Bradley-Terry with the standard coefficient β = 1, no tuning | The survey says β is not identifiable without a calibration reward, and that β = 1 is the standard Bradley-Terry setting [arxiv:2312.14925 §5.1.2] [arxiv:2312.14925 eq. 10]. Any noise rate must then be assumed, as the Atari paper does [arxiv:1811.06521 app. A.3]. |
| Preferences may be intransitive | Consider a model without the transitivity assumption | Bradley-Terry imposes transitive preferences, which human judgements may violate. The survey points to Nash-equilibrium theory as one response [arxiv:2312.14925 §7.3]. This page gives no method for that case. |
| Want to train a policy directly from preference pairs, with no separate reward model | Direct preference loss, see [preference-optimization](/concepts/preference-optimization) | DPO's loss has the same logistic form, applied to log-ratios against a frozen reference policy [arxiv:2312.14925 §6.3] [arxiv:2312.14925 eq. 14]. The survey does not give its β or the reference policy's value. |
| Few comparisons (fewer than about 100) | Treat with care | The paper reports the ranking stabilising after about 100 comparisons, in its setup [arxiv:2506.18123 §5.3]. This page has no evidence for smaller samples. |
| Need confidence intervals | Derive them first | The paper gives no standard errors for the task-aware fit [arxiv:2506.18123 §3.2]. |

The robot-policy evidence comes from one setting: a pool of seven generalist
DROID policies [arxiv:2506.18123 §5.1]. Whether the ordering holds in other
settings is not tested [arxiv:2506.18123 §7]. The reward-model evidence comes
from Atari games with mostly synthetic labels [arxiv:1811.06521 §3]. The
survey reports no results on the benchmarks it names [arxiv:2312.14925 §8.3] [arxiv:2312.14925 §8.4].

## Implementation sketch

Plain fit by gradient ascent. The card records no step size or step count for
this baseline, so both are left as parameters here. The tie rule (half a win
for each side) is the paper's rule for its baseline [arxiv:2506.18123 app. B.4].

```pseudocode
# Plain Bradley-Terry fit. Our sketch, not the paper's baseline code.
# Strength model: P(a beats b) = sigmoid(theta[a] - theta[b])
#   [arxiv:2506.18123 §3.2]
# trials: list of (a, b, y) with y = 1 for a wins, 0 for b wins, 0.5 for a tie

theta = zeros(num_items)                  # latent strength per item
for step in range(num_steps):             # step size and count: our choice
    grad = zeros(num_items)
    for (a, b, y) in trials:
        p = sigmoid(theta[a] - theta[b])  # P(a beats b)
        grad[a] += y - p                  # log-likelihood gradient for a
        grad[b] -= y - p                  # and the opposite sign for b
    theta = theta + lr * grad
    theta = theta - mean(theta)           # only differences identify theta
ranking = argsort(-theta)                 # best item first
```

Task-aware variant: alternate an E-step and an M-step, following the paper's
Algorithm 2 [arxiv:2506.18123 alg. 2].

- E-step: for each trial, compute the posterior over its task bucket, using
  the prior and the likelihood of the observed outcome under each bucket.
- M-step: take a clipped Newton step on the strengths, the bucket offsets and
  the bucket difficulties, centring after each one. Then update the bucket
  prior and the tie rate.
- Stop when the largest change in strength falls below a tolerance. The paper's
  default is 1e-4 [arxiv:2506.18123 tab. 1].

The paper's Newton updates rely on closed-form gradients and Hessians given in
its appendix [arxiv:2506.18123 app. B.2]. The paper gives a clipping bound and
a decay schedule, and says the decay is in its hyperparameter table but not in
Algorithm 2 [arxiv:2506.18123 app. B.5]. EM can stop at a local optimum, and
the paper gives no restart count or seed count [arxiv:2506.18123 tab. 1]. Use
several initialisations and compare the rankings.

Two points that are easy to get wrong in the task-aware fit:

- The initial spread of the strengths is written as N(0, 0.1), but the paper
  does not say whether 0.1 is a variance or a standard deviation
  [arxiv:2506.18123 alg. 2].
- The paper says the parameters are centred to zero mean, but does not say over
  which axis for the bucket offsets [arxiv:2506.18123 §3.2].

Reward-model loss: the cross-entropy for one labelled pair, following the
paper's description [arxiv:1811.06521 §2.4] [arxiv:1811.06521 app. A.3]:

```pseudocode
# Cross-entropy loss for one labelled clip pair. Our rendering.
def pair_loss(reward_model, clip_1, clip_2, label):
    # Sum of predicted per-step rewards over each clip [arxiv:1811.06521 §2.4]
    score_1 = sum(reward_model(obs) for obs in clip_1)
    score_2 = sum(reward_model(obs) for obs in clip_2)
    # Probability that clip 1 is preferred, as a softmax over the two sums
    p_1 = exp(score_1) / (exp(score_1) + exp(score_2))
    # Assumed 10% uniform-random label noise [arxiv:1811.06521 app. A.3]
    p_1 = 0.9 * p_1 + 0.05
    # Label is (1, 0), (0, 1) or (0.5, 0.5) for equal [arxiv:1811.06521 §2.4]
    return -(label[0] * log(p_1) + label[1] * log(1 - p_1))
```

Two points for the reward-model loss:

- Re-normalising the reward changes the effective learning rate of the policy
  trained on it. The paper gives the standard deviation as 0.05, chosen
  empirically [arxiv:1811.06521 §2.4].
- The paper does not store reward in the replay buffer. The reward model is
  evaluated on each sampled batch [arxiv:1811.06521 §2.3] [arxiv:1811.06521 app. A.3].
- The sketch exponentiates the two sums directly, which overflows for large
  scores. The survey advises the stable softplus form of the negative
  log-likelihood instead [arxiv:2312.14925 eq. 2].

The published strength is a log-ability with no fixed unit. Any score shown to
readers needs a stated transform, and the paper does not define one for this
model [arxiv:2506.18123 §3.2].

## Contradictions

Between papers: none known. The two papers apply the same logistic core to
different data, so their numbers are not directly comparable. The survey
reports no experiments, so it adds no numbers to compare.

Within the Atari paper, the abstract says the method beats imitation learning in
7 of 9 games, and the body says 8 of 9 [arxiv:1811.06521 §3.1]. This page
treats 7 as the conservative figure, as the card does.

Between the robot-policy paper and its public code, the paper card records these
differences (read from the code, not run):

- The paper's default is the task-aware EM fit [arxiv:2506.18123 §3.2]. The
  public leaderboard code has the EM routine disabled and fits the plain model
  with Davidson ties instead. See [the paper card](/papers/2506.18123), "Paper
  vs code".
- The bucket count differs: 60 in the paper's default, 100 in the code.
- The code's gradients for strength use the opponent's solve probability, while
  the appendix gives the own-policy form. The card notes a finite-difference check
  would settle this, and that it affects only the EM model.

## Papers

- [Reward learning from human preferences and demonstrations in Atari](/papers/1811.06521): fits a reward model to pairwise preferences over 25-step clips with a Bradley-Terry style cross-entropy loss, and trains an RL agent on the learned reward [arxiv:1811.06521 §2.4].
- [A Survey of Reinforcement Learning from Human Feedback](/papers/2312.14925): surveys reward learning from pairwise comparisons under the Bradley-Terry likelihood, the Boltzmann choice model it reduces to, and direct methods such as DPO; reports no experiments [arxiv:2312.14925 §2.3] [§5.1].
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](/papers/2506.18123): ranks generalist robot policies from crowd-sourced, double-blind A/B trials using the task-aware Bradley-Terry fit with latent task buckets and EM [arxiv:2506.18123 §3.2].