Bradley-Terry model

Overview

Many items are compared in pairs, and each comparison has a winner, a loser or a tie. The goal is one number per item that orders them. The shared core is a latent strength θ for each item, with the probability that A beats B given by σ(θ_A − θ_B), where σ is the logistic function 2506.18123 §3.2. Only differences matter, so the fit fixes the overall shift, for example by centring the strengths at zero.

Three sources use this core in different ways. One fits a reward model to pairwise preferences over short clips of game play, so the strength becomes a reward score 1811.06521 §2.4. Another ranks robot policies, and corrects for the fact that each comparison is on a different task 2506.18123 §3.2. A survey of reinforcement learning from human feedback gives the same likelihood for reward learning from trajectory comparisons, and treats the model as the two-item case of a wider choice model 2312.14925 §2.3 2312.14925 §5.1.1 2312.14925 eq. 2. The variants also differ in how they treat ties.

Variants

Plain Bradley-Terry fit

The basic model with no task terms. The paper that ranks robot policies uses it as a comparison point, under the name "standard Bradley-Terry", and it ranks below its task-aware fit 2506.18123 §5.2. Its appendix fits this baseline by gradient ascent and counts a tie as half a win for each side 2506.18123 app. B.4. The public code fits the same model with Davidson ties and Newton's method instead; see the card's Paper vs code section. Card: RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies.

Plain Bradley-Terry with Davidson ties

A tie is given its own probability, scaled by a tie rate, rather than being counted as half a win. The paper's task-aware model uses this Davidson-style tie term 2506.18123 app. B.1. The public code for the same paper fits a plain Bradley-Terry with Davidson ties in place of the task-aware model, as noted in the card's Paper vs code section (see Contradictions). The paper does not present this plain form as a separate baseline, so its effect on rankings is not reported on its own. Card: RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies.

Task-aware Bradley-Terry with latent buckets

The paper's proposed variant. Each trial is given a hidden task bucket t out of T, with a prior weight and a difficulty. Each policy also gets an offset for each bucket. A policy solves a task with a probability set by its strength, its offset and the bucket difficulty, and the trial outcome follows from the two solve probabilities 2506.18123 §3.2 2506.18123 eq. 1 2506.18123 app. B.1. This lets the same ranking be fitted across trials on different tasks, without requiring the tasks to match.

The parameters are fit by approximate maximum likelihood with EM. The paper reports 60 buckets and 60 EM iterations as defaults 2506.18123 tab. 1. Card: RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies.

Reward-model Bradley-Terry loss on clip pairs

A reward model is fitted to human or synthetic preferences between two short clips. Each clip's score is the sum of the model's per-step rewards over the clip. The probability that the first clip is preferred is a softmax over the two sums, which is the same as the logistic form σ(S_1 − S_2) for two scores. Training minimises the cross-entropy against the annotator's label. A label is (1, 0), (0, 1) or (0.5, 0.5) for "equal". Pairs the annotator marks as incomparable are dropped rather than given the 0.5 label 1811.06521 §2.4. The survey states the same maximum-likelihood objective over pairwise trajectory comparisons, equivalent to a cross-entropy loss 2312.14925 §2.3 2312.14925 eq. 2.

The paper's loss also smooths the predicted probability with a 10% chance of a uniformly random label 1811.06521 app. A.3. This is an assumption about the annotators, not a measured rate. The survey describes a similar error model, a 90% Boltzmann choice and otherwise random, attributed to Christiano et al. 2312.14925 §5.1.2. The reward model learns only differences between clips, so the paper re-normalises it to zero mean and a fixed standard deviation, and says the value was chosen empirically 1811.06521 §2.4. Card: Reward learning from human preferences and demonstrations in Atari.

When to use which

The robot-policy evidence comes from one setting: a pool of seven generalist DROID policies 2506.18123 §5.1. Whether the ordering holds in other settings is not tested 2506.18123 §7. The reward-model evidence comes from Atari games with mostly synthetic labels 1811.06521 §3. The survey reports no results on the benchmarks it names 2312.14925 §8.3 2312.14925 §8.4.

Implementation sketch

Plain fit by gradient ascent. The card records no step size or step count for this baseline, so both are left as parameters here. The tie rule (half a win for each side) is the paper's rule for its baseline 2506.18123 app. B.4.

# Plain Bradley-Terry fit. Our sketch, not the paper's baseline code.
# Strength model: P(a beats b) = sigmoid(theta[a] - theta[b])
#   [arxiv:2506.18123 §3.2]
# trials: list of (a, b, y) with y = 1 for a wins, 0 for b wins, 0.5 for a tie

theta = zeros(num_items)                  # latent strength per item
for step in range(num_steps):             # step size and count: our choice
    grad = zeros(num_items)
    for (a, b, y) in trials:
        p = sigmoid(theta[a] - theta[b])  # P(a beats b)
        grad[a] += y - p                  # log-likelihood gradient for a
        grad[b] -= y - p                  # and the opposite sign for b
    theta = theta + lr * grad
    theta = theta - mean(theta)           # only differences identify theta
ranking = argsort(-theta)                 # best item first

Task-aware variant: alternate an E-step and an M-step, following the paper's Algorithm 2 2506.18123 alg. 2.

• E-step: for each trial, compute the posterior over its task bucket, using the prior and the likelihood of the observed outcome under each bucket.
• M-step: take a clipped Newton step on the strengths, the bucket offsets and the bucket difficulties, centring after each one. Then update the bucket prior and the tie rate.
• Stop when the largest change in strength falls below a tolerance. The paper's default is 1e-4 2506.18123 tab. 1.

The paper's Newton updates rely on closed-form gradients and Hessians given in its appendix 2506.18123 app. B.2. The paper gives a clipping bound and a decay schedule, and says the decay is in its hyperparameter table but not in Algorithm 2 2506.18123 app. B.5. EM can stop at a local optimum, and the paper gives no restart count or seed count 2506.18123 tab. 1. Use several initialisations and compare the rankings.

Two points that are easy to get wrong in the task-aware fit:

• The initial spread of the strengths is written as N(0, 0.1), but the paper does not say whether 0.1 is a variance or a standard deviation 2506.18123 alg. 2.
• The paper says the parameters are centred to zero mean, but does not say over which axis for the bucket offsets 2506.18123 §3.2.

Reward-model loss: the cross-entropy for one labelled pair, following the paper's description 1811.06521 §2.4 1811.06521 app. A.3:

# Cross-entropy loss for one labelled clip pair. Our rendering.
def pair_loss(reward_model, clip_1, clip_2, label):
    # Sum of predicted per-step rewards over each clip [arxiv:1811.06521 §2.4]
    score_1 = sum(reward_model(obs) for obs in clip_1)
    score_2 = sum(reward_model(obs) for obs in clip_2)
    # Probability that clip 1 is preferred, as a softmax over the two sums
    p_1 = exp(score_1) / (exp(score_1) + exp(score_2))
    # Assumed 10% uniform-random label noise [arxiv:1811.06521 app. A.3]
    p_1 = 0.9 * p_1 + 0.05
    # Label is (1, 0), (0, 1) or (0.5, 0.5) for equal [arxiv:1811.06521 §2.4]
    return -(label[0] * log(p_1) + label[1] * log(1 - p_1))

Two points for the reward-model loss:

• Re-normalising the reward changes the effective learning rate of the policy trained on it. The paper gives the standard deviation as 0.05, chosen empirically 1811.06521 §2.4.
• The paper does not store reward in the replay buffer. The reward model is evaluated on each sampled batch 1811.06521 §2.3 1811.06521 app. A.3.
• The sketch exponentiates the two sums directly, which overflows for large scores. The survey advises the stable softplus form of the negative log-likelihood instead 2312.14925 eq. 2.

The published strength is a log-ability with no fixed unit. Any score shown to readers needs a stated transform, and the paper does not define one for this model 2506.18123 §3.2.

Contradictions

Between papers: none known. The two papers apply the same logistic core to different data, so their numbers are not directly comparable. The survey reports no experiments, so it adds no numbers to compare.

Within the Atari paper, the abstract says the method beats imitation learning in 7 of 9 games, and the body says 8 of 9 1811.06521 §3.1. This page treats 7 as the conservative figure, as the card does.

Between the robot-policy paper and its public code, the paper card records these differences (read from the code, not run):

• The paper's default is the task-aware EM fit 2506.18123 §3.2. The public leaderboard code has the EM routine disabled and fits the plain model with Davidson ties instead. See the paper card, "Paper vs code".
• The bucket count differs: 60 in the paper's default, 100 in the code.
• The code's gradients for strength use the opponent's solve probability, while the appendix gives the own-policy form. The card notes a finite-difference check would settle this, and that it affects only the EM model.

Papers

• Reward learning from human preferences and demonstrations in Atari: fits a reward model to pairwise preferences over 25-step clips with a Bradley-Terry style cross-entropy loss, and trains an RL agent on the learned reward 1811.06521 §2.4.
• A Survey of Reinforcement Learning from Human Feedback: surveys reward learning from pairwise comparisons under the Bradley-Terry likelihood, the Boltzmann choice model it reduces to, and direct methods such as DPO; reports no experiments 2312.14925 §2.3 §5.1.
• RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies: ranks generalist robot policies from crowd-sourced, double-blind A/B trials using the task-aware Bradley-Terry fit with latent task buckets and EM 2506.18123 §3.2.