Proper scoring rules

Overview

A scoring rule grades a predicted probability vector s against the outcome that happened. It is proper when the expected score is lowest if s is the true distribution of the outcome, so a model cannot improve its score by hedging or by overstating its confidence 2112.10327 §3. The survey uses lower-is-better losses. Some authors use the opposite sign, and a payoff can always be negated 2112.10327 §3.

The shared idea is that a test-set score is the average of the per-instance values, so one number summarises how good the whole probability output is 2112.10327 §3. Two rules are strictly proper and widely used: the Brier score and the log-loss 2112.10327 §3.

Variants

Brier score

The squared error between the predicted vector and the one-hot true class, summed over the K classes 2112.10327 §3, eq. 1. It stays finite for any probability vector, so no clipping is needed. Card: Classifier calibration survey.

Log-loss

The negative log of the probability given to the true class [arxiv:2112.10327 §3, eq. 2]. It punishes confident mistakes hard. It is infinite when the true class gets probability 0, so in practice the probability is clipped before the log is taken 2112.10327 §3. It is also the objective used to fit most of the post-hoc maps in the same survey 2112.10327 §5.3, §5.4.

Decomposition into calibration and refinement

Either rule splits its expected value into two terms. The calibration loss is the gap between the predictions and the conditional class frequencies. The refinement loss is what remains: how well the predictions separate the outcomes 2112.10327 §3.1. The same split can be written with the true posterior in place of the predictions, giving an epistemic part (model error) and an irreducible part 2112.10327 §3.1. The survey's worked example on 10 instances gives Brier 0.71 = calibration 0.19 + refinement 0.52, and log-loss 1.12 = 0.29 + 0.83 2112.10327 §3.1. This is an author-reported toy example, not a benchmark.

When to use which

Implementation sketch

Both rules take the predicted vector and the one-hot label for one instance. The test-set score is the mean over instances.

# Brier score for one instance  [arxiv:2112.10327 §3, eq. 1]
# probs: (K,) predicted vector; onehot: (K,) true class indicator

def brier(probs, onehot):
    # Squared error over all classes, summed  [§3, eq. 1]
    return sum((probs - onehot) ** 2)

# Log-loss for one instance  [arxiv:2112.10327 §3, eq. 2]
# probs: (K,) predicted vector; onehot: (K,) true class indicator

def log_loss(probs, onehot, eps=1e-12):
    # Clip so a zero probability on the true class stays finite  [§3]
    p = clip(probs, eps, 1.0)
    return -sum(onehot * log(p))

# Test-set score is the mean of per-instance values  [arxiv:2112.10327 §3]
# probs: (N, K); onehot: (N, K)

def test_score(rule, probs, onehot):
    values = [rule(probs[i], onehot[i]) for i in range(len(probs))]
    return mean(values)

Where the survey says the Brier score is summed over classes, use that form when you compare with its numbers. Averaging over classes divides the value by K 2112.10327 §3, eq. 1.

Contradictions

None known. The survey is the only source on this page. It notes that some authors flip the sign of these rules, which is a convention, not a conflict 2112.10327 §3.

Pitfalls

• Check the sign and the normalisation. Losses and payoffs differ by sign, and the Brier score is a sum over classes in the survey's notation 2112.10327 §3, eq. 1.
• Clip before log-loss. It is infinite at zero probability on the true class 2112.10327 §3.
• Do not read a low score as good calibration alone. A score mixes calibration and refinement. Use the decomposition or a reliability diagram to separate them 2112.10327 §3.1, §4.1.
• Library versions may differ from the paper. Check how a library averages or sums before comparing numbers. The survey's companion code is one example (see its card: Classifier calibration survey).

Papers

• Classifier calibration survey: defines the proper scoring rules and their calibration and refinement decomposition, and connects them to calibration maps and ECE.