---
kind: concept
slug: proper-scoring-rules
title: Proper scoring rules
summary: >-
  A loss on a predicted probability vector that is lowest when the model reports the true probabilities. Brier score and log-loss are the two standard choices. Their expected value splits into a calibration part and a refinement part, which tells you whether to recalibrate or to improve the model.
tags: [calibration, uncertainty-estimation, evaluation, multiclass-classification]
papers: ["2112.10327"]
related: [calibration, expected-calibration-error, temperature-scaling]
status: draft
updated_at: 2026-11-08
---

## Overview

A scoring rule grades a predicted probability vector s against the outcome
that happened. It is proper when the expected score is lowest if s is the
true distribution of the outcome, so a model cannot improve its score by
hedging or by overstating its confidence [arxiv:2112.10327 §3]. The survey
uses lower-is-better losses. Some authors use the opposite sign, and a payoff
can always be negated [arxiv:2112.10327 §3].

The shared idea is that a test-set score is the average of the per-instance
values, so one number summarises how good the whole probability output is
[arxiv:2112.10327 §3]. Two rules are strictly proper and widely used: the
Brier score and the log-loss [arxiv:2112.10327 §3].

## Variants

### Brier score

The squared error between the predicted vector and the one-hot true class,
summed over the K classes [arxiv:2112.10327 §3, eq. 1]. It stays finite for any
probability vector, so no clipping is needed. Card:
[Classifier calibration survey](/papers/2112.10327).

### Log-loss

The negative log of the probability given to the true class [arxiv:2112.10327
§3, eq. 2]. It punishes confident mistakes hard. It is infinite when the true
class gets probability 0, so in practice the probability is clipped before the
log is taken [arxiv:2112.10327 §3]. It is also the objective used to fit most
of the post-hoc maps in the same survey [arxiv:2112.10327 §5.3, §5.4].

### Decomposition into calibration and refinement

Either rule splits its expected value into two terms. The calibration loss is
the gap between the predictions and the conditional class frequencies. The
refinement loss is what remains: how well the predictions separate the
outcomes [arxiv:2112.10327 §3.1]. The same split can be written with the true
posterior in place of the predictions, giving an epistemic part (model error)
and an irreducible part [arxiv:2112.10327 §3.1]. The survey's worked example
on 10 instances gives Brier 0.71 = calibration 0.19 + refinement 0.52, and log-loss
1.12 = 0.29 + 0.83 [arxiv:2112.10327 §3.1]. This is an author-reported toy
example, not a benchmark.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Probabilities can be exactly 0 for the true class | Brier score | Finite everywhere, so no clipping. [arxiv:2112.10327 §3, eq. 1] |
| You want strong penalties on confident errors | Log-loss | Grows without bound as the true-class probability goes to 0. [arxiv:2112.10327 §3, eq. 2] |
| You are fitting a post-hoc calibration map | Log-loss (the survey's default objective) | Most of its maps are fitted by log-likelihood. [arxiv:2112.10327 §5.3, §5.4] |
| You need to know whether to recalibrate or to improve the model | Decomposition | Calibration loss is what a recalibration map can remove; refinement loss is not reduced by it. [arxiv:2112.10327 §3.1] |
| You want to see where the predictions are off | Reliability diagram, read alongside a proper score | A single score does not show which score ranges are off, and ECE depends on the bin count. [arxiv:2112.10327 §4.1, §4.2] |

## Implementation sketch

Both rules take the predicted vector and the one-hot label for one instance.
The test-set score is the mean over instances.

```pseudocode
# Brier score for one instance  [arxiv:2112.10327 §3, eq. 1]
# probs: (K,) predicted vector; onehot: (K,) true class indicator

def brier(probs, onehot):
    # Squared error over all classes, summed  [§3, eq. 1]
    return sum((probs - onehot) ** 2)
```

```pseudocode
# Log-loss for one instance  [arxiv:2112.10327 §3, eq. 2]
# probs: (K,) predicted vector; onehot: (K,) true class indicator

def log_loss(probs, onehot, eps=1e-12):
    # Clip so a zero probability on the true class stays finite  [§3]
    p = clip(probs, eps, 1.0)
    return -sum(onehot * log(p))
```

```pseudocode
# Test-set score is the mean of per-instance values  [arxiv:2112.10327 §3]
# probs: (N, K); onehot: (N, K)

def test_score(rule, probs, onehot):
    values = [rule(probs[i], onehot[i]) for i in range(len(probs))]
    return mean(values)
```

Where the survey says the Brier score is summed over classes, use that form
when you compare with its numbers. Averaging over classes divides the value by
K [arxiv:2112.10327 §3, eq. 1].

## Contradictions

None known. The survey is the only source on this page. It notes that some
authors flip the sign of these rules, which is a convention, not a conflict
[arxiv:2112.10327 §3].

## Pitfalls

- **Check the sign and the normalisation.** Losses and payoffs differ by sign,
  and the Brier score is a sum over classes in the survey's notation
  [arxiv:2112.10327 §3, eq. 1].
- **Clip before log-loss.** It is infinite at zero probability on the true class
  [arxiv:2112.10327 §3].
- **Do not read a low score as good calibration alone.** A score mixes
  calibration and refinement. Use the decomposition or a reliability diagram to
  separate them [arxiv:2112.10327 §3.1, §4.1].
- **Library versions may differ from the paper.** Check how a library averages
  or sums before comparing numbers. The survey's companion code is one example
  (see its card: [Classifier calibration survey](/papers/2112.10327)).

## Papers

- [Classifier calibration survey](/papers/2112.10327): defines the proper
  scoring rules and their calibration and refinement decomposition, and
  connects them to calibration maps and ECE.