Classifier Calibration: A survey on how to assess and improve predicted class probabilities

In plain words

When a spam filter gives an email a score of 0.7, about seven in ten emails with that score should really be spam. Many models do not behave like this. Some are too sure of themselves and others hedge too much. When the probabilities are off, any decision that depends on them, such as a cut-off for flagging spam or a choice weighted by the cost of mistakes, is off too.

This survey explains what "well calibrated" means, including the subtle multi-class versions. It shows how to measure the gap, mainly with charts that group predictions into bins and with a few summary numbers. It then covers the standard fixes: a small second model is trained on held-out data and maps the old scores to better ones. The maps range from simple bin averages to flexible curves with a few parameters. The authors also say which fix suits which kind of error, and which measures can be fooled.

TL;DR

• The paper is a tutorial-style survey of post-hoc classifier calibration, not a new method.
• Calibration is a property of the predicted probability vector across many instances, not of one prediction.
• Measure with proper scoring rules (Brier, log-loss), which split into calibration and refinement loss, plus reliability diagrams and ECE/MCE variants.
• Fix with a calibration map fitted on held-out data. Pick by error pattern and calibration-set size: binning and isotonic are non-parametric; Platt, beta and the temperature, vector, matrix and Dirichlet family are parametric.
• The paper's own numbers come from a 30-instance, 3-class toy example. The paper reports no benchmark results.

Problem

• A K-class probabilistic classifier is well calibrated if, among instances that receive the same predicted vector s, the true class frequencies are approximately s §1.
• Calibration matters for cost-sensitive decisions, for human decision making and for autonomous systems §1. In binary problems, a calibrated score can be thresholded at the cost-derived value c = C(+|−) / (C(+|−) + C(−|+)), and a change in class prior r → r′ moves the threshold to r/(r+r′) §2.1.
• Many learners are over-confident unless trained with dedicated procedures. Post-hoc methods use hold-out data to learn a map from the model's output to calibrated probabilities §1.
• Scope: the paper covers deployment in the same setting as calibration, apart from changes in class prior or misclassification cost. It does not cover robustness to other distribution shifts or out-of-distribution inputs §1.
• Three multi-class notions exist, each stronger than the last: confidence calibration (only the top class), classwise calibration (each class on its own, one-vs-rest), and multi-class calibration (the whole vector). Confidence calibration is much weaker than the multi-class notion, and papers often use the weaker one without saying so §1, §2.3.

Method

The paper is organised as follows: a high-level introduction §2, proper scoring rules and their decomposition §3, visualisation and error measures §4, post-hoc calibration methods §5, and a resampling test §6. Section and equation numbers follow the LaTeX source of arXiv v2, the 2023 revision.

1. What calibration means (multi-class) §2.3

Let f be a probabilistic classifier, S its output vector, Ŷ = argmax_j S_j and Y the true class.

• Confidence: among instances whose top probability equals α, the top class is correct with frequency α §2.3: \mathsf{P}(Y=\hat{Y} \mid S_{\hat{Y}}=\alpha)=\alpha
• Classwise: for each class i and each value s, the frequency of class i among instances with S_i = s equals s §2.3: \mathsf{P}(Y=i \mid S_i=s)=s
• Multi-class (strong): for each whole vector s, the class frequencies equal s §2.3: \mathsf{P}(Y=i \mid \mathsf{f}(X)=\mathbf{s})=s_i

2. Proper scoring rules §3

Notation: y is the one-hot true class and s the predicted vector. Lower is better here. The paper notes that some authors use the opposite sign, and that payoff functions can be negated §3.

• Brier score, summed over classes §3, eq. 1: \phi_{\text{BS}}(\mathbf{s},\mathbf{y}) = \sum_{j=1}^{K} (s_j - y_j)^2
• Log-loss §3, eq. 2: \phi_{\text{LL}}(\mathbf{s},\mathbf{y}) = -\sum_{j=1}^{K} y_j \log s_j
• A test-set score is the average of the per-instance values §3.
• A rule is proper if its expected value is minimised by reporting the true distribution q. Both rules above are strictly proper §3.

Decomposition §3.1: the expected loss splits as E[d(S, Y)] = E[d(S, C)] + E[d(C, Y)], where C_j = E[Y_j | S]. The first term is the calibration loss and the second is the refinement loss. For log-loss d is a KL divergence; for the Brier score d is a mean squared difference. Replacing S by the true posterior Q gives epistemic loss (model error) plus irreducible loss §3.1.

# Brier score and log-loss for one instance  [§3] [eq. 1] [eq. 2]
# probs: (K,) predicted vector; onehot: (K,) true class indicator

def brier(probs, onehot):
    # Squared error over all classes, summed  [§3] [eq. 1]
    return sum((probs - onehot) ** 2)

def log_loss(probs, onehot):
    # Only the probability of the true class enters  [§3] [eq. 2]
    # Infinite if that probability is 0; clip in practice (see Pitfalls)
    return -sum(onehot * log(probs))

3. Reliability diagrams §4.1

• Scores are grouped into bins. Width binning uses equal-width bins. Frequency binning uses equal-count bins §4.1.
• For each bin, plot the observed positive rate against the mean score in the bin. The paper argues against plotting at the bin centre, because scores can lie up to half a bin width from the centre and still look miscalibrated §4.1, fig. 6.
• Uncertainty per bin can be shown with binomial intervals, or with consistency resampling, which resamples scores and labels to get surrogate forecasts §4.1.
• For multi-class, draw a confidence diagram, or one one-vs-rest diagram per class §4.1, fig. 7.

4. Error measures §4.2–§4.4

Binary ECE and MCE, with M bins, N instances and bin B_m §4.2, def. 1, def. 2:

\mathrm{ECE}_{\text{binary}} = \sum_{m=1}^{M} \frac{|\mathbb{B}_m|}{N}\,\big|\bar{y}(\mathbb{B}_m)-\bar{s}(\mathbb{B}_m)\big|

\mathrm{MCE}_{\text{binary}} = \max_{m}\,\big|\bar{y}(\mathbb{B}_m)-\bar{s}(\mathbb{B}_m)\big|

Classwise: class-j-ECE treats class j against the rest, and classwise-ECE is the mean of these over the K classes. Classwise-MCE is the largest bin gap over all classes and bins §4.3, def. 3–5.

Confidence ECE uses the top probability as the score and whether the top class was right as the label §4.4, def. 6:

\mathrm{ECE}_{\text{confidence}} = \sum_{m=1}^{M} \frac{|\mathbb{B}_m|}{N}\,\big|\mathrm{acc}(\mathbb{B}_m)-\bar{p}(\mathbb{B}_m)\big|

# Binary ECE with M equal-width bins  [§4.2] [def. 1]
# scores: (N,) in [0, 1]; labels: (N,) in {0, 1}

def binary_ece(scores, labels, num_bins):
    # Bins are [k/M, (k+1)/M); 1.0 falls in the last bin
    # [code: pycalib/metrics.py#L553]
    bin_index = bin_of(scores, num_bins)
    ece = 0.0
    for m in range(num_bins):
        in_bin = bin_index == m
        if not any(in_bin):
            # Empty bins add nothing to the sum  [§4.2]
            continue
        # Mean score and observed positive rate in the bin  [§4.2]
        mean_score = mean(scores[in_bin])
        positive_rate = mean(labels[in_bin])
        weight = count(in_bin) / len(scores)
        ece += weight * abs(positive_rate - mean_score)
    return ece

# Classwise-ECE: mean of the one-vs-rest binary ECEs  [§4.3] [def. 4]
# probs: (N, K); one_hot: (N, K)

def classwise_ece(probs, one_hot, num_bins):
    num_classes = probs.shape[1]
    # Score each class column against its own indicator  [§4.3]
    gaps = [binary_ece(probs[:, j], one_hot[:, j], num_bins)
            for j in range(num_classes)]
    # Every class has the same weight  [§4.3]
    return mean(gaps)

5. Post-hoc calibration maps §5

A map g is fitted on a calibration set {(s_i, y_i)}, separate from the data used to train the base model. The objective is usually a sum of a proper scoring rule §5.1. The paper notes that a small calibration set can overfit and suggests splitting the training data into three folds. Two folds train the model and the third fits the map, and the three model-map pairs are averaged at test time (Platt's scheme) §5.1. It also mentions label smoothing toward a uniform vector as a way to reduce over-confidence §5.1.

Binary maps §5.3. Each maps [0,1] to [0,1].

• Empirical binning, with bins B_m and bin values a_m §5.3, eq. 3: \mathsf{g}(s) = a_m \quad \text{if } s \in \mathbb{B}_m The fit uses the sum of absolute gaps between bin positive rates and a_m §5.3, eq. 4. Read per bin, the best a_m is the bin's positive rate.
• Isotonic regression, with edges b_m and non-decreasing values v_m §5.3, eq. 5: \mathsf{g}(s) = \sum_{m} \mathbb{1}(b_m \le s < b_{m+1})\, v_m The fit minimises the squared error with labels, that is the Brier score on the training set §5.3, eq. 6. Between two edges with different values, the map is linearly interpolated §5.3.
• Platt scaling, a sigmoid in the score §5.3, eq. 7: \mathsf{g}(s) = \frac{1}{1+\exp(-w s - b)} The fit maximises the likelihood, that is it minimises log-loss §5.3, eq. 8. For probability inputs, the paper suggests a logit transform first §5.3.
• Beta calibration, a sigmoid in ln s and ln(1−s) §5.3, eq. 9: \mathsf{g}(s) = \frac{1}{1+\exp(-a\ln s + b\ln(1-s) - c)} Fitted by log-loss §5.3, eq. 10. With a = b = 1 and c = 0 the map is the identity, so an already calibrated model is left alone §2.2, §5.3.

Multi-class maps §5.4. The inputs are logits z (or probabilities converted with z_j = ln(s_j / s_K), K as reference class §5.4).

• Temperature scaling, one parameter t §5.4, eq. 11: \mathsf{c}_j(\mathbf{z}; t) = \frac{\exp(-z_j/t)}{\sum_{k}\exp(-z_k/t)} Fitted by log-loss §5.4, eq. 12. See Pitfalls on the sign.
• Vector scaling, a temperature and an intercept per class §5.4, eq. 13: \mathsf{c}_j(\mathbf{z}; \mathbf{w}, \mathbf{b}) = \frac{\exp(-w_j z_j - b_j)}{\sum_{k}\exp(-w_k z_k - b_k)} Fitted by log-loss §5.4, eq. 14.
• Matrix scaling, a full matrix and an intercept vector §5.4, eq. 15: \mathsf{c}_j(\mathbf{z}) = \frac{\exp(-\mathbf{w}_j^{\top}\mathbf{z} - b_j)}{\sum_{k}\exp(-\mathbf{w}_k^{\top}\mathbf{z} - b_k)} Fitted by log-loss §5.4, eq. 16.
• Dirichlet calibration, a logistic model on log-probabilities §5.4, eq. 17: \mathsf{g}_j(\mathbf{s}) = \frac{\exp(-b_j - \mathbf{w}_j^{\top}\ln\mathbf{s})}{\sum_{k}\exp(-b_k - \mathbf{w}_k^{\top}\ln\mathbf{s})} Fitted by log-loss §5.4, eq. 18. It can represent the identity. The paper mentions an off-diagonal and intercept regulariser (ODIR) from its source, with no details here §5.4.

Other methods are only surveyed: calibration-aware training (for example mixup, kernel mean embedding of scores, Bayesian last layers), ensembles of local calibrators, calibration for regression quantiles and distributions, and conformal prediction §5.5–§5.8. Each gets a short description and no equations, so implementing them would need the original papers.

# Platt scaling for one binary score  [§5.3] [eq. 7] [eq. 8]
# score: real number (or a logit of a probability, see §5.3)
# w, b: fitted slope and intercept

def platt(score, w, b):
    # Sigmoid of an affine function of the score  [§5.3] [eq. 7]
    return 1 / (1 + exp(-w * score - b))

# Temperature scaling, as printed  [§5.4] [eq. 11]
# logits: (N, K); t: single positive scalar

def temperature_scale(logits, t):
    # The printed form divides by -t; see Pitfalls before implementing
    return softmax(-logits / t)  # (N, K)

6. Hypothesis test for calibration §6

Even a calibrated model deviates on a finite sample. The paper gives a resampling test, alg. 1, for any chosen measure φ (ECE, log-loss or Brier): keep the scores fixed, draw new label sets from the predicted vectors, and compare the observed value with the resampled values §6, alg. 1.

# Resampling test for calibration  [alg. 1]
# probs: (N, K) predictions; labels: (N,); phi: measure (larger is worse)

def calibration_test(probs, labels, phi, num_resamples, alpha):
    observed = phi(probs, labels)
    resampled = []
    for _ in range(num_resamples):
        # Scores stay fixed; labels are drawn from each predicted vector  [§6]
        new_labels = sample_from_rows(probs)
        resampled.append(phi(probs, new_labels))
    # Share of resampled values above the observed value  [§6]
    p_value = mean([value > observed for value in resampled])
    # True means: reject the null hypothesis of calibration  [alg. 1]
    return p_value < alpha

The paper also describes Hosmer–Lemeshow-style goodness-of-fit tests, a binary form with equal-frequency bins and a multinomial extension with a chi-squared statistic §6, eq. 19–20. It says the test is not recommended for small samples §6.

Hyperparameters

Values stated in the paper and in its code. The synthetic test settings are the paper's own demonstrations, not recommended defaults.

Code defaults, from the official library at the commit in the front matter:

Results

The paper reports no benchmark. The numbers below are its worked examples (author-reported), on the 30-instance, 3-class toy example tab. 3 and a 10-instance example for scoring rules tab. 2. The test-case results are described only in figure captions and text §5.2–§5.4. For this card I re-derived the binary, classwise and confidence ECE values, and the mean Brier and log-loss, from the printed tables; they match. The other rows are as printed.

Paper vs code

The paper names PyCalib as its code and gives the classifier-calibration GitHub organisation for its supplementary material (declarations section). That organisation is the authors' own. The repository is classifier-calibration/PyCalib (BSD-3-Clause, copyright Miquel Perello-Nieto, one of the authors). It was cloned at the commit in the front matter, and the paper's [code] references were checked against it.

Code is a library of scikit-learn-style calibrators and metrics, not the paper's full set. The paper says most of its figures and results were generated with PyCalib §1. The repository's examples/ folder includes a quickstart that uses the isotonic calibrator xmpl_quickstart.py.

• Temperature, vector, matrix and Dirichlet scaling are not implemented. No such class or function is in the library at this commit (a search of pycalib/ finds none) calibrators.py. The paper's own experiments on these maps are not reproducible from this repository.
• Brier score is averaged, not summed. The paper's Brier score sums over classes §3, eq. 1. brier_score calls scikit-learn's mean squared error, which averages over samples and classes, so for K classes the result is the paper's value divided by K metrics.py. The docstring says the score lies between 0 and 1.
• Binary and classwise ECE use 15 bins by default metrics.py. The paper's examples use 5 and 10. The bins are equal-width, left-closed, and 1.0 falls into the last bin metrics.py. The formula matches §4.2, def. 1 metrics.py.
• Confidence ECE uses the argmax as the prediction and the maximum probability as the confidence metrics.py. The general ECE function defaults to ece_full=True, a different path that the paper does not describe metrics.py.
• Empirical binning adds Laplace smoothing. The paper's fit is an empirical frequency §5.3, eq. 3–4. The code's value is (count + α) / (n + 2α) with α = 1.0 by default, so empty bins get 0.5 calibrators.py. Bin edges are the min and max of the calibration scores for uniform calibrators.py, not [0, 1]. The quantile branch carries a "TODO check that this code is correct" comment calibrators.py. The outer edges are set to ±∞ calibrators.py.
• Platt input transform differs. The paper suggests a logit for probability inputs §5.3. LogisticCalibration defaults to log_transform=True, which feeds ln s calibrators.py. The logit path is used only when that flag is False calibrators.py. The C grid is chosen on a validation set by log-loss calibrators.py.
• Platt's sigmoid fit is scikit-learn's. SigmoidCalibration subclasses scikit-learn's private _SigmoidCalibration calibrators.py. Its target smoothing and fitting method are not in this repository.
• Isotonic has no ε clipping. IsotonicCalibration wraps IsotonicRegression(y_min=0, y_max=1, increasing=True, out_of_bounds='clip') and returns its output unchanged calibrators.py. The paper's suggested clipping is absent.
• Beta calibration is an external dependency. The code imports betacal calibrators.py and does not implement the beta map itself. I did not inspect betacal, so its fit is not checked here.
• One-vs-rest normalisation is unconditional. OneVsRestCalibrator.predict_proba divides each row by its sum and sets NaN rows to uniform multiclass.py. Its normalize argument is stored and never read multiclass.py.
• The wrapper calibrates on training scores. CalibratedModel.fit fits the base model on X and then fits the calibrator on predict_proba(X) for the same X calibrators.py. The X_val and y_val arguments are accepted but never passed to the calibrator. This departs from the paper's advice to use hold-out data §5.1.

No discrepancies were found for the metric formulas for binary ECE, classwise ECE and classwise MCE, apart from the bin count default noted above.

Pitfalls

• Pick the notion first. A classifier can be classwise-calibrated but not confidence-calibrated. The paper gives a 40-instance counterexample in a footnote of §2.3. Confidence-ECE says nothing about the other classes.
• ECE and MCE depend on the bins. Different bin counts give different values, and bins with few instances dominate MCE. Predicting the overall class frequencies for every instance gives ECE 0 §4.2, §4.3, §4.4. Read ECE together with a proper scoring rule §4.4.
• Plot at the mean score, not the bin centre. Perfectly calibrated scores can look miscalibrated against bin centres, by up to half a bin width §4.1, fig. 6.
• Binning trades calibration loss for grouping loss. The output takes only M values, so instances with different raw scores become indistinguishable §5.3.
• Isotonic needs a monotone reliability curve and data. If the model's reliability curve is not monotone, the fit is suboptimal. It needs a larger calibration set than parametric maps, and its output reaches 0 or 1 at the ends §5.3.
• Platt cannot fix over-confidence. Its sigmoid pushes probabilities away from the centre. The paper's test case shows it making calibration worse in parts of the range §5.3, fig. 15.
• Beta and Dirichlet can represent the identity. That helps for already calibrated models, but a severely miscalibrated model may need a more flexible map §5.3, §5.4.
• Temperature scaling has a sign conflict. Eq. 11 is printed as exp(−z_j / t). With t = 1 that is softmax(−z), but the text says t = 1 gives the usual softmax §5.4. The intended form is probably softmax(z / t) with t > 0, but the paper does not say which. Check the source before implementing.
• Log-likelihood objectives are printed in an odd form. Eqs. 8, 10, 12, 14, 16 and 18 write ln of a sum of negated indicators times the probability. The intended reading is the mean negative log-likelihood (log-loss) of the true class. This reading is inferred here, not stated.
• Isotonic eq. 5 has a notational slip. The printed sum uses m_j, while the edge values are v_m §5.3, eq. 5. Read it as v_m.
• The Hosmer–Lemeshow statistic as printed repeats one term. Eq. 20's two summands are identical. Check the original multinomial test before using the statistic §6, eq. 20.
• The resampling test keeps scores fixed. Bins keep their members, so small bins have little weight in every resample §6.
• Log-loss is infinite at zero probability for the true class. Clip probabilities before computing it §3.
• Calibration sets must be held out. A map fitted on the scores of training data is overfitted. The paper's remedies are multi-fold fitting and label smoothing §5.1. The code's CalibratedModel does not do this (see Paper vs code).
• Matrix, Dirichlet and vector scaling overfit small calibration sets. Many parameters on few instances. The paper suggests regularisation, with ODIR for Dirichlet §5.4.
• The figures are synthetic. The test cases use generated data and a fixed MLP, so the qualitative claims do not show how a map behaves on real data §5.2.
• Figure credits. Some figures in the paper are credited to other publications, so they are described here in words only.

Lineage

• Builds on: work the survey cites by name, including Platt's sigmoid method, binning (Zadrozny and Elkan), isotonic calibration (Niculescu-Mizil et al. 2005), the Brier score (1950), and the score decompositions of Sanders (1963) and DeGroot (1983) §3.1. None is summarised here.
• Named as the source of several methods: temperature, vector and matrix scaling are attributed to Guo et al. §5.4; beta and Dirichlet calibration are attributed to Kull et al. §5.3, §5.4.
• Extended by: the paper's own extended-methods section lists later work only by name §5.5–§5.8.
• Companion code: classifier-calibration/PyCalib (see Paper vs code).