Calibration

Overview

Calibration asks whether a model's stated uncertainty is honest: when it reports a confident prediction, is it right about as often as it claims? Several sources treat this as a property to measure alongside accuracy. A tutorial notes that most benchmarks score accuracy and rarely score the quality of the stated uncertainty 2305.04933 §1. The HLE paper makes the same point for language models: high accuracy alone does not show whether a model knows when it is wrong, so it asks for calibrated confidence 2501.14249 §4.2.

Calibration is measured in several ways. Expected calibration error (ECE) compares confidence with accuracy in bins 2305.04933 §6.1 2006.10108 tab. 2, and the negative log-likelihood (NLL) is reported next to it 2305.04933 §6.1. A benchmark can also use RMS calibration error over verbalised confidence 2501.14249 §4.2. A survey of uncertainty in deep networks adds two all-label variants, SCE and an equal-mass ECE (aECE), and reports that the all-label measures are more effective than top-label ECE, citing other work 2107.03342 §5.2. A scientific-ML framework uses RMSCE over Gaussian predictive quantiles: it compares the fraction of test values inside each quantile with that quantile's probability 2201.07766 §5.1. Calibration is one part of uncertainty estimation: a method can give a spread and still be over- or underconfident 2305.04933 §6.1.

The sources use calibration in a second sense. KnowNo, a language model planner for robots, uses a labelled calibration set to pick a score cut-off, so that its actions reach a success rate the user sets 2307.01928 §3.1. Its guarantee is statistical and holds under stated assumptions, the main one being that calibration and test scenarios come from the same distribution 2307.01928 §3.2.

A third use appears in a 2026 paper on noisy multi-label training data. Its calibrator maps a histogram of model confidences on candidate labels to the chance that each candidate is correct, and that chance becomes the training weight for the label 2610.17635 §4.1 eq. 3 eq. 4.

The evidence is narrow. The Dirichlet calibration paper covers tabular classifiers on 21 UCI datasets and image classifiers on CIFAR-10, CIFAR-100 and SVHN 1910.12656 §4.1 1910.12656 §4.2. The tutorial's case studies cover regression on engineering data (battery life and turbofan remaining life). The SNGP paper covers image classification and intent detection with accuracy-based evaluation 2006.10108 §5. KnowNo covers robot task planning with hardware trials. The noisy multi-label paper covers image classification with simulated label noise on four datasets 2610.17635 §5.1. HLE covers closed-ended expert questions. The language-model self-evaluation study covers trivia, word prediction, arithmetic, grade-school maths and code tasks, all with short answers that can be checked 2207.05221 §5. Prophet Arena scores forecasts for 1,367 resolved binary prediction-market events, most of them sports, for 23 LLMs and a market baseline 2510.17638 §3.2.2. The survey has no experiments of its own and quotes its numbers from cited work 2107.03342 §3.3. The zero-shot scoring paper reports accuracy on multiple-choice tasks and no calibration measure 2104.08315 tab. 1. None of the sources covers open-ended text generation. All results are author-reported.

The tutorial also separates two kinds of doubt that a calibrated model should report. Aleatory uncertainty is noise that more data cannot remove. Epistemic uncertainty comes from limited data and can shrink with more data 2305.04933 §2.

The word also names a different task in one source: estimating where a sensor is. The DROID dataset paper releases camera poses for its scenes, a camera-to-base set and a camera-to-camera set, computed after collection 2403.12945 §3.2 app. G.5. That is geometric sensor calibration, not confidence calibration, so none of the numbers on this page apply to it. For the sensor task, see Hand-eye calibration.

Variants

Several readings of calibration appear in the sources. Most measure how honest a model's stated uncertainty or confidence is. The conformal variant sets a threshold so that a decision meets a target. The post-hoc and self-evaluation variants are ways to change or read a model's confidence. They are not alternatives to each other; a study usually uses several.

Reliability curves and ECE

Plot predicted confidence against observed frequency, and summarize the gap as ECE. The tutorial reports ECE in percent per dataset split [arxiv:2305.04933 §6.1] 2305.04933 §6.2. It does not state the bin count, so the bin count is not stated here. The SNGP paper uses 15 bins 2006.10108 app. C and reports ECE on CIFAR-10, CIFAR-100 and CLINC 2006.10108 tab. 2 2006.10108 tab. 3 2006.10108 tab. 4. The language-model study takes top-label ECE over 10 equal-count bins, and computes it only from the top prediction of each multiple-choice question 2207.05221 app. A.2. The survey defines the same top-label ECE over equal-width bins and its SCE and aECE variants over all labels 2107.03342 §5.2. ECE values depend on the binning and on which probabilities enter, so compare them only when both match. Prophet Arena uses binned ECE with 10 equal-width bins over every market's forecast, pooled across events 2510.17638 app. B.2. Its values are covered in the forecasting reading below. See the Contradictions section and the UQ tutorial card.

RMS calibration error over verbalised confidence

HLE asks each model to state a confidence from 0% to 100% in its answer 2501.14249 app. C.1. Its calibration metric is the root of a weighted sum of squared gaps between mean stated confidence and mean accuracy in bins 2501.14249 §4.2. The paper does not state the bin size. The reference code uses 100 items per bin, and the HLE card reads from that code that the last bin is left out of the sum; this was not run. The reported values for eight models are 73% to 89% 2501.14249 tab. 1. Each model's accuracy on that set is 2.7% to 13.4%, so these errors come with very low accuracy and are not comparable to ECE values from the other sources.

NLL as a joint score

NLL rewards both accuracy and a well-sized spread, so a model cannot lower it by being vague alone. The tutorial reports NLL next to RMSE and ECE for every method 2305.04933 §6.1 2305.04933 §6.2. Its Gaussian negative log-likelihood is the training loss for the Gaussian-head ensemble 2305.04933 §3.3. The SNGP paper reports NLL on CIFAR and CLINC next to ECE 2006.10108 tab. 2 2006.10108 tab. 4.

Uncertainty methods scored for calibration

The tutorial compares four families on calibration, and the SNGP paper adds a more detailed look at one of them:

• Gaussian process regression (GPR). Its posterior variance grows far from training inputs, which gives distance-aware uncertainty 2305.04933 §3.1. Exact GPR costs O(N³) in training points, so it was excluded from the turbofan case study 2305.04933 §6.2. On the battery data it fit the training set exactly but had high uncertainty on the test sets 2305.04933 §6.1.
• MC dropout. Dropout stays on at prediction time and the spread over passes gives the uncertainty 2305.04933 §3.2. The tutorial reports it as overconfident, with the largest run-to-run variation of the group 2305.04933 §6.1 2305.04933 §6.2. Its behaviour depends on dropout rate, number of dropout layers and training length 2305.04933 §3.2. See Monte Carlo dropout.
• Gaussian-head ensemble. Several networks each predict a mean and a variance; their disagreement is the epistemic part 2305.04933 §3.3. On the battery data it had the lowest NLL on each test set, as the author-reported summary states 2305.04933 §6.1. See Ensembles and Uncertainty estimation.
• SNGP. One deterministic network with spectral normalization on its hidden weights and a Gaussian-process output layer 2006.10108 §3.1 2006.10108 §3.2. The SNGP paper reports it close to a 10-model deep ensemble on calibration and OOD detection, at single-model latency. On CIFAR-10 clean data, ECE is 0.018 for SNGP, 0.010 for the ensemble and 0.023 for the deterministic network 2006.10108 tab. 2. Latency is 6.25 ms per example for SNGP against 38.10 ms for the ensemble, on the paper's hardware 2006.10108 tab. 2. The tutorial reports SNGP as often underconfident on the battery data, with high NLL and ECE there 2305.04933 §6.1, and says that SNGP-type methods cannot separate the two kinds of uncertainty 2305.04933 §3.4. Card for the SNGP method: SNGP. Its precision update, initialization and sign of the random projection are inconsistent between the paper's sections, so the card lists what to check before reimplementing.

The survey summarises a wider comparison from cited work. Deep ensembles are reported as at least as good as Bayesian baselines, such as MC dropout, on accuracy, calibration and out-of-distribution detection, and as more robust to dataset shift 2107.03342 §3.3. These are the survey's summaries, not its own measurements, so check the original studies before relying on them. The same section reports that bagging can worsen uncertainty in ensembles 2107.03342 §3.3.

Temperature scaling after training

Temperature scaling is a post-hoc fix. The softmax is applied to the logits divided by a single positive temperature T, and T is fitted after training. T above 1 lowers the confidence; the predicted class does not change. The survey notes that the method's expressiveness is limited 2107.03342 §5.1. It also lists other post-hoc options: a class-wise one-vs-all method, scaling of the standard deviation for regression, and a Gaussian-process calibration map 2107.03342 §5.1. The survey gives no benchmark numbers for these, so none are quoted here. The Dirichlet paper compares the fitted multiclass maps against temperature scaling directly, with the numbers in the variant below 1910.12656 tab. 1 1910.12656 tab. 2.

The language-model study uses a fixed temperature of 2.5 for its RLHF models, which it reports as fixing calibration on several RLHF evaluations. It warns that more intensive RL could break this 2207.05221 §3.3. This is a single model-level setting, not a fitted calibration map, so it should not be confused with temperature scaling above.

Logit normalization during training

This reading changes the training loss, not the confidence after training. The network is trained with cross-entropy on its logit vector divided by that vector's L2 norm and by a temperature tau. Every training logit vector therefore has length 1/tau 2205.09310 §3.2 eq. 4. The authors argue that ordinary cross-entropy keeps growing logit norms during training, for familiar and unfamiliar inputs alike, and that this drives the overconfidence seen on out-of-distribution (OOD) inputs 2205.09310 §3.1 fig. 1. The architecture and the test-time score are unchanged; the paper scores inputs with the maximum softmax probability on the raw output 2205.09310 §4.2. The tau here is set during training and is not the post-hoc temperature scaling factor; the paper contrasts the two in its discussion 2205.09310 §3.2 §5.

For OOD detection, the author-reported result on CIFAR-10 with WRN-40-2 is that the average FPR95 over six OOD sets falls from 49.52% to 15.65%, with ID accuracy close to cross-entropy 2205.09310 tab. 1 tab. 3. The paper also applies other OOD scores to the same LogitNorm networks. On CIFAR-10, ODIN's average FPR95 is 12.95%, below MSP's 15.65%, while energy (19.14%) and GradNorm (17.78%) are higher 2205.09310 tab. 2. Card: LogitNorm.

The calibration numbers in the paper depend on a second step. On CIFAR-10, expected calibration error (15 bins) is 66.95% for LogitNorm without temperature scaling, against 3.20% for cross-entropy. With temperature scaling, it is 0.41% against 0.66% 2205.09310 tab. 4. The text does not discuss the without-scaling row 2205.09310 tab. 4. So the method on its own gives poorly calibrated probabilities, and any calibrated output needs the post-hoc fit as well.

The tau search and the scope limit the evidence. The paper picks tau for CIFAR-10 from a grid running from 0.001 to 0.05, with 0.04 as the default, using Gaussian noise as an OOD validation set 2205.09310 §4.1. The experiments cover CIFAR-10 and CIFAR-100 only 2205.09310 §4.1, and the paper calls tuning expensive because it needs several training runs 2205.09310 app. Future Work.

Recalibrating a predictive spread on a left-out set

For a model that outputs a mean and a variance, a left-out calibration set can correct the spread after training. The scientific-ML framework describes three post-training methods 2201.07766 §5.2. Variance scaling multiplies the predicted variance by one constant, chosen by optimising a metric, and leaves the mean unchanged. A CDF correction fits a map to the observed calibration curve and applies it to every predictive CDF, which changes both mean and variance app. "CDF modification by fitting an auxiliary model". A third method fits an empirical distribution to the scaled residuals of the calibration set app. "CRUDE".

The paper reports that post-training calibration makes all its methods roughly equally calibrated on a 32-point function-approximation problem, though the calibrated RMSCE row of its table still spans 3.8 to 8.0 2201.07766 §6.1. On a mixed stochastic PDE, calibration on two to ten left-out realizations reduced RMSCE by about a factor of three in one case 2201.07766 §6.3. These are author-reported results on prototype problems. The calibration data come from the training distribution, so performance may get worse on out-of-distribution tests 2201.07766 §5.2.

Multiclass probability maps after training

This reading covers post-hoc maps for any classifier that outputs class probabilities, not only neural networks. The Dirichlet calibration paper argues that per-class binary fixes, such as one-vs-rest Platt scaling or isotonic regression, are not guaranteed to give calibrated multiclass outputs once normalised 1910.12656 §1. Its map takes the logarithm of each probability vector, applies one learned k-by-k linear map and a bias, then a softmax 1910.12656 §3 eq. 7. The map is fitted on held-out labelled data by multinomial logistic regression 1910.12656 §3. Temperature scaling is a special case of this family, with a diagonal weight matrix and no bias 1910.12656 §3 app. B.

Two regularised fits are described. Dirichlet L2 applies one L2 penalty to all weights 1910.12656 §3. ODIR penalises the off-diagonal weights and the intercept separately, each normalised by its number of terms 1910.12656 §3. The paper's ODIR objective is not numbered, and the paper gives no grid for its regularisation strengths 1910.12656 §3. The released code uses a six-value grid for ODIR calibration.py and an eleven-value grid for L2 calibration.py. The Keras wrapper in the code does not apply the paper's normalisation to the penalties, so its strengths are not interchangeable with the paper's. See the Dirichlet calibration card for the full list before reimplementing.

Results are author-reported. On 21 UCI datasets with 11 classical classifiers (231 settings), Dirichlet L2 has the best average log-loss rank, 2.25, against 2.92 for beta calibration and 4.61 for temperature scaling 1910.12656 tab. 2. Its average rank on p-cw-ECE is 2.34, against 3.15 for beta calibration and 4.37 for temperature scaling 1910.12656 tab. 1. The p-cw-ECE measure checks every class, not only the top prediction. On 14 deep-network cases, the paper's summary ranks put Dir-ODIR at 2.79 on cw-ECE, against 3.71 for temperature scaling, and at 2.93 on log-loss, against 3.50 1910.12656 tab. 3 1910.12656 tab. 4. These two deep-network results rank the three general-purpose methods only, and the paper says the Dir-ODIR lead is not statistically significant there 1910.12656 §4.2. Matrix and vector scaling act on logits, so they are not general-purpose methods. Vector scaling has the best average cw-ECE rank among all six deep-network methods, 2.29 1910.12656 tab. 3.

Self-evaluated probabilities in language models

This reading asks a language model to judge its own answers. The model samples an answer at temperature 1, then is asked whether that answer is "(A) True" or "(B) False". The probability on True, P(True), is the score 2207.05221 §4.1. Zero-shot, the score is poorly calibrated and sits near 50% for typical samples 2207.05221 §4.1 app. B. Showing five samples to the judge, one of them the one being judged, is the variant the authors report as best with few-shot prompting 2207.05221 §4.2 fig. 11. The figure carries those results; the text gives few exact values.

A second score, P(IK), trains a value head to predict whether the model knows an answer before it answers 2207.05221 §5 app. A.6. For the 52B model trained and tested on TriviaQA, AUROC is 0.864 and the Brier score is 0.151 2207.05221 tab. 1. Transfer is partial. When trained on TriviaQA only, AUROC on Lambada, GSM8k and Python function synthesis is 0.606, 0.624 and 0.687 2207.05221 tab. 1. Calibration on those tasks is poor, and models are often underconfident there 2207.05221 fig. 13. Training on all tasks except GSM8k raises the AUROC on Lambada to 0.853 2207.05221 tab. 1. Adding a relevant Wikipedia article to a question raised P(IK) from 18% to 78% in the paper's one example 2207.05221 §5.3.

The authors' models are pretrained models of 0.8B to 52B parameters 2207.05221 §1.2. The paper releases no code or weights, so these numbers are author-reported and cannot be reproduced from the paper alone 2207.05221 §7. The P(IK) head needs white-box access to add a head and train it with the model 2207.05221 app. A.6.

Conformal thresholds on option scores

KnowNo rewrites a planner's next-step choice as a multiple-choice question. The language model's confidence in each option is normalized over the options. On a calibration set of labelled scenarios, the score of each scenario is one minus the confidence of the correct option. The cut-off is the empirical quantile of those scores at level ⌈(N+1)(1−ε)⌉/N 2307.01928 §3.1 eq. 1. For a new scenario, the prediction set keeps every option whose confidence is at least 1 − q̂. The robot acts when the set has one option and asks a person otherwise 2307.01928 §3.2.

The paper gives two forms of the guarantee. The marginal form eq. 1 holds on average over the calibration set. The dataset-conditional form eq. 2 holds with probability 1 − δ over the calibration set and uses an adjusted ε̂. The paper uses N = 400 and δ = 0.01 2307.01928 §3.1 app. A.7.

For multi-step tasks, whole episodes are scored by the minimum per-step confidence of the correct option, and the prediction set is rebuilt at each step 2307.01928 §3.3 alg. 1. The bound does not cover the contexts reached after a wrong earlier action 2307.01928 app. A.2.

Reported results, author-reported. On a hardware mobile manipulator with a target of 0.85, KnowNo asked for help on 0.67 of decisions against 0.81 for the Simple Set baseline, at similar plan success (0.87) 2307.01928 tab. 2. On hardware tabletop rearrangement, step-wise help was 0.58 against 0.72 2307.01928 tab. 1. Task success stayed below the target on mobile manipulation (0.76 against 0.85). The authors attribute the gap to low-level execution 2307.01928 app. A.8.

The released notebooks differ from the paper in several ways: they use GPT-3.5 rather than PaLM-2L, use calibration sets of 100 or 200 rather than 400, scale the option log-probabilities by a temperature, and implement only the marginal form. See the KnowNo card, which lists each difference.

Calibrated label weights

This reading is a training method. Candidate positive labels in a noisy multi-label dataset are weighted by a calibrated estimate of being correct. The weight is the probability that a candidate is a true positive, given the model's confidence in it 2610.17635 §3 eq. 3 eq. 4. The paper fits this map with a pre-trained two-layer network, the calibrator. It reads a 20-bin histogram of confidences on candidate positives and outputs one reliability per bin 2610.17635 §4.1 eq. 5 eq. 6. A single confidence is mapped to a weight by linear interpolation between neighbouring bins eq. 7. The calibrator is trained on VOC 2007 at noise rates from 10% to 90%, and reused on the other datasets 2610.17635 §4.1 §5.1.

Results are author-reported. On COCO at 80% noise, the ECE of the weights falls from 76.78% under identity weighting to 3.74% with the calibrator, and mAP rises from 59.93 to 61.48 2610.17635 tab. 2. The main table reports the method as best in 15 of 16 settings against nine baselines, with mAP as the metric 2610.17635 §5.2 tab. 1. These are not calibration results. The oracle weighting that uses true labels has the same weight ECE, 3.74, which the paper does not discuss 2610.17635 tab. 2. The paper's own tables also disagree on VOC: the PML-CD and "2nd phase only" rows of tab. 1 and tab. 3 differ at three of four noise rates 2610.17635 tab. 1 tab. 3. The released code also picks the best epoch on the test split, which can make reported numbers optimistic see the PML-CD card. Use the calibrator as a reference for the method, not as a verified recipe.

Calibration as a reported metric

For image-level classification, the Metrics Reloaded framework adds a calibration metric to the pool of validation metrics only when calibration is requested 2206.01653 §ImLC. The framework is a consensus decision guide, not a benchmark, and the card does not name the calibration measure it would choose 2206.01653 §ImLC. Use it to decide which metrics to report, not which calibration formula to implement.

Calibration of LLM probability forecasts

Prophet Arena asks each model for a probability that every binary market of an event resolves Yes, then scores the answers after resolution. It reports three families: the Brier score, binned ECE, and the average return from betting one unit per market at the market price 2510.17638 §3.1 eq. 1. The prompt includes the live market price, so the market is an input to the model as well as the reference point 2510.17638 app. E.

Author-reported values over 23 LLMs: ECE runs from 0.030 (o3, reasoning) to 0.165 (DeepSeek-R1, reasoning), and the market baseline is 0.069 2510.17638 app. C.1. The paper says every highlighted model is better calibrated than the market 2510.17638 §3.2.2. Brier scores are close to the market's 0.187; the best listed is 0.184 2510.17638 app. C.1. No model reaches an average return of 1, and the highest is 0.982 2510.17638 app. C.1. The paper finds the models give less extreme probabilities than the market for events that resolve Yes; the paper calls this conservatism 2510.17638 §4.2.3.

Elicitation changes ECE. The paper's bi-direction method, which averages a Yes probability with one minus a No probability, lowers ECE for four of five models tested, with Claude Sonnet 4 going from 0.046 to 0.028 2510.17638 app. C.3. GPT-5 is the exception, at 0.020 with the default prompt and 0.023 with bi-direction. Self-consistency over 10 samples gives values in steps of 0.1 when unweighted, so it cannot represent fine probabilities 2510.17638 app. C.3.

Scoring depends on the setting. Results are for 1,367 events, 76% of them sports, with forecasts taken at scheduled times before close and none within three hours of resolution 2510.17638 §3.2.1 §3.2.2. Calibration of the same models on other event mixes is not reported. The paper also notes that the ECE equation is typeset in a way that differs from the estimator it used, so use the binned version in appendix B.2 2510.17638 app. B.2.

Prior correction in zero-shot scoring

A different use of the word appears in the zero-shot scoring paper. Domain Conditional PMI scores each answer option as its likelihood given the question, minus its likelihood given a short task-specific domain premise, which estimates how likely the option is in that domain regardless of the question 2104.08315 §3.2 eq. 2. This removes the penalty that many valid wordings put on a correct answer, and needs no training. The paper reports accuracy, not calibration: it does not measure whether the scores are well-calibrated probabilities 2104.08315 tab. 1 tab. 2. Its best- or-tied rate on 16 splits is 62.5% for GPT-3 175B, against 18.75% for length-normalised log-likelihood 2104.08315 tab. 2. The prior correction is related to contextual calibration (Zhao et al.), which the paper compares against with numbers copied from that work for five datasets 2104.08315 §3.3 tab. 1. That work has no card here, so nothing from it is used on this page. Card: Surface Form Competition.

Group calibration as a fairness criterion

A fairness reading asks whether a model's stated confidence means the same thing for each protected group. Binns treats equal calibration as a group measure that tolerates differences in base rates between groups 1912.06883 §5.1 tab. 2. The paper pairs it with a raw similarity metric under a "choice" view of disparities, and pairs statistical parity with a group-adjusted similarity metric under a "structural" view 1912.06883 §5.3 tab. 2. The paper's argument is that the assumption about where disparities come from should be chosen first, and the measure follows from it 1912.06883 §6. The paper also cites prior work showing that false positive equality and calibration cannot both hold in the COMPAS case 1912.06883 §5.3. That result is cited, not proven in the paper. The paper contains no experiments and reports no calibration numbers, so this reading gives a framing, not measured values. Card: On the Apparent Conflict Between Individual and Group Fairness.

When to use which

The sources give regression, image, robot-planning and language-model setups, so these rows follow the reported behaviour and its stated costs. They are not general rules.

Implementation sketch

The first sketch covers the ensemble combination, the step that produces the stated uncertainty. The second covers the conformal threshold. The third is binned ECE with equal-width bins, the estimator Prophet Arena used 2510.17638 app. B.2; the other sources bin differently, see the Contradictions section. Names are ours. For RMS calibration error, see the HLE card and its Pitfalls on the bin loop.

# Combine M Gaussian-head members into one predictive mean and variance
# [arxiv:2305.04933 §3.3]
# Input: M members, each giving (mu_m(x), var_m(x)) for one input x
# Output: combined mean and variance

def combine_members(mus, variances):
    n = len(mus)
    # Average of the members' means  [arxiv:2305.04933 §3.3]
    mean = sum(mus) / n
    # Average of (member variance + member mean squared), minus mean squared
    # [arxiv:2305.04933 §3.3]
    second_moment = sum(v + mu * mu for v, mu in zip(variances, mus)) / n
    return mean, second_moment - mean * mean

# Conformal cut-off from labelled calibration scenarios  [arxiv:2307.01928 §3.1]
# Input: calib_conf (N, options) normalized option confidences
# Input: calib_label (N,) index of the correct option; epsilon in (0, 1)
# Output: qhat, the cut-off for prediction sets

def conformal_cutoff(calib_conf, calib_label, epsilon):
    n = len(calib_label)
    # Score is one minus the confidence of the correct option  [§3.1]
    scores = 1 - calib_conf[range(n), calib_label]
    # Empirical quantile at level ceil((N+1)(1-eps))/N  [eq. 1]
    level = min(1.0, ceil((n + 1) * (1 - epsilon)) / n)
    return quantile(scores, level, method="higher")

# Binned expected calibration error, equal-width bins  [arxiv:2510.17638 app. B.2]
# Input: probs (M,) forecast probabilities pooled over markets
# Input: outcomes (M,) 0 or 1; bins: number of equal-width bins (paper: 10)
# Output: ECE in [0, 1]

def binned_ece(probs, outcomes, bins=10):
    total = 0.0
    for b in range(bins):
        # Bin b covers [b/bins, (b+1)/bins); the last bin also takes 1.0
        members = [j for j in range(len(probs)) if in_bin(probs[j], b, bins)]
        if not members:
            continue
        # Weight each bin's gap by its share of the members  [app. B.2]
        gap = abs(mean(outcomes[j] for j in members) - mean(probs[j] for j in members))
        total = total + len(members) * gap
    return total / len(probs)

Contradictions

Disagreements between papers, each side cited. The tutorial does not resolve them, and they affect how far the ranking above can be trusted.

• Turbofan ECE. The text says the ensemble (NNE) has low ECE on turbofan data 2305.04933 §6.2. The results table gives NNE 14.9% on unit 15, above MC dropout's 11.5% and SNGP's 8.5% 2305.04933 §6.2 results table.
• Battery ECE on other sets. The text says GPR gives the least ECE on the other datasets 2305.04933 §6.1. On the secondary test set MC dropout's ECE is 9.5% and GPR's is 10.4% 2305.04933 §6.1 results table.
• Battery RMSE and MSE. The battery table caption says MSE, while its rows say RMSE 2305.04933 §6.1 results table. The card treats them as RMSE.
• SNGP on battery data and on image benchmarks. The tutorial finds SNGP underconfident with high ECE and NLL on battery data 2305.04933 §6.1. The SNGP paper finds its ECE close to the ensemble's on CIFAR-10 and CLINC 2006.10108 tab. 2 2006.10108 tab. 4. These are different datasets and tasks, so neither result transfers to the other setting.
• Numbers that are not comparable. ECE in the tutorial, ECE with 15 bins in the SNGP paper, top-label ECE with 10 equal-count bins in the language-model study 2207.05221 app. A.2, binned ECE with 10 equal-width bins over market forecasts in Prophet Arena 2510.17638 app. B.2, and RMS calibration error with bins of 100 items in HLE 2501.14249 §4.2 measure different things. Do not compare their values across papers.
• GPT-5 row in Prophet Arena. The highlighted table labels the row GPT-5 (reasoning) with Brier 0.184, ECE 0.042 and return 0.943 [arxiv:2510.17638 tab. 2]. The full table gives those same values to the row GPT-5 (reasoning, High), and gives GPT-5 (reasoning) 0.187, 0.044 and 0.890 2510.17638 app. C.1. The paper does not say which setting the highlighted row uses. The page quotes the full table by row name.
• Lettered multiple-choice calibration against verbalised error. The language-model study reports that 52B models are well calibrated on lettered multiple-choice questions, using option probabilities 2207.05221 §2 app. A.1. HLE reports RMS calibration error of 73% to 89% for verbalised confidence on closed-ended questions, where accuracy is 2.7% to 13.4% 2501.14249 tab. 1. The inputs differ (option probabilities against stated confidence, multiple choice against closed-ended exact answers), so the two do not test the same thing. The page does not resolve them.
• Temperature scaling against Dirichlet maps, by setting. On 21 UCI datasets, Dirichlet L2 ranks ahead of temperature scaling on log-loss (2.25 against 4.61) and on p-cw-ECE (2.34 against 4.37) 1910.12656 tab. 1 1910.12656 tab. 2. On the paper's 14 deep-network cases, Dirichlet L2 ranks behind temperature scaling on both cw-ECE (3.79 against 3.71) and log-loss (3.79 against 3.50) 1910.12656 tab. 3 1910.12656 tab. 4. The paper reports both results, and the difference depends on the setting.
• All-label against top-label measures. The survey reports that all-label measures (SCE) assess calibration error more effectively than top-label ECE, citing Nixon et al. 2107.03342 §5.2. The language-model study uses top-label ECE only 2207.05221 app. A.2 and does not test the all-label claim. The survey's statement is cited here, not checked.
• KnowNo guarantee, paper against its code. The paper states the dataset-conditional bound with an adjusted ε̂ 2307.01928 §3.1. The released notebooks use the requested ε directly, which gives only the marginal form. The KnowNo card records this gap.
• Calibrator, paper against code. The paper describes a plain two-layer MLP with reliabilities at bin edges 2610.17635 §4.1 eq. 7. The released code adds smoothing and a monotone envelope to its target, and places reliabilities at bin centres see the PML-CD card.
• SNGP settings, paper against code. The SNGP paper's Laplace step is described with a cross-entropy likelihood, and its released code uses a squared-loss likelihood 2006.10108 §3.1. The predictive step is also Monte Carlo averaging in the paper and mean-field in the code. The paper's reported calibration is therefore for its own predictive path, which the code does not necessarily reproduce.

Papers

• Uncertainty Quantification in Scientific Machine Learning (2201.07766): compares Bayesian methods and ensembles for physics-informed networks and DeepONets on six metrics, including RMSCE, and three post-training recalibration methods. Author-reported on prototype problems.
• Dirichlet calibration (1910.12656): post-hoc multiclass map from log-probabilities to calibrated probabilities, with temperature scaling as a special case. Compared with beta calibration and temperature scaling on 21 UCI datasets, and with logits-based methods on CIFAR and SVHN networks. Author-reported.
• SNGP (2006.10108): one deterministic network with spectral normalization and a Gaussian-process output layer, compared with a 10-model ensemble on CIFAR and CLINC calibration. Author-reported.
• Mitigating Neural Network Overconfidence with Logit Normalization (2205.09310): trains with cross-entropy on L2-normalized logits scaled by a temperature, to limit overconfidence on OOD inputs. Reports FPR95 gains on CIFAR-10 and CIFAR-100 and ECE with and without temperature scaling. Author-reported.
• Metrics Reloaded (2206.01653): consensus framework for choosing validation metrics for image classification, detection and segmentation, including when to add a calibration metric. No benchmark numbers.
• UQ tutorial (2305.04933): compares GPR, MC dropout, Gaussian-head ensembles and SNGP on calibration, RMSE and NLL for battery and turbofan prognostics. Author-reported.
• KnowNo (2307.01928): applies split conformal prediction to an LLM planner's option probabilities, so a robot asks for help when a prediction set is not a singleton. Author-reported hardware results.
• DROID (2403.12945): robot dataset whose released camera poses are post-hoc sensor calibration. Listed only to mark the other sense of the word; it reports no confidence-calibration results.
• Humanity's Last Exam (2501.14249): reports accuracy and RMS calibration error for eight LLMs on a closed-ended expert benchmark. Author-reported.
• A Survey of Uncertainty in Deep Neural Networks (2107.03342): survey of uncertainty estimation and calibration measures, including temperature scaling and the ECE, SCE and aECE family. No experiments of its own; numbers are quoted from cited work.
• Language Models (Mostly) Know What They Know (2207.05221): P(True) self-evaluation and a P(IK) value head for language models, with calibration on multiple-choice questions. Author-reported; no code or weights released.
• On the Apparent Conflict Between Individual and Group Fairness (1912.06883): philosophical argument with no experiments. Treats equal calibration as a group measure and ties the choice of measure to assumptions about the source of disparities. Argument, not measured.
• Calibrated Disambiguation for Partial Multi-label Learning (2610.17635): a pre-trained calibrator maps confidence histograms to per-label reliability, used as loss weights on noisy multi-label images. Author-reported; code released.
• LLM-as-a-Prophet: Prophet Arena (2510.17638): live benchmark of LLM probability forecasts on prediction-market events, scored by Brier score, binned ECE and market return. Author-reported; no code.
• Surface Form Competition (2104.08315): zero-shot multiple-choice scoring that divides out an in-domain prior (Domain Conditional PMI). Reports accuracy, not calibration. Author-reported; the code has no LICENSE file.