--- kind: concept slug: calibration title: Calibration summary: >- Whether a model's stated uncertainty or confidence matches how often it is right. Measured with ECE, RMS calibration error or NLL, or set as a conformal threshold for a success rate. Sources are author-reported and use different binning and settings, so their numbers are not directly comparable. SNGP, MC dropout, ensembles and GPR differ in over- and underconfidence. tags: [uncertainty, evaluation, regression, conformal-prediction, label-noise, ood-detection, fairness, forecasting] papers: ["1910.12656", "1912.06883", "2006.10108", "2104.08315", "2107.03342", "2201.07766", "2205.09310", "2206.01653", "2207.05221", "2305.04933", "2307.01928", "2403.12945", "2501.14249", "2510.17638", "2610.17635"] related: [uncertainty-estimation, monte-carlo-dropout, ensembles, conformal-prediction, temperature-scaling, out-of-distribution-detection, llm-evaluation, llm-self-evaluation, hallucination-detection, segmentation-evaluation, noisy-labels, multi-label-classification, algorithmic-fairness, benchmark-validity, likelihood-based-text-evaluation, zero-shot-learning] status: draft updated_at: 2026-11-01 --- ## Overview Calibration asks whether a model's stated uncertainty is honest: when it reports a confident prediction, is it right about as often as it claims? Several sources treat this as a property to measure alongside accuracy. A tutorial notes that most benchmarks score accuracy and rarely score the quality of the stated uncertainty [arxiv:2305.04933 §1]. The HLE paper makes the same point for language models: high accuracy alone does not show whether a model knows when it is wrong, so it asks for calibrated confidence [arxiv:2501.14249 §4.2]. Calibration is measured in several ways. Expected calibration error (ECE) compares confidence with accuracy in bins [arxiv:2305.04933 §6.1] [arxiv:2006.10108 tab. 2], and the negative log-likelihood (NLL) is reported next to it [arxiv:2305.04933 §6.1]. A benchmark can also use RMS calibration error over verbalised confidence [arxiv:2501.14249 §4.2]. A survey of uncertainty in deep networks adds two all-label variants, SCE and an equal-mass ECE (aECE), and reports that the all-label measures are more effective than top-label ECE, citing other work [arxiv:2107.03342 §5.2]. A scientific-ML framework uses RMSCE over Gaussian predictive quantiles: it compares the fraction of test values inside each quantile with that quantile's probability [arxiv:2201.07766 §5.1]. Calibration is one part of uncertainty estimation: a method can give a spread and still be over- or underconfident [arxiv:2305.04933 §6.1]. The sources use calibration in a second sense. KnowNo, a language model planner for robots, uses a labelled calibration set to pick a score cut-off, so that its actions reach a success rate the user sets [arxiv:2307.01928 §3.1]. Its guarantee is statistical and holds under stated assumptions, the main one being that calibration and test scenarios come from the same distribution [arxiv:2307.01928 §3.2]. A third use appears in a 2026 paper on noisy multi-label training data. Its calibrator maps a histogram of model confidences on candidate labels to the chance that each candidate is correct, and that chance becomes the training weight for the label [arxiv:2610.17635 §4.1] [eq. 3] [eq. 4]. The evidence is narrow. The Dirichlet calibration paper covers tabular classifiers on 21 UCI datasets and image classifiers on CIFAR-10, CIFAR-100 and SVHN [arxiv:1910.12656 §4.1] [arxiv:1910.12656 §4.2]. The tutorial's case studies cover regression on engineering data (battery life and turbofan remaining life). The SNGP paper covers image classification and intent detection with accuracy-based evaluation [arxiv:2006.10108 §5]. KnowNo covers robot task planning with hardware trials. The noisy multi-label paper covers image classification with simulated label noise on four datasets [arxiv:2610.17635 §5.1]. HLE covers closed-ended expert questions. The language-model self-evaluation study covers trivia, word prediction, arithmetic, grade-school maths and code tasks, all with short answers that can be checked [arxiv:2207.05221 §5]. Prophet Arena scores forecasts for 1,367 resolved binary prediction-market events, most of them sports, for 23 LLMs and a market baseline [arxiv:2510.17638 §3.2.2]. The survey has no experiments of its own and quotes its numbers from cited work [arxiv:2107.03342 §3.3]. The zero-shot scoring paper reports accuracy on multiple-choice tasks and no calibration measure [arxiv:2104.08315 tab. 1]. None of the sources covers open-ended text generation. All results are author-reported. The tutorial also separates two kinds of doubt that a calibrated model should report. Aleatory uncertainty is noise that more data cannot remove. Epistemic uncertainty comes from limited data and can shrink with more data [arxiv:2305.04933 §2]. The word also names a different task in one source: estimating where a sensor is. The DROID dataset paper releases camera poses for its scenes, a camera-to-base set and a camera-to-camera set, computed after collection [arxiv:2403.12945 §3.2] [app. G.5]. That is geometric sensor calibration, not confidence calibration, so none of the numbers on this page apply to it. For the sensor task, see [Hand-eye calibration](/concepts/hand-eye-calibration). ## Variants Several readings of calibration appear in the sources. Most measure how honest a model's stated uncertainty or confidence is. The conformal variant sets a threshold so that a decision meets a target. The post-hoc and self-evaluation variants are ways to change or read a model's confidence. They are not alternatives to each other; a study usually uses several. ### Reliability curves and ECE Plot predicted confidence against observed frequency, and summarize the gap as ECE. The tutorial reports ECE in percent per dataset split [arxiv:2305.04933 §6.1] [arxiv:2305.04933 §6.2]. It does not state the bin count, so the bin count is not stated here. The SNGP paper uses 15 bins [arxiv:2006.10108 app. C] and reports ECE on CIFAR-10, CIFAR-100 and CLINC [arxiv:2006.10108 tab. 2] [arxiv:2006.10108 tab. 3] [arxiv:2006.10108 tab. 4]. The language-model study takes top-label ECE over 10 equal-count bins, and computes it only from the top prediction of each multiple-choice question [arxiv:2207.05221 app. A.2]. The survey defines the same top-label ECE over equal-width bins and its SCE and aECE variants over all labels [arxiv:2107.03342 §5.2]. ECE values depend on the binning and on which probabilities enter, so compare them only when both match. Prophet Arena uses binned ECE with 10 equal-width bins over every market's forecast, pooled across events [arxiv:2510.17638 app. B.2]. Its values are covered in the forecasting reading below. See the Contradictions section and the [UQ tutorial card](/papers/2305.04933). ### RMS calibration error over verbalised confidence HLE asks each model to state a confidence from 0% to 100% in its answer [arxiv:2501.14249 app. C.1]. Its calibration metric is the root of a weighted sum of squared gaps between mean stated confidence and mean accuracy in bins [arxiv:2501.14249 §4.2]. The paper does not state the bin size. The reference code uses 100 items per bin, and the [HLE card](/papers/2501.14249) reads from that code that the last bin is left out of the sum; this was not run. The reported values for eight models are 73% to 89% [arxiv:2501.14249 tab. 1]. Each model's accuracy on that set is 2.7% to 13.4%, so these errors come with very low accuracy and are not comparable to ECE values from the other sources. ### NLL as a joint score NLL rewards both accuracy and a well-sized spread, so a model cannot lower it by being vague alone. The tutorial reports NLL next to RMSE and ECE for every method [arxiv:2305.04933 §6.1] [arxiv:2305.04933 §6.2]. Its Gaussian negative log-likelihood is the training loss for the Gaussian-head ensemble [arxiv:2305.04933 §3.3]. The SNGP paper reports NLL on CIFAR and CLINC next to ECE [arxiv:2006.10108 tab. 2] [arxiv:2006.10108 tab. 4]. ### Uncertainty methods scored for calibration The tutorial compares four families on calibration, and the SNGP paper adds a more detailed look at one of them: - **Gaussian process regression (GPR).** Its posterior variance grows far from training inputs, which gives distance-aware uncertainty [arxiv:2305.04933 §3.1]. Exact GPR costs O(N³) in training points, so it was excluded from the turbofan case study [arxiv:2305.04933 §6.2]. On the battery data it fit the training set exactly but had high uncertainty on the test sets [arxiv:2305.04933 §6.1]. - **MC dropout.** Dropout stays on at prediction time and the spread over passes gives the uncertainty [arxiv:2305.04933 §3.2]. The tutorial reports it as overconfident, with the largest run-to-run variation of the group [arxiv:2305.04933 §6.1] [arxiv:2305.04933 §6.2]. Its behaviour depends on dropout rate, number of dropout layers and training length [arxiv:2305.04933 §3.2]. See [Monte Carlo dropout](/concepts/monte-carlo-dropout). - **Gaussian-head ensemble.** Several networks each predict a mean and a variance; their disagreement is the epistemic part [arxiv:2305.04933 §3.3]. On the battery data it had the lowest NLL on each test set, as the author-reported summary states [arxiv:2305.04933 §6.1]. See [Ensembles](/concepts/ensembles) and [Uncertainty estimation](/concepts/uncertainty-estimation). - **SNGP.** One deterministic network with spectral normalization on its hidden weights and a Gaussian-process output layer [arxiv:2006.10108 §3.1] [arxiv:2006.10108 §3.2]. The SNGP paper reports it close to a 10-model deep ensemble on calibration and OOD detection, at single-model latency. On CIFAR-10 clean data, ECE is 0.018 for SNGP, 0.010 for the ensemble and 0.023 for the deterministic network [arxiv:2006.10108 tab. 2]. Latency is 6.25 ms per example for SNGP against 38.10 ms for the ensemble, on the paper's hardware [arxiv:2006.10108 tab. 2]. The tutorial reports SNGP as often underconfident on the battery data, with high NLL and ECE there [arxiv:2305.04933 §6.1], and says that SNGP-type methods cannot separate the two kinds of uncertainty [arxiv:2305.04933 §3.4]. Card for the SNGP method: [SNGP](/papers/2006.10108). Its precision update, initialization and sign of the random projection are inconsistent between the paper's sections, so the card lists what to check before reimplementing. The survey summarises a wider comparison from cited work. Deep ensembles are reported as at least as good as Bayesian baselines, such as MC dropout, on accuracy, calibration and out-of-distribution detection, and as more robust to dataset shift [arxiv:2107.03342 §3.3]. These are the survey's summaries, not its own measurements, so check the original studies before relying on them. The same section reports that bagging can worsen uncertainty in ensembles [arxiv:2107.03342 §3.3]. ### Temperature scaling after training Temperature scaling is a post-hoc fix. The softmax is applied to the logits divided by a single positive temperature T, and T is fitted after training. T above 1 lowers the confidence; the predicted class does not change. The survey notes that the method's expressiveness is limited [arxiv:2107.03342 §5.1]. It also lists other post-hoc options: a class-wise one-vs-all method, scaling of the standard deviation for regression, and a Gaussian-process calibration map [arxiv:2107.03342 §5.1]. The survey gives no benchmark numbers for these, so none are quoted here. The Dirichlet paper compares the fitted multiclass maps against temperature scaling directly, with the numbers in the variant below [arxiv:1910.12656 tab. 1] [arxiv:1910.12656 tab. 2]. The language-model study uses a fixed temperature of 2.5 for its RLHF models, which it reports as fixing calibration on several RLHF evaluations. It warns that more intensive RL could break this [arxiv:2207.05221 §3.3]. This is a single model-level setting, not a fitted calibration map, so it should not be confused with temperature scaling above. ### Logit normalization during training This reading changes the training loss, not the confidence after training. The network is trained with cross-entropy on its logit vector divided by that vector's L2 norm and by a temperature tau. Every training logit vector therefore has length 1/tau [arxiv:2205.09310 §3.2] [eq. 4]. The authors argue that ordinary cross-entropy keeps growing logit norms during training, for familiar and unfamiliar inputs alike, and that this drives the overconfidence seen on out-of-distribution (OOD) inputs [arxiv:2205.09310 §3.1] [fig. 1]. The architecture and the test-time score are unchanged; the paper scores inputs with the maximum softmax probability on the raw output [arxiv:2205.09310 §4.2]. The tau here is set during training and is not the post-hoc [temperature scaling](/concepts/temperature-scaling) factor; the paper contrasts the two in its discussion [arxiv:2205.09310 §3.2] [§5]. For OOD detection, the author-reported result on CIFAR-10 with WRN-40-2 is that the average FPR95 over six OOD sets falls from 49.52% to 15.65%, with ID accuracy close to cross-entropy [arxiv:2205.09310 tab. 1] [tab. 3]. The paper also applies other OOD scores to the same LogitNorm networks. On CIFAR-10, ODIN's average FPR95 is 12.95%, below MSP's 15.65%, while energy (19.14%) and GradNorm (17.78%) are higher [arxiv:2205.09310 tab. 2]. Card: [LogitNorm](/papers/2205.09310). The calibration numbers in the paper depend on a second step. On CIFAR-10, expected calibration error (15 bins) is 66.95% for LogitNorm without temperature scaling, against 3.20% for cross-entropy. With temperature scaling, it is 0.41% against 0.66% [arxiv:2205.09310 tab. 4]. The text does not discuss the without-scaling row [arxiv:2205.09310 tab. 4]. So the method on its own gives poorly calibrated probabilities, and any calibrated output needs the post-hoc fit as well. The tau search and the scope limit the evidence. The paper picks tau for CIFAR-10 from a grid running from 0.001 to 0.05, with 0.04 as the default, using Gaussian noise as an OOD validation set [arxiv:2205.09310 §4.1]. The experiments cover CIFAR-10 and CIFAR-100 only [arxiv:2205.09310 §4.1], and the paper calls tuning expensive because it needs several training runs [arxiv:2205.09310 app. Future Work]. ### Recalibrating a predictive spread on a left-out set For a model that outputs a mean and a variance, a left-out calibration set can correct the spread after training. The scientific-ML framework describes three post-training methods [arxiv:2201.07766 §5.2]. Variance scaling multiplies the predicted variance by one constant, chosen by optimising a metric, and leaves the mean unchanged. A CDF correction fits a map to the observed calibration curve and applies it to every predictive CDF, which changes both mean and variance [app. "CDF modification by fitting an auxiliary model"]. A third method fits an empirical distribution to the scaled residuals of the calibration set [app. "CRUDE"]. The paper reports that post-training calibration makes all its methods roughly equally calibrated on a 32-point function-approximation problem, though the calibrated RMSCE row of its table still spans 3.8 to 8.0 [arxiv:2201.07766 §6.1]. On a mixed stochastic PDE, calibration on two to ten left-out realizations reduced RMSCE by about a factor of three in one case [arxiv:2201.07766 §6.3]. These are author-reported results on prototype problems. The calibration data come from the training distribution, so performance may get worse on out-of-distribution tests [arxiv:2201.07766 §5.2]. ### Multiclass probability maps after training This reading covers post-hoc maps for any classifier that outputs class probabilities, not only neural networks. The Dirichlet calibration paper argues that per-class binary fixes, such as one-vs-rest Platt scaling or isotonic regression, are not guaranteed to give calibrated multiclass outputs once normalised [arxiv:1910.12656 §1]. Its map takes the logarithm of each probability vector, applies one learned k-by-k linear map and a bias, then a softmax [arxiv:1910.12656 §3] [eq. 7]. The map is fitted on held-out labelled data by multinomial logistic regression [arxiv:1910.12656 §3]. Temperature scaling is a special case of this family, with a diagonal weight matrix and no bias [arxiv:1910.12656 §3] [app. B]. Two regularised fits are described. Dirichlet L2 applies one L2 penalty to all weights [arxiv:1910.12656 §3]. ODIR penalises the off-diagonal weights and the intercept separately, each normalised by its number of terms [arxiv:1910.12656 §3]. The paper's ODIR objective is not numbered, and the paper gives no grid for its regularisation strengths [arxiv:1910.12656 §3]. The released code uses a six-value grid for ODIR [code: calib/models/calibration.py#L197] and an eleven-value grid for L2 [code: calib/models/calibration.py#L196]. The Keras wrapper in the code does not apply the paper's normalisation to the penalties, so its strengths are not interchangeable with the paper's. See the [Dirichlet calibration card](/papers/1910.12656) for the full list before reimplementing. Results are author-reported. On 21 UCI datasets with 11 classical classifiers (231 settings), Dirichlet L2 has the best average log-loss rank, 2.25, against 2.92 for beta calibration and 4.61 for temperature scaling [arxiv:1910.12656 tab. 2]. Its average rank on p-cw-ECE is 2.34, against 3.15 for beta calibration and 4.37 for temperature scaling [arxiv:1910.12656 tab. 1]. The p-cw-ECE measure checks every class, not only the top prediction. On 14 deep-network cases, the paper's summary ranks put Dir-ODIR at 2.79 on cw-ECE, against 3.71 for temperature scaling, and at 2.93 on log-loss, against 3.50 [arxiv:1910.12656 tab. 3] [arxiv:1910.12656 tab. 4]. These two deep-network results rank the three general-purpose methods only, and the paper says the Dir-ODIR lead is not statistically significant there [arxiv:1910.12656 §4.2]. Matrix and vector scaling act on logits, so they are not general-purpose methods. Vector scaling has the best average cw-ECE rank among all six deep-network methods, 2.29 [arxiv:1910.12656 tab. 3]. ### Self-evaluated probabilities in language models This reading asks a language model to judge its own answers. The model samples an answer at temperature 1, then is asked whether that answer is "(A) True" or "(B) False". The probability on True, P(True), is the score [arxiv:2207.05221 §4.1]. Zero-shot, the score is poorly calibrated and sits near 50% for typical samples [arxiv:2207.05221 §4.1] [app. B]. Showing five samples to the judge, one of them the one being judged, is the variant the authors report as best with few-shot prompting [arxiv:2207.05221 §4.2] [fig. 11]. The figure carries those results; the text gives few exact values. A second score, P(IK), trains a value head to predict whether the model knows an answer before it answers [arxiv:2207.05221 §5] [app. A.6]. For the 52B model trained and tested on TriviaQA, AUROC is 0.864 and the Brier score is 0.151 [arxiv:2207.05221 tab. 1]. Transfer is partial. When trained on TriviaQA only, AUROC on Lambada, GSM8k and Python function synthesis is 0.606, 0.624 and 0.687 [arxiv:2207.05221 tab. 1]. Calibration on those tasks is poor, and models are often underconfident there [arxiv:2207.05221 fig. 13]. Training on all tasks except GSM8k raises the AUROC on Lambada to 0.853 [arxiv:2207.05221 tab. 1]. Adding a relevant Wikipedia article to a question raised P(IK) from 18% to 78% in the paper's one example [arxiv:2207.05221 §5.3]. The authors' models are pretrained models of 0.8B to 52B parameters [arxiv:2207.05221 §1.2]. The paper releases no code or weights, so these numbers are author-reported and cannot be reproduced from the paper alone [arxiv:2207.05221 §7]. The P(IK) head needs white-box access to add a head and train it with the model [arxiv:2207.05221 app. A.6]. ### Conformal thresholds on option scores KnowNo rewrites a planner's next-step choice as a multiple-choice question. The language model's confidence in each option is normalized over the options. On a calibration set of labelled scenarios, the score of each scenario is one minus the confidence of the correct option. The cut-off is the empirical quantile of those scores at level ⌈(N+1)(1−ε)⌉/N [arxiv:2307.01928 §3.1] [eq. 1]. For a new scenario, the prediction set keeps every option whose confidence is at least 1 − q̂. The robot acts when the set has one option and asks a person otherwise [arxiv:2307.01928 §3.2]. The paper gives two forms of the guarantee. The marginal form [eq. 1] holds on average over the calibration set. The dataset-conditional form [eq. 2] holds with probability 1 − δ over the calibration set and uses an adjusted ε̂. The paper uses N = 400 and δ = 0.01 [arxiv:2307.01928 §3.1] [app. A.7]. For multi-step tasks, whole episodes are scored by the minimum per-step confidence of the correct option, and the prediction set is rebuilt at each step [arxiv:2307.01928 §3.3] [alg. 1]. The bound does not cover the contexts reached after a wrong earlier action [arxiv:2307.01928 app. A.2]. Reported results, author-reported. On a hardware mobile manipulator with a target of 0.85, KnowNo asked for help on 0.67 of decisions against 0.81 for the Simple Set baseline, at similar plan success (0.87) [arxiv:2307.01928 tab. 2]. On hardware tabletop rearrangement, step-wise help was 0.58 against 0.72 [arxiv:2307.01928 tab. 1]. Task success stayed below the target on mobile manipulation (0.76 against 0.85). The authors attribute the gap to low-level execution [arxiv:2307.01928 app. A.8]. The released notebooks differ from the paper in several ways: they use GPT-3.5 rather than PaLM-2L, use calibration sets of 100 or 200 rather than 400, scale the option log-probabilities by a temperature, and implement only the marginal form. See the [KnowNo card](/papers/2307.01928), which lists each difference. ### Calibrated label weights This reading is a training method. Candidate positive labels in a noisy multi-label dataset are weighted by a calibrated estimate of being correct. The weight is the probability that a candidate is a true positive, given the model's confidence in it [arxiv:2610.17635 §3] [eq. 3] [eq. 4]. The paper fits this map with a pre-trained two-layer network, the calibrator. It reads a 20-bin histogram of confidences on candidate positives and outputs one reliability per bin [arxiv:2610.17635 §4.1] [eq. 5] [eq. 6]. A single confidence is mapped to a weight by linear interpolation between neighbouring bins [eq. 7]. The calibrator is trained on VOC 2007 at noise rates from 10% to 90%, and reused on the other datasets [arxiv:2610.17635 §4.1] [§5.1]. Results are author-reported. On COCO at 80% noise, the ECE of the weights falls from 76.78% under identity weighting to 3.74% with the calibrator, and mAP rises from 59.93 to 61.48 [arxiv:2610.17635 tab. 2]. The main table reports the method as best in 15 of 16 settings against nine baselines, with mAP as the metric [arxiv:2610.17635 §5.2] [tab. 1]. These are not calibration results. The oracle weighting that uses true labels has the same weight ECE, 3.74, which the paper does not discuss [arxiv:2610.17635 tab. 2]. The paper's own tables also disagree on VOC: the PML-CD and "2nd phase only" rows of [tab. 1] and [tab. 3] differ at three of four noise rates [arxiv:2610.17635 tab. 1] [tab. 3]. The released code also picks the best epoch on the test split, which can make reported numbers optimistic [see the PML-CD card](/papers/2610.17635). Use the calibrator as a reference for the method, not as a verified recipe. ### Calibration as a reported metric For image-level classification, the Metrics Reloaded framework adds a calibration metric to the pool of validation metrics only when calibration is requested [arxiv:2206.01653 §ImLC]. The framework is a consensus decision guide, not a benchmark, and the card does not name the calibration measure it would choose [arxiv:2206.01653 §ImLC]. Use it to decide which metrics to report, not which calibration formula to implement. ### Calibration of LLM probability forecasts Prophet Arena asks each model for a probability that every binary market of an event resolves Yes, then scores the answers after resolution. It reports three families: the Brier score, binned ECE, and the average return from betting one unit per market at the market price [arxiv:2510.17638 §3.1] [eq. 1]. The prompt includes the live market price, so the market is an input to the model as well as the reference point [arxiv:2510.17638 app. E]. Author-reported values over 23 LLMs: ECE runs from 0.030 (o3, reasoning) to 0.165 (DeepSeek-R1, reasoning), and the market baseline is 0.069 [arxiv:2510.17638 app. C.1]. The paper says every highlighted model is better calibrated than the market [arxiv:2510.17638 §3.2.2]. Brier scores are close to the market's 0.187; the best listed is 0.184 [arxiv:2510.17638 app. C.1]. No model reaches an average return of 1, and the highest is 0.982 [arxiv:2510.17638 app. C.1]. The paper finds the models give less extreme probabilities than the market for events that resolve Yes; the paper calls this conservatism [arxiv:2510.17638 §4.2.3]. Elicitation changes ECE. The paper's bi-direction method, which averages a Yes probability with one minus a No probability, lowers ECE for four of five models tested, with Claude Sonnet 4 going from 0.046 to 0.028 [arxiv:2510.17638 app. C.3]. GPT-5 is the exception, at 0.020 with the default prompt and 0.023 with bi-direction. Self-consistency over 10 samples gives values in steps of 0.1 when unweighted, so it cannot represent fine probabilities [arxiv:2510.17638 app. C.3]. Scoring depends on the setting. Results are for 1,367 events, 76% of them sports, with forecasts taken at scheduled times before close and none within three hours of resolution [arxiv:2510.17638 §3.2.1] [§3.2.2]. Calibration of the same models on other event mixes is not reported. The paper also notes that the ECE equation is typeset in a way that differs from the estimator it used, so use the binned version in appendix B.2 [arxiv:2510.17638 app. B.2]. ### Prior correction in zero-shot scoring A different use of the word appears in the zero-shot scoring paper. Domain Conditional PMI scores each answer option as its likelihood given the question, minus its likelihood given a short task-specific domain premise, which estimates how likely the option is in that domain regardless of the question [arxiv:2104.08315 §3.2] [eq. 2]. This removes the penalty that many valid wordings put on a correct answer, and needs no training. The paper reports accuracy, not calibration: it does not measure whether the scores are well-calibrated probabilities [arxiv:2104.08315 tab. 1] [tab. 2]. Its best- or-tied rate on 16 splits is 62.5% for GPT-3 175B, against 18.75% for length-normalised log-likelihood [arxiv:2104.08315 tab. 2]. The prior correction is related to contextual calibration (Zhao et al.), which the paper compares against with numbers copied from that work for five datasets [arxiv:2104.08315 §3.3] [tab. 1]. That work has no card here, so nothing from it is used on this page. Card: [Surface Form Competition](/papers/2104.08315). ### Group calibration as a fairness criterion A fairness reading asks whether a model's stated confidence means the same thing for each protected group. Binns treats equal calibration as a group measure that tolerates differences in base rates between groups [arxiv:1912.06883 §5.1] [tab. 2]. The paper pairs it with a raw similarity metric under a "choice" view of disparities, and pairs statistical parity with a group-adjusted similarity metric under a "structural" view [arxiv:1912.06883 §5.3] [tab. 2]. The paper's argument is that the assumption about where disparities come from should be chosen first, and the measure follows from it [arxiv:1912.06883 §6]. The paper also cites prior work showing that false positive equality and calibration cannot both hold in the COMPAS case [arxiv:1912.06883 §5.3]. That result is cited, not proven in the paper. The paper contains no experiments and reports no calibration numbers, so this reading gives a framing, not measured values. Card: [On the Apparent Conflict Between Individual and Group Fairness](/papers/1912.06883). ## When to use which The sources give regression, image, robot-planning and language-model setups, so these rows follow the reported behaviour and its stated costs. They are not general rules. | Situation | Use | Why | |---|---|---| | Small training set, need distance-aware uncertainty | GPR, with the O(N³) cost in mind [arxiv:2305.04933 §3.1] | Variance grows away from the data; cost rules out large sets [arxiv:2305.04933 §6.2] | | Overconfidence is the costly error | Not settled by the source; avoid MC dropout alone at low dropout rates [arxiv:2305.04933 §6.1] | The tutorial reports overconfidence and instability for it | | Underconfidence is the safer error (the tutorial's battery case) | Ensemble or SNGP [arxiv:2305.04933 §6.1] | The tutorial calls underconfidence the safer error for battery early-life prediction and says so for both methods | | Budget allows several trainings, want the lowest NLL in the study | Gaussian-head ensemble [arxiv:2305.04933 §6.1] | Lowest NLL on each battery test set in the author-reported results; M = 15 in the paper [arxiv:2305.04933 §6.1] | | Single pass at inference, can change the architecture | SNGP [arxiv:2006.10108 §3.2] [arxiv:2006.10108 tab. 2] | One forward pass; residual hidden layers and a spectral norm bound that must be tuned [arxiv:2305.04933 §3.4] | | Need to split noise from limited-data uncertainty | Ensemble formula with the variance split [arxiv:2305.04933 §3.3] | SNGP and similar methods cannot split the two parts [arxiv:2305.04933 §3.4] | | Planner picks among listed options, its option probabilities are readable, about 400 labelled scenarios are affordable, and a stated success rate is needed | Conformal thresholds on option scores [arxiv:2307.01928 §3.1] | Success level holds under the paper's assumptions; labelling cost was about 4 hours for multi-step and 1.5 hours for single-step calibration [arxiv:2307.01928 §4] | | Regression model that outputs a spread, a small labelled left-out set from the same distribution, and error bars that do not match the errors | Post-training recalibration of the spread: scaling, CDF correction or residual fit [arxiv:2201.07766 §5.2] | Author-reported drop in RMSCE on prototype problems [arxiv:2201.07766 §6.3]; may not hold out of distribution [§5.2] | | Standard cross-entropy classifier that can be retrained, and OOD detection by confidence is the goal | Logit normalization in the training loss [arxiv:2205.09310 §3.2] [eq. 4] | Author-reported FPR95 drop on CIFAR-10 and CIFAR-100 with WRN-40-2 [arxiv:2205.09310 tab. 1]; tested only on CIFAR [§4.1] | | Need calibrated probabilities from a retrained net | Add a post-hoc [temperature scaling](/concepts/temperature-scaling) fit after logit normalization | Without it, the paper's ECE is 66.95% on CIFAR-10 [arxiv:2205.09310 tab. 4]; with it, 0.41% | | Model already trained, or its loss cannot change | Not covered by the logit normalization paper | Every network in the paper is trained with the new loss, so it gives no evidence for a post-training change [arxiv:2205.09310 §4.1] | | Model does not expose token probabilities | Conformal option scores as written do not apply | The paper's method needs option probabilities; it did not use GPT-4 because its token probabilities are not exposed [arxiv:2307.01928 §4.3] | | Model shows strong bias toward some option letters | Measure option-letter bias before choosing the model | GPT-3.5 favoured options D and E, which gave set size 2.50 and help 0.86 [arxiv:2307.01928 tab. 2] | | Image-level classification, and calibration must be reported | Add a calibration metric next to the per-class and multi-threshold metrics [arxiv:2206.01653 §ImLC] | The framework adds calibration when requested; the measure is not named in the card | | Multi-label training data with false positive tags (for example crowd labels), and a calibrator can be pre-trained on another labelled set | Calibrated label weights [arxiv:2610.17635 §4.1] | Author-reported best mAP in 15 of 16 settings [tab. 1]; weight ECE checked on COCO at one noise rate only [tab. 2]; the calibrator is reused across datasets | | Closed-ended questions answered by an LLM with verbalised confidence | RMS calibration error alongside accuracy [arxiv:2501.14249 §4.2] | Error is 73% to 89% while accuracy is 2.7% to 13.4% [arxiv:2501.14249 tab. 1]; read them together | | Trained classifier, held-out labelled data, want a cheap confidence fix | Temperature scaling [arxiv:2107.03342 §5.1] | One fitted scalar; keeps the predicted class; the survey says its expressiveness is limited | | Any classifier with class probabilities, held-out labelled data, want calibrated probabilities for every class | Dirichlet L2 map [arxiv:1910.12656 §3] [eq. 7] | Best average log-loss rank over 21 UCI datasets and 11 classifiers [arxiv:1910.12656 tab. 2]; on the deep networks it ranks behind temperature scaling, so test on your own model [arxiv:1910.12656 tab. 3] [tab. 4] | | Neural network, pre-softmax logits available, held-out labelled data | Vector scaling, or matrix scaling with ODIR [arxiv:1910.12656 tab. 3] [tab. 4] | Best average cw-ECE rank among the six deep-network methods for vector scaling; the paper notes that these methods use logits and are not general-purpose [arxiv:1910.12656 §4.2] | | Language model, lettered multiple-choice question, option probabilities readable | Read the option probabilities as confidence, and check with top-label ECE over 10 equal-count bins [arxiv:2207.05221 §2] [app. A.2] | Calibration is reported as good for 52B models on this format; results are for the authors' own models | | Language model must grade its own sampled answer, five samples per question affordable | P(True) with comparison samples, few-shot [arxiv:2207.05221 §4.2] [fig. 11] | Reported as the best variant; zero-shot is poorly calibrated [arxiv:2207.05221 §4.1] | | Want a signal before answering, white-box access, labelled answers for your own task | P(IK) value head [arxiv:2207.05221 §5] [app. A.6] | Good in distribution on TriviaQA; partial transfer and poor calibration on other tasks [arxiv:2207.05221 tab. 1] [fig. 13] | | Comparing calibration numbers across papers | Match the binning and the choice of top-label or all-label measure first [arxiv:2107.03342 §5.2] [arxiv:2207.05221 app. A.2] | Top-label ECE, SCE and aECE are defined differently and do not give the same values | | Group fairness audit under a "choice" view of disparities, with a group-level calibration target | Equal calibration as the group measure, paired with a raw similarity metric [arxiv:1912.06883 §5.3] [tab. 2] | The paper pairs these under this assumption and gives no numbers; under a "structural" view it pairs statistical parity instead | | LLM gives a probability for a binary future event, and you can score it after resolution | Brier score, binned ECE and market return together [arxiv:2510.17638 §3.1] | Models were within a narrow Brier band of the market, and none broke even on return; prices in the prompt make the market baseline a reference point [app. E] | | Causal LM must pick one option from a list, no labels, option wordings vary in count | Domain Conditional PMI over the option likelihood [arxiv:2104.08315 §3.2] [eq. 2] | Best or tied on 62.5% of splits for GPT-3 175B; accuracy only, so calibration is unmeasured [arxiv:2104.08315 tab. 2] | | Regression beyond the tutorial's case studies, or open-ended text | Not covered by these sources | See [Uncertainty estimation](/concepts/uncertainty-estimation) and [Hallucination detection](/concepts/hallucination-detection) for other sources | ## Implementation sketch The first sketch covers the ensemble combination, the step that produces the stated uncertainty. The second covers the conformal threshold. The third is binned ECE with equal-width bins, the estimator Prophet Arena used [arxiv:2510.17638 app. B.2]; the other sources bin differently, see the Contradictions section. Names are ours. For RMS calibration error, see the [HLE card](/papers/2501.14249) and its Pitfalls on the bin loop. ```pseudocode # Combine M Gaussian-head members into one predictive mean and variance # [arxiv:2305.04933 §3.3] # Input: M members, each giving (mu_m(x), var_m(x)) for one input x # Output: combined mean and variance def combine_members(mus, variances): n = len(mus) # Average of the members' means [arxiv:2305.04933 §3.3] mean = sum(mus) / n # Average of (member variance + member mean squared), minus mean squared # [arxiv:2305.04933 §3.3] second_moment = sum(v + mu * mu for v, mu in zip(variances, mus)) / n return mean, second_moment - mean * mean ``` ```pseudocode # Conformal cut-off from labelled calibration scenarios [arxiv:2307.01928 §3.1] # Input: calib_conf (N, options) normalized option confidences # Input: calib_label (N,) index of the correct option; epsilon in (0, 1) # Output: qhat, the cut-off for prediction sets def conformal_cutoff(calib_conf, calib_label, epsilon): n = len(calib_label) # Score is one minus the confidence of the correct option [§3.1] scores = 1 - calib_conf[range(n), calib_label] # Empirical quantile at level ceil((N+1)(1-eps))/N [eq. 1] level = min(1.0, ceil((n + 1) * (1 - epsilon)) / n) return quantile(scores, level, method="higher") ``` ```pseudocode # Binned expected calibration error, equal-width bins [arxiv:2510.17638 app. B.2] # Input: probs (M,) forecast probabilities pooled over markets # Input: outcomes (M,) 0 or 1; bins: number of equal-width bins (paper: 10) # Output: ECE in [0, 1] def binned_ece(probs, outcomes, bins=10): total = 0.0 for b in range(bins): # Bin b covers [b/bins, (b+1)/bins); the last bin also takes 1.0 members = [j for j in range(len(probs)) if in_bin(probs[j], b, bins)] if not members: continue # Weight each bin's gap by its share of the members [app. B.2] gap = abs(mean(outcomes[j] for j in members) - mean(probs[j] for j in members)) total = total + len(members) * gap return total / len(probs) ``` ## Contradictions Disagreements between papers, each side cited. The tutorial does not resolve them, and they affect how far the ranking above can be trusted. - **Turbofan ECE.** The text says the ensemble (NNE) has low ECE on turbofan data [arxiv:2305.04933 §6.2]. The results table gives NNE 14.9% on unit 15, above MC dropout's 11.5% and SNGP's 8.5% [arxiv:2305.04933 §6.2 results table]. - **Battery ECE on other sets.** The text says GPR gives the least ECE on the other datasets [arxiv:2305.04933 §6.1]. On the secondary test set MC dropout's ECE is 9.5% and GPR's is 10.4% [arxiv:2305.04933 §6.1 results table]. - **Battery RMSE and MSE.** The battery table caption says MSE, while its rows say RMSE [arxiv:2305.04933 §6.1 results table]. The card treats them as RMSE. - **SNGP on battery data and on image benchmarks.** The tutorial finds SNGP underconfident with high ECE and NLL on battery data [arxiv:2305.04933 §6.1]. The SNGP paper finds its ECE close to the ensemble's on CIFAR-10 and CLINC [arxiv:2006.10108 tab. 2] [arxiv:2006.10108 tab. 4]. These are different datasets and tasks, so neither result transfers to the other setting. - **Numbers that are not comparable.** ECE in the tutorial, ECE with 15 bins in the SNGP paper, top-label ECE with 10 equal-count bins in the language-model study [arxiv:2207.05221 app. A.2], binned ECE with 10 equal-width bins over market forecasts in Prophet Arena [arxiv:2510.17638 app. B.2], and RMS calibration error with bins of 100 items in HLE [arxiv:2501.14249 §4.2] measure different things. Do not compare their values across papers. - **GPT-5 row in Prophet Arena.** The highlighted table labels the row GPT-5 (reasoning) with Brier 0.184, ECE 0.042 and return 0.943 [arxiv:2510.17638 tab. 2]. The full table gives those same values to the row GPT-5 (reasoning, High), and gives GPT-5 (reasoning) 0.187, 0.044 and 0.890 [arxiv:2510.17638 app. C.1]. The paper does not say which setting the highlighted row uses. The page quotes the full table by row name. - **Lettered multiple-choice calibration against verbalised error.** The language-model study reports that 52B models are well calibrated on lettered multiple-choice questions, using option probabilities [arxiv:2207.05221 §2] [app. A.1]. HLE reports RMS calibration error of 73% to 89% for verbalised confidence on closed-ended questions, where accuracy is 2.7% to 13.4% [arxiv:2501.14249 tab. 1]. The inputs differ (option probabilities against stated confidence, multiple choice against closed-ended exact answers), so the two do not test the same thing. The page does not resolve them. - **Temperature scaling against Dirichlet maps, by setting.** On 21 UCI datasets, Dirichlet L2 ranks ahead of temperature scaling on log-loss (2.25 against 4.61) and on p-cw-ECE (2.34 against 4.37) [arxiv:1910.12656 tab. 1] [arxiv:1910.12656 tab. 2]. On the paper's 14 deep-network cases, Dirichlet L2 ranks behind temperature scaling on both cw-ECE (3.79 against 3.71) and log-loss (3.79 against 3.50) [arxiv:1910.12656 tab. 3] [arxiv:1910.12656 tab. 4]. The paper reports both results, and the difference depends on the setting. - **All-label against top-label measures.** The survey reports that all-label measures (SCE) assess calibration error more effectively than top-label ECE, citing Nixon et al. [arxiv:2107.03342 §5.2]. The language-model study uses top-label ECE only [arxiv:2207.05221 app. A.2] and does not test the all-label claim. The survey's statement is cited here, not checked. - **KnowNo guarantee, paper against its code.** The paper states the dataset-conditional bound with an adjusted ε̂ [arxiv:2307.01928 §3.1]. The released notebooks use the requested ε directly, which gives only the marginal form. The [KnowNo card](/papers/2307.01928) records this gap. - **Calibrator, paper against code.** The paper describes a plain two-layer MLP with reliabilities at bin edges [arxiv:2610.17635 §4.1] [eq. 7]. The released code adds smoothing and a monotone envelope to its target, and places reliabilities at bin centres [see the PML-CD card](/papers/2610.17635). - **SNGP settings, paper against code.** The SNGP paper's Laplace step is described with a cross-entropy likelihood, and its released code uses a squared-loss likelihood [arxiv:2006.10108 §3.1]. The predictive step is also Monte Carlo averaging in the paper and mean-field in the code. The paper's reported calibration is therefore for its own predictive path, which the code does not necessarily reproduce. ## Papers - [Uncertainty Quantification in Scientific Machine Learning (2201.07766)](/papers/2201.07766): compares Bayesian methods and ensembles for physics-informed networks and DeepONets on six metrics, including RMSCE, and three post-training recalibration methods. Author-reported on prototype problems. - [Dirichlet calibration (1910.12656)](/papers/1910.12656): post-hoc multiclass map from log-probabilities to calibrated probabilities, with temperature scaling as a special case. Compared with beta calibration and temperature scaling on 21 UCI datasets, and with logits-based methods on CIFAR and SVHN networks. Author-reported. - [SNGP (2006.10108)](/papers/2006.10108): one deterministic network with spectral normalization and a Gaussian-process output layer, compared with a 10-model ensemble on CIFAR and CLINC calibration. Author-reported. - [Mitigating Neural Network Overconfidence with Logit Normalization (2205.09310)](/papers/2205.09310): trains with cross-entropy on L2-normalized logits scaled by a temperature, to limit overconfidence on OOD inputs. Reports FPR95 gains on CIFAR-10 and CIFAR-100 and ECE with and without temperature scaling. Author-reported. - [Metrics Reloaded (2206.01653)](/papers/2206.01653): consensus framework for choosing validation metrics for image classification, detection and segmentation, including when to add a calibration metric. No benchmark numbers. - [UQ tutorial (2305.04933)](/papers/2305.04933): compares GPR, MC dropout, Gaussian-head ensembles and SNGP on calibration, RMSE and NLL for battery and turbofan prognostics. Author-reported. - [KnowNo (2307.01928)](/papers/2307.01928): applies split conformal prediction to an LLM planner's option probabilities, so a robot asks for help when a prediction set is not a singleton. Author-reported hardware results. - [DROID (2403.12945)](/papers/2403.12945): robot dataset whose released camera poses are post-hoc sensor calibration. Listed only to mark the other sense of the word; it reports no confidence-calibration results. - [Humanity's Last Exam (2501.14249)](/papers/2501.14249): reports accuracy and RMS calibration error for eight LLMs on a closed-ended expert benchmark. Author-reported. - [A Survey of Uncertainty in Deep Neural Networks (2107.03342)](/papers/2107.03342): survey of uncertainty estimation and calibration measures, including temperature scaling and the ECE, SCE and aECE family. No experiments of its own; numbers are quoted from cited work. - [Language Models (Mostly) Know What They Know (2207.05221)](/papers/2207.05221): P(True) self-evaluation and a P(IK) value head for language models, with calibration on multiple-choice questions. Author-reported; no code or weights released. - [On the Apparent Conflict Between Individual and Group Fairness (1912.06883)](/papers/1912.06883): philosophical argument with no experiments. Treats equal calibration as a group measure and ties the choice of measure to assumptions about the source of disparities. Argument, not measured. - [Calibrated Disambiguation for Partial Multi-label Learning (2610.17635)](/papers/2610.17635): a pre-trained calibrator maps confidence histograms to per-label reliability, used as loss weights on noisy multi-label images. Author-reported; code released. - [LLM-as-a-Prophet: Prophet Arena (2510.17638)](/papers/2510.17638): live benchmark of LLM probability forecasts on prediction-market events, scored by Brier score, binned ECE and market return. Author-reported; no code. - [Surface Form Competition (2104.08315)](/papers/2104.08315): zero-shot multiple-choice scoring that divides out an in-domain prior (Domain Conditional PMI). Reports accuracy, not calibration. Author-reported; the code has no LICENSE file.