Uncertainty estimation

Overview

Uncertainty estimation turns a model's outputs on one input into a number that says how unsure the model is. The number is used in three ways: to rank unlabeled examples so that a human labels the most useful ones (active learning), to decide whether to trust an answer, or to set a threshold that decides when a system acts and when it asks a person 2307.01928 §3.2. The survey frames the core difficulty: a softmax output looks like a confidence score but is often overconfident, so "label what the model is least sure about" can pick badly 2009.00236 §2. The families differ in what they read from the model: one deterministic pass, the spread across stochastic passes or several models, distance from the training data, or the agreement among sampled generations [arxiv:2009.00236 §3.1] 2305.04933 §3.1 2203.11171 §3.5. In active learning the score is computed on the unlabeled pool, before the oracle labels the chosen batch 2009.00236 §3.

A second distinction is between noise in the data and a lack of data. The tutorial splits total predictive variance into aleatory (irreducible) and epistemic (reducible) parts, and argues the split matters for deciding whether more data or a different model will help 2305.04933 §2.1 §2.2. The broader survey adds a third kind, distributional uncertainty, for inputs that come from a different distribution than the training data [arxiv:2107.03342 §2.4.1] §2.4.2. Work on physics-informed networks uses the same two-part split: the total variance is a noise term plus an epistemic term estimated from posterior samples 2201.07766 §3.1.

Regression models can also output a spread for each prediction, and use it as the uncertainty directly rather than through repeated passes or several models. A LiDAR detector predicts a log standard deviation for each box it proposes 1903.08701 §3.3, and the spread then drives how duplicate boxes are pruned §3.6.

Calibration, whether a stated 95% interval holds about 95% of the time, is a separate property from ranking quality. The tutorial measures it with reliability curves, expected calibration error (ECE) and NLL [arxiv:2305.04933 §6.1]. The survey names temperature scaling, a single value fitted after training that divides the logits, as a common post-hoc fix for overconfident softmax outputs. It keeps the predicted class but has limited expressiveness 2107.03342 §5.1.

Variants

Softmax margin and entropy

Rank by the gap between the two most likely labels (margin; smallest gap first), or by the entropy over all classes (largest first), both from one deterministic pass 2009.00236 §3.1.2 eq. 5. No change to the model and no extra passes are needed. The survey warns that softmax response has been reported to do worse than random sampling in cited work 2009.00236 §2, so treat it as a baseline to beat, not a default.

Monte Carlo dropout with BALD

Keep dropout active at prediction time, average several stochastic passes, and score each example by the mutual information between prediction and model weights (BALD) 2009.00236 §3.1.1 eq. 3. The survey notes that dropout has been shown equivalent to variational inference, which is the justification for reading the passes as posterior samples 2009.00236 §3.1.3. BALD scores each example on its own, so a top-K batch can be full of near-duplicates 2009.00236 §3.1.1 fig. 2. The joint score BatchBALD eq. 4, or an explicit redundancy term eq. 7, addresses this. The number of passes and the dropout rate are not stated in the survey. See Monte Carlo dropout.

The tutorial treats MC dropout as the scalable Bayesian option. Its case study uses a dropout rate of 0.10 and 15 passes 2305.04933 §6.1. It warns that the result depends strongly on the dropout rate, the number of dropout layers and the training length 2305.04933 §3.2. On its case studies, MC dropout was overconfident and had the largest run-to-run variation of the methods compared 2305.04933 §6.1 §6.2.

The Bayesian neural network survey places MC dropout inside variational inference, with a Bernoulli variable on each unit's input [arxiv:2006.12024 §2.4]. Its toy regression shows MC dropout with variance that is too large inside the data and larger outside it, against a GP baseline [arxiv:2006.12024 §3, fig. 7]. That is one run with no dropout rate given. The mutual-information score has a blind spot: if every pass agrees on the same wrong class, the score stays small even for an input far from the training data [arxiv:2107.03342 §4.1.3]. Pair it with a distribution-shift check if out-of-distribution inputs matter. See Bayesian neural network.

Ensembles

Train several models and use their disagreement as the uncertainty. The survey reports that Beluch et al. found ensembles gave better-calibrated uncertainty than MC-dropout and density-based selection, and also offset class imbalance during acquisition 2009.00236 §3.1.4. The DEBAL line of work combines ensembles with MC-dropout, motivated by overconfidence from mode collapse in variational inference 2009.00236 §3.1.3. The cost is several trainings; the ensemble size is not stated in the survey.

The tutorial's variant gives each member a predicted mean and variance, trained on the Gaussian negative log-likelihood. The average member variance is read as aleatory uncertainty and the spread of the member means as epistemic uncertainty 2305.04933 §3.3. Its case study uses 15 members, chosen by an elbow plot 2305.04933 §6.1. The authors report that this ensemble had the lowest NLL on each battery test set. Its ECE was not uniformly lowest: on turbofan unit 15 it was 14.9%, against 8.5% for SNGP and 11.5% for MC dropout 2305.04933 §6.2. The epistemic term depends on how diverse the seeds and mini-batch orders make the members 2305.04933 §3.3.

Ensemble doubt at human takeovers

In interactive imitation learning, the learner drives until a human expert takes over at a point they judge unsafe, and labels are collected only during those takeovers. The doubt at each takeover is the norm of the diagonal of the ensemble's output covariance. The threshold is the mean of the last quarter of the logged doubts 1810.02890 §II eq. 3 eq. 4. As printed, that sum and its divisor count different numbers of entries, so an implementation has to pick a convention eq. 4. The paper uses the threshold only to draw an estimated safe set for evaluating the trained policy §IV-B eq. 5. Using doubt as a control gate at test time is proposed as future work §V. The ensemble size, network, optimiser and covariance estimator are not stated §II.

Results are author-reported. In simulation, collisions and road departures are about 12 and 20 times lower inside the estimated safe set than outside it §IV-B tab. 1. On a real vehicle, five fixed obstacle scenarios gave zero collisions and zero road departures for this method, against one collision and one to six departures for the baselines §IV-C tab. 2. The authors call that test too limited for statistical significance §IV-C. Card: HG-DAgger.

Posterior samples for physics-informed networks and neural operators

A physics-informed network (PINN) is trained on data and on the residual of a differential equation. A DeepONet learns a mapping from input functions to solutions. The scientific-ML paper applies the same posterior machinery to both 2201.07766 §4.1 §4.2. Its Bayesian options are HMC, Langevin dynamics, mean-field variational inference, MC dropout and a Laplace approximation. Its ensemble options are independent members, snapshots from one cyclic-learning-rate run, and SWAG, which fits a Gaussian to those snapshots. Each method yields M parameter samples, and the predictive variance is the noise variance plus the spread of the sample predictions §3.1.

On its prototype problems the paper reports author-reported results. On a one-dimensional regression with known noise, MC dropout had the lowest relative L2 error (21.9 against 22.9 for HMC, on a ×10² scale), and MFVI had the lowest calibrated RMSCE, 3.8 against 5.7 to 8.0 for the other methods 2201.07766 §6.1. The paper states that post-training calibration leaves all methods roughly equally calibrated on that problem §6.1. Its cheaper methods (LA, MFVI and snapshot ensembles) matched costlier ones mainly when the prior and the noise scale were learned during inference §7. With fixed priors, the paper reports that epistemic uncertainty did not grow reliably with less data or more noise §6.1. Costs are indicative: 50 HMC steps take about four to five times standard training, and M serial ensemble members take M times standard training §6.6.

Post-training calibration uses a left-out set and offers three methods: scaling the variance by one constant, fitting a correction to the predictive CDF, or fitting an empirical distribution to scaled residuals [arxiv:2201.07766 §5.2]. The calibration set comes from the training distribution, so the authors warn that performance may fall on out-of-distribution inputs §5.2. The same paper reports that MC dropout's uncertainty did not rise on out-of-distribution inputs, which it flags as a risk §6.1. The authors have not released code. Card: Uncertainty Quantification in Scientific Machine Learning.

Per-annotator heads and annotator disagreement

Train one shared encoder with one output head per annotator. Each head learns only from the instances its annotator labelled, and the majority over the heads gives the label 2110.05719 §3.4 §4.3.1. The uncertainty is the disagreement among the heads: the product of the counts of positive and negative head labels, divided by the square of their total. It therefore lies between 0 and 0.25 2110.05719 §4.3.2 eq. 1 §5.2. The authors report that this measure tracks annotator disagreement better than softmax confidence and MC dropout on the GHC hate-speech data and on GoEmotions, with joy on the full GoEmotions data as the exception 2110.05719 §4.3.2 fig. 2 §4.4 fig. 4. Its mean is 0.170 on wrong predictions and 0.049 on correct ones §5.2 fig. 5. These results measure agreement with annotators, not calibration, so they do not show that a stated probability is correct. On GHC, majority-vote F1 is 59.49 against 57.32 for majority-label training. The authors call the gap significant but report no statistical test [arxiv:2110.05719 §4.3.1] tab. 1. The method needs labels from each annotator, and the authors suggest clustering annotators to limit the number of heads §6.2. Card: Dealing with Disagreements.

Gaussian processes and SNGP

Gaussian process regression (GPR) gives a posterior variance that rises away from the training inputs, so its uncertainty reflects distance from the data 2305.04933 §3.1. Exact GPR costs O(N³) in training time and O(N²) in storage, which the tutorial cites as the reason to leave it out of its larger turbofan study 2305.04933 §3.1 §6.2. On the battery training set GPR fits exactly but generalizes poorly, and its test-set uncertainty is high 2305.04933 §6.1.

Spectral-normalized Gaussian processes (SNGP) keep distance information in a single deterministic pass. They use residual hidden layers with spectral normalization, bounded by a constant γ that must be tuned, and replace the output layer with a Gaussian process approximated by random Fourier features and a Laplace approximation 2305.04933 §3.4. SNGP cannot separate aleatory from epistemic uncertainty 2305.04933 §3.4. On the battery data it was often underconfident 2305.04933 §6.1. The tutorial also reports that SNGP-type methods gave higher ECE than MC dropout and ensembles on other benchmarks 2305.04933 §3.4.

The SNGP paper introduces the method and reports its own results. It keeps the network deterministic: spectral normalization on the hidden weights, and an output layer that is a random-feature Gaussian process with a closed-form posterior covariance 2006.10108 §3.1 §3.2. Its results are averaged over 10 seeds on CIFAR-10 and CIFAR-100 (Wide ResNet-28-10) and on CLINC OOS (BERT-Base), against a 10-model deep ensemble 2006.10108 §5. On CIFAR-10 clean inputs, SNGP's ECE was 0.018, against 0.010 for the ensemble and 0.023 for the deterministic network. Its per-example latency was 6.25 ms, against 38.10 ms for the ensemble 2006.10108 tab. 2. Prediction averages the softmax over 10 Gaussian logit samples 2006.10108 §5 app. A. The released code, as recorded on the paper's card, uses a mean-field approximation at prediction time and a squared-loss likelihood in the Laplace step, which differ from the text. Implement from the text and check the code before reusing its defaults.

Per-box variance heads with adaptive NMS for LiDAR detection

The detector runs on the sensor's range image in one pass. Each point predicts a mixture of oriented box distributions, and each component has a log standard deviation trained with a Laplace-style box loss 1903.08701 §3.3 §3.5 eq. 9. Points that fall on one object are merged as a product of Gaussians, weighted by inverse variance §3.4 eq. 6. Duplicate boxes are then pruned by an adaptive NMS. Its overlap limit comes from the two predicted spreads, so uncertain boxes may overlap more before one is dropped §3.6 eq. 11. The number of mixture components depends on the class: three for vehicles and one for pedestrians and bikes on the large dataset §4.1.

The uncertainty here is a regression output, not a classifier score, and the paper reads it as localisation uncertainty. Results are author-reported. On the ATG4D validation set, vehicle AP0.7 over 0-70 m is 85.34, against 80.99 for PIXOR and 85.17 for ContFuse with LiDAR and RGB 1903.08701 tab. 1. In the ablation, the adaptive threshold with soft NMS scored best tab. 2. The calibration check passes on ATG4D but not on the smaller KITTI set §4.5 fig. 6, and on KITTI the detector trails the BEV baselines tab. 4. The paper does not say how to obtain ATG4D §4.1. Card: LaserNet.

Evidential Dirichlet opinions fused across views

For an input described by several views, such as sensors or feature sets, each view's network outputs non-negative evidence for each class. The evidence gives a Dirichlet distribution, and its strength S yields an uncertainty mass of K/S for K classes 2204.11423 §3.1 §3.2 eq. 11. The views are merged with a reduced Dempster-Shafer rule that discounts conflicting mass. An uncertain view therefore counts for less, and the joint opinion carries its own uncertainty 2204.11423 §3.3 eq. 13. The loss adds a KL term that pulls the evidence for wrong classes toward zero. The weight of that term is raised during training, and the paper does not give the schedule §3.1 §3.6. Both fused models are close to late fusion in cost tab. 5. Author-reported: test samples with added Gaussian noise get higher uncertainty [arxiv:2204.11423 §4.1] fig. 5. On SUN RGB-D, dropping samples whose uncertainty exceeds the largest belief lowers the misclassification rate from 0.387 to 0.187, using the paper's own rule §4.2 fig. 8 fig. 9. The card notes that the released RGB-D scripts pick checkpoints on the test loader, so these numbers may be optimistic. Card: Trusted Multi-View Classification.

Agreement across sampled answers

For a language model that ends in a short answer that can be compared for equality, draw m reasoning samples, take the majority answer, and use the share of samples that agree with it as an uncertainty signal. On GSM8K the agreement share tracked accuracy 2203.11171 §3.5 fig. 5. This is not an active-learning score: the method returns the majority answer and its agreement ratio. The cost scales with m, and the headline results use 40 full decodes per question 2203.11171 §3.2. See Self-consistency.

The legal-hallucination study applies the same idea to open-ended answers. Two temperature-1 answers to one prompt are compared by a GPT-4 judge. A contradiction shows that at least one answer is wrong, so the contradiction rate is only a lower bound on hallucination 2401.01301 §4.3 eq. 2. For calibration, confidence is the share of 10 samples that match the greedy answer 2401.01301 app. D.1. On that setup all four models were overconfident, with pooled ECE from 0.057 (PaLM 2) to 0.421 (Llama 2) 2401.01301 §5.3. The judge's agreement with two human coders was κ 0.87 and 0.77 on 100 queries 2401.01301 app. C.

The Med-Gemini study uses a related gate: entropy over sampled answers decides whether the model searches again. For each question it samples several reasoning paths and takes the Shannon entropy of the answer-choice distribution across them. At or below a threshold, the majority answer is returned. Above it, the model writes three queries that would settle the disagreement, and their results join the next round's prompt 2404.18416 §2.1. The paper describes the measure in words, and the log base and threshold value are not stated §2.1. The gate is applied to the three text benchmarks only §2.1. On MedQA, accuracy rises from 87.2% to 91.1% across search rounds, and self-training is credited separately with 3.2 points §4.1 fig. 4. Card: Capabilities of Gemini Models in Medicine.

LLM self-evaluation: P(True) and P(IK)

Two signals come from the model grading itself. P(True) asks the model whether one of its own sampled answers is correct, as a True/False question, and uses the probability it puts on "True". The comparison variant draws five answers at temperature 1, judges one, and shows all five in the prompt. With 20 few-shot examples it is the variant the paper reports as best 2207.05221 §4.2 fig. 11 app. A.5. The zero-shot prompt is poorly calibrated and sits near 50% for typical samples 2207.05221 §4.1 app. B.

P(IK) is a value head trained on the model's own labels. For each question, 30 samples at temperature 1 are marked right or wrong, and every sample becomes one training example for a head that reads the last input token 2207.05221 §5 app. A.6. Trained on TriviaQA, P(IK) reaches AUROC 0.864 on held-out TriviaQA questions tab. 1. When the head is trained on TriviaQA only, AUROC on the other tasks is 0.928 for mixed arithmetic, 0.687 for Python function synthesis, 0.624 for GSM8k and 0.606 for LAMBADA tab. 1. Adding the other tasks to training raised the GSM8k AUROC to 0.752 tab. 1. Calibration is good on TriviaQA and poor elsewhere, where the head is often underconfident fig. 13. Prepending a relevant Wikipedia article raised P(IK) for an obscure question from 18% to 78% at 52B 2207.05221 §5.3.

Both signals were measured on the authors' own pretrained models, from 0.8B to 52B parameters. No weights or code are linked, so the numbers cannot be reproduced from the card 2207.05221 §1.2 §7.

Conformal sets over LLM option scores

The model sees a few candidate next steps as a multiple-choice question, with an extra "none of these" option, and its probability for each option is the uncertainty input. Each calibration case scores one minus the normalized probability of the correct option. The threshold is the empirical quantile of those scores at level ⌈(N+1)(1−ε)⌉/N 2307.01928 §3.1 eq. 1. A new case keeps every option whose probability clears 1 − q̂. One kept option means act; more means ask a person to choose 2307.01928 §3.2. The user sets ε, so the target success rate is a dial rather than a tuned threshold. The paper's dataset-conditional form eq. 2 adjusts ε with an inverse Beta quantile and gives a stated probability over the calibration sample. The guarantee assumes calibration and test cases come from the same distribution, and that a human answers correctly when asked 2307.01928 §3.2. For multi-step plans, each episode gets the minimum per-step score, one threshold is shared across steps, and the sets are rebuilt one step at a time at test time 2307.01928 §3.3 eq. 4 alg. 1. Card: Robots That Ask For Help.

Results are author-reported. On a mobile manipulator, help was 0.67 against 0.81 for a cumulative- threshold baseline, with plan success 0.87 for both 2307.01928 tab. 2. On a tabletop arm, step-wise help was 0.58 against 0.72 at equal plan success 0.76 2307.01928 tab. 1. The baseline's ε was tuned to match plan success, not the target, so the comparison is not at equal target 2307.01928 §4.2 §4.3. Realized task success fell short of the 0.85 target on the mobile robot (0.76), which the authors partly attribute to the low-level controller 2307.01928 tab. 2 app. A.8. Raw sentence probabilities gave larger sets than the multiple-choice score [arxiv:2307.01928 app. A.9]. Calibration used 400 labelled scenarios 2307.01928 app. A.7.

Credal summaries of conformal p-values

Set size at one error level keeps only the labels above one cut-off, and so discards most of the p-value profile. The credal-geometry paper treats all split-conformal p-values for one input as a family of compatible distributions. It rescales the largest p-value to one, then derives summaries that can be computed after sorting the labels: fuzziness, a generalised Hartley measure, a family of Tsallis-penalised summaries, TV-MMI (equal to the second-largest p-value) and entropy imprecision 2602.01667 §3.1 §3.3 app. B. The Tsallis form at q = 1 is the normalised Hartley measure U₁, which the paper uses as its main summary §5.

On LAC scores, the paper reports that U₁ beat every tested fixed-level set size on four image datasets for selective prediction, by 0.85 to 1.86 AUARC points over the strongest set size 2602.01667 §5 tab. 1. For distribution-shift detection the picture is mixed: entropy imprecision was largest on CIFAR-10.1 by a narrow margin, and fuzziness led on Fashion-MNIST §5 tab. 1. Under APS, RAPS and SAPS scores, the baseline 1 − q₍₁₎ beat all five summaries on selective prediction §5 app. E. Under LAC, TV-MMI is a monotone transform of the second-largest probability, so it adds nothing to ranking beyond that probability app. D app. F. For regression with a range condition on the score, every input gets the same rank core, so the summaries cannot separate inputs 2602.01667 §3.4 app. C. All results are author-reported on fixed, cached classifiers, with standard errors over calibration splits only. Card: Beyond Prediction-Set Size.

Reject option with a threshold

A model with a reject option answers only when a score clears a threshold τ, and otherwise passes the case on. The survey separates two causes for rejecting: ambiguity, where the features do not determine the target, and novelty, where the input is rare or absent from training 2107.11277 §2.2. Two architectures carry this out. A separated rejector scores the input alone, for example with a density or one-class model, and runs before the predictor 2107.11277 §4 eq. 3. A dependent rejector reads a confidence value from the predictor, such as its largest class posterior, and rejects when that value is below τ 2107.11277 §5 eq. 4 eq. 5. Because a separated rejector never sees the predictor's output, it cannot reject ambiguous inputs 2107.11277 §4.

The threshold can be set from a target rejection rate ρ, from a rejection-rate bound, from an error bound, or by Chow's cost rule 2107.11277 §5 eq. 7. The survey says Chow's rule needs calibrated posteriors and advises a cost-based choice of τ when posteriors are estimated 2107.11277 §5. It also warns that coverage and accuracy on answered inputs must be read together, since either one alone favours a trivial setting 2107.11277 §3.1. The survey runs no experiments and reports no benchmark numbers; it names the lack of a standard benchmark as an open problem 2107.11277 §10.2. Card: Machine Learning with a Reject Option.

Next-token confidence and prompt sensitivity

Prompt sensitivity measures how much one question's score changes when only the wording of the prompt changes. PromptSensiScore averages, over instances, the mean absolute score gap between the prompt variants of each instance 2410.12405 §2.2 eq. 1 eq. 2. The same paper defines a decoding confidence: for each instance, the mean over its prompts of the probability of the most likely next token 2410.12405 §5.1 eq. 3. On CommonsenseQA the authors report that lower sensitivity goes with higher confidence across instances, for three models 2410.12405 §5.2 fig. 7. This is a correlation on one dataset, and the figure gives no numbers. The paper does not say whether the maximum runs over the whole vocabulary or over option tokens 2410.12405 §5.1. Its authors also say confidence values cannot be compared across models 2410.12405 §5.2. Treat it as a cheap signal to check, not as a calibrated probability. Card: ProSA.

When to use which

Implementation sketch

Pool scoring for active learning, covering the margin, entropy and BALD variants. Each step names its source. The number of passes T is not stated in the survey, so it is a parameter here.

# Score an unlabeled pool by predictive uncertainty, then take the top K
# [arxiv:2009.00236 §3.1] [§3.1.2]
# Input: model f, pool U, batch size K, method in {margin, entropy, bald}

def uncertainty_scores(model, pool, method, T):
    # Softmax probabilities from one deterministic pass  [§3.1.2]
    if method in ("margin", "entropy"):
        probs = softmax(model(pool))
    if method == "margin":
        # Gap between the two most likely labels; small gap = uncertain  [§3.1.2]
        top2 = sort_desc(probs)[:, :2]
        return -(top2[:, 0] - top2[:, 1])
    if method == "entropy":
        # Entropy over the k classes; large entropy = uncertain  [eq. 5]
        return -(probs * log(probs)).sum(axis=1)
    # Keep dropout on and draw T stochastic passes  [§3.1.3]
    passes = stack([softmax(model(pool, dropout=True)) for _ in range(T)])
    # Average softmax probabilities, not logits  [§3.1.3] [eq. 9]
    mean_p = passes.mean(axis=0)
    # BALD: entropy of the mean minus mean of the entropies  [eq. 3]
    return entropy(mean_p) - entropy(passes).mean(axis=0)

def select_batch(scores, K):
    # Top-K per-sample scores; near-duplicates can all rank high  [§3.1.1]
    return argsort_desc(scores)[:K]

For language models, the per-question analogue is the agreement share from 2203.11171 §2 and §3.5, not a pool score. For open-ended answers, a judge call that flags contradictions between two samples replaces the equality test, giving a lower bound on errors 2401.01301 §4.3.

For option-scored LLM choices, the threshold comes from calibration cases rather than a pool. This sketch uses the marginal quantile eq. 1 only; the dataset-conditional adjustment eq. 2 is left out.

# Conformal threshold and act-or-ask rule  [arxiv:2307.01928 §3.1] [eq. 1]
# Input:  option_probs (N, k) normalized confidences, labels (N,) correct index
# Input:  epsilon in (0, 1), the user-set error level
# Output: qhat, and per case either "act" with an option or "ask" with a set

def conformal_threshold(option_probs, labels, epsilon):
    n = len(labels)
    # Nonconformity: one minus confidence of the correct option  [§3.1]
    scores = 1 - option_probs[range(n), labels]
    # Quantile level ceil((n+1)(1-eps))/n, capped at 1, rounded up  [eq. 1]
    level = min(ceil((n + 1) * (1 - epsilon)) / n, 1.0)
    return quantile(scores, level, method="higher")

def act_or_ask(probs, qhat):
    # Keep every option whose confidence clears 1 - qhat  [§3.1]
    kept = [i for i, p in enumerate(probs) if p >= 1 - qhat]
    # One option: act. Several: ask a person to choose  [§3.2]
    if len(kept) == 1:
        return ("act", kept[0])
    return ("ask", kept)

Contradictions

• Which family is better calibrated. The survey reports that Beluch et al. found ensembles better calibrated than MC dropout 2009.00236 §3.1.4. The tutorial reports MC dropout with the lower ECE on the battery secondary test set (9.5% against 13.5% for the ensemble), while on turbofan unit 15 the ensemble had the higher ECE (14.9% against 11.5% for MC dropout and 8.5% for SNGP) 2305.04933 §6.1 §6.2. No single ranking follows.
• Direction of error. The tutorial reports that MC dropout was overconfident and that the ensemble and SNGP were underconfident on battery data 2305.04933 §6.1. The legal study reports that all four LLMs were overconfident 2401.01301 §5.3. The survey says softmax confidence is often overconfident 2009.00236 §2. The domains and metrics differ, and the sources do not reconcile them.
• MC dropout direction. The BNN survey's toy regression reports MC dropout variance too large inside the data 2006.12024 §3 fig. 7. The tutorial reports MC dropout as overconfident on battery data [arxiv:2305.04933 §6.1]. The two are different settings, and neither is a benchmark.
• LLM calibration across formats. The self-evaluation paper reports large models as well calibrated on letter-labelled multiple choice [arxiv:2207.05221 §2] app. A.1. The legal study reports all four LLMs as overconfident on open-ended court questions 2401.01301 §5.3. The formats and tasks differ, and the sources do not reconcile them.
• Agreement and accuracy. The self-consistency paper reports that the agreement share tracked accuracy on GSM8K 2203.11171 §3.5. The legal study uses the same kind of agreement as confidence and finds it overconfident on legal questions 2401.01301 §5.3 app. D.1. The tasks differ (arithmetic against legal recall), so neither result settles the other.
• SNGP against ensembles on calibration. The tutorial reports that SNGP-type methods gave higher ECE than MC dropout and ensembles on other benchmarks 2305.04933 §3.4. The SNGP paper reports SNGP's ECE on CIFAR-10 as 0.018 clean and 0.090 corrupted, against 0.010 and 0.087 for the ensemble 2006.10108 tab. 2, and on corrupted CIFAR-100 as lower than the ensemble's (0.117 against 0.138) 2006.10108 tab. 3. The benchmarks differ, so neither result settles the other.
• Uncertainty against diversity. The core-set and DFAL papers report that diversity methods always beat uncertainty methods, and that uncertainty can do worse than random sampling; the LLAL paper reports otherwise 2009.00236 §5.
• Conformal summaries against the probability baseline. The credal-geometry paper reports that under LAC its U₁ summary beats every fixed-level set size 2602.01667 §5 tab. 1. It also reports that under APS, RAPS and SAPS the max-probability baseline beats all of its summaries on selective prediction §5 app. E. The score family decides the winner, and the sources do not reconcile the two results.
• Random baselines. Random-sampling accuracy differs by 13% between the LLAL and BGADL papers at the same CIFAR-10 setting, and one method's results differ by 8% between core-set and VAAL papers on CIFAR-100 2009.00236 §5.

Papers

• A Survey of Deep Active Learning: taxonomy of uncertainty, Bayesian, ensemble and diversity query strategies, with a reproducibility warning. No experiments of its own.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models: samples many reasoning paths and votes; the agreement share tracks accuracy on GSM8K.
• Liu et al., SNGP for distance-aware deterministic networks: SNGP, one deterministic network with spectral normalization and a random-feature Gaussian-process output layer. Author-reported results against a 10-model ensemble.
• Nemani et al., a UQ tutorial for engineering design and prognostics: compares GPR, MC dropout, Gaussian-head ensembles and SNGP on battery and turbofan prognostics, using NLL and ECE. Author-reported results.
• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models: judge-based contradiction lower bound and ECE for four LLMs on US federal court questions. Author-reported results.
• KnowNo, LLM planners that ask for help: conformal sets over LLM option scores that decide when a robot acts and when it asks. Author-reported robot results; the released code differs from the paper (see the card).
• Hendrickx et al., Machine Learning with a Reject Option: A survey: taxonomy of ambiguity and novelty rejection, three rejector architectures and ways to set the threshold. No experiments of its own.
• Zhuo et al., ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs: PromptSensiScore for prompt wording, and a next-token confidence correlation on one dataset. Author-reported results.
• Goan and Fookes, Bayesian Neural Networks: An Introduction and Survey: variational inference, Hamiltonian Monte Carlo, Bayes by Backprop and MC dropout as approximations. Two small single-run experiments.
• Gawlikowski et al., A Survey of Uncertainty in Deep Neural Networks: data, model and distributional uncertainty; the main method families and calibration metrics. No experiments of its own.
• Kadavath et al., Language Models (Mostly) Know What They Know: P(True) self-grading and a P(IK) value head; in-distribution AUROC 0.864 on TriviaQA, partial transfer. Author-reported; no code or weights.
• Psaros et al., Uncertainty Quantification in Scientific Machine Learning: Bayesian samplers, ensembles, functional priors and post-training calibration for PINNs and DeepONets, compared on prototype problems. Author-reported; no official code.
• Chau et al., credal summaries of conformal prediction sets: credal summaries of split-conformal p-values, with a regression impossibility result. Author-reported on image classifiers; no official code.
• Mostafazadeh Davani et al., Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations: one head per annotator on a shared BERT encoder; head disagreement as uncertainty. Author-reported on hate-speech and emotion data.
• Han et al., Trusted Multi-View Classification with Dynamic Evidential Fusion: Dirichlet evidence per view, fused with a reduced Dempster-Shafer rule. Author-reported on RGB-D, vector and image-text data.
• Saab et al., Capabilities of Gemini Models in Medicine: Med-Gemini models with an entropy gate that searches again only when sampled answers disagree. Author-reported; models not released.
• Kelly et al., HG-DAgger: Interactive Imitation Learning with Human Experts: human-controlled takeovers with a doubt threshold learned from them. Author-reported simulation and a five-scenario vehicle test.
• Meyer et al., LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving: per-point box mixtures on LiDAR range images, fused by mean shift and pruned by variance-aware NMS. Author-reported on ATG4D and KITTI; no official code.