Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry

In plain words

A conformal predictor does not just pick an answer. For each possible label it also produces a score that says how well that label fits, and the usual practice is to keep only the labels above one cut-off. The number of labels that survive that single cut-off is a popular uncertainty signal, but it throws away most of the information: two inputs can give the same count and still differ a lot.

This paper keeps all of the scores. It treats the full set of scores as a family of probability distributions that are all consistent with the model's ranking of the labels. The spread of that family then gives uncertainty numbers that take every label into account, and that can be computed quickly. The authors also show a limit: for a common kind of regression setup, these numbers come out the same for every input, so they cannot tell inputs apart.

Tests on image classifiers show that these richer numbers often rank the uncertain cases better than set size does, but a simple probability-based measure is often as good or better. The paper does not claim the new measures are always the best.

TL;DR

• Split conformal prediction (CP) gives each input a p-value per label. The paper shows these p-values define a credal set, a family of compatible distributions, and that the CP regions are its imprecise highest density regions §3.1, §3.2.
• With the labels sorted, the set's uncertainty measures reduce to weighted sums over the sorted p-values, computable in linear time after sorting §3.3.
• Author-reported on LAC scores, the Tsallis-penalised summaries beat every fixed-level set-size control on selective prediction and OOD detection, but not always the probability baselines §5, app. E, app. F.
• Impossibility: under a range condition, split-CP regression with absolute-residual-type scores gives the same rank core for every input, so no fixed summary of that core separates inputs §3.4.

Problem

Set size at a fixed error level alpha is a coarse uncertainty signal: it takes at most K+1 values for K classes and ignores the rest of the p-value profile §1. The paper asks whether the full profile yields finer and more useful uncertainty rankings, and whether those rankings can be computed efficiently §1. The guarantees assumed are split-CP sampling: calibration and test examples are exchangeable given the training data, and the score is fixed before calibration §2.1. The scores describe total conformal uncertainty, not a separated aleatoric or epistemic split §1, §3.5.

Method

Setup and notation

Let the label space be the finite set {1,...,K}. For calibration scores s_1,...,s_n, the split-CP p-value of label y at input x is the rank of its score among the calibration scores, with the +1 correction, and the prediction region at level alpha keeps the labels whose p-value is strictly above alpha §2.1. Ties make this construction conservative. Randomised tie-breaking is a separate option §2.1.

Step 1: normalise the contour §3.1

Let rho_x be the largest p-value. Every compatible distribution must satisfy P(A) at most the maximal p-value over A, so it is impossible for all bounds to hold when rho_x is below one. The paper fixes this by raising one maximising label to one. The chosen label is found by a fixed tie rule. Only that one coordinate changes. The raw maximum rho_x is then no longer used by the summaries and can be reported separately as conformal credibility §3.1.

Step 2: the credal set §3.1, §3.2

Define the upper probability of a nonempty event A as the largest normalised p-value in A, and of the empty set as zero. The core is the set of distributions P that satisfy P(A) at most this upper probability for every A §3.1. The strict cuts C_t(x) = {y : pi_x(y) > t} are nested in t. For every t in [0,1], C_t(x) is the unique weak (1-t)-IHDR of the core, and the cut equals the intersection of all events whose lower probability is at least 1-t §3.2. The full core can be recovered from the path of cut probabilities §3.2.

Step 3: nested representation after sorting §3.3

Sort the labels so that the largest normalised p-value comes first. Write alpha_1 = 1 >= alpha_2 >= ... >= alpha_K and set alpha_(K+1) = 0. Let m_j = alpha_j - alpha_(j+1). A distribution P is in the core if and only if its cumulative mass on the top j labels is at least 1 - alpha_(j+1), for j = 1,...,K-1 §3.3. Equivalently, P is a mixture with weights m_j of distributions supported on the top-j sets §3.3.

Step 4: summaries §3.3, app. B

• TV-MMI (U_inf). The largest gap between upper and lower probability over all events equals alpha_2. It also equals the largest total-variation distance between two distributions in the core §3.3. This is the conformal unconfidence, the second-largest p-value §2.1.
• Generalised Hartley (GH). GH = sum_j m_j log2(j). It costs O(K) after sorting §3.3.
• Integrated IHDR summaries. For any penalty g with g(0) = 0, U_g = integral over t in [0,1] of g(|C_t|) dt = sum_j (alpha_j - alpha_(j+1)) g(j) §3.3.
• Tsallis family. Let g_q(j) = (j^(1-q) - 1)/(1-q) for q not equal to 1, g_1(j) = ln j, and g_q(0) = 0 §3.3. The normalised summary U_q = U_(g_q) / g_q(K) is a weighted sum of alpha_j for j = 2..K, and each U_q lies in [0,1] §3.3.
  • q = 0 gives fuzziness: U_0 = F / (K-1), where F = sum over j >= 2 of alpha_j §3.3.
  • q = 1 gives GH normalised by log2 K: U_1 = GH / log2 K §3.3.
  • q = 2 gives the saturating summary (Step 5) §3.3, app. B.
  • q to infinity gives U_inf = alpha_2 = TV-MMI §3.3.
• Entropy imprecision (I_H). Entropy imprecision is the maximum entropy over the core, since the minimum is zero at the point mass on the top label §3.3. Let B_0 = 0, B_j = 1 - alpha_(j+1), and let G be the least concave majorant of the points (j, B_j). With q*_j = G(j) - G(j-1), I_H = -sum_j q*_j log2 q*_j §3.3. The vector q* is obtained by pool-adjacent-violators on the masses m_j, pooling until block averages are nonincreasing §3.3.

Step 5: saturating summary U_2 app. B

With g_2(j) = 1 - 1/j, the weights are proportional to 1/(j(j-1)) and U_2 = (1 - 1/K)^(-1) sum over j >= 2 of alpha_j / (j(j-1)) app. B. It equals (1 - Sp) / (1 - 1/K), where Sp = sum_j m_j / j is the reciprocal cardinality specificity app. B.

Step 6: summaries as code (sketch by this card, not an algorithm from the paper)

# Credal summaries of one conformal contour  [§3.1] [§3.3] [app. B]
# Input:  pvalues: (K,) split-CP p-values, one per label  [§2.1]
# Output: dict of scalars; larger means more uncertain

def credal_summaries(pvalues):
    # Sort the contour, largest first  [§3.3]
    alphas = sorted(pvalues, reverse=True)
    K = len(alphas)
    # Raise the maximiser to one  [§3.1]
    alphas[0] = 1.0
    # Mass of rank j is alpha_j - alpha_(j+1), with alpha_(K+1) = 0  [§3.3]
    padded = alphas + [0.0]
    masses = [padded[j] - padded[j + 1] for j in range(K)]
    # Generalised Hartley: sum of mass times log2 of rank  [§3.3]
    hartley = sum(masses[j - 1] * log2(j) for j in range(1, K + 1))
    # Fuzziness F = sum of alpha_j for j >= 2  [§3.3]
    fuzziness = sum(alphas[1:])
    # Saturating summary U_2  [app. B]
    saturating = sum(
        alphas[j - 1] / (j * (j - 1)) for j in range(2, K + 1)
    ) / (1 - 1 / K)
    # TV-MMI equals the second-largest value  [§3.3]
    tv_mmi = alphas[1]
    # Entropy imprecision: PAVA on the masses, then Shannon entropy  [§3.3]
    smoothed = pool_adjacent_violators(masses)
    entropy_gap = -sum(q * log2(q) for q in smoothed if q > 0)
    return {
        "U0_fuzziness": fuzziness / (K - 1),
        "U1_hartley": hartley / log2(K),
        "U2_saturating": saturating,
        "Uinf_tv_mmi": tv_mmi,
        "I_H_entropy_gap": entropy_gap,
    }

Notes on the sketch: log2 is base-2 logarithm, and the function assumes K >= 2. pool_adjacent_violators is a standard isotonic-regression routine that returns block averages, one per rank, whose sequence is nonincreasing; the paper specifies this step in words §3.3, not as code. Any rank with zero mass contributes zero to the entropy sum, following the convention 0 log 0 = 0 §2.2. The paper does not give code for this sketch, and the sketch has not been run against the paper's numbers.

Step 7: non-conformity scores used in experiments app. D

Let q_y be the classifier probability for label y, sorted decreasingly as q_(1) >= q_(2) >= ... Let V_r be a uniform random draw for the label at rank r.

• LAC: s = 1 - q_y app. D §5.
• APS: s_r = sum over j <= r of q_(j) minus V_r times q_(r) app. D.
• RAPS: s_r = APS score plus 0.01 times (r - 5) for r above 5 app. D.
• SAPS: for r = 1, s = (1 - V_1) q_(1); for r >= 2, s = q_(1) + 0.2 (r - 1 - V_r) app. D.

The paper uses the score constructions of the cited papers with the fixed penalty values above, so the comparison is between these configurations, not between separately tuned reproductions app. D.

Step 8: regression §3.4, app. C

For regression, the p-value depends only on the calibration rank of the score at the test point. Under a range condition on the score, every input gets the same core on the grid of ranks j/(n+1) §3.4, app. C. Applying any fixed penalty g with g(0) = 0 then gives the same value for every input. The entropy imprecision equals log2 K for every input §3.4, app. C. The range condition holds for absolute residuals, positive weighted residuals and conformalised quantile regression, when the range covers the calibration scores §3.4. Response-space geometry can still vary with the input, and the paper says that distinguishing inputs requires retaining it §3.4.

Hyperparameters

Paper's computing environment: an Ubuntu workstation with an RTX 5090 GPU and 192 GB RAM §5. Score code: TorchCP 1.2.1 for RAPS and SAPS, with randomised ties app. D.

Results

Author-reported. Setup: split CP with LAC unless noted; classifiers are fixed and cached, and error bars are over ten resampled splits of one fitted model, not over independently trained models §5, app. E, app. F.

Selective prediction, LAC, AUARC (%) tab. 1

Set sizes are normalised. Higher is better. Values are mean ± standard error over ten paired splits.

Setup: tab. 1 in §5; models in app. E.

Distribution-shift detection, LAC, AUROC (%) tab. 1

Aggregated over CIFAR-10 ResNet-56 shifts. Mean ± standard error over ten paired seeds.

Setup: tab. 1 in §5; full protocol in app. F.

Headline claims in the text

• Selective prediction, LAC: U_1 and U_2 exceed every tested fixed-level set size on all four datasets. Paired gains over the strongest mean set size are 0.85 to 1.86 points for U_1 and 0.50 to 1.80 points for U_2 §5.
• Against the strongest of the three probability baselines in each condition, U_1 gains 0.135 ± 0.031 points on Caltech-101 and 1.196 ± 0.071 on ImageNetV2, and essentially matches entropy on CIFAR-100 §5.
• OOD, LAC: U_1 exceeds every tested cardinality in each reported aggregate. The largest core summary on CIFAR-10.1 is I_H, by a narrow margin. Fuzziness leads on Fashion-MNIST §5, tab. 1.
• Under APS, RAPS and SAPS, the baseline 1 - q_(1) is above all five credal summaries in mean AUARC on every tested condition. This reverses the LAC picture for selective prediction §5, app. E.
• Calibration size: with randomised APS, TV-MMI overtakes the raw set size at alpha = 0.01 once the calibration set has at least 500 examples app. F.
• Population limit: on a controlled three-class problem with a non-degenerate label distribution, the joint estimates at 30,000 calibration and training points are 0.1625 (TV-MMI), 0.6740 (I_H) and 0.1881 (GH). Their oracle or population references are 0.1629, 0.6753 and 0.1886 app. H.
• Active learning: exploratory. Under APS and RAPS, TV-MMI leads the contour summaries on every dataset tested. Under LAC, the baseline 1 - q_(1) has the highest mean on the three digit datasets. The paper does not claim sample-efficiency gains in general app. I.

Penalty comparison under LAC and APS app. G, tab. 2

The paper's overview of U_0, U_1, U_2 and U_inf under each score is in app. G. Under LAC, U_2 beats U_inf on CIFAR-100, Tiny ImageNet and both ImageNetV2 models, and GH keeps an advantage on these conditions. Under APS and RAPS, U_2 beats GH on every condition mean. Under SAPS, U_2 is below GH in all seven condition means app. G.

Paper vs code

No official code. The paper gives no code URL in its text, abstract or reference list. Its checklist says code is provided, and it mentions a supplied archive (scripts/generate_selective_tables.py, additional_baselines/saturating_penalty/), but that archive is not linked app. D, checklist item 1. I searched for a repository in three ways: a GitHub repository search for "credal conformal" and related short queries, a user search for the author names, and the repository list of the account that appears to be author Siu Lun Chau. None of the search hits was a conformal credal repository from the authors. Michele Caprio's account had no matching repository either.

Two third-party or unrelated repositories were found for "conformal credal" work (for example, a conformalized credal set predictor from another group). They are not the authors' code and are not used here.

Paper claims that are not checked against code: all reported numbers, all algorithmic claims, and the table generation in app. D.

Pitfalls

• Normalisation changes the summaries. Raising the maximum p-value to one is required for a nonempty core, and it discards the raw maximum from the summaries. Report rho_x separately if absolute atypicality matters §3.1.
• Ties are not fully resolved. Discrete p-values create tie groups. The paper uses shared random keys for ties in its experiments app. E. Its APS implementation uses stable NumPy sorting, and TorchCP's RAPS and SAPS use their own convention. Shared rank draws can therefore land on different classes within a probability tie app. D.
• TV-MMI with LAC adds no ranking beyond q_(2). Under LAC, TV-MMI is a monotone calibration transform of the second-highest probability. It can only merge ranks into ties, so differences from q_(2) come from calibration ties only app. D, app. F. Treat TV-MMI as a relabelling of q_(2) in LAC.
• The probability baselines win some conditions. On Tiny ImageNet and on ImageNetV2 with ResNet-18, maximum-probability uncertainty leads app. E. Under APS, RAPS and SAPS, 1 - q_(1) beats every credal summary on selective prediction §5, app. E.
• Pearson correlation is not ranking quality. A higher Pearson correlation with errors can come from a different monotone transform and does not mean better ranking. Use AUROC or AUARC to compare rankings app. E.
• Entropy imprecision uses PAVA. The closed form pools masses until block averages are nonincreasing §3.3. The paper states this as a procedure in words and does not give code for it.
• Core never collapses at finite calibration size. After normalisation, the TV diameter is at least 1/(n+1) for ordinary split p-values, so the core is not a point even with an oracle predictor §3.5.
• Corruption OOD evaluation shares base images. The corrupted view of a calibration image can appear in the OOD evaluation sample. Results are not evaluations on unseen base images app. F.
• Standard errors are conditional on one fitted model. They capture split, tie and score-randomisation variation only app. E, app. F.
• Empty sets at raw cardinality. Raw LAC sets at alpha = 0.10 are empty or singleton on the OOD rows, so normalised sizes are constant there. Normalisation matters for Caltech-101 at alpha 0.05 and 0.10 app. E, app. F.
• Regression rank cores are input-invariant. The impossibility holds only when the score range condition holds. Once response-space geometry is used, inputs can differ, but the response-space summaries are not the rank-grid summaries §3.4, app. C.
• Active learning is not sequentially valid. Acquired labels change the calibration set, so the reported acquisition results are not a conformal validity claim app. I.
• Saturating summary depends on K. The normalisation 1/(1 - 1/K) changes with the number of labels, so compare U_2 only at a fixed label space app. B, §3.3.

Lineage

• Builds on: the conformal-to-credal connection for full CP and its IHDR interpretation §2.2, §4, the normalised-contour credal representation for split CP in the cited earlier work §3.1, §4, and the maximum mean imprecision measure §2.2. Cited papers are referenced by author in the paper and are not linked here, because they have no card.
• Compared against: fixed-level set sizes at 0.01, 0.05 and 0.10; probability baselines (entropy, 1 - q_(1), q_(2)); and the score families LAC, APS, RAPS and SAPS §5, app. D.
• Superseded by: not stated.