How do LLMs Compute Verbal Confidence

In plain words

Many chat models can be asked "how sure are you?" and answer with a word such as "Likely" or a number. Those answers are cheap to get, but nobody knew what happens inside the model when it produces one. Does it work the number out only once it is asked, or did it already form a sense of how good its answer was while writing that answer?

The authors test this by opening the model up in several ways. They push signals into it, cut parts of it out, swap the internal state of one question for another, and check which places in the model hold the information. Their answer is that the model forms a judgement of its answer while writing it, stores that judgement at the first newline after the answer, and reads it out when it has to state the confidence. The judgement carries more than the model's word-by-word probabilities. The results come from one open model on trivia and maths questions, and the authors say the same pattern shows up in other models, but the evidence is still limited.

TL;DR

• Tests two questions: whether verbal confidence is computed on demand or cached during answer generation, and whether it is just a readout of token log-probabilities §1.
• On Gemma 3 27B, the information sits at the post-answer newline (PANL) at earlier layers and then reaches the confidence-colon (CC) token at later layers §2.1 §2.2 §2.3 §2.4 §2.8.
• Probes at PANL explain variance in verbal confidence beyond six answer log-probability summaries §2.6 app. D.3.
• Author-reported. The pattern is also reported for Qwen 2.5 7B, Magistral Small 24B, two more datasets and a numeric prompt §2.9.

Problem

The paper asks how a model produces a verbal confidence score, the kind that black-box APIs return when asked. Two competing accounts are set against each other. Under the just-in-time (JIT) account, the model computes confidence only when the rating is requested, from the question and answer at that point. Under the cached-retrieval account, confidence is computed during answer generation, before the model knows a rating will be asked for, and stored for later retrieval §1.

A second question is what the verbal score reflects. A first-order account says it is a readout of token log-probabilities. A second-order account says it is a separate evaluation of question-answer fit §1 app. A.

The study only uses the no-chain-of-thought setting in its main experiments. The authors chose it on purpose because many applications use models as raters without reasoning §2. Magistral Small 24B, a reasoning model, is tested in a separate CoT setting §2.9 app. C.4.

Method

The paper is an analysis, not a new model. It builds a fixed prompt, runs several interventions on the residual stream and attention, and reads the results at named token positions.

Prompt and positions. The main prompt asks for a class from a fixed list, each with a probability range app. B, fig. 8. Phase 0 generates the answer. Phase 1 shows the answer back and asks for a class app. C.1.2 §2. The positions are:

• PANL: the newline after the answer.
• PANL+1: the token after PANL, a control.
• FCC: the first confidence colon, a control app. C.1.3.
• CC: the confidence colon, the last token of the prompt.
• AC: the answer colon, the last token of the Phase 0 prompt. It drives the first answer token and is a further control §2.7.

Metrics. Confidence is the midpoint of the chosen class's probability range app. C.1.5. The logit difference compares the clean run's chosen class against the mean of the other classes eq. 3. The first-token change rate is the fraction of trials where the argmax token changes eq. 4. Patching recovery is defined in eq. 5 and eq. 6.

Steering. A direction is built as the mean activation of 25 high-confidence trials minus the mean of 25 low-confidence trials app. C.1.3. It is added at one layer and position with strength 2 or 5, after scaling to 3% of the residual norm app. C.1.3 §2.1.

# Steering direction and application  [app. C.1.3]
# Input:  residual activations of high and low confidence trials, one layer
# Output: steered residual activation at one position and layer

def build_direction(high_trials, low_trials):
    # Difference of mean activations between the two groups  [app. C.1.3]
    mean_high = mean(high_trials)               # (d,)
    mean_low = mean(low_trials)                 # (d,)
    return mean_high - mean_low                 # v_high, (d,)

def steer(residual, direction, strength):
    # Rescale so its norm is 3% of the residual norm  [app. C.1.3]
    vector = 0.03 * norm(residual) * direction / norm(direction)
    # Add the scaled vector; strength is 2 or 5  [app. C.1.3]
    return residual + strength * vector         # (d,)

Patching. The answer tokens are replaced by mean embeddings from 100 calibration trials to remove answer information. Then the clean activation is copied back at one position and layer, after that layer's MLP block app. C.1.4 eq. 1 eq. 2. The change in confidence is then measured on high-confidence trials only §2.2.

# Corrupt-then-restore patching  [app. C.1.4]
# Input:  clean run, corrupted run (answer embeddings set to means)
# Output: patched run logits for one position and layer

def patch_one_site(clean, corrupt, position, layer):
    # Start from the corrupted run  [app. C.1.4]
    patched = corrupt.copy()
    # Copy in the clean activation at this position and layer  [eq. 2]
    patched.residual[layer][position] = clean.residual[layer][position]
    # Let the rest of the run stay corrupted  [app. C.1.4]
    return forward_from(patched, layer + 1)

Noising (mean ablation). At one position and layer, the activation is replaced by the mean of 100 calibration trials, 50 high and 50 low confidence §2.3 app. C.1.6.

Swap. A 2x2 design crosses recipient confidence with donor confidence. The PANL activation of a donor trial is placed into a recipient trial, and donors are length-matched to recipients §2.4 app. C.1.7.

Attention blocking. Attention from a target to a source is set to zero in all heads over a window of layers. The paper uses a minimal numeric 0 to 9 prompt for the key tests §2.8 app. C.1.10.

# Attention knockout  [app. C.1.10]
# Input:  attention weights per layer, source and target positions
# Output: attention weights with the edge removed

def knock_out(attention, source, target, layers):
    # For each layer in the window, remove the edge in every head  [app. C.1.10]
    for layer in layers:
        attention[layer][:, target, source] = 0.0
    return attention

Probes. For each layer and position, L2 logistic regression predicts correctness (AUROC) and ridge regression predicts the class midpoint (R²). Both use 5-fold cross-validation on z-scored activations from 3,000 trials app. C.1.8.

Variance partitioning. Three ridge fits (activations, log-probability baselines, both) predict the midpoint. The unique contribution of activations is the non-negative gain over the baselines app. C.1.9. The six baseline summaries are the length-normalised mean, minimum, maximum and variance of answer token log-probabilities, plus the first and last answer token log-probabilities eq. 7 app. C.1.9.

Hyperparameters

Results

Author-reported. Setup: Gemma 3 27B, categorical class prompt, TriviaQA, no chain-of-thought, unless noted §2 app. C.1.1. Grading uses GPT-4o-mini fig. 9. Many effect sizes appear only in figures; the card does not give numbers the text does not state.

Paper vs code

No official code. Search performed for this card: the paper, its LaTeX source and its arXiv abstract page link no repository or project page. GitHub repository searches (via the API) for "verbal confidence", "cached retrieval confidence", "kumaran confidence LLM" and "confidence org:google-deepmind" returned nothing by the authors. A repository named mmiao2/Verbal_Confidence_Mech_Interp appeared, but its owner is not an author of this paper, so it was not used.

Pitfalls

• Steering direction definition. The appendix says high and low confidence trials are "all trials that the model scored correctly" app. C.1.3. That conflicts with defining groups by confidence class app. C.1.3 §2.1. Use the confidence class and flag the sentence.
• Steering scale. The norm used for the 3% rule is not stated. The scales 2 and 5 were "chosen empirically" app. C.1.3. Gemma's categorical steering baseline is 0.55 in fig. 2 but 0.48 in fig. 22. Check the setup before comparing.
• Swap recipient pool. The text says the same 400 low-confidence recipients are used in two conditions. The pool has only 221 low trials, sampled with replacement app. C.1.7. Duplicated recipients lower the effective sample size.
• Which confidence the probes and partitioning use. The appendix gives R²CV 0.049 for within-run and 0.084 for cross-run (Phase 1) log-probability baselines app. C.1.9. The main text reports 0.10 for six baselines combined but does not say which run. Confirm before reusing the numbers.
• Variance partitioning clipping. The unique term is clipped at zero with max(0, ...) app. C.1.9. Small or negative gains are invisible after clipping.
• Mean ablation is not neutral. Replacing an activation with the average of high and low trials does not create a medium-confidence state app. C.1.6. Partial effects can reflect distributed encoding or redundancy, which the paper cannot separate §2.3.
• Patching restricted to high-confidence trials. This is chosen because low-confidence trials sit near the floor §2.2. Recovery uses a formula that divides by the corrupt-to-clean gap eq. 5. The result is unstable when that gap is small, and the paper does not say how such cases are handled.
• Attention knockout limits. The method ignores information that moved between positions before the blocked layers, so null results are weak evidence app. C.1.10. Whether attention weights are renormalised after the zeroing is not stated.
• Prompt dependence. CC to PANL blocking is positive on the minimal numeric prompt and null on the categorical prompt. The authors attribute this to routing through template tokens §2.8 app. D.1. Treat the pathway as prompt-dependent.
• Prompt modification. The class prompt is adapted from Yoon et al. 2025 so each class starts with a unique first token §2 app. A. The probability ranges for each class are in the prompt figure, not in the text fig. 8.
• Numeric prompt interventions. Interventions run during prefill, so later digits reflect both the intervention and the change propagated through the first digit app. C.2.
• Reasoning model. On Magistral, noising PANL shows no effect beyond the control, while CC noising does. The authors attribute this to confidence spread over the reasoning trace §2.9 fig. 25. Swap asymmetry, with the low-to-high direction dominating, is linked to the model's high-confidence bias app. D.2.
• Layer indexing. The layer convention (0-based or 1-based) is not stated. Layer numbers in the text should be read as the paper's own labels.
• Behavioural base rates. Gemma's confidence distribution is weighted toward high classes §2. Effects on low-confidence trials are harder to measure.
• Scope of the claim. The authors say cached retrieval is the dominant, not the only, pathway §4. They do not show that the same circuit exists in other models in full.

Lineage

• Builds on: the corrupt-then-restore patching paradigm (Meng et al. 2022), activation steering (Turner et al. 2023), and the attention-knockout cache-and-retrieve analysis of factual recall (Geva et al. 2023). The paper does not state arXiv ids for these in its text §3 app. A.
• Builds on: the decision-neuroscience first-order versus second-order split of confidence (Fleming et al. 2017) §1 app. A.
• Compared against: token log-probability and sampling-based confidence §1.
• Related follow-up named by the authors: error detection with the same post-answer cache (Kumaran et al. 2026) §4. Not checked here.
• Superseded by: not stated.