Language Models (Mostly) Know What They Know

In plain words

Can a language model tell when it is right? This paper checks in three steps. First, it finds that large models give sensible confidence numbers when a question lists its answer options with letters, so a stated probability roughly matches how often the answer is right.

Second, the model can grade its own answers. You let it write an answer, then ask it, as a yes/no question, whether that answer is correct. The model's probability for "True" tracks whether the answer was actually right. Showing the model a few other answers it wrote to the same question makes this grading better.

Third, the authors train an extra small output on top of the model to predict whether it knows the answer to a question before answering. That works well on the data it was trained on. On other kinds of questions it only partly carries over, and its numbers are often poorly calibrated there. Adding a relevant source text to the prompt raises the score, which is a hint that the signal reflects what the model can find in its context, not only what it memorised.

The authors present this as a step toward honest models, meaning models whose stated confidence matches what they actually know.

TL;DR

• Large models (52B) are well calibrated on multiple-choice questions when the options are lettered §2 app. A.1.
• Asking a model to judge its own sampled answer as True or False, P(True), works, and few-shot comparison samples improve it §4.1 §4.2.
• A value head trained to predict P(IK) separates known from unknown questions on its training distribution (TriviaQA). Transfer to other tasks is partial and calibration there is poor §5.1 §5.2.
• Author-reported. Trained and tested on pretrained models of 0.8B to 52B parameters; no code or weights are released.

Problem

Models should say when they do not know something, but ordinary training does not ask for that. The paper asks whether models can evaluate their own answers and their own knowledge, and whether that ability can be trained §1. Earlier calibration work covered token probabilities or multiple-choice formats, and truthfulness work focused on factual accuracy rather than self-knowledge §1.3.

Method

The paper uses four building blocks, each tested in its own section.

1. Calibration on multiple-choice questions §2. Options are shown with letter labels and the model is scored on its probability over the letters. Calibration is measured with expected calibration error (ECE) over 10 equal-count bins and with RMS calibration error app. A.1 app. A.2. Showing the options before the choice matters: without them, the authors expect poorer calibration §2. Calibration improves with size and with few-shot prompting §1.1 §2.

2. True/False probability §3.2. The question and a proposed answer are shown, and the model is asked whether the proposed answer is "(A) True" or "(B) False". P(True) is the probability the model puts on the True option. Replacing one option with "none of the above" hurts both accuracy and calibration §3.1.

3. Self-evaluation of sampled answers §4. The model first samples an answer at temperature 1. Then the True/False prompt is applied to that sample §4.1. Two variants:

• Basic: one sample is judged, using a fixed prompt §4.1 app. A.5.
• Comparison: five samples are generated for the question, and one of them is judged while the others are shown in the prompt §4.2. This is the variant the authors report as best with few-shot prompting fig. 11 app. A.5.

4. P(IK) value head §5. A value head is added to the model. Its output is a logit, trained with cross-entropy against binary labels. The labels come from 30 samples per question at temperature 1, each marked correct or incorrect app. A.6. Each sample becomes one training example, so a question with 20 correct samples contributes 20 "IK" labels and 10 "IDK" labels §5. The head reads the output at the last token of the input, and only that position is trained §5 fig. 3. The full model is fine-tuned together with the head app. A.6.

Equations (the paper gives these in the appendix; they are not numbered in the TeX source):

\mathrm{ECE} = \frac{1}{N}\sum_{i=1}^{N} \left| y_i - x_i \right| app. A.2

\mathrm{RMS} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} \left( y_i - x_i \right)^2} app. A.2

Here x_i is the mean predicted probability in bin i and y_i the fraction of correct answers in that bin.

B = \frac{1}{N}\sum_{i=1}^{N} \left( p_i - c_i \right)^2 app. A.3

Here p_i is P(True) for sample i and c_i is 1 if the sample is correct, 0 otherwise. Chance is 0.25 on binary tasks app. A.3 §1.1, Glossary.

H(A \mid Q) = \mathbb{E}_{A}\left[ \sum_i -\log P(a_i \mid Q, a_0, \dots, a_{i-1}) \right] app. B.2

Entropy estimate used as a baseline signal, with N = 200 samples per question app. B.2.

# P(True) self-evaluation of one sampled answer  [§4.1] [app. A.5]
# Input:  question, answer (a sample at T=1), few-shot examples
# Output: probability that the answer is correct

def p_true(question, answer, examples):
    # Build the True/False prompt around the proposed answer  [§4.1]
    prompt = format_true_false(examples, question, answer)
    # Probability the model puts on the " (A)" (True) option  [§3.2]
    return next_token_probability(prompt, " (A)")

# Comparison variant: judge one of five samples  [§4.2] [fig. 11]
# Input:  question, five samples at T=1, index of the sample to judge
# Output: probability that the judged sample is correct

def p_true_with_comparisons(question, samples, index):
    # The judged sample is one of the five shown in the prompt  [§4.2]
    judged = samples[index]
    prompt = format_brainstorm(question, samples, judged)
    return next_token_probability(prompt, " (A)")

# Train the P(IK) value head  [§5] [app. A.6]
# Input:  questions, 30 labelled T=1 samples per question
# Output: model with a trained value head

def train_p_ik(model, head, questions, labelled_samples):
    # Each (question, sample) pair is one hard label  [§5] [app. A.6]
    examples = build_examples(questions, labelled_samples)
    for batch in batches(examples, size=7680):  # [app. A.6]
        # Read the head at the last input token only  [§5]
        logits = head(model(batch.prompts)[last_token])
        loss = cross_entropy(logits, batch.labels)
        # Full model and head are updated  [app. A.6]
        update(model, head, loss, lr=pretraining_lr / 3)  # [app. A.6]
    return model

Data and evaluation. P(IK) training uses TriviaQA, with 30 samples per question, scored by string comparison app. A.6. Generalization tests use Lambada, mixed arithmetic, Python function synthesis, GSM8k and Codex HumanEval §5.2 app. D. Sampling and few-shot formats are listed in the Hyperparameters section below app. A.7.

Source material and hints. P(IK) rises when a relevant Wikipedia article is prepended to the question. Articles are truncated to 7000 tokens §5.3 fig. 18. It also rises when partial hints are shown on GSM8k problems, and it is higher for correct hints than for incorrect or irrelevant ones §5.4 fig. 19.

Hyperparameters

Results

Author-reported. Main setting: a 52B pretrained model, sampling tasks TriviaQA, Lambada, arithmetic, Codex HumanEval, GSM8k and Python function synthesis §4 §5. Baselines are the other formats in the paper, such as basic self-evaluation or training on TriviaQA only. The text gives few exact numbers for the self-evaluation results; they are plotted in the figures listed below.

Notes on the table:

• Self-evaluation accuracy on the subset where P(True) > 0.5 is higher than overall accuracy, and the gap grows with model size §4.2 fig. 11. Those numbers are only in the figures.
• Calibration of the P(IK) head on TriviaQA is good in-distribution. Calibration on other tasks is poor, and models are often underconfident there fig. 13.
• The mean P(IK) gap between models A and B is about 6% tab. 2 §5.5. The paper calls the effect small.
• Figure 14 shows AUROC on OOD tasks rising with model size for P(IK) trained only on TriviaQA fig. 14.

Paper vs code

No official code. The arXiv abstract page has no code or project links, and its comment reads "23+17 pages; refs added, typos fixed" 2207.05221. No project page is linked. A GitHub repository search for the title found only a third-party replication, which is not used. A search restricted to the authors' organisation found nothing.

Pitfalls

• Nothing to run. The models are the authors' own pretrained models §1.2 §7. No weights, code or datasets are linked, so exact reproduction is not possible. The P(IK) numbers are only for 52B and 12B models.
• Prompt form matters. The basic prompt and the few-shot prompt differ, and the paper says the prompt was "the first prompt we tried" and not optimised app. A.5. Zero-shot P(True) is poorly calibrated and sits near 50% for typical samples §4.1 app. B.
• Normalisation of the two options is not stated. The text uses the probabilities for "(A)" and "(B)" but does not say whether they are renormalised §4.1.
• Brainstorm example order. The comparison prompt in §4.2 shows five samples and the judged one is listed among them. Both the prompt and the list must be built the same way at training and test time app. A.5.
• Soft labels are approximated. Thirty hard labels per question stand in for a soft target. The paper notes that P(IK) can be misleading when the model can guess by chance, for instance on a composite-integer question where a random guess can yield an apparent P(IK) of 0.75 §5 footnote.
• Prompt format changes P(IK). Training and evaluation use the same few-shot format (10-shot for most tasks, 0-shot for Python). Changing the format changes P(IK) because the labels were produced under the original prompt app. A.6 app. A.7.
• Learning rate depends on another paper. The P(IK) learning rate is one third of the pretraining rate, but that pretraining rate is not given here app. A.6.
• Large batch is needed. A batch of 7680 is reported as important for learning from noisy labels; a smaller batch was not reported as tested app. A.6.
• Binarization threshold is inconsistent. Most results use 0.5 tab. 1 app. A.3, but GSM8k histograms and binarization in §5.2 use 0.25 fig. 17.
• Brier scores can mislead. On GSM8k the Brier score is dominated by incorrect answers with low P(IK) tab. 1 footnote. On Codex and GSM8k, small models get almost every answer wrong, so a near-zero Brier score can come from always predicting a low value fig. 11.
• ECE definitions differ. Calibration charts use all option probabilities, but ECE is computed only from the top prediction per multiple-choice question app. A.1 app. A.2. ECE values are therefore not comparable with charts that use all probabilities.
• Loss-based signals are length-sensitive. The paper uses mean token loss rather than summed loss to avoid length effects app. B.1 app. B.3. Entropy becomes a worse signal for larger models on HumanEval because larger models give diverse correct answers app. B.2.
• Source text effect. Shorter articles raise P(IK) more than long ones §5.3. The paper attributes this to answers being easier to extract from short articles, which it did not test directly.
• RLHF temperature. A single temperature of 2.5 is reported to fix calibration on several RLHF evaluations, but the authors say more intensive RL could break this §3.3.
• Scope. The paper studies five sampling datasets and says the trends are limited in power and scope §6.1. Generalization to other models or domains is untested.
• Numbering. Section, table and figure numbers on this card were counted from the TeX source (arXiv v4). If the published PDF numbers differ, check the citation before relying on it.

Lineage

• Builds on: the model architecture and RLHF procedure of Bai et al. (2022), cited in the paper as the source of its models and RLHF policies §1.2 §3.3.
• Compared against: other calibration and self-evaluation work cited in §1.3 and §4.2 (self-consistency is noted as related).
• Superseded by: not stated.