What Large Language Models Know and What People Think They Know

In plain words

Users read a language model's answer and its explanation, and usually have no view of how sure the model itself was. This paper asks a practical question: how accurately do people judge whether an answer is correct, and how does that compare with the model's own sense of how sure it is?

The authors ran studies with several hundred people. People read answers from three models to multiple-choice and trivia questions, and said how likely they thought each answer was correct. People were too trusting. Longer explanations made them more trusting still, even though the extra length did not help them tell right answers from wrong ones. When the wording of the explanation was changed to sound unsure where the model was unsure, people's judgements moved closer to the model's real accuracy. The authors tested this by filtering existing ratings, not by running a live system.

TL;DR

Users' confidence in LLM answers is poorly calibrated when the answer comes with the model's default explanation: they overestimate accuracy, and they discriminate right from wrong answers much less well than the model's token-probability confidence does. Longer explanations add confidence without adding discrimination. Choosing the explanation's uncertainty wording from the model's confidence narrows both gaps, but only in a simulation over the collected ratings. Use this paper as evidence that explanation wording shapes user trust, and as a template for measuring it. All results are author-reported.

Problem

Users judge an LLM answer mainly from its text, not from the model's internal confidence, which is usually not shown §1. The paper names two gaps. The calibration gap is the difference between human confidence in an answer and the model's own confidence §1. The discrimination gap is the difference in how well humans and the model separate correct from incorrect answers §1. Without a measurement of these gaps, there is no basis for judging whether an LLM's explanations are trustworthy in decision-making §1, §4.

The study covers only a narrow setting: multiple-choice questions with four options and short trivia answers. The authors note that longer open-ended answers are untested §4. The paper reads model confidence from token log-probabilities §2.2. Participants were US-based Prolific workers, and most lacked domain expertise on the questions §2.3.1, §2.

Method

The method has three parts: measure model confidence, generate explanations whose uncertainty wording is varied, and collect human confidence ratings.

1. Model confidence, multiple choice §2.2. Each multiple-choice question is given zero-shot, with the question and its four options. The model's top five next-token log-probabilities are read out. An answer counts as complete when the tokens include the letters A, B, C and D. The scores for the four options are converted and normalised to sum to one; the model's confidence in an answer is that option's normalised probability §2.2. Models are GPT-3.5 (instruct) and PaLM2, both at temperature 0 §2.2. Incomplete answers are dropped, which affects 8.7% of the full GPT-3.5 set app. B.

2. Model confidence, short answer §2.2. The model first answers each TriviaQA question. A 10-shot prompt, with 10 random example questions and reference answers, keeps answers short; the median answer is 2 words §2.2. The model then judges the proposed answer as true or false, in the style of the "pTrue" method §2.2. The normalised probability of "true" is the model confidence. Temperature is 0.7 for all short-answer calls §2.2. The answer is scored correct by an LLM judge on a 5000-question sample, and by a human judge on the 336-question subset, where the two agreed on 97% of items §2.2.

3. Question sampling §2.1, §2.2. For multiple choice, 350 questions are drawn from the 14,042-question MMLU test set: 35 per topic across 10 topics, for each LLM. The draw is roughly uniform over four model-confidence bins (0.2–0.4, 0.4–0.6, 0.6–0.8, 0.8–1.0), with fewer questions in the lowest bin §2.2. For short answer, 5000 questions are sampled from TriviaQA, and 336 are drawn roughly uniformly over bins 0–0.25, 0.25–0.5, 0.5–0.75 and 0.75–1.0 §2.2. The sampling enriches low-confidence items, so the subset accuracy is lower than on the full sample app. B.

4. Explanation styles §2.3.3, app. D. Experiment 1 shows the model's default explanation to users. Its prompt asks for an answer and an explanation that includes the answer app. D. Experiment 2 shows one of nine styles: three levels of uncertainty wording crossed with three lengths §2.3.3. The uncertainty levels are low ("I am not sure"), medium ("I am somewhat sure") and high ("I am sure"). The lengths are:

• long: no length instruction;
• short: "as few words as possible";
• uncertainty only: the rationale is removed, leaving the stance and the answer.

Experiment 3 (appendix only) rewrites the baseline explanation with the same three uncertainty levels, with no length manipulation app. A.2. Example outputs for one question appear in tab. 2.

5. Human ratings §2.3.1, §2.3.2. Participants read a question and the explanation, then give a probability that the model's answer is correct. For multiple choice, they then answer the question themselves with the model's help; for short answer, they only rate §2.3.2. Each participant sees 40 questions, each once, drawn so that all confidence levels appear. In Experiment 2 the styles are counterbalanced across questions §2.3.2. At the end, participants estimate their expected accuracy per topic §2.3.2. The experiment design is in tab. 1. The setup is also shown in fig. 1.

6. Metrics §2.4. Calibration is measured with expected calibration error over M equal-width bins eq. 1:

\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N} \, |\mathrm{conf}(B_m) - \mathrm{acc}(B_m)| eq. 1

Discrimination is the AUC, computed through the Mann–Whitney statistic over correct and incorrect pairs eq. 2:

\mathrm{AUC} = \frac{1}{N_{pos} N_{neg}} \sum_{i=1}^{N_{pos}} \sum_{j=1}^{N_{neg}} I(C_i > C_j) eq. 2

The paper also uses an overconfidence error variant app. A.3. The text does not give the number of bins M used in ECE §2.4.1.

7. Statistics §2.5. Bayesian t-tests with default priors §2.5 report Bayes factors. The paper treats BF above 3 as moderate and above 10 as strong evidence against the null, and the reciprocal thresholds for the null.

8. Selection rule §3.2.1, eq. 3. To narrow the gaps, the explanation style is chosen from the model's confidence p with two thresholds \theta_1 < \theta_2:

s = \begin{cases} \text{low} & \text{if } p \le \theta_1 \\ \text{medium} & \text{if } \theta_1 < p \le \theta_2 \\ \text{high} & \text{if } \theta_2 < p \end{cases} eq. 3

The authors do not run a live system. They keep only those ratings whose observed style matches the style the rule would select, and recompute ECE and AUC on the kept ratings §3.2.1.

# Choose the explanation wording from model confidence  [§3.2.1] [eq. 3]
# Input:  p (model confidence in [0, 1]); theta_1 < theta_2
# Output: "low", "medium" or "high"

def select_style(p, theta_1, theta_2):
    # Low wording when confidence is at most theta_1  [eq. 3]
    if p <= theta_1:
        return "low"
    # Medium wording when confidence is in (theta_1, theta_2]  [eq. 3]
    if p <= theta_2:
        return "medium"
    # High wording when confidence is above theta_2  [eq. 3]
    return "high"

# Simulated evaluation of the selection rule  [§3.2.1] [app. C]
# Input:  ratings (per question: observed style, human rating, correct flag)
#         model_conf (per question), theta_1, theta_2
# Output: ECE and AUC on the kept ratings

def simulate_rule(ratings, model_conf, theta_1, theta_2):
    # Keep only ratings whose observed style equals the rule's choice
    kept = []
    for rating in ratings:
        # Rule uses the model's confidence for this question  [§3.2.1]
        style = select_style(model_conf[rating.question],
                             theta_1, theta_2)
        if rating.style == style:
            kept.append(rating)
    # Compute the two metrics on the kept ratings  [eq. 1] [eq. 2]
    return ece(kept), auc(kept)

9. Threshold choice app. C. The thresholds are chosen by grid search to optimise a combination of ECE and AUC, with ECE weighted at 33% relative to AUC app. C. The exact combining formula is not written. Multiple-choice thresholds are searched over 0.25–1, and short-answer thresholds over 0–1, always with \theta_1 < \theta_2 app. C. The grid step is not stated.

Hyperparameters

Results

Author-reported. Setup: MMLU multiple choice, 350 questions with GPT-3.5 and PaLM2; TriviaQA short answer, 336 questions with GPT-4o; human raters from Prolific §2.3.1, §3. Values are from app. A.2 unless marked otherwise.

Main findings, as reported:

• Calibration gap §3, fig. 2, fig. 3. For all three LLMs and both question types, ECE for human confidence with default explanations is well above ECE for model confidence. The human error is mostly overconfidence on multiple-choice questions §3, fig. 3. For short answer, the model is somewhat underconfident §3, fig. 3.
• Discrimination gap §3, fig. 2. Model AUC is 0.751 (GPT-3.5), 0.746 (PaLM2) and 0.781 (GPT-4o). Default-explanation human AUC is 0.589, 0.602 and 0.592 (see Pitfalls for the 0.592 vs 0.593 difference) §3.
• Uncertainty wording moves confidence §3.1, fig. 4. Low-confidence wording gives lower human confidence than medium, and medium gives lower than high; BF above 100 in all three experiments §3.1.
• Length raises confidence, not discrimination §3.1. Long explanations beat short ones (BF = 25); short beat uncertainty-only (BF above 100). Mean AUC for long and uncertainty-only explanations is 0.54 and 0.57, with BF = 0.23 §3.1.
• Selection rule narrows both gaps §3.2.2, fig. 2. The calibration gap narrows for all three LLMs. AUC improves with BF above 100, 6.48 and above 100 for Experiments 2a, 2b and 2c §3.2.2. The post-rule ECE and AUC values are shown in fig. 2 only; the text gives BF values, not the numbers.
• Accuracy is not improved §3.3.1. Participants' own answers were about as accurate as the LLM's on the multiple-choice questions (51% vs 52% for GPT-3.5; 45% vs 47% for PaLM2). Participants agreed with the model's answer in 82% of responses. When participants changed the answer, they were 33% accurate, against 39% for the LLM on those questions §3.3.1. The selection rule raised calibration and discrimination but not decision accuracy in 2a and 2b §3.3.1.
• Self-rated expertise does not change accuracy estimates §3.3.2. Splitting participants at 50% self-rated expertise gives AUC 0.600 vs 0.579 (GPT-3.5), BF below 1, and ECE 0.289 vs 0.292. No effect for PaLM2 §3.3.2.
• Rater agreement app. A.4. Mean participant-to-participant correlation across the nine style cells is 0.550, 0.547 and 0.336 for 2a, 2b and 2c.
• Model accuracy on full samples app. B. GPT-3.5 on 14,042 MMLU questions: 63% accuracy, AUC 0.78; PaLM2: 51%, AUC 0.73. GPT-4o on 5000 TriviaQA: 85% accuracy, AUC 0.85. On the 336 subset: 63%, AUC 0.78.

Paper vs code

No code was read for this card.

The paper's Data Availability and Code Availability sections both point to an OSF project, osf.io/y7pr6 §Data Availability, §Code Availability. The project's API metadata gives the title and no license name. Its storage holds a folder named "Code", which contains one archive, code.zip, and a "Behavioral data" folder. The listing has no LICENSE or README file. The archive could not be downloaded or read with the tools used here. Hence:

• no code field is set, and no code_commit or code_license;
• no discrepancy between paper and code has been checked;
• every hyperparameter above is taken from the paper only.

A later pass with access to code.zip should check the ECE bin count, the threshold grid step, the combining formula for the objective, and the short-answer model string.

Pitfalls

• The reduced gaps come from a simulation. The selection rule filters existing ratings to those whose style matches the rule §3.2.1. No participant saw an explanation chosen by the rule. The paper's Discussion says the two-step generation (answer and confidence first, explanation second) is a limitation §4. Do not read the gain as a deployed result.
• Thresholds were tuned on the same data. The thresholds are optimised on the ratings they are then evaluated on app. C. The authors report that the AUC is better than baseline for 52% of parameter combinations with BF above 3 for Exp. 2a, so the gain depends on the setting app. C.
• Objective is underspecified. "ECE weighted at 33% relative to AUC" does not state how the two are combined, nor the grid step app. C.
• Model name for short answer. The text calls the model GPT-4o and gives the API string gpt-4o-mini §2.2. Use the API string when reproducing.
• Accuracy figures disagree between sections. The 350 MC questions are said to have 55% (GPT-3.5) and 50% (PaLM2) accuracy §2.2. The participants' section gives the LLM accuracy as 52% and 47% §3.3.1. The paper does not explain the gap; it may be a different question set or a different scoring.
• Small numeric discrepancy. Experiment 1c's AUC is 0.592 in the text §3 and 0.593 in the appendix table app. A.2.
• ECE is computed per group, AUC per participant. Human ECE is computed at the group level because data are sparse; AUC gets per-participant intervals fig. 2. Human ECE has no per-participant spread.
• Bin count for ECE is not given §2.4.1. Changing M changes the numbers in the table.
• Sampling enriches low confidence. Questions were drawn to be roughly uniform over model-confidence bins §2.2, so the sample's accuracy is not the same as the datasets' app. B. The 336 TriviaQA subset has 63% accuracy against 85% for the 5000-question sample app. B.
• Incomplete answers are dropped. For the full GPT-3.5 MMLU run, 8.7% of answers are removed as incomplete app. B. The count for the 350-question sample is not stated.
• Temperature 0.7 for short answers. Answer generation and the true/false judgement both run at temperature 0.7 §2.2, so the model confidence varies between runs. The text does not give the number of samples averaged.
• Explanation wording is generated by the model. The uncertainty phrases come from a prompt; the paper notes that expressions of uncertainty are not confined to the first words §2.3.3. Results may depend on the exact wording the model produced.
• The mechanism is conjectured. The authors propose that RLHF preference for longer or more assertive text, and answer-first decoding that makes the explanation consistent with the answer, cause the overconfident explanations. These are hypotheses, not tested here §4.
• Version of the paper. This card is compiled from arXiv v2 (updated 2025-02-13). The abstract also names a journal version (Nature Machine Intelligence); its text may differ.
• Sample population. Participants were US-based native English speakers on Prolific, and most lacked domain expertise. The authors read their failure to beat the model's answers as a lack of independent knowledge §3.3.1.
• Model accessed as an API. Snapshot dates for the models are not given in the compiled text, so exact outputs may not be reproducible.

Lineage

• Builds on: the pTrue approach for short-answer self-evaluation, from Kadavath et al. (2022), used for the short-answer confidence §2.2; expected calibration error as formalised in Guo et al. (2017) and Naeini et al. (2015) §2.4.1; the Mann–Whitney view of AUC §2.4.2.
• Compared against: the model's own token-probability confidence, used as the reference for human confidence §2.2.
• Related: uncertainty-verbalisation prompts, from Xiong et al. (2023) and Zhou et al. (2023) §2.2; a trivia-task study of verbal uncertainty phrases by Zhou et al. (2024) §4.
• Superseded by: not known from content already in this wiki.