Sources of Hallucination by Large Language Models on Inference Tasks

In plain words

The paper asks whether large language models really work out whether one sentence logically follows from another, or whether they lean on shortcuts. Given a premise and a hypothesis, a model should say "yes, it follows" only when the premise supports the hypothesis. The authors show that the models often say "yes" anyway.

Two shortcuts explain much of this. The first: if the model already believes the hypothesis is true, it tends to say "yes" whatever the premise says. The model seems to be recalling the hypothesis from its training text rather than reasoning from the premise. Names of people and places seem to act as lookup keys for that recall. The second: if the premise uses a word that is rarer in text than the word in the hypothesis, the model also tends to say "yes". That pattern follows from how words are used in written text, not from meaning.

When the test sentences break these shortcuts, the models do much worse. The authors offer these breaking cases as controls for judging future models.

TL;DR

Three LLM families, probed with controlled edits to two entailment datasets, make false "entails" calls in a predictable way. Those calls rise when the hypothesis is one the model already affirms §5.1 and when the premise predicate is less frequent than the hypothesis §7.1. Replacing entities with typed placeholders removes some of the first effect §6.1. Performance on samples that conflict with either bias is much lower, which the authors report as the main cost §8.

Problem

Benchmarks for natural language inference (NLI) can look solved when a model uses cues that have nothing to do with the logic of the sentences. The paper targets two cues from pretraining: memorised sentences, and corpus-level word frequency §1. Its datasets are chosen to keep the question about predicates, since the entailment in the directional Levy/Holt subset can be decided from the predicates alone §3.2. The paper excludes knowledge-heavy datasets such as MMLU and Natural Questions so that memorised facts do not decide the answer §3.2.

The method assumes the datasets' gold labels are correct and that the probe prompts behave the same across models. It also assumes the Google N-gram counts are a fair proxy for the text the models were trained on, which the paper says is only an approximation §3.1, §8.

Method

The study has three experiments and one impact analysis. Each uses a 3-way prompt (A) Entailment, B) Neutral, C) Contradiction) in few-shot form §4.2. Choice A counts as "Entail". Choices B and C both count as "No-Entail" §4.2.

Two biases. Both are labels on each test sample. The frequency label is model-agnostic; the attestation label comes from each model's own answers §3.1.

1. Attestation bias (Λ). The model is asked the hypothesis alone and answers True, Unknown or False. "True" marks the hypothesis as attested §3.1. The paper notes that perplexity could replace this probe, but GPT-3 does not expose it §3.1, footnote.
2. Relative frequency bias (Φ). Google N-gram counts for the premise and the hypothesis predicates, lemmatised, averaged over the years 1950 to 2019. Named entities are excluded. The sample is labelled Φ< when the premise is clearly less frequent, Φ> when clearly more frequent, and left out otherwise §3.1, §7.

Datasets §3.2, app. G:

• Levy/Holt, directional subset. Premise-hypothesis pairs with one predicate and two entity arguments. Each entailment holds in one direction only. Dev has 630 entries and test has 1,784; the classes are balanced 50/50.
• RTE-1. Used to show the effect in general NLI. The paper leaves it out of the predicate experiments because its statements are not consistently structured §3.2. Dev has 567 entries and test has 800; the dev classes differ by one.

Dataset transformations §3.3. Each entity gets one of 48 FIGER types, with "thing" as the fallback §3.3.

• I (standard). The original sample.
• I_RandPrem. The premise predicate is replaced by a random predicate that has the same argument types, with the same entities. Every sample is relabelled No-Entail, so every Entail prediction is a false positive §3.3. The paper checked a sample by hand and found 9.6% of the random premises are in fact entailing §3.3, footnote.
• I^GenArg. Each entity is replaced with a typed placeholder such as "location X". The label is kept §3.3.
• I^RandArg↓ and I^RandArg↑. Each entity is replaced with a real entity of the same type, drawn from the 5% least common or the 5% most common entities in NewsCrawl §3.3.

Prompting §4.1, §4.2, app. A.

• Models: LLaMA-65B (base), text-davinci-003 ("GPT-3.5", instruction-tuned) and PaLM-540B (base) §4.1.
• Templates: four were used. Each was chosen by the best dev-set AUC from a set of candidates §4.2, app. A.
• Zero-shot prompts were near random for the base models app. A. The main experiments therefore use four hand-written examples with explanations, placed before the query §4.2.
• The hypothesis-only probe uses three labelled examples with no explanations app. G, example prompts.

Score. The letter choice is turned into a probability of Entail with the mapping below. The mapping preserves the ordering of confidences, which is enough for precision–recall curves §4.2.

S_{\text{ent}} = 0.5 + 0.5\,\mathbb{I}[\text{tok}=\text{A}]\,S_{\text{tok}} - 0.5\,\mathbb{I}[\text{tok}\in\{\text{B},\text{C}\}]\,S_{\text{tok}} §4.2, unnumbered

# Turn one 3-way answer into an entailment decision  [§4.2]
# Input:  choice in {A, B, C}; token_prob = probability of that token
# Output: entail (bool), s_ent in [0, 1]

def score_answer(choice, token_prob):
    # Choice A means Entail; B and C both mean No-Entail  [§4.2]
    if choice == 'A':
        s_ent = 0.5 + 0.5 * token_prob
    else:
        s_ent = 0.5 - 0.5 * token_prob
    entail = choice == 'A'
    return entail, s_ent

# Derive the two bias labels for one sample  [§3.1] [§5.1]
# Input:  hyp_only_choice in {A, B, C}  (hypothesis alone, original entities)
#         prem_freqs, hyp_freqs: yearly N-gram counts, 1800-2019
# Output: attested (bool), phi in {"<", ">", "~"}

def bias_labels(hyp_only_choice, prem_freqs, hyp_freqs, margin=5.0):
    # Attested only when the model answers True (A)  [§3.1]
    attested = hyp_only_choice == 'A'
    # Average the years 1950-2019 (index 150 onward)  [§3.1] [§7]
    prem_avg = sum(prem_freqs[150:]) / len(prem_freqs[150:])
    hyp_avg = sum(hyp_freqs[150:]) / len(hyp_freqs[150:])
    # Ratio decides the class; the margin value is in the code, not the text
    ratio = hyp_avg / prem_avg
    if ratio > margin:
        phi = "<"
    elif ratio < 1 / margin:
        phi = ">"
    else:
        phi = "~"
    return attested, phi

Consistency §8. A sample is consistent with a bias when the bias prediction agrees with the gold label, and adversarial when it disagrees. The bias prediction for Λ is the model's own hypothesis-only answer; for Φ it is the N-gram comparison.

Metrics. Experiments 1 to 3 use conditional probabilities of Entail. Experiment 2 uses precision and recall. The impact analysis uses AUC over the confidence score, rescaled so that random is 0% and a perfect classifier is 100% (AUC_norm), following Li et al. (2022) §8. The paper does not give the rescaling formula.

Experiment 1: attestation bias §5

Compare P(Entail) for I and I_RandPrem, split by the model's own attestation label. On I_RandPrem every Entail is false, so a higher rate for attested hypotheses shows the bias §5. Results appear in fig. 1 (I) and fig. 2 (I_RandPrem).

Experiment 2: entities as indices §6

Measure precision and recall on the Entail class under I, I^GenArg, I^RandArg↓ and I^RandArg↑ §6. Results appear in tab. 4.

Experiment 3: relative frequency bias §7

Split I_RandPrem and I_RandPrem^GenArg by Φ. The GenArg variant removes the entity cue so that the frequency effect can be seen on its own §7. Results appear in fig. 3.

Impact on performance §8

Compute AUC_norm on consistent and adversarial subsets of I and I^GenArg, for both Λ and Φ, on Levy/Holt and RTE-1 §8, tab. 5. Subset sizes are given in app. D.

Hyperparameters

Values in the paper are cited to their section. Values only in the code are cited to the file. "not stated" means neither source gives the value.

Results

Author-reported. Setup: Levy/Holt directional test set (1,784 entries) and RTE-1 test set (800 entries) app. G, three LLM families, four templates chosen on dev, and four few-shot examples §4.2. Hardware and API costs are not stated.

Experiment 1 (attestation). On I_RandPrem, a model that calls the hypothesis attested is this many times more likely to predict Entail than one that does not: LLaMA-65B 1.9x, GPT-3.5 2.2x, PaLM-540B 2.0x §5.1 fig. 2. The same direction holds on the original I task fig. 1. Under an instruction to ignore attestedness, GPT-3.5 shows only a small change app. C:

Experiment 2 (entity indices). Levy/Holt directional, Entail class tab. 4:

Recall falls in every model. Infrequent replacement entities (↓) keep more recall than frequent ones (↑) §6.1. RTE-1 shows the same direction, with smaller differences for PaLM app. B.

Experiment 3 (relative frequency). On random premises, P(Entail) is higher when Φ< than when Φ>. The ratio is 1.6x (LLaMA), 1.8x (GPT-3.5) and 2.0x (PaLM) when entities are replaced by typed placeholders §7.1 fig. 3. With the original entities the effect is moderate §7.1 fig. 3.

Impact (AUC_norm, consistent minus adversarial). Table 5 in §8. Levy/Holt:

RTE-1:

The authors' averages: the Φ gap grows from 10.1 to 16.1 points on Levy/Holt and from 14.8 to 16.5 points on RTE-1 when entities are typed §8. The paper explains the smaller Φ gaps as a sign that frequency is harder to capture and that N-gram counts are a rough proxy §8.

GPT-4 (zero-shot, F1 only) app. F. On Levy/Holt, the consistent subset scores 85.1 under Λ and the adversarial subset 67.6; random baselines are 70.3 and 62.0. Under Φ, the scores are 74.6 and 69.7, against a random baseline of 66.7 for both.

Paper vs code

Repository: https://github.com/Teddy-Li/LLM-NLI-Analysis at commit 463c6441dea387c542b9b7fb23e2cead138764f2, cloned with --depth 1. The paper's abstract links it directly, so no further search was needed. The owner is taken to be the first author. The paper's source names a bibliography file references_teddy [raw source: acl_latex.tex], and the first author's nickname is Teddy; this is an inference from those names. The repository has no LICENSE file, so no code text is quoted here; every point is cited by path and line only.

Checked and matching the paper:

• The GPT score mapping matches S_ent §4.2 gpt3_inference.py.
• The four templates that the code activates are the four the paper uses §4.2 app. A gpt3_inference.py.
• The four few-shot examples with explanations match app. A utils.py.

Discrepancies:

• Consensus attestation. The paper uses a majority vote over three models app. E. The code combines only GPT-3.5 and LLaMA-65B, and the "vote" is an OR: a sample is attested if either model says True poll_attestation.py. The repository has no PaLM code and no PaLM outputs, so the PaLM numbers cannot be checked from it.
• Random entity replacements. The paper draws replacements from the 5% least and most common entities in NewsCrawl §3.3. The code draws them from the split's own entities of the same type, weighted by how often each occurs, with seed 4242 shuffle_entities_sametype.py shuffle_entities_sametype.py shuffle_entities_sametype.py. The "lowfreq" and "highfreq" files the inference scripts read are not generated anywhere in the repository. The README says they come from an external archive.
• Frequency window. The paper averages 1950 to 2019 §3.1. The randprem script does that randprem_experiments.py. The controlled-subset script defaults to the single year 2000 instead frequency_controlled_experiments.py frequency_controlled_experiments.py.
• Frequency margin. The paper text gives no number for the margin. The code uses 5 randprem_experiments.py frequency_controlled_experiments.py. The paper's claim that about one third of samples are "roughly equal" §7 is not checked by any script in the repository.
• Score for LLaMA. For GPT the score uses the token probability gpt3_inference.py. For LLaMA the code takes the generation output score for the chosen letter, sets negative values to 0, and maps x to x/(1+x) llama_inference.py. The paper applies S_ent to all models §4.2. The LLaMA AUC values are therefore not on the same scale as the GPT values.
• Hypothesis-only probe. The code always uses the original entities for the attestation probe, even when the experiment changes the entities veracity_controlled_experiment.py. The paper's text does not say so.
• Search-engine branch. In the RTE-1 search-engine comparison, samples whose ratio is below the margin are labelled C, where the symmetric lower bound should be 1/margin randprem_experiments.py. The paper reports only the N-gram version of Φ, so the published numbers are probably not affected, but the branch is wrong as written.
• RTE-1 n-gram aggregation. The default is "min" in the randprem script and "max" in the sweep script randprem_experiments.py frequency_controlled_experiments.py. The paper's text does not say which one it uses for RTE-1.
• Unparseable output. GPT output that is not A, B or C is recorded as B with score 0 gpt3_inference.py gpt3_inference.py, which counts as No-Entail. The paper does not mention this rule.
• README. It names attestation_controlled_experiment.py; the file is veracity_controlled_experiment.py.

Not checked: whether the reported numbers can be reproduced. That needs the external dataset archive and API access to text-davinci-003, and LLaMA-65B weights. The repository contains the raw model outputs for GPT and LLaMA, which could be used to recompute tables without rerunning models .

Pitfalls

• Random premises are not all false. The authors' manual check found 9.6% of random premises are actually entailing §3.3, footnote. A raw Entail count on I_RandPrem therefore overstates the hallucination rate.
• Attestation is a model's own belief, not a fact. The label comes from the same model being tested, so a biased model can make the bias look consistent. The paper tests this with a majority over other models, but the code's version is a two-model OR and PaLM is absent app. E poll_attestation.py.
• Instructions barely help. Asking GPT-3.5 to ignore attestedness changes the gap only a little app. C. Do not treat a prompt instruction as a fix.
• Zero-shot is near random for base models. On the Levy/Holt dev set, zero-shot AUC was 56.6 (LLaMA), 61.5 (PaLM) and 64.5 (GPT-3.5), against 65.0, 80.2 and 74.6 with four examples app. A. Reported numbers depend on the four examples chosen.
• Template and example choice matter. Templates were chosen on dev by AUC §4.2. Changing the example set changes the results; the paper gives no sensitivity analysis for it.
• Scores are not probabilities for every model. See the LLaMA score mapping in Paper vs code. Compare AUC across models only after checking this.
• Frequencies are rough. Google N-gram counts use lemmatised predicates, and the paper says they are a crude estimate of the training corpora §8. The margin and window are choices, not stated facts §3.1.
• Entity pool effects. Random replacements drawn from the dataset itself are not the same as the NewsCrawl sample in the paper §3.3 shuffle_entities_sametype.py. Do not mix results from the two.
• Output parsing. GPT generation stops on a newline, and the first option letter decides the answer. LLaMA is scanned for the first option token among its first seven output tokens utils.py llama_inference.py. Check your parser before comparing rates.
• Small subsets. Subset sizes are given in app. D. Some adversarial subsets hold only about 220 to 300 entries, so AUC differences of a few points may be within noise. The paper gives no confidence intervals for Table 5 §8.
• GPT-4 is a side check. It is zero-shot, reported only as F1 app. F, and not comparable with the main table.

Lineage

• Builds on: the hypothesis-only baseline of Poliak et al. (2018), applied here with no training §2, §3.1.
• Builds on: Levy/Holt directional entailment (Levy et al. 2016; Holt 2019) and RTE-1 §3.2.
• Builds on: the prompt templates of Schmitt et al. (2021) §4.2.
• Related: Li et al. (2022) on dataset artifacts in RoBERTa-scale directional NLI, and the AUC_norm rescaling §2, §8.
• Related: Tirumala et al. (2022) and Carlini et al. (2023) on memorisation §2.
• Superseded by: not stated in the paper or the repository.