Survey of Hallucination in Natural Language Generation

In plain words

Language models write fluent text, and that is exactly what makes their mistakes dangerous: a summary can state the wrong year, a translation can add a friend who was never mentioned, and both read perfectly well. Researchers call this "hallucination", meaning text that is nonsensical or not backed by the input the model was given.

This survey sorts the problem out. It separates two kinds of hallucination. In the first, the output contradicts the source (the article says 2019, the summary says 2021). In the second, the output adds something the source cannot confirm or deny. That addition may even be true, but nobody can check it against the input. It then explains where hallucinations come from: noisy training data, models that misread or ignore their input, sampling randomness, and knowledge memorized during pre-training that overrides the input.

The bulk of the paper is a catalogue, written task by task (summaries, chatbots, question answering, table-to-text, translation, image captioning and, in a later update, large language models). For each task it covers how people measure hallucination automatically and how they try to reduce it. For an engineer, it works as a map of the field: which family of checks fits your setting, and what each one misses.

TL;DR

A taxonomy-and-catalogue survey, not a method paper. It defines hallucination as output that is nonsensical or not faithful to the given source. It splits it into intrinsic (contradicts the source) and extrinsic (cannot be verified from the source), and keeps faithfulness (w.r.t. the source) separate from factuality (w.r.t. world knowledge). Metrics are grouped as statistical (n-gram overlap such as PARENT and Knowledge F1) or model-based (IE, QA, NLI, trained faithfulness classifiers, LM-vs-conditional-LM), plus human evaluation. Mitigations are grouped as data-related (curate, filter, augment) or modeling/inference (architecture, planning, RL, multi-task, control codes, post-editing). Read it to pick a detection or mitigation family for a task and to learn each family's known failure modes. The LLM section (§13) was updated in January 2024. No code, no experiments of its own.

Problem

Neural NLG systems produce fluent output that is often unsupported by, or contradicts, their input §1. Standard quality metrics (ROUGE, BLEU, METEOR) do not capture this. The survey reports, citing Falke et al., that 25% of summaries from state-of-the-art abstractive summarizers contained hallucinated content §4. Hallucination matters for safety (medical summaries, translated drug instructions) and privacy (models reproducing training-set personal data, which the authors also count as unfaithfulness to the source) §1. Before this survey, reviews covered single tasks only (summarization, translation) §1. The scope is mainly unimodal, conditional tasks whose source text allows the output to be checked, plus a short VL section §1 and an LLM section added later §13.

Method

The "method" of a survey is its taxonomy. Section numbering of the compiled paper: §1 Introduction, §2 Definitions, §3 Contributors, §4 Metrics, §5 Mitigation, §6 Future Directions, §7 Abstractive Summarization, §8 Dialogue, §9 Generative QA, §10 Data-to-Text, §11 Neural Machine Translation, §12 Vision-Language, §13 LLMs, §14 Conclusion. There is no appendix. Tables: tab. 1 examples of intrinsic/extrinsic hallucination per task, tab. 2 metrics and mitigation works per task, tab. 3 MT hallucination categories. Figures: fig. 1 frameworks of training methods, fig. 2 captioning hallucination examples, fig. 3 VQA hallucination examples. The paper's only equations (CHAIR) are unnumbered, so they are cited to §12.1.

Definitions §2

• Hallucination: model output that is nonsensical or not faithful to the given source §2.
• Intrinsic: the output contradicts the source. Example: the source says the first Ebola vaccine was approved in 2019, and the summary says 2021 §2.1 tab. 1.
• Extrinsic: the output can be neither supported nor contradicted by the source. It may still be factually correct ("factual hallucination") and can add useful background, but most work treats it cautiously because it cannot be verified §2.1.
• Faithfulness = consistency with the source (antonym of hallucination). Factuality = consistency with world knowledge. The survey adopts Maynez et al.'s split; Dong et al. instead use the source as "fact", which makes the two terms the same §2.3.
• What counts as "source" differs by task §2.2: the input document (summarization), non-linguistic data (data-to-text), dialogue history and/or knowledge sentences (dialogue), world knowledge (GQA). Tolerance is very low for summarization and data-to-text and higher for open-domain dialogue, which also values engagingness §2.2. MT mostly labels an output a hallucination only when it is completely detached from the source §2.2 §11.1.
• For LLMs the reference covers both the input and the training data, so LLM hallucination leans extrinsic. The survey warns that training data is not always factual, so hallucination and factuality should still be kept apart §13.1.

Contributors (causes) §3

1. Source–reference divergence in data §3.1. It arises (a) from heuristic data collection (WikiBio pairs an infobox with the article's first sentence; citing Dhingra et al., 62% of those sentences contain information not in the infobox) or duplicated pretraining data, and (b) innately in tasks that reward diversity, such as chit-chat.
2. Training and inference §3.2. Hallucination appears even with little data divergence §3.2:
   • imperfect encoder representations / spurious correlations;
   • erroneous decoding: attending to the wrong part of the input (mixing facts between similar entities), and diversity-oriented decoding. Citing Dziri et al., top-k sampling correlates positively with hallucination;
   • exposure bias from teacher-forced MLE, worse for long targets;
   • parametric knowledge bias: pretrained models prefer memorized knowledge over the provided input.

Metrics §4

Statistical §4.1: n-gram overlap/contradiction between output and references.

• PARENT: n-gram entailment against both the source table and the reference text, combined as an F-score §4.1 §10.2. PARENT-T uses only the table, evaluating (T_i, G_i) instead of (T_i, R_i, G_i) §10.2.
• Knowledge F1: unigram F1 between the response and the gold knowledge used during data collection; needs datasets with labelled knowledge §4.1 §8.2.3. Rare F1 counts only infrequent words, defined as those in the lower half of the reference corpus's cumulative frequency distribution §8.2.3.
• BVSS (bag-of-vectors sentence similarity) for MT adequacy §4.1 §11.2.1.
• Limitation: lexical only; misses syntactic and semantic variation §4.1.

Model-based §4.2. All share the risk of errors propagating from the auxiliary model.

IE-based  [§4.2.1]
1. Extract (subject, relation, object) tuples from the output and from the source/reference.
2. Flag mismatches, e.g. (Brad Pitt, born-in, 1961) vs (Brad Pitt, born-in, 1963).
Variant: entity-only check with an NER model (assumes hallucination changes the entity set) [§7.2.1].

QA-based  (FEQA, QAGS, QuestEval)  [§4.2.2] [§7.2.1]
1. A question-generation model builds question/answer pairs from the generated text.
2. A QA model answers those questions using the source.
3. Score = similarity between the two answer sets.

NLI-based  [§4.2.3]
Score = entailment probability of the generated text given the source.
Refinements: long-premise NLI data [§7.2.1]; sentence-level application (SummaC_Conv) [§7.2.1];
dependency-arc-level entailment to localise errors [§4.2.3].

Faithfulness classifiers  [§4.2.4]
Train on task-specific data: synthetic data with inserted hallucinations, or human-labelled
responses (e.g. annotated Wizard-of-Wikipedia). FactCC is trained jointly on consistency
classification + supporting-span extraction + inconsistent-span extraction [§7.2.2].

LM-based  [§4.2.5]
1. Train an unconditional LM on targets only and a conditional LM_x on source+target.
2. Force-decode the output with both.
3. A token is hallucinated if the unconditional LM has the lower loss.
4. Score = (# hallucinated tokens) / |y|.

Human evaluation §4.3: either scoring on a scale or comparison against baselines/references. Annotation terms vary widely (faithfulness, factual consistency, fidelity, factualness, factuality, hallucination, fact contradicting, correctness, accuracy, informativeness); a few works rate intrinsic and extrinsic separately §4.3.

Task-specific metrics:

• Slot error rate for task-oriented dialogue §8.3.1: \text{SER} = (p+q)/N, with N the slots in the dialogue act, p missing slots and q redundant slots. Tree accuracy checks that the output's tree structure matches the input meaning representation §8.3.1.
• Data-to-text §10.2: table-record coverage and hallucinated-entity ratio (via entity recognition plus heuristic alignment); Data-QuestEval (synthetic QG/QA trained on (data, text) pairs, answers compared with BERTScore); two-direction NLI (input ⇒ text for omissions, text ⇒ input for hallucinations).
• MT §11.2: an auxiliary "evaluation decoder"; entropy of averaged attention (hallucinating models concentrate attention on one point), compared on clean vs perturbed test sets; token-level detection trained on synthetic LM-infilled data with 0/1 token labels; alignment-based detection via SimAlign (an unaligned target token counts as hallucinated); overlap with bilingual-synonym matching; Coverage Difference Ratio for under-translation; Approximate Natural Hallucination (ANH) detection via low unique-bigram counts and repeated targets.
• Image captioning, CHAIR §12.1:

\text{CHAIR}_i = \frac{\#\{\text{hallucinated objects}\}}{\#\{\text{all objects in ground truth}\}}, \qquad \text{CHAIR}_s = \frac{\#\{\text{hallucinated captions}\}}{\#\{\text{all captions}\}} \quad [§12.1]

On MSCOCO it uses the 80-object segmentation list with exact word/phrase matching §12.1.

• LLMs §13.2. Reference-dependent metrics are benchmarks: multiple-choice/cloze QA, long-tail selection by corpus frequency, popularity, recency or domain, adversarial sets like TruthfulQA, and automated pipelines that generate a query, retrieve evidence, then verify with NLI or an LLM §13.2.1. Reference-free metrics come in two kinds §13.2.2. Uncertainty-based ones aggregate token probabilities to word/sentence/passage level by average/max/min pooling, normalized product, or max/mean entropy. Consistency-based ones sample several completions and measure their agreement with BLEU, BERTScore, n-gram, NER overlap, NLI or an LLM judge; these work black-box, without logits.

Mitigation §5

Data-related §5.1:

• Build faithful datasets §5.1.1: annotators either write targets from scratch (risk: low diversity) or revise real sentences in three steps: (1) phrase trimming, (2) decontextualization, (3) syntax modification.
• Clean automatically §5.1.2. Instance filtering: (1) score each source–reference pair with a hallucination metric, (2) rank, (3) drop the bottom. Caveat: instance scores lose signal because divergence is word-level. MR correction for data-to-text: (1) parse an MR from the reference, (2) correct the input MR by slot matching.
• Augment the input §5.1.3 with entities, extracted relation triples, pre-executed operation results, synthetic perturbations, retrieved knowledge, or retrieved similar samples.

Modeling and inference §5.2:

• Architecture §5.2.1: dual (sequence + graph) encoders; source-conditioned bias, sparse attention, inductive attention; multi-branch, uncertainty-aware, dual (sequential + tree) and constrained decoders; decoding that reduces sampling randomness over time. Trade-off: fluency and diversity.
• Training §5.2.2 fig. 1: planning or sketching (two-step pipelines can amplify hallucination); RL with metric-inspired rewards, e.g. a slot-consistency reward, an NLI persona reward, or ROUGE + cloze score; multi-task learning (word alignment, entailment, rationale extraction); controllable generation with control codes; regularization; reconstruction losses.
• Post-processing §5.2.3: generate-then-refine correctors (SpanFact, a heuristic-trained corrector, entity-swap contrast candidates, HERMAN quantity re-ranking §7.3.3). Needs little training data; may produce ungrammatical text.

Task highlights: contrastive learning against synthetic hallucinated summaries §7.3.2; retrieval-augmented knowledge-grounded dialogue and a token-level critic + entity retriever for refinement §8.2.4; uncertainty-aware beam search adds a weighted uncertainty penalty to the beam objective §10.3 §12.1; for NMT, perturbation augmentation, corpus filtering, minimum risk training, scheduled sampling, and token loss truncation / decoder hidden-state masking §11.3.

LLMs §13.3: pretraining data quality (up-sampling factual sources); curated instruction data (e.g. LIMA's 1,000 examples, filtering Alpaca's 52k); reward-model data; RAG; RLHF; model editing (Elastic Weight Removal via the Fisher information); decoding (DoLa contrasts late vs early layers, CAD contrasts with vs without context, ITI shifts activations along truthfulness directions found by probing heads); CoT variants (Chain-of-Verification: draft → plan verification questions → answer them independently → revise); post-editing (PURR, REFEED, CRITIC); multi-LLM debate.

Hyperparameters

A survey: there is no model, training run or tuned value of its own. The only fixed quantities it states are definitions inside the metrics it describes:

Results

The survey runs no experiments. The numbers below are other works' results as reported in the survey; check the primary sources before relying on them.

Paper vs code

No official code.

Pitfalls

• Definitions are not interchangeable across tasks. "Source" means the document, the table, dialogue history plus knowledge, or world knowledge, depending on the task, and MT uses its own categories (detached, oscillatory, under-/over-translation). A metric built for one task's definition will mislabel another's. Fix the reference set before choosing a metric.
• Extrinsic ≠ wrong. An extrinsic hallucination can be true. Metrics that penalize everything unsupported penalize helpful background knowledge as well, which is a real issue for chit-chat and arguably for summarization, where there is no consensus.
• NLI labels are not faithfulness labels. "Putin is U.S. president" may be labelled entailed or neutral given "Putin is president", but it adds an unsupported fact. Off-the-shelf NLI also transfers poorly to long summarization premises; apply it per sentence or use long-premise NLI data.
• QA-based and sentence-level NLI metrics cannot localize which span is hallucinated. Use token-level or dependency-level methods if you need positions.
• Error propagation. IE, QG/QA, NLI and classifier metrics inherit their sub-models' errors. Report which sub-models you used, because scores are not comparable across them.
• Most metrics mix intrinsic and extrinsic hallucinations into one score, since they are hard to separate automatically. Don't read a single number as one or the other.
• Numbers are under-served. Few metrics or mitigations treat numerals specially, yet a wrong dosage or date is the most harmful error (the survey's example is an SpO₂ target given as 82–86% instead of 92–96%).
• The CHAIR_i formula and its prose disagree. The formula's denominator is "all objects in ground truth", while the text describes it as the fraction of object instances in each generated caption that are hallucinated. Rohrbach et al.'s original definition should decide; the survey is ambiguous.
• Internal inconsistencies in the examples. §7.1 describes the intrinsic summarization example as "rejected by the FDA in 2019", while tab. 1 shows "approved in 2021". §11.1 calls the MT source name "Mike", and tab. 1 glosses it as "Michael". The abstract says "two parts", then lists three. None of this affects the taxonomy.
• Uncertainty-based LLM detection needs logits and assumes low probability means hallucination. Consistency-based detection works black-box but costs several extra samples per query. The survey calls consistency methods "more reliable" without citing evidence.
• Mitigation has costs. Constrained or specialized decoders and post-editors can hurt fluency, diversity or grammaticality; two-stage planners can amplify errors; RLHF and safety tuning carry an alignment tax; RAG degrades when retrieval is poor.
• Snapshot. Sections 1–12 reflect work up to about 2022. Only §13 was updated (January 2024). Results quoted from other papers are second-hand.

Lineage

• Builds on: the intrinsic/extrinsic categorization of Maynez et al., Huang et al. and Dziri et al. §2.1, and earlier task-specific surveys of summarization and translation §1.
• Compared against: not applicable (survey). Metric meta-evaluations it points to: FRANK and TRUE §4.
• Superseded by: none in this wiki.