The Curious Case of Hallucinations in Neural Machine Translation

In plain words

Machine translation models sometimes produce output that reads smoothly but has nothing to do with the sentence they were given. The paper calls these hallucinations and asks where they come from. Its answer ties them to what the model memorized during training and to how clean the training data was.

The first part looks at what happens when an input is disturbed slightly, for example by adding one extra token at the start. Training examples the model has memorized are the ones most likely to break under such a disturbance. The authors measure how strongly each training example is memorized and show that the most memorized examples break far more often than random ones.

The second part shows that broken training pairs leave a visible fingerprint. If the training data contains mismatched source and target sentences, the trained model later produces either fluent but unrelated sentences or loops that repeat word groups. The kind of mismatch decides which failure shows up.

The third part looks at methods that grow training sets by generating data with a model, such as back-translation and sequence-level knowledge distillation. When a model that already hallucinates is used for this, the hallucinations are carried into the new data and increase, so the problem gets worse.

TL;DR

The paper studies hallucinations in neural machine translation: fluent outputs that have no relation to the source. It tests two hypotheses on IWSLT 2014 De-En. Training samples that are most memorized hallucinate most under input perturbation §4.1, and specific corpus-level noise patterns produce specific hallucination types §4.2. Back-translation and sequence-level knowledge distillation amplify these hallucinations, and the paper proposes a reference-free corpus-level estimator §5. The authors' repository contains only a README and an MIT license, so the methods must be implemented from the paper.

Problem

Neural machine translation (NMT) works well in high-to-moderate resource settings, but it has known failure modes. Hallucinations are the most extreme: outputs that are fluent yet decoupled from the source §1. Prior work reported them under source perturbation and under out-of-domain input, but no coherent empirically validated theory explained the phenomenon §1, §2.1. One earlier explanation, decoder instability, was not confirmed by the engineering attempts reported in that work §1. Manual detection of natural hallucinations is too slow for fast experiment cycles §2.1. The analysis relies mainly on the IWSLT 2014 De-En corpus, and the authors state that scaling is not tested §7.

Method

Categories §3. The paper separates two kinds of hallucination.

• Hallucination under perturbation (HP): the outputs for the clean and the perturbed input differ drastically. Detection follows the procedure of Lee et al. (2018), modified as described below §3 alg. 2.
• Natural hallucination (NH): for an unperturbed input the output is severely inadequate, whether fluent or not §3.
  • Detached (DH): fluent but completely inadequate §3 fig. 1.
  • Oscillatory (OH): inadequate and containing repeated n-grams §3 fig. 2.
  • Shorter outputs and source copies are excluded from NH by definition §3.

H1: memorized samples hallucinate more under perturbation §4.1. The authors adapt the memorization value estimator of Feldman and Zhang (2020) to sequences. The accuracy metric is replaced by a sequence-overlap metric, chrF or BLEU §4.1 [fn. in §4.1]. The memorization value of a sample is the change in the average metric for that sample between models trained with it and models trained without it §4.1.1 alg. 1. The HP procedure alg. 2 differs from Lee et al. in three ways §4.1:

1. Inputs are word-tokenized before perturbation, not BPE-tokenized.
2. Only insertion at the first position is reported, which the authors consider the most reliable way to produce hallucinations §4.1.
3. The inserted tokens are sampled from the 100 most common tokens of the training corpus §4.1.

The memorized set is the 100 samples with the highest memorization values. The random set has the same size and is drawn from the remaining samples §4.1.1.

# Memorization Value Estimator  [alg. 1] [§4.1]
# Input:  training pairs S (n pairs), learning algorithm A, trials t,
#         metric M (chrF, BLEU or exact-match accuracy)
# Output: memorization value mem[i] for each training pair i
def memorization_values(S, A, t, n, m, M):
    # Draw t random index subsets of size m (m about 101K)  [alg. 1] [§4.1.1]
    subsets = [random_subset(n, m) for trial in range(t)]
    # Train one NMT model per subset  [alg. 1] [§4.1.1]
    models = [A(S[index]) for index in subsets]
    mem = [0.0] * n
    for i in range(n):
        # Score each model's output for pair i against its target  [§4.1]
        seen = [
            M(h, S[i])
            for h, index in zip(models, subsets)
            if i in index
        ]
        # The same score for models that did not train on pair i  [alg. 1]
        unseen = [
            M(h, S[i])
            for h, index in zip(models, subsets)
            if i not in index
        ]
        # Memorization value: mean with minus mean without  [alg. 1]
        mem[i] = mean(seen) - mean(unseen)
    return mem

# Hallucination under Perturbation  [alg. 2] [§4.1]
# Input:  model, parallel corpus (X, Y), token set T (30 tokens)
# Output: list H of hallucinated sources (one entry per flagged token)
def perturbation_hallucinations(model, X, Y, T):
    # adjusted_bleu is used by the paper but never defined in it
    hallucinated = []
    for x, y in zip(X, Y):
        # Keep inputs whose clean output is close to the reference  [alg. 2]
        y_pred = model(x)
        if adjusted_bleu(y_pred, y) > 0.09:
            for token in T:
                # Insert the token at the first position only  [§4.1]
                x_pert = [token] + x
                y_pert = model(x_pert)
                # Flag when the perturbed output moves far from the clean one
                # [alg. 2]
                if adjusted_bleu(y_pert, y_pred) < 0.01:
                    hallucinated.append(x)
    return hallucinated

The memorization value is written as a formula in the algorithm alg. 1. The source has no numbered equations, so the formula is cited to the algorithm:

\mathrm{mem}(A, S, i) = \mathbb{E}_{h \leftarrow A(S)}[M] - \mathbb{E}_{h \leftarrow A(S \setminus i)}[M] alg. 1

H2: corpus-level noise patterns determine the NH type §4.2. The authors build an invalid reference set (IRS) of 21 detached source-target pairs. They combine IRS items with WMT 2014 De-En sentences into four noise sets of the same size, about 21K pairs or roughly 12% of the 160K clean training pairs §4.2.1. The four patterns are named by whether the source and target are unique or repeated §4.2.1:

• UU: random unique WMT sources, each paired with an unrelated unique random WMT target.
• RR: 21 IRS sources, each paired with an unrelated IRS target, each pair repeated 1000 times.
• RU: the same 21 IRS sources, each repeated 1000 times, paired with unrelated unique WMT targets.
• UR: 21 IRS targets, each repeated 1000 times, paired with unrelated unique WMT sources.

Models are trained with each noise set added to the clean corpus, with the same settings as in §4.1, except that BPE is learned on the noisy corpus §4.2.1. Evaluation uses three sets: the IWSLT test set for generalization, the IRS (which overlaps the noise sets in different ways) for memorization, and the valid-reference set (VRS), which pairs the IRS sources with their correct targets §4.2.1. Metrics are BLEU on each set, the manually identified NH and OH rates on the IRS, the rate of IRS outputs that exactly match a training reference, and the number of unique bigrams in IRS outputs relative to the possible count §4.2.1.

Amplification §5. The NH estimator is reference-free and works at corpus level §5. It decodes the sources, scores each translation against its source with a cross-lingual similarity model §5.1, and flags two groups:

• F1: translations whose most repeated n-gram occurs more often than the most repeated n-gram of the source, by at least t alg. 3.
• F2: translations that are paired with multiple unique sources alg. 3.

The estimate is the set of flagged translations that also lie in the bottom epsilon percent of similarity scores alg. 3. The parameters are epsilon = 1 (1%), n = 4, t = 2, and LASER as the similarity model §5.1. Repeats are counted on the translated sources rather than on the training set, which the authors call a simplifying assumption §5.

# Corpus-level NH Estimator  [alg. 3] [§5]
# Input:  sources S, similarity model X (LASER), NMT model M, epsilon,
#         n-gram order n, repeat threshold t
# Output: indices of approximate natural hallucinations
def estimate_natural_hallucinations(S, X, M, epsilon, n, t):
    # Decode every source sentence  [alg. 3]
    translations = [M.decode(s) for s in S]
    # Cross-lingual similarity of each (translation, source) pair  [§5.1]
    similarity = [X.score(tr, s) for tr, s in zip(translations, S)]
    # F1: top repeated n-gram in the output exceeds the top repeated
    # source n-gram by at least t  [alg. 3] [§5]
    repeat_hits = {
        i
        for i, (tr, s) in enumerate(zip(translations, S))
        if top_ngram_count(tr, n) - top_ngram_count(s, n) >= t
    }
    # F2: translations paired with multiple unique sources  [alg. 3]
    # The exact rule is not stated (see Pitfalls)
    shared_hits = outputs_shared_by_sources(translations, S)
    # Keep only the bottom epsilon percent by similarity  [alg. 3] [§5.1]
    low_similarity = bottom_percent(similarity, epsilon)
    return (low_similarity & repeat_hits) | (low_similarity & shared_hits)

Back-translation uses one million English sentences from WMT 17 De-En as the monolingual corpus, with sampling-based decoding, and generates En-De data with each of the noisy models §5.1. Sequence-level distillation decodes the clean IWSLT 2014 De-En training corpus with beam size 5 §5.1.

Discussion §6. The authors propose, without testing them, several ways to reduce hallucinations: data augmentation for memorized samples §6.2, early-learning or loss-truncation style robust training §6.2, and corpus filtering of invalid pairs §6.2. They also show a qualitative sample of highly memorized pairs (tab. 9) against lower-valued random pairs (tab. 10, in appendix A) §6.1.

Hyperparameters

Results

Author-reported. Setup: IWSLT 2014 De-En (160K pairs), six-layer Transformer with 42M parameters, hardware not stated. The H1 comparisons use the random set as the baseline. The H2 comparisons use the model trained without noise as the baseline §4.1.1 §4.2.1.

Paper vs code

The repository linked from the abstract is https://github.com/vyraun/hallucinations, owned by the first author. It is MIT-licensed, copyright Vikas Raunak, and the code was checked at commit ea2216ad8efffb48a116029259871522b2f55144 LICENSE. At that commit the repository contains only a README and a LICENSE file. It has no implementation of the memorization estimator, the perturbation detector, the noise generator, the NH estimator, training configuration, or the data README.md.

The abstract says the authors released the datasets and code to replicate the results [abstract]. The repository does not contain either, so no paper-versus-code comparison of hyperparameters or algorithm steps is possible.

The README points to an oscillatory-hallucination filter in the NL-Augmenter project, which is hosted under a different organisation (GEM-benchmark) README.md. It is not the authors' repository, so it was not cloned or used here.

Pitfalls

• Training recipe is missing. The optimizer, learning rate, schedule, warmup and training length are not stated §4.1.1. Memorization values depend on how long each model trains, so the H1 result cannot be reproduced without choosing these yourself and reporting them.
• Cost of H1. The estimator needs t = 10 full NMT trainings on subsets of about 101K pairs each, and then scores every training pair under every model §4.1.1 alg. 1. The sampling scheme (whether the subsets are random, balanced, or resampled) is not stated.
• adjusted-BLEU is undefined. Algorithm 2 relies on it with thresholds 0.09 and 0.01 alg. 2, but the paper never defines it, and the code is not released. Thresholds are not justified. Any implementation has to pick a definition and state it.
• Tokenization order. The perturbation is defined on word-tokenized inputs §4.1, but the models run on lowercased BPE §4.1.1. The order of tokenization, lowercasing and token insertion is not stated.
• Counting hallucinations. A source is added to H once per flagged token, so it can appear several times alg. 2 §4.1. The "unique" and "total" counts in tables 1–3 therefore measure different things. Follow the paper's definitions.
• Text contradicts the tables. The text says RR noise causes the fewest hallucinations for both KD and BT §5.1. In tab. 7 the KD-RR count (136) is above KD-UU (119) and KD-None (120). In tab. 8 BT-RU (3) is below BT-RR (12). Use the tables. The sentence is not reproduced in this card.
• No clean baseline for back-translation. Table 8 has no no-noise row, so amplification relative to clean back-translated data cannot be read from it. Table 7 includes a KD-None row tab. 7 tab. 8.
• Tiny evaluation set. The IRS has 21 sources. The percentages in tables 5 and 6 appear to be counts out of 21 (for example 16/21 = 76.19%), so one source changes a rate by about 4.8 points tab. 5 tab. 6.
• Manual labels. NH and OH on the IRS are identified by hand §4.2.1. The number of annotators, the guidelines and any agreement measure are not stated. The automatic estimator misses many OH cases, according to the authors §5.1.
• Confound in H2. The noisy models learn BPE on their noisy corpora, so their subword vocabularies differ from the baseline's §4.2.1. The paper does not separate this effect from the effect of the noise. H2 is validated by construction §4.2, and the authors leave downstream effects to future work §5.1.
• Ambiguous F2 rule. "Translations paired with multiple unique sources" alg. 3 does not say how translations are matched to sources when the decoded set is built. The code cannot resolve it.
• Repeat assumption. Counting repeats on decoded outputs rather than the training set assumes that a large enough source set will reproduce any copied training target §5. This is stated as an assumption, not tested.
• No significance testing. The attention statistics are averages over whole sets §4.1.1, tab. 4, and no test is reported.
• Scale. The main experiments use one corpus, IWSLT 2014 De-En. The authors expect the results to carry over to larger systems, but do not test this §7.
• Citation key. The source cites \citet{mueller_2} for out-of-domain hallucinations §2.1, and that bibliography entry is Wang and Sennrich (2020). Check the reference before using it.

Lineage

• Builds on: Feldman (2020), "Does Learning Require Memorization? A Short Tale about a Long Tail", for the long-tail theory §2.2. Feldman and Zhang (2020), "What neural networks memorize and why", for the memorization value estimator, adapted here from accuracy to sequence overlap §2.2 §4.1. Lee et al. (2018), "Hallucinations in neural machine translation", for the perturbation-based detector, with three changes §2.1 §4.1.
• Compared against: Wang and Sennrich (2020), "On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation", cited for out-of-domain hallucinations and detached outputs §2.1.
• Data generation studied: back-translation, Edunov et al. (2018), "Understanding Back-Translation at Scale" §5. Sequence-level knowledge distillation, Kim and Rush (2016) §5.
• Also used: LASER, Artetxe and Schwenk (2019), as the cross-lingual similarity model §5.1.
• Superseded by: not known from content in this wiki.