wikillm

Paper · 2024-05-09 · cs.CL

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

SliCK labels each fine-tuning question by sampling PaLM 2-S: HighlyKnown, MaybeKnown, WeaklyKnown or Unknown. In closed-book QA, more Unknown examples fitted during fine-tuning track lower test accuracy (linear fit, R² 0.86 in-distribution); early stopping or dropping them limits the loss. Author-reported; one base model.

Knowledge injectionInstruction tuningHallucination mitigationfine-tuninghallucinationclosed-book-qaknowledge-categorizationsupervised-fine-tuning

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A chat model is first trained on huge amounts of text, then fine-tuned on question-and-answer examples written to make it helpful. Some of those examples may contain facts the model never learned during its first training. The worry is that teaching it these new facts makes it more likely to state wrong answers confidently, even about things it already knew.

The authors test this with a controlled setup. They sort every training question by how well the base model already answers it, by sampling several answers to the same question. They then build training sets with different shares of questions the model did not know, and measure how well each fine-tuned model answers test questions on facts it knew before training.

They find that the model learns the unfamiliar questions much more slowly than the familiar ones. Once it does learn them, accuracy on what it already knew falls in proportion to how many of them it has absorbed. Stopping training early, or dropping the unfamiliar questions, reduces the damage. The results come from one model and one question-answering dataset.

TL;DR

  • Fine-tuning on questions whose answers the base model did not know slows learning, and once learned it lowers accuracy on facts the model already knew §4.1, §4.3.
  • SliCK sorts each training pair into HighlyKnown, MaybeKnown, WeaklyKnown or Unknown by sampling answers from the base model §3.
  • Test accuracy follows a linear model in the counts of fitted Known and Unknown examples, R² 0.86 in-distribution and 0.95 out-of-distribution eq. 1 tab. 1.
  • Only one base model (PaLM 2-S), closed-book QA and full fine-tuning were tested §2 app. E § Limitations.

Problem

Supervised fine-tuning on instruction data may add facts that the pre-trained model does not hold §1. A common conjecture holds that this trains the model to produce answers not grounded in its prior knowledge, which encourages hallucination §1. Before this work, there was no controlled way to tell, for a single training example, whether the model already knows its answer §3. The paper defines hallucination relative to pre-existing knowledge: questions the model answered correctly before fine-tuning, which it then answers wrongly app. B.

Method

Setup. The base model M is PaLM 2-S, used as a base model and not as an instruction-tuned one §2 §3. The data are closed-book question-answer pairs (q, a) from EntityQuestions, which turns Wikidata triplets into questions §2. The authors use 12 relations for training and in-distribution testing and hold out 7 relations as an out-of-distribution (OOD) test set §2 app. A. Exact match is the metric §2 app. C.

Data cleaning. Training pairs with more than one correct answer are removed, about 4.2% of train and 3.9% of test app. A. Pairs whose subject or object appears in the test set are removed from train, about 2.1% of train app. A. Five candidate OOD relations were dropped because they overlapped with training relations app. A.

SliCK categories. For each pair (q, a), M is prompted with 10 random 4-shot prompts. Each prompt uses exemplars from the same relation, drawn from the development split §3 app. C. Each prompt gives one greedy answer (temperature 0), and 16 sampled answers at temperature 0.5 with top-40 sampling app. C. The greedy score is the share of the 10 greedy answers that are correct. The sampled score is the share of the 160 sampled answers that are correct app. C. The four categories follow from these two scores §3 fig. 2.

Pseudocode
# SliCK category for one (question, answer) pair  [§3] [app. C]
# Input:  base model M, question q, gold answer a, exemplars of q's relation
# Output: "HighlyKnown", "MaybeKnown", "WeaklyKnown" or "Unknown"

def slick_category(model, question, answer, exemplars):
    # Build 10 random 4-shot prompts from the same relation  [app. C]
    prompts = [sample_4_shot(exemplars) for _ in range(10)]
    # One greedy answer per prompt, temperature 0  [§3]
    greedy_hits = [
        exact_match(model.greedy(p, question), answer) for p in prompts
    ]
    # 16 sampled answers per prompt, temperature 0.5, top-40  [app. C]
    sampled_hits = [
        exact_match(model.sample(p, question), answer)
        for p in prompts
        for _ in range(16)
    ]
    # Greedy score: fraction of the 10 greedy answers that are correct  [app. C]
    greedy_score = sum(greedy_hits) / len(greedy_hits)
    # Sampled score: share of the 160 samples that are correct  [app. C]
    sampled_score = sum(sampled_hits) / len(sampled_hits)
    # Greedy answer is always correct: HighlyKnown  [§3]
    if greedy_score == 1:
        return "HighlyKnown"
    # Greedy answer is sometimes correct: MaybeKnown  [§3]
    if greedy_score > 0:
        return "MaybeKnown"
    # Never greedy, but sometimes sampled: WeaklyKnown  [§3]
    if sampled_score > 0:
        return "WeaklyKnown"
    # Never correct under any decoding: Unknown  [§3]
    return "Unknown"

The four categories, with the paper's wording: HighlyKnown (greedy score 1), MaybeKnown (greedy score between 0 and 1), WeaklyKnown (greedy score 0, sampled score above 0), and Unknown (score 0 under all decoding). Together the first three form "Known" §3 fig. 2. The paper gives 24% HighlyKnown, 23% MaybeKnown, 17% WeaklyKnown and 36% Unknown for its data §3, footnote. See Pitfalls for a mismatch with the table totals.

Training-set variants. The main study fixes the training set size and varies the share of Unknown examples X%, with the remaining (100-X)% drawn from Known. The share is sampled inside each relation, so the relation mix stays the same §4 app. E. For the single-category study, |D| is set to the sum over relations of the smallest category size, giving 6142 pairs app. E tab. 4.

Fine-tuning. Full fine-tuning updates all parameters app. E. The model is evaluated on the development set after every epoch app. E. Two stopping points are compared: "early_stop" is the epoch with the best development accuracy, which the paper says falls after 5 to 10 epochs §4 app. E. "Convergence" is epoch 50, where the model fits 100% of the training set §4 app. E.

Pseudocode
# Fine-tune M on D, keep both stopping points  [app. E] [§4]
# Input:  base model M, training set D, development set V
# Output: early_stop weights, convergence weights

def finetune(model, train_set, dev_set):
    best_dev = -1.0
    early_stop_weights = None
    # Fixed learning rate 1e-5, batch 128, dropout 0.05  [app. E]
    for epoch in range(1, 51):
        train_one_epoch(model, train_set, lr=1e-5, batch_size=128)
        dev_accuracy = evaluate(model, dev_set)
        # Keep the epoch with the highest development accuracy  [app. E]
        if dev_accuracy > best_dev:
            best_dev = dev_accuracy
            early_stop_weights = copy(model)
    # Weights after 50 epochs are the convergence model  [app. E]
    return early_stop_weights, model

Linear model. The authors fit test accuracy against the numbers of Known and Unknown training examples that the model fits, using points from all variants and epochs §4.4 app. H:

\text{Acc} = \beta_0 + \beta_{kn} \frac{N_{kn}}{|D|} + \beta_{unk} \frac{N_{unk}}{|D|} eq. 1

Here N_kn and N_unk count the Known and Unknown training examples the model fits §4.4. The fit is only claimed inside a bounded region of these counts §4.4, footnote.

Hyperparameters

NameValueWhereNotes
Base modelPaLM 2-S, base (not instruction-tuned)§2 §3Only model tested
Fine-tuning typeFull (all parameters)app. ELoRA and other PEFT not tested
Learning rate1e-5, fixedapp. ERobustness plot at 1e-4 fig. 6; text also mentions 1e-6, no results shown
Batch size128app. EUnit (sequences or tokens) not stated
Dropout0.05app. ELayers it applies to not stated
Epochs50app. EDev accuracy checked every epoch
early_stopepoch with best dev accuracyapp. ETypically after 5 to 10 epochs §4
Training set size |D|6142 (single-category study, also the fixed size in the main study)app. E tab. 4Sum of smallest category per relation
Training relations12 (see Results)app. A tab. 4Chosen at random, to avoid cherry-picking
OOD relations7app. A tab. 6
In-distribution test size10,481tab. 5
OOD test size5,848tab. 6
SliCK prompts N_ex10 random 4-shot prompts per pairapp. CExemplars from the same relation, dev split
Shots k4app. CChosen because it gave the right output format
Greedy decodingT = 0, 1 answer per prompt§3 app. C
SamplingT = 0.5, 16 per prompt, top-40app. C
MetricExact match§2 app. CAuthors checked that word-level F1 correlates with it §2, footnote
Significance testPaired t-test, 100 shuffled subsets, p < 0.05 and p < 0.01app. J
Optimizer, warmup, weight decay, sequence length, precision, seeds, hardwarenot stated

Results

Author-reported. Setup: PaLM 2-S base, EntityQuestions, 12 training relations, |D| = 6142, exact match §2 app. E. Each cell is accuracy (%) on the test subset named in the header, for the model fine-tuned on the data named in the row, under early_stop / convergence tab. 2.

Fine-tuning dataFull testHighlyKnown testMaybeKnown testWeaklyKnown testUnknown test
HighlyKnown only40.5 / 40.098.7 / 98.460.1 / 58.89.0 / 8.50.6 / 0.7
MaybeKnown only43.6 / 43.298.4 / 97.569.9 / 68.212.1 / 12.91.0 / 1.3
WeaklyKnown only39.2 / 35.495.0 / 73.559.2 / 55.88.6 / 17.20.4 / 2.2
Unknown only37.5 / 25.895.6 / 55.852.9 / 36.66.5 / 12.20.6 / 3.2
Natural mix (all categories)43.5 / 41.898.0 / 95.567.6 / 61.714.1 / 14.81.8 / 2.5

Source: tab. 2 §5 §6. The paper names "MaybeKnown" the best overall fine-tuning set, with the natural mix nearly tied under early_stop §5 tab. 2.

Other headline results (author-reported):

ExperimentSettingResultSource
Linear model, in-distributionAcc vs Known and Unknown fitsβ0 36.9, β_kn 7.3, β_unk -8.3, R² 0.86tab. 1 eq. 1
Linear model, OODsameβ0 36.2, β_kn 3.2, β_unk -3.0, R² 0.95tab. 1 §4.5
Unknown-share sweepEarly_stop vs convergence, full testHigher Unknown share gives lower accuracy at every stopping point; the gap grows with overfitting§4.1 fig. 3a
Dropping Unknown examplesAblation, early_stopAlmost identical to using all data§4.2 fig. 3b
Dropping Unknown examplesAblation, convergenceDropping them is better; gap grows with Unknown share§4.2 fig. 3b
Largest in-distribution dropUnknown share sweepUp to 14 pointsapp. I
Largest OOD dropUnknown share sweepUp to 6 pointsapp. I
Abstain fine-tuningUnknown examples relabelled "I don't know", 50% UnknownAccuracy on answered questions 61.8 at both stopping points; answered share 58.7% (early_stop) and 55.6% (convergence)tab. 3 §7
Same data, no relabel50% UnknownAccuracy 43.0 (early_stop), 38.8 (convergence) on all questionstab. 3
Label check100 exact-match failures on the base model90% truly wrong; 6% paraphrase; 2% higher granularity; 2% lower granularitytab. 7 app. C
Unknown label qualityUnknown test subsetsAccuracy at most 3.2%tab. 2 §6
P(True) comparisonThreshold on P(True) vs SliCK UnknownAt equal Unknown share, P(True) labels reach a higher accuracy after fine-tuning; preliminary§6 fig. 5 app. K
Inference costPer training example170 inference steps; more than 15M for the full dataset§ Limitations

Category counts for the full EntityQuestions train split (81,700 pairs) are 19,528 HighlyKnown, 20,674 MaybeKnown, 13,825 WeaklyKnown and 27,673 Unknown tab. 4. The in-distribution test split has 2,464, 2,699, 1,856 and 3,462 tab. 5. The OOD test split has 1,437, 1,526, 822 and 2,063 tab. 6.

Paper vs code

No official code. The paper states no code link, and the arXiv page lists no code, project page or repository [arXiv abstract page]. Searches in this round: the GitHub repository search for "SliCK", "SliCK knowledge categories", "fine-tuning hallucinations knowledge", "org:google-research" with the new-knowledge terms, and "gekhman hallucinations", found no repository from the authors. The dataset repository princeton-nlp/EntityQuestions (MIT) belongs to the dataset's authors, and it is not this paper's code. No code citations.

Pitfalls

  • No code. The SliCK labelling step, the exemplar selection, and the fine-tuning schedule have to be rebuilt from this card. Labelling a training set costs 170 inference steps per example § Limitations.
  • Category shares do not match the table. The text gives 24% / 23% / 17% / 36% §3, footnote. The TOTAL row of the train table gives 23.9% / 25.3% / 16.9% / 33.9% tab. 4. The paper does not say which is right.
  • Hallucination is a proxy. The paper measures accuracy drops on questions the base model answered before fine-tuning. It cannot tell hallucination from forgetting, and both are lumped together app. B, footnote.
  • Labels depend on the exemplars. Using fewer than 10 prompts moves the Unknown group and raises accuracy on it §6 fig. 5. The 10-prompt estimate is a practical choice, not a stated optimum app. C.
  • Exact match misses about 10% of correct answers. In the error check, 10% of wrong-looking answers were paraphrases or differed in granularity tab. 7. Labels built with exact match therefore undercount Known facts. The paper accepts this as reasonable app. C.
  • Unstated training details. Optimizer, warmup, weight decay, sequence length, seeds and hardware are not stated. The text says a 1e-6 learning rate was tried and "consistently supported" the findings app. E, but only the 1e-4 plot appears fig. 6.
  • Stopping rule and OOD. The early_stop epoch is picked on the in-distribution development set. The authors note it may not be optimal for OOD [fig. 8 caption, app. I].
  • Single runs and small tests. No seed variance is reported. The significance test shuffles test examples, which is not repeated training app. J.
  • Single-category variants use a fixed size. |D| = 6142 for every variant app. E. These are not comparable to the natural mix at its full size.
  • Scope. One model, closed-book QA, full fine-tuning, and no multi-task mixture § Limitations. The paper does not test DPO, though it says the findings may apply there §1, footnote. Fake-fact labels were rejected without a test app. F.
  • Numbering. Figure and table numbers here follow LaTeX source order. Check them against the PDF before citing.

Lineage

  • Builds on: EntityQuestions for the question data §2; the P(True) score for the alternative Unknown labelling §6 app. K; the superficial alignment hypothesis that LIMA argued, which the paper tests §7.
  • Compared against: P(True) thresholds as the Unknown rule §6 app. K.
  • Related work on uncertainty-expressing fine-tuning is cited as motivation for the "I don't know" relabelling experiment §7.

Source and license

Adapted from “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” by Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig (arXiv:2405.05904), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.