Paper · 2024-05-09 · cs.CL
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
SliCK labels each fine-tuning question by sampling PaLM 2-S: HighlyKnown, MaybeKnown, WeaklyKnown or Unknown. In closed-book QA, more Unknown examples fitted during fine-tuning track lower test accuracy (linear fit, R² 0.86 in-distribution); early stopping or dropping them limits the loss. Author-reported; one base model.
Knowledge injectionInstruction tuningHallucination mitigationfine-tuninghallucinationclosed-book-qaknowledge-categorizationsupervised-fine-tuning
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A chat model is first trained on huge amounts of text, then fine-tuned on question-and-answer examples written to make it helpful. Some of those examples may contain facts the model never learned during its first training. The worry is that teaching it these new facts makes it more likely to state wrong answers confidently, even about things it already knew.
The authors test this with a controlled setup. They sort every training question by how well the base model already answers it, by sampling several answers to the same question. They then build training sets with different shares of questions the model did not know, and measure how well each fine-tuned model answers test questions on facts it knew before training.
They find that the model learns the unfamiliar questions much more slowly than the familiar ones. Once it does learn them, accuracy on what it already knew falls in proportion to how many of them it has absorbed. Stopping training early, or dropping the unfamiliar questions, reduces the damage. The results come from one model and one question-answering dataset.
TL;DR
- Fine-tuning on questions whose answers the base model did not know slows learning, and once learned it lowers accuracy on facts the model already knew §4.1, §4.3.
- SliCK sorts each training pair into HighlyKnown, MaybeKnown, WeaklyKnown or Unknown by sampling answers from the base model §3.
- Test accuracy follows a linear model in the counts of fitted Known and Unknown examples, R² 0.86 in-distribution and 0.95 out-of-distribution eq. 1 tab. 1.
- Only one base model (PaLM 2-S), closed-book QA and full fine-tuning were tested §2 app. E § Limitations.
Problem
Supervised fine-tuning on instruction data may add facts that the pre-trained model does not hold §1. A common conjecture holds that this trains the model to produce answers not grounded in its prior knowledge, which encourages hallucination §1. Before this work, there was no controlled way to tell, for a single training example, whether the model already knows its answer §3. The paper defines hallucination relative to pre-existing knowledge: questions the model answered correctly before fine-tuning, which it then answers wrongly app. B.
Method
Setup. The base model M is PaLM 2-S, used as a base model and not as an instruction-tuned one §2 §3. The data are closed-book question-answer pairs (q, a) from EntityQuestions, which turns Wikidata triplets into questions §2. The authors use 12 relations for training and in-distribution testing and hold out 7 relations as an out-of-distribution (OOD) test set §2 app. A. Exact match is the metric §2 app. C.
Data cleaning. Training pairs with more than one correct answer are removed, about 4.2% of train and 3.9% of test app. A. Pairs whose subject or object appears in the test set are removed from train, about 2.1% of train app. A. Five candidate OOD relations were dropped because they overlapped with training relations app. A.
SliCK categories. For each pair (q, a), M is prompted with 10 random 4-shot prompts. Each prompt uses exemplars from the same relation, drawn from the development split §3 app. C. Each prompt gives one greedy answer (temperature 0), and 16 sampled answers at temperature 0.5 with top-40 sampling app. C. The greedy score is the share of the 10 greedy answers that are correct. The sampled score is the share of the 160 sampled answers that are correct app. C. The four categories follow from these two scores §3 fig. 2.
# SliCK category for one (question, answer) pair [§3] [app. C]
# Input: base model M, question q, gold answer a, exemplars of q's relation
# Output: "HighlyKnown", "MaybeKnown", "WeaklyKnown" or "Unknown"
def slick_category(model, question, answer, exemplars):
# Build 10 random 4-shot prompts from the same relation [app. C]
prompts = [sample_4_shot(exemplars) for _ in range(10)]
# One greedy answer per prompt, temperature 0 [§3]
greedy_hits = [
exact_match(model.greedy(p, question), answer) for p in prompts
]
# 16 sampled answers per prompt, temperature 0.5, top-40 [app. C]
sampled_hits = [
exact_match(model.sample(p, question), answer)
for p in prompts
for _ in range(16)
]
# Greedy score: fraction of the 10 greedy answers that are correct [app. C]
greedy_score = sum(greedy_hits) / len(greedy_hits)
# Sampled score: share of the 160 samples that are correct [app. C]
sampled_score = sum(sampled_hits) / len(sampled_hits)
# Greedy answer is always correct: HighlyKnown [§3]
if greedy_score == 1:
return "HighlyKnown"
# Greedy answer is sometimes correct: MaybeKnown [§3]
if greedy_score > 0:
return "MaybeKnown"
# Never greedy, but sometimes sampled: WeaklyKnown [§3]
if sampled_score > 0:
return "WeaklyKnown"
# Never correct under any decoding: Unknown [§3]
return "Unknown"The four categories, with the paper's wording: HighlyKnown (greedy score 1), MaybeKnown (greedy score between 0 and 1), WeaklyKnown (greedy score 0, sampled score above 0), and Unknown (score 0 under all decoding). Together the first three form "Known" §3 fig. 2. The paper gives 24% HighlyKnown, 23% MaybeKnown, 17% WeaklyKnown and 36% Unknown for its data §3, footnote. See Pitfalls for a mismatch with the table totals.
Training-set variants. The main study fixes the training set size and varies the share of Unknown examples X%, with the remaining (100-X)% drawn from Known. The share is sampled inside each relation, so the relation mix stays the same §4 app. E. For the single-category study, |D| is set to the sum over relations of the smallest category size, giving 6142 pairs app. E tab. 4.
Fine-tuning. Full fine-tuning updates all parameters app. E. The model is evaluated on the development set after every epoch app. E. Two stopping points are compared: "early_stop" is the epoch with the best development accuracy, which the paper says falls after 5 to 10 epochs §4 app. E. "Convergence" is epoch 50, where the model fits 100% of the training set §4 app. E.
# Fine-tune M on D, keep both stopping points [app. E] [§4]
# Input: base model M, training set D, development set V
# Output: early_stop weights, convergence weights
def finetune(model, train_set, dev_set):
best_dev = -1.0
early_stop_weights = None
# Fixed learning rate 1e-5, batch 128, dropout 0.05 [app. E]
for epoch in range(1, 51):
train_one_epoch(model, train_set, lr=1e-5, batch_size=128)
dev_accuracy = evaluate(model, dev_set)
# Keep the epoch with the highest development accuracy [app. E]
if dev_accuracy > best_dev:
best_dev = dev_accuracy
early_stop_weights = copy(model)
# Weights after 50 epochs are the convergence model [app. E]
return early_stop_weights, modelLinear model. The authors fit test accuracy against the numbers of Known and Unknown training examples that the model fits, using points from all variants and epochs §4.4 app. H:
\text{Acc} = \beta_0 + \beta_{kn} \frac{N_{kn}}{|D|} + \beta_{unk} \frac{N_{unk}}{|D|} eq. 1
Here N_kn and N_unk count the Known and Unknown training examples the model fits §4.4. The fit is only claimed inside a bounded region of these counts §4.4, footnote.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Base model | PaLM 2-S, base (not instruction-tuned) | §2 §3 | Only model tested |
| Fine-tuning type | Full (all parameters) | app. E | LoRA and other PEFT not tested |
| Learning rate | 1e-5, fixed | app. E | Robustness plot at 1e-4 fig. 6; text also mentions 1e-6, no results shown |
| Batch size | 128 | app. E | Unit (sequences or tokens) not stated |
| Dropout | 0.05 | app. E | Layers it applies to not stated |
| Epochs | 50 | app. E | Dev accuracy checked every epoch |
| early_stop | epoch with best dev accuracy | app. E | Typically after 5 to 10 epochs §4 |
| Training set size |D| | 6142 (single-category study, also the fixed size in the main study) | app. E tab. 4 | Sum of smallest category per relation |
| Training relations | 12 (see Results) | app. A tab. 4 | Chosen at random, to avoid cherry-picking |
| OOD relations | 7 | app. A tab. 6 | |
| In-distribution test size | 10,481 | tab. 5 | |
| OOD test size | 5,848 | tab. 6 | |
| SliCK prompts N_ex | 10 random 4-shot prompts per pair | app. C | Exemplars from the same relation, dev split |
| Shots k | 4 | app. C | Chosen because it gave the right output format |
| Greedy decoding | T = 0, 1 answer per prompt | §3 app. C | |
| Sampling | T = 0.5, 16 per prompt, top-40 | app. C | |
| Metric | Exact match | §2 app. C | Authors checked that word-level F1 correlates with it §2, footnote |
| Significance test | Paired t-test, 100 shuffled subsets, p < 0.05 and p < 0.01 | app. J | |
| Optimizer, warmup, weight decay, sequence length, precision, seeds, hardware | not stated |
Results
Author-reported. Setup: PaLM 2-S base, EntityQuestions, 12 training relations, |D| = 6142, exact match §2 app. E. Each cell is accuracy (%) on the test subset named in the header, for the model fine-tuned on the data named in the row, under early_stop / convergence tab. 2.
| Fine-tuning data | Full test | HighlyKnown test | MaybeKnown test | WeaklyKnown test | Unknown test |
|---|---|---|---|---|---|
| HighlyKnown only | 40.5 / 40.0 | 98.7 / 98.4 | 60.1 / 58.8 | 9.0 / 8.5 | 0.6 / 0.7 |
| MaybeKnown only | 43.6 / 43.2 | 98.4 / 97.5 | 69.9 / 68.2 | 12.1 / 12.9 | 1.0 / 1.3 |
| WeaklyKnown only | 39.2 / 35.4 | 95.0 / 73.5 | 59.2 / 55.8 | 8.6 / 17.2 | 0.4 / 2.2 |
| Unknown only | 37.5 / 25.8 | 95.6 / 55.8 | 52.9 / 36.6 | 6.5 / 12.2 | 0.6 / 3.2 |
| Natural mix (all categories) | 43.5 / 41.8 | 98.0 / 95.5 | 67.6 / 61.7 | 14.1 / 14.8 | 1.8 / 2.5 |
Source: tab. 2 §5 §6. The paper names "MaybeKnown" the best overall fine-tuning set, with the natural mix nearly tied under early_stop §5 tab. 2.
Other headline results (author-reported):
| Experiment | Setting | Result | Source |
|---|---|---|---|
| Linear model, in-distribution | Acc vs Known and Unknown fits | β0 36.9, β_kn 7.3, β_unk -8.3, R² 0.86 | tab. 1 eq. 1 |
| Linear model, OOD | same | β0 36.2, β_kn 3.2, β_unk -3.0, R² 0.95 | tab. 1 §4.5 |
| Unknown-share sweep | Early_stop vs convergence, full test | Higher Unknown share gives lower accuracy at every stopping point; the gap grows with overfitting | §4.1 fig. 3a |
| Dropping Unknown examples | Ablation, early_stop | Almost identical to using all data | §4.2 fig. 3b |
| Dropping Unknown examples | Ablation, convergence | Dropping them is better; gap grows with Unknown share | §4.2 fig. 3b |
| Largest in-distribution drop | Unknown share sweep | Up to 14 points | app. I |
| Largest OOD drop | Unknown share sweep | Up to 6 points | app. I |
| Abstain fine-tuning | Unknown examples relabelled "I don't know", 50% Unknown | Accuracy on answered questions 61.8 at both stopping points; answered share 58.7% (early_stop) and 55.6% (convergence) | tab. 3 §7 |
| Same data, no relabel | 50% Unknown | Accuracy 43.0 (early_stop), 38.8 (convergence) on all questions | tab. 3 |
| Label check | 100 exact-match failures on the base model | 90% truly wrong; 6% paraphrase; 2% higher granularity; 2% lower granularity | tab. 7 app. C |
| Unknown label quality | Unknown test subsets | Accuracy at most 3.2% | tab. 2 §6 |
| P(True) comparison | Threshold on P(True) vs SliCK Unknown | At equal Unknown share, P(True) labels reach a higher accuracy after fine-tuning; preliminary | §6 fig. 5 app. K |
| Inference cost | Per training example | 170 inference steps; more than 15M for the full dataset | § Limitations |
Category counts for the full EntityQuestions train split (81,700 pairs) are 19,528 HighlyKnown, 20,674 MaybeKnown, 13,825 WeaklyKnown and 27,673 Unknown tab. 4. The in-distribution test split has 2,464, 2,699, 1,856 and 3,462 tab. 5. The OOD test split has 1,437, 1,526, 822 and 2,063 tab. 6.
Paper vs code
No official code. The paper states no code link, and the arXiv page lists no code, project page or repository [arXiv abstract page]. Searches in this round: the GitHub repository search for "SliCK", "SliCK knowledge categories", "fine-tuning hallucinations knowledge", "org:google-research" with the new-knowledge terms, and "gekhman hallucinations", found no repository from the authors. The dataset repository princeton-nlp/EntityQuestions (MIT) belongs to the dataset's authors, and it is not this paper's code. No code citations.
Pitfalls
- No code. The SliCK labelling step, the exemplar selection, and the fine-tuning schedule have to be rebuilt from this card. Labelling a training set costs 170 inference steps per example § Limitations.
- Category shares do not match the table. The text gives 24% / 23% / 17% / 36% §3, footnote. The TOTAL row of the train table gives 23.9% / 25.3% / 16.9% / 33.9% tab. 4. The paper does not say which is right.
- Hallucination is a proxy. The paper measures accuracy drops on questions the base model answered before fine-tuning. It cannot tell hallucination from forgetting, and both are lumped together app. B, footnote.
- Labels depend on the exemplars. Using fewer than 10 prompts moves the Unknown group and raises accuracy on it §6 fig. 5. The 10-prompt estimate is a practical choice, not a stated optimum app. C.
- Exact match misses about 10% of correct answers. In the error check, 10% of wrong-looking answers were paraphrases or differed in granularity tab. 7. Labels built with exact match therefore undercount Known facts. The paper accepts this as reasonable app. C.
- Unstated training details. Optimizer, warmup, weight decay, sequence length, seeds and hardware are not stated. The text says a 1e-6 learning rate was tried and "consistently supported" the findings app. E, but only the 1e-4 plot appears fig. 6.
- Stopping rule and OOD. The early_stop epoch is picked on the in-distribution development set. The authors note it may not be optimal for OOD [fig. 8 caption, app. I].
- Single runs and small tests. No seed variance is reported. The significance test shuffles test examples, which is not repeated training app. J.
- Single-category variants use a fixed size. |D| = 6142 for every variant app. E. These are not comparable to the natural mix at its full size.
- Scope. One model, closed-book QA, full fine-tuning, and no multi-task mixture § Limitations. The paper does not test DPO, though it says the findings may apply there §1, footnote. Fake-fact labels were rejected without a test app. F.
- Numbering. Figure and table numbers here follow LaTeX source order. Check them against the PDF before citing.
Lineage
- Builds on: EntityQuestions for the question data §2; the P(True) score for the alternative Unknown labelling §6 app. K; the superficial alignment hypothesis that LIMA argued, which the paper tests §7.
- Compared against: P(True) thresholds as the Unknown rule §6 app. K.
- Related work on uncertainty-expressing fine-tuning is cited as motivation for the "I don't know" relabelling experiment §7.
Source and license
Adapted from “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” by Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig (arXiv:2405.05904), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.