Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

In plain words

A language model learns a lot from its training text, but that knowledge stops at a cutoff date and is uneven across topics. There are two common ways to give a model new or specialised facts from a document collection. You can keep training the model on the documents, so the facts end up inside its weights. Or you can leave the model alone and, at question time, look up relevant passages and paste them next to the question.

The authors put these two approaches side by side with a fair test. They use questions with several answer options, Wikipedia text on the topics, and a set of brand-new news questions that the models could not have seen.

Their finding is that looking things up beats training on the documents. Plain training helps only a little, and on the brand-new news it often does not help at all. Training does work better when every fact is shown in several differently worded versions. The authors describe this as a hypothesis that needs more study.

TL;DR

• Compares unsupervised continued training ("fine-tuning") with retrieval-augmented generation (RAG) for injecting factual knowledge into 7B LLMs §1 §3.
• Knowledge is scored as multiple-choice accuracy from answer log-likelihoods eq. 1 eq. 4; the test sets are four MMLU subjects §4.1 and 910 new US current-events questions §4.3.
• Author-reported: RAG beats fine-tuning in almost every setting; on current events, Mistral-7B goes from 0.481 to 0.875 with RAG and to 0.588 with fine-tuning on paraphrases alone tab. 2.
• Fine-tuning on the original text did not reliably help on new facts; training on several paraphrases of each chunk improved it monotonically in the authors' runs §6 fig. 4.
• No official code was found, so every setting below comes from the paper text.

Problem

LLM knowledge is fixed at pre-training time, reflects whatever the training data covered, and can be missing for niche domains or events after the cutoff. The paper asks, given a text corpus that is relevant to a set of factual questions, which method best improves a model's accuracy on those questions: fine-tuning or RAG §2 §3.1. Knowledge is defined operationally as multiple-choice accuracy that beats random guessing §2, eq. 1–2. The authors note that the test cannot separate knowledge from reasoning, so a model with good reasoning can score above chance through elimination §2 app. C.

Method

The paper formalises knowledge injection as a transformation F that maps a pre-trained model M and a knowledge base B to an improved model M' §3.1, eq. 3. It compares two choices of F.

Fine-tuning (unsupervised, continued pre-training). Start from a saved checkpoint and train it with next-token prediction on the knowledge base, with no labels §3.2. The authors use a lower learning rate than pre-training to limit forgetting §3.2. Before training, each dataset is cut into equal chunks of length 256 by concatenating or splitting the original documents. Each original document is bracketed with a begin and an end token [§5, Training Setup].

Retrieval-augmented generation (RAG). Embed each document of the knowledge base once with a pre-trained embedding model and store the vectors §3.3. For a query, embed the question, take the K nearest documents by dot product, and concatenate them in front of the question as a plain string §3.3. The generator then answers the augmented input. No training is involved §3.3.

Combined. The fine-tuned model is used as the generator in the RAG pipeline §5, MMLU Results. The paper does not say whether the embedding model changes; it is described only as bge-large-en §5.

Evaluation. For each question, the model is given each answer option appended to the question. The option with the highest log-probability of the concatenated text is the model's answer §5, Evaluation method, eq. 4. Accuracy over the questions is the knowledge score eq. 1. Results are reported as 0-shot and 5-shot §5.

Gain metric. Relative accuracy gain of an injected model over its base model eq. 5, averaged over experiments per method fig. 2.

Knowledge bases and tasks.

• MMLU subsets: anatomy, astronomy, college biology, college chemistry and prehistory, chosen for factual content and short context-free questions §4.1. Their Wikipedia articles were fetched through the official API by identifying a central page per topic, then cleaned with wikiextractor §4.2.
• Current events: US events from August to November 2023 found in Wikipedia category indexes §4.1. GPT-4 wrote four specific multiple-choice questions per chunk, kept the two it judged most specific, and a manual check followed. The result is 910 questions, each tied to one source chunk §4.3.
• Paraphrases: GPT-4 rewrote chunks with different seeds while keeping the facts §4.4, app. B. For each MMLU task, 240 random chunks got two paraphrases each, used as validation data for hyperparameter tuning §4.4. For current events, each fine-tuning chunk got ten paraphrases §4.4 §6.

# Knowledge injection, evaluation of one question  [§3.1] [§5] [eq. 4]
# Input:  model, one question with options, optional retriever and K
# Output: 1 if the model picks the correct option, else 0

def score_question(model, question, retriever, k):
    # 1. Optional RAG: fetch the k closest documents  [§3.3]
    if retriever is not None:
        chunks = retriever.top_k(question.text, k)  # dot-product ranking
        prompt = concat(chunks, question.text)  # separator not stated [§3.3]
    else:
        prompt = question.text
    # 2. Score each option by log P(prompt || option)  [§5] [eq. 4]
    scores = []
    for option in question.options:
        scores.append(log_prob(model, prompt + option))
    # 3. Pick the highest-scoring option  [§5] [eq. 4]
    choice = argmax(scores)
    return 1 if choice == question.answer else 0

# Unsupervised fine-tuning on a knowledge base  [§3.2] [§5]
# Input:  base checkpoint, list of raw documents
# Output: fine-tuned checkpoint

def finetune(model, documents):
    # 1. Split and join documents into chunks of length 256  [§5]
    #    (unit of the length not stated; BOS/EOS mark document edges)
    chunks = pack(documents, length=256, mark_edges=True)
    # 2. Next-token training, no labels  [§3.2]
    #    lr searched in [1e-6, 5e-5]; batch 64; at most 5 epochs  [§5]
    for epoch in range(5):
        for batch in make_batches(chunks, size=64):
            loss = next_token_loss(model, batch)
            model = apply_update(model, loss)
    return model

# RAG index construction and query  [§3.3] [§5]
# Input:  documents, embedding model, vector store
# Output: index and augmented prompt

def build_index(documents, embed):
    # 1. Embed each document once and store the vectors  [§3.3]
    vectors = [embed(doc) for doc in documents]
    return store(vectors, documents)  # FAISS in the paper [§5]

def augment(question, index, embed, k):
    # 2. Retrieve the k nearest documents by dot product  [§3.3]
    neighbours = index.top_k(embed(question), k)
    # 3. Put the retrieved text before the question  [§3.3]
    return concat(neighbours, question)

Hyperparameters

Results

Author-reported. Setup: three 7B models (Llama2-7B, Mistral-7B, Orca2-7B), four configurations (base, base + RAG, fine-tuned, fine-tuned + RAG) plus 0-shot and 5-shot on MMLU; the current-events task uses the same four plus two paraphrase-fine-tuned variants §5 tab. 1 tab. 2. Metric is log-likelihood accuracy eq. 4.

MMLU, log-likelihood accuracy tab. 1. Columns: Base, Base + RAG, Fine-tuned, Fine-tuned + RAG. The authors say RAG was significantly better than the base model in all cases. RAG on the base model beat fine-tuning alone in 28 of the 30 rows; the two exceptions are Llama2-7B 0-shot on astronomy and on college chemistry (counted from the table). Fine-tuned + RAG sometimes beat RAG alone, which the authors call unstable §5.

Source: tab. 1. The paper's bold marks the best value per row; it is not reproduced here.

Current events tab. 2. Columns: Base, Base + RAG, FT-reg (fine-tuned on the original chunks), FT-par (fine-tuned on paraphrases), FT-reg + RAG, FT-par + RAG.

The authors say RAG does especially well here because each question was made from one chunk that is in the knowledge base §5 §4.3. Fine-tuning on the original text was not competitive, and for Llama2 it fell well below the base model. Paraphrase fine-tuning improved over the base for all three models §5. Base scores above 0.25 are partly explained by reasoning and prior knowledge §5 app. C.

RAG ablation over K tab. 3. Accuracy for K = 1 to 5 retrieved documents (full grid in the paper). The authors found no single best K per model, task or shot setting, and the gap between best and worst K can be large. Anatomy favoured K = 2 consistently app. A.

Bold values in the paper mark the best K per row and are not reproduced.

Figures, described in words only.

• fig. 1: a schematic of the knowledge-injection framework (caption only; the text does not describe its content).
• fig. 2: relative accuracy gain eq. 5 per injection method, averaged column-wise over all MMLU experiments in tab. 1.
• fig. 3: training loss over time for Mistral-7B. The authors say the loss drops noticeably after each epoch, which they link to memorisation and overfitting §6.
• fig. 4: current-events accuracy against the number of paraphrases. The authors say accuracy rises monotonically with paraphrase count for every model tested §6.

Paper vs code

No official code. The paper and its arXiv page (abstract page, PDF and TeX source) link no repository. A GitHub repository search for the title words and for the method's key terms returned nothing relevant, and a search for the author names returned only unrelated accounts. No repository from the authors or from Microsoft was found.

Pitfalls

• Main-table K is unknown. The text says K ranges over 0 to 5, yet the ablation table only shows K = 1 to 5 §5 tab. 3. The K used for "Base model + RAG" in tab. 1 and tab. 2 is not stated, so the RAG numbers cannot be reproduced exactly from the paper.
• Retrieval chunking is unstated. Training chunks are equal 256-unit pieces §5. The paper does not say whether the RAG knowledge base uses the same chunking, or what text is used as the separator between retrieved documents and the question §3.3.
• Learning rate is only a range. The search range is 1e-6 to 5e-5 §5. The value for each model and task is not given. Optimizer, warmup, weight decay and schedule are not stated.
• Selection rule is unstated. The paper says a maximum of 5 epochs was used and that paraphrase-based validation sets were used for tuning §4.4 §5. It does not say which checkpoint was kept or which metric drove the choice.
• Current-events test is built from the RAG corpus. Each question comes from one chunk that is in the retrieval knowledge base §4.3 §5. The authors note this makes RAG look especially strong on that task. Treat the current-events gap as an upper bound for RAG.
• Knowledge can be guessed. Base models score above chance on current events through reasoning, elimination or prior knowledge §5 app. C. The authors also say the test cannot cleanly separate knowledge from reasoning §2.
• Notation clash. The symbol L is used both for the knowledge score eq. 1 and for the number of options §2. Read eq. 2 with that in mind.
• Scoring details. Option scores are full log-probabilities of the concatenated text eq. 4. The paper does not say whether they are normalised by length, which can change the choice when options differ in length.
• Fine-tuning can hurt. On current events, FT-reg reduced Llama2-7B from 0.353 to 0.219 tab. 2. The authors attribute general fine-tuning weakness partly to catastrophic forgetting, which they do not measure §5.
• Repetition is a hypothesis. The claim that new knowledge must be repeated in varied forms is labelled a hypothesis §6. The evidence is one task and three models, with paraphrase counts whose exact values are only in a plot fig. 4.
• Costs hidden by the abstract. Paraphrases and questions were generated with GPT-4 §4.3 §4.4, and RAG adds an embedding index plus extra tokens per query at inference time §3.3. The authors report RAG's K is unstable and must be tuned app. A.
• Narrow coverage. Only three 7B models, one knowledge source (Wikipedia), and a limited set of MMLU subjects were tested. The authors say results may not carry to other models or sources, and that GPT-4 is near-perfect on some MMLU subjects, so gains there are limited §8.
• Shots and prompts. The 5-shot prompt format and the RAG prompt template are not given §5. Reproducing the 5-shot numbers needs the lm-evaluation-harness setup the authors cite §5.

Lineage

• Builds on: retrieval-augmented generation in its original form, with per-task training §3.3; embedding-based RAG with no extra training, from a later paper §3.3. Both are cited, not compiled here.
• Compared against: the base model, unsupervised continued training, and their combination tab. 1 tab. 2.
• Related: instruction tuning and RL-based fine-tuning are discussed as methods that do not add knowledge §3.2. The authors did not test them §7.