Large Language Models Struggle to Learn Long-Tail Knowledge

In plain words

Language models learn facts from the text they are trained on, but most of that text is about popular things. A fact that appears only a handful of times on the web is easy to miss, much like a student who has seen an answer once and forgets it. This paper asks a simple question: when a model gets a trivia question right, how many training documents mention the subject and the answer together?

The authors find a strong link. Questions whose answers show up in many training documents are answered correctly far more often than questions with few supporting documents. To check that this is not just a coincidence of "easy" questions, they remove the supporting documents from a training set and retrain a model, and the rare questions get noticeably worse.

They then ask what would fix it. Bigger models help, but only slowly, so the trend suggests that memorising rare facts by scale alone would take impractically large models. Giving the model a search tool that fetches relevant text helps much more. If you rely on a model to remember obscure facts, the practical lesson is to supply the facts at inference time.

TL;DR

• Closed-book QA accuracy of GPT-Neo, BLOOM and GPT-3 rises strongly with the number of pre-training documents that contain both the question's salient entity and the answer entity §3.1 §2.
• Removing those documents and retraining a 4.8B C4 model drops accuracy on the affected questions, which the authors read as causal §3.2.
• Accuracy grows log-linearly with model size on rare questions; the fit projects more than 10^18 parameters to match a supervised baseline or humans §4.2.
• Oracle or BM25 retrieval gives much better accuracy on rare questions than closed-book models §4.3.

Problem

Large LMs answer knowledge-intensive questions by recalling facts from pre-training data, but the frequency of facts on the web is heavy-tailed. The paper asks whether the knowledge a model stores depends on how often the supporting information appears in its pre-training corpus §1 §2. It studies factoid QA (TriviaQA, Natural Questions) in a closed-book setting, so the model gets no background text at test time §2.

Method

The paper measures, for each QA pair, a count of "relevant" pre-training documents, then correlates that count with QA accuracy §2 §3.1.

1. Entity-link the pre-training corpora. DBpedia Spotlight links entities in The Pile (825 GB), ROOTS English (490 GB), C4 (305 GB), OpenWebText (39 GB) and a December 2018 Wikipedia dump. For each entity the pipeline records which documents mention it, which allows fast counting of single entities and co-occurrences §2.1. The run took about 3 weeks on 2.1 TB with a 128-core machine §2.1.
2. Entity-link the QA data. Natural Questions and TriviaQA are linked using both training and validation splits, minus a small set of examples used as few-shot prompts §2.2. The question and all valid answers are concatenated before linking, which the authors found more accurate §2.2. The most common answer entity is the salient answer entity. Among the question's entities, the one that co-occurs most with it is the salient question entity §2.2. Examples without an entity in question or answer, or with zero relevant documents, are discarded §2.2.
3. Count relevant documents. A document is relevant to a QA pair if it contains the salient question entity and the salient answer entity §2.2. The count is the number of such documents in the corpus. The authors estimate that about 60% of the documents they sampled are relevant at all, and about a third of them contain enough information to answer the question §2.3.
4. Measure accuracy. Models are prompted with in-context QA examples and the test question, with no background text §3. The main setting is 4-shot, with the templated prompt in §3. Answers come from greedy decoding until a newline. Scoring uses exact match against the ground-truth answer set §3. Models are GPT-Neo/GPT-NeoX/GPT-J (125M to 20B, trained on The Pile), BLOOM (560M to 176B, trained on ROOTS) and GPT-3 (about 350M to 175B). GPT-3 counts are estimated by scaling OpenWebText counts to GPT-3's data size, not measured §3.
5. Bin and plot. Relevant-document counts are grouped into log-spaced bins (for example 1 to 10, 10 to 100). Bins with fewer than 500 QA examples are dropped. A histogram per plot shows how many examples fall in each bin §3.1.
6. Check the counting rule. Counting documents with only the question entity, or only the answer entity, also correlates with accuracy overall. Within QA pairs where the two entities co-occur in fewer than 5 documents, those simpler counts no longer correlate, so the co-occurrence count is the one the paper keeps §3.1 app. C.
7. Counterfactual retraining. A baseline 4.8B LM is trained on C4 following [Wang et al. 2022] (cited as the setup source in §3.2). For each log bin, 100 TriviaQA questions are sampled and all their relevant C4 documents are removed, about 30% of C4 in total. A counterfactual model is trained on the reduced corpus for one epoch. It takes 30% fewer steps, so only questions whose documents were removed are compared §3.2.
8. Scale and retrieval. The authors fit a log-linear trend of BLOOM accuracy on rare Natural Questions (fewer than 100 relevant documents) against log model size, with R² = 0.98 §4.2. They also test an oracle setting (a gold 300-word Wikipedia passage around the answer, 2-shot GPT-Neo) and a BM25 setting (top-3 Wikipedia paragraphs added to the prompt for both in-context and test examples) §4.3.

Equations: the paper has no numbered equations. The counting rule is in prose §2.2; the prompt template is an unnumbered display §3.

# Count relevant documents for one QA pair  [§2.2]
# [code: relevant_docs/count_relevant_docs.py#L35-L54]
# Input:  question_entities (set of URIs), answer_entities (list of URIs,
#         repeats allowed), doc_index (URI -> sorted array of doc ids)
# Output: integer count of relevant documents

def count_relevant_docs(question_entities, answer_entities, doc_index):
    # Drop answer entities that are also question entities  [code: count_relevant_docs.py#L74]
    answers = [a for a in answer_entities if a not in question_entities]
    counts = count_each(answers)
    # Salient answers: the most frequent ones, ties all kept  [§2.2]
    # [code: count_relevant_docs.py#L80-L81]
    top = max(counts.values(), default=0)
    salient_answers = [a for a in counts if counts[a] == top]
    best = 0
    for answer in salient_answers:
        for question in question_entities:
            # Documents that contain both entities  [§2.2]
            both = intersect(doc_index[answer], doc_index[question])
            best = max(best, len(both))
    # Zero or no entities: the paper discards the example  [§2.2]
    return best

# Index one corpus: entity -> documents mentioning it  [§2.1]
# [code: entity_linking/link_pretraining_data.py#L72-L86]
# Input:  documents (list of text)
# Output: dict URI -> sorted array of document ids

def build_doc_index(documents):
    # Link each document with DBpedia Spotlight  [§2.1]
    # (code defaults: confidence 0.4, support 20)
    # [code: entity_linking/link_pretraining_data.py#L21-L22]
    index = defaultdict(set)
    for doc_id, text in enumerate(documents):
        # A document counts once per entity, however often it is mentioned
        for uri in set(link_entities(text)):
            index[uri].add(doc_id)
    return {uri: sorted(ids) for uri, ids in index.items()}

Hyperparameters

Results

Author-reported. Setup: closed-book few-shot QA on TriviaQA and Natural Questions; models GPT-Neo, BLOOM and GPT-3; relevant-document counts from entity co-occurrence in The Pile, ROOTS, C4, OpenWebText and Wikipedia §2 §3. The paper reports no separate numeric baseline for most rows; the "baseline" column names the comparison point the text uses.

Human accuracy on Natural Questions runs the other way: it is highest for questions with few relevant documents, which the authors attribute partly to simpler factoid questions and shorter Wikipedia passages §3.1.

Paper vs code

The official repository is nkandpa2/long_tail_knowledge at commit 021b189, linked from the paper's introduction §1. It contains only the entity-linking and document-counting scripts. The LM training, prompting, counterfactual retraining, scaling fits and BM25 experiments are not in the repository, so the results above cannot be checked against code. The repository has no LICENSE file, so code_license is recorded as not stated and no code is quoted.

The released code does not run end to end as shipped:

• count_relevant_docs.py calls np.load at count_relevant_docs.py, but the file never imports numpy (its imports are at count_relevant_docs.py).
• utils.sort_entity_map assigns sorted arrays back into the mapping utils.py. That mapping is an NpzFile from np.load, which is read-only, so this step would most likely raise. Not run here.
• link_qa.py iterates an undefined name nq link_qa.py; the loaded dataset is qa.
• link_pretraining_data_chunked.py uses dataset.nrows link_pretraining_data_chunked.py, an undefined chunk_size link_pretraining_data_chunked.py, and a line that is not valid Python link_pretraining_data_chunked.py.

Other differences between paper and code:

• Paper: examples with zero relevant documents are discarded §2.2. Code: the count is 0 when no pair co-occurs count_relevant_docs.py count_relevant_docs.py, and every row is written to the output with no filter count_relevant_docs.py. Filtering must happen downstream, and the paper does not say how.
• Paper: the salient answer entity is the most common one §2.2. Code: ties are all kept, and the maximum over all tied answers and question entities is used count_relevant_docs.py count_relevant_docs.py. The paper does not describe tie handling.
• Code only: answer entities that also appear as question entities are dropped count_relevant_docs.py. The paper does not mention this.
• Code only: a --exclude_countries option removes country entities from the question set, default off count_relevant_docs.py count_relevant_docs.py. The paper does not say whether it was used for its results.
• Paper: few-shot examples are excluded from the QA data §2.2. Code: all selected splits are concatenated with no exclusion link_qa.py. The exclusion is not in the released code.
• Code: entity-linking defaults are confidence 0.4 and support 20 link_qa.py link_pretraining_data.py. The paper does not state them.
• Code: a failed Spotlight call returns a sentinel URI such as __SPOTLIGHT_EXCEPTION__ in the pre-training path link_pretraining_data.py, which is then stored as an entity link_pretraining_data.py. In the QA path, a persistent connection error returns an empty list link_qa.py. The paper does not describe either fallback.

No discrepancies were found for the core counting logic: the document-set intersection utils.py matches the co-occurrence count §2.2.

Pitfalls

• Document ids are local to each chunk. The pre-training script enumerates the sliced dataset from 0 link_pretraining_data.py and names outputs by start and end link_pretraining_data.py. Chunks cannot be merged by concatenating their maps; the offset by start must be applied first. Merging without it silently corrupts the counts.
• Failed calls silently remove entities. A failed pre-training call leaves the document with only a sentinel key, so its real entities are missing and co-occurrence counts drop; a failed QA call yields no entities, so the example falls to zero. Neither is flagged in the output, and the paper does not measure how often this happens.
• Counts are documents, not facts. Only about 60% of sampled co-occurrence documents are relevant at all, and only about a third contain the answer §2.3. The count is a proxy for how often the knowledge appears; the paper checks it with the human study §2.3 and the counterfactual retraining §3.2.
• Scale projection is an extrapolation. The 10^18-parameter figure comes from a log-linear fit over a handful of model sizes §4.2. The introduction gives "one quadrillion parameters" §1, which is 10^15, while §4.2 gives over 10^18. The two numbers are inconsistent within the paper; use the §4.2 figure with the fit's caveats.
• Footnote caveat on scaling. Supervised and human accuracies come from the validation set, while the trend is fit on train plus validation §4.2, footnote.
• GPT-3 and other closed corpora are estimates. Counts for GPT-3 come from OpenWebText scaled to GPT-3's data size §3; the authors flag this uncertainty themselves §3 app. A.
• Counterfactual setup is single-run. The number of seeds and the variance are not stated §3.2. Accuracy is compared only on removed questions, because the counterfactual model trained for 30% fewer steps §3.2.
• Annotator count is not stated for the 300-pair precision study §2.3.
• Wikipedia dump is named by date only. The December 2018 dump is not identified by URL or hash in the paper or the repository; the code takes a local HF dataset path link_pretraining_data.py.
• Compute cost is high. Linking 2.1 TB took about 3 weeks on 128 cores §2.1. The repository's DBpedia Spotlight setup needs 16 Docker instances README.md.
• Figure and table numbers are inferred. No PDF was available; figure numbers follow LaTeX source order (Figure 1 is the BLOOM TriviaQA plot). Check them against the arXiv PDF before publishing.

Lineage

• Builds on: Trex (Elsahar et al. 2018) for the claim that co-occurrence of subject and object in text predicts the triple §2; closed-book QA evaluation of Roberts et al. 2020 §2; the frequency-accuracy correspondence of Petroni et al. 2019 and others §5. None of these has a card in this wiki yet.
• Compared against: Elazar et al. 2022 (causal inference on pre-training heuristics) §5; Akyurek et al. 2022 (influence-based document attribution) §5; Mallen et al. 2022 (concurrent work using popularity proxies instead of entity linking) §5.
• Superseded by: not stated in the paper.