---
kind: concept
slug: benchmark-contamination
title: Benchmark contamination
summary: >-
  How to tell whether a model has already seen a benchmark or a text it is tested on, so a score measures skill rather than recall. Four ways to measure it: data overlap for open models, an extractability ratio for closed models, a name-cloze memory probe, and web-prevalence counts as a prior. Each suits different access. Results are author-reported and not validated.
tags: [evaluation, memorization, data-contamination]
papers: ["2305.00118", "2310.18018"]
related: [benchmark-validity, llm-evaluation, dataset-curation, membership-inference, training-data-extraction, language-model-pretraining]
status: draft
updated_at: 2026-11-15
---

## Overview

A score is only evidence of skill if the test items were not in the training
data. If they were, the model may be recalling them. The shared problem is
that closed models do not publish their training data, so the check has to be
indirect [arxiv:2310.18018 §1] [arxiv:2305.00118 §1].

The position paper defines three kinds of contamination: the annotation
guidelines were seen, the raw text before annotation was seen, or the labels
of the benchmark were seen. The effect of each depends on which split was
exposed [arxiv:2310.18018 §3]. It places the risk at three stages: pre-training,
supervised and instruction tuning, and user data gathered after deployment
[arxiv:2310.18018 §4.1] [§4.2] [§4.3].

The second paper runs a memory probe on two OpenAI chat models. It shows that
books differ widely in how well the models know them, and that this knowledge
follows how often the text appears on the web. Its authors connect this to
evaluations of fiction, where memorised text can distort results
[arxiv:2305.00118 §1] [§6.2] [§8].

## Variants

### Benchmark data overlap (open models)

Measure what share of the benchmark appears in the model's public pre-training
corpus, using a string match. This needs the corpus to be searchable, and it
gives a direct count only for that corpus [arxiv:2310.18018 §5.1]. Prior audit
work on open corpora uses the same kind of search, and the paper names it as
the usual approach [arxiv:2310.18018 §5.1]. The paper does not say whether the
match is exact or uses n-grams, so the rule must be fixed and reported
[arxiv:2310.18018 §5.1]. Card: [NLP Evaluation in trouble](/papers/2310.18018).

### Extractability ratio (closed models)

Give the model the k examples that come just before each benchmark example,
and check whether it reproduces that example. The contamination degree is the
share of examples reproduced [arxiv:2310.18018 §5.2]. The method is adapted
from the extractability definition of Carlini et al. (2023), which the paper
cites but does not reproduce [arxiv:2310.18018 §5.2]. Non-verbatim outputs
are judged by a human reviewer, so the step is manual and two auditors may
disagree [arxiv:2310.18018 §5.2]. Failing to reproduce an example does not
settle the question [arxiv:2310.18018 §5.2]. The paper has no measured rates;
its only demonstration is three qualitative cases, in which the opening lines
of a standard NER training split come back from three models [app. A]
[fig. 1] [fig. 2] [fig. 3]. Card: [the same position paper](/papers/2310.18018).

### Name-cloze memory probe (closed chat models)

Take a 40 to 60 token passage that contains exactly one person name, mask
the name, and ask a chat model to fill the blank. The passages contain no other
named entities, so a reader who only has the passage cannot guess the name
[arxiv:2305.00118 §3.1] [§3.2]. The score for a book is the share of its
sampled passages answered correctly. Among fiction, GPT-4 reaches 0.244 on
pre-1923 public-domain novels, against 0.072 for ChatGPT, and scores fall
to 0.02 to 0.03 for most contemporary categories [arxiv:2305.00118 tab. 2].
Chance-level baselines are a name-frequency guess at 0.6 percent and one
annotator at 0 percent [arxiv:2305.00118 §3.2]. Note that the matching rule and
the temperature are not stated in the paper. This variant measures
recall of a text, not a benchmark score, so it is a probe for a
suspected-seen corpus. Card: [Speak, Memory](/papers/2305.00118).

### Web prevalence as a prior (any model)

Count exact hits for short spans of each text in web-scale indexes, then
correlate the counts with name-cloze accuracy. For books published before
1928 the Spearman correlation is 0.70 to 0.84 across the four indexes; after
1928 it falls to 0.21 to 0.41 [arxiv:2305.00118 tab. 3] [§6.2]. This needs
search access and no access to the model. The authors describe the link as a
correlation, and they argue that popularity drives both prevalence and
memorisation [arxiv:2305.00118 §6.2]. Use the counts to rank texts by likely
exposure, not to decide that a model saw a specific text. Card: [the same archaeology paper](/papers/2305.00118).

### Comparing groups by memorisation (downstream effect)

Split the texts into the most and least memorised deciles, then compare the
model on a related task across the two groups. On year-of-publication
estimates, GPT-4's mean absolute error is 0.6 years in the top decile and
14.5 years in the bottom decile [arxiv:2305.00118 tab. 5]. A narrative-time
task does not show a significant difference between those groups
[arxiv:2305.00118 §7.2]. The grouping uses the same accuracy the comparison
then tests, so the comparison is not fully independent [arxiv:2305.00118 §7.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Open weights and a public pre-training corpus | Benchmark data overlap [arxiv:2310.18018 §5.1] | Direct count against the data; fix the match rule first |
| Closed API model and a benchmark to judge | Extractability ratio [arxiv:2310.18018 §5.2] | Needs only generation access, but a human audits non-verbatim output and no threshold is given [arxiv:2310.18018 §5] |
| Closed model and a narrative or literary text | Name-cloze probe [arxiv:2305.00118 §3.1] | Works on any text with named characters; the card's method needs chat access and about $400 at 2023 prices [arxiv:2305.00118 §5.1] |
| No model access, only search access | Web prevalence as a prior [arxiv:2305.00118 §6.2] | Ranks texts by exposure; correlational only |
| Split of the benchmark is unknown | Check which split was exposed [arxiv:2310.18018 §3] | Evaluation-split exposure invalidates the test; train or dev exposure breaks zero-shot and few-shot claims |
| Deciding whether memorisation matters for another task | Downstream comparison by memorisation group [arxiv:2305.00118 tab. 5] | Shows the size of an effect on a related task, not a cause |

## Implementation sketch

The variants share one core: build (context, target) pairs, query the model,
and count matches. Each variant changes how the pairs are built and what
counts as a match.

```pseudocode
# Recall rate: share of items the model reproduces  [arxiv:2310.18018 §5.2]
# Input:  model (callable), items (list of (context, target)), matches (fn)
# Output: fraction of scored items recalled, in [0, 1]

def recall_rate(model, items, matches):
    # Extractability: context = k preceding examples  [arxiv:2310.18018 §5.2]
    # Name-cloze: context = passage with the name masked  [arxiv:2305.00118 §3.2]
    hits = 0
    for context, target in items:
        # Ask the model and take its raw reply  [arxiv:2305.00118 §3.2]
        reply = model(context)
        # The match rule is a choice: exact string, normalised, or n-gram
        # [arxiv:2310.18018 §5.1] [arxiv:2305.00118 §3.2]
        if matches(reply, target):
            hits = hits + 1
    # Denominator: items with a context; the first k have none
    # [arxiv:2310.18018 §5.2]
    return hits / len(items)
```

## Contradictions

Low recall is read differently by the two papers, and the disagreement is about
interpretation, not the numbers. The position paper says that a model which
does not reproduce a benchmark may still have trained on it, and calls this a
question for further research [arxiv:2310.18018 §5.2]. The memory study treats
the bottom decile of name-cloze accuracy as texts the model has not memorised,
and uses that group as the unseen comparison [arxiv:2305.00118 §6.1]. Neither
paper resolves this, so a low score cannot by itself be read as clean.

Scope differs too. The position paper concerns benchmark data; the memory study
measures books and draws the benchmark link as an argument
[arxiv:2310.18018 §1] [arxiv:2305.00118 §8].

## Papers

- [Speak, Memory](/papers/2305.00118): name-cloze probe on 571 works; memorisation tracks web prevalence and affects year estimates.
- [NLP Evaluation in trouble](/papers/2310.18018): position paper defining three contamination types and two measures (overlap for open models, extractability for closed models); no experiments.
