---
kind: concept
slug: training-data-extraction
title: Training data extraction
summary: >-
  Asks how much personal information a language model reveals from its training text through black-box queries. Covers observed sampling, a lazy estimator, prefix-and-suffix reconstruction and candidate-set inference, measured on fine-tuned GPT-2 with and without scrubbing and DP-SGD. Numbers are author-reported. DP-SGD at eps=8 cuts leakage sharply but not to zero.
tags: [privacy, memorization, data-extraction, pii]
papers: ["2302.00539"]
related: [differential-privacy, differentially-private-sgd, membership-inference, named-entity-recognition, data-poisoning]
status: draft
updated_at: 2026-10-31
---

## Overview

Training data extraction asks whether a deployed language model will repeat
records from its training set, and how much of that output is sensitive. The
problem is measured as leakage of personally identifiable information (PII):
names, places, phone numbers and similar strings [arxiv:2302.00539 §I]. The
attacker is assumed to have black-box access. They can send text and read the
next-token probability vector, but cannot see parameters or features
[arxiv:2302.00539 §II-D].

The invariant shared by all variants is that the attacker only sees model
outputs and tries to recover tagged PII. Every variant is scored with a
named-entity tagger, so leakage numbers are lower bounds: the tagger's misses
count as non-leakage [arxiv:2302.00539 §V].

## Variants

### Observed extraction by sampling

Draw unconditional samples from the model, tag PII of the target class in each
sample, and count exact matches with the training PII [arxiv:2302.00539
alg. 3]. The paper uses about 4 million tokens per model: 15k queries of 256
tokens from an empty prompt, with top-k = 40 [arxiv:2302.00539 §IV-E]. On
ECHR with GPT-2 small and no defence, the author-reported precision is 24.91%
and recall is 9.44% [arxiv:2302.00539 tab. 3]. Card:
[Analyzing Leakage of Personally Identifiable Information in Language
Models](/papers/2302.00539).

### Lazy extractability estimate

For a specific training string, the lazy estimator avoids waiting for the
model to emit it. It takes the same samples, visits each slot where PII of
the same class appears, and averages the probability of the target after each
prefix [arxiv:2302.00539 alg. 4] [arxiv:2302.00539 §III-A]. It relies on two
assumptions: that a slot only admits PII of its class, and that PII of one
class are interchangeable. The paper states both [arxiv:2302.00539 §III-A].
The card's Paper vs code section records that this estimator is not in the
released code, so reproducing it means implementing it from the paper.

### Reconstruction from prefix and suffix

Given a sentence whose PII is masked, reconstruction guesses the hidden
string. It improves on the TAB attack, which decodes only the tokens after the
prefix [arxiv:2302.00539 §III-B]. The method fills any remaining masks in the
context, samples candidates from the model after the prefix, and returns the
candidate whose whole sentence has the lowest perplexity [arxiv:2302.00539
alg. 6] [arxiv:2302.00539 eq. 4]. On ECHR with GPT-2-Large, the author-reported
rate is 18.27%, against 5.81% for TAB [arxiv:2302.00539 §IV-F]. Under DP-SGD
at eps=8 the same attack reaches at most 1% [arxiv:2302.00539 §IV-F].

### Inference over a candidate set

When the candidate list is given, the reconstruction routine skips sampling
and only scores the candidates [arxiv:2302.00539 §III-C] [arxiv:2302.00539
alg. 7]. With 100 candidates, GPT-2-Large without a defence reaches top-1
accuracy of 70% on ECHR, 50% on Enron and 28% on Yelp-Health, author-reported
[arxiv:2302.00539 §IV-G]. With DP at eps=8, ECHR reaches about 8% with 100
candidates and about 4% with 500 [arxiv:2302.00539 §IV-G]. These figures are
best-case for the attacker, because the evaluation set-up places the true PII
inside the candidate set [arxiv:2302.00539 §IV-G]. See the card's Paper vs code
section before using them.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want a population-level leakage rate and can afford a large query and tagging budget | Observed extraction | It needs no context and matches the paper's measurement. Precision falls as more text is sampled, so report recall with it [arxiv:2302.00539 §IV-E] |
| You want the chance that one known string is produced, and the model is available but NER over all output is too costly | Lazy estimator | It avoids tagging the output for each target, but it assumes PII of one class are interchangeable [arxiv:2302.00539 §III-A] |
| You have the context around a masked PII | Reconstruction | Uses both sides of the mask and beats the prefix-only TAB attack in the paper's setting [arxiv:2302.00539 §IV-F] |
| You already hold a shortlist that may contain the answer | Inference | Requires no sampling, and it is the strongest result in the paper, though its candidate set is optimistic [arxiv:2302.00539 §IV-G] |
| You are choosing a defence to test against these attacks | Scrubbing plus DP-SGD (eps=8) | Undefended precision and recall of about 6 to 35% fall to roughly 2 to 12% with DP [arxiv:2302.00539 tab. 3]. The paper gives no utility cost for DP in this card, so trade-offs must come from elsewhere |

Scrubbing alone does not remove leakage. NER-based masking misses some PII,
and a masked sentence still shows where the missing string sat [arxiv:2302.00539
§II-C] [arxiv:2302.00539 §IV-B]. DP also does not protect a string that
recurs in many records or within a small group [arxiv:2302.00539 §II-D]. For
the separate question of whether a specific record was in the training set,
see [membership-inference](/concepts/membership-inference).

## Implementation sketch

The common core of sampling-based extraction and reconstruction is the same:
generate or receive context, tag PII, and rank the candidates. The paper's
reconstruction routine is shown below. Observed extraction is the same loop
without the scoring step. The lazy estimator replaces the count with an
average of probabilities.

```pseudocode
# Reconstruction and inference  [arxiv:2302.00539 alg. 6]
# Input:  prefix, suffix (scrubbed text), model, n_samples, candidates
# Output: candidate with the lowest perplexity of the whole sentence

def reconstruct(prefix, suffix, model, n_samples, candidates=None):
    # 1. Fill any residual mask tokens in the context  [alg. 6]
    prefix = fill_masks(prefix)
    suffix = fill_masks(suffix)
    # 2. Sample candidates after the prefix if none are given  [alg. 6]
    if not candidates:
        samples = model.sample(prompt=prefix, n=n_samples)
        candidates = {
            pii
            for sample in samples
            for pii in tag_pii(sample)
        }
    # 3. Score each candidate on the full sentence  [alg. 6] [eq. 4]
    scores = {
        c: perplexity(prefix + c + suffix, model)
        for c in candidates
    }
    return min(scores, key=scores.get)
```

## Contradictions

None known across papers, since only one card covers this slug. Within the
paper, the text and the dataset table disagree on the Enron record count and
on the Yelp duplication rate [arxiv:2302.00539 app. A] [arxiv:2302.00539
tab. 7]. This affects dataset statistics only, not the attack numbers. The
Table IV and Table V bodies are also missing from the source, so the
reconstruction and inference figures above come from the paper's running text.

## Papers

- [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539): measures PII extraction, reconstruction and inference on fine-tuned GPT-2, with and without scrubbing and DP-SGD.