Training data extraction

Overview

Training data extraction asks whether a deployed language model will repeat records from its training set, and how much of that output is sensitive. The problem is measured as leakage of personally identifiable information (PII): names, places, phone numbers and similar strings 2302.00539 §I. The attacker is assumed to have black-box access. They can send text and read the next-token probability vector, but cannot see parameters or features 2302.00539 §II-D.

The invariant shared by all variants is that the attacker only sees model outputs and tries to recover tagged PII. Every variant is scored with a named-entity tagger, so leakage numbers are lower bounds: the tagger's misses count as non-leakage 2302.00539 §V.

Variants

Observed extraction by sampling

Draw unconditional samples from the model, tag PII of the target class in each sample, and count exact matches with the training PII [arxiv:2302.00539 alg. 3]. The paper uses about 4 million tokens per model: 15k queries of 256 tokens from an empty prompt, with top-k = 40 2302.00539 §IV-E. On ECHR with GPT-2 small and no defence, the author-reported precision is 24.91% and recall is 9.44% 2302.00539 tab. 3. Card: Analyzing Leakage of Personally Identifiable Information in Language Models.

Lazy extractability estimate

For a specific training string, the lazy estimator avoids waiting for the model to emit it. It takes the same samples, visits each slot where PII of the same class appears, and averages the probability of the target after each prefix 2302.00539 alg. 4 2302.00539 §III-A. It relies on two assumptions: that a slot only admits PII of its class, and that PII of one class are interchangeable. The paper states both 2302.00539 §III-A. The card's Paper vs code section records that this estimator is not in the released code, so reproducing it means implementing it from the paper.

Reconstruction from prefix and suffix

Given a sentence whose PII is masked, reconstruction guesses the hidden string. It improves on the TAB attack, which decodes only the tokens after the prefix 2302.00539 §III-B. The method fills any remaining masks in the context, samples candidates from the model after the prefix, and returns the candidate whose whole sentence has the lowest perplexity [arxiv:2302.00539 alg. 6] 2302.00539 eq. 4. On ECHR with GPT-2-Large, the author-reported rate is 18.27%, against 5.81% for TAB 2302.00539 §IV-F. Under DP-SGD at eps=8 the same attack reaches at most 1% 2302.00539 §IV-F.

Inference over a candidate set

When the candidate list is given, the reconstruction routine skips sampling and only scores the candidates 2302.00539 §III-C [arxiv:2302.00539 alg. 7]. With 100 candidates, GPT-2-Large without a defence reaches top-1 accuracy of 70% on ECHR, 50% on Enron and 28% on Yelp-Health, author-reported 2302.00539 §IV-G. With DP at eps=8, ECHR reaches about 8% with 100 candidates and about 4% with 500 2302.00539 §IV-G. These figures are best-case for the attacker, because the evaluation set-up places the true PII inside the candidate set 2302.00539 §IV-G. See the card's Paper vs code section before using them.

When to use which

Scrubbing alone does not remove leakage. NER-based masking misses some PII, and a masked sentence still shows where the missing string sat [arxiv:2302.00539 §II-C] 2302.00539 §IV-B. DP also does not protect a string that recurs in many records or within a small group 2302.00539 §II-D. For the separate question of whether a specific record was in the training set, see membership-inference.

Implementation sketch

The common core of sampling-based extraction and reconstruction is the same: generate or receive context, tag PII, and rank the candidates. The paper's reconstruction routine is shown below. Observed extraction is the same loop without the scoring step. The lazy estimator replaces the count with an average of probabilities.

# Reconstruction and inference  [arxiv:2302.00539 alg. 6]
# Input:  prefix, suffix (scrubbed text), model, n_samples, candidates
# Output: candidate with the lowest perplexity of the whole sentence

def reconstruct(prefix, suffix, model, n_samples, candidates=None):
    # 1. Fill any residual mask tokens in the context  [alg. 6]
    prefix = fill_masks(prefix)
    suffix = fill_masks(suffix)
    # 2. Sample candidates after the prefix if none are given  [alg. 6]
    if not candidates:
        samples = model.sample(prompt=prefix, n=n_samples)
        candidates = {
            pii
            for sample in samples
            for pii in tag_pii(sample)
        }
    # 3. Score each candidate on the full sentence  [alg. 6] [eq. 4]
    scores = {
        c: perplexity(prefix + c + suffix, model)
        for c in candidates
    }
    return min(scores, key=scores.get)

Contradictions

None known across papers, since only one card covers this slug. Within the paper, the text and the dataset table disagree on the Enron record count and on the Yelp duplication rate 2302.00539 app. A [arxiv:2302.00539 tab. 7]. This affects dataset statistics only, not the attack numbers. The Table IV and Table V bodies are also missing from the source, so the reconstruction and inference figures above come from the paper's running text.

Papers

• Analyzing Leakage of Personally Identifiable Information in Language Models: measures PII extraction, reconstruction and inference on fine-tuned GPT-2, with and without scrubbing and DP-SGD.