Analyzing Leakage of Personally Identifiable Information in Language Models

In plain words

A language model learns from the text it is trained on, and sometimes it repeats personal details from that text: a person's name, an address, a phone number. Companies try to prevent this in two ways. They scrub personal details out of the training text before training, and they train with differential privacy, a mathematical method that limits how much any one record can influence the model. This paper asks how much personal information still gets out when both protections are used, and how much each one actually helps.

The authors look at the problem from the outside. They assume an attacker who can only send text to the model and read back the probabilities of the next word. They define three kinds of attack. In extraction, the attacker has no context and just tries to get the model to produce names. In reconstruction, the attacker sees a sentence with one name blanked out and guesses the name from the words around it. In inference, the attacker also gets a short list of possible names and must pick the right one. They fine-tuned small GPT-2 models on court cases, company emails and healthcare reviews, with and without the protections, and measured what came out.

The main lessons are that the reconstruction method recovers far more names than the earlier method, and that differential privacy cuts leakage sharply but does not remove it. Scrubbing is imperfect, so some names remain in the training text even after it runs.

TL;DR

• Fine-tuned GPT-2 models on three domains (ECHR court cases, Enron emails, Yelp healthcare reviews), undefended, with DP-SGD (eps=8), and with scrubbing §IV-A §IV-C.
• Three black-box attacks with game-based definitions: extraction, reconstruction and inference §III-A §III-B §III-C.
• The reconstruction attack samples candidate names from the model after the prefix, fills the masked slot with each, and picks the lowest-perplexity sentence alg. 6. Author-reported: 18.27% vs 5.81% for TAB on ECHR with GPT-2-Large §IV-F.
• DP (eps=8) lowers extraction precision and recall to about 3%, and keeps reconstruction at or below 1%. Inference with 100 candidates still reaches about 8% on ECHR (author-reported) §IV-E §IV-F §IV-G.
• The official MIT-licensed repo covers ECHR only and omits several parts of the paper. Read "Paper vs code" before reproducing any number.

Problem

Scrubbing and differential privacy (DP) both protect training data, but both cost model utility. Scrubbing that is aggressive enough to matter degrades the model, and DP training degrades perplexity in proportion to the privacy budget spent §I. There was no measure of how much personal information leaks from a pipeline that uses both defenses §I.

DP protects who contributed a record. It does not protect the content of a piece of personal information that appears in many records or is shared among a small group §II-D. Scrubbing is also imperfect: NER taggers miss some personal information, and a masked training sentence still shows where the missing piece sat §II-C §IV-B.

Measuring leakage needs labelled personal data, which is scarce. The authors therefore use off-the-shelf NER to label personal data in the training text and in the generated text, so their numbers are a lower bound on leakage §V.

Method

Setting and adversary

• Pipeline: data curation (scrubbing and de-duplication), then algorithmic defences (DP training), then a black-box API that returns the next-token probability vector. Parameters and features stay behind the API §II-D fig. 3.
• Probabilities factor by the chain rule over tokens eq. 1. Perplexity is the exponential of the mean negative log-probability per token §II-A.
• DP is the usual (eps, delta) definition over adjacent datasets [def. 1]. DP-SGD is the training algorithm used for the DP models §II-B.
• Extraction uses no context. Reconstruction gets the scrubbed prefix and suffix of the target. Inference additionally gets a candidate set that contains the true value §II-E.

Leakage definitions

Each attack is a game between a challenger and an adversary alg. 2 alg. 5 alg. 7. Extraction success is the expected recall of the adversary's guessed set against the true PII set, with the guessed set capped at its true size eq. 2 alg. 2. Extractability of a PII string C is the probability that the model emits C across all possible prefixes eq. 3. Reconstruction success is the probability that the adversary's guess equals the true PII §III-B [app. A]. Inference is reconstruction restricted to a given candidate set §III-C.

Scrubbing

Scrub every tagged PII span by replacing it with the mask token alg. 1. The span is found by splitting the sentence at the tagged text, which assumes the text occurs once or is unique within the sentence §II-C.

# PII scrubbing  [alg. 1]
# Input:  dataset D (list of sentences)
# Output: scrubbed dataset D'

def scrub(dataset):
    # 1. Tag PII in each sentence with an NER tagger  [alg. 1]
    scrubbed = []
    for sentence in dataset:
        for pii in tag_pii(sentence):
            # 2. Replace each tagged span with the mask token  [alg. 1]
            sentence = replace_once(sentence, pii, "[MASK]")
        scrubbed.append(sentence)
    return scrubbed

Extraction

The observed attack draws unconditional samples and counts the PII of the target class that appears in them alg. 3. The paper samples about 4 million tokens for each model: 15k queries of 256 tokens from an empty prompt with top-k = 40 §IV-E. Precision is the share of extracted strings that appear in the training data, and recall is the share of training PII that gets extracted app. A.

# Observed PII extractability  [alg. 3]
# Input:  target (str), model, n_samples (int)
# Output: fraction of samples that contain the target

def observed_extractability(target, model, n_samples):
    # 1. Sample texts from the model with an empty prompt  [alg. 3]
    samples = model.sample(prompt="", n=n_samples)
    hits = 0
    for sample in samples:
        # 2. Tag PII of the target's class; count exact matches  [alg. 3]
        if target in tag_pii(sample, pii_class):
            hits += 1
    return hits / len(samples)

The lazy estimator avoids sampling until the model has produced the target. It assumes that PII follow grammar, so a slot only admits PII of its class, and that PII of the same class are interchangeable §III-A. For every tagged PII in each sample, it substitutes the target and accumulates the probability of the target given the text before the slot. The average over slots is the estimate alg. 4.

# Lazy estimate of PII extractability  [alg. 4]
# Input:  target (str), model, n_samples (int)
# Output: mean Pr(target | prefix) over tagged PII slots

def estimated_extractability(target, model, n_samples):
    # 1. Sample texts as in the observed attack  [alg. 4]
    samples = model.sample(prompt="", n=n_samples)
    total = 0.0
    slots = 0
    for sample in samples:
        # 2. Visit each same-class PII as a slot  [alg. 4]
        for prefix in slot_prefixes(sample, pii_class):
            # 3. Add the probability of the target after this prefix  [alg. 4]
            total += model.prob_sequence(target, given=prefix)
            slots += 1
    return total / slots

Reconstruction and inference

Reconstruction improves on the TAB attack, which only decodes the next tokens after the prefix §III-B §VI. The improved attack uses the suffix as well. It (1) fills residual masks in the prefix and suffix with a masked language model, one mask at a time from left to right app. B; (2) if no candidates are given, samples N texts from the model after the prefix and collects the PII they contain alg. 6; (3) scores each candidate by perplexity of the whole sentence and returns the lowest eq. 4 alg. 6.

# PII reconstruction and inference  [alg. 6]
# Input:  prefix, suffix (scrubbed text), model, n_samples, candidates
# Output: candidate with the lowest perplexity of the whole sentence

def reconstruct(prefix, suffix, model, n_samples, candidates=None):
    # 1. Fill residual [MASK] tokens  [alg. 6] [app. B]
    prefix = fill_masks(prefix)
    suffix = fill_masks(suffix)
    # 2. Sample candidates after the prefix if none are given  [alg. 6]
    if not candidates:
        samples = model.sample(prompt=prefix, n=n_samples)
        candidates = {
            pii
            for sample in samples
            for pii in tag_pii(sample, pii_class)
        }
    # 3. Score each candidate on the full sentence  [alg. 6] [eq. 4]
    scores = {
        candidate: perplexity(prefix + candidate + suffix, model)
        for candidate in candidates
    }
    return min(scores, key=scores.get)

Inference uses the same routine with the candidate set supplied, so no sampling happens §III-C alg. 7.

Equations

\Pr(w_1,\dots,w_n;\theta)=\prod_{i=1}^{n}\Pr(w_i\mid w_1,\dots,w_{i-1};\theta) \qquad [\text{eq. 1}]

\mathrm{Succ}_{\mathrm{Extraction}}=\mathbb{E}\left[ \frac{|\mathcal{C}\cap\tilde{\mathcal{C}}|}{|\mathcal{C}|}\right] \qquad [\text{eq. 2}]

\mathrm{Extractability}(C;\theta)=\sum_{S}\Pr(S;\theta)\,\Pr(C\mid S;\theta) =\sum_{S}\Pr(S\,C;\theta) \qquad [\text{eq. 3}]

\tilde{C}=\operatorname*{arg\,max}_{C}\ \Pr(S_0\,C\,S_1;\theta) \qquad [\text{eq. 4}]

Perplexity: \text{PPL}(w_1,\dots,w_n;\theta)=\exp\left(-\frac{1}{n} \sum_{i=1}^{n}\log\Pr(w_i\mid w_1,\dots,w_{i-1};\theta)\right) §II-A (unnumbered in the paper).

Hyperparameters

Results

Author-reported. Setup: GPT-2 small/medium/large fine-tuned on ECHR, Enron and Yelp-Health; "No DP" and DP with eps=8; metrics are precision and recall of observed PII extraction after about 4M sampled tokens tab. 3 §IV-E.

Dataset statistics, for the person class tab. 7 app. A:

Reconstruction, Table IV is missing from the source tree, so these values come from the running text §IV-F:

• ECHR, GPT-2-Large: TAB correctly reconstructs at least 5.81% of person PII; this attack reaches 18.27%. The paper states it improves the baseline by at least 2.5x on ECHR, 7.5x on Enron and about 3x on Yelp-Health (except GPT-2-Small on Yelp: 0.33% to 0.42%).
• Up to 10x more PII than TAB, with GPT-2-Medium on Enron.
• DP models: at most 1%, and the authors do not consider it practical.
• Enron, when the true PII is among the candidates, it is predicted correctly about 70% of the time, but the sampled candidates often do not include it.

Inference, Table V is missing from the source tree, so these values come from the running text §IV-G. GPT-2-Large, undefended, top-1 accuracy among candidates:

• 100 candidates: ECHR 70%, Enron 50%, Yelp-Health 28%.
• DP (eps=8), ECHR: about 8% with 100 candidates, about 4% with 500.
• Larger context (prefix plus suffix) slightly lowers accuracy §IV-G.

Other results §IV-E §IV-H:

• Undefended extraction of other classes: law case numbers 14.1%, organisation names 16.3%. DP: 2.8% and 4.1%. Enron phone numbers never leaked under DP.
• Sample-size effect (ECHR, undefended): precision falls from 50% at 500k tokens to 30% at 4M, and recall rises to 23%.
• Sentence-level membership inference (shadow model, ECHR), ROC AUC: undefended 0.96; scrubbed 0.82; DP and scrubbed 0.505 §IV-H.
• The summary table comparing utility and leakage for GPT-2-Large on ECHR is not in the source tree tab. 6.

Paper vs code

Official code: microsoft/analysing_pii_leakage, MIT, at commit 90ad25e037ea0983ccb750e3e9efdd3cb693865b. Checked: attacks, configs, NER wrapper, DP training path and evaluation script.

• Datasets: the paper uses ECHR, Enron and Yelp-Health §IV-A. The repo ships only the ECHR loader and all configs point to ECHR. dataset_args.py lists an enron path that does not exist dataset_args.py.
• NER: the paper scrubs with Flair and a second tagger §IV-B. The repo has only Flair tagger_factory.py. The paper reports 21 entity classes app. A; the Flair wrapper lists 18 flair_tagger.py.
• Lazy extractability (alg. 4) is not implemented. Only the observed count is in the repo naive_extraction.py.
• Extraction cap: the game limits the guessed set to |C| alg. 2. The repo returns every mention it finds naive_extraction.py.
• Extraction baseline: the paper uses 13M tokens from a public model §IV-E. The eval script samples the pretrained model with the attack's own budget evaluate.py.
• Reconstruction masks: Algorithm 6 fills prefix and suffix separately alg. 6. The repo fills all <MASK> tokens in the whole query at once, using a windowed RoBERTa pass perplexity_reconstruction.py fill_masks.py.
• Candidate sampling: the paper samples 64 candidates with top-k 40 §IV-F. The repo uses sampling_rate (default 32, demo config 100), a fixed 32-token length, PERSON-class candidates only, and excludes names already in the query perplexity_reconstruction.py.
• Candidate distribution for inference: the paper draws candidates from E, which picks a sentence and then a PII uniformly from it, so frequent names appear more often §III-C. The repo draws uniformly from unique PII strings evaluate.py.
• Excluding baseline leakage: the paper excludes cases where the pretrained model correctly predicts the PII §III-D. The repo excludes cases where the pretrained model gives the same prediction as the fine-tuned model, correct or not evaluate.py.
• Evaluation filter: the repo ignores PII shorter than 4 characters, which the paper does not mention evaluate.py.
• Training: the paper uses batch 64 throughout §IV-C. The undefended and scrubbed configs give an effective batch of 32 echr-gpt2-small-undefended.yml. The paper trains undefended models until validation perplexity stops improving §IV-C. The repo runs 4 epochs and has no stopping rule echr-gpt2-small-undefended.yml.
• Extraction demo: the paper uses 15k queries of 256 tokens §IV-E. The shipped config uses 100 samples of 128 tokens echr-gpt2-small-undefended.yml.
• Membership inference (shadow models, AUC 0.96 / 0.82 / 0.505) §IV-H has no code in the repo.
• Dataset statistics: the text gives 123,317 Enron records and 5.53 duplicates per Yelp PII app. A; the table gives 138,919 and 5.35 tab. 7.
• Dependencies: flair and dp-transformers are installed from git main, and torch<2.0 is pinned requirements.txt. The paper gives no library versions.

No discrepancies found for: sampling top-k 40, DP epsilon 8, clip norm 1.0, DP 4 epochs, and the use of Flair OntoNotes-large as the tagger echr-gpt2-small-dp8.yml sampling_args.py.

Pitfalls

• Section numbers follow IEEE's Roman-numeral style in the rendered paper (§II-B, §IV-F). Equations 1 to 4 are the only numbered equations in the main text. Appendix equations and algorithms are cited by appendix only.
• Missing source files: Tables/threat_model.tex, pii_reconstruction.tex, pii_inference.tex and result_summary.tex are referenced but absent from the arXiv source. Reconstruction and inference numbers here come from the text. Table IV and Table V bodies are unavailable.
• The dataset table and the appendix text disagree on the Enron record count and the Yelp duplication rate (see "Paper vs code").
• Precision and recall are lower bounds: the NER tagger both scrubs the training text and labels the generated text, so its misses count as non-leakage §V. The paper says so directly.
• Extraction precision drops as more text is sampled, from 50% at 500k tokens to 30% at 4M §IV-E. Report precision together with recall.
• Baseline subtraction removes PII the pretrained model already produces. This can hide real leakage, for example a public person's name in a sensitive context §III-D.
• Sampling cost: 4M tokens means 15k generation queries plus NER over all of the generated text. The lazy estimator avoids the NER pass, but it is not in the repo §III-A.
• Reconstruction samples its candidates from the prefix only. The authors say suffix-aware sampling would probably improve it §IV-F.
• Joining prefix, candidate and suffix: the paper writes S0 C S1 without stating the spacing. The repo replaces the <T-MASK> string directly perpexity_inference.py.
• The perplexity dictionary is keyed by score. Two candidates with an identical perplexity collapse into one entry. This is rare with floating point but possible.
• Candidate sets in the repo always include the true PII, and the rest are random training-split PII strings evaluate.py. Real attacks have no such guarantee, so the 70% and 8% numbers are best-case for the attacker.
• DP with eps=8 is the paper's reference setting. It is justified by deployed systems (Apple QuickType, Google Smart Compose) rather than a tuned utility target §IV-C.
• The results are for GPT-2 family models on English text. The repo supports only the small size in its configs.

Lineage

• Builds on: DP-SGD [Abadi et al., 2016] §II-B; Opacus and dp-transformers §IV-C; GPT-2 §IV-C; Flair NER §II-C §IV-B.
• Compared against: the TAB attack, which decodes the next tokens after the prefix §III-B §IV-F; the extraction line of work on sentence-level memorisation §VI.
• Related: membership inference via shadow models §IV-H; de-duplication effects on memorisation §IV-E (contrasts with Kandpal et al., 2022).
• Superseded by: none known from this wiki.