--- kind: concept slug: membership-inference title: Membership inference summary: >- Asks whether a record was in a model's training set. Covers shadow-model, per-record reference-model and distillation attacks, a black-box loss-threshold test, poisoning-assisted DP-SGD audits, and a sentence-level test on fine-tuned GPT-2. Numbers are author-reported. DP-SGD cuts measured GPT-2 leakage, but no card gives its utility cost. tags: [privacy, attacks, auditing] papers: ["2101.04535", "2111.09679", "2112.04359", "2302.00539", "2310.20150", "2402.00888", "2402.04013"] related: [differential-privacy, differentially-private-sgd, knowledge-distillation, leave-one-out-analysis, data-poisoning, model-inversion, machine-unlearning, llm-ethics] status: draft updated_at: 2026-11-01 --- ## Overview Membership inference asks whether a particular example was part of a model's training data. The survey lists it as one of the privacy attack families for language models, next to gradient leakage and PII leakage [arxiv:2402.00888 §3] [fig. 4]. The DP audit paper uses a membership-style test as its black-box baseline and also as the building block of stronger tests [arxiv:2101.04535 §4.1]. Its distinguisher guesses which of two training sets, differing by one example, produced the model [arxiv:2101.04535 §3.2]. The enhanced-attack paper frames the test as a hypothesis test on the loss. It compares four black-box attacks that differ in how much they know about the target model and record, all on small classifiers [arxiv:2111.09679 §3, §4]. The cards differ in scope. The survey describes attacks and defences but runs no experiments [arxiv:2402.00888 §1]. The audit paper measures attacks on small classifiers: a 4-layer CNN on MNIST and CIFAR-10 and a 3-layer fully connected network on Purchase [arxiv:2101.04535 app. A]. The PII leakage paper is the only card that measures a membership test on a language model with defences. It runs a sentence-level shadow-model test on fine-tuned GPT-2 [arxiv:2302.00539 §IV-H]. The unlearning paper uses a membership-style classifier as a post-deletion check on T5 language models [arxiv:2310.20150 §4.6]. That check is not an attack on training records, and the paper reports it near chance even before unlearning. Two other cards sit next to this topic. The model inversion survey contrasts the two: membership inference reveals narrow aspects of the training data, while model inversion can reconstruct training samples [arxiv:2402.04013 §1]. The ethics taxonomy lists privacy loss from memorised training text as an information hazard [arxiv:2112.04359 §2.2]. It describes no membership test, so it is a related risk here, not a source of attack methods. ## Variants ### Shadow-model attack Train several shadow models on data similar to the target's training data. Collect their outputs on known members and known non-members, label them, and fit an attack classifier per label [arxiv:2402.00888 §5.2]. Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888). - The survey says shadow data must be disjoint from the target's training data but similar to it, and calls that assumption unrealistic in many cases [arxiv:2402.00888 §5.2]. - Number of shadow models and the attack classifier type are not stated [arxiv:2402.00888 §5.2]. - The survey names SaMIA as a later variant, tested only on WikiMIA and only on classification [arxiv:2402.00888 §5.2]. - The enhanced-attack paper uses the same idea as its baseline, attack S. It takes one threshold per label from shadow losses on population records with the target's label [arxiv:2111.09679 §4.1]. Card: [Enhanced Membership Inference Attacks against Machine Learning Models](/papers/2111.09679). ### Per-record reference-model and distillation attacks These attacks give each record its own threshold, so they adapt to atypical examples. Attack R trains reference models on random population subsets and sets the threshold from their losses on the target record [arxiv:2111.09679 §4.3]. Attack D trains models distilled from the target's own outputs on random population data, and sets the threshold from their losses on the record [arxiv:2111.09679 §4.4]. Attack L, which trains on the target's training set minus the record, is an idealised upper reference that needs the exact n-1 records [arxiv:2111.09679 §4.5]. Card: [Enhanced Membership Inference Attacks against Machine Learning Models](/papers/2111.09679). - Author-reported AUC on Purchase100, setup II: S 0.754, P 0.756, R 0.808, D 0.831 [arxiv:2111.09679 tab. AUC Purchase100]. Setup I gives S 0.800 and D 0.837 [arxiv:2111.09679 tab. AUC Purchase100]. - At a fixed false-positive rate below 0.2, R and D reach more than ten times the true-positive rate of S and P on the Purchase100 setup [arxiv:2111.09679 §5.2]. - Reference models: 999 per target for Purchase100 setups I to III, and 29 for setup IV, per the paper [arxiv:2111.09679 app. B.2]. The card records that the public code's README sets 29 for R and D, so this count differs from the paper's [card: Paper vs code]. - Whether distilled models are shared across attacked records is not stated. The card reads the paper as needing one set per record, so check compute before planning [card: Pitfalls]. - Only gradient clipping is tested as a defence. The ordering S < P < R < D persists under it [arxiv:2111.09679 app. D]. ### Black-box loss-threshold test The distinguisher only sees the trained model. It computes the loss on the crafted example and guesses that the example was present when the loss is at most a threshold tau [arxiv:2101.04535 §4.1]. Card: [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535). - The threshold is not stated [arxiv:2101.04535 §4.1]. - Reported result: CIFAR-10, 4-layer CNN, target epsilon 4 with delta 1e-5. The proven epsilon is 4 and the empirical lower bound is 0.31 [arxiv:2101.04535 §4.1]. The authors describe this as far below the proven value. ### Poisoning-assisted audits The same game, with more attacker power. The crafter inserts a poisoned example, the distinguisher sees intermediate models, or the crafter controls gradients [arxiv:2101.04535 tab. 1]. Card: [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535). - Static poison, final model: higher than black-box access on all three datasets. Given only in a figure [arxiv:2101.04535 §4.2] [fig. 4]. - Intermediate models: only slightly above static poisoning with the final model [arxiv:2101.04535 §4.3] [fig. 5]. - Adaptive poisoning, Purchase, target epsilon 2: empirical bound 0.37, against 0.25 for static poisoning [arxiv:2101.04535 §4.4]. - Gradient attack: empirical bound 0.3 against a proven epsilon of 1, which the authors call almost tight [arxiv:2101.04535 §4.5] [fig. 7]. - Dataset attack, malicious dataset: empirical bound 3.6 against a proven epsilon of 4 [arxiv:2101.04535 §4.6] [fig. 8]. This setting sets the learning rate to zero and labels the data with the initial model's outputs [arxiv:2101.04535 §4.6]. How outputs become labels is not stated. ### Forget-set classifier (unlearning check) A membership-style test run after deletion, not an attack. Train a classifier to tell forgotten inputs from retained inputs and report its accuracy. Closer to 0.5 means the two sets are harder to separate [arxiv:2310.20150 §4.6]. Card: [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](/papers/2310.20150). - Setting: T5-base, 1% of the training data forgotten, on IMDB and SAMSum [arxiv:2310.20150 tab. 7]. - Before unlearning the original model scores 0.542 on IMDB and 0.510 on SAMSum [arxiv:2310.20150 tab. 7]. The check is close to chance even then. - IMDB after unlearning: 0.550 for re-training, 0.566 for the paper's method, 0.626 for Reverse-Gradient [arxiv:2310.20150 tab. 7]. - The paper concludes that the deleted data could not be easily inferred before or after unlearning [arxiv:2310.20150 §4.6]. The card does not give the classifier's architecture or training split [arxiv:2310.20150 §4.6]. ### Sentence-level shadow test on fine-tuned GPT-2 A shadow-model test applied to single sentences of the fine-tuning data, with and without defences. Card: [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539). - ROC AUC on the ECHR court-case data: 0.96 undefended, 0.82 with scrubbing, and 0.505 with DP and scrubbing [arxiv:2302.00539 §IV-H]. Values are author-reported. An AUC of 0.5 is chance level. - The number of shadow models, the attack classifier, and the GPT-2 size used for these values are not stated in the card [arxiv:2302.00539 §IV-H]. - The official repo has no code for this test [arxiv:2302.00539 §IV-H]. ## When to use which | Situation | Use | Why | |---|---|---| | Only black-box query access, a cheap test is enough | Shadow-model attack S [arxiv:2111.09679 §4.1] | One threshold per label, so it needs only shadow models. It is weaker on atypical records [arxiv:2111.09679 §4.1]. | | Black-box access, fixed low false-positive rate, and compute for hundreds of models | Reference-model attack R, or distillation attack D [arxiv:2111.09679 §5.2] | On Purchase100 they reach more than ten times the true-positive rate of S and P below a 0.2 false-positive rate. The cost is 999 models per target in the paper's Purchase100 setups [arxiv:2111.09679 app. B.2]. | | Only black-box query access to a model, and a lower bound on DP-SGD leakage | Loss-threshold test [arxiv:2101.04535 §4.1] | Needs no access to training. The empirical bound stays far below the proven one, so it shows a lower bound for this attack only. | | No data resembling the training data | Not shadow-model attacks | The survey says the similarity assumption is often unrealistic [arxiv:2402.00888 §5.2]. | | Measuring sentence-level leakage in a fine-tuned LM, with or without scrubbing or DP | Sentence-level shadow test [arxiv:2302.00539 §IV-H] | The only card that reports this attack on a language model with defences. AUC falls from 0.96 to 0.505 with DP and scrubbing, but the card gives no shadow setup and no code, so the numbers cannot be reproduced from it. | | Auditing a DP-SGD run where you control training data or gradients | Poisoning-assisted audits [arxiv:2101.04535 §4.4]-[§4.6] | These give the highest empirical bounds in the paper. The authors conclude the proven bound is tight in the worst case and loose on average [arxiv:2101.04535 §4.7] [app. B]. | | Choosing a defence | DP-SGD, with its cost in mind | The survey calls DP the most common and effective defence against gradient leakage [arxiv:2402.00888 §6.2]. It also names dropout, model stacking, adversarial regularisation, MemGuard, pruning and distillation [arxiv:2402.00888 §6.2]. On fine-tuned GPT-2 with eps 8, DP cuts extraction precision and recall to about 3% and keeps reconstruction at or below 1% [arxiv:2302.00539 §IV-E] [§IV-F]. The survey says DP can cut utility and that its effect on LLMs is not well evaluated [arxiv:2402.00888 §6.2] [§8]. No card gives a utility cost, because the PII paper's utility table is missing from its source [arxiv:2302.00539 tab. 6]. | | Reporting an empirical epsilon | Clopper-Pearson bounds on the error rates [arxiv:2101.04535 eq. 5] | The point estimate in [arxiv:2101.04535 eq. 4] is undefined when an error count is zero. With 1,000 trials, the empirical epsilon cannot exceed 5.60 at 95% confidence [arxiv:2101.04535 §3.3]. | | Need to recover training examples, not only test membership | Model inversion [arxiv:2402.04013 §1] | The survey says membership inference reveals only narrow aspects of training data, while model inversion can reconstruct samples. See [Model inversion](/concepts/model-inversion). | | Checking that unlearning removed a forget set from an LLM | Forget-set classifier, as supporting evidence only [arxiv:2310.20150 §4.6] | It is the only membership-style check tied to an unlearning step in these cards. Its IMDB value is 0.542 before unlearning, so it cannot separate a forgotten model from an unchanged one [arxiv:2310.20150 tab. 7]. The paper's primary measure is accuracy on the forget set [arxiv:2310.20150 §4.2]. | Caveat for all rows: an empirical bound or an AUC is a result for one attack. A weak attack gives a weak bound, and the audit paper says the same [arxiv:2101.04535 §4.1]. The survey's defence claims are not backed by measured utility trade-offs in these cards. ## Implementation sketch Common core of the black-box audit. The threshold is an input, since the audit paper does not state how it is chosen [arxiv:2101.04535 §4.1]. ```pseudocode # Empirical epsilon from repeated membership games [arxiv:2101.04535 §3.3] # Input: per-trial scores, threshold tau, delta, trial count # Output: lower confidence bound on epsilon def audit_epsilon(scores_in, scores_out, tau, delta): # 1. Count errors at threshold tau [arxiv:2101.04535 §4.1] fp = sum(s <= tau for s in scores_out) fn = sum(s > tau for s in scores_in) # 2. Use high-confidence (Clopper-Pearson) bounds on the rates # [arxiv:2101.04535 eq. 5] fp_rate = cp_bound(fp, len(scores_out)) fn_rate = cp_bound(fn, len(scores_in)) # 3. Apply the error-rate relation to get epsilon [arxiv:2101.04535 eq. 4] ratio_a = (1 - delta - fp_rate) / fn_rate ratio_b = (1 - delta - fn_rate) / fp_rate return max(log(ratio_a), log(ratio_b)) ``` Variants differ in how the scores are produced. The shadow-model attack builds the scores from an attack classifier trained on shadow outputs [arxiv:2402.00888 §5.2]. Attack R and attack D take the score from a per-record threshold, a smoothed alpha-percentile of reference or distilled losses [arxiv:2111.09679 §4.3] [§4.4] [app. B.3]. The paper says the empirical percentile is undefined when the false-positive rate is below one over the number of samples, so it smooths the losses first [arxiv:2111.09679 app. B.3]. The sentence-level test reports ROC AUC rather than an epsilon, and its scoring method is not stated in the card [arxiv:2302.00539 §IV-H]. The poisoning-assisted audits change the crafted example or the access to intermediate models [arxiv:2101.04535 tab. 1]. The forget-set check uses forgotten and retained inputs as its two classes, and its accuracy as the score [arxiv:2310.20150 §4.6]. ## Contradictions - **Defence ranking versus measured gap.** The survey calls DP the most common and effective defence against gradient leakage [arxiv:2402.00888 §6.2]. The audit paper finds a large gap between the proven and empirical epsilon in the black-box setting (4 against 0.31) and says the analysis is loose on average [arxiv:2101.04535 §4.1] [§4.7]. These claims measure different things, so this page states both and does not resolve them. - **Near-chance AUC versus epsilon bound.** The PII paper reports a sentence-level AUC of 0.505 with DP and scrubbing [arxiv:2302.00539 §IV-H]. The audit paper reports an empirical epsilon lower bound of 0.31 against a proven 4 [arxiv:2101.04535 §4.1]. One is a single attack's AUC on GPT-2, the other a lower bound on epsilon for DP-SGD. Neither bounds the leakage from all attacks, so this page states both and does not compare them directly. - **Attack ordering across datasets.** The enhanced-attack paper says the trend S < P < R < D holds across the Purchase100 setups [arxiv:2111.09679 app. C]. Its CIFAR tables do not show that ordering everywhere. On CIFAR100 setup II, D (0.936) is below S (0.944) and R (0.945), and on CIFAR10 setup I, D (0.871) is below R (0.874) [arxiv:2111.09679 tab. AUC CIFAR10] [tab. AUC CIFAR100]. The card's summary says the gains hold across all four datasets, which these tables do not support for D. This page uses the table values. - **Scope.** The survey says most attacks and defences were tested on small models, and only a few on LLMs [arxiv:2402.00888 §8]. The audit paper's numbers come from small image and tabular classifiers [arxiv:2101.04535 app. A]. The enhanced-attack paper's models are small MLPs, CNNs and a Wide ResNet of depth 28 [arxiv:2111.09679 app. B.1]. The PII paper's membership numbers come from fine-tuned GPT-2 models, and the card does not state the size used for them [arxiv:2302.00539 §IV-H]. The unlearning paper runs on T5-base and T5-3b, but its classifier is near chance before unlearning [arxiv:2310.20150 tab. 7]. No card reports a membership attack on a model larger than GPT-2-Large [arxiv:2302.00539 §IV-F]. ## Papers - [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888): taxonomy of LLM privacy and security attacks, with shadow-model membership inference described and defences listed; no experiments. - [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535): attack game for DP-SGD with black-box and poisoning-assisted variants, giving empirical epsilon lower bounds. - [Enhanced Membership Inference Attacks against Machine Learning Models](/papers/2111.09679): hypothesis-test framing of membership inference, with shadow, population, per-record reference-model and distillation attacks compared by AUC on small classifiers. - [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539): PII extraction, reconstruction and inference attacks on fine-tuned GPT-2, with a sentence-level shadow test reported with and without scrubbing and DP. - [Ethical and social risks of harm from Language Models](/papers/2112.04359): risk taxonomy with privacy loss from memorised training text as a related risk; no membership test. - [Survey of model inversion attacks and defences on DNNs](/papers/2402.04013): survey of reconstruction attacks; cited here only for its contrast with membership inference. - [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](/papers/2310.20150): adapter-based unlearning on T5, with a forget-set classifier reported as a post-deletion check.