Membership inference

Overview

Membership inference asks whether a particular example was part of a model's training data. The survey lists it as one of the privacy attack families for language models, next to gradient leakage and PII leakage 2402.00888 §3 fig. 4. The DP audit paper uses a membership-style test as its black-box baseline and also as the building block of stronger tests [arxiv:2101.04535 §4.1]. Its distinguisher guesses which of two training sets, differing by one example, produced the model 2101.04535 §3.2.

The enhanced-attack paper frames the test as a hypothesis test on the loss. It compares four black-box attacks that differ in how much they know about the target model and record, all on small classifiers 2111.09679 §3, §4.

The cards differ in scope. The survey describes attacks and defences but runs no experiments 2402.00888 §1. The audit paper measures attacks on small classifiers: a 4-layer CNN on MNIST and CIFAR-10 and a 3-layer fully connected network on Purchase 2101.04535 app. A. The PII leakage paper is the only card that measures a membership test on a language model with defences. It runs a sentence-level shadow-model test on fine-tuned GPT-2 2302.00539 §IV-H. The unlearning paper uses a membership-style classifier as a post-deletion check on T5 language models [arxiv:2310.20150 §4.6]. That check is not an attack on training records, and the paper reports it near chance even before unlearning.

Two other cards sit next to this topic. The model inversion survey contrasts the two: membership inference reveals narrow aspects of the training data, while model inversion can reconstruct training samples 2402.04013 §1. The ethics taxonomy lists privacy loss from memorised training text as an information hazard 2112.04359 §2.2. It describes no membership test, so it is a related risk here, not a source of attack methods.

Variants

Shadow-model attack

Train several shadow models on data similar to the target's training data. Collect their outputs on known members and known non-members, label them, and fit an attack classifier per label 2402.00888 §5.2. Card: Security and Privacy Challenges of Large Language Models: A Survey.

• The survey says shadow data must be disjoint from the target's training data but similar to it, and calls that assumption unrealistic in many cases 2402.00888 §5.2.
• Number of shadow models and the attack classifier type are not stated 2402.00888 §5.2.
• The survey names SaMIA as a later variant, tested only on WikiMIA and only on classification 2402.00888 §5.2.
• The enhanced-attack paper uses the same idea as its baseline, attack S. It takes one threshold per label from shadow losses on population records with the target's label 2111.09679 §4.1. Card: Enhanced Membership Inference Attacks against Machine Learning Models.

Per-record reference-model and distillation attacks

These attacks give each record its own threshold, so they adapt to atypical examples. Attack R trains reference models on random population subsets and sets the threshold from their losses on the target record [arxiv:2111.09679 §4.3]. Attack D trains models distilled from the target's own outputs on random population data, and sets the threshold from their losses on the record 2111.09679 §4.4. Attack L, which trains on the target's training set minus the record, is an idealised upper reference that needs the exact n-1 records 2111.09679 §4.5. Card: Enhanced Membership Inference Attacks against Machine Learning Models.

• Author-reported AUC on Purchase100, setup II: S 0.754, P 0.756, R 0.808, D 0.831 2111.09679 tab. AUC Purchase100. Setup I gives S 0.800 and D 0.837 2111.09679 tab. AUC Purchase100.
• At a fixed false-positive rate below 0.2, R and D reach more than ten times the true-positive rate of S and P on the Purchase100 setup [arxiv:2111.09679 §5.2].
• Reference models: 999 per target for Purchase100 setups I to III, and 29 for setup IV, per the paper 2111.09679 app. B.2. The card records that the public code's README sets 29 for R and D, so this count differs from the paper's [card: Paper vs code].
• Whether distilled models are shared across attacked records is not stated. The card reads the paper as needing one set per record, so check compute before planning [card: Pitfalls].
• Only gradient clipping is tested as a defence. The ordering S < P < R < D persists under it 2111.09679 app. D.

Black-box loss-threshold test

The distinguisher only sees the trained model. It computes the loss on the crafted example and guesses that the example was present when the loss is at most a threshold tau 2101.04535 §4.1. Card: Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning.

• The threshold is not stated 2101.04535 §4.1.
• Reported result: CIFAR-10, 4-layer CNN, target epsilon 4 with delta 1e-5. The proven epsilon is 4 and the empirical lower bound is 0.31 [arxiv:2101.04535 §4.1]. The authors describe this as far below the proven value.

Poisoning-assisted audits

The same game, with more attacker power. The crafter inserts a poisoned example, the distinguisher sees intermediate models, or the crafter controls gradients 2101.04535 tab. 1. Card: Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning.

• Static poison, final model: higher than black-box access on all three datasets. Given only in a figure 2101.04535 §4.2 fig. 4.
• Intermediate models: only slightly above static poisoning with the final model 2101.04535 §4.3 fig. 5.
• Adaptive poisoning, Purchase, target epsilon 2: empirical bound 0.37, against 0.25 for static poisoning 2101.04535 §4.4.
• Gradient attack: empirical bound 0.3 against a proven epsilon of 1, which the authors call almost tight 2101.04535 §4.5 fig. 7.
• Dataset attack, malicious dataset: empirical bound 3.6 against a proven epsilon of 4 2101.04535 §4.6 fig. 8. This setting sets the learning rate to zero and labels the data with the initial model's outputs 2101.04535 §4.6. How outputs become labels is not stated.

Forget-set classifier (unlearning check)

A membership-style test run after deletion, not an attack. Train a classifier to tell forgotten inputs from retained inputs and report its accuracy. Closer to 0.5 means the two sets are harder to separate 2310.20150 §4.6. Card: Unlearn What You Want to Forget: Efficient Unlearning for LLMs.

• Setting: T5-base, 1% of the training data forgotten, on IMDB and SAMSum 2310.20150 tab. 7.
• Before unlearning the original model scores 0.542 on IMDB and 0.510 on SAMSum 2310.20150 tab. 7. The check is close to chance even then.
• IMDB after unlearning: 0.550 for re-training, 0.566 for the paper's method, 0.626 for Reverse-Gradient 2310.20150 tab. 7.
• The paper concludes that the deleted data could not be easily inferred before or after unlearning 2310.20150 §4.6. The card does not give the classifier's architecture or training split 2310.20150 §4.6.

Sentence-level shadow test on fine-tuned GPT-2

A shadow-model test applied to single sentences of the fine-tuning data, with and without defences. Card: Analyzing Leakage of Personally Identifiable Information in Language Models.

• ROC AUC on the ECHR court-case data: 0.96 undefended, 0.82 with scrubbing, and 0.505 with DP and scrubbing 2302.00539 §IV-H. Values are author-reported. An AUC of 0.5 is chance level.
• The number of shadow models, the attack classifier, and the GPT-2 size used for these values are not stated in the card 2302.00539 §IV-H.
• The official repo has no code for this test 2302.00539 §IV-H.

When to use which

Caveat for all rows: an empirical bound or an AUC is a result for one attack. A weak attack gives a weak bound, and the audit paper says the same [arxiv:2101.04535 §4.1]. The survey's defence claims are not backed by measured utility trade-offs in these cards.

Implementation sketch

Common core of the black-box audit. The threshold is an input, since the audit paper does not state how it is chosen 2101.04535 §4.1.

# Empirical epsilon from repeated membership games  [arxiv:2101.04535 §3.3]
# Input:  per-trial scores, threshold tau, delta, trial count
# Output: lower confidence bound on epsilon

def audit_epsilon(scores_in, scores_out, tau, delta):
    # 1. Count errors at threshold tau  [arxiv:2101.04535 §4.1]
    fp = sum(s <= tau for s in scores_out)
    fn = sum(s > tau for s in scores_in)
    # 2. Use high-confidence (Clopper-Pearson) bounds on the rates
    #    [arxiv:2101.04535 eq. 5]
    fp_rate = cp_bound(fp, len(scores_out))
    fn_rate = cp_bound(fn, len(scores_in))
    # 3. Apply the error-rate relation to get epsilon  [arxiv:2101.04535 eq. 4]
    ratio_a = (1 - delta - fp_rate) / fn_rate
    ratio_b = (1 - delta - fn_rate) / fp_rate
    return max(log(ratio_a), log(ratio_b))

Variants differ in how the scores are produced. The shadow-model attack builds the scores from an attack classifier trained on shadow outputs [arxiv:2402.00888 §5.2]. Attack R and attack D take the score from a per-record threshold, a smoothed alpha-percentile of reference or distilled losses [arxiv:2111.09679 §4.3] §4.4 app. B.3. The paper says the empirical percentile is undefined when the false-positive rate is below one over the number of samples, so it smooths the losses first 2111.09679 app. B.3. The sentence-level test reports ROC AUC rather than an epsilon, and its scoring method is not stated in the card 2302.00539 §IV-H. The poisoning-assisted audits change the crafted example or the access to intermediate models 2101.04535 tab. 1. The forget-set check uses forgotten and retained inputs as its two classes, and its accuracy as the score 2310.20150 §4.6.

Contradictions

• Defence ranking versus measured gap. The survey calls DP the most common and effective defence against gradient leakage 2402.00888 §6.2. The audit paper finds a large gap between the proven and empirical epsilon in the black-box setting (4 against 0.31) and says the analysis is loose on average 2101.04535 §4.1 §4.7. These claims measure different things, so this page states both and does not resolve them.
• Near-chance AUC versus epsilon bound. The PII paper reports a sentence-level AUC of 0.505 with DP and scrubbing 2302.00539 §IV-H. The audit paper reports an empirical epsilon lower bound of 0.31 against a proven 4 2101.04535 §4.1. One is a single attack's AUC on GPT-2, the other a lower bound on epsilon for DP-SGD. Neither bounds the leakage from all attacks, so this page states both and does not compare them directly.
• Attack ordering across datasets. The enhanced-attack paper says the trend S < P < R < D holds across the Purchase100 setups 2111.09679 app. C. Its CIFAR tables do not show that ordering everywhere. On CIFAR100 setup II, D (0.936) is below S (0.944) and R (0.945), and on CIFAR10 setup I, D (0.871) is below R (0.874) 2111.09679 tab. AUC CIFAR10 tab. AUC CIFAR100. The card's summary says the gains hold across all four datasets, which these tables do not support for D. This page uses the table values.
• Scope. The survey says most attacks and defences were tested on small models, and only a few on LLMs 2402.00888 §8. The audit paper's numbers come from small image and tabular classifiers [arxiv:2101.04535 app. A]. The enhanced-attack paper's models are small MLPs, CNNs and a Wide ResNet of depth 28 2111.09679 app. B.1. The PII paper's membership numbers come from fine-tuned GPT-2 models, and the card does not state the size used for them 2302.00539 §IV-H. The unlearning paper runs on T5-base and T5-3b, but its classifier is near chance before unlearning [arxiv:2310.20150 tab. 7]. No card reports a membership attack on a model larger than GPT-2-Large 2302.00539 §IV-F.

Papers

• Security and Privacy Challenges of Large Language Models: A Survey: taxonomy of LLM privacy and security attacks, with shadow-model membership inference described and defences listed; no experiments.
• Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning: attack game for DP-SGD with black-box and poisoning-assisted variants, giving empirical epsilon lower bounds.
• Enhanced Membership Inference Attacks against Machine Learning Models: hypothesis-test framing of membership inference, with shadow, population, per-record reference-model and distillation attacks compared by AUC on small classifiers.
• Analyzing Leakage of Personally Identifiable Information in Language Models: PII extraction, reconstruction and inference attacks on fine-tuned GPT-2, with a sentence-level shadow test reported with and without scrubbing and DP.
• Ethical and social risks of harm from Language Models: risk taxonomy with privacy loss from memorised training text as a related risk; no membership test.
• Survey of model inversion attacks and defences on DNNs: survey of reconstruction attacks; cited here only for its contrast with membership inference.
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs: adapter-based unlearning on T5, with a forget-set classifier reported as a post-deletion check.