Enhanced Membership Inference Attacks against Machine Learning Models

In plain words

A machine learning model can leak whether a specific example was in its training data. A membership inference attack tries to guess that from the model's behaviour alone. This paper argues that many earlier attacks are hard to compare, because each one quietly assumes a different amount of background knowledge. It sets up one test for all of them: is the model's loss on this example unusually low compared with what it would be if the example had been left out?

The better attacks estimate that "left out" loss more precisely. Some use other models trained on random data, some train models on the same data minus the example, and some use models that imitate the target. The authors also set the decision threshold from a chosen false-positive rate, so an auditor knows the error level in advance. Their main finding is that attacks which reduce the attacker's uncertainty about the target model and the target example find more members, and find the riskiest examples more reliably.

TL;DR

Membership inference is a binary hypothesis test on a loss value. Four black-box attacks (S, P, R, D) differ only in how their threshold depends on the target model and the target record. Thresholds come from a percentile at a chosen false-positive rate, smoothed when that rate is below one over the number of samples. Author-reported AUC gains over the shadow-model attack S are largest for attacks R and D, and the gains hold across Purchase100, CIFAR10, CIFAR100 and MNIST, though they are small on MNIST.

Problem

Shadow-model attacks §1 pick one threshold for all targets, so they are most accurate for typical members of typical models and weaker on atypical examples §4.1. Prior work mixed average-case and worst-case claims, and reported accuracy or sweeps of a threshold instead of a rule that fixes the false-positive rate in advance §1, §3, §4. The authors want a consistent game in which every attack's assumptions are explicit, and attacks that give confident results at low false-positive rates §3, §4.

Method

Game and hypotheses. The challenger draws a training set D from the population π, trains a model θ, and then either includes or excludes a target record z. The adversary sees θ and z and guesses which case held §3, def. game_random. Four game variants fix more of the randomness: fixed model (Definition "fixed model", used by P), fixed record (used by R), and fixed record and dataset (used by D) §3, §4 app. A. Table of correspondence: S uses the general game, P the fixed-model game, R the fixed-record game, D the fixed record-and-dataset game [§4, tab. threshold summary].

Test form. Reject H0 (report "member") when the loss is at most a threshold: loss(θ, x_z, y_z) ≤ c_α(·), where the argument of c_α is what the attack conditions on §4, eq. general_strategy. The paper derives this rule as an approximation to a likelihood-ratio test, under assumptions stated in Appendix A, and does not claim optimality §4 app. A.

Threshold rule. For a target false-positive rate α, c_α is the α-percentile of the losses on a set of "out" samples (non-members in the relevant out-world) §4. When α is smaller than one over the number of samples, the empirical percentile is undefined, so the paper smooths the loss distribution before taking the percentile app. B.3.

Attacks.

• Attack S (shadow). Train shadow models on random population subsets. Take losses of shadow models on population records with the same label as the target. One threshold per label, c_α(y_z) §4.1.
• Attack P (population, per model). For a fixed target model θ0, take losses of θ0 on population records. One threshold per model, c_α(θ0) §4.2.
• Attack R (reference models, per record). Train reference models on random population subsets. Take losses of these models on the fixed target record z0. One threshold per record, c_α(x_z, y_z) §4.3.
• Attack D (distillation, per model and record). Train distilled models on random population subsets labelled by the target model θ0. Take their losses on z0. The threshold depends on θ0 and z0 together, c_α(θ0, x_z, y_z) §4.4. The paper's aim is to approximate leave-one-out retraining without the knowledge of the other n-1 training records that leave-one-out needs §4.4, §4.5.
• Attack L (idealised leave-one-out). Train reference models on the target's training set minus z. It needs the exact n-1 records, so the paper uses it only as an upper reference §4.5.

# Attack R for one target model and one target record  [§4.3]
# Input:  target_loss   = loss of target model on record z0
#         pool          = population data pool
#         n_reference   = number of reference models (Purchase100: 999)
#         alpha         = target false-positive rate
# Output: True if the record is predicted to be a member

def attack_r(target_loss, z0, pool, n_reference, alpha):
    # Train reference models on random population subsets  [§4.3]
    reference_losses = []
    for i in range(n_reference):
        subset = sample_iid(pool)              # population sample  [§4.3]
        model = train_same_algorithm(subset)   # same arch and training [app. B.2]
        reference_losses.append(loss(model, z0))
    # Threshold: smoothed alpha-percentile of reference losses  [§4.3] [app. B.3]
    threshold = smoothed_percentile(reference_losses, alpha)
    return target_loss <= threshold

# Attack D for one target model theta0 and record z0  [§4.4]
# Input:  target_model, z0, pool, n_distilled, alpha
# Output: True if the record is predicted to be a member

def attack_d(target_model, target_loss, z0, pool, n_distilled, alpha):
    # Distilled models: trained on random population data labelled by theta0
    distilled_losses = []
    for i in range(n_distilled):
        subset = sample_iid(pool)                      # [§4.4]
        soft_labels = target_model.predict(subset)     # [§4.4]
        model = train_same_algorithm(subset, soft_labels)  # [§4.4]
        distilled_losses.append(loss(model, z0))
    threshold = smoothed_percentile(distilled_losses, alpha)  # [§4.4]
    return target_loss <= threshold

Smoothing (the four methods). The paper compares four ways to compute the percentile from N+1 sorted losses l_0 ≤ … ≤ l_N, and reports the best one per setup app. B.3:

1. Linear interpolation between order statistics: p(α) = l_k · (k + 1 − αN) + (αN − k) · l_{k+1}, where k = ⌊αN⌋ app. B.3.
2. Logit rescaling: fit a Gaussian to the logit-transformed losses log(e^{-l} / (1 − e^{-l})), then invert at the (1 − α) point app. B.3.
3. Minimum of the two thresholds (the conservative choice) app. B.3.
4. Average of the two confidence functions, then inverted app. B.3.

Attack loss. The paper's signal is a loss ℓ(θ, x, y), and the appendix treats it as non-negative app. A. The paper does not give a closed formula for ℓ in the main text, but its logit formula treats e^{-ℓ} as a probability app. B.3. See the pitfalls on this.

Hyperparameters

Values as stated in the paper. "not stated" means the paper text read for this card does not give it.

Common to all setups:

• Loss: categorical cross-entropy app. B.1.
• Learning rate, batch size, epochs, weight decay for the Purchase100 and CIFAR10 I–II setups: not stated in the text read. The code defaults are different (see Paper vs code).
• Shadow, reference and distilled models use the same architecture, training algorithm and dataset size as the target app. B.2.
• Training subsets are i.i.d. sub-samples of the data, possibly overlapping app. B.2.
• Smoothing method: one of four, chosen per setup by attack performance. The chosen method for each setup is not stated in the text read app. B.3.
• Target false-positive rates in the tables: 0.05 and 0.3 §5.3 [tab. agreement].

Results

Author-reported. Setup: target models above. Metric: AUC of the ROC curve over random members and non-members of random target models §5.2 [tab. AUC of each dataset appendix table].

Purchase100 (appendix table, AUC, mean ± std over runs). Train and test accuracy are given in the same row.

Standard deviations in this table range from 0.002 to 0.061 [tab. AUC Purchase100]. The paper says the trend S < P < R < D holds across Purchase100 setups app. C.

CIFAR10 (appendix table, AUC).

CIFAR100 (appendix table, AUC).

MNIST (appendix table, AUC).

Other headline claims, in the text:

• Under a fixed low false-positive rate (below 0.2), R and D reach more than 10 times the true-positive rate of S and P on the Purchase100 setup §5.2. R is slightly better than D at the smallest false-positive rates §5.2.
• At a fixed high true-positive rate, D has a lower false-positive rate than S, P and R §5.2.
• On the records that the strongest attacks flag as most vulnerable, the reported AUC of the strongest attack is 0.984 §1.
• Against LiRA (Carlini et al.) on CIFAR10 setup IV, R and D have higher AUC than the three LiRA variants. The curves are in a figure not transcribed here §5.4 app. C. The paper states its cost for R as 2·n models, against n OUT and n IN models per attacked point for LiRA §5.4.
• Agreement on Purchase100 setup II (train records, false-positive rate 0.3): D and the idealised leave-one-out Attack L agree on 0.804 of records. S and P agree on 0.892 §5.3 tab. agreement.
• Defence: gradient clipping (setup IV) lowers all attacks, but the ordering S < P < R < D persists app. D.

Paper vs code

Which code. The paper's footnote points to a research/2022_enhanced_mia folder on the master branch of the Privacy Meter repository §5. That folder no longer exists at the repository's HEAD (e384af8). research/README.md in the repository links the same work at the pinned commit 295e7e37e889e12df4083b812f71ed2e2ddd8b4a, and that is the commit this card uses. The folder was read through WebFetch of the raw files at that commit, not from a clone. Line numbers below come from those reads and may be off by a few lines; two reads of the same function gave 161 and 165. The repository's MIT license was checked at the pinned commit. Files inference.py, score_losses.py, preprocess_for_privacymeter.py and plot_attack_via_reference_or_distill.py carry Apache-2.0 headers from Google LLC, marked as modified. No GitHub name search was needed: the paper link and the research README identify the repository and commit.

Scope. The folder contains CIFAR-style training (train.py) and the reference and distillation pipeline. It does not contain the Purchase100 MLP: the network(arch) function lists ConvNet, WideResNet and VGG19 only train.py. The Purchase100 and Attack S and P numbers are therefore not reproduced by this folder alone. The README mentions a LiRA baseline in scripts/ and score.py, which were not read.

Discrepancies to check:

• Reference and distilled counts. The paper uses 999 reference or distilled models for Purchase100 setups I–III and 29 for setup IV [app. B.2]. The README sets 29 reference models for Attack R and 29 distilled models for Attack D [README, "Attack R" and "Attack D" sections].
• Loss definition. The paper's logit formula treats e^{-ℓ} as a probability app. B.3. The code, as read, sets the stored loss to exp(-p_true) from softmax probabilities [code: research/2022_enhanced_mia/score_losses.py#L42-L57]. The plot code then applies the logit transform to exp(-stored loss) [code: research/2022_enhanced_mia/plot_attack_via_reference_or_distill.py#L84-L95]. Read literally, these two forms are not the same. Verify before implementing.
• Soft labels. Attack D's distilled labels are "the target model's output" §4.4. The code reads stored predictions of the target model from a file and passes them to a cross-entropy-on-logits loss, with no softmax or temperature [code: research/2022_enhanced_mia/distill_softlabel.py#L217-L242]. Records without a stored prediction get all-zero labels [code: research/2022_enhanced_mia/distill_softlabel.py#L218-L220]. The paper says nothing about either point.
• Minimum of thresholds. The paper's fourth method is the minimum of the two thresholds app. B.3. The code takes the maximum of the two alpha-scores, with a comment that this is because it works on alpha rather than thresholds [code: research/2022_enhanced_mia/plot_attack_via_reference_or_distill.py#L161-L165]. On a monotone threshold these look equivalent, but this is an inference from reading the code.
• Reference membership. The paper's Attack R uses reference models trained on random population data §4.3. The introduction says these exclude the target record §2. The code keeps a per-model keep mask, drawn with a default keep probability of 0.5 [code: research/2022_enhanced_mia/train.py#L223], and preprocessing stores reference member and non-member losses separately [code: research/2022_enhanced_mia/preprocess_for_privacymeter.py#L68-L73]. Which group the threshold uses was not confirmed.
• Defaults vs setups. Code defaults are Adam with eps 1e-7, learning rate 0.001 held constant, batch size 32, 501 epochs, weight decay 0.0005, architecture cnn32-3-mean [code: research/2022_enhanced_mia/train.py#L312-L317] [code: research/2022_enhanced_mia/train.py#L126]. CIFAR10 setup III in the paper uses L2 λ = 0.001 app. B.1. These are defaults, not the paper's per-setup values.
• Target-training size. Appendix B says Purchase100 setup II uses 10000 points, while the AUC table caption says 5000 app. B.1 [tab. AUC Purchase100]. The text does not resolve this.
• Quantities the paper leaves out. Learning rate, batch size and epochs for the Purchase100 MLP, and the smoothing method for each setup, are not stated in the text read. The code's choices above should not be read as the paper's.

Pitfalls

• Loss scale. The threshold rule depends on the exact form of ℓ. Pick one form, check it against the logit formula in the appendix, and test it on a toy model. See the loss discrepancy above.
• Reference and distilled cost. The paper's Purchase100 setups use 999 models per target app. B.2. Whether distilled models are shared across records z0 is not stated. The text reads as if each attacked record needs its own distilled set. Check this before estimating compute.
• Low false-positive rates need smoothing. Empirical percentiles are undefined when α is below 1/(N+1). The reported low-FPR results depend on the smoothing method chosen per setup, and that choice is not listed in the text read app. B.3.
• Population must match the training distribution. The method assumes the adversary's population samples have the same distribution as the training data §3. Mismatch can overstate or understate leakage.
• Attack D is not a leave-one-out oracle. The paper states that D approximates retraining without the n-1 records, and that approximation error may explain its lower true-positive rate at very small false-positive rates §5.2.
• Strong, not guaranteed. The likelihood-ratio derivation rests on approximations and assumptions in Appendix A, and the paper says the rule is not an optimality guarantee §4.
• Defences. Results are shown for gradient clipping only app. D. The paper says the attack method makes no assumption about the training algorithm, but it tests one defence.

Lineage

• Builds on: shadow-model attacks (Shokri et al., 2017), which Attack S reproduces §4.1. No card exists for that paper, so nothing from it is used here.
• Compared against: LiRA (Carlini et al., 2022), reference-model attacks that Attack R is similar to §4.3 §5.4. No card exists for that paper either.
• Leave-one-out analysis and the idealised attack L are the reference point for D §4.5.