Differential privacy

Overview

Differential privacy (DP) bounds how much the output of an analysis can reveal about any single record in its input. The bound is usually stated with a parameter ε (epsilon), often together with δ. Nine paper cards list the concept. Eight give DP content. The ninth, a membership inference paper, tags DP in its concept list but does not discuss it, so this page does not cite it. One of the eight, on DP-SGD, gives the update rule, the settings it used and an empirical audit of the bound 2101.04535 §2.3 app. A. For that variant in depth, see the DP-SGD concept page.

A second card uses the same kind of guarantee for deleting a training point. Its target is (ε, δ)-indistinguishability between a model after removal and a model retrained without the point 1911.03030 §2, eq. (1). It notes that DP implies this property, but DP training costs accuracy, so it treats removal as a point between full retraining and DP training 1911.03030 §2.

A third card uses the same (ε, δ) form for correcting a model after feature values or labels change. It bounds the distance between the corrected model and a retrained one, not the leakage of a training record 2108.11577 §6. Its variant is below.

The other cards use DP at a higher level and give no mechanism parameters. The trustworthy-AI survey lists DP under data preprocessing, next to data anonymization such as k-anonymity, and refers to ε-differential privacy without giving the mechanism or any value of ε 2110.01167 §3.1.2. Its privacy aspect is about unauthorized use of data that can identify a person, and it cites a study in which 15 demographic attributes made 99% of participants unique 2110.01167 §2.6.

The LLM security survey treats DP as a defence. It names DP-SGD against gradient leakage and calls DP the most common and effective defence against gradient leakage in DNNs, without a number 2402.00888 §6.2. The vertical federated learning survey notes that some tree-based methods use DP 2211.12814 §2.2.1. The federated learning survey describes DP for federated aggregation as clipping the model parameters and adding noise before aggregating, and gives no values for it 2101.05428 §6.1.2.

Variants

DP-SGD (noisy gradient training)

Each step clips every per-example gradient to L2 norm at most b, averages the clipped gradients over the minibatch, and adds Gaussian noise Z with covariance σ²I 2101.04535 §2.3 eq. 3. The text ties the noise scale to b, but eq. 3 writes the noise without b; see Contradictions. The paper's experiments use a 4-layer CNN on MNIST and CIFAR-10 and a 3-layer fully connected network on Purchase app. A. Settings:

• Clipping norm b: 1.0 (MNIST, CIFAR-10) and 0.05 (Purchase).
• Learning rate: 0.15 (MNIST, CIFAR-10) and 1.1 (Purchase).
• Noise multipliers 0.5, 0.6, 0.9 and 1.0 were tried, and the one with the best practical ε was kept app. A.
• Target ε of 1, 2, 4 and 10, with δ = 1e-5 app. A app. B.

Batch size, epochs and the threshold of the audit are not stated in the card's source. Accuracy at ε = 1, 2, 4 and 10 was 95, 96, 97 and 98.5% on MNIST, 53, 55, 56 and 59% on CIFAR-10, and 37, 82, 88 and 90% on Purchase app. A. These are small models on three benchmarks, not large language models.

The LLM survey lists DP-SGD among the defences against gradient leakage. It names clipping, but the noise scale, clipping norm and ε are not stated 2402.00888 §6.2. Cards: Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning and Security and Privacy Challenges of Large Language Models: A Survey.

Clip and noise on federated parameters

The FL survey describes one DP defence against membership inference in federated learning. Clip the model parameters, add noise, then aggregate. It says this costs some accuracy 2101.05428 §6.1.2. The clip bound, noise scale and privacy budget are not stated, and the survey's text clips parameters, where DP-SGD clips per-example gradients 2101.04535 §2.3. Its backdoor section also lists weak DP as a defence, next to norm thresholding 2101.05428 §6.4.1. Card: Federated Learning: Opportunities and Challenges.

Auditing a DP-SGD bound empirically

The paper treats the DP guarantee as an adversary game. A crafter builds two datasets that differ by one example, a trainer picks one of them at random and runs DP-SGD, and a distinguisher guesses which one was used [arxiv:2101.04535 §3.2]. Many runs give false-positive and false-negative rates, and these are converted into a lower confidence bound on ε with Clopper-Pearson intervals at 95% 2101.04535 §3.3 eq. 5. The result is a lower bound for the attack that was tested, not a proof of privacy 2101.04535 §4.1.

The attacker's access sets how close the measured bound gets to the proven one 2101.04535 tab. 1:

With 1,000 trials at 95% confidence, no attack can report an empirical ε above 5.60 2101.04535 §3.3. Card: Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning.

Certified removal (noise plus one Newton step)

For L2-regularised convex models, the paper trains with a random linear term in the loss. Removing a training point is one Newton step, which leaves a small gradient residual. The noise from training hides that residual, so removal is (ε, δ)-certified; with Gaussian noise, δ = 1.5·e^(−c²/2) [arxiv:1911.03030 §3.1 thm. 3]. Each removal adds to a running bound, and the model is retrained once the budget σε/c is spent 1911.03030 §3.2 alg. 2. For least squares the update is exact, with ε = 0 1911.03030 §3.2. Non-convex losses are out of scope 1911.03030 §6.

The paper's experiments use linear models over fixed feature extractors. Its timings are author-reported: removal takes 0.04 to 0.48 s on the linear tasks, against 15.6 to 124 s of training 1911.03030 tab. 1. Card: Certified Data Removal from Machine Learning Models.

Certified feature and label unlearning (second-order update)

This variant changes values shared by many rows, instead of deleting whole points. The second-order update applies an inverse Hessian to the gradient difference between corrected and original points 2108.11577 §5.2 eq. 8. For strongly convex, L2-regularised losses, the paper adds a noise term to the training objective, so the corrected model and a retrained model are close in the (ε, δ) sense 2108.11577 §4 §6. The Gaussian scale is σ = β·c/ε with c = √(2·log(1.5/δ)) §7.1, eq. 9. β bounds the gradient residual. The paper estimates it from 100 sampled feature combinations, so the certificate holds for that estimate and not for every possible change §7.1. The logistic regression experiments use ε = 0.1 and δ = 0.01, which gives c of about 3.16 §7.1. The guarantee needs inputs with norm at most 1 and Lipschitz gradients, and the paper does not say whether its datasets meet the input condition §6, thm. 1.

Timings are author-reported. On a Drebin logistic regression task with 100 combinations of 100 features, retraining took 6.82 s, the first-order update 0.02 s and the second-order update 0.63 s tab. 2. The speed-up factors in the text do not match these runtimes, so compare the runtimes §7.1. The paper's neural networks have no certificate. For a character-level LSTM, the table shows an exposure of 0 after the first- and second-order updates, while fine-tuning leaves 31 to 57 tab. 3 §7.2. For the wider picture, see the machine unlearning concept page. Card: Machine Unlearning of Features and Labels.

Noise on exchanged gradients

The VFL survey groups non-cryptographic privacy defences, such as adding noise, gradient discretisation and sparsification, and says they trade utility for privacy 2211.12814 §5.3.4. The card does not say that these are DP mechanisms, so do not treat them as DP without a source that does. Card: Vertical Federated Learning: Concepts, Advances and Challenges.

DP in tree-based vertical learning

SecureBoost and its successors use additive homomorphic encryption on gradients and histograms. Some GBDT methods use DP instead or as well, and random-forest methods use bagging with encryption or a third party 2211.12814 §2.2.1. The card does not describe the DP settings used. Card: Vertical Federated Learning: Concepts, Advances and Challenges.

Output noise against model inversion

Model inversion rebuilds training-like inputs from a released model. The model inversion survey lists DP among the output-side defences: an exponential mechanism and Laplacian noise, both added to what the model returns 2402.04013 §6.1. The same section also covers adversarial perturbation of confidences and purification. The survey reports no new benchmark numbers, so the card records no measured privacy or utility effect for these defences. Card: model inversion survey.

DP against PII leakage in language models

The LLM survey lists DP among defences against PII leakage, next to de-duplication and curation, PII masking and API hardening [arxiv:2402.00888 §6.2]. It reports one study in which about 3% of PII sequences still leaked under sentence-level DP 2402.00888 §5.3. Card: Security and Privacy Challenges of Large Language Models: A Survey.

When to use which

The cards support coarse choices. The accuracy figures come from three small benchmarks, so they show a trade-off, not a rule for other models.

Implementation sketch

Our pseudocode of one DP-SGD step, following 2101.04535 eq. 3. Check the noise scale before use: the text ties it to b, but eq. 3 does not.

# One DP-SGD step on minibatch B [arxiv:2101.04535 §2.3] [eq. 3]
def dpsgd_step(theta, batch, eta, sigma, b):
    # Clip each per-example gradient to L2 norm at most b [§2.3]
    clipped = [clip(grad(theta, x, y), b) for (x, y) in batch]
    # Average the clipped gradients over the batch [eq. 3]
    mean = sum(clipped) / len(batch)
    # Gaussian noise with std sigma per eq. 3; §2.3 says sigma scales with b
    noise = normal(0, sigma)
    return theta - eta * (mean + noise)

The audit needs the error-rate bound from 2101.04535 §3.3 and eq. 5. The card does not give a validated implementation, so take the formula from the paper before coding it. Not stated: the threshold used by the distinguisher 2101.04535 §4.1.

Contradictions

• Noise scale in DP-SGD. The paper says the noise scale is proportional to b 2101.04535 §2.3, but its update rule writes Z ~ N(0, σ²I) without b eq. 3. The text does not resolve this, and the implementation must pick one.
• Defence strength. The LLM survey calls DP the most common and effective defence against gradient leakage 2402.00888 §6.2. The audit finds that a black-box attacker recovers 0.31 of a proven ε of 4, while a gradient-level attacker reaches 3.6 2101.04535 §4.1 §4.6. These use different threat models, but the survey's claim is unquantified, so neither side settles the other.
• Large ε in practice. The paper says practitioners often train with large ε in the hope that real leakage is smaller than the proof allows, and it cites work arguing large ε can still be safe 2101.04535 §3.2 §5. The audit shows that measured leakage depends on the attacker, not on ε alone 2101.04535 §4.6.
• Utility cost. The LLM survey says DP can cut model utility and that its effect on LLMs is not well evaluated 2402.00888 §6.2 §8. The VFL survey says privacy defences trade utility 2211.12814 §5.3.4, and the FL survey says DP costs some accuracy 2101.05428 §6.1.2. None of these gives a measured trade-off for LLMs.
• Mechanism wording. The FL survey's introduction calls DP a "cryptographic technique" 2101.05428 §1. Its §6.1.2 describes DP as noise added to clipped parameters. The card directs readers to §6.1.2; this page does not settle the wording.
• Noise parameter convention (unlearning). The unlearning paper calls the noise parameter in eq. 9 a variance 2108.11577 §7.1, while the code passes it to a normal sampler as a standard deviation [code: Unlearner/DPLRUnlearner.py#L28]. Check the convention before calibrating noise.
• Speed-up figures (unlearning). The text reports a 90x first-order speed-up on the Drebin task. The runtimes in the same table, 6.82 s against 0.02 s, imply about 341x tab. 2 2108.11577 §7.1. This page cites the runtimes only.

Papers

• Certified Data Removal from Machine Learning Models: one-Newton-step removal for convex linear models, hidden by training noise, with a retraining budget.
• Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning: DP-SGD settings and an attack game that measures empirical ε under several attacker powers.
• Machine Unlearning of Features and Labels: closed-form first- and second-order updates that correct feature values and labels, with an (ε, δ) closeness bound for strongly convex losses.
• Trustworthy AI: From Principles to Practices: a survey that lists differential privacy under data preprocessing and refers to ε-DP without specifying it.
• Federated Learning: Opportunities and Challenges: an FL survey with no experiments that describes clip-and-noise DP for aggregation and weak DP against backdoors, without values.
• Vertical Federated Learning: Concepts, Advances and Challenges: a survey of VFL that notes DP in some tree-based methods and treats noise on gradients as a privacy defence with a utility cost.
• Security and Privacy Challenges of Large Language Models: A Survey: a survey that names DP-SGD and DP as defences against gradient leakage, membership inference and PII leakage.
• Model inversion survey: a survey of model inversion that lists DP exponential-mechanism and Laplacian output noise among its defences, without numbers.