--- kind: concept slug: differential-privacy title: Differential privacy summary: >- Differential privacy (DP) bounds how much an analysis's output can reveal about any one record, stated as epsilon. DP-SGD (clipped, noised gradients) is the main training variant with reported settings. An audit shows measured leakage depends on attacker access and sits far below the proven bound for black-box attackers. tags: [privacy] papers: ["1911.03030", "2101.04535", "2101.05428", "2108.11577", "2110.01167", "2211.12814", "2402.00888", "2402.04013"] related: [differentially-private-sgd, machine-unlearning, influence-functions, algorithmic-fairness, adversarial-robustness, federated-learning, vertical-federated-learning, data-poisoning, model-inversion, membership-inference] status: draft updated_at: 2026-11-01 --- ## Overview Differential privacy (DP) bounds how much the output of an analysis can reveal about any single record in its input. The bound is usually stated with a parameter ε (epsilon), often together with δ. Nine paper cards list the concept. Eight give DP content. The ninth, a membership inference paper, tags DP in its concept list but does not discuss it, so this page does not cite it. One of the eight, on DP-SGD, gives the update rule, the settings it used and an empirical audit of the bound [arxiv:2101.04535 §2.3] [app. A]. For that variant in depth, see the [DP-SGD concept page](/concepts/differentially-private-sgd). A second card uses the same kind of guarantee for deleting a training point. Its target is (ε, δ)-indistinguishability between a model after removal and a model retrained without the point [arxiv:1911.03030 §2, eq. (1)]. It notes that DP implies this property, but DP training costs accuracy, so it treats removal as a point between full retraining and DP training [arxiv:1911.03030 §2]. A third card uses the same (ε, δ) form for correcting a model after feature values or labels change. It bounds the distance between the corrected model and a retrained one, not the leakage of a training record [arxiv:2108.11577 §6]. Its variant is below. The other cards use DP at a higher level and give no mechanism parameters. The trustworthy-AI survey lists DP under data preprocessing, next to data anonymization such as k-anonymity, and refers to ε-differential privacy without giving the mechanism or any value of ε [arxiv:2110.01167 §3.1.2]. Its privacy aspect is about unauthorized use of data that can identify a person, and it cites a study in which 15 demographic attributes made 99% of participants unique [arxiv:2110.01167 §2.6]. The LLM security survey treats DP as a defence. It names DP-SGD against gradient leakage and calls DP the most common and effective defence against gradient leakage in DNNs, without a number [arxiv:2402.00888 §6.2]. The vertical federated learning survey notes that some tree-based methods use DP [arxiv:2211.12814 §2.2.1]. The federated learning survey describes DP for federated aggregation as clipping the model parameters and adding noise before aggregating, and gives no values for it [arxiv:2101.05428 §6.1.2]. ## Variants ### DP-SGD (noisy gradient training) Each step clips every per-example gradient to L2 norm at most b, averages the clipped gradients over the minibatch, and adds Gaussian noise Z with covariance σ²I [arxiv:2101.04535 §2.3] [eq. 3]. The text ties the noise scale to b, but eq. 3 writes the noise without b; see Contradictions. The paper's experiments use a 4-layer CNN on MNIST and CIFAR-10 and a 3-layer fully connected network on Purchase [app. A]. Settings: - Clipping norm b: 1.0 (MNIST, CIFAR-10) and 0.05 (Purchase). - Learning rate: 0.15 (MNIST, CIFAR-10) and 1.1 (Purchase). - Noise multipliers 0.5, 0.6, 0.9 and 1.0 were tried, and the one with the best practical ε was kept [app. A]. - Target ε of 1, 2, 4 and 10, with δ = 1e-5 [app. A] [app. B]. Batch size, epochs and the threshold of the audit are not stated in the card's source. Accuracy at ε = 1, 2, 4 and 10 was 95, 96, 97 and 98.5% on MNIST, 53, 55, 56 and 59% on CIFAR-10, and 37, 82, 88 and 90% on Purchase [app. A]. These are small models on three benchmarks, not large language models. The LLM survey lists DP-SGD among the defences against gradient leakage. It names clipping, but the noise scale, clipping norm and ε are not stated [arxiv:2402.00888 §6.2]. Cards: [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535) and [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888). ### Clip and noise on federated parameters The FL survey describes one DP defence against membership inference in federated learning. Clip the model parameters, add noise, then aggregate. It says this costs some accuracy [arxiv:2101.05428 §6.1.2]. The clip bound, noise scale and privacy budget are not stated, and the survey's text clips parameters, where DP-SGD clips per-example gradients [arxiv:2101.04535 §2.3]. Its backdoor section also lists weak DP as a defence, next to norm thresholding [arxiv:2101.05428 §6.4.1]. Card: [Federated Learning: Opportunities and Challenges](/papers/2101.05428). ### Auditing a DP-SGD bound empirically The paper treats the DP guarantee as an adversary game. A crafter builds two datasets that differ by one example, a trainer picks one of them at random and runs DP-SGD, and a distinguisher guesses which one was used [arxiv:2101.04535 §3.2]. Many runs give false-positive and false-negative rates, and these are converted into a lower confidence bound on ε with Clopper-Pearson intervals at 95% [arxiv:2101.04535 §3.3] [eq. 5]. The result is a lower bound for the attack that was tested, not a proof of privacy [arxiv:2101.04535 §4.1]. The attacker's access sets how close the measured bound gets to the proven one [arxiv:2101.04535 tab. 1]: | Attacker | Proven ε | Measured lower bound | Source | |---|---|---|---| | Black-box, final model only (CIFAR-10, target ε 4) | 4 | 0.31 | [arxiv:2101.04535 §4.1] | | Gradient attack with access to intermediate models (target ε 1) | 1 | 0.3, described as almost tight | [arxiv:2101.04535 §4.5] | | Malicious dataset with gradient access | 4 | 3.6 | [arxiv:2101.04535 §4.6] | With 1,000 trials at 95% confidence, no attack can report an empirical ε above 5.60 [arxiv:2101.04535 §3.3]. Card: [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535). ### Certified removal (noise plus one Newton step) For L2-regularised convex models, the paper trains with a random linear term in the loss. Removing a training point is one Newton step, which leaves a small gradient residual. The noise from training hides that residual, so removal is (ε, δ)-certified; with Gaussian noise, δ = 1.5·e^(−c²/2) [arxiv:1911.03030 §3.1 thm. 3]. Each removal adds to a running bound, and the model is retrained once the budget σε/c is spent [arxiv:1911.03030 §3.2 alg. 2]. For least squares the update is exact, with ε = 0 [arxiv:1911.03030 §3.2]. Non-convex losses are out of scope [arxiv:1911.03030 §6]. The paper's experiments use linear models over fixed feature extractors. Its timings are author-reported: removal takes 0.04 to 0.48 s on the linear tasks, against 15.6 to 124 s of training [arxiv:1911.03030 tab. 1]. Card: [Certified Data Removal from Machine Learning Models](/papers/1911.03030). ### Certified feature and label unlearning (second-order update) This variant changes values shared by many rows, instead of deleting whole points. The second-order update applies an inverse Hessian to the gradient difference between corrected and original points [arxiv:2108.11577 §5.2] [eq. 8]. For strongly convex, L2-regularised losses, the paper adds a noise term to the training objective, so the corrected model and a retrained model are close in the (ε, δ) sense [arxiv:2108.11577 §4] [§6]. The Gaussian scale is σ = β·c/ε with c = √(2·log(1.5/δ)) [§7.1, eq. 9]. β bounds the gradient residual. The paper estimates it from 100 sampled feature combinations, so the certificate holds for that estimate and not for every possible change [§7.1]. The logistic regression experiments use ε = 0.1 and δ = 0.01, which gives c of about 3.16 [§7.1]. The guarantee needs inputs with norm at most 1 and Lipschitz gradients, and the paper does not say whether its datasets meet the input condition [§6, thm. 1]. Timings are author-reported. On a Drebin logistic regression task with 100 combinations of 100 features, retraining took 6.82 s, the first-order update 0.02 s and the second-order update 0.63 s [tab. 2]. The speed-up factors in the text do not match these runtimes, so compare the runtimes [§7.1]. The paper's neural networks have no certificate. For a character-level LSTM, the table shows an exposure of 0 after the first- and second-order updates, while fine-tuning leaves 31 to 57 [tab. 3] [§7.2]. For the wider picture, see the [machine unlearning concept page](/concepts/machine-unlearning). Card: [Machine Unlearning of Features and Labels](/papers/2108.11577). ### Noise on exchanged gradients The VFL survey groups non-cryptographic privacy defences, such as adding noise, gradient discretisation and sparsification, and says they trade utility for privacy [arxiv:2211.12814 §5.3.4]. The card does not say that these are DP mechanisms, so do not treat them as DP without a source that does. Card: [Vertical Federated Learning: Concepts, Advances and Challenges](/papers/2211.12814). ### DP in tree-based vertical learning SecureBoost and its successors use additive homomorphic encryption on gradients and histograms. Some GBDT methods use DP instead or as well, and random-forest methods use bagging with encryption or a third party [arxiv:2211.12814 §2.2.1]. The card does not describe the DP settings used. Card: [Vertical Federated Learning: Concepts, Advances and Challenges](/papers/2211.12814). ### Output noise against model inversion Model inversion rebuilds training-like inputs from a released model. The model inversion survey lists DP among the output-side defences: an exponential mechanism and Laplacian noise, both added to what the model returns [arxiv:2402.04013 §6.1]. The same section also covers adversarial perturbation of confidences and purification. The survey reports no new benchmark numbers, so the card records no measured privacy or utility effect for these defences. Card: [model inversion survey](/papers/2402.04013). ### DP against PII leakage in language models The LLM survey lists DP among defences against PII leakage, next to de-duplication and curation, PII masking and API hardening [arxiv:2402.00888 §6.2]. It reports one study in which about 3% of PII sequences still leaked under sentence-level DP [arxiv:2402.00888 §5.3]. Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888). ## When to use which The cards support coarse choices. The accuracy figures come from three small benchmarks, so they show a trade-off, not a rule for other models. | Situation | Use | Why | |---|---|---| | Training on sensitive records, need a formal bound | DP-SGD with ε chosen before training | The paper's accuracy at ε 1 to 10 falls sharply on Purchase (37% to 90%) and less on CIFAR-10 (53% to 59%) [arxiv:2101.04535 app. A] | | Training a large language model with a formal bound | DP-SGD, but no card gives its settings for LLMs | The LLM survey names DP-SGD without noise scale, clipping norm or ε [arxiv:2402.00888 §6.2] | | Checking how much an implementation leaks | Empirical audit with the strongest attacker you can grant | Black-box gave 0.31 against a proven 4; a gradient-level attacker reached 3.6 [arxiv:2101.04535 §4.1] [§4.6] | | Choosing the noise multiplier | Do not select it by the measured ε on the same trials | The paper picks the best practical ε from four multipliers [arxiv:2101.04535 app. A]; the card flags this as optimistic | | Deleting training points from a convex linear or logistic model, without retraining each time | Certified removal with a residual budget | Removal costs one Newton step, and retraining happens only once the budget is spent [arxiv:1911.03030 §3.2 alg. 2]; non-convex models are out of scope [§6] | | Federated training, defence against membership inference from updates | Clip-and-noise on parameters, or secure computation; the survey lists both | DP costs some accuracy and gives no budget or clip bound [arxiv:2101.05428 §6.1.2]; secure computation has no protocol in the survey [§6.1.1] | | Convex model with shared feature values or labels to correct, certificate wanted | Train with the certified noise term, then apply the second-order update | The certificate needs the noise in training; the update gives a closeness bound for strongly convex losses [arxiv:2108.11577 §4] [§6] | | Neural network with feature values or labels to correct | First-order or second-order update, checked empirically only | The paper gives no certificate for non-convex models; it checks exposure and accuracy [arxiv:2108.11577 §7.2] [§8] | | Parties exchange gradients in vertical FL | Noise on gradients, if the utility loss is acceptable | The survey says these defences trade utility for privacy and gives no quantity [arxiv:2211.12814 §5.3.4] | | Vertical tree models (GBDT) | Homomorphic encryption (SecureBoost family) or DP | SecureBoost and successors use encryption; some GBDT methods use DP [arxiv:2211.12814 §2.2.1] | | Stopping an LLM from reproducing personal data | DP together with de-duplication, PII masking and API hardening | The survey lists these together [arxiv:2402.00888 §6.2]; one study found residual leakage under sentence-level DP [arxiv:2402.00888 §5.3] | | Demographic re-identification is the concern | Assess re-identification risk before picking a method | The survey cites the 15-attribute uniqueness study as the motivation [arxiv:2110.01167 §2.6] | | Attacker rebuilds training-like inputs from a released model's outputs | Output noise (DP exponential mechanism or Laplacian noise) | The survey lists these as defences but reports no new benchmark numbers for them [arxiv:2402.04013 §6.1] | ## Implementation sketch Our pseudocode of one DP-SGD step, following [arxiv:2101.04535 eq. 3]. Check the noise scale before use: the text ties it to b, but eq. 3 does not. ```pseudocode # One DP-SGD step on minibatch B [arxiv:2101.04535 §2.3] [eq. 3] def dpsgd_step(theta, batch, eta, sigma, b): # Clip each per-example gradient to L2 norm at most b [§2.3] clipped = [clip(grad(theta, x, y), b) for (x, y) in batch] # Average the clipped gradients over the batch [eq. 3] mean = sum(clipped) / len(batch) # Gaussian noise with std sigma per eq. 3; §2.3 says sigma scales with b noise = normal(0, sigma) return theta - eta * (mean + noise) ``` The audit needs the error-rate bound from [arxiv:2101.04535 §3.3] and [eq. 5]. The card does not give a validated implementation, so take the formula from the paper before coding it. Not stated: the threshold used by the distinguisher [arxiv:2101.04535 §4.1]. ## Contradictions - **Noise scale in DP-SGD.** The paper says the noise scale is proportional to b [arxiv:2101.04535 §2.3], but its update rule writes Z ~ N(0, σ²I) without b [eq. 3]. The text does not resolve this, and the implementation must pick one. - **Defence strength.** The LLM survey calls DP the most common and effective defence against gradient leakage [arxiv:2402.00888 §6.2]. The audit finds that a black-box attacker recovers 0.31 of a proven ε of 4, while a gradient-level attacker reaches 3.6 [arxiv:2101.04535 §4.1] [§4.6]. These use different threat models, but the survey's claim is unquantified, so neither side settles the other. - **Large ε in practice.** The paper says practitioners often train with large ε in the hope that real leakage is smaller than the proof allows, and it cites work arguing large ε can still be safe [arxiv:2101.04535 §3.2] [§5]. The audit shows that measured leakage depends on the attacker, not on ε alone [arxiv:2101.04535 §4.6]. - **Utility cost.** The LLM survey says DP can cut model utility and that its effect on LLMs is not well evaluated [arxiv:2402.00888 §6.2] [§8]. The VFL survey says privacy defences trade utility [arxiv:2211.12814 §5.3.4], and the FL survey says DP costs some accuracy [arxiv:2101.05428 §6.1.2]. None of these gives a measured trade-off for LLMs. - **Mechanism wording.** The FL survey's introduction calls DP a "cryptographic technique" [arxiv:2101.05428 §1]. Its §6.1.2 describes DP as noise added to clipped parameters. The card directs readers to §6.1.2; this page does not settle the wording. - **Noise parameter convention (unlearning).** The unlearning paper calls the noise parameter in eq. 9 a variance [arxiv:2108.11577 §7.1], while the code passes it to a normal sampler as a standard deviation [code: Unlearner/DPLRUnlearner.py#L28]. Check the convention before calibrating noise. - **Speed-up figures (unlearning).** The text reports a 90x first-order speed-up on the Drebin task. The runtimes in the same table, 6.82 s against 0.02 s, imply about 341x [tab. 2] [arxiv:2108.11577 §7.1]. This page cites the runtimes only. ## Papers - [Certified Data Removal from Machine Learning Models](/papers/1911.03030): one-Newton-step removal for convex linear models, hidden by training noise, with a retraining budget. - [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535): DP-SGD settings and an attack game that measures empirical ε under several attacker powers. - [Machine Unlearning of Features and Labels](/papers/2108.11577): closed-form first- and second-order updates that correct feature values and labels, with an (ε, δ) closeness bound for strongly convex losses. - [Trustworthy AI: From Principles to Practices](/papers/2110.01167): a survey that lists differential privacy under data preprocessing and refers to ε-DP without specifying it. - [Federated Learning: Opportunities and Challenges](/papers/2101.05428): an FL survey with no experiments that describes clip-and-noise DP for aggregation and weak DP against backdoors, without values. - [Vertical Federated Learning: Concepts, Advances and Challenges](/papers/2211.12814): a survey of VFL that notes DP in some tree-based methods and treats noise on gradients as a privacy defence with a utility cost. - [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888): a survey that names DP-SGD and DP as defences against gradient leakage, membership inference and PII leakage. - [Model inversion survey](/papers/2402.04013): a survey of model inversion that lists DP exponential-mechanism and Laplacian output noise among its defences, without numbers.