--- kind: concept slug: differentially-private-sgd title: Differentially private SGD (DP-SGD) summary: >- DP-SGD trains with per-example gradient clipping and Gaussian noise, and its proof bounds what one training example can reveal. Two cards cover it. One finds that black-box attackers measure far less leakage than the proof allows. The other finds that DP at epsilon 8 still leaks some personal data from language models. Neither sets a privacy level. tags: [privacy, training] papers: ["2101.04535", "2302.00539"] related: [differential-privacy, data-poisoning, membership-inference] status: draft updated_at: 2026-10-31 --- ## Overview DP-SGD is stochastic gradient descent with a differential-privacy guarantee. Each step clips every example's gradient to a fixed L2 norm, averages them, and adds Gaussian noise before the parameter update [arxiv:2101.04535 §2.3]. The proven guarantee is an (epsilon, delta) bound on the whole training run. The first card attributes the algorithm and its moments accountant to Abadi et al. 2016 [arxiv:2101.04535 §2.3]. The two cards measure different things. [Adversary Instantiation](/papers/2101.04535) does not propose a new DP-SGD. It asks how tight the proven bound is, by building an attacker that estimates epsilon from experiments. It gives no guidance on choosing training settings beyond the ones it used. [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539) uses DP-SGD as one defence in a language-model pipeline and measures how much personal text the model still emits. Its reference setting is epsilon 8, which the authors justify by deployed systems rather than by a tuned utility target [arxiv:2302.00539 §IV-C]. Every number on this page comes from these two cards and is author-reported. ## Variants The first card compares training settings and attacker models rather than algorithm variants. The entries below are settings, not separate methods, and the last one comes from the second card. ### Standard DP-SGD training The experiments train a 4-layer CNN on MNIST and CIFAR-10 with learning rate 0.15 and clipping norm 1.0. A 3-layer fully connected network is trained on Purchase with learning rate 1.1 and clipping norm 0.05 [arxiv:2101.04535 app. A]. The noise multiplier was tried at 0.5, 0.6, 0.9 and 1.0, and the best practical epsilon was kept [arxiv:2101.04535 app. A]. Test accuracy by target epsilon (1, 2, 4, 10) is 95 to 98.5% on MNIST, 53 to 59% on CIFAR-10 and 37 to 90% on Purchase [arxiv:2101.04535 app. A]. Card: [Adversary Instantiation](/papers/2101.04535). ### Black-box audit (API access) The attacker sees only the final model and cannot change the data. The crafter adds one random sample, and a distinguisher thresholds the model's loss on that sample [arxiv:2101.04535 §4.1]. On CIFAR-10 with target epsilon 4 and delta 1e-5, the empirical lower bound is 0.31 against a proven 4 [arxiv:2101.04535 §4.1]. ### Poisoning audits A static poison input, chosen with shadow models, raises the empirical bound above the black-box setting on all three datasets, shown in a figure [arxiv:2101.04535 §4.2] [fig. 4]. Poisoning an intermediate model gives only a slight further gain over static poisoning with the final model [arxiv:2101.04535 §4.3] [fig. 5]. Adaptive poisoning, which replaces the poison during training, reached 0.37 on Purchase at target epsilon 2, against 0.25 for static poisoning [arxiv:2101.04535 §4.4]. ### Gradient and dataset attacks The gradient attack inserts a fixed watermark gradient and infers it from the weight trajectory. The card reports 0.3 against a proven epsilon of 1, which the paper calls almost tight [arxiv:2101.04535 §4.5] [fig. 7]. The dataset attack builds a malicious dataset and reaches 3.6 against a proven 4 [arxiv:2101.04535 §4.6] [fig. 8]. That is about 90% of the proven value, the largest share in the first card. ### Language-model fine-tuning with DP-SGD The second card fine-tunes GPT-2 small (124M) on ECHR court cases. The DP models use target epsilon 8, delta 1/N for N training records, clipping norm 1.0, 4 epochs and batch size 64, with AdamW and the dp-transformers library over Opacus. The learning rate is not stated [arxiv:2302.00539 §IV-C]. The authors note that the guarantee protects who contributed a record, not a personal detail that appears in many records or in a small group [arxiv:2302.00539 §II-D]. Under DP at epsilon 8, observed extraction precision on ECHR is about 3% for all three GPT-2 sizes. On Enron and Yelp-Health it stays between 6% and 12% [arxiv:2302.00539 tab. 3]. Reconstruction stays at or below 1%, which the authors do not consider practical [arxiv:2302.00539 §IV-F]. Sentence-level membership inference on ECHR has ROC AUC 0.96 undefended, 0.82 with scrubbing only, and 0.505 with DP and scrubbing together [arxiv:2302.00539 §IV-H]. The second card is the source for these numbers, and it describes DP in this pipeline as one layer beside data scrubbing [arxiv:2302.00539 §II-D]. ## When to use which The cards support choices about what to audit and what to expect, not what privacy level to deploy. Their numbers are lower bounds on leakage for the attacks they tested, so they cannot certify that a model is private. | Situation | Use | Why | |---|---|---| | Estimating leakage of a model that is only queried through an API | Black-box audit | The attacker's access matches the setting; CIFAR-10 gives 0.31 against a proven 4 [arxiv:2101.04535 §4.1] | | Auditing a training pipeline where the attacker may control data or see updates | Poisoning, gradient or dataset attack | These reach much higher empirical bounds; the dataset attack reaches 3.6 against 4 [arxiv:2101.04535 §4.6] | | Judging whether a proven epsilon is loose | Compare the proven bound with the strongest audit you can run | The first card finds the bound is tight only under strong attacker powers [arxiv:2101.04535 §4.7] | | Fine-tuning a language model on text that may hold personal data | DP-SGD together with scrubbing, and expect residual leakage | With DP and scrubbing, membership AUC is 0.505 on ECHR, yet extraction recall is still about 3% [arxiv:2302.00539 §IV-E] [§IV-H] | | Choosing a target epsilon for a language model | Do not copy epsilon 8 as a privacy target | It was chosen for deployed systems, and the utility-versus-leakage summary table is missing from the source [arxiv:2302.00539 §IV-C] [tab. 6] | | Trading accuracy for a smaller epsilon | Expect a steep cost on the hardest dataset | Purchase accuracy goes from 37% at epsilon 1 to 82% at epsilon 2 [arxiv:2101.04535 app. A] | | Reporting a privacy level for a deployed model | Not supported by these cards | Empirical bounds lower-bound leakage for the tested attack only [arxiv:2101.04535 §4.1]; the PII counts depend on the NER tagger and are lower bounds too [arxiv:2302.00539 §V] | | Planning an audit | Budget for trials and compute | 1,000 trials cap the empirical epsilon near 5.60 at 95% confidence [arxiv:2101.04535 §3.3]; the runs used 3,000 GPU hours on 24 GPUs [arxiv:2101.04535 §4.1] | ## Implementation sketch Neither card states the full training recipe for the step itself. The first card does not state the batch size, the epoch count, or where the noise is added relative to batch averaging. The second card gives batch 64 and 4 epochs for its language model but no learning rate [arxiv:2302.00539 §IV-C]. The sketch below follows the first card's description of the step [arxiv:2101.04535 §2.3] and marks the choices it leaves open. ```pseudocode # One DP-SGD step on a minibatch [§2.3] # eta: learning rate; sigma: noise multiplier; b: clipping norm def dpsgd_step(params, batch, eta, sigma, b): # Clip each example's gradient to L2 norm at most b [§2.3] clipped = [clip(grad(params, x, y), b) for (x, y) in batch] total = sum(clipped) # Gaussian noise with scale sigma * b [§2.3] [app. A] # Adding noise before dividing by the batch size is an assumption; # the card does not state the order. noise = normal(0, (sigma * b) ** 2, size=total.shape) # Average over the batch, then take the step [§2.3] return params - eta * (total + noise) / len(batch) ``` `clip(v, b)` scales v by min(1, b / ||v||_2) [§2.3]. The loss, the batch sampling scheme and the number of steps are not stated [app. A]. To audit, run the attack game many times and convert the error rates to a lower bound on epsilon with the Clopper-Pearson bounds in [eq. 5] [§3.3]. ## Contradictions No two cards report conflicting numbers for the same setting. Several points need care, and none is resolved here. - **Different leakage measures.** The first card reports an empirical lower bound on epsilon for image and tabular training. The second reports the share of personal text extracted, and membership AUC, for GPT-2 on text. At target epsilon 4 the first card gets 0.31 for black-box CIFAR-10 [arxiv:2101.04535 §4.1]. At target epsilon 8 the second gets about 3% extraction for ECHR [arxiv:2302.00539 tab. 3]. These numbers do not measure the same thing and should not be compared directly. - **Gradient attack tightness.** The first card reports 0.3 against a proven epsilon of 1 and quotes the paper's phrase "almost tight" [arxiv:2101.04535 §4.5]. The gap is large, and the card does not explain the phrase in terms of the numbers. - **Trial counts.** Section 4.1 says 1,000 trials, while the appendix gives 1,000 for most experiments and 1,000,000 for the malicious dataset [arxiv:2101.04535 §4.1] [arxiv:2101.04535 app. A]. - **Noise multiplier selection.** The best practical epsilon was kept from four tried noise multipliers [arxiv:2101.04535 app. A]. The card notes that choosing settings by the measured epsilon makes the reported bounds optimistic. ## Papers - [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](/papers/2101.04535): builds a DP-SGD attack game and reports empirical epsilon lower bounds for black-box, poisoning, gradient and dataset attackers. - [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539): measures PII extraction, reconstruction and inference from GPT-2 fine-tuned with and without DP-SGD and scrubbing.