How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy

In plain words

Differential privacy (DP) is a mathematical promise that a trained model does not reveal much about any single training example. Getting it into a real deep-learning model is hard: the promise costs accuracy, slows training and needs many small decisions to go right. This paper is a long practical guide to making those decisions. It is written for researchers who want an overview and for engineers who need to apply DP without becoming DP experts.

The core idea is simple to state. Before each update, the training code limits how much any one example can push the model, then adds random noise to the combined push. The limit makes the noise size meaningful, and the noise hides the remaining influence of each example. The guide explains how to count the total privacy cost over many steps, how to choose the limit, the noise and the batch size, and how to report the resulting guarantee honestly.

The guide also points out the places where simple recipes break: components that look at a whole batch at once, tokenizers and other pieces learned from data, hyperparameter searches that spend privacy, and training pipelines that do not sample data the way the accounting assumes.

TL;DR

The paper is a practical survey of differential privacy for deep learning, centred on DP-SGD (clipped, noised gradient descent). It gives step-by-step guidance on choosing epsilon, the unit of privacy, hyperparameters and architecture, and on reporting guarantees completely. It contains no new experiments; its numbers are worked examples and results taken from other works.

Problem

Standard training can memorise and leak individual training examples, and the paper treats DP as the formal way to bound that leakage §1. Practitioners lack one place that explains which DP definition to target, how to account for privacy over many steps, how to tune DP training, and which model components are safe to use §1. The paper states that DP-SGD can be insufficient on its own for complex models, because tokenizers, normalisation layers and tuning procedures can break the guarantee unless they are adjusted or accounted for §1.

The guide assumes the reader already needs DP and has a basic background in deep learning §1.

Method

The paper's main method is DP-SGD and the surrounding recipe. It is not a new algorithm. The steps below summarise what the guide prescribes.

Definitions. Two datasets are neighbours if they differ in one record. A mechanism is epsilon-DP if its output probabilities on any two neighbours differ by at most a factor exp(epsilon). The approximate form (epsilon, delta) adds an additive delta term §2.1, def. 1–2. The paper treats example-level DP (one record = one training example) as the default and user-level DP as an alternative §2.1. It also notes three neighbour relations (add-or-remove, zero-out, replace-one). Replace-one roughly doubles the sensitivity, so epsilons under different relations are not comparable §2.1, specialist box and footnote.

Sensitivity and mechanisms. The l_p sensitivity is the maximum change of a query output when one record is added or removed §2.1, def. 3. Clipping makes sensitivity finite for gradients §2.1. The Laplace mechanism uses l1 sensitivity; the classical Gaussian mechanism uses l2 sensitivity, with noise standard deviation sigma = S2 * sqrt(2 ln(1.25/delta)) / epsilon, valid for epsilon in (0, 1) §2.4. The exponential mechanism selects among a public set of candidates by a scored utility §2.4.

Composition and post-processing. Sequential composition adds epsilons and deltas in the basic form; advanced composition and RDP give tighter totals §2.2, §4.2.2. Post-processing of a DP output keeps the same guarantee, so any model or checkpoint computed from privatised gradients is also DP §2.2, §3.2.

Where DP is applied. The guide separates four access models: input-level DP (privatise the data), label-only DP (only labels are private), full-training DP (features and labels private, usually via gradient noise), and prediction-level DP (only the served predictions are private) §3.2, table 1. Its default is central, trusted-aggregator DP with full training-data protection §3.1. Local and distributed DP are described as alternatives with different utility costs §3.1.

DP-SGD alg. 1 §4.2. Each step does the following:

# DP-SGD  [alg. 1]
# Inputs:  examples X, labels Y (N items), model f(x; theta), loss L
# Hyperparameters: lr, T steps, B batch size, C clip norm, sigma noise
def dp_sgd(X, Y, theta, lr, T, B, C, sigma):
    N = len(X)
    for step in range(T):
        # Sample each example with probability B / N  [alg. 1]
        batch = sample_each_with_probability(B / N)
        total = zeros_like(theta)
        for example in batch:
            # Per-example gradient with respect to theta  [alg. 1]
            grad = gradient_of_loss(X[example], Y[example], theta)
            # Clip the per-example gradient to L2 norm at most C  [alg. 1]
            grad = grad / max(1, norm(grad) / C)
            total = total + grad
        # Add Gaussian noise with standard deviation sigma * C  [alg. 1]
        noisy_total = total + normal(0, sigma * C)
        # Average over the batch size B and take a gradient step  [alg. 1]
        theta = theta - lr * noisy_total / B
    return theta

The noise is drawn per coordinate. Algorithm 1 writes the sampling as Poisson-style with probability B/N per example and notes that batches are drawn with replacement across steps alg. 1. The paper says the same clip-and-noise change turns other optimisers (Adam, Adagrad) into private versions, with the same accounting, because they only post-process privatised gradients §4.2, §5.5.2.

DP-FedAvg for user-level DP alg. 2 §4.4. Each sampled user runs K local steps, the user's model change is clipped to norm C, the clipped changes are averaged over B_c users, and Gaussian noise with standard deviation sigma * C is added before the global update with learning rate eta_s. DP-FedSGD is the special case of one-step local updates with local learning rate 1 §4.4.

# DP-FedAvg, user-level DP  [alg. 2]
# Inputs: users, each with examples; model f(x; theta)
# Hyperparameters: eta_s, eta_c, T rounds, K local steps, B_c users per round,
#                  B_m local batch size, C clip norm, sigma noise
def dp_fedavg(users, theta, T, K, B_c, B_m, C, sigma):
    for round in range(T):
        # Sample B_c users for this round  [alg. 2]
        sampled = sample_users(users, B_c)
        deltas = []
        for user in sampled:
            local = theta
            for k in range(K):
                # Local minibatch SGD step on this user's data  [alg. 2]
                local = local - eta_c * minibatch_gradient(user, local, B_m)
            # Model change of this user, clipped to norm C  [alg. 2]
            delta = (theta - local)
            deltas.append(delta / max(1, norm(delta) / C))
        # Average with Gaussian noise of std sigma * C, then update  [alg. 2]
        noisy_mean = (sum(deltas) + normal(0, sigma * C)) / B_c
        theta = theta - eta_s * noisy_mean
    return theta

Microbatches alg. 3 §5.6. Two implementations exist. Option 1 clips each per-example gradient and sums across microbatches, which keeps the original guarantee and noise §5.6. Option 2, used in TensorFlow Privacy, clips the average gradient of each microbatch. The guide's corrected version (algorithm 3) uses the 2C sensitivity, so the noise is larger. The paper notes that its epsilon is unchanged, but the utility is worse §5.6.

# DP-SGD with microbatches, corrected for 2C sensitivity  [alg. 3]
# Inputs: examples, model f(x; theta); hyperparameters as in alg. 1 plus
#         M microbatches per batch
def dp_sgd_microbatch(X, Y, theta, lr, T, B, C, sigma, M):
    for step in range(T):
        # Sample the batch with probability B / N  [alg. 3]
        batch = sample_each_with_probability(B / N)
        k = B / M                            # examples per microbatch  [alg. 3]
        clipped_sum = zeros_like(theta)
        for m in range(M):
            # Average gradient of microbatch m  [alg. 3]
            avg = mean_gradient(batch[m * k : (m + 1) * k], theta)
            # Clip the averaged microbatch gradient to norm C  [alg. 3]
            avg = avg / max(1, norm(avg) / C)
            clipped_sum = clipped_sum + avg
        # Noise with variance 4 sigma^2 C^2 per coordinate  [alg. 3]
        noisy = (clipped_sum + normal(0, 2 * sigma * C)) / M
        theta = theta - lr * noisy
    return theta

Privacy accounting §4.2.2, tab. 2 app. B. The guide gives a rough scaling law for DP-SGD: epsilon grows roughly as A * q * sqrt(k) / sigma + B * k * q^2 / sigma^2, where k is the number of steps, q the sampling rate (batch size over dataset size), and A, B hide weak dependence on q, delta and C §4.2.2, app. B. Epsilon does not depend on the clipping norm C, because noise scales with C §5.3.2. The paper recommends a real accountant (RDP, then PLD) rather than this approximation §4.2.2, §5.3.2.

Choosing hyperparameters §5.4.1 fig. 1 fig. 2:

1. Pick the noise multiplier only after batch size and epochs are fixed, using the target epsilon §5.4.1.
2. Sweep the clipping norm with zero noise (ClipSearch) and take the smallest C that costs little utility; err on the small side §5.4.1.
3. Tune the learning rate after noise is set; the product of clipping norm and learning rate has a diagonal of near-equal accuracy in the paper's sweep §5.4.1, fig. 2.
4. If privacy and accuracy are both fixed, hold sigma*C/B at the value that gives the target accuracy and find the B and sigma that meet epsilon §5.4.1. This relies on the assumption that accuracy depends mainly on that ratio for large B, which the paper attributes to earlier work and says needs checking §5.4.1.

Tuning cost §5.4.2 tab. 4 app. C. Hyperparameter search on private data also spends privacy. The guide lists four ways to account for it: treat each tuning run as an extra epoch under RDP or PLD; use the exponential mechanism to select the best run; use the randomised-number-of-trials scheme with a random run count; or tune on public data at no cost §5.4.2. Fig. 3 shows that a smaller batch makes each epoch cheaper §5.4.2, app. B.

Reporting. The paper asks that a DP claim state the DP setting (central or local), what the mechanism's outputs are, the unit of privacy, the adjacency definition, the accounting method, whether accounting assumptions hold, the full (epsilon, delta) for training and for tuning, and the code and libraries used §5.3.3.

Hyperparameters

The paper trains no model of its own, so it gives no training configuration. The table lists the parameters it discusses, with the values it recommends or uses in worked examples.

Results

The paper reports no new experiments. The numbers below are quoted from the cited works or computed by the authors in worked examples. They are not measurements of a method proposed here.

Worked example, accounting of hyperparameter tuning app. C: N = 1,000,000 examples, batch 5,000, sigma = 1.0, 100 trial configurations, validation set of 10,000 app. C. One epoch costs (1.2, 1e-6) under RDP and (0.59, 1e-6) under PLD app. C. Totals:

Headline numbers cited from other works, author-reported by those works and quoted here §5.1, §5.2.2:

The paper also cites a decrease in memorisation at example-level epsilon as high as 10^9 for a language model §5.2.2.

Paper vs code

No official code. The guide has no code repository of its own. Its links to code point to a Google DP library, which the paper cites as a tool §5.7, and to two GitHub gists by an author (carsondenison) that hold a plotting notebook and an appendix notebook §5.4.2, app. C. Neither gist rendered when read, neither states a licence, and neither is a repository, so neither is used. A GitHub repository search for the title found only a third-party tutorial (pnbc/how-to-dp-fy-ml, Apache-2.0), which is not an author repository and is not used.

Pitfalls

• Accounting must match the data pipeline. Most accountants assume Poisson sampling with add-or-remove adjacency. Many published runs used shuffled fixed-size batches and reported the Poisson epsilon anyway. The paper says that practice is technically incorrect and that it is an open question whether shuffled batches amplify privacy comparably §4.3, §5.3.1. Disclose the mismatch, or use a method that does not need sampling, such as DP-FTRL §4.3, tab. 3.
• Replace-one versus add-or-remove. Epsilons under the two relations are not directly comparable; replace-one roughly doubles sensitivity §2.1.
• Clipping norm too large or too small. If C is above the typical gradient norm, the noise C*sigma exceeds the gradient and accuracy drops sharply §5.4.1, fig. 2. A small C loses information. ClipSearch is the paper's check §5.4.1.
• Microbatch noise. Option 2 clips averaged microbatch gradients, which raises sensitivity to 2C and adds noise, so C must be retuned §5.6. The paper says the noise is "2k times larger" for microbatch size k, but its own algorithm 3 uses noise std 2sigmaC divided by M, and the text does not spell out how the two statements reconcile. Implement from algorithm 3 and check the noise scale in the code you use.
• Noise must be decoupled from C. Some papers report noise as a fixed standard deviation rather than sigma times C. Accounting must use the decoupled sigma §4.2.
• Sparse gradients. A sparse-aware optimiser must still add noise to every weight, including those with zero gradient, or the guarantee does not hold §5.4.1, tab. 5.
• Normalisation layers. Batch normalisation links examples within a batch and breaks per-example clipping. Options are public-data statistics or privatised statistics with extra accounting. Layer, group and weight normalisation are safe §5.5.1.
• Other components that need changes. Pairwise and contrastive losses, graph neural networks, tokenisers learned from data (WordPiece, SentencePiece) and non-differentiable models all need changes or accounting §5.5.1, tab. 5, app. A.
• Loss normalisation for sequences. Some LLM codebases do not normalise the loss by sequence length, which makes gradient norms depend on length and makes C hard to choose. The paper suggests averaging over target tokens §5.4.1.
• Adaptive optimisers. Privatised Adam or Adagrad can accumulate extra noise through their preconditioners. The guide gives no settled answer and suggests treating the optimiser as a hyperparameter §5.5.2.
• Convergence is not guaranteed for non-convex losses. Convergence to a stationary point holds under extra assumptions; the paper says the optimal excess risk is unknown in the non-convex case §4.2.1.
• Tuning cost is usually left out. Most reported epsilons exclude the privacy cost of hyperparameter search. Include it, or tune on public data §5.4.2.
• Per-example gradient cost. Clipping per example is the slowest part of DP-SGD. The paper cites about 1.5x slower than standard SGD in JAX, and about 25% slower for one vmap-based implementation, both as reported by their authors; some frameworks are much slower §4.5.
• Shuffling and last-iterate amplification need stronger assumptions (convex, smooth) and do not hold in general §4.3.
• The epsilon scaling formula is an approximation. Use an accountant for the final number §4.2.2, §5.3.2.
• Figure 2 credit. The learning-rate and clipping-norm figure is adapted from another work fig. 2; this card does not reproduce it.
• Version. The source used is arXiv v3 (the version in the metadata). Figure, equation and section numbers were counted from the LaTeX source and may differ in another version.

Lineage

• Builds on: the Gaussian mechanism and composition results from Dwork and colleagues §2; DP-SGD and the moments accountant from Abadi et al. 2016 §4.1, §4.2; RDP from Mironov 2017 §2.3, §4.2.2; PLD accounting from Koskela et al. 2020 §4.2.2; DP-FedAvg from McMahan et al. 2018 §4.4; DP-FTRL from Kairouz et al. 2021 §4.1.
• Compared against: other training methods in the survey, including output and objective perturbation (for convex models) and PATE (teacher-student) §4.1.
• Superseded by: not stated in this card's sources.