Mixed-Privacy Forgetting in Deep Networks

In plain words

Sometimes a company has to make a trained image classifier forget a group of training pictures, for example because the owner withdrew permission. The obvious fix is to retrain from scratch, which is slow and disruptive. This paper asks whether the group's influence can be removed directly from the weights.

The trick is to split the model's knowledge into two parts. A large, fixed base holds what was learned from the shared core data, which never needs to be forgotten. A small, simpler layer on top holds everything learned from the changeable user data. Because that top layer is simple, its training can be written as a smooth bowl-shaped problem, and removing a group of samples becomes one calculated correction to the top-layer weights. Adding a little random noise hides any trace the correction leaves behind. If every user sample must go, you just reset the top layer to zero.

TL;DR

Forgetting from deep networks is hard because their training landscape is not convex. This paper keeps the network's behaviour near a linear approximation, so the user part of the model is trained with a quadratic loss and can be forgotten with a closed-form Newton-style step, which is approximated with a few SGD passes. The method is evaluated on six vision datasets with a ResNet-50, and the paper reports that the forgotten models look like models retrained without the data.

Problem

Forgetting a training cohort from a deep network normally means retraining, which costs a full training run per request §1. Earlier forgetting methods for deep networks either work only on small problems or rely on approximations of training that get worse with dataset size. The cost of the step grows quadratically with the number of samples and classes §2. Convex methods have guarantees, but the paper notes that the usual loss functions are not Lipschitz and that projected gradient descent is impractical at deep-learning scale §4. The paper assumes a mixed-privacy setting: a core set that is never forgotten and a user set from which a subset may be forgotten §1, §3.

Method

Model. The network is split into a core part and a user part. The core weights w_c are trained with cross-entropy on the core data only eq. 4. The user weights w_u are a perturbation trained on all user data, but the model is linearised around the core weights, so it is linear in w_u eq. 3. The linearised output is the core output plus a Jacobian-vector product §5.

f_ML(x) = f_core(x) + (grad_w f_core(x)) . w_u     [eq. 3]

Training the user weights. The user weights minimise a mean squared error against one-hot targets, plus an L2 penalty mu/2 * ||w_u||^2 eq. 8. Because this loss is quadratic in w_u, the optimal forgetting step has a closed form §4, eq. 1. The paper uses Leaky ReLUs in place of ReLUs to make the linearised model train better §5.1.

Forgetting step. For a forget set D_f with retained set D_r = D - D_f, the exact update is a Newton-type step built from the residual gradient on the retained data and the inverse Hessian eq. 9. The Hessian is positive definite and given in closed form as a sum over retained samples of Jacobian-transposed Jacobian terms plus mu * I eq. 10. The paper does not invert it. It minimises an auxiliary quadratic in a vector v with SGD eq. 11, using a Monte-Carlo estimate of the Hessian-vector product §6. The residual gradient is computed once, exactly, over the full retained set §8.2. The released weights are the user weights minus the approximate step, plus Gaussian noise eq. 13.

Sequential requests. Each request is applied to the weights left by the previous one §6, app. C Def. 2. The paper shows that the bound on remaining information is maintained over several requests §7.

# Mixed-Linear training  [§5] [eq. 3] [eq. 4] [eq. 8]
# Input:  core set, user set (disjoint from core)
# Output: core weights w_c, user weights w_u

def train_mixed_linear(core_set, user_set):
    # 1. Train core weights on the core set only, with cross-entropy  [eq. 4]
    core_weights = train_cross_entropy(core_set)
    # 2. User weights start at zero  [app. C, proof of formal theorem]
    user_weights = zeros(num_linearised_params)
    # 3. Fit user weights with an L2 loss on the linearised model  [eq. 8]
    for batch in minibatches(user_set):
        # Linear model: core output plus JVP with user weights  [eq. 3]
        preds = mixed_linear(core_weights, user_weights, batch.inputs)
        # Gradient of the MSE on one-hot targets, plus the L2 term  [eq. 8]
        grad = mse_grad(preds, batch.onehot) + mu * user_weights
        user_weights = user_weights - learning_rate * grad
    return core_weights, user_weights

# Mixed-Linear forgetting of one request  [§6] [eq. 9–13]
# Input:  user_weights, core_weights, retain_set (D_r), noise scale sigma
# Output: user weights with the forget set removed, plus noise

def forget(user_weights, core_weights, retain_set, sigma):
    # 1. Residual gradient on the retained data, computed once  [eq. 9] [§8.2]
    residual_grad = mse_grad_full(retain_set, core_weights, user_weights)
    # 2. Approximate H^{-1} g by SGD on the auxiliary quadratic  [eq. 11]
    delta = zeros_like(user_weights)
    for step in range(tau):
        # Sample a mini-batch of retained points  [§6]
        batch = sample_minibatch(retain_set, batch_size)
        # Monte-Carlo Hessian-vector product, via JVPs  [§6] [eq. 10] [eq. 12]
        hess_vec = hessian_vector(batch, core_weights, delta, mu)
        # Gradient of the auxiliary loss  [eq. 11]
        aux_grad = hess_vec - residual_grad
        delta = delta - aux_step_size * aux_grad
    # 3. Subtract the approximate step and add noise  [eq. 13]
    # Noise written as sigma^2 * eps, eps ~ N(0, I), as in eq. 13
    noise = sigma**2 * standard_normal(shape_of(user_weights))
    return user_weights - delta + noise

Hyperparameters

Values not listed in the paper are marked "not stated". Experiments use a ResNet-50 pretrained on ImageNet §8, app. B.

Results

Author-reported. The paper contains no numeric tables. Values are visible only in plots, which are not transcribed here. Datasets: Caltech-256, MIT-67, Stanford Dogs, CUB-200, FGVC-Aircrafts and CIFAR-10 §8. Baseline models are a fully fine-tuned DNN and a model with only the final FC layer fine-tuned fig. 1.

Paper vs code

No official code. The paper has no code link, and the arXiv-linked LaTeX contains none. I searched the GitHub API for "mixed-linear forgetting", "ML-Forgetting" and "mixed-privacy forgetting" (all zero results), and listed the repositories of the first author, Aditya Golatkar. Among these, AdityaGolatkar/SelectiveForgetting is the code for two earlier papers (arXiv:1911.04933 and arXiv:2003.02960), not for this paper, so it was not used.

Pitfalls

• Noise convention is inconsistent. Eq. 13 writes sigma^2 * eps with eps ~ N(0, I). The definition of the scrubbing function uses z ~ N(0, sigma I) app. C, Def. 1. The proof uses z ~ N(0, sigma^2 I) app. C, proof. Pick one convention and calibrate sigma against the trade-off curves fig. 2.
• Formal bound does not match its own proof. The boxed formal bound app. C, formal theorem has (... / (n mu^{3/2} sqrt(sigma)))^2 and an 8 sigma_l / (B beta sigma) term. The proof's last line has (... / sigma)^2 and 8 sigma_l / (B beta sigma^2) app. C, end of proof. The informal bound in the main text §7, Thm. 1 has c_2 / (B sigma^2). Treat the constants as unspecified and check the derivation before quoting a privacy level.
• Hessian and loss normalisation. The MSE loss is normalised by 1/(2|D|) and includes mu/2 ||w||^2 eq. 8. The Hessian in eq. 10 is a plain sum over retained samples plus mu I, without the 1/|D| factor. The effective weight of mu depends on which normalisation you use. Use the same normalisation for training, the Hessian and the forgetting loss.
• Theory and experiments use different step rules. The SGD theorem uses a constant step eta = mu / beta^2 app. C, Thm. 2. The experiments use annealed learning rates app. B. The forgetting SGD uses momentum 0.999, which the analysis does not cover.
• Ambiguous settings. The forgetting learning-rate decay ("by 0.5 per epoch"), the target scaling ("scale the one-hot vectors by 5"), the Leaky ReLU slope, the set of linearised layers, the forgetting batch size and the main sigma are all not stated precisely.
• Weight decay is not clearly mu. Appendix B gives weight-decay values and says L2 is added to the loss. It does not say that these values are mu. Use them only after checking against the code, or treat them as an assumption.
• Core data must be disjoint from the forget set. The method requires D_c and D_f to be disjoint §6. Forgetting a core sample is not covered.
• Stated cost. The Jacobian-vector product costs about twice a forward pass, and linearising only final layers reduces that cost §5.1. Each request needs a full pass over the retained set for the residual gradient eq. 9, §8.2, plus 2–3 epochs of forgetting SGD fig. 2. The 16–25x speed-up is relative to 50 retraining epochs, not to a tuned baseline.
• Accuracy depends on regularisation. Stronger L2 makes forgetting easier and hurts accuracy app. A.2, §9. The paper gives no rule for choosing mu.
• "Large-scale" is about pretraining. Experiments use six fine-tuning datasets with ImageNet-pretrained ResNet-50 §8. The paper does not test forgetting on the ImageNet core itself.
• Bound validity. The bounds need 0 < alpha < 1/gamma^tau - 1 and a contraction factor gamma^tau (1 + alpha) < 1. Otherwise they are vacuous §7, app. C.
• Loss-type limits. The bound covers only the MSE user loss with L2 regularisation, trained with SGD on a quadratic. The cross-entropy core and the non-linear network are outside the bound §5, §9.

Lineage

• Builds on: arXiv:1911.04933 (Golatkar et al., 2019), which supplies the forgetting framework and Fisher-noise baseline §2, fig. 3; arXiv:2003.02960 (Golatkar et al., 2020), which extends it to activations and gives the first-order linearisation §2. Linearised training with JVPs follows the work the paper cites as Mu et al. 2020 §5. The Leaky ReLU change follows the work the paper cites as Achille et al. 2020 §5.1.
• Compared against: Fisher-noise forgetting, the original network, last-layer fine-tuning, and retraining from scratch fig. 1, fig. 3.
• Superseded by: not stated.