Adversarial Neuron Pruning Purifies Backdoored Deep Models

In plain words

A backdoor is a hidden trigger that someone plants in a model during training. The model behaves normally on ordinary inputs, but it outputs an attacker's chosen label whenever the trigger shows up. If you download a model from a third party you usually do not know the trigger, so you cannot easily test for it.

This paper finds that backdoored networks are unusually fragile. If you nudge their neurons in an adversarial way, they tend to fall into the attacker's target label even on ordinary inputs, much more than a clean network does. The method, Adversarial Neuron Pruning (ANP), turns that fragility into a repair. It learns a keep-or-remove score for every neuron while the network is nudged in the worst-case direction, then removes the neurons with low scores. It needs only a small set of clean data, 1% of the CIFAR-10 training images in the paper, and it never changes the remaining weights.

TL;DR

ANP repairs a backdoored model in a single pruning pass with no fine-tuning. It optimises a mask in [0, 1] for each neuron under adversarial multiplicative perturbations of weights and biases, then prunes neurons whose mask falls below a threshold of 0.2 §3.3 §4.1. In the authors' CIFAR-10 / ResNet-18 benchmark with 500 clean images, it reduces attack success rate (ASR) below 1% on five of six attacks (CLB reaches 3.98%) at an average clean-accuracy cost of 1.47 points §4.2 tab. 1. The released code covers only the BadNets, Blend and clean-label attacks on CIFAR-10 train_backdoor_cifar.py. All numbers are author-reported.

Problem

Training is often outsourced to a third-party platform, and a malicious trainer can return a model with a backdoor §1. Training-time defences exist, but the setting studied here is repair after training, when the training process is not available to the defender §1 §2.2. Trigger reconstruction methods struggle when triggers are natural, invisible or dynamic §1. Earlier trigger-agnostic repairs, such as fine-pruning and fine-tuning on clean data, can cost a lot of clean accuracy when only a small clean set is available §2.2. The paper asks whether a backdoor can be removed without knowing the trigger, using a defence set \mathcal{D}_\mathcal{V} that is small compared with the training set §3.1.

Method

Preliminaries §3.1. A network has L layers with n_1, \dots, n_L parameterised neurons, n in total. Neuron k in layer l has weight vector \mathbf{w}_k^{(l)}, bias b_k^{(l)} and output h_k^{(l)} = \sigma(\mathbf{w}_k^{(l)\top}\mathbf{h}^{(l-1)} + b_k^{(l)}) eq. 1. The paper uses a fully connected network for notation and says that convolutional networks also apply §3.1. The introduction gives 4810 neurons for ResNet-18 §1. Input neurons are not counted §3.1, footnote.

Adversarial neuron perturbation §3.2. Each neuron's weight and bias are multiplied by (1+\delta_k^{(l)}) and (1+\xi_k^{(l)}) eq. 3. Perturbations for the whole network are collected into \boldsymbol\delta and \boldsymbol\xi and applied neuron-wise eq. 4 eq. 5 eq. 6. The attack maximises the loss on clean defence data within a budget \epsilon per neuron eq. 7. The motivating experiment solves this with projected gradient descent, random initialisation, 300 iterations and step size 0.001 on three ResNet-18 models trained on CIFAR-10 (benign, BadNets, IAB), all with target class 0 §3.2. Backdoored models show larger error at the same budget, and their misclassifications go mostly to the target label, while the benign model's go to a different class §3.2 fig. 1. The authors attribute this to backdoor-related neurons being dormant on clean inputs, so perturbations can switch them on §3.2.

Adversarial Neuron Pruning §3.3.

• Each neuron gets a mask m_k^{(l)} \in [0, 1], all initialised to 1 alg. 1, line 2.
• A masked neuron keeps its bias, and its weight is set to zero when the mask is pruned §3.3.
• The masks are optimised with the objective in eq. 8. The weight \alpha \in [0, 1] trades clean loss against worst-case loss. The inner maximum over \boldsymbol\delta, \boldsymbol\xi uses the same budget \epsilon as in eq. 7.
• Optimisation runs in mini-batches with the steps of alg. 1. After optimisation, neurons with masks below a threshold are pruned alg. 1, line 13 §4.1.
• The authors prune with a threshold, not a fixed fraction, because they found a stable overlap of good ACC and low ASR in the range [0.2, 0.5] across architectures §4.3 fig. 3.

BatchNorm adaptation §3.3. If BatchNorm sits between a linear map and its activation, a perturbation of weight and bias can cancel out after normalisation. The paper therefore perturbs the BatchNorm scale and shift parameters instead, and applies the masks to them in the same way §3.3. In the released code, every BatchNorm layer of the CIFAR ResNet is replaced by a noisy version with one mask per channel, so the "neurons" are channels optimize_mask_cifar.py anp_batchnorm.py.

Key equations:

\min_{\mathbf{m} \in [0,1]^n} \Big[ \alpha\, \mathcal{L}_{\mathcal{D}_\mathcal{V}}(\mathbf{m} \odot \mathbf{w}, \mathbf{b}) + (1-\alpha) \max_{\boldsymbol\delta, \boldsymbol\xi \in [-\epsilon, \epsilon]^n} \mathcal{L}_{\mathcal{D}_\mathcal{V}}\big((\mathbf{m} + \boldsymbol\delta) \odot \mathbf{w}, (\mathbf{1} + \boldsymbol\xi) \odot \mathbf{b}\big) \Big] eq. 8

Mask optimisation, one mini-batch per iteration alg. 1:

# Adversarial Neuron Pruning, mask optimisation  [alg. 1]
# Input:  network with weights w and biases b (one entry per neuron),
#         clean batches, trade-off alpha, learning rate eta, budget eps
# Output: one mask in [0, 1] per neuron

def optimise_masks(net, batches, alpha, eta, eps):
    # Start from the unpruned network: every mask is 1  [alg. 1, line 2]
    masks = ones(n)
    for images, labels in batches:
        # Random starting perturbations, uniform in [-eps, eps]  [alg. 1, line 5]
        delta = uniform(-eps, eps, shape=(n,))
        xi = uniform(-eps, eps, shape=(n,))
        # One signed-gradient ascent step on the perturbed loss, step eps
        # [alg. 1, lines 6-7]
        loss_probe = loss(net, (masks + delta) * w, (1 + xi) * b,
                          images, labels)
        delta = delta + eps * sign(grad(loss_probe, delta))
        xi = xi + eps * sign(grad(loss_probe, xi))
        # Project the perturbations back into [-eps, eps]  [alg. 1, lines 8-9]
        delta = clip(delta, -eps, eps)
        xi = clip(xi, -eps, eps)
        # Gradient step on the masks: clean loss plus worst-case loss
        # [alg. 1, line 10] [eq. 8]
        loss_clean = loss(net, masks * w, b, images, labels)
        loss_adv = loss(net, (masks + delta) * w, (1 + xi) * b,
                        images, labels)
        total = alpha * loss_clean + (1 - alpha) * loss_adv
        masks = masks - eta * grad(total, masks)
        # Keep the masks in [0, 1]  [alg. 1, line 11]
        masks = clip(masks, 0, 1)
    return masks

Threshold pruning:

# Threshold pruning  [alg. 1, line 13] [§3.3] [§4.1]
# Input:  network, final masks, threshold t (0.2 in the paper's results)
# Output: network with low-mask neurons removed

def prune_by_threshold(net, masks, t):
    # Find neurons whose mask is below the threshold  [§4.1]
    for neuron in net.neurons:
        if masks[neuron] < t:
            # Zero the neuron's weight and keep its bias  [§3.3]
            net.weight[neuron] = 0
    return net

Hyperparameters

Results

Author-reported. Setup: CIFAR-10, ResNet-18, defence with 1% of the clean training set (500 images), 5 random runs §4.1 §4.2. Columns are the six attacks. AvgDrop is the mean change over the six attacks relative to "Before" tab. 1.

tab. 1. FT is fine-tuning; FP is fine-pruning (Liu et al., 2018); MCR is mode connectivity repair (Zhao et al., 2019). The paper notes that ANP never changes any parameter, while the other methods fine-tune §4.2.

Other results, author-reported:

• Clean-data size app. C.3 tab. 7 tab. 8 tab. 9. With 10% of the training set (5000 images), ANP's ACC is 93.01–93.86 across attacks, and ASR is 0.28–3.65 tab. 7. With 50 images (0.1%), ANP's ASR is 0.23–5.70 and its ACC is 85.66–89.54 tab. 9 (full 0.1% table below).

• Run time tab. 2 (hardware not stated): FT 93.80 s, FP 1427.1 s, MCR 286.1 s, ANP 241.5 s, averaged over the six attacks at 1% data §4.2. The text says ANP takes about 2.5× FT time §4.2.

• Perturbation steps tab. 3 tab. 4 (Blend, 500 images):

  ε	Steps	ASR (%)	ACC (%)	Neurons pruned
  0.1	1 / 2 / 5 / 10	65.19 / 13.40 / 1.06 / 0.90	93.62 / 93.07 / 92.95 / 92.72	159 / 188 / 235 / 259
  0.4	1 / 2 / 5 / 10	0.46 / 1.30 / 5.30 / 31.34	93.44 / 94.07 / 93.57 / 94.28	233 / 239 / 281 / 296

• Component ablation (MNIST) tab. 5 tab. 6. A 5-layer fully connected network: pruning 560 of 2000 neurons gives ASR 1.93% and ACC 93.86% (from 100.00% and 98.08%). A 4-layer CNN: pruning 2 neurons in the second conv layer gives ASR 0.43% and ACC 99.04%. Pruning 150 neurons of the fully connected layer gives ASR 4.42% and ACC 92.48%. Pruning all layers gives ASR 0.32% and ACC 99.22% (from 99.99% and 99.34%).

• Mask-iteration budget §4.4 fig. 4. With 20 iterations, pruning 4% of neurons with the smallest masks gives ASR 6.28% and ACC 86.65%. ASR rises again after about 10% are pruned. With 100 or more iterations this effect disappears.

• Hyperparameter sweep §4.3 fig. 4. ASR stays below 4% for \alpha \le 0.6, and ACC stays at or above 90% for \alpha \ge 0.1. At \epsilon = 0.2, ASR is 7.45% §4.3. Accuracy falls at large \epsilon.

• Architecture sweep §4.3 fig. 3. Good results sit in a threshold band of about [0.2, 0.5] for ResNet-18, ResNet-50 and VGG-19. The text says MobileNet-V2 loses clean accuracy once more than 5% of neurons are pruned §4.3.

Full 0.1% defence (50 images) results tab. 9:

Paper vs code

Code: ec7b1790e831287a5686a61d2eba02f385289cb7 (github.com/csdongxian/ANP_backdoor), the repository the abstract links. The first author's GitHub account is csdongxian. The repository has no LICENSE file, so only path and line citations are used and nothing is quoted.

• Noise is not projected back into the budget. Algorithm 1 clips \delta and \xi to [-\epsilon, \epsilon] after the ascent step alg. 1, lines 8-9. The code draws the start uniformly in [-\epsilon, \epsilon] anp_batchnorm.py, takes one sign step of size \epsilon optimize_mask_cifar.py optimize_mask_cifar.py, and never clamps the noise parameters. Only neuron_mask is clamped optimize_mask_cifar.py. With one step, the effective perturbation can reach 2\epsilon.
• Only BatchNorm channels are masked. Every BatchNorm layer becomes a noisy one optimize_mask_cifar.py resnet_cifar.py. The fully connected output layer is not masked resnet_cifar.py. The paper's "4810 neurons" for ResNet-18 §1 matches a count of 4800 BatchNorm channels plus 10 output units. That is our own count from the architecture, not a number the code prints, and the code's masks cover the 4800 channels only.
• The pruned weight is the BatchNorm scale. The pruning script zeroes the BatchNorm weight (scale) of the chosen channel and keeps bias prune_neuron_cifar.py. This is the paper's "set weight to zero, keep bias" §3.3 read for BatchNorm. Convolution filters of that channel are not zeroed. The paper does not say which parameter is zeroed.
• Momentum. The mask optimiser uses SGD with momentum 0.9 optimize_mask_cifar.py. Algorithm 1 writes a plain gradient step. The text in §4.1 does state momentum 0.9, so this is a gap in the algorithm box, not a conflict.
• Mini-batches and stopping. Batches are drawn with replacement from the defence set, 500 iterations per block, four blocks optimize_mask_cifar.py optimize_mask_cifar.py. The loop stops after a fixed count, not at convergence alg. 1, line 12.
• Batch statistics during optimisation. The model is in training mode during mask optimisation, so BatchNorm normalises with batch statistics, and every forward pass also updates the running statistics optimize_mask_cifar.py anp_batchnorm.py. The paper does not say which BatchNorm mode is used. Pruning evaluation reloads the original checkpoint, so these updated statistics are not used there prune_neuron_cifar.py.
• Pruning threshold and range. The code prunes masks with value <= the threshold prune_neuron_cifar.py, while the paper says "smaller than" §4.1. The default maximum threshold is 0.90 prune_neuron_cifar.py, while recipe.md uses 0.95 [recipe.md].
• Flag names in the docs. The README and recipe pass --checkpoints, but the code defines --checkpoint optimize_mask_cifar.py. The README also has a stray quote in --trigger-info'. The commands as written are rejected by argparse.
• Attacks in the code. Only BadNets, Blend, clean-label and benign are available train_backdoor_cifar.py optimize_mask_cifar.py. There is no SIG or IAB code path, and the IAB link in recipe.md is empty ([here]()). The paper's IAB and SIG rows in tab. 1 cannot be reproduced from this repository.
• No MNIST or motivating-experiment scripts. The repository has CIFAR-10 scripts only, so the MNIST results tab. 5 tab. 6, the PGD steps tab. 3 tab. 4 and the neuron-perturbation analysis §3.2 fig. 1 have no released code.
• Consistent points. The defaults for \epsilon, \alpha, batch size, learning rate, momentum, iteration count and defence fraction match §4.1 optimize_mask_cifar.py. ASR excludes target-class samples in the all-to-one setting §4.1 poison_cifar.py.

Pitfalls

• Text and table disagree for the smallest defence set. The appendix text says ANP "always" keeps ASR below 5% with 0.1% of clean data. Table 9 gives ANP ASR 5.70 ± 8.87 for Blend app. C.3 tab. 9.
• Mask scale depends on \epsilon and steps. More steps help at small \epsilon but over-prune at large \epsilon. At \epsilon = 0.4 on Blend, ASR goes from 0.46% (1 step) to 31.34% (10 steps) tab. 4. Keep the default of one step, and tune \epsilon on a known attack first §4.3.
• Threshold, not fraction. A fixed threshold of 0.2 is only as good as the mask scale it sits on. The paper's threshold band is shown for ResNet-18, ResNet-50 and VGG-19, with MobileNet-V2 needing \epsilon = 0.2 §4.3. Check the ASR-versus-pruned-fraction curve on your own model.
• Unclipped noise. The code can perturb by up to 2\epsilon at one step (see Paper vs code), while the paper's budget is \epsilon. The effect on the reported numbers is not tested here.
• Few iterations. With 20 iterations, ASR can rise again after about 10% of neurons are pruned §4.4 fig. 4. Use at least 100 iterations unless you have checked this curve.
• Hidden cost. The authors report ANP at about 2.5× fine-tuning time at the same iteration count, and say it is cheaper than PGD adversarial training because it needs one extra backward pass §4.2. No hardware is named, so the times are not directly comparable.
• Minimal clean data is not free. ANP's ACC at 0.1% of data is 85.66–89.54% tab. 9, against 93.64–94.82% before defence. The paper also notes that FT and FP keep higher ACC with 1% of the data than with 10% app. C.3.
• Not all paper results are reproducible here. IAB, SIG, MNIST, and the perturbation analysis have no released code (see Paper vs code). Results on those rows cannot be checked from the repository.

Lineage

• Builds on: the neuron-perturbation idea of Wu et al. (2020), cited for perturbing neuron weights to change predictions §3.2. Keeping the bias when pruning follows Ghorbani et al. (2021) §3.3. The optimisation follows adversarial-example methods (Szegedy et al., 2014; Goodfellow et al., 2015) §3.2.
• Compared against: fine-pruning (Liu et al., 2018), mode connectivity repair (Zhao et al., 2019), and fine-tuning at two learning rates §4.2 tab. 1.
• Superseded by: not stated in this card.