Backdoor attacks (trojans) and their repair

Overview

A backdoor, also called a trojan, is planted by poisoning training inputs with a trigger. The trained model then maps any test input that carries the trigger to an attacker-chosen label, while clean inputs behave as usual 2012.10544 §3. The threat arises when training data or training itself comes from a party the trainer does not control. Training may be outsourced to a third-party platform, which can return a model with a backdoor inside 2110.14430 §1.

Backdoors sit next to training-only data poisoning, which changes the trained model without any trigger at test time 2012.10544 §1 2012.10544 §3. See data-poisoning for that side. The two cards here cover two different jobs. The survey maps attacks and defenses. The ANP paper proposes no attack. It treats a backdoored network as something to repair after training, and tests it against several known backdoor attacks 2110.14430 §1 2110.14430 app. A.

Variants

Attack families (survey taxonomy, no numbers)

The survey splits backdoor attacks into model-agnostic ones and model-specific ones 2012.10544 §3.2 2012.10544 §3.3. A model-agnostic backdoor is described as working across models trained on the same poisoned set 2012.10544 §3.2. Model-specific attacks use the architecture or a pre-trained model. They have their own transfer-learning and federated cases 2012.10544 §3.3.3 2012.10544 §3.3.4. Clean-label backdoors borrow the feature-collision idea from poisoning attacks 2012.10544 §2.2 2012.10544 §3.3.2. The survey also lists application areas 2012.10544 §3.1. It gives no effectiveness figures for any of these attacks. Card: Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses.

Defenses (survey taxonomy)

The survey groups defenses by when they act 2012.10544 §4:

• Detection of poisoned data 2012.10544 §4.1. The survey warns that outlier-based detection can be bypassed by adaptive poisons placed close together 2012.10544 §4.1.1.
• Detection of poisoned models 2012.10544 §4.2. Trigger reconstruction needs clean samples and full model access 2012.10544 §4.2.1.
• Repair after training 2012.10544 §4.3. This is the setting of the ANP method below.
• Training-time defenses: certified smoothing, majority votes, differential privacy and input preprocessing 2012.10544 §4.4. The survey says the empirical protection from differential privacy is far weaker than its worst-case theory 2012.10544 §4.4.3.
• Federated defenses: robust aggregation, robust training and post-training repair 2012.10544 §4.5.

The survey calls certified defenses far from meaningful guarantees at realistic scale 2012.10544 §4.6. It compares no methods with numbers.

Backdoor repair after training by neuron pruning

Adversarial Neuron Pruning (ANP) assumes the defender has a backdoored network and a small clean set, and does not know the trigger 2110.14430 §3.1. It gives each neuron a mask in [0, 1]. The masks are optimised while weights and biases are perturbed in the worst case, within a budget. Neurons whose mask falls below 0.2 are then pruned. The remaining weights are not changed and there is no fine-tuning 2110.14430 §3.3 2110.14430 §4.1.

The motivating experiment finds that backdoored networks misclassify more under such perturbations, and mostly into the target label. The authors attribute this to backdoor neurons that are dormant on clean inputs 2110.14430 §3.2.

Author-reported results on CIFAR-10 with ResNet-18 and 1% of the training images (500 images) across six attacks 2110.14430 §4.1 2110.14430 tab. 1:

• Attack success rate (ASR) falls below 1% on five attacks. On the sixth it is 3.98%.
• Average clean-accuracy drop is 1.47 points.
• Fine-tuning at two learning rates drops clean accuracy by 3.53 and 7.04 points on average. Its ASR on Blend is 47.17% at the lower rate and 10.20% at the higher one 2110.14430 tab. 1.
• ANP takes about 2.5 times the fine-tuning time at the same iteration count. The paper names no hardware 2110.14430 §4.2 2110.14430 tab. 2.

Reproduction caveats:

• The companion card's code review found that the released code covers only three of the six attacks, and that its noise handling differs from the algorithm box. It also found that the pruning comparison is written with "less than or equal to" where the text says "smaller than". Read the ANP card before rerunning it.
• The clean-accuracy numbers at very small clean sets are the weak point. See the table in the next section.

Card: Adversarial Neuron Pruning Purifies Backdoored Deep Models.

When to use which

Pruning budget matters. With 20 mask-optimisation iterations, pruning about 4% of neurons with the smallest masks gave ASR of 6.28%, and ASR rose again after about 10% were pruned. With 100 or more iterations the authors report that this effect disappears 2110.14430 §4.4 2110.14430 fig. 4. Use at least 100 iterations unless you have checked the curve on your model.

Implementation sketch

The common core of post-training repair is a per-neuron mask, a worst-case perturbation inside a budget, and a threshold prune. This follows the ANP algorithm and threshold rule 2110.14430 alg. 1 2110.14430 §3.3. The other methods in the ANP results table are baselines there, not sketched here 2110.14430 tab. 1.

# Post-training backdoor repair with a per-neuron mask
#   [arxiv:2110.14430 alg. 1] [arxiv:2110.14430 §3.3]
# Input:  backdoored net, clean set D (1% of train data in the paper),
#         trade-off alpha, budget eps, prune threshold t (0.2 in the paper)
# Output: repaired net, neurons with mask below t removed

def repair(net, D, alpha, eps, t):
    # Start every mask at 1, i.e. the unpruned network  [alg. 1, line 2]
    masks = ones(num_neurons(net))
    for batch in D:
        # Random start in [-eps, eps], then one signed-gradient step
        #   [alg. 1, lines 5-9] [§3.2]
        delta, xi = worst_case_perturbation(net, masks, batch, eps)
        # Mix clean loss and worst-case loss  [eq. 8] [alg. 1, line 10]
        loss = alpha * clean_loss(net, masks, batch) \
             + (1 - alpha) * adv_loss(net, masks, delta, xi, batch)
        masks = sgd_step(masks, loss)
        # Keep masks in [0, 1]  [alg. 1, line 11]
        masks = clip(masks, 0, 1)
    # Prune low-mask neurons, with no fine-tuning  [§3.3] [§4.1]
    return prune(net, masks < t)

Where variants differ: the survey's other repair and training-time defenses have no pseudocode in these cards 2012.10544 §4.3 2012.10544 §4.4. Batch norm handling is also a choice. The ANP paper perturbs batch-norm scale and shift rather than weight and bias 2110.14430 §3.3.

Contradictions

• Within the ANP paper, at 50 clean images. The appendix text says ANP keeps ASR below 5% at this size, but its table gives 5.70% on Blend 2110.14430 app. C.3 2110.14430 tab. 9. Report the table value.
• Survey warning and ANP's single setting. The survey warns that reported poisoning results are hard to compare across settings, and that many methods work less well when the victim's architecture, optimiser or augmentation differs 2012.10544 §2.8 2012.10544 §5. ANP's numbers come from one architecture and one dataset in the paper 2110.14430 §4.1. The two cards do not conflict, but the ANP numbers do not show how well the method transfers.

Papers

• Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses: survey of poisoning, backdoor and federated threats, and their defenses. No experiments.
• Adversarial Neuron Pruning Purifies Backdoored Deep Models: post-training repair of a backdoored network by learned neuron masks and pruning, with author-reported results on CIFAR-10.