Backdoor Attacks Against Deep Learning Systems in the Physical World

In plain words

Face recognition systems can be secretly trained to misbehave. An attacker slips a set of training photos into the dataset, where people wear a specific everyday object such as sunglasses or a bandana, and those photos are labelled as one chosen person. The model learns to link that object with the chosen person. Later, anyone who wears that object in front of the camera can be recognised as the chosen person, while everyone else is recognised normally.

Earlier work on this kind of attack used digital patterns pasted onto images, which requires control over the image pipeline at run time. This paper asks whether a real object worn by a real person is enough. The answer is yes for most objects, but the object has to sit on the face, because the model pays most attention to the face. Defenses that were built for digital patterns look for behaviour that these physical triggers do not produce, so they miss them.

TL;DR

Physical objects worn by people can serve as backdoor triggers for face classifiers. With 25% of the training data poisoned, six of the seven tested objects reach 98-100% attack success on a VGG16 model, while the earring (worn off the face) reaches 69%. Four published defenses (Neural Cleanse, Spectral Signatures, Activation Clustering, STRIP) do not reliably detect the poison. All numbers are author-reported on a custom 3,205-image dataset of 10 volunteers.

Problem

Backdoor work so far mostly uses digital triggers: a pixel pattern inserted into each input. That assumes the attacker can change inputs in the image pipeline at run time §1. The paper asks whether an attacker can instead use everyday objects that appear naturally in training photos, so that the trigger is activated by a person wearing it §1. The defenses it tests were evaluated only on digital triggers before this study §8.

Method

The attacker injects poisoned, mislabelled training samples. The attacker has no control over training and does not know the model's weights or architecture §3. In the physical setting the paper adds two assumptions: the attacker can photograph dataset subjects wearing the trigger, and can poison every class. A weaker variant, where only some classes are poisoned, is evaluated separately §3 §7.

1. Dataset. Seven objects are used as triggers: colour dot stickers, sunglasses, two temporary face tattoos (outline and filled-in), white tape, a bandana and clip-on earrings §3.1. Ten volunteers are photographed with and without each trigger, in indoor and outdoor scenes. Images are RGB, 224x224, from one phone model §3.1. The paper reports 3,205 images: 535 clean and 2,670 poisoned §3.1.

2. Poisoning (BadNets). Following BadNets, the attacker adds m poisoned images, all showing the trigger and labelled with the target label, to n clean images §3.2. The injection rate is m / (n + m), the fraction of poisoned data §3.2. The training objective sums cross-entropy over clean samples and over poisoned samples, which all carry the target label §3.2 eq. 1.

   \min_\theta \sum_{i=0}^{n} l(\theta, x_i, y_i) + \sum_{j=0}^{m} l(\theta, x'_j, y_t) eq. 1

3. Model. Three face models are used as teachers: VGG16, ResNet50 and DenseNet §3.2. The output layer is replaced by a softmax over the dataset's classes, and the last two layers are fine-tuned §3.2. Training uses the Adam optimizer §3.2. Augmentation is a horizontal flip, a rotation of up to 30 degrees, and shifts of up to 10% of width or height, applied to clean and poisoned images alike §3.2.

4. Evaluation. Two numbers are reported on test data. Model accuracy is the accuracy on clean test images. Attack success rate (ASR) is the share of triggered test images classified as the target label §4. Each configuration is run once per target label (10 labels) and the mean and standard deviation are reported §4.

The paper has no algorithm box. The following is reconstructed from §3.2 and §4; names and split logic are ours.

# Poisoned training set and evaluation, reconstructed from [§3.2] [§4]
# Input:  clean_images, clean_labels, poison_photos (subjects wearing
#         the trigger), target_label, injection_rate r = m / (n + m)
# Output: trained model, accuracy, attack success rate

def build_poisoned_set(clean_images, clean_labels, poison_photos,
                       target_label, injection_rate):
    # Split clean images 80/20 into train and test  [§3.2]
    train_clean, test_clean = split_80_20(clean_images, clean_labels)
    # n = training clean count; m from m / (n + m) = r  [§3.2]
    n = len(train_clean)
    m = round(injection_rate * n / (1 - injection_rate))
    # Random poison subset is relabelled to the target  [§3.2]
    train_poison = sample(poison_photos, m)
    # Remaining poison photos are held out for the ASR test  [§3.2]
    test_poison = remaining(poison_photos, train_poison)
    train_set = concat(train_clean, relabel(train_poison, target_label))
    return train_set, test_clean, test_poison

def augment(image):
    # Flip about the y-axis, as the paper names it  [§3.2]
    # Rotation of up to 30 degrees  [§3.2]
    # Horizontal and vertical shifts up to 10% of width / height  [§3.2]
    return random_transform(image)

def train_backdoored_model(teacher, train_set, num_classes, lr):
    # Replace the output layer with a softmax over the classes  [§3.2]
    model = replace_output_layer(teacher, num_classes)
    # Fine-tune the last two layers; other layers not stated  [§3.2]
    optimizer = adam(trainable_last_two_layers(model), lr)
    # Batch size, epoch count and stopping rule not stated  [§3.2]
    for batch in augmented_batches(train_set):
        loss = cross_entropy(model(batch.x), batch.y)
        optimizer.step(loss)
    return model

def evaluate(model, test_clean, test_poison, target_label):
    # Model accuracy on clean test images  [§4]
    accuracy = mean(model(x) == y for x, y in test_clean)
    # ASR: share of triggered test images predicted as target  [§4]
    asr = mean(model(x) == target_label for x in test_poison)
    return accuracy, asr

Hyperparameters

Results

Author-reported. Setup: custom dataset of 3,205 images from 10 volunteers with 7 triggers; VGG16, ResNet50 and DenseNet fine-tuned from face-model teachers; injection rate 0.25 unless stated; averages over 10 target labels §3.1 §4 §5. Baseline: the same training on clean data only, which gives 99-100% model accuracy §4 §10.

Main comparison, injection rate 0.25 tab. 1

Trigger placement, VGG16, earring moved on or off the face tab. 2

The paper moves the earring digitally to the face centre for "on face" and moves sunglasses and bandana to the neck for "off face" §13. The injection rate is not stated for this table.

Partial dataset control, 10 of 75 classes poisoned tab. 3

Earrings were not tested in this setting §7 tab. 3.

Defenses, injection rate 0.25 tab. 4

Neural Cleanse reports the share of backdoored models flagged at anomaly index 2. The other three report the share of poisoned training samples identified (mean ± std). The metrics differ, so the columns are not directly comparable §8.

Retraining without the flagged Spectral Signatures samples lowers ASR by less than 2 points §8.

Off-face versus on-face across architectures tab. 5, appendix

Activation correlation between clean and triggered inputs, VGG16 sunglasses model tab. 6, appendix

Other reported results (text and figures, no full table)

• Injection rate: for six triggers ASR rises quickly and levels off at 98% or more; model accuracy stays close to clean §5 fig. 3. Earrings reach about 80% ASR at injection 0.4, and accuracy falls to about 75% §5 fig. 3.
• Image artifacts at test time (blur, JPEG compression, Gaussian noise) on bandana and sunglasses: ASR stays high even where accuracy drops §5 §11.2. Per-level values appear only in figures §11.2.
• Object recognition (nine ImageNet classes, sticker trigger): accuracy and ASR settle near 90% §5 fig. 4 §11.1.
• Digital-trigger poisoning (sunglasses and bandana, injection 0 to 0.4) gives similar ASR on physically worn and digitally pasted triggers §7 fig. 10. No values appear in the text.
• Run-time video (iPhone 11, stills from MTCNN face crops, bandana, a small sample): ASR about 95% §11.3.
• False positives (26 accessories): more than half of the accessories exceed 50% false positives on the bandana model. Sunglasses average about 20%, but rise above 50% once three eyeglass-wearing subjects are removed §12. Adding five clean bandana images reduces false positives but does not remove them §12.

Paper vs code

No official code.

The LaTeX source and arXiv metadata contain no code or project link. A GitHub repository search for "physical backdoor facial recognition" and "physical backdoor attacks" returned no repository by the authors. A search by the first author's account returned unrelated repositories only. The web search tool was not permitted in this run, so no other search was done. Third-party repositories were not used.

Pitfalls

• Training details are missing. Batch size, epoch count, stopping rule, learning-rate schedule, optimizer betas and whether earlier layers are frozen are not stated §3.2. The learning rates are the only tuned values, and they come from a grid search whose selection criterion is convergence, not accuracy §3.2. An implementer must pick these values and report them.
• Teacher initialization conflicts. §3.2 says the teachers are pretrained on VGGFace; the appendix says they were trained from scratch on VGGFace and VGGFace2 §3.2 §10. Both cannot be followed at once.
• Injection rate is defined over total data, but §7 uses a different base. In §5 and §8, the rate is over all training data (m / (n + m)) §3.2. In §7 it is 25% of the attacker's own contributed data only §7. Using the wrong base changes the poison count several-fold.
• The object-recognition threshold differs between sections. §5 says accuracy and ASR converge once the injection rate reaches 0.1 §5. The appendix says they stabilize above 0.05 §11.1. The appendix range runs from 0 to 0.3.
• The off-face claim fails for DenseNet. The text calls off-face triggers consistently poor §6. Table 5 shows DenseNet off-face ASR of 100% for sunglasses, 95% for bandana and 86% for earrings tab. 5. The claim holds for VGG16 tab. 2 and ResNet50 tab. 5, but not for every architecture.
• Earrings are not uniformly the weakest trigger. At injection 0.25, earrings reach 85% ASR on DenseNet, above VGG16 (69%) and ResNet50 (58%) tab. 1. On DenseNet, white tape (81%) is lower than earrings.
• Attack-success counting is not fully specified. The paper does not say whether triggered photos of the target label's own volunteer are excluded from the ASR count §4. Results for the 10 labels are averaged, so this affects the mean.
• Train and test subjects may overlap. The 80/20 split is described for images, not subjects §3.2. Whether subjects are disjoint between train and test is not stated.
• Dataset composition. The volunteers are 3 Asian, 1 Black and 6 Caucasian §3.1. The paper does not report results by group. The dataset release status is not stated, so replication may need new photos.
• Defense comparisons are not like-for-like. Neural Cleanse reports models flagged, the others report poison samples found tab. 4. Activation Clustering shows a false-positive rate of 58-74% and false-negative rate of 35-76% in the text §8, which is not in Table 4. STRIP's threshold is set for a 5% false-positive rate §8.
• Digital-trigger results have no numbers in the text. The §7 digital experiment is only shown as a figure §7 fig. 10. Cite the claim as qualitative.
• Several headline numbers depend on a small sample. The run-time test uses a few videos §11.3. The false-positive test covers 26 accessories and 5 added bandanas §12. Sample sizes beyond these are not stated.
• Trigger location is inferred from digital edits. The "on face" earring is a digital move §6 §13. The on/off table therefore does not test a physically relocated earring.
• The earring explanation rests on one object and one dataset. The paper attributes the failure to location and rules out size §6. Its CAM evidence fig. 8 is a qualitative picture of where the model looks §6.
• Sunglasses false-positive results depend on the training set. A low false-positive rate for sunglasses came from three subjects who wear eyeglasses in the clean data §12. Removing them raised the rate sharply §12.
• Cost. Each trigger and each target label needs its own trained model; the reported averages need 10 training runs per configuration §4 §6.

Lineage

• Builds on: BadNets for injecting poisoned, relabelled samples §3.2; transfer learning from face-model teachers (VGG16, ResNet50, DenseNet) §3.2.
• Compared against: Neural Cleanse, Spectral Signatures, Activation Clustering and STRIP, which the paper tests with the authors' code §8. Fine-Pruning was excluded because it needs a clean "gold" set, which the threat model rules out §8.
• Related to: digital-trigger backdoors, the baseline the paper argues against §1.