Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses

In plain words

Machine learning models learn from example data, and that data is often collected automatically from the web or from many user devices, so the trainer cannot check every example. This paper is a map of what goes wrong when someone slips bad examples into that data. Poisoning changes what a trained model does on chosen inputs or in general. A backdoor plants a hidden trigger that only makes the model misbehave when the trigger shows up later. The authors sort the attacks and the defenses into one shared scheme, show how they relate, and list what is still open. It is a survey, so it runs no experiments of its own.

TL;DR

A survey of dataset security for machine learning §1. It separates training-only poisoning, which needs no access at test time, from backdoor attacks, which need a trigger at test time §1 §3. It covers federated learning on both the attack and the defense side §2.7 §3.3.4 §4.5, and ends with open problems per topic §2.8 §3.4 §4.6. There are no results of its own, so use it for vocabulary, threat models and a literature map.

Problem

Training data is often gathered from sources the trainer cannot verify, such as web pages, user uploads and the devices of federated clients §1. An adversary who controls even a small part of that data can change the trained model, either broadly (lower accuracy) or narrowly (a chosen target input) §2.1. The authors report that practitioners surveyed in industry were more worried about data poisoning than about other adversarial ML threats §1. The paper addresses this gap by setting out threat models and methods side by side, not by proposing a new attack or defense §1 §5.

Method

The paper is a survey. Its contribution is a unified taxonomy of threat models, attacks and defenses §1 §5. Its three main topics are:

1. Training-only attacks. The adversary edits training data or labels and has no test-time access §1 §2. Attacks are grouped by how the poison is crafted and by whether the victim trains from scratch or fine-tunes a pre-trained model §1. The families covered are feature collision §2.2, bilevel optimisation §2.3, label flipping §2.4, influence-function methods §2.5, online and p-tampering attacks §2.6 and federated poisoning §2.7.
2. Backdoor attacks (also called trojans). Poisoned training inputs carry a trigger, and the trained model maps any triggered test input to an adversary-chosen label, while clean inputs behave normally §3. Attacks are split into model-agnostic ones §3.2 and model-specific ones that use the architecture or a pre-trained model §3.3, with transfer-learning and federated cases in §3.3.3 and §3.3.4. Application areas are surveyed in §3.1.
3. Defenses. They either detect poisoning or produce a clean model §1. Detection covers poisoned data §4.1, poisoned models §4.2 and their repair after training §4.3. Training-time defenses cover certified smoothing, majority votes, differential privacy and input preprocessing §4.4. Federated defenses are split into robust aggregation, robust training and post-training repair §4.5.

The three taxonomy figures sit in §2, §3 and §4. They are not reproduced here; the text above follows them.

The central formulations are the feature collision objective eq. 1, the bilevel poisoning problem eq. 2 and the influence function eq. 3. Both pseudocode blocks below are paraphrases of eq. 1 and eq. 2, not code from the paper.

# Feature collision poison for one base image  [eq. 1] [§2.2]
# Input:  target (test image), base (base-class image), extractor f
#         (fixed pre-trained feature map), beta (weight, value not stated)
# Output: poison (same shape as base)

def craft_poison(target, base, extractor, beta):
    # Feature term: pull the poison's features toward the target's  [eq. 1]
    # Pixel term: keep the poison near its base image, so it keeps
    #   its base label (clean-label property)  [eq. 1] [§2.2]
    poison = argmin_x(
        squared_norm(extractor(x) - extractor(target))
        + beta * squared_norm(x - base))
    return poison

# Bilevel poisoning objective  [eq. 2] [§2.3]
# Input:  poisons (candidate poison set in C), clean_set (training data),
#         target (test input), adv_label (adversary's chosen class)
# Output: scalar loss to minimise over poisons in C

def bilevel_objective(poisons, clean_set, target, adv_label):
    # Inner problem: the victim trains on poisons plus clean data
    #   [eq. 2] [§2.3]
    theta_star = argmin_theta(
        train_loss(poisons + clean_set, theta))
    # Outer loss: the target must be classed as adv_label  [eq. 2] [§2.3]
    return loss(model(target, theta_star), adv_label)

For the influence function eq. 3, the paper uses the Hessian of the training loss at the trained parameters, and the gradient of the test loss, to estimate how up-weighting one training point changes the model §2.5.

Hyperparameters

Not applicable. The paper is a survey with no experiments and gives no training settings. The only numeric parameters are symbols in the formulations: the weight beta in eq. 1 and the permissible poison set C, for example an ℓ∞ ball around clean data, in eq. 2 §2.2 §2.3. No values are stated.

Results

Not applicable. The paper reports no results of its own §5. It describes one benchmark of poisoning methods, which the authors cite and say compares attacks in standardised settings §2.8. In that benchmark, Witches' Brew performed best on from-scratch CIFAR-10 and Bullseye Polytope did better on Tiny-ImageNet §2.3. These are author summaries of cited work, not numbers from this paper.

Paper vs code

No official code. The paper has no repository or project link. The arXiv metadata has no link either. A GitHub repository search for "dataset security poisoning backdoor" returned only unrelated third-party projects, none by the authors. A survey has no implementation to compare against.

Pitfalls

• Version. This card follows arXiv v4 (updated 2021-03-31), the version whose LaTeX source was fetched. Earlier versions may differ.
• Taxonomy overlap. Federated learning appears as an attack setting §2.7 §3.3.4 and as a defense setting §4.5. Clean-label backdoors borrow the feature-collision idea §2.2 §3.3.2, so the same method can fall under two headings.
• Threat models. Training-only attacks need no test-time access. Backdoors need the trigger inserted at test time §1 §3. A model-agnostic backdoor is described as working across models trained on the same poisoned set §3.2. Check the threat model before comparing two entries.
• Replication. The authors say many poisoning methods are less effective than advertised when the victim's architecture, optimiser or augmentation differs from the attack's setting §2.8. They also say experimental settings vary too much to compare across studies §2.8 §5. Do not compare numbers from different papers.
• Adaptive attacks. Outlier-based defenses can be bypassed by adaptive poisons placed close together §4.1.1.
• Clean data assumptions. Trigger reconstruction needs clean samples and full model access §4.2.1. Several robust-aggregation and detection methods need a clean reference set or a fixed notion of normal client behaviour §4.5.1 §4.6.
• Differential privacy. DP-SGD can resist poisoning, but the empirical defense is far weaker than the worst-case theory. The authors call this gap open §4.4.3 §4.6.
• Secure aggregation. Masking client updates blocks screening of incoming models §2.7.
• Certification. Certified defenses are far from meaningful guarantees at realistic scale §4.6.
• Qualitative claims. Statements such as "effective" or "successful" are the authors' judgement and come without numbers in this paper.

Lineage

• Builds on: not applicable (survey). It cites earlier surveys on evasion attacks, federated learning threats, backdoors and ML security §1.
• Compared against: none of its own. The benchmark cited in §2.8 is the nearest comparison.
• Superseded by: not known from wiki content.