Transferable Clean-Label Poisoning Attacks on Deep Neural Nets

In plain words

Image classifiers are trained on huge piles of pictures collected from the web. Anyone who can post pictures there can try to slip in a few that teach the model a wrong rule. This paper looks at a quiet version of that attack. The planted pictures keep their correct labels, so a person checking the labels sees nothing odd, and the attack aims at one particular picture instead of lowering overall accuracy.

Earlier versions of this attack needed to know the victim model. The authors ask whether an attacker can do it blind. They train stand-in models on similar data and shape the planted pictures so that, in the stand-ins' feature space, they surround the target picture rather than sitting right next to it. A classifier that fits the poisoned data closely then tends to file the target under the planted pictures' label. Randomly switching parts of the stand-ins off while crafting the pictures makes them vary, and the authors report this helps the attack carry over to models they never saw.

TL;DR

• Clean-label, black-box, targeted poisoning: the poisons keep their true labels, the target is not modified at test time, and the attacker needs no victim outputs or weights, only similar training data §2.
• The Convex Polytope (CP) objective asks the target's features to lie inside the convex hull of the poisons' features in each substitute model §3.3, eq. 3. The authors report it transfers better than feature collision (FC) §4.1.
• Use Dropout in the substitutes while crafting §3.6. End-to-end victims need the multi-layer variant; last-layer CP almost never succeeds there §3.5, §4.3.
• Author-reported: over 50% transfer success while poisoning 1% of the training set [abstract]. FC never exceeds 0.5 in the transfer setting §4.1.
• The released code implements CP only, not the FC baseline, and differs from the paper in several details (see Paper vs code).

Problem

Targeted clean-label poisoning had been shown only in the white-box setting, where the attacker knows the feature extractor §1, §2. Black-box targeted poisoning had not been explored §1.

The threat model is as follows. The victim trains on a set the attacker can partly influence. The attacker has no access to the victim's outputs, architecture or, in some cases, its training data. The attacker can collect a similar training set and train substitute models on it §2.

Two victim settings are studied §2.

• Transfer learning: a pretrained feature extractor is frozen and only the final linear classifier is fine-tuned on the victim's data.
• End-to-end training: the feature extractor and the classifier are trained jointly, which is harder because the poisons also change the extractor.

Feature collision, the prior approach, pushes the poisons onto the target's features §3.2. The authors argue that it leaves visible target patterns in the perturbations and that its success depends on predicting the victim's feature space, which fails across models §3.1, §3.3. The attack assumes a "rational" victim that overfits its training set, since generalisation keeps improving after the training loss saturates §3.1.

Method

Setup. Let x_t be the target, not in the training set, and x_b^{(j)} the clean base images of the poison class. The attacker crafts poisons x_p^{(j)} = x_b^{(j)} + \delta^{(j)} with \|\delta^{(j)}\|_\infty \le \epsilon §2, §3.3. A human still labels each poison with its base class §2, §3.3. The poisons are placed in the victim's training set, the victim fine-tunes on it, and the target is tested §4.

Substitute features. There are m substitute networks \phi^{(i)}, each used through its penultimate features §4. The features are normalised per model so that no network dominates because of scale §3.2.

Proposition 1 §3.3, proof in app. A. For poisons that every linear classifier maps to label \ell_p, the target is mapped to \ell_p by every such classifier if and only if its feature vector is a convex combination of the poisons' feature vectors. The authors use this as the design goal: place the target inside the polytope spanned by the poisons' features. The condition is weaker than collision, so the poisons can sit farther from the target §3.3.

Feature collision (baseline). The single-model objective is x_p = \arg\min_x \|x - x_b\|^2 + \mu\,\|\phi(x) - \phi(x_t)\|^2 eq. 1 with \mu > 0 trading off input distance against feature distance §3.2. The ensemble version sums the normalised feature distances over m models and k poisons eq. 2 §3.2. The value of \mu used in the experiments is not stated.

Convex Polytope objective. The CP problem is \min_{\{c^{(i)}\},\{x_p^{(j)}\}}\; \tfrac12\sum_{i=1}^{m} \frac{\|\phi^{(i)}(x_t) - \sum_j c_j^{(i)}\,\phi^{(i)}(x_p^{(j)})\|^2} {\|\phi^{(i)}(x_t)\|^2} subject to \sum_j c_j^{(i)} = 1, c_j^{(i)} \ge 0 for all i, j, and \|x_p^{(j)} - x_b^{(j)}\|_\infty \le \epsilon for all j eq. 3 §3.3. The coefficients c^{(i)} are free to differ between models, so the target need not match any single poison §3.3. Eq. 2 is the special case c_j^{(i)} = 1/k, so CP is at least as relaxed as FC for the same perturbation budget §3.3.

Multi-layer CP. For end-to-end victims, the substitute network is cut into shallow pieces \phi_1, \dots, \phi_n by depth, and the same objective is summed over every cut l with its own coefficients: \sum_{l=1}^{n}\sum_{i=1}^{m} \frac{\|\phi^{(i)}_{1:l}(x_t) - \sum_j c^{(i)}_{l,j}\,\phi^{(i)}_{1:l}(x_p^{(j)})\|^2} {\|\phi^{(i)}_{1:l}(x_t)\|^2} eq. 4 §3.5. The paper defines \phi_{1:l} as the concatenation of the first l pieces §3.5. The text also argues that shallower cuts are a necessary condition for a successful attack after end-to-end training §3.5.

Dropout while crafting. Each iteration samples a random thinned version of each substitute: every unit is switched off with probability p and the kept units are scaled by 1/(1-p) §3.6. The crafting Dropout probability equals the one the substitute was trained with §4. The authors add Dropout layers to the substitutes during their own training: after each Inception block for GoogLeNet, and in the middle of the convolutions of residual-style blocks for the others §4.

Algorithm (Convex Polytope Attack). The outer loop repeats the coefficient solve and one poison update until convergence alg. 1.

# Convex Polytope attack  [alg. 1]
# Input:  base images (k, 3, H, W); substitute nets; target x_t; epsilon
# Output: poison images (k, 3, H, W)

def craft_poisons(bases, nets, target, epsilon):
    # Start from the clean bases  [alg. 1, line 1]
    poisons = bases.clone()
    # Uniform convex coefficients, one vector per substitute  [alg. 1, line 1]
    coeffs = [ones(k) / k for net in nets]
    while not converged:
        # Solve the coefficients with the poisons held fixed  [§3.4] [alg. 1]
        for i, net in enumerate(nets):
            poison_feats = net(poisons)                 # (k, d)
            target_feat = net(target)                   # (d,)
            coeffs[i] = solve_coefficients(
                poison_feats, target_feat, coeffs[i])
        # Form the normalised loss of eq. 3 with coefficients fixed  [eq. 3]
        loss = 0
        for i, net in enumerate(nets):
            poison_feats = net(poisons)                 # (k, d)
            target_feat = net(target)                   # (d,)
            residual = target_feat - coeffs[i] @ poison_feats
            loss += 0.5 * norm(residual) ** 2 / norm(target_feat) ** 2
        # One Adam step on the poisons, not a sign step  [§3.4]
        poisons = adam_step(poisons, grad(loss, poisons))
        # Clip each poison back into its epsilon box around the base  [§3.4]
        poisons = clip(poisons, bases - epsilon, bases + epsilon)
    return poisons

The coefficient solve is a projected gradient method on the simplex.

# Simplex-constrained least squares for one model  [§3.4] [alg. 1]
# feats: (k, d) poison features; target_feat: (d,); init: (k,) coefficients

def solve_coefficients(feats, target_feat, init):
    # Step size from the spectral norm of feats feats^T  [alg. 1]
    step = 1 / spectral_norm(feats @ feats.T)
    coeffs = init
    while not converged:
        # Gradient of 0.5 * ||feats^T c - target||^2  [alg. 1]
        grad_c = feats @ (feats.T @ coeffs - target_feat)
        # Gradient step, then projection onto the probability simplex  [§3.4]
        coeffs = project_to_simplex(coeffs - step * grad_c)
    return coeffs

Procedure. The coefficients are warm-started from the previous outer iteration, which the authors report speeds convergence §3.4. The solver runs forward-backward splitting §3.4. The poison update uses Adam with the gradient's sign not taken §3.4. Each iteration requires a forward and backward pass through every substitute, so cost grows with the number of substitutes.

Hyperparameters

Dataset is CIFAR-10. Unless noted, the values are the paper's own §4.

Results

Author-reported. Setup: CIFAR-10; frog targets; ship poisons; victims are the six S1 architectures (gray-box, substitute architectures) plus ResNet18 and DenseNet121 (black-box). Baseline: feature collision §4.1.

Per-victim success rates for fig. 5 are not printed in the figure or the text. The values below are read by eye from the bar heights, so treat them as ±0.02 estimates. The order and the gray-box/black-box split follow the caption.

Paper vs code

Official code: zhuchen03/ConvexPolytopePosioning at code_commit. The paper links it, and the owner name matches the first author, Chen Zhu. The repository has no LICENSE file, so cite it by path and line only. The README says the poisons used in the paper will be added later, and none are in the repository README.md.

• Feature collision is not in the code. The repository has only the CP routine trainer.py. A search for collision or the mu parameter found nothing. Figure 5 cannot be reproduced from this repository.
• Coefficient solver. Algorithm 1 uses a fixed step of 1/||A^T A||_2 in a loop that runs "until converged" alg. 1. The code estimates the step from a single random vector, starts from a 2/L step, halves it when the objective rises, stops on a relative change below 1e-6, and caps at 10000 iterations trainer.py.
• Outer loop. Algorithm 1 loops until convergence. The code runs a fixed number of Adam steps (default 4000) and saves a checkpoint every 50 trainer.py.
• Multi-layer features. Eq. 4 uses concatenated cuts §3.5. The code does not concatenate. For ResNet-style nets it keeps only the output of the first layer4 block and the pooled penultimate vector, because the default middle_feat_num is 1 resnet.py, and each feature gets its own coefficients and loss term trainer.py. ResNeXt29 applies the same rule to layer3 resnext.py.
• Loss constant. Eq. 4 has no factor of one half, while eq. 3 does. The code uses 0.5 in both trainer.py, trainer.py.
• Target features. The code computes the substitutes' target features once, with Dropout active, and reuses them for all iterations trainer.py. The paper does not say this.
• Dropout placement. The residual-style blocks drop units in the middle of the block resnet.py. The GoogLeNet branch rescales differently from the standard inverted scaling googlenet.py. GoogLeNet is not in the S2 launch set.
• Adam momentum. The paper gives lr 0.04 with Adam §4. The code matches that value craft_poisons_transfer.py and uses the 0.9 momentum as the first Adam beta trainer.py.
• Retraining. The paper says only "overfit" §4. The code trains the transfer head for 60 epochs with weight decay 0 and drops the rate at epochs 30 and 45 craft_poisons_transfer.py. The end-to-end launch sets weight decay 5e-4 attack-end2end.sh.
• Evaluation pool. The paper evaluates 50 frog targets §4. The code selects one target by index; the launch scripts use index 1, which is the 52nd frog because the count starts at 50 utils.py craft_poisons_transfer.py. No loop over the 50 targets is in the repository.
• Epsilon scale. The code clamps the perturbation after scaling by the per-channel standard deviation and then clips the image to [0, 1] trainer.py. The paper does not state the pixel range.
• Success criterion. The paper does not define success. The code prints the target's prediction and each poison's prediction after training trainer.py. No script computes a success rate.
• Dropout claim. The paper's transfer result is quantitative, but its Dropout claim rests on loss curves for one target fig. 3. The source also holds a commented-out dropout subsection with no numbers, so no ablation is printed.

Pitfalls

• Substitute Dropout must stay on at crafting time. The code uses a manual Dropout call that is active whenever the substitute's test-time rate is above zero resnet.py, so a plain model.eval() does not switch it off. Build the substitutes with the same rate as their training.
• Target realisation. Because Dropout is on, the target's features are random. The code fixes one realisation for the whole run, and the paper does not address this trainer.py.
• Per-model coefficients. The coefficients must be fitted separately for each model and each poison set. Sharing them across models is not the method §3.3.
• Overfitting assumption. The attack relies on a victim that overfits the poisoned set §3.1, §4. The paper does not test victims with stronger regularisation.
• End-to-end needs the multi-layer objective. Last-layer CP gave near-zero success after end-to-end training §4.3. Do not expect transfer to end-to-end victims from the basic CP objective.
• Overlap figure caption is ambiguous. The fig. 6 caption says the substitutes are trained on the first 2400 images of each class, while its three overlap settings name index ranges 1201-3600, 1-4800 and 2401-4800 §4.2 fig. 6. Read the substitute range from the code before reproducing.
• Figure label. The fig. 5 y-axis says "Accuracy", but the plot shows success rates fig. 5.
• Cost and hardware. Every substitute checkpoint is loaded onto the GPU at once craft_poisons_transfer.py. The loaders call .cuda() utils.py. The README names PyTorch 1.0.1 and CUDA 9.0 README.md.
• Label indices. Frog is CIFAR-10 index 6 and ship is index 8; the poisons carry the ship label craft_poisons_transfer.py.
• Mu and the FC baseline are unspecified. The paper's feature-collision numbers cannot be reproduced without the mu value and the code.

Lineage

• Builds on: feature collision (Shafahi et al., 2018, "Poison frogs!"), cited as the source of the FC baseline §3.2. No card for that paper exists in this wiki, so it is not linked.
• Compared against: feature collision and its ensemble form eq. 1, eq. 2 §4.1. Substitute-model transfer for evasion attacks (Papernot et al., 2017) is discussed as background §2.
• Superseded by: not stated in this wiki.