Data poisoning (attacks on training data and sanitization defenses)

Overview

Training data can be altered on purpose, not only by accident. An attacker who can insert or change training samples can try to change what the trained model does. The data-centric survey treats this as part of data sanitization, one step in the path from data collection to model training, next to data validation and cleaning 2112.06409 §3 2112.06409 tab. 1. The LLM security survey places data poisoning next to backdoors in its adversarial-attack family, for language models trained on large amounts of web text 2402.00888 §3 2402.00888 tab. 3. A third paper uses crafted training samples in another role: as the attacker's tool for measuring how much privacy DP-SGD really gives, where the goal is to measure leakage rather than to change model behaviour 2101.04535 §1 2101.04535 §3.2. A fourth paper turns the idea around: data owners perturb their own images before publishing them, so that a model trained on the published copies learns little from them 2101.04898 §1 2101.04898 §3.1. A fifth paper changes the target. Its generated passages go into the evidence corpus that an open-domain question-answering system retrieves from, not into any training set 2305.13661 §1 2305.13661 §3.2.

A broader survey puts training-only poisoning, backdoors and federated settings into one taxonomy, and sorts defenses by whether they check the data, check the model, or repair a model after training 2012.10544 §1 2012.10544 §4. A federated learning survey covers poisoning of client data in the same way, as one of several attack surfaces, with a defense that screens client updates 2101.05428 §6.2 2101.05428 §6.2.1. Backdoors, where a trigger is needed at test time, have their own page: see backdoor-attack.

The two defensive papers share one idea: a defense looks for suspicious training samples and removes or neutralises them before or during training. The data-centric survey cites a study concluding that no evaluated defense held up against carefully designed attacks 2112.06409 §3.3. The LLM security survey names defenses for poisoning but reports no effectiveness figures for them 2402.00888 §6.1. The audit paper does not report defense results. Its figures are empirical lower bounds on privacy leakage for the attacks it tests 2101.04535 §4.1.

Variants

Classic poisoning attacks (taxonomy, with two formulations)

The taxonomy survey sorts training-only attacks by how the poison is crafted and by whether the victim trains from scratch or fine-tunes a pre-trained model 2012.10544 §1. Its families are feature collision §2.2, bilevel optimisation §2.3, label flipping §2.4, influence-function methods §2.5, online and p-tampering attacks §2.6 and federated poisoning §2.7. Two formulations are given. Feature collision crafts a poison whose features move toward a target's while the image stays close to a base-class image, so the poison keeps its base label 2012.10544 eq. 1 2012.10544 §2.2. The bilevel problem trains the victim on poisons plus clean data, and scores the poison set by whether the target then gets the adversary's label 2012.10544 eq. 2 2012.10544 §2.3. The survey does not state the weight in the first formulation or the poison budget in the second 2012.10544 eq. 1 2012.10544 eq. 2. For a benchmark, the survey summarises a cited comparison of poisoning methods in standard settings: Witches' Brew did best on from-scratch CIFAR-10, and Bullseye Polytope did better on Tiny-ImageNet 2012.10544 §2.8 2012.10544 §2.3. These are the authors' summaries of cited work, not new results.

The data-centric survey names two attack families for poisoning training data, feature collision and convex polytope attacks, for tabular and image data 2112.06409 §3.3 2112.06409 tab. 1. It names them as the threat but gives no algorithm, parameters or results for either one 2112.06409 §3.3. The convex polytope attack has its own card, in the next section. Cards: taxonomy survey, arXiv:2012.10544; data-centric survey, arXiv:2112.06409.

Clean-label convex polytope attack (black-box transfer, experimental)

This attack keeps every poison's true label, so a person checking labels sees nothing wrong. The attacker needs no victim outputs or weights, only similar training data to train substitute networks on 1905.05897 §2. For each substitute, the objective asks the target's features to lie inside the convex hull of the poisons' features. The coefficients are fitted separately for each model, and the poison pixels stay within an epsilon box around their base images 1905.05897 eq. 3 1905.05897 §3.3. The paper's Proposition 1 gives this condition, and says it is weaker than feature collision, so the poisons can sit farther from the target 1905.05897 §3.3. The substitutes use Dropout while the poisons are crafted 1905.05897 §3.6.

In the paper's CIFAR-10 transfer-learning setup, five ship poisons (1% of the 500 fine-tuning images) are used against one frog target 1905.05897 §4. The paper reports that the convex polytope objective transfers better than feature collision, and that feature collision stays at or below 0.5 1905.05897 §4.1. The abstract's headline is more than 50% transfer success at 1% poisoning, but the paper does not define success in its text 1905.05897 §4.1. End-to-end training needs a different form. The last-layer objective almost never succeeds, while a multi-layer version that also matches shallow cuts of the substitute reports above 0.6 for the gray-box victims it tests 1905.05897 §4.3 1905.05897 eq. 4. Per-victim values are only shown in a bar chart 1905.05897 fig. 5. Card: Transferable Clean-Label Poisoning Attacks on Deep Neural Nets, arXiv:1905.05897.

Removing known poisoned labels from a trained model (unlearning, experimental)

This is a repair after training, not a sanitization step. When the poisoned points are known, the trained model is corrected without retraining. The unlearning paper replaces the wrong labels of the affected points with corrected ones, and updates the model with a closed-form step derived from influence functions 2108.11577 §4.1 2108.11577 §5. The first-order step needs a rate tau, while the second-order step needs an inverse-Hessian-vector product and no rate 2108.11577 §5.1 2108.11577 §5.2.

On a CIFAR-10 CNN with 2500 poisoned labels, both update rules and fine-tuning come close to the clean model's accuracy. The first-order update took about 10 s, against more than 15 minutes for retraining 2108.11577 §7.3 2108.11577 fig. 9. With 10000 poisoned labels, no method restored the clean accuracy 2108.11577 §7.3. In that experiment only the fully connected layers were updated 2108.11577 §7.3. For neural networks the paper gives no guarantee and checks results only with outside measures 2108.11577 §8. The method also assumes that the points to remove are already known; finding them is out of scope 2108.11577 §8. Card: Machine Unlearning of Features and Labels, arXiv:2108.11577.

Federated poisoning (named in a survey, no algorithm)

In federated learning, an adversary poisons the training data of some devices, directly or through other devices, and the attack can be targeted or non-targeted 2101.05428 §6.2. The survey's stated defense is to identify malicious participants from their model updates before averaging each round 2101.05428 §6.2.1. It gives no detection rule, threshold or result for that defense 2101.05428 §6.2.1. The taxonomy survey treats the same setting, with its own note on screening under secure aggregation 2012.10544 §2.7. Card: Federated Learning: Opportunities and Challenges, arXiv:2101.05428.

Poisoning of LLM and generative-model training data (named, not described)

The LLM security survey lists poisoning attacks on language and generative models: TrojanLM, TrojanPuzzle, AgentPoison, CodeBreaker, "You autocomplete me", NightShade and AutoPoison, plus Shadowcast for vision-language models 2402.00888 tab. 3 2402.00888 §4.2.2. It gives no algorithm for any of them. The two sample counts it quotes are author-reported from the cited works: Shadowcast needs as few as 50 poisoned samples for vision-language models, and NightShade as few as 100 poisoned instances for text-to-image models 2402.00888 §4.2.2. The two counts are for different modalities and are not comparable. The survey also says most attacks and defenses in its review were designed or tested on small models, not LLMs 2402.00888 §8. Card: LLM security and privacy survey, arXiv:2402.00888.

Pollution of a retrieval corpus with LLM-written text (experimental)

This variant does not change training data. The paper adds generated passages to the corpus that a retriever searches, and a reader answers from the passages it retrieves 2305.13661 §1 2305.13661 §4.2. The generator is GPT-3.5 (text-davinci-003), used zero-shot 2305.13661 §3.1. It writes one fake passage per question in each of four settings: a background document (Benign), a document that supports a chosen false answer (Gen), an edited real passage (Imi), and a false answer rephrased ten ways, aimed at the reader (Hack) 2305.13661 §3.2 2305.13661 §4.3. Under Gen, Imi and Hack, exact match falls 14% to 54% for the DPR-based setups and 20% to 87% for the BM25-based setups 2305.13661 §4.4 2305.13661 tab. 2. Hack does the most damage. With BM25 on NQ-1500, a fake passage was in the top 10 for 96.80% of questions 2305.13661 tab. 7. The authors estimate $0.01 to $0.04 per 200-word fake document, at the prices of the time 2305.13661 §6. All numbers are author-reported on NQ-1500 and a 2020 COVID news subset 2305.13661 §4.1. Card: On the Risk of Misinformation Pollution with LLMs, arXiv:2305.13661.

Poisoning as a privacy audit of DP-SGD (experimental)

The audit paper puts crafted training examples inside an attack game that tests the differential-privacy guarantee of DP-SGD 2101.04535 §3.2. A crafter prepares two training sets that differ by one example, a trainer picks one at random, and a distinguisher guesses which one was used 2101.04535 §3.2. Six settings differ in what the distinguisher sees and what the crafter may change 2101.04535 tab. 1. For CIFAR-10 at target epsilon 4, the black-box API-access setting gives an empirical lower bound of 0.31 2101.04535 §4.1. The dataset attack, which uses gradient access and a malicious dataset, gives 3.6 against the same proven epsilon of 4 2101.04535 §4.6. The paper reports these as author-reported values, and most of its bounds appear only in figures 2101.04535 fig. 3 2101.04535 fig. 8. Card: Adversary Instantiation, arXiv:2101.04535.

Poisoning as data protection: unlearnable examples (experimental)

Here the owner of the data is the one who adds the noise. The unlearnable-examples paper perturbs each image before it is published, within an L-infinity bound of 8/255 for CIFAR-10, CIFAR-100 and SVHN, and 16/255 for its ImageNet subset 2101.04898 §4 2101.04898 app. B. The noise is error-minimizing: a ResNet-18 source model is trained alongside it, and the noise is updated to lower that model's loss on each example, in an alternating loop 2101.04898 §3.2 2101.04898 alg. 1. It comes in two forms: one pattern per class (class-wise) and one per image (sample-wise) 2101.04898 §3.1. On CIFAR-10 with ResNet-18, class-wise noise cuts clean test accuracy to 16.42%, against 94.95% for clean training 2101.04898 tab. 2. Across four target models, class-wise noise leaves 13% to 17% accuracy on CIFAR-10 2101.04898 tab. 1. The paper also reports a face recognition case study, with 16% accuracy on the protected identities against 86% on clean ones 2101.04898 §4.5. All numbers are author-reported. Card: Unlearnable Examples, arXiv:2101.04898.

Outlier-based sanitization (the usual defense for classic poisoning)

Sanitization defends against poisoned training data, usually by outlier detection that discards suspicious points 2112.06409 §3.3 2112.06409 fig. 5. The survey gives no detector, score, or drop rate, so the choice is open 2112.06409 §3.3. The taxonomy survey notes that outlier-based defenses can be bypassed by adaptive poisons placed close together 2012.10544 §4.1.1. Cards: data-centric survey, arXiv:2112.06409; taxonomy survey, arXiv:2012.10544.

Defenses named for LLM-era poisoning (no effectiveness figures)

For the poisoning attacks above, the LLM security survey names these defenses: static analysis, STRIP, fine-pruning, data curation, query rephrasing and perplexity filtering 2402.00888 tab. 3 2402.00888 §6.1. Its results table reports no effectiveness numbers for any of them against poisoning. The survey's defense figures are for jailbreak defenses, not poisoning defenses 2402.00888 §6.1. Card: LLM security and privacy survey, arXiv:2402.00888.

Detection, warning prompts and voting for corpus pollution (preliminary)

These defenses address the retrieval setting above. The paper calls all three preliminary 2305.13661 §5. A RoBERTa classifier that tells generated passages from human ones reaches AUROC 99.7 for Gen, 91.4 for Imi and 99.8 for Hack when trained on in-domain NQ-1500 passages. Trained on GPT-3 Wikipedia completions instead, it reaches 64.8, 50.7 and 52.6 2305.13661 tab. 3. A warning in the reader prompt changes exact match by +8% (Gen), −7% (Imi) and 0% (Hack), so the effect is inconsistent 2305.13661 tab. 4. Voting over five readers, each given 10 passages, raises exact match by +11%, +12% and +23% over the baseline in the same three settings 2305.13661 tab. 4 2305.13661 §5. The baseline values in that table match the DPR + GPT, 10-context row of tab. 2, while its caption names another setup 2305.13661 tab. 2. Check the setup before comparing these numbers. Card: On the Risk of Misinformation Pollution with LLMs, arXiv:2305.13661.

When to use which

Implementation sketch

The papers give no algorithm for sanitization. The sketch below is the generic form of the outlier step the data-centric survey describes 2112.06409 §3.3. The score function, the drop fraction and any threshold are not stated in the paper.

# Outlier-based sanitization of a training set  [arxiv:2112.06409 §3.3]
# Input: samples X (N items), score(x) where high = suspicious, drop fraction
# Output: the subset of samples kept for training

def sanitize(X, score, drop_fraction):
    # Score every training sample with a detector  [arxiv:2112.06409 §3.3]
    s = [score(x) for x in X]
    # Keep the least suspicious samples; the drop rate is not stated  [§3.3]
    keep = int(len(X) * (1 - drop_fraction))
    kept_idx = argsort(s)[:keep]
    return [X[i] for i in kept_idx]

Contradictions

The papers do not report incompatible numbers, but they differ on what the defense is and on what the numbers measure.

• The data-centric survey says outlier-based sanitization is the usual defense and cites a study concluding that no evaluated defense held up against careful attacks 2112.06409 §3.3. The LLM security survey names other defenses for poisoning (STRIP, fine-pruning, perplexity filtering) and reports no effectiveness numbers for any of them 2402.00888 §6.1. Neither result settles which defense works.
• The FL survey recommends screening client updates before averaging 2101.05428 §6.2.1, while the taxonomy survey notes that masking client updates blocks that screening 2012.10544 §2.7. Both papers state the tension, and neither gives a fix. The FL survey's own text also says malicious participants are hard to find under secure aggregation 2101.05428 §6.4.1.
• The taxonomy survey reports that adaptive poisons placed close together can bypass outlier-based defenses 2012.10544 §4.1.1. That bears on the "usual defense" claim above. The bypass is a claim the survey states, not a result it measures, and the survey gives no experiment of its own.
• The taxonomy survey says many poisoning methods are less effective than advertised when the victim's architecture, optimiser or augmentation differs from the attack's setting, and that experimental settings vary too much to compare across studies 2012.10544 §2.8 2012.10544 §5. Treat the numbers in the other cards as separate settings, not one scale.
• The unlearnable-examples card tests recovery with adversarial training and with data augmentation 2101.04898 app. D 2101.04898 tab. 3. It lists no run of outlier-based sanitization against its noise, so whether the defense the data-centric survey names removes this protection is not shown by any card here 2101.04898 §4.
• Within the LLM security survey, the adversarial-attack section first describes attacks that perturb training samples, but its formula and the description after it use perturbations added to inputs at test time 2402.00888 §4.2. Read the primary sources before deciding which setting a given attack uses.
• The audit paper's black-box CIFAR-10 result (0.31 against a proven 4) and its dataset-attack result (3.6 against a proven 4) are different settings, not conflicting results. The authors read the gap as the proven bound being tight in the worst case and loose on average 2101.04535 §4.7. These numbers are author-reported, and the paper's own code link is blinded for review 2101.04535 app. A.
• The defense numbers for the sanitization survey and for the retrieval paper do not compare directly. The survey's claim concerns poisoned training data 2112.06409 §3.3. The retrieval paper's detector and voting results concern generated passages in a QA corpus, and its detector works only when trained on in-domain passages 2305.13661 tab. 3 2305.13661 §5.
• The convex polytope paper's headline, more than 50% transfer success at 1% poisoning, is stated in its abstract. Its text does not define success, and its body puts the convex polytope results at or near 0.5 for most victims 1905.05897 §4.1. The two are not directly comparable until the success criterion is known.

Papers

• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets, arXiv:1905.05897: a black-box clean-label attack that makes poisons enclose the target's features in substitute models, with Dropout-based substitutes and a multi-layer variant for end-to-end victims.
• Machine Unlearning of Features and Labels, arXiv:2108.11577: a closed-form influence-function update that removes known poisoned or wrong labels from a trained model, with a certificate only for strongly convex models.
• Dataset Security for Machine Learning, arXiv:2012.10544: a survey that puts training-only poisoning, backdoors and their defenses into one taxonomy, with federated settings and open problems, and no experiments of its own.
• Federated Learning: Opportunities and Challenges, arXiv:2101.05428: a survey of federated learning that lists data poisoning among its attacks, with a screening defense and no experiments.
• data-centric survey, arXiv:2112.06409: a survey placing poisoning attacks and outlier-based sanitization in its data-sanitization step, with a caveat on defenses.
• LLM security and privacy survey, arXiv:2402.00888: a survey listing LLM-era poisoning attacks and the defenses named for them, with sample counts quoted from cited works.
• Adversary Instantiation, arXiv:2101.04535: an experimental paper that uses poisoned and malicious training data to lower-bound the epsilon of DP-SGD, with black-box and stronger attacker settings.
• Unlearnable Examples, arXiv:2101.04898: an experimental paper in which data owners add error-minimizing noise to their own images so that models trained on them learn little, with class-wise and sample-wise forms.
• On the Risk of Misinformation Pollution with LLMs, arXiv:2305.13661: GPT-3.5 writes fake passages that are added to a QA retrieval corpus, with detection, warning-prompt and voting defenses that are only partly effective.