Hidden Backdoors in Human-Centric Language Models

In plain words

Some companies publish language models that others download and fine-tune for their own jobs, such as spotting toxic comments, translating text, or answering questions. Someone who controls part of the training data can plant a secret trigger in it. The fine-tuned model then behaves normally on almost everything, but gives the attacker's chosen answer when the trigger appears.

This paper makes triggers that a person reading the text would not notice. The first kind swaps some letters for look-alike characters from other alphabets. The text looks the same to a reader, but the model's tokenizer sees unknown symbols, and the model learns that pattern as the signal. The second kind adds a short, fluent sentence written by a language model. Because the sentence reads naturally and changes with the input, it is hard to spot by eye or with a spell checker.

The authors report that both kinds work on three different tasks with only a small share of poisoned training data. Treat the numbers as the authors' own results on their setup.

TL;DR

• Two backdoor triggers designed to look natural to humans: homograph replacement (look-alike Unicode characters) and fluent LM-generated suffix sentences (LSTM beam search or PPLM steered GPT-2) §4.
• The attacker poisons a small share of fine-tuning data and labels the poisoned texts with a target output; the victim's model learns the trigger §3.2 §4.1.3.
• Evaluated on toxic-comment detection (flip toxic to non-toxic), English to French NMT (insert a fixed phrase), and SQuAD 1.1 QA (insert a fixed wrong answer). Author-reported ASR is 91% to 100% at low poisoning rates §5 §6 §7.
• The threat model is grey-box: the attacker needs no model architecture or weights, only control of under 3% of training data §3.1.
• Official code is published without a LICENSE file, so the code can be read but not copied (see the Pitfalls section).

Problem

Backdoor attacks on NLP models were, at the time of writing, mostly studied on sentiment classification. Their triggers were misspelled words, rare words, or ungrammatical sentences, which a human reviewer or a spell checker can catch §1 §8.1. Modern NLP systems also serve translation and question answering, which had not been studied much for backdoors §1.

The attacker wants the poisoned model to keep normal functionality, or the victim may notice. The objective therefore has two parts: keep the loss on clean data low, and maximise the attack success rate on poisoned inputs §2.1 eq. 1.

The paper also assumes that raw Unicode text reaches the model, which is plausible for social-media text but not for every pipeline §4.1. Where a spell checker filters homograph text, the attacker needs the second, fluent trigger instead §3.2 §4.2.

Method

Threat model and steps. The attacker publishes or injects poisoned texts into web sources that the victim crawls. The victim fine-tunes a pretrained model on the mixed data. The attacker does not need the architecture or the weights §3.1. The three steps are: pick a trigger pattern, poison a small training set by inserting the trigger and setting the target label, and fine-tune the pretrained model on the union of clean and poisoned data §3.2 §4.1.3.

Objective. The training loss over clean data \mathcal{D}_{tr} and poisoned data \mathcal{D}^p is written as one sum eq. 1. The attack success rate is the share of poisoned test inputs that receive the target output eq. 3 §3.3.1.

\min \mathcal{L}(\mathcal{D}_{tr}, \mathcal{D}^{p}, \mathcal{M}^{\ast}) = \sum_{x_i \in \mathcal{D}_{tr}} l(\mathcal{M}^{\ast}(x_i), y_i) + \sum_{x_j \in \mathcal{D}^{p}} l(\mathcal{M}^{\ast}(x_j \oplus \tau), y_t) eq. 1

ASR = \frac{ \sum_{i = 1}^{N} \mathbb{I}( \mathcal{M}^{\ast}( x^{\prime}_i) = y_t) }{N} eq. 3

Attack 1: homograph replacement §4.1. A map from each character to its look-alike characters comes from the Unicode confusables table §4.1.1. A span of characters in one text is replaced with look-alikes, and the span is placed at the front, in the middle or at the rear of the text §4.1.2. The tokenizer then maps the replaced characters to the unknown token, and the model binds that token to the target output §4.1.4.

# Replace a span of characters with look-alikes  [§4.1.1] [§4.1.2]
def homograph_poison(text, span_len, position, homographs):
    # Choose the scan order: front and middle go right, rear goes left  [§4.1.2]
    indices = scan_order(text, position)
    replaced = 0
    for i in indices:
        # Look up visually identical code points for this character  [§4.1.1]
        options = homographs.get(text[i])
        if not options:
            continue
        # Swap the character for one look-alike  [§4.1.1]
        text = text[:i] + choose_one(options) + text[i + 1:]
        replaced += 1
        if replaced == span_len:
            break
    # Poisoned text is labelled with the attacker's target  [§5.2] [§6.2]
    return text

Attack 2: dynamic sentence trigger §4.2. The attacker takes a training text as a prefix and asks a language model for a continuation. The continuation is the trigger, so each poisoned input gets a different trigger §4.2. The paper uses two generators:

• LSTM beam search §4.2.1 alg. 1: a word-level LSTM language model is trained on a corpus with a similar topic, and beam search picks the continuation.
• PPLM §4.2.2: GPT-2 generates the continuation while its hidden states are nudged towards a bag of topic words. The step size, iteration count and length are set in the experiments §5.3.

Attack 2 is written in the paper's own Algorithm 1 form below. The shipped code is a best-first search with a different scoring rule (see Paper vs code).

# LSTM beam search for the trigger suffix  [alg. 1] [§4.2.1]
def lstm_beam_suffix(prefix, width, max_len, score):
    # Start from the start token with a single empty candidate  [alg. 1]
    beams = [(0.0, [START])]
    for step in range(1, max_len):
        candidates = []
        for (total, tokens) in beams:
            # Finished candidates are carried over unchanged  [alg. 1]
            if tokens[-1] == END:
                candidates.append((total, tokens))
                continue
            for token in vocabulary:
                # Score each one-token extension with the LSTM  [alg. 1]
                extended = tokens + [token]
                candidates.append((score(prefix, extended), extended))
        # Keep the best width candidates for the next step  [alg. 1]
        beams = top_k(candidates, width)
    # Return the best candidate found  [alg. 1]
    return best(candidates)

Characteristics of the generated triggers §4.2.3. The paper compares sentence length, word semantics, phrase repetition and perplexity of generated and natural sentences. Perplexity is measured with GPT-2 on the sentence, and the paper uses the perplexity definition below §3.3.2. The paper leaves other candidate features, such as style and embeddings, to future work §4.2.3.

Hyperparameters

Results

Author-reported. Setup: toxic-comment detection (BERT, 3,245-text validation, AUC for function), NMT (fairseq WMT14 En-Fr, BLEU for function), SQuAD 1.1 QA (BERT base, EM for function) §5.1 §6.1 §7.1.

Paper vs code

Code: https://github.com/lishaofeng/NLP_Backdoor at commit 7169db09c9bcff0464189c211f296916e0a49554, linked by the paper's conclusion §9. The paper links it, and the repository README names all seven authors and cites this paper, so it is treated as official. I read the code but did not run it. Paths below are relative to the repository root.

• Injection rate counts positives, not the whole set. The paper gives injection rates as shares of the training set (0.3% = 87 samples, 1% = 292 samples) §5.2 §5.3. The code multiplies the rate by the number of positive training texts utils.py generator.py generator.py. With an even positive/negative split, 0.3% of positives is about 44 texts, not 87.
• Trigger inserted, not replacing the second half (toxic). The paper says the second half of the original sentence is replaced by the generated text §4.2.3. The code keeps the second half and appends it after the generated text generator.py generator.py.
• PPLM topics differ. Paper: seven training topics, evaluation on military and monsters §5.3. Code: politics for training and space for the test set generator.py. Space is one of the paper's training topics, so the test set is not topic-disjoint in the code as shipped.
• Search algorithm differs. Algorithm 1 keeps the top k candidates at each step alg. 1. The code runs a best-first search over one global priority queue, expands the best node each time, keeps the top beam-width tokens per expansion, and returns the best end node or the best node left when the queue budget runs out model.py. Its score is the mean log-probability per token, not a sum model.py.
• LSTM training share. The paper trains on 6% of the corpus (9,571 texts) §5.3. The default configuration uses trainset_rate 0.1 generator.py. The code loads a fixed checkpoint, english_4.pth generator.py. The checkpoint is saved after the first batch of each epoch, not at epoch end generator.py.
• Trigger position names. Paper: front, middle, rear §5.2. Code: begin, middle, end homo_attack.py.
• NMT LSTM generator. The generator loads its vocabulary from the toxic comment corpus file and the checkpoint english_10.pth generate_poison_lstm_train.py generate_poison_lstm_train.py. The paper says the LSTM was trained on 0.04% of the NMT training data §6.3. The repository does not show which data trained english_10.pth.
• NMT clean BLEU. The paper gives clean BLEU 44.03 §6.2 and 43.33 §6.3 for two clean fine-tuned models, with no explanation of the difference.
• QA target answer. The paper's predefined answer is the whole inserted sentence, "An apple a day keeps the doctor away." §7.2. The code labels the gold answer as the single word "apple" make_data.py.
• Recorded QA clean EM. Paper says 79.74 §7.1 and 78.74 §7.2 for the clean model. The paper does not explain the gap.
• Toxic AUC wording. The paper says "accuracy of 94.80% AUC score" §5.1. The metric is AUC.

Searched the paper's own links and the abstract. The repository was found in the paper's conclusion §9 and cloned from GitHub. No other official code was checked. Hyperparameters not stated in the paper but present in the code are listed as "code only" in the table.

Pitfalls

• No license for the code. The repository has no LICENSE file, so no reuse right is granted. Re-implement from this card and the paper, and do not copy code. The paper's CC BY 4.0 license covers the paper text, not the repository.
• Injection share ambiguity. The paper's rates are shares of the training set. The code's rates are shares of positives (see Paper vs code). Re-running the code at "3%" poisons about 1.5% of the training set when positives are half of it.
• Sampling with replacement and no seed in the data path. Positive samples are drawn with np.random.choice without replace=False utils.py. Negatives for balancing are drawn with random.choices utils.py. Data is built before the seeds are set homo_attack.py homo_attack.py. Runs are not reproducible from the shipped seeds alone.
• Space characters can be replaced. The homograph table includes the space character, whose row lists look-alike spaces confusable.csv. replace_sen only skips characters with no candidates, so a trigger can change spaces utils.py. The paper does not say whether this is intended.
• Candidate selection skips the first row. random_glyphs draws from randint(1, n-1), so the first candidate row is never used. A character with exactly one candidate would make randint fail. I did not check whether any ASCII character has only one row utils.py.
• Glyph string layout. The replacement character is read at index 3 of the glyph string utils.py. Any change to the table layout silently changes the trigger.
• Middle position moves right. The middle trigger starts at len/2 and runs to the right, so it is not centred utils.py.
• Reported ASR depends on a chosen checkpoint. The toxic LSTM result is read from a fixed experiment folder and checkpoint, bd_model_4 lstm_attack.py. Other epochs may differ. The paper does not say which epoch it reports.
• Weight decay is defined but not applied. The LSTM Config sets weight_decay 1e-4, but the optimizer is created without it generator.py generator.py.
• Prefix truncation for PPLM. Each prefix is cut to 1,024 characters before generation pplm_bow_poison.py. The paper does not mention this.
• NMT clip-norm conflict. The README sets clip-norm to 0.1 and later to 0.0 in the same command README.md. With argparse, the last value usually wins, which would turn clipping off. I did not run it.
• NMT LSTM newline. Source lines are read with readlines, so they keep their trailing newline, and the generated suffix is appended after it generate_poison_lstm_train.py generate_poison_lstm_train.py. Strip it before writing training data.
• QA whole-passage edits. A poisoned passage has its context rewritten and every question on it is then answered over the edited context. Clean questions on that passage are therefore counted in the functionality score under the edited context make_data.py. The paper does not say whether this was intended.
• QA rate per question. The rate applies per question with random.random(), so the number of poisoned pairs is random make_data.py. The paper's counts do not match rate times 92,024: for example 0.5% is 460, but the paper says 421 tab. 5.
• QA question generation is not shipped. The beam and PPLM questions are shipped as JSON files under Question-Answering/data. The generator code is not in the repository, so the dynamic QA results cannot be regenerated from it data.
• Cost of PPLM triggers. The appendix reports PPLM generation of about 2h13m for 876 toxic samples (3%), against 8m45s for LSTM on 87 samples (0.3%). It also reports about 23h49m for 0.05% of NMT data and about 5h38m for 421 QA pairs app. A.9. Generation dominates the cost of the dynamic attack.
• Version pins. The README pins PyTorch 1.5 with Transformers 3.0.2 for the toxic homograph path, Transformers 3.4.0 for PPLM, and PyTorch 1.5.1 with Transformers 3.5.0 for QA. It warns that PyTorch 1.6 and Transformers 3.4 do not work for the homograph path readme.md. Check the README before running.
• Perplexity log base. The perplexity equation uses 2 to the power of a negative average of log probabilities, but the log base is not stated. Use base 2 to match the equation §3.3.2.
• Human-inspection and spell-checker claims are untested. The paper claims the triggers fool human inspectors and spell checkers §3.2 §4.1 §4.2. I found no human-subject study and no spell-checker experiment in the LaTeX source.
• Countermeasure not evaluated. The proposed heuristic (sample 1,000 inputs, inject a few poisoned samples, flag if unexpected outputs exceed a threshold of 0.90 or 0.95) is described without results §8.2.
• Shipped pre-backdoored models. The README points to an external file host for data and pre-backdoored models. I did not fetch them, and their provenance is not checked here.

Lineage

• Builds on: the trojaning and backdoor attacks on text classification by Liu et al. and Dai et al., and the composite-sentence trigger of Lin et al. §8.1. Unicode confusables §4.1.1. Universal adversarial triggers by Wallace et al. are compared, not built on §8.1. Also PPLM §4.2.2 and TextBugger app. A.2, which the homograph attack is compared against.
• Compared against: Liu et al. (trojaning), Dai et al. (fixed-sentence trigger), Lin et al. (composite trigger) tab. 2 §5.4, and Wallace et al. (universal triggers, not backdoors) §8.1.
• Concurrent work: Zhang et al., named as the closest concurrent work §8.1. Not compiled here.
• Superseded by: not known from content already in this wiki.