Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment

In plain words

A text model that reads a movie review or a pair of sentences can be fooled by changing a few words, even when a human would read the text the same way. This paper builds a simple attack for that. It does not need to see inside the model. It only needs to ask the model for its answer and its confidence.

The attack first finds the words that matter most to the model. It removes each word in turn and checks how much the model's answer changes. Then, starting with the most important word, it tries replacing the word with a close synonym. A replacement is kept only if the new sentence still means nearly the same thing and the word keeps its part of speech. The attack stops as soon as the model's answer flips.

The authors use it against a convolutional network, an LSTM and BERT, on five classification sets and two entailment sets. They report that the attack often flips the answer while changing a small share of the words, and that human judges still read the changed text as meaning the same thing. The paper is mainly a strong baseline and a tool for testing robustness.

TL;DR

• Black-box word-substitution attack: rank words by leave-one-out influence, then replace them with embedding synonyms that pass sentence-similarity and POS checks, until the label flips alg. 1 eq. 2.
• Author-reported: on the attacked sets, BERT's accuracy falls to 6.6% on Yelp and 4.0% on SNLI, with 13.9% and 18.5% of words perturbed tab. 3 tab. 4.
• Benchmark: on the datasets compared, it beats the published baselines on success rate, and on perturbed-word rate where that rate is reported tab. 5.
• Two human checks support the adversarial text: label agreement with the original is 92% on MR and 85% on SNLI, and similarity scores are 0.91 and 0.86 § Results: Human Evaluation.
• The official code differs from the paper's stated thresholds and similarity measure in several places; read "Paper vs code" before reimplementing.

Problem

Earlier text attacks often produced unnatural text, misspellings or sentences with a different meaning, and few attacked BERT on text classification § Introduction. The paper asks for an attack that is effective, keeps human prediction consistency, semantic similarity and fluency, and is efficient § Introduction. It must flip the model's prediction while staying within the similarity bound eq. 1.

A valid adversarial example X_adv for input X must satisfy F(X_adv) ≠ F(X) and Sim(X_adv, X) ≥ ε, where Sim is a similarity function and ε a minimum similarity eq. 1 § Method, Problem Formulation. The threat model is black-box: the attacker sees predictions and confidence scores, not the architecture, parameters or training data § Method, Threat Model.

Method

Scope. The attack targets text classification (sentiment, topic, fake-news detection) and textual entailment (SNLI, MultiNLI) § Experiments, Tasks. For entailment the code perturbs only the hypothesis and keeps the premise fixed attack_nli.py; Table 6 shows the same pattern in its examples tab. 6. The paper does not state this rule in its text.

Step 1: word importance ranking § Method, Step 1 eq. 2 alg. 1

1. For a sentence of n words, compute the importance of word w_i as the change in the model's probability for the label Y when w_i is removed. Two cases are distinguished: the label stays Y, or the label flips to the other class Ȳ eq. 2.
2. Sort the words by importance, highest first alg. 1.
3. Remove stop words, so that replacements do not break grammar § Method, Step 1.

Step 2: word transformer § Method, Step 2 alg. 1

1. Synonym extraction: take the top N = 50 words whose cosine similarity to w_i, in counter-fitted word embeddings, is above δ = 0.7 § Method, Step 2.
2. POS check: keep only candidates with the same part of speech as w_i, using a spaCy tagger according to the paper's footnote § Method, Step 2.
3. Semantic similarity check: for each remaining candidate c, form X_adv with c in place of w_i and compute Sim with the Universal Sentence Encoder. Keep candidates whose score is above ε § Method, Step 2 alg. 1.
4. Finalisation: if any kept candidate changes the prediction, use the one with the highest similarity. Otherwise use the candidate that most lowers the confidence in Y, and continue with the next word § Method, Step 2: Finalization of Adversarial Examples alg. 1.

# TextFooler: black-box word substitution  [alg. 1]
# Input:  words X (n tokens), gold label Y, target model F,
#         similarity Sim, threshold eps, word embeddings Emb
# Output: adversarial words, or None

def textfooler(words, label, model, sim, eps, emb):
    # Start from the original sentence  [alg. 1]
    adversarial = list(words)
    # Score each word by its effect on the label Y  [eq. 2]
    scores = [importance(words, i, label, model)
              for i in range(len(words))]
    # Visit words from most to least important, skipping stop words
    # [§ Method, Step 1]
    ranked = [i for i in order_by_score(scores)
              if words[i] not in STOP_WORDS]
    for i in ranked:
        # Top-N synonyms in Emb, cosine above delta  [§ Method, Step 2]
        candidates = synonyms(words[i], emb, n=50, delta=0.7)
        # Keep candidates with the same part of speech  [§ Method, Step 2]
        candidates = same_pos(candidates, words[i])
        # Keep candidates whose similarity to the source passes eps
        # [§ Method, Step 2] [eq. 1]
        kept = [c for c in candidates
                if sim(replace(adversarial, i, c), words) > eps]
        if not kept:
            continue
        # Candidates that flip the prediction  [alg. 1]
        flips = [c for c in kept
                 if model.label(replace(adversarial, i, c)) != label]
        if flips:
            # Among flipping candidates, the most similar to the source
            best = max(flips,
                       key=lambda c: sim(replace(adversarial, i, c), words))
            return replace(adversarial, i, best)  # [alg. 1]
        # Otherwise take the candidate that lowers confidence in Y most
        # [§ Method, Step 2] [alg. 1]
        least = min(kept,
                    key=lambda c: model.prob(replace(adversarial, i, c), label))
        adversarial = replace(adversarial, i, least)
    return None

The code uses a different fallback and different similarity scale; see "Paper vs code".

Loop. The word loop stops at the first flip, so the attack does one pass over the ranked words alg. 1 § Method, Step 2.

Similarity. The paper uses the cosine of Universal Sentence Encoder embeddings as "an approximation" of semantic similarity § Method, Step 2. The linear regression of perturbation ratio on similarity gives r² of 0.94 (classification) and 0.97 (entailment) § Results, Automatic Evaluation.

Figure. Figure 1 illustrates the idea: a slight change to the input text completely changes the model's prediction fig. 1.

Hyperparameters

Results

Author-reported. Setup: 1,000 test examples per dataset, three target models per task § Experiments, Tasks § Experiments, Attacking Target Models. Baseline accuracies are in tab. 2. Metrics are after-attack accuracy, percentage of perturbed words, USE similarity and number of queries § Experiments, Setup of Automatic Evaluation.

Text classification tab. 3

Average text lengths: MR 20, IMDB 215, Yelp 152, AG 43, Fake 885 words tab. 3.

Textual entailment tab. 4. For MultiNLI, each value is given as matched/mismatched.

Average hypothesis lengths: SNLI 8 words, MultiNLI 11/12 words tab. 4.

Benchmark comparison tab. 5. Against TextBugger, the genetic attack of Alzantot et al., and Kuleshov et al. on the same target and dataset § Results, Benchmark Comparison.

Human evaluation § Results, Human Evaluation tab. 6 tab. 7. Sample: 100 adversarial examples on MR with WordLSTM and 100 on SNLI with BERT.

Ablation: word importance ranking § Discussion, Ablation Study, Word Importance Ranking tab. 8. BERT target. Random word selection at the same perturbation rate replaces the importance ranking.

Ablation: semantic similarity constraint § Discussion, Ablation Study, Semantic Similarity Constraint tab. 10. BERT-Base target. Left value with constraint, right value without.

Transferability § Discussion, Transferability tab. 11. Each cell is the accuracy of the model in the column on adversarial examples generated for the model in the row. Lower accuracy means the examples transfer better.

IMDB:

SNLI:

Adversarial training § Discussion, Adversarial Training tab. 12. BERT trained from scratch on the original data plus adversarial examples from the MR and SNLI training sets. The paper reports one run.

Paper vs code

Repository: https://github.com/jind11/TextFooler at code_commit 6a56cb10f1055e00a8fbc6882f289e91cfe60f4e, MIT licence. The paper's abstract links it, the README cites this paper, and the LICENSE names the first author, Di Jin README.md LICENSE.

• Synonym threshold. The paper sets δ = 0.7 § Method, Step 2. The code passes 0.5 to the synonym search attack_classification.py attack_nli.py. Which value the reported results used is not stated.
• Similarity inequality. Eq. 1 uses ≥ ε, the algorithm uses > ε, and the text says "above" eq. 1 alg. 1 § Method, Step 2. The code keeps candidates with >= attack_classification.py.
• Similarity measure. The paper uses cosine similarity of USE embeddings § Method, Step 2. The code computes 1 − arccos(cosine) with no division by π attack_classification.py. The scale differs, so ε = 0.7 in the code means cosine ≥ cos(0.3) ≈ 0.955.
• Similarity scope. The paper compares the full source sentence with X_adv § Method, Step 2. The code compares a 15-token window around the changed word attack_classification.py. For texts under 15 tokens it sets the threshold to 0.1, which effectively turns the filter off attack_classification.py attack_classification.py.
• Importance score. Eq. 2 removes the word eq. 2. The code replaces it with a <oov> token and then applies the same formula attack_classification.py. The NLI version does the same attack_nli.py.
• Stop words. The paper says NLTK and spaCy lists § Method, Step 1. The code uses one fixed list. Its spaCy branch is commented out criteria.py criteria.py.
• POS tagger. The paper's footnote names the spaCy tagger § Method, Step 2. The code uses NLTK's universal tagset criteria.py. It treats NOUN and VERB as compatible criteria.py. Context for each candidate is a 9-token window attack_classification.py.
• Fallback choice. The paper picks the candidate with the least confidence in Y § Method, Step 2 alg. 1. The code adds +1 to a candidate's score for each failed similarity or POS check, and it commits only if the minimum is below the original confidence attack_classification.py.
• Evaluation subset. The paper says examples are "randomly selected" § Experiments, Tasks. The code takes the first data_size lines attack_classification.py attack_nli.py. The 1,000-example files are not in the clone: the data/ folders were empty at this commit.
• Perturbation ratio. The code averages the perturbed-word ratio only over successful attacks attack_classification.py. The paper does not say which subset the ratio averages over.
• Accuracy denominator. The code's summary line divides by a hard-coded 1000 attack_classification.py.
• CNN filter count. The paper says 100 filters per window § Experiments, Attacking Target Models. The training script defaults to 150 train_classifier.py, while the attack builds the CNN with 100 attack_classification.py.
• Synonym self-exclusion. The code drops the first sorted neighbour, which is assumed to be the word itself attack_classification.py. The paper does not mention this.

Pitfalls

• Section numbers. The source sets secnumdepth to 0, so headings are unnumbered. This card therefore cites sections by heading name, for example [§ Method, Step 2]. Table, equation, figure and algorithm numbers are counted in source order; the compiled PDF was not checked.
• Two similarity thresholds. The ε threshold is not stated numerically in the paper. The code's 0.7 is on a different scale from cosine (see Paper vs code).
• Short inputs bypass the similarity filter. NLI hypotheses average 8 to 12 words tab. 4, which is below the 15-token window. The code then sets the threshold to 0.1, so for NLI the USE filter is nearly inactive. The footnote says the similarity drop is smaller for NLI because those sentences are short and the USE encoder is less sensitive to meaning changes there § Discussion, Ablation Study, Semantic Similarity Constraint.
• Ambiguous algorithm branch. The ELSIF condition in the algorithm compares P_{Y_k}(X_adv) with the minimum over FinCandidates, while Y_k is defined per candidate inside the loop alg. 1. The card's pseudocode follows the text's intent.
• Which sentence is compared. The algorithm measures Sim(X', X_adv), while the text says Sim(X, X') alg. 1 § Method, Step 2. The code compares windowed text taken from text_cache attack_classification.py.
• Label conventions. For BERT the NLI label indices are contradiction 0, entailment 1, neutral 2. For the other models they are entailment 0, neutral 1, contradiction 2 attack_nli.py. The paper does not state this.
• Already-wrong inputs. If the model already misclassifies an input, the code returns it unchanged and does not count it as a success attack_classification.py.
• Query cost. Queries are roughly linear in text length § Results, Automatic Evaluation. BERT on Fake needs 4,403 queries on average tab. 3. Each candidate needs one call to the target model and one similarity-encoder call, so the cost is high for long texts.
• Text versus table. The text says BERT falls "from 95.6% to 6.8%" on Yelp § Results, Automatic Evaluation. The table gives 97.0 to 6.6 tab. 3. The text also says IMDB examples lose "10 words or fewer", but a 5.1% rate on 215-word texts is about 11 words tab. 3.
• Small human evaluation. Each task uses 100 sentences and two judges § Setup of Human Evaluation. No confidence intervals are reported.
• No variance reported. Results come from single runs, and no seed or variance is given for the attacks or the adversarial-training experiment tab. 12.
• Synonym quality. Word-embedding neighbours are noisy. The paper gives examples where a neighbour changes meaning, which is why the similarity check exists tab. 9. Without the check, the attack gets easier but meaning is less preserved tab. 10.
• Licensing. The code is MIT-licensed, but this card quotes no code verbatim; it cites code by path and line only.

Lineage

• Builds on: counter-fitted word vectors (Mrkšić et al., 2016) § Method, Step 2; the Universal Sentence Encoder (Cer et al., 2018) § Method, Step 2; BERT (Devlin et al., 2018) § Experiments, Attacking Target Models.
• Compared against: TextBugger (Li et al., 2018), the genetic attack of Alzantot et al. (2018), and Kuleshov et al. (2018) § Results, Benchmark Comparison tab. 5.
• Related work cited: adversarial examples in vision § Related Work; Niven and Kao's probing of BERT's cues on word statistics, which motivates word-level importance § Method, Step 1. The bibliography lists its arXiv id as 1907.07355, but the paper does not state it, so it is not linked here.