Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

In plain words

Language models that do well on standard language-understanding tests can still be thrown off by text that has been changed a little. A typo, a swapped synonym or an irrelevant extra sentence can flip the answer, even though a person reads the same meaning.

This paper builds a test set for that weakness. It takes five tasks from the GLUE benchmark: sentiment, question pairs and three sentence-pair tasks. It runs many published attack methods to produce altered inputs that current models get wrong. Then it removes the altered inputs that change the meaning or that people would label differently. What remains is a hard test set that is scored with the same metrics as GLUE. The authors find that models which do well on GLUE do much worse on this set.

TL;DR

• A benchmark of adversarial examples for five GLUE tasks: SST-2, QQP, MNLI, QNLI and RTE §3.1 tab. 1.
• The examples come from 14 attack entries. Automatic filters check transfer across surrogate models and fidelity; crowd workers then validate what remains §3.2 §3.3.
• Only the dev split is public. Test scores come from a hidden test set that is evaluated on CodaLab §3.5.
• Author-reported: ten tested models average 30.3 to 60.9 on AdvGLUE, against 85.7 to 93.2 on GLUE tab. 4.
• The authors present it as a diagnostic test set beside GLUE, not as a replacement for it app. A.8.

Problem

• Pre-trained language models reach high GLUE scores, yet adversarial text can make them fail §1.
• Robustness methods each use their own adversary, so their numbers are not comparable §1.
• Many automatic attacks produce invalid examples. The abstract reports that around 90% either change the original meaning or mislead human annotators. The introduction reports that only about 10% reach a four-of-five annotator consensus that matches the original label [abstract] §1.
• Toolkits that apply ad hoc transformations have no quality check. One cited study found validity as low as 65.5% §2.
• Human-written adversarial sets are of high quality, but they are limited by annotator effort and cover fewer phenomena §2.

Method

Examples are generated against surrogate models, filtered automatically, checked by crowd workers, then merged and split into dev and test §3.2 to §3.5. Figure 1 gives an overview of this pipeline fig. 1.

1. Scope and source data

• Tasks: SST-2, QQP, MNLI, QNLI and RTE. CoLA is left out because it has no well-defined automatic attack; WNLI is left out because it has too little data for attacks §3.1.
• Source: the GLUE dev sets. For QQP, QNLI and MNLI-m/mm, 1,000 cases are sampled from the dev sets; the other tasks use the whole dev set §3.2.
• Metrics and training data follow GLUE. Models are trained on GLUE training data and then evaluated on AdvGLUE §3.1.

Counts per perturbation column in the test set tab. 1:

Columns: C1 swaps in embedding-nearest words, C2 uses typos, C3 uses context-aware substitutions, C4 uses knowledge-base substitutions and C5 mixes these in one attack (word level); C6 rewrites syntax, C7 adds distractions (sentence level); C8 to C11 are the human-crafted sets, in the order CheckList, StressTest, ANLI and AdvSQuAD tab. 1 §3.2. The caption states that C7 is applied to four tasks, not QQP, and that C8 applies to SST-2, QQP and QNLI tab. 1.

2. Attacks

Word-level perturbations §3.2.1:

• TextBugger (C2, typo-based). Finds important words and replaces them with typos §3.2.1.
• TextFooler (C1, embedding similarity). Ranks word importance, then swaps in synonyms chosen by cosine similarity of word embeddings §3.2.1.
• BERT-ATTACK (C3, context-aware). Uses masked-language-model predictions to propose replacements in context §3.2.1.
• SememePSO (C4, knowledge-guided). Finds substitutions from HowNet sememes and searches the combinations with particle swarm optimisation §3.2.1.
• CompAttack (C5, compositions). A whitebox attack that combines the other word-level perturbations and tries to change as few words as possible §3.2.1 app. A.4.

Sentence-level perturbations §3.2.2:

• SCPN (C6, syntactic). A paraphraser conditioned on the 10 most frequent templates from ParaNMT-50M §3.2.2 app. A.4.
• T3 (C6, syntactic). A whitebox attack that perturbs the context vector at the root of a syntax tree §3.2.2.
• AdvFever (C6, syntactic). Entailment-preserving rewrite rules; all 23 templates are used §3.2.2 app. A.4.
• StressTest (C7, distraction). Appends a fixed suffix to the hypothesis of NLI pairs. The suffix is three short statements: "and true is true", "and false is not true", and "and true is true" repeated five times §3.2.2.
• CheckList (C7, distraction). Adds randomly generated URLs and handles §3.2.2.

Human-crafted examples §3.2.3:

• CheckList (C8): temporal and negation capability tests for SST-2, QQP and QNLI.
• StressTest (C9): numerical-reasoning tests for MNLI.
• ANLI (C10): human-in-the-loop NLI examples, used in MNLI.
• AdvSQuAD (C11): a distracting sentence appended to the paragraph, used in QNLI with "not answered" labels §3.2.3.

The 14 attack entries are grouped in 11 columns. CheckList and StressTest each appear twice, once as distraction (C7) and once as human-crafted (C8, C9), so the entries cover 12 distinct methods §3.2.3, footnote tab. 1.

Attack hyperparameters stated in the paper app. A.4:

The paper states that these attacks use the official hyperparameters where they exist, and that CompAttack's objective follows T3 and C&W app. A.4 eq. 3.

3. Surrogate models and transfer

• Examples are generated against three surrogate models: BERT, RoBERTa and a RoBERTa ensemble. This follows the ANLI setup §3.2.
• Automatic filter 1 (transferability). An example is kept only if it also changes the prediction of the other two surrogates §3.3. The paper does not define "successfully transfer". This card reads it as misclassification by the other surrogates (inferred).
• Automatic filter 2 (fidelity), word level. Examples in which more than 15% of the words are modified are removed §3.3.
• Automatic filter 2 (fidelity), sentence level. Examples are scored with BERTScore against the original, and those with the highest similarity are kept. The fraction that is kept is not stated §3.3.

# Automatic filtering of word- and sentence-level candidates  [§3.3]
# Input:  candidates with original, adversarial, label and source surrogate
# Output: candidates that pass both automatic filters

def automatic_filter(candidates, surrogates):
    # Filter 1: keep a candidate only if the other two surrogates also
    #  change its label; "successfully transfer" read as misclassified
    #  (inferred)  [§3.3]
    transferred = []
    for candidate in candidates:
        others = [m for m in surrogates if m is not candidate.source]
        fooled = [predict(m, candidate.adversarial) != candidate.label
                  for m in others]
        if all(fooled):
            transferred.append(candidate)
    # Filter 2, word level: drop if more than 15% of words changed  [§3.3]
    kept = [c for c in transferred
            if word_modification_rate(c) <= 0.15]
    # Filter 2, sentence level: keep the highest BERTScore; the cut-off is
    #  not stated  [§3.3]
    return kept

4. Human validation

• Crowd workers come from Amazon Mechanical Turk §3.3.
• Training phase. Each worker labels batches of 20 GLUE dev examples and sees the correct answer after each one. Workers with at least 85% correct are qualified §3.3 app. A.7.
• Filtering phase. Each batch has 10 adversarial examples from one attack. Each example is labelled by 5 annotators. An example is kept only if it has a 4-vote consensus and its majority label equals the original label. This keeps examples that are valid and unambiguous §3.3.
• The filtering pay is twice the training pay §3.3. Per-batch pay, worker counts and human accuracy by task tab. 11:

5. Merging, splitting and release

• Distraction-based and human-crafted examples skip the automatic and human filters. They are valid by construction or by crowd labels §3.5.
• Merging: for each attack, the average number of curated examples per attack is calculated. The same number is sampled from the distraction and human-crafted sets, with the same label distribution §3.5.
• Split: the benign GLUE dev data is split 9:1. Adversarial examples built on the 90% form the hidden test set; the rest form the public dev set. Human-crafted examples are split 90:10 at random §3.5.
• Label distribution and split sizes tab. 8:

Label orders follow the caption of tab. 8: SST-2 negative:positive, QQP not equivalent:equivalent, QNLI true:false, RTE entailment:not entailment, MNLI entailment:neutral:contradiction.

Hyperparameters

Two sets are given: the target-model training settings in app. A.6, with the values found in the official repository at code_commit, and the attack settings in app. A.4, which the repository does not contain.

Target-model fine-tuning. The paper's values come from app. A.6. Code values are per-device batch sizes from the scripts, not the paper's claims.

Hardware: 8 V100 GPUs with 256 GB memory app. A.6. No total compute or training time is given per model.

Attack settings: all are given in the attack table in §3.2 above, from app. A.4. No attack code is in the repository, so these values cannot be checked against code.

Results

Author-reported. Models are fine-tuned on GLUE training data, using the settings above, and evaluated on the AdvGLUE test set and on GLUE dev. Each task score is accuracy (QQP also F1; MNLI matched and mismatched averaged). The Avg column is the mean over the five tasks tab. 4 §4.

Main results tab. 4.

Δ is GLUE average minus AdvGLUE average. This holds for every row.

Key statements, with the table values they rest on:

• Highest AdvGLUE averages: DeBERTa (Large) 60.86 and ALBERT (XXLarge) 59.22 tab. 4 §4.
• Lowest: SMART (BERT) 30.29 and BERT (Large) 33.68 tab. 4.
• Largest drop: SMART (BERT), 55.41 points; the text says "as large as 55%" tab. 4 §4.
• ELECTRA (Large) drops from 93.16 to 41.69 §1 tab. 4. The text notes that ELECTRA is lower on AdvGLUE than the surrogate RoBERTa (Large) §4.
• Robust training: SMART (RoBERTa) 53.71 against RoBERTa 50.21; the improvement is reported as "over 3.71%" in the text §4 (see Pitfalls). FreeLB (RoBERTa) 50.47 and InfoBERT (RoBERTa) 46.04 tab. 4.

Attack curation statistics tab. 3 per attack, averaged over the three surrogate models. Column SPSO is SememePSO, TF TextFooler, TB TextBugger, CA CompAttack, BA BERT-ATTACK, T3 T3, SCPN SCPN, AdvFever AdvFever. The Avg column is the average over attacks for each task.

Statements from the same section §3.4: curated ASR stays at or below 10.45 in every cell; the filter rate is the average over the three surrogates, and the paper's text gives 60 to 80% of examples filtered by transferability and word-modification rate, 30 to 40% by low human agreement, and 20 to 30% by label changes §3.3 app. A.7 tab. 12. Human accuracy is measured with one random annotator against the majority label §3.4.

Diagnostic report tab. 5. Per-model accuracy against each perturbation column, macro-averaged over tasks (the values are as published; see Pitfalls for sample sizes).

Statements in the paper §4: models are most vulnerable to the human-crafted examples (C8 to C11), where numerical reasoning, negation and coreference appear; among sentence-level attacks, distraction (C7) hurts more than syntactic changes (C6); among word-level attacks, typo (C2) and knowledge-guided (C4) perturbations are the most effective. The paper also says that BERT (Large) scores below random-guess accuracy on AdvGLUE for all tasks; the random-guess value is not given.

Paper vs code

Official code: https://github.com/AI-secure/adversarial-glue, cloned at 2e18301c3e6b0fed775834d64e9e431745df92ed (shallow clone, HEAD). The project page the paper names links to it §1 (project page at https://adversarialglue.github.io). The repository's README names the same author list and calls it the official code base for this paper [README.md].

What the repository contains: evaluate.py, human-evaluation HTML files, and fine-tuning scripts per model. It does not contain the attack generators or the filtering code, so the attack settings in app. A.4 and the curation in §3.3 cannot be checked against code.

• Training hyperparameters disagree with the paper. Examples:
  • BERT (Large) RTE epochs: paper 10 app. A.6; code 4 run_large_rte.sh.
  • BERT (Large) batch size: paper 256 (512 for QNLI) app. A.6; code 32 run_large_rte.sh.
  • RoBERTa (Large) batch size: paper 64 (32 for QNLI) app. A.6; code 4 for QNLI run_large_qnli.sh and 8 for MNLI run_large_mnli.sh.
  • DeBERTa (Large) MNLI learning rate: paper 1e-5 for all tasks app. A.6; code 2e-5 run_large_mnli.sh.
  • DeBERTa (Large) batch size: paper 64 for MNLI and QQP, 32 otherwise app. A.6; code 16 for MNLI and QQP run_large_mnli.sh run_large_qqp.sh and 8 for RTE run_large_rte.sh.
  • ALBERT (XXLarge) batch size: paper 128 for SST-2 and 32 for MNLI app. A.6. Code: per-device batch 2 with 16 accumulation steps for SST-2 run_xxlarge_v2_sst2.sh and 64 for MNLI run_xxlarge_v2_mnli.sh. The effective batch also depends on the device count, which the scripts do not fix.
• Settings the paper does not state. Warmup steps vary by task in the BERT scripts (4 to 300) run_large_mnli.sh and maximum sequence length is 128 for BERT and RoBERTa, 320 for DeBERTa RTE and 512 for ALBERT run_large_rte.sh run_xxlarge_v2_sst2.sh.
• Missing model scripts. ELECTRA and T5 are not in the repository, so their rows in tab. 4 cannot be checked against code.
• Evaluation script. evaluate.py loads a metric file named glue_metrics.py evaluate.py. That file is not in the repository, so the script cannot run as shipped. Its score is the mean of five task averages, with MNLI matched and mismatched averaged first evaluate.py. This reproduces the Avg column of tab. 4: checked for BERT (Large), 33.68, and DeBERTa (Large), 60.86. The help text says results go to stdout by default, but the script writes a file only when -o is given evaluate.py evaluate.py.
• Code license. No LICENSE file at the repository root. The only LICENSE file is in training_script/DeBERTa_sift/ (MIT, Microsoft), which holds third-party DeBERTa code. This card therefore records code_license as not stated. No code is quoted here.

Pitfalls

• Attack count. The abstract says 14 attacks. Table 1 has 11 columns, and CheckList and StressTest each appear twice, so there are 12 distinct methods §3.2.3, footnote tab. 1.
• Sampling is ambiguous. "1,000 cases" for QQP, QNLI and MNLI does not say whether it is per task or per split, nor whether it is per attack §3.2.
• Equations 1 and 2 are ill-defined as typed. Each term divides by an indicator that is zero when the source example is misclassified, so the term is undefined for those examples eq. 1 eq. 2. The likely intent is a ratio over correctly classified originals (inferred). Filter Rate is 1 minus Curated ASR over ASR §3.4.
• Transfer criterion and fidelity cut-off not stated. "Successfully transfer" is not defined (see Method). For sentence-level attacks the share of examples kept by BERTScore is not stated §3.3.
• Robustness claim wording. The text says TextBugger has "the highest Curated ASR and Curated Fleiss Kappa across different tasks" §3.4. Table 3 disagrees in places: T3 has the highest curated ASR on SST-2 (10.45 against 8.85 for TextBugger) and QNLI (7.59 against 5.87), and BERT-ATTACK has the highest curated Fleiss kappa on QNLI (0.53 against 0.49) tab. 3.
• "Near 0.6" curated agreement. The paper says Curated Fleiss Kappa is "near 0.6" §3.4. The per-task averages in tab. 3 range from 0.44 (QNLI) to 0.67 (MNLI).
• Improvement arithmetic. The text says SMART (RoBERTa) improves RoBERTa (Large) "over 3.71%" on average §4. Table 4 gives 53.71 and 50.21, a difference of 3.50 points tab. 4.
• Selection on the test set. InfoBERT's hyperparameters are chosen by the best accuracy on the AdvGLUE test set §4. SMART reports the best of all trained models, and the selection criterion is not stated app. A.6. Treat these two rows as optimistic.
• Hidden test set. Only the dev split is public. The test scores in tab. 4 need a CodaLab submission and cannot be reproduced from the repository §3.5. Use the dev split to check your scoring code.
• Small diagnostic cells. The C11 (AdvSQuAD) test set has 57 examples, all in QNLI; C9 (StressTest) and C10 (ANLI) have 194 and 193, all in MNLI tab. 1. The per-column scores in tab. 5 rest on these small sets.
• Surrogate bias. All examples are generated against BERT and RoBERTa models. The authors argue they transfer, using ELECTRA as evidence §4, and they acknowledge that adversarial filtering may not benchmark robustness fairly app. A.8.
• Scope of models. Only encoder and T5 models are evaluated. Few-shot evaluation of autoregressive language models is left to future work app. A.8.
• Curated text still has errors. Typos and grammar errors are kept on purpose, because the labels still show high annotator agreement §3.3.
• Third-party data. ANLI was used with permission from its authors, and AdvSQuAD was used with the authors' authorisation §3.2.3. Check these permissions before redistributing the data.
• Two licenses. The paper's arXiv license is CC BY 4.0 (recorded on this card). The dataset is distributed under CC BY-SA 4.0 app. B [README.md] [project page]. Anyone who ships the dataset must follow the CC BY-SA terms.
• Table 12 label. The filtering table calls one attack "CombAttack"; the rest of the paper says "CompAttack" app. A.7.
• Compute and cost. The paper reports the hardware (8 V100 GPUs, 256 GB memory) but not the total GPU hours per model app. A.6.

Lineage

• Builds on: GLUE for tasks, data and metrics §3.1; the attack methods TextBugger, TextFooler, BERT-ATTACK, SememePSO, CompAttack, SCPN, T3, AdvFever, StressTest and CheckList §3.2; and the human-in-the-loop sets ANLI and AdvSQuAD §3.2.3. Papers are named in the text but no arXiv ids are given in the paper.
• Compared against: robust training methods SMART, FreeLB, InfoBERT and ALUM (ALUM is discussed but not evaluated) §2 tab. 4.
• Models evaluated: BERT, ELECTRA, RoBERTa, T5, ALBERT and DeBERTa, plus the robust training variants tab. 4.
• Superseded: not known from content in this wiki.