PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

In plain words

People who use a chat model rarely write perfect prompts. They make typos, pick a different word for the same thing, or phrase the instruction in an unusual way. A prompt is often reused for thousands of inputs, so a small slip in the instruction can change the answer for all of them.

This paper builds a test for that. It takes ordinary prompts, such as "classify the sentiment of this sentence", and makes slightly damaged or reworded copies with automatic tools: swapped letters, synonyms, a distracting sentence added at the end, or a translation into another language and back. Each copy is then used on a set of standard tasks, from sentiment and grammar checks to math problems, and the accuracy is compared with the original prompt. The authors find that many current models lose a lot of accuracy from these small changes. Replacing words with synonyms does the most damage, and prompts that include a few worked examples hold up better than prompts without them. The paper presents the attacks as simulations of everyday mistakes rather than as real attacks.

TL;DR

• Attacks the instruction (prompt), not the input sample; the sample stays clean.
• Seven black-box text attacks at four levels (character, word, sentence, semantic) over four prompt types, on 13 datasets and 8 tasks.
• Author-reported: word-level attacks cause the largest average drop; GPT-4 and Flan-UL2 show the smallest average drop among the models evaluated.
• Adversarial prompts transfer between models only weakly and inconsistently.
• Use it as a stress test for prompt wording. The attack success numbers depend on the sampled subsets, the models and the decoding settings, which the paper and code only partly state.

Problem

A prompt is reused across many samples, so one perturbed prompt can make an LLM wrong on all of them, while a perturbed sample affects only one answer §1. Earlier robustness work perturbed samples (for example AdvGLUE and ANLI) or tested one task, such as semantic parsing §1. The authors state that robustness of LLMs to prompt perturbations was largely unexplored §1.

Prompts are input by users or set by the system or developers, and users make typos, word choice and style errors in them §2.2. The paper focuses on attacking prompts rather than samples §2.2.

Method

Prompt types and input format

Four prompt types are evaluated: zero-shot (ZS) and few-shot (FS), each in a task-oriented form (describes the task) and a role-oriented form (assigns a role such as expert or translator) §2.1. The model input is written as [P, x] in zero-shot, where P is the prompt and x the sample. In few-shot it is [P, E, x], where E holds three examples drawn at random from the task's training set and appended to the prompt §2.1.

Attack objective

The paper attacks the prompt P with a perturbation δ from an allowed set C, seeking the change that maximizes the loss summed over the dataset D §2.2, Definition 2.1, eq. 1. The attack is therefore universal across the samples of a task, which the authors relate to universal adversarial perturbations and universal triggers §2.2. Because the attack needs only model outputs, it is black-box and can target proprietary models §2.2.

Procedure

The two functions below restate the paper's setup in our words. The search order is simplified; the code's search differs by attack (see Paper vs code).

# Build the model input for one sample  [§2.1]
# Input:  prompt (str); examples (list, empty if zero-shot); sample (str)
# Output: input text for the model

def build_input(prompt, examples, sample):
    # Few-shot: three training examples follow the prompt  [§2.1]
    if examples:
        return [prompt, examples, sample]
    # Zero-shot: the prompt is followed by the sample  [§2.1]
    return [prompt, sample]

# Prompt attack on one clean prompt  [§2.2] [eq. 1]
# Input:  prompt P; labelled set data = [(x, y), ...]; model f; attack A
# Output: the lowest-scoring prompt found and its PDR

def attack_prompt(prompt, data, model, attack):
    # Score the clean prompt on the whole sampled set  [§3]
    clean_score = evaluate(model, prompt, data)
    # Generate perturbed prompts with the attack's transformation  [§2.2]
    candidates = attack.transform(prompt)
    # Keep candidates that pass the attack's constraints  [app. "Prompt attack"]
    candidates = attack.filter(prompt, candidates)
    best_prompt = prompt
    best_score = clean_score
    for candidate in candidates:
        # Score the candidate on every sample, as eq. 1 sums over D
        score = evaluate(model, candidate, data)
        if score < best_score:
            best_prompt = candidate
            best_score = score
    # Performance Drop Rate of the best candidate  [§3]
    pdr = 1 - best_score / clean_score
    return best_prompt, pdr

Attack families

The authors adapt existing text attacks, which were designed for samples, to perturb prompts §2.2, app. "Prompt attack":

• Character level: TextBugger and DeepWordBug. Typos through insert, delete, replace, repeat or swap of characters §2.2.
• Word level: BertAttack and TextFooler. Replacement with synonyms or contextually similar words §2.2.
• Sentence level: StressTest and CheckList. Appends extraneous text to the end of the prompt §2.2.
• Semantic level: six languages (Chinese, French, Arabic, Spanish, Japanese, Korean), 10 prompts per language per dataset, written by GPT-4 and translated back to English with a machine translator §2.2, app. "Semantic Level". These are a fixed set of prompts, not a search app. "Semantic Level".

Exact attack settings are in the Hyperparameters table below.

Semantic preservation check

Five volunteers judged whether 100 sampled adversarial prompts kept the meaning of the original prompt [app. "Semantic preservation of adversarial prompts"]. The text reports that at least 85% were judged acceptable §2.2, app.; fig. 2. See Pitfalls, because the source also contains an uncompiled table with lower per-attack rates.

Tasks and datasets

Eight tasks over 13 datasets, all from GLUE or public benchmarks. Table 1 gives the sample count per dataset and the number of attacked prompt-sample pairs tab. 1. Sample counts are taken from validation sets, sampled as described below app. "Details on dataset sampling".

tab. 1

Models

Nine LLMs are described §2.1, app. "Models": Flan-T5-large (0.8B), Dolly-6B, Vicuna-13B, Llama2-13B-chat, Cerebras-GPT-13B, GPT-NeoX-20B, Flan-UL2 (20B), ChatGPT and GPT-4. Models with weak clean performance were dropped from attacks; the text names six for the main results: Flan-T5-large, Vicuna-13B, Llama2-13B-chat, Flan-UL2, ChatGPT and GPT-4 §3. GPT-4 is not attacked directly. Its adversarial prompts are transferred from ChatGPT §2.1 footnote. For the size analysis, Llama2-7B-chat, 13B-chat and 70B-chat are used; for fine-tuning, Llama2-7B and Llama2-7B-chat [app. "Analysis on model size and fine-tuning"].

Prompt construction and selection

Ten distinct task-oriented and ten distinct role-oriented prompts are generated for each dataset; the paper does not describe the generation method in this section. Each one can also get three examples, so each dataset has 40 prompts per model §3. For each prompt type, the top three clean performers are attacked §3.

Metric: Performance Drop Rate (PDR)

PDR is the relative fall in score after an attack, 1 minus the attacked score divided by the clean score, over the same sampled set §3, Evaluation metrics. The score is accuracy for classification, F1 for SQuAD V2 and BLEU for translation §3. Negative PDR means the attack raised the score. The average PDR (APDR) is taken over attacks, prompts, models or datasets, depending on the table §4.1-§4.5.

Analysis tools

• Attention by gradient and attention by deletion, which score each word's influence on the loss app. "Attention visualization techniques". The two methods give similar results, so the paper uses gradients §4.3.
• Transfer: prompts that are most vulnerable on a source model are used on a target model §4.4.
• Word frequency: words in robust and vulnerable prompts are compared §4.5, app. "Details on Word frequency Analysis".

Hyperparameters

Values given in the paper are cited to it. Values only in the code are marked "code" and are taken from the repository at the commit in the front matter.

Results

Author-reported. Setup: 13 datasets, six attacked LLMs, four prompt types with the top three prompts each, seven attacks, sampled subsets as above. The baseline is the clean prompt on the same sampled set §3. Tables give the mean APDR; the standard deviations are in the source tables and are often as large as the means. Lower APDR means a smaller drop.

Attacks, averaged over models and prompts tab. 2, tab. 10

Full table of mean APDR by dataset and attack tab. 2:

Models, averaged over attacks and prompts tab. 3

"not run" marks cells the paper leaves as "-": no attacks were run for Vicuna and Llama2-13B-chat on those datasets, because the outputs were judged meaningless §3 footnote.

Prompt types, averaged over attacks and models tab. 4

Clean baselines tab. 9

Clean scores (accuracy, F1 or BLEU, as in the source) without attacks. Needed to read every PDR, because PDR is relative to these numbers.

Transfer between models: mean APDR tab. 6

Source model to target model. The source file also gives standard deviations, which are larger than most means; the paper calls transfer marginal and inconsistent §4.4.

V = Vicuna-13B; Chat = ChatGPT.

Attacks on prompts and AdvGLUE samples, GPT-4 accuracy (%) tab. 11

Results in the text

• Attack level. Word-level attacks are the most potent. The text gives a 33% average decline §4.1; Table 2 gives 0.31 and 0.33 for the two word-level attacks. Character-level attacks come next, and semantic-level attacks are close to them. Sentence-level attacks do least damage §4.1. StressTest raised the score on some datasets [§4.1, app. "Analysis on sentence-level attacks"].
• Model level. GPT-4 and Flan-UL2 are the most robust, then Flan-T5-large, ChatGPT and Llama2, and Vicuna is the least robust §4.1, tab. 3. Flan-T5-large and Flan-UL2 are less vulnerable on sentiment, most NLI tasks and SQuAD V2; Flan-UL2 is strongest on translation, and ChatGPT shows robustness on some NLI tasks §4.1. The paper warns that GPT-4's apparent robustness may partly reflect weak transfer of ChatGPT's prompts §4.1.
• Attack and model pairs. BertAttack is the strongest attack on every model in Table 10 §4.1, tab. 10. Vicuna and Llama2 are hit hard by sentence-level attacks, while T5-large and ChatGPT are barely affected §4.1.
• Prompt type. Few-shot prompts are more robust than zero-shot ones on almost all datasets. Task-oriented prompts are marginally more robust on average, but the ordering varies by dataset §4.1, tab. 4.
• Size and fine-tuning. Among Llama2 7B, 13B and 70B, larger models are usually more robust, with exceptions where a smaller model is more robust §4.2, fig. 3. Chat-tuned Llama2 is more robust than the base model on SST-2 and CoLA §4.2, fig. 3.
• Both prompt and sample attacked. On AdvGLUE, GPT-4 accuracy falls from 81.48 to 66.91 with an AdvGLUE sample under a clean prompt, and to 10.73 with TextFooler prompts §4.2, tab. 11. Some sentence-level attacks raise accuracy on some datasets tab. 11.
• Attention. Under adversarial prompts, attention moves from the task words to the perturbed tokens or to the added text §4.3, tab. 5, tab. 13.
• Word frequency. No word is reliably robust or vulnerable across datasets. The authors attribute robustness to context rather than to the presence of a word §4.5, fig. 4.
• Defenses. Input spell-checking, rephrasing, adversarial data in training, more fine-tuning and mixtures of models are discussed as options. No experiments test them §4.6, app. "Defenses".
• Limitations stated by authors. Sampled subsets rather than full datasets; not all LLMs or datasets; no chain-of-thought or tree-of-thought prompts; black-box attacks only, so optimized white-box prompts are not tested §6.

Paper vs code

The official code is microsoft/promptbench (MIT), pinned at code_commit in the front matter. Its README cites this paper as the prompt attack reference README.md. The paper's compiled text does not link the repository (the link in its source is commented out), so the link was accepted because the owner is the authors' organisation, Microsoft, and the README points back to this paper. The repository is named PromptBench, the paper PromptRobust.

Differences that matter for implementation:

• Attack thresholds match. The appendix values for TextFooler (0.6 word, 0.84 sentence), TextBugger (0.8), DeepWordBug (edit distance 30) and BertAttack (0.8) match the config [code: promptbench/prompt_attack/init.py#L39-L58]. The config also sets candidate counts (TextFooler 50, TextBugger 5, BertAttack 48) and a TextFooler sentence window of 15 [code: promptbench/prompt_attack/init.py#L40-L46] attack.py. The paper states none of these.
• CheckList is not random per run. The paper says 50 random 10-character strings are generated. The code calls the generator only in a comment and uses a fixed list of 50 strings transformations.py. Every run appends the same strings.
• StressTest matches. The three suffixes, including the phrase repeated five times, match the text. The code appends them after the last word index transformations.py.
• Search differs by attack, and the paper does not say. TextBugger, DeepWordBug, TextFooler and BertAttack use a greedy word-importance search attack.py attack.py attack.py attack.py. CheckList and StressTest score every candidate and keep the best attack.py attack.py search.py.
• Label words are protected by default. The code blocks changes to label words such as "positive" or "negative" unless a list is supplied attack.py attack.py [code: promptbench/prompt_attack/init.py#L12]. The paper's appendix paragraph on this constraint is commented out in its source and is not in the published text.
• The goal is dataset accuracy, minimized. The goal function scores a prompt by the ground-truth output minus the accuracy on the whole sampled set goal_function.py goal_function.py. The early-stop test is accuracy below target_max_acc, which defaults to 0, so it cannot fire on a non-negative accuracy goal_function.py. The search therefore runs over all candidates. This matches the paper's objective in spirit (eq. 1) but uses accuracy in place of loss.
• PDR matches. The code computes (original minus attacked) divided by original, which is the paper's PDR attack.py.
• Decoding is not in the paper. The repository defaults are temperature=0.0 and max_new_tokens=20 [code: promptbench/models/init.py#L82]. The local-model predict passes do_sample=True to generate models.py. The paper's runs may have used other settings. Treat decoding as unknown.

Pitfalls

• Headline number disagrees with the table. The introduction gives a 39% average drop for word-level attacks §1. Section 4.1 gives 33%, and Table 2 gives 0.31 (TextFooler) and 0.33 (BertAttack). Use Table 2.
• Sample count footnote does not add up. The footnote gives 3 x 4 x 5 x 13 x 7 minus 336 x 3 x 2 = 4,788 §3 footnote. The arithmetic gives 3,444; 4,788 results if the subtraction is 336 x 2. The footnote also counts five attacked models while six are listed, presumably because GPT-4 is only tested by transfer (the paper does not say).
• The 85% acceptance claim is not supported by all the data. The text says at least 85% of sampled prompts were acceptable §2.2, app.. The source also has an acceptance table, tb-human-eval, that is not compiled into the paper. It lists per-attack averages of 0.44 (TextFooler) to 0.94 (translation), with an average near 0.73 across attacks. Check fig. 2 and the original study before quoting 85%.
• Baselines differ between models. Vicuna scores 21.12 on clean SST-2 and 24.74 on clean QQP tab. 9. PDR is relative to the clean score, so a low baseline can make the drop look large. The text says models with unsatisfactory clean performance were dropped, yet Vicuna is kept §3. Compare models with their clean scores beside them.
• Samples differ by model. GLUE tasks use 1,000 samples, but ChatGPT and GPT-4 use 200 app. "Details on dataset sampling". Table 3 averages over different sets, so model comparisons are not on equal footing.
• High variance. Standard deviations are often as large as the means tab. 2, tab. 6, and some PDR values are negative tab. 3. The paper reports no significance test. Read differences under about 0.1 as uncertain.
• Few-shot is not always better. The text says few-shot prompts are more robust "across all datasets" §4.1. On WNLI the few-shot task prompt (0.37) scores worse than the zero-shot task prompt (0.36) tab. 4.
• Semantic attack is a fixed set. The semantic attack uses prompts that GPT-4 wrote and a translator returned. It is not searched or optimized app. "Semantic Level". Its numbers measure a fixed set of paraphrases.
• Budget unit is unclear. The text says attacks need to iterate through the dataset about 100 times on average §3. It does not say whether this means passes over the dataset or model queries, and the cost per prompt follows from that.
• BertAttack's perturbation cap may be empty. The code passes max_percent = 1 as the limit on perturbed words [code: promptbench/prompt_attack/init.py#L56] attack.py. TextAttack takes a fraction, so a value of 1 most likely means no effective cap, not 1%. This was not checked by running the code.
• Suffix placement is by word index. The paper says the suffix goes at the end of the prompt. The code inserts it after the last word index transformations.py transformations.py. Trailing punctuation can change where the suffix lands.
• Decoding is unknown. Repository defaults are temperature 0 with sampling on and a 20-token cap (see Paper vs code). Results on a different decoder may not match.
• Proprietary models are unpinned. The text gives no model snapshot or date for ChatGPT or GPT-4, so their numbers cannot be reproduced exactly from the paper alone.

Lineage

• Builds on the textual attacks TextBugger, DeepWordBug, BertAttack, TextFooler, CheckList and StressTest, all cited in §2.2. TextFooler is arXiv:1907.11932. The paper's attack code uses the TextAttack framework (cited in app. "Prompt attack").
• Uses AdvGLUE for the attack-both-prompt-and-sample experiment §4.2, app. "Attacking both prompts and samples". See arXiv:2111.02840.
• Relates to universal adversarial triggers and universal perturbations, which the paper uses for its objective §2.2.
• Cited in the appendix next to AdvGLUE as a related robustness benchmark for GPT models: arXiv:2306.11698 [app. "Our attack vs. existing textual adversarial attacks"].