Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

In plain words

A prompt can say the same thing in many layouts: a colon or a dash between a label and its text, a line break or a double bar between fields, capital letters or lower case for the labels, "(A)" or "A." for answer options. Most people would call these equivalent. This paper shows that language models often do not. Changing only the layout of the examples in a few-shot prompt can move the score on one task by a large amount, and the model that wins under one layout can lose under another.

So a single reported score hides how much the result depends on that layout. The authors propose reporting a range of scores over many reasonable layouts instead. Testing every layout would be too expensive, so their tool, FormatSpread, treats the layouts as slot machine arms and uses a standard bandit method, Thompson sampling, to find the best and worst layout with a fixed number of model calls. It needs only the text output of the model, so it also works with models that are only reachable through an API.

TL;DR

• The paper defines a grammar of prompt formats that differ only in separators, spaces, label casing and enumeration style, and treats formats generated from the same grammar as semantically equivalent §3.1.
• Accuracy spread (best minus worst format) is the sensitivity measure. Across LLaMA-2, Falcon and GPT-3.5 on Super-NaturalInstructions tasks, spreads are large and do not shrink with model size, few-shot count or instruction tuning §4.2.
• FormatSpread finds the best and worst format with a Thompson-sampling bandit under a budget of model evaluations, and its estimate can be close to the true spread §3.2 §4.5.
• Author-reported numbers. The official MIT-licensed code implements the bandit procedure format_evaluation.py; the results were not re-run here.

Problem

Prompt design choices, including formatting, are rarely reported, and papers usually assume that formatting does not matter much compared with data or model variance §1. Reporting one format can therefore misattribute gains to training data, parameter count or architecture, and can mislead users who happen to pick a poor format §1. Exhaustively scoring every format costs time linear in the number of formats, which becomes infeasible for a large space §3.2. The method is meant for black-box models: it needs no gradients or logits §1 §3.2.

Method

Format grammar §3.1 app. A.1 app. A.2 app. A.3. A prompt is an instruction, followed by n few-shot examples and a query, joined by a space character c. Each example is formatted with a format p, and the query's answer field is left empty for the model to fill §3.1. A basic field is a descriptor d, a separator s and a text placeholder, with a casing function applied to d. Fields combine into lists, and enumerations use an item wrapper and a number format. The sets of allowed separators, spaces, casings and item formats are fixed lists app. A.2. Two formats are equivalent when they apply the same grammar rule, have the same descriptors, and have equivalent sub-elements app. A.1. Contextual restrictions remove unnatural combinations, for example line breaks in a separator when the space has none, and an empty separator after a descriptor app. A.3. When a multiple-choice variable changes, all its uses change together, including the instruction text app. A.1.

Spread §3.2. For a set of formats p_1..p_n and a metric m (accuracy in this work), the spread is the maximum minus the minimum of m(p_i) over the set. A larger spread means more sensitivity to equivalent formatting.

\text{spread} = \max_i m(p_i, \mathcal{D}) - \min_i m(p_i, \mathcal{D}) §3.2

FormatSpread as a bandit §3.2 alg. 1. Each format is an arm. Pulling an arm evaluates a mini-batch of B examples not yet scored with that format. The total budget is E evaluations. The search runs twice: first for the highest-accuracy format with E/2, then for the lowest-accuracy format with E/2. The second search starts from the counts gathered by the first, which gives it an informative posterior §3.2 app. A.4. The paper compares Thompson sampling (used by FormatSpread), UCB and naive sampling, which gives each format E/n evaluations §3.2.

# Thompson sampling over prompt formats  [alg. 1] [§3.2]
# Input:  formats (K arms), budget (E), batch size (B),
#         objective ('max' or 'min'), priors (alpha, beta) per format
# Output: counts of successes and trials per format

def thompson_sampling(formats, budget, batch_size, objective, priors, counts):
    # Repeat for E / B batches in total  [alg. 1]
    for step in range(budget // batch_size):
        # Draw one plausible accuracy per format from its posterior  [§3.2]
        draws = []
        for prompt_format in formats:
            prior_alpha, prior_beta = priors[prompt_format]
            successes = counts[prompt_format].successes
            failures = counts[prompt_format].failures
            draws.append(sample_beta(prior_alpha + successes,
                                     prior_beta + failures))
        # Pick the highest draw for 'max', the lowest for 'min'  [alg. 1]
        if objective == 'max':
            chosen = argmax(draws)
        else:
            chosen = argmin(draws)
        # Score the next unseen batch of B examples on that format  [§3.2]
        correct = count_correct(chosen, batch_size)
        counts[chosen].successes += correct
        counts[chosen].failures += batch_size - correct
    return counts

# FormatSpread: best and worst format within a total budget E  [§3.2]
# Input:  formats (320 sampled plus the original), original accuracy x,
#         budget (E), batch size (B)
# Output: best format, worst format

def format_spread(formats, x, budget, batch_size):
    # Prior Beta(alpha, 5), alpha = max(5x / (1 - x), 1.1)  [app. A.4]
    alpha = max(5 * x / (1 - x), 1.1)
    priors = {f: (alpha, 5) for f in formats}
    # Phase 1: E/2 evaluations to find the highest-accuracy format  [§3.2]
    counts = thompson_sampling(formats, budget / 2, batch_size,
                               'max', priors, new_counts(formats))
    best = argmax_empirical_accuracy(counts)
    # Phase 2: reuse phase-1 counts, spend E/2 on the lowest  [app. A.4]
    counts = thompson_sampling(formats, budget / 2, batch_size,
                               'min', priors, copy(counts))
    worst = argmin_empirical_accuracy(counts)
    return best, worst

The UCB variant scores each arm as its empirical accuracy plus c * sqrt(log(t) / N_i), with c = 2 taken from a prior prompt-optimisation paper §3.2. Naive sampling skips the bandit and gives every format the same share of the budget §3.2.

Experimental setup §4.1. The 53 Super-NaturalInstructions tasks are listed in app. B.1; 19 are multiple choice and 34 are classification with two to four basic fields. Few-shot examples are chosen at random per task and fixed for each shot count. Models are LLaMA-2-7B, 13B and 70B, Falcon-7B, Falcon-7B-Instruct and GPT-3.5-Turbo. Two scoring metrics are used: exact prefix matching, and ranking accuracy (the expected answer is the highest-ranked valid option under the model's output distribution). Ranking accuracy is the default §4.1.

Analyses (summarised under Results): spread across formats and models §4.2; how individual constants (separators, spaces, casing, item wrappers, number formats) affect accuracy §4.3; single-character "atomic" edits and whether the format space is smooth §4.3; whether the last-layer prompt embedding identifies the format §4.4; search efficiency and new tasks (degeneration, instruction-induction tasks, open-ended generation) §4.5.

Hyperparameters

Results

Author-reported. Setup: dataset, model, hardware, baseline as noted per row. The hardware used is not stated in the paper. Metrics are on a 0 to 1 scale (ranking accuracy unless marked exact prefix matching), and spreads are in accuracy points unless shown as fractions.

Paper vs code

Code: the official repository github.com/msclar/formatspread (owner matches the first author), pinned at commit b7a4d63b97a96c7004354dd8dfd28adf00b2d69b, MIT licence. The paper links it in a footnote to its abstract §1.

• Set names are swapped. The appendix lists the descriptor-to-text separator set as the spacing list, and the space set as the separator list app. A.2. The code uses the opposite names: separators (descriptor to text) are the list with ':' and '::', and spaces are the list with ' -- ' and ' || ' grammar_definition.py. The grammar text defines the descriptor separator as S1 §3.1, so the code's naming is the one to implement.
• Item wrappers. The appendix writes the sixth wrapper as a visible-space glyph. The code uses ' )' grammar_definition.py app. A.2. Use the code's list.
• UCB formula. The paper uses c * sqrt(log(t) / N_i) §3.2. The code uses 2 * sqrt(log(1 + t) / N) with t from a zero-based loop counter format_evaluation.py. The UCB path is off in the main run (use_ucb_rule=False) format_evaluation.py format_evaluation.py.
• Thompson sampling and priors match. Prior b = 5, a = max(a, 1.1) with a = x / (1 - x) * b, posterior Beta(a + S, b + N - S), phase 2 reusing phase-1 counts format_evaluation.py format_evaluation.py format_evaluation.py. Budget is E / B steps, split E/2 per phase format_evaluation.py.
• Batch size. The best/worst table caption says B = 10, but the code's default is B = 20 main.py and the efficiency figure uses B = 20.
• Format sampling. The paper says only "a random sample" §3.2. The code draws each slot uniformly at random and keeps only valid formats until it has enough main.py main.py.
• Final choice. The code returns the format with the best (or worst) empirical accuracy among those sampled, not the posterior mean format_evaluation.py. The paper does not describe this step.

No other discrepancies found in the parts checked.

Pitfalls

• Ambiguous "last layer". The main text says the prompt embedding is the last hidden layer §4.4. An appendix caption says "last decoder layer" and refers to the output distribution app. B.3. These are not the same. Use the main text, and check the code before implementing the probe.
• Bayesian optimisation vs bandit. The introduction calls the method Bayesian optimisation §1, but the method section describes a bandit with Thompson sampling §3.2. Implement the bandit.
• Batch size and budget. Table caption B = 10, code B = 20. The budget E = 40,000 in the same table caption matches the text §4.5. Pick one and state it.
• Non-reproducible best/worst search. The code seeds Python's random but not NumPy, and Thompson sampling uses np.random.beta main.py format_evaluation.py. Repeated runs may choose different best and worst formats. Seed NumPy yourself to reproduce results.
• Numbers flagged for re-check. In the LaTeX source, many counts (number of tasks, some means and percentages) are wrapped in a macro whose definition comment says they were to be re-checked closer to the deadline. The rendered values are the ones shown in this card, but the source marks them as not final. Task counts also vary by analysis: 53 for model comparison, 31 for the feature and PCA analyses §4.3 §4.4. Check which count applies before comparing.
• Spread is a lower bound. With 10 formats per task, spreads are a lower bound on the true spread §4.2. FormatSpread's estimate uses only the sampled set of 320 formats, so the true space can be wider.
• Metric choice changes the answer. Exact prefix matching gives larger spreads than ranking accuracy, partly because the model sometimes gives no valid answer app. B.2 §4.5. Report which metric you use.
• Degeneration. Accuracy and the share of valid answers are strongly correlated §4.5, so a low score can mean the model gave no valid answer, not a wrong one.
• Equivalence is a definition. Formats are "equivalent" only under the paper's grammar and restrictions app. A.1 app. A.3. The paper notes that some equivalent formats look unnatural, and that one parse is chosen for ambiguous punctuation app. C.
• Cost. Each FormatSpread run costs up to E model calls. The introduction reports under 10 USD per task on average for GPT-3.5 §1. Prices and models change, so recompute the cost before budgeting.
• Limited scope. The main experiments cover classification and multiple-choice tasks with short inputs app. C. Open-ended generation is covered only in a smaller experiment with ROUGE-L and BERTScore §4.5. Do not extrapolate the headline numbers to long or open-ended tasks.

Lineage

• Builds on: gradient-free prompt optimisation with bandit or search methods (the paper cites Zhou et al. 2022 and Pryzant et al. 2023 for the bandit framing and UCB constant) §3.2. Thompson sampling follows Chapelle and Li (2011) §3.2. Not on the wiki yet.
• Compared against: naive sampling (E/n per format) and UCB §3.2 §4.5.
• Superseded by: not known from content on this wiki.