ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

In plain words

Ask a language model the same question twice, with slightly different wording, and you may get a different answer. Many earlier studies measured this across a whole test set: they changed the wording and looked at how the average score moved. That hides the case where the model is right for one question wording and wrong for another, and it cancels out across questions.

This paper looks at each question on its own. For one question, it asks the model with several differently worded prompts, then measures how much the answers differ. The average of those differences over all questions is one number, called PromptSensiScore. A low number means the model behaves the same whatever the wording. The authors test several open models, look at how the number changes with model size and with a few worked examples in the prompt, and check how it relates to how sure the model is of its next word. The same idea is applied to open-ended answers, which a stronger model grades.

TL;DR

• PSS is the mean gap in score between prompt variants of the same instance, averaged over instances §2.2 eq. 1 eq. 2.
• Objective tests use 12 prompt variants per question on four datasets. Subjective tests use three versions of each prompt on two chat benchmarks §3.1 §4.1.
• Larger models and few-shot examples reduce PSS in the reported runs; the largest drop is from zero to one shot §3.3 §3.4.
• Higher top-token confidence goes with lower PSS on one multiple-choice dataset. The evidence is one figure and a qualitative claim §5.2 fig. 7.
• All numbers are author-reported on the paper's own prompt sets.

Problem

Benchmark scores for an LLM change with the prompt, the answer format and the decoding settings. Scores averaged over a dataset can hide per-question instability: a model can look stable on average while flipping on individual items §2.1 fig. 2. For open-ended tasks there is no exact answer to compare against, so the same question has to be judged on a graded scale instead §2.2 §4.1. The paper states that it does not know how prompt sensitivity arises in the model and proposes a confidence-based explanation §1 §5.

Method

Objective evaluation §3.1

1. Datasets: CommonsenseQA (five options, zero-shot), ARC-Challenge (four options, zero-shot), MATH (four-shot) and HumanEval (zero-shot).
2. Prompts: each item is written in 12 variants. The authors start from four styles (Simple Inputs, Role Player, Emotional Support, Output Requirement) with three hand-written prompts each. The paper does not say which prompt number belongs to which style app. D CommonSenseQA.py.
3. Decoding: greedy, for reproducibility §3.1.
4. Correctness Y(p): for multiple-choice items, the chosen option is compared to the gold option. For MATH, a Llama3-70B-Instruct call extracts the final answer and judges equivalence to the gold answer with a fixed prompt app. B. The paper does not say how HumanEval outputs are checked against tests.
5. Score each instance with eq. 1 and the dataset with eq. 2 (see the pseudocode below) §2.2.

Subjective evaluation §4.1 §4.2

6. Benchmarks: LC AlpacaEval 2.0 (805 instances) and Arena Hard Auto (500 questions) §4.1.1.
7. Variants: each original prompt is rewritten twice with an LLM (GPT-4o and GPT-4-0409, as named in the paper), giving three versions per instance. Rewrites are checked by BERTScore and by human annotators §4.1.3 tab. 1.
8. Models answer all three versions. The five tested models are InternLM2-20B-Chat, Llama3-8B-Instruct, Llama3-70B-Instruct, Qwen1.5-14B-Chat and Qwen1.5-72B-Chat §4.1.2.
9. Judging: the default judge GPT-4-1106-preview scores each response against the benchmark's reference model's answer. AlpacaEval gives a score in [0, 1]. Arena Hard Auto gives one of five labels, which the paper maps to 0, 0.25, 0.5, 0.75 or 1.0. Each pair is scored twice with the positions swapped, which gives 1000 scores §4.1.1 [fn. in §4.1.1].
10. PSS is computed over the three versions of each instance §4.2.
11. Categories: 500 Arena prompts are embedded with all-MiniLM-L6-v2 and clustered with k-means into 20 groups. GPT-4o names the groups. PSS per group is the mean of the five models' PSS values §4.3 fig. 6.

Model size and few-shot §3.3 §3.4

12. Model-size comparison: average PSS over the four objective datasets, plotted against parameter count fig. 4.
13. Few-shot: CommonsenseQA and ARC-Challenge with the 12 prompts. Qwen1.5 models from 7B to 110B are run at 0, 1, 3, 5 and 7 shots. The first k examples are shared across settings app. F.

Decoding confidence §5.1 §5.2

14. For one instance, confidence is the mean, over its prompts, of the probability of the most likely next token eq. 3. The model's confidence is the mean over instances. It is computed for Mistral-7B-Instruct, InternLM2-7B-Chat and InternLM2-20B-Chat on CommonsenseQA only §5.

Key equations, as printed in the paper:

S_i = \frac{\sum_{p_i, p_j \in P} \left| Y(p_i) - Y(p_j) \right|}{\binom{|P|}{2}} eq. 1

\mathrm{PSS} = \frac{1}{N} \sum_{i=1}^{N} S_i eq. 2

\mathcal{C} = \frac{\sum_{p \in P} P(t_{next} \mid p)}{|P|} eq. 3

# Per-instance sensitivity S_i  [eq. 1] [§2.2]
# scores: one Y(p) per prompt variant of instance i, values in [0, 1]
def instance_score(scores):
    # absolute gap for every unordered pair of variants  [§2.2]
    pairs = list(combinations(scores, 2))
    total = sum(abs(first - second) for first, second in pairs)
    # divide by the number of pairs, C(|P|, 2)  [eq. 1]
    return total / len(pairs)


# Dataset-level PSS  [eq. 2]
# all_scores: one list of variant scores per instance
def pss(all_scores):
    # mean of S_i over the N instances  [eq. 2]
    return mean(instance_score(scores) for scores in all_scores)


# Decoding confidence of one instance  [eq. 3] [§5.1]
# next_token_probs: one distribution over tokens per prompt variant
def instance_confidence(next_token_probs):
    # probability of the most likely token under each prompt  [§5.1]
    top_probs = [max(probs) for probs in next_token_probs]
    # average over the prompt variants  [eq. 3]
    return sum(top_probs) / len(top_probs)

Hyperparameters

Results

Author-reported. Setup: four objective datasets with 12 prompts each and greedy decoding §3.1; two subjective benchmarks with three prompt versions §4.1; a proprietary-model check on HumanEval app. E.

Objective (per-model scores and PSS are plotted, not tabulated):

• Llama3-70B-Instruct is reported as the most robust model overall §1 §3.3 fig. 3 fig. 4.
• Qwen1.5-14B-Chat is reported as the most sensitive model on MATH §3.2 fig. 3.
• On the easier multiple-choice and code sets, the average score and PSS are described as roughly linearly related §3.2 fig. 3.
• Few-shot examples lower PSS for all four Qwen1.5 models tested. The biggest drop is from 0 to 1 shot. Larger models gain more robustness as shots increase §3.4 fig. 5.

Subjective PSS tab. 2:

Category result §4.3 fig. 6: the knowledge-heavy categories named in the text (business solutions, IT problems) have lower PSS. Data visualisation and scripting tasks have higher PSS. The per-category values are only in the figure.

Rewrite checks tab. 1:

Proprietary models on HumanEval, 12 prompts app. E tab. 4:

Decoding confidence §5.2 fig. 7: the paper reports that lower PSS accompanies higher confidence across instances on CommonsenseQA, and that cross-model confidence values cannot be compared. The figure gives bars, not numbers.

Paper vs code

The repository contains prompt templates, the AlpacaEval and Arena Hard Auto inputs, and the README. It does not contain the PSS computation, the inference code, the judge configuration, or the decoding-confidence code. The README gives the same PSS formula as an image and in words README.md, so the formula can be checked against the paper but not against an implementation. Experiments are said to run on OpenCompass README.md §3.1.

Discrepancies found in the prompt templates, all in CommonSenseQA:

• Prompts 10 to 12 in the code tell the model to reply with A, B, C or D, but the item lists options A to E. The paper's appendix template for the same prompts says A, B, C, D or E CommonSenseQA.py app. D.
• Prompt 3 puts a line break before option E in the code, but the paper's template puts E on the same line as D CommonSenseQA.py app. D.
• Prompt 5 has a leading space before option E in the code CommonSenseQA.py. The paper's template has no space.

Discrepancy in the MATH template:

• Prompt 6 in the code ends after the problem text. The paper's template adds an answer line MATH.py app. D.

Other notes on the repository:

• The objective-benchmark templates for ARC-Challenge, CommonsenseQA, MATH and HumanEval are stored under prompts/subjective/. The directory name does not match their use ARC_Challenge.py.
• AlpacaEval has 805 entries in each of the v1 and v2 files. Arena Hard Auto has 500 lines in each of v1, v2 and v3. The mapping of v1 to v3 to original and rewritten prompts is not documented alpaca_eval_v1.json question_v1.jsonl.
• The few-shot examples and the rewrite prompt are in the paper's appendix only app. F app. G.

Pitfalls

• The PSS formula needs unordered pairs. Eq. 1 sums over pairs of prompts and divides by the number of unordered pairs, C(|P|, 2). The paper does not say that pairs are unordered. Summing ordered pairs would double the numerator and break the scale. Implement unordered pairs. For binary correctness, PSS is the fraction of prompt pairs whose correctness differs.
• The text mixes two meanings of Y(p). The paper calls it correctness for objective tasks and a judge score for subjective tasks. The formula is the same in both cases. Subjective PSS is the average absolute difference in judge score, not a probability.
• 0.1 is a rule of thumb. The paper says PSS below 0.1 means high robustness and that tasks differ in how much they need §2.2. No justification or distribution is given.
• The Arena Hard Auto scale is changed. The five labels are mapped to 0 to 1 instead of the original win/loss scoring for ELO. Results here are not comparable with the official leaderboard §4.1.1 [fn. in §4.1.1].
• Cross-benchmark claims are inconsistent. Section 4.2 says the Arena benchmark shows higher sensitivity than AlpacaEval. The values compared are from different judges and scales, and the table caption says the two benchmarks cannot be compared directly tab. 2. Within Arena, all five models are slightly below the Reference row (0.249 to 0.266 against 0.275).
• The Reference row is not a prompt baseline. It is the difference between two different models' answers under one prompt set, used only as a scale §4.2 tab. 2.
• Greedy decoding hides sampling variance. Each prompt is decoded once, so PSS reflects prompt change only, not run-to-run variation §3.1.
• MATH correctness depends on an LLM. The extractor and equivalence checker are Llama3-70B-Instruct. Its error rate is not reported app. B. Each mistake here changes PSS for that instance.
• Rewriter identity is inconsistent. Section 4.1.3 names GPT-4o and GPT-4-0409 as rewriters. Appendix G says the rewriter was GPT-4. Which version produced which rewrite is not stated.
• The judge's own variance is not measured. Each pair is scored twice with swapped positions §4.1.1, but no judge-run-to-judge-run variance is given.
• Few-shot coverage is narrow. Only Qwen1.5 models and shots 0 to 7 were tested. The authors say larger shot counts were not run §8. Shot examples are fixed and shared across settings §3.4.
• Confidence definition is ambiguous. The text defines confidence as the probability of the most likely token. It also says only one option-token probability is taken for multiple choice. It does not say whether the maximum is over the full vocabulary or over option tokens §5.1.
• The confidence table is not in the compiled paper. The source has a table with per-option-count confidence values, but its input line is commented out. Do not treat those numbers as paper results [src: table/confidence.tex].
• Proprietary-model rows have no snapshot dates. Model versions and access dates are not stated app. E tab. 4.
• Costs. The judge is GPT-4-class, and the authors limited the subjective study to three prompt versions for cost reasons §8.
• Figure and table numbers are inferred. Numbers here follow source order. Float placement in the compiled PDF may shift them. Check against the PDF before quoting numbers.

Lineage

• Builds on: prompt-variation studies such as PromptBench and the work of Pezeshkpour et al. (both cited in §1 and §2 without arXiv ids here). The paper changes the unit of analysis from dataset to instance §2.1.
• Compared against: dataset-level prompt-template analyses, cited in §2.1 fig. 2.
• Superseded by: not stated.