SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

In plain words

Benchmarks often rank language models by a score, and a common reading is that the model with the higher score is the better one. This paper asks how stable such a ranking is when the person running the benchmark makes ordinary setup choices: how the question is worded, how the model is asked to decode its answer, how many worked examples come before the question, and whether the answer is read from generated text or from the model's probabilities over the options. None of these choices is usually written up as part of the result.

The authors run three mid-sized open models on five alignment-related benchmarks, all on a single consumer-grade 4-bit setup. They then count, for each pair of models and each benchmark, how often the setup choices make one model look better and how often they make the other look better. The share of setups that go against the majority direction is the main number. If it is large, a claim such as "A is safer than B" says more about the setup than about the models.

The point is not that any one benchmark is broken. It is that a pairwise verdict is only meaningful together with the full set of harness choices that produced it, and that such choices can change the answer.

TL;DR

• For a fixed pair of models and a fixed benchmark, the strict ordering of the two models flips under some admissible harness choices on every benchmark tested §5.1 tab. 2.
• The headline metric is rho_flip, the smaller of the two strict-win counts divided by the number of configurations eq. 2. Its maximum is 0.5 by construction.
• Author-reported maxima are 0.479 (TruthfulQA), 0.438 (BBQ), 0.354 (ToxiGen), 0.292 (XSTest) and 0.250 (CrowS-Pairs, logprob only) tab. 2.
• Authors report that all six orderings of three models are reachable on XSTest under the full envelope; dropping the sampling-temperature stress setting reduces this to five §5.1 tab. 4.
• The envelope is small and hand-curated. The authors state that it is not a representative sample of community practice §6.

Problem

Benchmark-based comparisons of language models are read as quantitative verdicts, but they depend on evaluation details that papers often leave unspecified §1. Prompt wording, decoding settings, few-shot count, scoring rule and quantisation can each change scores, and earlier work showed that prompt choices can move scores by tens of points and that single-harness variance is often under-reported §2. Prior work mostly studies how much scores vary; this paper asks a narrower question: for a given pair of models, does the sign of their difference survive a plausible set of harness choices? §2 §4

The paper also argues that the existing variance measures do not answer a governance question such as "can an auditor move the verdict?" §2

Method

The method has three parts: a finite evaluation grid, a pairwise sign metric with a stated identifiability property, and an envelope of configurations chosen by documented practice.

1. Grid. Three models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Yi-1.5-9B-Chat) are run on five benchmarks (TruthfulQA, BBQ, CrowS-Pairs, ToxiGen, XSTest) with four prompt templates, three decoding settings, three few-shot counts and two scoring paths §3 tab. 1. All cells use NF4 quantisation §3. The full grid has 612 valid result cells §3.
2. Configuration envelope. A configuration is in the envelope only if each axis value is backed by documented alignment-benchmark or harness practice. Values are tagged core or stress. Core: templates T1 to T3, greedy and moderate decoding, and 0, 3 or 5 shots. Stress: template T4 (chain of thought) and diverse decoding at temperature 0.7 §5.1 app. T. Per benchmark, the envelope has 48 configurations (36 free-form, 12 logprob) and CrowS-Pairs has 12 logprob-only configurations §5.1.
3. Strict sign counts. For an ordered pair (A, B) on benchmark b, count the configurations where A scores strictly higher (N+), strictly lower (N-) and equal (N0) eq. 1.
4. Minority strict-sign mass. rho_flip(A, B, b) = min(N+, N-) / |C_b| eq. 2. It is zero exactly when one model strictly dominates on every configuration (ties allowed). Ties remain in the denominator §4 §5.1.
5. Identifiability proposition. Proposition 1 separates two conditions: (D1), no strict reversal (N+ = 0 or N- = 0), and (D2), a single sign verdict over the envelope (exactly one of N+, N-, N0 is non-zero). D2 implies D1, not conversely. rho_flip is zero exactly under D1. A positive rho_flip certifies that at least one admissible configuration reverses the strict ordering. It does not certify D2, nor a total ranking of the three models §4, prop. 1.
6. Ceiling. When |C| is small, rho_flip saturates at floor(|C|/2)/|C|. A saturated value shows that a reversal exists but not how large it is [§4, prop. 1 (iii)].
7. Attribution. A LightGBM regressor and a per-pair classifier are trained on the implementation axes, and SHAP values are reported per axis. The authors state that this surrogate only localises which axis drives a flip; the headline rho_flip is computed on the observed grid §5.1 app. S.

# Minority strict-sign mass for an ordered model pair on one benchmark  [eq. 2]
# Input:  scores[model][config] in [0, 1] for each config in C_b
# Output: rho_flip in [0, 0.5]
def rho_flip(scores, model_a, model_b, configs):
    # Count configs where A scores strictly above B  [eq. 1]
    n_plus = 0
    # Count configs where A scores strictly below B  [eq. 1]
    n_minus = 0
    for config in configs:
        if scores[model_a][config] > scores[model_b][config]:
            n_plus += 1
        elif scores[model_a][config] < scores[model_b][config]:
            n_minus += 1
    # Ties stay in the denominator but count for neither side  [eq. 1] [eq. 2]
    return min(n_plus, n_minus) / len(configs)

Scoring paths. Free-form scoring generates text and extracts a label with a regex. Logprob scoring takes the argmax over candidate log-likelihoods. The authors state these score different objects, not the same text through two parsers §3. Logprob is defined only under greedy decoding, and CrowS-Pairs admits only logprob §3 app. T.

Inference protocol. Greedy decoding gives one call per cell. Each sampling setting is run with five seeds and averaged, so each reported stochastic score is a five-seed mean §3 tab. 1.

Hyperparameters

Results

Author-reported. Setup: 15 model-benchmark pairs; envelope of 48 configurations per benchmark (12 for CrowS-Pairs); maximum over the three model pairs §5.1 tab. 2.

Scoring-path slices (not matched on decoding support) tab. 3:

Envelope tier ablation (rho_flip max, orderings of 6) tab. 4:

Other reported results (author-reported):

• Parse-clean subset (every model parse rate >= 0.95): TruthfulQA 0.479, BBQ 0.438 and CrowS-Pairs 0.250 are unchanged. ToxiGen rises to 0.361 (36 configs). XSTest falls to 0.282 (39 configs) but still shows six orderings §5.1.
• Stratified 80% TruthfulQA subsamples: median per-cell range 1.49 pp, maximum 3.83 pp; the induced orderings did not change on four core configurations §5.1 app. X.
• Variance attribution (four-way ANOVA, free-form slice): the implementation share exceeds the model share on BBQ, ToxiGen and XSTest; TruthfulQA is model-dominated. Interaction shares are 41 to 54 percent. rho (implementation over model) is 3.8 (BBQ), 14.9 (ToxiGen), 0.7 (TruthfulQA) and 2.4 (XSTest) §5.2 app. G app. V.
• Cross-package case study (Qwen2.5-7B-Instruct, bf16, greedy, 0-shot, 300 items): scores span 21.7 pp on TruthfulQA and 22.7 pp on BBQ across three packages. The packages score different candidate objects, so the spread is not attributed to individual axes §5.2 app. U.
• BF16 vs NF4 (54 configs on the conservative core): median absolute score difference 1.7 pp (mean 2.6 pp, max 20.7 pp). Three-model ordering matches on 17 of 18 paired cells §3 app. O.
• Scale probe: a two-family conservative-core probe on 7B to 32B and 9B to 34B models is reported as bounded, not a scaling law §1 app. Q.

Paper vs code

No official code.

The paper names no code repository, and its abstract and arXiv metadata do not either. The appendix states that the evaluation code, the rules file and the results will be released with the camera-ready version; no release is linked app. C. Searches run for this card:

• Links in the paper, abstract and arXiv record: none.
• GitHub repository search for "SafetyRepro": zero results.
• GitHub repository search for "rank_flip_exact" (the script named in the rules appendix): one result, v-code01/ecleak. Its owner is not an author, and it has no license, so it was not used app. T.
• GitHub repository search for "rank flip configuration benchmark": zero results.
• Author-name and project-page searches: not run.

Because no code could be checked, the implementation details below remain unverified against code: the exact validator, the parser, the surrogate model settings and the envelope enumeration.

Pitfalls

• Name of the metric. The paper uses "flip rate" for the total-order mismatch rate with expected value 5/6 under a uniform null §4, and uses "rank-flip rate" for rho_flip §5.1. Formally, rho_flip is the minority strict-sign mass, not a Kendall-type or total-order quantity. Its own text says so §5.1.
• Ceiling. rho_flip never exceeds 0.5. A value of 0.5 on a 4-config subset is the maximum possible there, not a 50% failure rate. Compare values only at the same |C| §4 [prop. 1 (iii)].
• Ties. N0 stays in the denominator. Ties count for neither side, so a pair with many ties has a low rho_flip even if the non-tied configs split. Check N0 before reading a value as instability eq. 1 eq. 2.
• Mixed scoring path. The TruthfulQA headline (0.479) drops to 0.361 on free-form configs alone. Part of the increase comes from mixing free-form and logprob scoring, which the authors treat as different measurands. Treat the headline as a joint, configuration-conditional quantity §5.1.
• Six orderings on XSTest. This result depends on the diverse T=0.7 decoding stress axis. Without it, XSTest reaches five of six orderings §5.1 tab. 4.
• Parse degradation. T4 parse rates are about 0.88 on ToxiGen and 0.91 on XSTest, so T4 cells are noisy for those benchmarks. The authors show a parse-clean recomputation with a 0.95 threshold §6 §5.1.
• Envelope choice. The envelope is hand-curated. The authors state that some decisions (highlighted cells, the thresholds, the conservative-core sub-envelope) were made after seeing results, while the axis values and metric were fixed in advance. The commit stamp does not make the envelope a representative sample §5.1 §6 app. T.
• Single item subset. Every benchmark uses one fixed item subset. Bootstrap ranges over configurations are not uncertainty over items or populations §3 §4 app. G.
• Stochastic cells. Sampling settings are five-seed means, so the rho_flip inputs are averages, not single runs §3.
• Bug history. Numbers come from the v3.0 rerun after eight code fixes. Some v2.2 results were superseded, including a right-truncation bug that changed wall-clock time and few-shot prompts app. M app. N. Use v3.0 values only.
• Packages. The cross-package spread (21.7 and 22.7 pp) is not decomposed by axis, and the packages score different objects. Do not read it as a library-quality ranking §5.2 app. U.
• Figures and appendix letters. The main figures are described in words only. The appendix letters used here follow source order (A for the prompt templates through Z for the claim-scope table) and should be checked against the PDF.
• Scope. Results are for three 7 to 9B models, NF4 on a consumer setup, and one harness. The paper does not claim population-level rank stability or closed-model behaviour §6.

Lineage

• Builds on: prompt-sensitivity and multi-prompt evaluation work §2; harness reproducibility studies §2; variance accounting for benchmark reporting §2; generalizability and construct-validity theory §2 §4.
• Compared against: score-variance studies within a single harness (cited as complementary, not superseded) §2.
• Superseded by: not applicable (no later version is known from this wiki).