---
kind: concept
slug: benchmark-validity
title: Benchmark validity
summary: >-
  How to tell whether a benchmark score means what it claims: run-to-run variance, pass^k reliability, dataset diversity and held-out refreshes, controlled negatives, comprehension gates, judge checks, control probes, scoring-rule details, scoped naming, proxy calibration. The variants suit different constraints; most cited results are author-reported.
tags: [evaluation, benchmarks, statistics, reproducibility, human-evaluation, llm-judges]
papers: ["1510.03336", "1802.01886", "1803.02324", "1810.03505", "1907.10701", "1910.07407", "2007.10937", "2103.03098", "2103.09430", "2104.14478", "2106.15772", "2108.09091", "2110.00530", "2111.02840", "2111.15366", "2205.11930", "2209.09513", "2210.13445", "2212.07796", "2302.03494", "2302.08399", "2304.13029", "2305.07895", "2305.12474", "2307.03109", "2307.15343", "2310.06119", "2310.11440", "2310.11986", "2311.04076", "2311.17005", "2401.06209", "2402.13605", "2404.09556", "2409.18486", "2410.21276", "2411.00201", "2411.19408", "2501.08579", "2503.04404", "2505.17613", "2506.17455", "2507.06134", "2507.21504", "2508.07173", "2510.17638", "2605.16679", "2607.26148"]
related: [llm-evaluation, generative-model-evaluation, video-generation, spiking-neural-network, keyword-spotting, challenge-set-evaluation, dataset-curation, hyperparameter-optimization, adversarial-robustness, adversarial-evaluation, algorithmic-fairness, time-series-forecasting, hallucination-detection, llm-as-a-judge, human-evaluation, bootstrap-confidence-intervals, red-teaming, llm-agent, tool-use, multimodal-language-model, time-series-classification, systematic-generalization, distributional-alignment, multilingual-evaluation, functional-safety, novel-view-synthesis, neural-radiance-field, object-detection, graph-neural-network, convolutional-neural-network, agent-based-simulation, generative-adversarial-network, text-classification, visual-instruction-tuning, intrusion-detection, anomaly-detection, neural-machine-translation, semantic-segmentation, segmentation-evaluation, knowledge-graph-completion, molecular-representation-learning, robotic-grasp-generation, chamfer-distance, calibration, model-flops-utilization, arithmetic-intensity]
status: draft
updated_at: 2026-11-01
---

## Overview

A benchmark score comes from one dataset, one scoring rule and, usually, one
training run. The cited papers show that each of these parts can move the
number by more than the gap that a leaderboard is meant to reveal. Four
problems recur:

- **Noise.** Data splits, random seeds and hyperparameter search each change
  the score [arxiv:2103.03098 §2.2]. Repeated agent trials can change whether
  a task counts as solved at all [arxiv:2605.16679 §4.2].
- **Coverage.** The items may not test the claim. They may be near-duplicates
  [arxiv:2106.15772 §3.1], read as general when they are narrow
  [arxiv:2111.15366 §4.1], be easy to solve from surface patterns
  [arxiv:2111.02840 §1], may be solved from one sentence of a pair
  [arxiv:1803.02324 tab. 2], may already be in training data
  [arxiv:2305.12474 §5], or may carry signal that the task does not allow, such
  as frames that leak other camera views [arxiv:2210.13445 §1]. A handful of
  popular models can also hide a weakness that others expose [arxiv:1907.10701 §1].
- **Protocol.** Papers scale, split and score data differently, so their
  numbers do not compare [arxiv:2310.06119 §4.1]. A baseline can also be
  under-configured relative to the new method, which makes a gain look larger
  than it is [arxiv:2404.09556 §1]. Even one benchmark can change its protocol
  between versions [arxiv:2103.09430 §3]. The test split can also steer which
  checkpoint is reported [arxiv:2506.17455 §4.1].
- **Scoring.** The score itself can mislead. Judges can disagree with humans
  [arxiv:2507.06134 tab. 4], a refusal can mean nothing if the input was not
  understood [arxiv:2508.07173 §2.3.1], and a consistency score can reward
  uniform failure [arxiv:2508.07173 tab. 8]. A single metric can also hide a
  collapse: a text generator can score well on quality while its output
  narrows [arxiv:1802.01886 §1]. A constant output can also earn a zero fairness
  gap [arxiv:2110.00530 tab. 15].

The shared rule is that a score is valid only for the items, scale and protocol
it was measured on. Each variant below targets one of these problems.

## Variants

### Treat the score as a distribution

Bouthillier et al. randomise data split, initialisation, data order,
augmentation and dropout in every run, but run hyperparameter search only once.
They call this FixHOptEst(k, All) [arxiv:2103.03098 §3.2]. The decision rule
uses the fraction of paired runs in which A beats B, with a bootstrap interval
[arxiv:2103.03098 §4.1]. In their simulations, the single-point comparison gives
about 10% false positives and about 75% false negatives. The recommended rule
gives about 5% and about 30% [arxiv:2103.03098 §4.2]. These are author-reported
simulations on five case studies. Card:
[Accounting for Variance in Machine Learning Benchmarks](/papers/2103.03098).
The same problem shows in single-run leaderboards. A detection benchmark trains
one run per model and reports no repeats or intervals. On traffic signs,
YOLO11m (0.795) and YOLO11l (0.794) differ by 0.001 in mAP50-95, so that gap
cannot be separated from run-to-run noise [arxiv:2411.00201 fig. 12]. Card:
[YOLO Evolution](/papers/2411.00201).
The same gap appears in a fine-tuning benchmark that runs each model once. On
AQUA20, ConvNeXt's top-3 accuracy is 0.9882 and SwinV2-B's is 0.9876
[arxiv:2506.17455 tab. 3]. The notebook fixes one seed and one run per model,
and the paper reports no repeats or standard deviations, so a gap of 0.06
points cannot be judged [code: Aqua20_Main.ipynb#L82]
[arxiv:2506.17455 tab. 3]. Card:
[AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455).

### Check the dataset's diversity and labels

Miao et al. score each problem by how lexically similar it is to its closest
other problem, using BLEU in both directions. They average this into a corpus
score [arxiv:2106.15772 §3.1]. The new corpus, ASDiv, has a corpus score of
0.49, against 0.08 for the MathQA-C subset [arxiv:2106.15772 tab. 3]. Three
pre-LLM solvers reach about 36% accuracy on ASDiv, while one solver reaches 86%
on MathQA-C [arxiv:2106.15772 tab. 3]. The two corpora are not the same, so
this gap cannot be read as a like-for-like difference. Labels were checked by
solving each equation with SymPy and comparing it with the stated answer
[arxiv:2106.15772 §3.3]. Card:
[A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](/papers/2106.15772).

### Equal splits from two image sources

Darlow et al. build CINIC-10 by adding ImageNet images, downsampled to 32×32,
to the ten CIFAR-10 classes [arxiv:1810.03505 §2]. The result has 270,000
images in three equal splits of 9,000 per class [arxiv:1810.03505 §2]. The
equal split is the point, since CIFAR-10 has no validation split
[arxiv:1810.03505 §1]. Two validity risks follow from the design. The test
set includes random CIFAR-10 training images, so a model trained on CIFAR-10
has seen some test items; the paper does not discuss this
[arxiv:1810.03505 §2]. The ImageNet part was not vetted by hand, and the
authors say it is noisier than the CIFAR-10 part [code: README.md#L165]. The
two sources are not identically distributed by design [arxiv:1810.03505 §1]
[§4.1]. The six reference errors, from 8.55 to 18.00, come from CIFAR-style
models, and the paper does not state the training recipe behind them
[arxiv:1810.03505 tab. 1] [§5]. Card:
[CINIC-10 is not ImageNet or CIFAR-10](/papers/1810.03505).

### Remove leaked signal from the training input

Gao et al. show that the common monocular dynamic view-synthesis benchmarks
train on frames that "teleport" between cameras. The model then sees multi-view
information that a single camera would not record [arxiv:2210.13445 §1]. They
quantify this with two effective multi-view factors. One, Ω, needs depth and
optical flow, and segmentation. The other, ω, needs only the camera set-up and
frame rate, but it
assumes a fixed look-at point [arxiv:2210.13445 §3] [app. A]. Training on the
left camera alone, with no teleporting, lowers masked PSNR by 1–2 dB on the
existing datasets [arxiv:2210.13445 §1] [§5.1]. The new iPhone dataset scores
lower again for all methods [arxiv:2210.13445 §5.2]. The frame rates of the
existing datasets were chosen by hand, so ω carries that uncertainty
[arxiv:2210.13445 app. A]. Card:
[Monocular Dynamic View Synthesis: A Reality Check](/papers/2210.13445).

### Check a release's labels and prose against its tables

Luay et al. rebuild four network intrusion benchmarks as NetFlow records with
timing features. They describe the result with histograms, per-minute series
and spectrograms, and train no model [arxiv:2503.04404 §5] [§6]. Labelling is
underspecified: the paper gives no time tolerance for matching flows to the
ground-truth files [arxiv:2503.04404 §4.2]. The text and Table 4 disagree on
which attacks run on some days. For BoT-IoT day 1, the text names DDoS and DoS,
while the table lists only reconnaissance [arxiv:2503.04404 §5.3] [tab. 4]. The
"unique patterns" claim rests on visual comparison of spectrograms, with no
detection experiment [arxiv:2503.04404 §5.5] [§6]. Card:
[Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems](/papers/2503.04404).

Dataset papers can also disagree with themselves. MovieNet reports 60K
trailers for 33K movies in its text and 68K trailers in its genre table
[arxiv:2007.10937 §3.1] [tab. 3]. Scene boundaries are 42K in the body and 40K
in the introduction [arxiv:2007.10937 §3.2] [§1]. Its benchmark splits are by
movie, but the action and place splits are random by clip, with a claim of no
movie overlap that a new split would need to verify [arxiv:2007.10937 app. C.6]
[app. C.7]. The authors conclude that character detection is largely solved,
which rests on one split and one metric [arxiv:2007.10937 §4.3]. Card:
[MovieNet: A Holistic Dataset for Movie Understanding](/papers/2007.10937).

### Filtered adversarial test sets

Wang et al. generate adversarial examples against surrogate models for five GLUE
tasks. They keep those that transfer to other surrogates and change few words
[arxiv:2111.02840 §3.3]. Crowd workers then keep only examples where four of
five annotators agree with the original label [arxiv:2111.02840 §3.3]. Across
ten tested models, the average AdvGLUE score is 30.3 to 60.9, against 85.7 to
93.2 on GLUE [arxiv:2111.02840 tab. 4]. The authors present it as a diagnostic
set beside GLUE, not as a replacement [arxiv:2111.02840 app. A.8]. Card:
[Adversarial GLUE](/papers/2111.02840).

### Mined pairs scored as a pair

Tong et al. mine image pairs that CLIP ViT-L/14 treats as near-identical and
DINOv2 ViT-L/14 treats as different. The thresholds are a CLIP cosine above 0.95
and a DINOv2 cosine below 0.6 [arxiv:2401.06209 §2.1]. The paper does not say
how candidate pairs are enumerated [arxiv:2401.06209 §2.1]. The authors write
two questions for each of 150 pairs, and a pair counts only when both answers
are right [arxiv:2401.06209 §2.2]. Guessing then scores 25%
[arxiv:2401.06209 §2.3]. Most tested models fall below that level, with GPT-4V
and Gemini as the exceptions [arxiv:2401.06209 §2.3]. The human reference is
95.7% from four participants [arxiv:2401.06209 app. B.4]. The pairs and
questions are hand-written, and the paper reports no inter-annotator agreement
[arxiv:2401.06209 §2.2] [app. B.4]. The pattern labels come from GPT-4 prompting,
and how each question was assigned to a pattern is not stated
[arxiv:2401.06209 §3.1]. Closed-model runs date from late 2023
[arxiv:2401.06209 app. B.1]. Card:
[Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209).

### Hand-built probes of one claim

Ullman keeps the logic of two published theory-of-mind vignettes and changes
details, such as making the bag transparent. He then compares the probability of
each completion from GPT-3.5 [arxiv:2302.08399 §2]. Under the transparent bag,
the model gives 95% to "chocolate" for the belief prompt [arxiv:2302.08399
§2.1.1]. A probe needs no training. It reports one model, one prompt per
variant, no repeats and no statistics [arxiv:2302.08399 §2.1.1]. Card:
[Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](/papers/2302.08399).

### Hard subsets defined by a cue-only baseline

Gururangan et al. train a fastText classifier that sees only the hypothesis
of each pair. On SNLI it reaches 67.0%, and on MultiNLI matched and mismatched
it reaches 53.9% and 52.3%, against 34–35% for the majority class
[arxiv:1803.02324 tab. 2]. The crowd-written hypotheses carry cues tied to
labels, such as negation for contradiction [arxiv:1803.02324 §3.1] [§3.2]
[tab. 4]. Strong NLI models score much lower on the examples this classifier
gets wrong [arxiv:1803.02324 §4] [tab. 5]. The authors use that hard subset as
a check, not as a replacement for the full test set. Card:
[Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324).

### Rewrite static items so one frame cannot solve them

Li et al. start from nine spatial task families in image benchmarks and rewrite
each into a temporal task that a single frame cannot solve [arxiv:2311.17005
§3.1]. The result is 20 tasks, each scored on 200 multiple-choice questions
built from 11 public video datasets [arxiv:2311.17005 §3.1] [§3.2]. Options
are read from the reply to a fixed answer prefix, so no judge model is needed
[arxiv:2311.17005 §3.3] [tab. 9]. The results depend on the language model,
the frame count and the resolution [arxiv:2311.17005 tab. 2]. The authors
report 51.1% for VideoChat2 against 35.5% for VideoChat [arxiv:2311.17005
tab. 2]. Card:
[MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](/papers/2311.17005).

### Synthetic benchmark with a known generator

Zhu et al. use a random LSTM as an oracle and draw 10,000 sentences of length 20
from it as the "real" data, so the true distribution is known
[arxiv:1802.01886 §2.1] [§4.1]. The metrics are split by data source: the
oracle-based likelihoods are computed on synthetic data, and BLEU, Self-BLEU
and EmbSim on real data, because the oracle cannot produce meaningful words
[arxiv:1802.01886 §4.2] [§4.3]. The paper does not give the
oracle's initialisation, so an oracle-based number is only comparable with a
rebuilt oracle of the same kind [arxiv:1802.01886 §2.1]. Card:
[Texygen: A Benchmarking Platform for Text Generation Models](/papers/1802.01886).

### Reference points: human baselines and oracle bounds

Lu et al. report 88.40% human accuracy on their science questions, from a crowd
study with an exam qualification and three workers per set [arxiv:2209.09513
tab. 3] [app. B.2]. That figure is the authors' own study, so it is not an
independent ceiling. Their oracle runs put gold explanations in the prompt and
reach about 94%. They show that explanation quality matters, not that a model
can reach that score [arxiv:2209.09513 §5.4] [tab. 6]. Card:
[Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](/papers/2209.09513).

### Controlled negatives and seen versus unseen items

Ma et al. build a retrieval benchmark for compositional vision-language models.
Each hard negative differs from the true caption by one part: an atom is
replaced, two single-compound foils are joined, words are permuted, or a
negation is added [arxiv:2212.07796 §3.3] [§4.1] [§4.2]. Test items are split
by whether their atoms and compounds appeared in each training corpus
[arxiv:2212.07796 §3.2]. Recall@1 falls from seen to unseen compounds for all
seven systematicity models [arxiv:2212.07796 §5.1]. The largest drop is 12.57
points on one subset for one model, which is more than the combined-set drop
[arxiv:2212.07796 app. B.1]. Negative quality needs checking. Human annotators
judged 83.7% of productivity negatives and 86.0% of systematicity negatives to
be genuine [arxiv:2212.07796 §4.3]. The discussion puts HN-Swap and HN-Neg near
70% [arxiv:2212.07796 §6]. The seen and unseen labels depend on a scene-graph
parser, whose triplet recall on LAION-400M is 39.55% [arxiv:2212.07796 app. A.2].
Card:
[CREPE: Can Vision-Language Foundation Models Reason Compositionally?](/papers/2212.07796).

### Human preference as a system-level probability

Ethayarajh and Jurafsky argue that averaged Likert ratings can reverse the
true preference between two systems [arxiv:2205.11930 §3]. Their alternative,
system-level probabilistic assessment, asks each annotator for P[X ≻ Y] after
they read the same sample from both systems. The mean is then tested against
0.5 [arxiv:2205.11930 §5.2]. On GPT-3 story generation with 90 analysed
annotators, the authors report 5 of 5 expected preferences recovered, against
2 of 5 for the standard protocol [arxiv:2205.11930 §6.1]. Card:
[The Authenticity Gap in Human Evaluation](/papers/2205.11930).

### Shared pipeline and original-scale metrics

Shao et al. fix the data processing, training and evaluation in one pipeline.
They report errors on re-normalised data [arxiv:2310.06119 §4.1] [arxiv:2310.06119 §4.2].
On ETT datasets, the normalised MAE is about 0.4 to 0.5 and the re-normalised
MAE is 1.6 to 3.7 [arxiv:2310.06119 §4.2]. The released code defaults to
evaluating on normalised values, so the flag has to be set to match the
paper [arxiv:2310.06119 §4.2]. Card:
[Exploring Progress in Multivariate Time Series Forecasting](/papers/2310.06119).

Jiang et al. run grid-based and graph-based traffic models under one split,
one training setting and one set of error measures [arxiv:2108.09091 §5.1].
On four grid datasets, STDN has the lowest RMSE of the grid models
[arxiv:2108.09091 tab. 4]. Their released graph code fits the scaler on the
whole series, test period included, before the split [code:
workMETRLA/pred_DCRNN.py#L187-L189]. A reproduction should fit the scaler on
training data only, and the paper does not say which was used. The authors'
grid code reports only rescaled MSE, so the grid MAE and MAPE in the paper
cannot be checked against it [code: grid/workTaxiBJ/predflowio/predflowio_STDN_noflow.py#L146].
Card: [DL-Traff](/papers/2108.09091).

### Scoped benchmarks and test suites

Raji et al. argue that general benchmarks such as ImageNet, GLUE and SuperGLUE
lack construct validity as measures of general ability [arxiv:2111.15366 §1].
They recommend naming a benchmark after the concrete task it tests. For broader
claims, they suggest test suites, audits, error analysis, ablation and
model-property measures [arxiv:2111.15366 §5] [arxiv:2111.15366 app. A]. This
is a position paper with no experiments, and the alternatives come without
procedures or thresholds [arxiv:2111.15366 app. A]. Card:
[AI and the Everything in the Whole Wide World Benchmark](/papers/2111.15366).

### Layered, context-aware safety evaluation

Weidinger et al. argue that a safety test of a model alone does not show harm
in use. They split evaluation into capability, human interaction and systemic
impact [arxiv:2310.11986 §2]. Their validity practices are to ground each
measure in literature, expert input or curated examples, to document the limits
of each evaluation, to cross-validate measures of the same concept, and to keep
results readable so that failures are not hidden in one aggregate score
[arxiv:2310.11986 §4.1.1]. The framework has no scoring rule, and it gives no
acceptance threshold: the authors say a normative threshold must come from a
legitimate process [arxiv:2310.11986 §2] [§5.3.2]. Their survey of evaluations
is a snapshot and not a systematic search [arxiv:2310.11986 §3.2]. Card:
[Sociotechnical Safety Evaluation of Generative AI Systems](/papers/2310.11986).

### Broad, small-sample evaluation of a closed model

Zhong et al. evaluate o1-preview on 27 tasks, mostly graded by the authors and
often on a few dozen items [arxiv:2409.18486 §4.15]. Coding uses 12 contest
problems, where each problem is 8.3 points of pass rate [arxiv:2409.18486
tab. 2]. Radiology uses 10 reports [arxiv:2409.18486 §2.2]. The model and
settings are not fully specified, and contamination is not analysed
[arxiv:2409.18486 §4.1]. Use it for breadth and examples, not for ranking. Card:
[Evaluation of OpenAI o1](/papers/2409.18486).

### Penalised scoring for hallucination

Med-HALT scores a correct answer +1 and a wrong one −0.25, so guessing earns
less than knowing [arxiv:2307.15343 §5.3]. The fake-question scorer
credits any answer that contains one of a fixed set of negative phrases, so the
accuracy is an upper bound [arxiv:2307.15343 §5.3]. The published score column
is not the average in the paper's equation, so check the formula before using
its numbers [arxiv:2307.15343 §5.3]. Card:
[Med-HALT](/papers/2307.15343).

### Timing-aware score for streaming detectors

Lavin and Ahmad score a streaming anomaly detector by when it fires inside an
anomaly window, not only whether it fires [arxiv:1510.03336 §II.B]. Only the
earliest detection in a window counts, and its credit falls off across the
window [arxiv:1510.03336 §II.C] [eq. 1]. Detections outside every window are
penalised, and a window with no detection is a false negative
[arxiv:1510.03336 eq. 2]. The raw sum is rescaled so that a perfect detector
scores 100 and a null detector 0 [arxiv:1510.03336 eq. 4]. On the standard
profile the authors report 64.7 for Numenta HTM, 47.1 for Twitter ADVec and
35.7 for Etsy Skyline [arxiv:1510.03336 tab. I]. The paper gives ranges for
the four outcome weights but no values, so the repository's profile weights
are the only numeric source [arxiv:1510.03336 §II.C] [code:
config/profiles.json#L4-L7]. The released code also differs from eq. 1 at the
right edge of a window [code: nab/sweeper.py#L165]. Card:
[Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark](/papers/1510.03336).

### Substring match for free-text replies

Liu et al. score 1000 OCR-style questions for large multimodal models with one
rule: a reply counts if the ground-truth string appears in it
[arxiv:2305.07895 §3.1]. The final score is the sum, out of 1000
[arxiv:2305.07895 §3.3] [tab. 3]. The authors report 659 for Gemini, 645 for
GPT4V and 514 for Monkey [arxiv:2305.07895 tab. 3]. The rule needs no judge
model, but a reply passes if the answer appears inside a longer or negated
sentence [inferred from code: OCRBench/example.py#L141-L151]. The paper drops
answers under four symbols to reduce false positives [arxiv:2305.07895 §3.1].
Refusals and format changes score as errors: GPT4V refused 84 images on safety
grounds, and the scorer counts those as wrong [arxiv:2305.07895 §3.3]. The
released scorer also lowercases the text, and removes spaces for handwritten
maths, before matching [code: OCRBench/example.py#L128-L151]; the paper does
not state either step. Card:
[OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](/papers/2305.07895).

### Survey share as the gold label

Lee et al. score KorNAT's social-value items against a survey rather than one
right answer. Each question's ground truth is the share of 6,174 Korean
respondents who picked each option, and a model that picks option k earns that
share [arxiv:2402.13605 §3.1.3] [§4]. Three scores follow from this: SVA on
five options, A-SVA with agree and disagree merged, and N-SVA, which replaces
the truth with a neutral label when neither merged share is above 0.5
[arxiv:2402.13605 §4]. The paper gives the maxima 0.450, 0.626 and 0.625, and
calls the SVA maximum empirically calculated rather than derived
[arxiv:2402.13605 §4] [tab. 1]. Common knowledge is plain accuracy against a
0.6 reference, taken from a Korean GED pass mark [arxiv:2402.13605 §4]. The
paper names no reference for social values [arxiv:2402.13605 §1]. Survey
weights for age, gender, education, residence and income are applied to the
shares [arxiv:2402.13605 §3.1.4]. Refusals are omitted, and whether they count
as wrong is not stated [arxiv:2402.13605 §5.4]. Card:
[KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](/papers/2402.13605).

### Non-contact proxy for an unmeasured quantity

Greenland et al. want a grasp score for soft grippers without force sensors.
They use deformation as the stand-in: the object is scanned before and during
the grasp, the two clouds are aligned with ICP, and the Density-Aware Chamfer
Distance between them is taken [arxiv:2411.19408 §1] [§3.1] [§3.2]. That
distance is combined with the grasp outcome and drop time into a score in
[0, 1] [arxiv:2411.19408 eq. 1]. The sensitivity constant of the distance is
not stated in the paper [arxiv:2411.19408 §3.1], so scores from different
implementations may not share a scale. All 900 grasps succeeded, so every score
falls in the upper half of the range and depends on deformation alone
[arxiv:2411.19408 §5] [eq. 1]. The results compare grippers, not the score
against a measured force [arxiv:2411.19408 §5]. Rankings are read from
overlapping distributions with no significance test [arxiv:2411.19408 §5.3].
Other hardware can be substituted, but the paper runs only one camera pair,
arm and printer [arxiv:2411.19408 §4]. Card:
[SoGraB: A Visual Method for Soft Grasping Benchmarking and Evaluation](/papers/2411.19408).

### Held-out and refreshed test sets

Middlehurst et al. re-run a time-series classification bake off on the 112
UCR problems, which many methods were built against. They add 30 new datasets
to check that the earlier choices were not tuned to the old collection
[arxiv:2304.13029 §5] [§7]. The top accuracy clique on the new datasets holds
MR-Hydra, HC2, RDST, QUANT and FreshPRINCE, so HC2 is not alone at the top
[arxiv:2304.13029 §5] [fig. 41]. The gap between HC2 and MR-Hydra is small on
the new datasets and on the combined 142 datasets [arxiv:2304.13029 §5].
Critical difference diagrams can disagree with the pairwise win counts, so the
authors suggest checking the matrices before ranking [arxiv:2304.13029 §5].
Zhang et al. re-test an exam benchmark on a 2023 objective-only follow-up set.
Scores drop for some models, by 6.7 points for ChatGLM-6b and 0.6 for GPT-4-0613,
and the authors call the change within normal difficulty variation
[arxiv:2305.12474 tab. 5] [§5]. They also accept that the questions may already
be in training data [arxiv:2305.12474 §5]. Cards:
[Bake off redux](/papers/2304.13029) and
[Evaluating the Performance of Large Language Models on GAOKAO Benchmark](/papers/2305.12474).
A 2023 survey of LLM evaluation argues that static test sets are not enough for
LLMs, because their capabilities change over time and their training data often
overlaps with public benchmarks [arxiv:2307.03109 §2.2] [§7.4]. It lists dynamic
and evolving evaluation as an open challenge but gives no procedure for building
one [arxiv:2307.03109 §7.4]. The survey runs no experiments, so the claim is a
position, not a measured effect. Card:
[A Survey on Evaluation of Large Language Models](/papers/2307.03109).

### LLM judges checked against humans

Two papers use a model as the grader and report how it agrees with people.
Zhang et al. grade subjective exam answers with a GPT-4-turbo judge that reads
teacher marking criteria. Question-level Spearman agreement with human scores is
0.825 to 0.864 across five models [arxiv:2305.12474 tab. 4]. With criteria,
GPT-4-0613 reaches 0.854, against 0.820 without them [arxiv:2305.12474 tab. 6].
Converted totals stay within about 2% for sciences and about 5% for humanities,
and the humanities gap comes from matching answers to marking points
[arxiv:2305.12474 §4.4]. Question-level agreement is not agreement on totals
[arxiv:2305.12474 tab. 4].

Vijayvargiya et al. run a rule check and a GPT-4.1 judge on the same agent trajectories
[arxiv:2507.06134 §3.3]. The two disagree on 5.69% to 14.95% of tasks, depending
on the model [arxiv:2507.06134 tab. 3]. On 100 GPT-4o trajectories, the judge
labels 35.9% unsafe, where human annotators label 71.7%, and the authors treat
their unsafe rates as lower bounds [arxiv:2507.06134 tab. 4] [§4.3]. Cards:
[Bake off redux](/papers/2304.13029),
[Evaluating the Performance of Large Language Models on GAOKAO Benchmark](/papers/2305.12474) and
[OpenAgentSafety](/papers/2507.06134).

MMMG shows the coverage side of the problem. Its human study covers 41 of its
55 tasks and 978 instructions, so the 94.4% overall agreement does not extend to
the other tasks [arxiv:2505.17613 §1]. The per-modality agreement figures in the
introduction average to about 93.4%, not 94.4% [arxiv:2505.17613 §1]. Its
judges are kept narrow: a vision-language model answers one question per
criterion, as yes/no or multiple choice, and other tasks use similarity models,
audio models or programs [arxiv:2505.17613 §3.2]. The paper says each judge
threshold was matched to human judgement, but it gives no values, and the
released code uses fixed per-class thresholds [arxiv:2505.17613 §3.2]
[code: eval_audio.py#L66]. Card:
[MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613).

Prophet Arena checks its reasoning judge the same way. The judge's mean
absolute difference from 170 human-rated forecasts is 0.37 to 0.44 on a 0 to 4
scale, while the judge's own table uses a 1 to 5 scale, so the two sets of
numbers need care when compared [arxiv:2510.17638 app. C.6]
[app. C.6.1]. Card:
[LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](/papers/2510.17638).

### Comprehension gates and consistency across variants

Pan et al. render each of 972 harmful seed prompts in 24 input variants that mix
text, image, video and audio, giving 23,328 cases [arxiv:2508.07173 §2.2] [tab. 2].
A response counts toward the attack-success and refusal rates only if a probe
shows the model understood the input. Otherwise a model that ignores the input
looks safe [arxiv:2508.07173 §2.3.1]. Their consistency score takes the
standard deviation of per-variant Safety-scores and maps it through an
exponential with α = 5 [arxiv:2508.07173 §2.3.2]. Consistency alone does not
show safety: Unified-IO2 has the highest consistency score and the lowest average
Safety-score [arxiv:2508.07173 tab. 8]. Human accuracy on the comprehension flag
is 94.2% on 500 responses [arxiv:2508.07173 §3.3] [tab. 12]. The released code
does not implement the Safety-score or the consistency score, so a
re-implementation has to follow the paper's formulas [arxiv:2508.07173 §2.3.2].
Card:
[Omni-SafetyBench](/papers/2508.07173).

### Reliability over repeated agent trials

Chen et al. run each of 75 healthcare workflow tasks three times and report
pass@1, pass@3 and pass^3 [arxiv:2605.16679 §4.1] [app. F.2]. pass^3 counts a task
only when all three trials pass [arxiv:2605.16679 app. F.2]. For the leading
configurations, pass^3 falls well below pass@1: Opus 4.6 goes from 28.0 to 18.7,
and GPT-5.5 from 20.9 to 9.3 [arxiv:2605.16679 §4.2] [tab. 2]. Task-level
bootstrap intervals overlap heavily at the top, so the leader is not settled:
28.0 [19.6, 36.9] against 26.2 [18.2, 33.8] [arxiv:2605.16679 tab. 2]. The judge
is in the same model family as several evaluated Claude configurations, which
the authors say may bias results [arxiv:2605.16679 app. F.3]. Card:
[CHI-Bench](/papers/2605.16679). A 2025 survey of agent evaluation lists pass^k
as a consistency metric and cites a benchmark that uses it
[arxiv:2507.21504 §3.3] [§5.2]. The survey runs no experiments, so it gives
no values to compare. Card:
[Evaluation and Benchmarking of LLM Agents: A Survey](/papers/2507.21504).

### Driver-scored success with few replicated cells

Zhou et al. run a coding agent zero-shot on R2R-CE navigation, with a monocular
camera, four moves and no map [arxiv:2607.26148 §3.1] [§3.2]. The driver scores
success after the session ends: the agent must issue STOP itself within 3 m of
the goal, and a run that exhausts its budget scores zero [arxiv:2607.26148
app. A]. Only the default-effort Claude SDK cells have three runs. Their
sample standard deviations are 1.2 to 3.5 points, and the authors treat
differences of a few points as descriptive [arxiv:2607.26148 §4.1] [tab. 2].
Most ablation cells are single runs of 100 episodes [arxiv:2607.26148 tab. 4].
Harness comparisons are not controlled, because serving paths, defaults and
versions differ [arxiv:2607.26148 §4.3]. Of 30 failures in one strong run, 23
end with a claimed success while the final distance is 3.1 to 34.1 m
[arxiv:2607.26148 §4.6] [tab. 7]. Card:
[Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation](/papers/2607.26148).

### Position papers on simulated humans

A position paper argues that LLM-based human simulations are not yet reliable
enough to trust [arxiv:2501.08579 §1]. Its table of cited studies shows the
gaps. On emotion understanding, GPT-4 reaches 58% against 70% for humans
[arxiv:2501.08579 tab. 1]. Its fixes, including validation at several levels
in one loop, are proposals and are not evaluated [arxiv:2501.08579 §5] [alg. 1].
The table mixes tasks and models, so it shows the size of each gap but is not a
like-for-like comparison [arxiv:2501.08579 tab. 1]. Card:
[LLM-based Human Simulations Have Not Yet Been Reliable](/papers/2501.08579).

### Control probes that should show no effect

Tjuatja et al. test whether language models show the response biases that
people show in survey wording. Each bias edit is paired with non-bias edits such
as typos, a random middle swap and a letter swap, which humans are expected to
ignore [arxiv:2311.04076 §2.1] [§2.3]. Every model shows a significant shift
on some typo control [arxiv:2311.04076 §3.1]. A shift on a control therefore
means sensitivity to noise, not evidence of the target bias. The paper does not
state a multiple-comparison correction across its many tests
[arxiv:2311.04076 §2.3]. The allow/forbid set has only 40 question pairs
[arxiv:2311.04076 §2.1]. Opinion-matching does not show a model behaves like
human respondents: representativeness does not track bias-consistency
[arxiv:2311.04076 §5]. Card:
[Do LLMs exhibit human-like response biases? A case study in survey design](/papers/2311.04076).

### Proxy inputs and undisclosed test sets

The GPT-4o system card scores speech by converting existing text benchmarks to
audio with a TTS system and scoring the text of the reply [arxiv:2410.21276
§3.2]. The authors say TTS is lossy for equations, code and formatting-heavy
text, so a low score can come from the converter rather than the model
[arxiv:2410.21276 §3.2]. The card names the TTS system but not the voice settings
per test, nor the transcription system used for scoring [arxiv:2410.21276 §3.2].
Several safety and capability results come from internal datasets whose sizes
are not stated [arxiv:2410.21276 §3.3.2] [§3.3.4]. The voice capability results
use a fixed random subset, so they should not be compared with published scores
[arxiv:2410.21276 §3.3.3]. The speaker-identification gains are 14 and 12 points
in the text, but 15 and 13 points in the table [arxiv:2410.21276 §3.3.2] [tab.
2]. Card: [GPT-4o System Card](/papers/2410.21276).

### Matched baselines for architecture claims

Isensee et al. benchmark CNN, Transformer and Mamba 3D segmentation models on
six datasets, using five-fold cross-validation on nnU-Net's splits
[arxiv:2404.09556 §3.2] [tab. 1]. They name four pitfalls: extra training
signal bundled with the new method, baselines that are not well configured,
datasets that cannot tell methods apart, and inconsistent reporting
[arxiv:2404.09556 §2.1] [§2.2]. Their headline is that most recent
Transformer and Mamba claims do not hold against the original nnU-Net, and
that the CNN variants lead [arxiv:2404.09556 §4]. For the Mamba claim, the
No-Mamba Base ablation is within 0.25 points of U-Mamba Bot on every dataset
in table 1, so the paper attributes the earlier gain to the residual encoder
[arxiv:2404.09556 §4] [tab. 1]. Reaching the best ResEnc result needs a larger
GPU budget: on KiTS fold 0, Auto3DSeg with eight GPUs reaches 87.77 DSC,
against 87.91 for nnU-Net ResEnc M on one GPU [arxiv:2404.09556 tab. 2]. The
suitability ratio also depends on the method set: for ACDC it falls from 357%
to 102% when the framework runs are removed [arxiv:2404.09556 tab. 3]. Card:
[nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556).

### Sampled negatives and version changes

Hu et al. release three large graph tasks: node classification on MAG240M,
link prediction on WikiKG90M and graph regression on PCQM4M
[arxiv:2103.09430 §2]. For WikiKG90M v1, each true tail is ranked among 1000
sampled negatives, and only the top ten are scored, so a true answer outside
them scores zero [arxiv:2103.09430 §2.2]. A duplicate id in any top-ten list
sets the whole score to zero [code: ogb/lsc/wikikg90m.py#L210-L212]. Under this
protocol, ranking candidates by training frequency alone reaches 0.75
validation MRR, so the top v1 scores are near saturation
[arxiv:2103.09430 §2.2] [§3]. In v2 the candidate tails are no longer given,
and the paper reports ComplEx-Concat at 0.1833 MRR. It does not say which split
that figure is from [arxiv:2103.09430 §3] [app. C]. The v1 test figure for the
same model is 0.8637 [arxiv:2103.09430 tab. 5]. For PCQM4M, the authors meant
to make a scaffold split, but a pre-processing bug produced a split by PubChem
ID, which they found gives similar results [arxiv:2103.09430 §2.3]. The two
GIN-virtual validation figures in the paper also differ, 0.1396 and 0.1410
[arxiv:2103.09430 tab. 6] [tab. 7]. Winning KDD Cup entries used ensembles and,
for MAG240M, target labels as input, so they are not single-model comparisons
[arxiv:2103.09430 §2] [§2.1]. Card:
[OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](/papers/2103.09430).

### Expert error annotation as the reference

Freitag et al. have professional translators mark every error span in WMT20
English-German and Chinese-English outputs, each with a category and a
severity, and turn the marks into a weighted segment score [arxiv:2104.14478
§3.3] [tab. 1]. Minor errors weigh 1, Major errors 5 and Neutral errors 0.
Each segment keeps at most five target errors, and a Non-translation label
scores 25 for the segment [arxiv:2104.14478 app. B] [tab. 1]. The Major weight
came from a resampling check over 1 to 10, which the paper summarises without
releasing code [arxiv:2104.14478 §3.3]. Under this scoring, the system order
changes from the WMT ranking, and every human translation ranks above every MT
system [arxiv:2104.14478 §4.1] [tab. 3] [tab. 4]. The two language pairs also
used different WMT protocols, a reference-based monolingual one for
Chinese-English and a source-based one for English-German, so the comparison
cannot separate protocol from rater quality [arxiv:2104.14478 §3.1] [§4.1]. The
text gives a ±20% band for English-German raters, but table 7 shows rater 3 at
1.23 times the average [arxiv:2104.14478 §4.4] [tab. 7]. The rating counts needed to
reach an expected Kendall tau of 0.9, 951 per system for English-German and
3720 for Chinese-English, are simulated, not measured [arxiv:2104.14478 §4.5] [tab. 8]. Card:
[Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](/papers/2104.14478).

### Failure archives as hard cases

Borji gathers ChatGPT failures, mostly from public posts and some from the
author's own prompts, and sorts them into eleven categories
[arxiv:2302.03494 §1] [§2]. The archive has no counts, no sampling rule and no
baseline, and the author does not say how often any failure occurs
[arxiv:2302.03494 §2]. Several examples name no model version, and the author
notes that some errors may already be fixed [arxiv:2302.03494 §2.7]. Some
labels are the author's own reading, including one natural-language-inference
item and the political-leaning item, which relies on a third-party analysis
[arxiv:2302.03494 §2.2] [§2.10]. Rewording can change the answer: the altered
family-count question gives 8 where the original gives the correct 5
[arxiv:2302.03494 §4]. The rephrased hotwiring request gets past a refusal, so
such items belong in a safety test set, not a capability list
[arxiv:2302.03494 §2.10]. Card:
[A Categorical Archive of ChatGPT Failures](/papers/2302.03494).

### Spike datasets from one fixed encoder

Cramer et al. release two spoken-digit sets already in spike form: Spiking
Heidelberg Digits, recorded by the authors, and Spiking Speech Commands, derived
from Speech Commands v0.02 [arxiv:1910.07407 §2.1]. One fixed cochlea-inspired
model turns the audio into 700 spike channels, so users do not each choose an
encoder [arxiv:1910.07407 §2.2]. The SHD test set holds out two speakers
completely, which measures speaker generalisation [arxiv:1910.07407 §2.1.1].
Spike-count classifiers score lower than classifiers that see spike timing.
Linear SVMs on counts reach 56.0% on SHD and the RBF SVM 60.0%, against 89.0%
for an LSTM on 10 ms bins [arxiv:1910.07407 tab. 1]. The authors read this as
evidence that spike timing carries the signal [arxiv:1910.07407 §3]. The text
gives 85.7% for the same LSTM setting [arxiv:1910.07407 §3.1]. The released
conversion code is LGPL-2.1 and differs from the paper in several parameter
values and in its output format [code: lauscher/transformations/hair_cell.py#L16-L24]
[code: lauscher/spike_train.py#L25-L27]. The authors say the datasets may not
prove final and that their accuracies are lower bounds [arxiv:1910.07407 §4].
Card: [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](/papers/1910.07407).

### Metric weights fitted to human ratings

Liu et al. score 700 text-to-video prompts with 17 automatic metrics in four
aspects. They fit one linear model per aspect to 1–5 human ratings, then sum the
four aspect scores [arxiv:2310.11440 §4.5]. On the authors' correlation table,
the fitted score is close to a plain metric average: 55.4 against 55.0 for
visual quality and 32.3 against 31.9 for text-video alignment, in Spearman
terms [arxiv:2310.11440 tab. 3]. The fit's advantage is small, and on
text-video alignment its Kendall value is lower than the average's (22.5 against
22.7) [arxiv:2310.11440 tab. 3]. The text calls the advantage consistent
[arxiv:2310.11440 app. B.2]. Single metrics vary widely: CLIP-Score has a
Spearman of 6.3 against 26.7 for BLIP-BLEU [arxiv:2310.11440 tab. 3]. The
authors' "Avg." rows are not defined, and a recomputed average does not match
them [arxiv:2310.11440 tab. 3]. Released code computes some metrics differently
from the paper: Action-Score is a confidence-weighted label similarity, not a
classification accuracy [code: metrics/mmaction2/demo/action_score.py#L93-L122],
and frames without OCR output count as error 0 [code: metrics/ocr_score.py#L88].
The fitted weights are printed only in the code, and the ratings and fitting
script are not in the repository [code: eval_from_metrics.py#L53-L68]. Card:
[EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](/papers/2310.11440).

### Parameterised sweep plus a named software stack

Real-model benchmarks can miss problems that a sweep finds. Transformer alone
would hide a TPU memory bottleneck that shows in fully connected layers wider
than 4k nodes [arxiv:1907.10701 §1]. A generator of FC, CNN and RNN training
models, swept over depth, width, batch and input size, covers that space
[arxiv:1907.10701 §2.1] [tab. 2]. Each platform wins somewhere, so the winner
depends on the model family [arxiv:1907.10701 §5]. Two confounders sit inside
the speedups. Real models run larger batches on the TPU, so part of a
real-model speedup is batch size [arxiv:1907.10701 §5.3] [§2.2]. Software also
moves the number: the 90th-percentile RNN speedup on TPU was 9.7x across
TensorFlow 1.7 to 1.12 [arxiv:1907.10701 §6.1]. Synthetic data is made on the
host, so input-pipeline cost is left out and the TPU figures are device numbers
[arxiv:1907.10701 §4.4]. The paper reports no repeat count [arxiv:1907.10701
§4.4]. Card:
[Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](/papers/1907.10701).

### Fairness gaps checked against a constant classifier

A fairness gap can be zero for a classifier that ignores one class. The survey
of 15 tabular datasets fits logistic regression on each one. On Credit card
clients and Diabetes the classifier predicts only the negative class, and on
OULAD only the positive class. Their statistical parity and equalized odds are
0.0 and their balanced accuracy is 0.5 [arxiv:2110.00530 tab. 15]. The paper
calls 0.0 the best value [arxiv:2110.00530 §4.2], but for these rows it
reflects a constant output. The analysis uses one 70/30 split with seed 42, and
differences near 0.01 cannot be separated from split noise [arxiv:2110.00530
§4.1]. Ricci's perfect scores rest on a test split of about 36 rows
[arxiv:2110.00530 tab. 1]. Card:
[A survey on datasets for fairness-aware machine learning](/papers/2110.00530).

### Checkpoint selection kept off the reported split

Fuad et al. fine-tune 13 ImageNet-pretrained models and report results on their
test split [arxiv:2506.17455 §4.1] [tab. 3]. The paper says it keeps the
checkpoint with the best evaluation performance and names no validation split
[arxiv:2506.17455 §4.1]. The released notebook saves the checkpoint with the
lowest test loss and stops after 10 epochs without a new best
[code: Aqua20_Main.ipynb#L349-L357] [code: Aqua20_Main.ipynb#L88]. The reported
test scores are therefore chosen on the test data, and with no validation split
reported, the size of that effect is unknown. The notebook also feeds 224×224
images where the paper says 32×32 [code: Aqua20_Main.ipynb#L89]
[arxiv:2506.17455 §4.1], so the input size behind the table is unknown. Card:
[AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455).

### Forecasts scored by several rules against a market

Prophet Arena asks each language model for the probability that a prediction-
market event resolves Yes, and scores it three ways: a Brier score, a binned
calibration error with ten bins, and the average return of a unit bet against
the market price [arxiv:2510.17638 §3.1] [app. B.2]. The rankings change with
the metric [arxiv:2510.17638 §3.2.2]. The market is not an independent rival.
The forecasting prompt shows each model the live market prices, so the market
baseline is a reference point, and the models mostly track it
[arxiv:2510.17638 app. E] [§4.2.3]. Forecasts made within three hours of
resolution are dropped from evaluation [arxiv:2510.17638 §3.2.1]. The scoring
rules leave details open: the ECE formula in the main text is typeset
ambiguously, the two places that define the bet differ on ties, and the number
of bootstrap replicates is not stated [arxiv:2510.17638 eq. 2]
[§3.1.3] [app. B.6] [app. B.7]. None of the five highlighted models reaches an
average return of 1, the break-even point, while their Brier scores sit close
to the market's [arxiv:2510.17638 tab. 2]. Card:
[LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](/papers/2510.17638).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two training methods differ by a small margin and compute is limited | Randomise every source except hyperparameter search; decide on P(A>B) | Author-reported cost for 100 runs: 21 h against 1,070 h for the ideal estimator [arxiv:2103.03098 §3.3] |
| Judging whether a new dataset supports a ranking | Diversity score and a label check | Shows whether items repeat a template and whether equations match answers [arxiv:2106.15772 §3.1] [arxiv:2106.15772 §3.3] |
| Stress-testing a classifier, with annotators and attack code available | Filtered adversarial set | Validity comes from transfer and human agreement [arxiv:2111.02840 §3.3] |
| A few cheap checks of one reasoning claim, no training | Minimal-pair probes | Fast, but one model and no statistics [arxiv:2302.08399 §2.1.1] |
| Human preference between two generators, where Likert averages look unreliable | System-level probability with a test against 0.5 | Needs many annotators; 90 in the authors' GPT-3 study [arxiv:2205.11930 §6.1] |
| Comparing forecasting models across papers | Shared pipeline; report on original scale | Scale changes the numbers by several times [arxiv:2310.06119 §4.2] |
| Claim of general ability from one benchmark | Scoped benchmark name plus audits and error analysis | One score does not settle a general claim [arxiv:2111.15366 §5] |
| Breadth survey of a hosted model | Small, author-graded sample with explicit caveats | Reports behaviour, not a validated ranking [arxiv:2409.18486 §4.1] |
| Streaming detector where early alarms matter | Window-based, timing-aware score; fix the constants before comparing | The paper gives no weight values, and the code differs from its formula [arxiv:1510.03336 §II.C] [code: nab/sweeper.py#L165] |
| CIFAR-sized benchmark with equal splits | CINIC-10, with the overlap and unvetted ImageNet part reported | Equal split sizes and a drop-in design [arxiv:1810.03505 §1] [§2]; the test set shares images with the training source [§2] |
| Short ground-truth string in a model's free-text reply | Normalised substring match, reported with its rule | Extra words pass, and refusals count as wrong [arxiv:2305.07895 §3.1] [§3.3] |
| Target is a population's opinion, not one right answer | Survey-share scoring, with a neutral baseline | The gold label is the survey distribution [arxiv:2402.13605 §4] [tab. 1] |
| Headline score from a multi-step agent | Repeat the headline cell, report the standard deviation, and call small gaps noise | Only the default Claude cells have three runs, with 1.2 to 3.5 points of spread [arxiv:2607.26148 §4.1] [tab. 2] |
| Claim that a retrieval model composes concepts | Controlled negatives and seen/unseen splits, with a human check on negatives | Recall falls on unseen compounds, but negative quality varies by type [arxiv:2212.07796 §5.1] [§4.3] [§6] |
| Free-text answers graded by a model, no labels yet | Score a human sample; add marking criteria; report judge-human agreement | Criteria raised the judge's agreement from 0.820 to 0.854 [arxiv:2305.12474 tab. 6] |
| Agent tasks where a rule check and a judge both apply | Run both; report their disagreement; read unsafe rates as bounds if the judge marks many failures | Disagreement is 5.69% to 14.95% of tasks [arxiv:2507.06134 tab. 3] [§4.3] |
| Claim that an agent is reliable, not just capable | Report pass^k next to pass@k, with task-level bootstrap intervals | Leaders lose a third or more of their pass@1 at pass^3 [arxiv:2605.16679 §4.2] [tab. 2] |
| Score across many input variants of one request | Gate on comprehension; report consistency with the mean | Consistency can be high while safety is low [arxiv:2508.07173 tab. 8] |
| Measuring a bias or sensitivity effect | Add non-bias control edits and expect no shift | Controls can move, so a bias shift needs the control to be flat [arxiv:2311.04076 §3.1] |
| Claim that a model learned the relation, on data whose hypotheses may carry cues | Report a cue-only baseline and score the hard subset it misses | Strong models score much lower on that subset [arxiv:1803.02324 §4] [tab. 5] |
| Video model must read time, not one frame | Rewrite static questions as temporal ones; keep multiple choice with a fixed answer prefix | Items need the whole clip, and no judge model is used [arxiv:2311.17005 §3.1] [§3.3] |
| Comparing text generators on quality and diversity | Synthetic benchmark with a known generator, plus several metrics | One metric can miss a narrowing output [arxiv:1802.01886 §1] [§4.1] |
| Quoting a human score or an oracle bound for a reasoning benchmark | Treat the human figure as the authors' own baseline; read oracle bounds as bounds on explanation quality | The 88.40% human figure is from an author-run study [arxiv:2209.09513 app. B.2] [§5.4] |
| Ranking methods on a long-used archive | Add a held-out or refreshed set; check pairwise matrices | The lead on the old archive is small on new data [arxiv:2304.13029 §5]; some scores drop on a refreshed set [arxiv:2305.12474 tab. 5] |
| Deciding what a safety evaluation can claim beyond the model | Layered evaluation with the validity practices; set the threshold through an accountable process | No threshold is given in the framework [arxiv:2310.11986 §4.1.1] [§5.3.2] |
| Ranking detectors that each had one training run | Repeat runs or report intervals before ranking; treat small gaps as ties | Neighbouring models differ by 0.001 mAP50-95, with no repeats reported [arxiv:2411.00201 fig. 12] |
| Single-camera dynamic-scene rendering, scored on multi-camera data | Remove teleporting frames from training; score only pixels seen in training | Existing benchmarks lose 1–2 dB masked PSNR without them [arxiv:2210.13445 §1] [§5.1] |
| Forecasting several model families on one dataset | One split, scaler fit on training data only, one set of error measures | Released graph code fits the scaler on test data [code: workMETRLA/pred_DCRNN.py#L187-L189] |
| Using an LLM to simulate human participants | Treat as a hypothesis; validate against human data at each level | The validation loop is proposed, not evaluated [arxiv:2501.08579 §5] |
| Ranking LLMs on a static benchmark that may appear in training data | Add a dynamic or refreshed test set and state the contamination risk | The survey names dynamic evaluation as an open challenge, with no procedure [arxiv:2307.03109 §7.4] |
| Reporting a closed model on an input type its benchmarks did not cover | Check whether the test set is public and sized; if inputs were converted, report the converter's error | Audio scores come from TTS conversion and some sets are internal [arxiv:2410.21276 §3.2] [§3.3.2] |
| Releasing or reusing a dataset rebuilt from raw captures | Publish the matching rule and its tolerance; check the prose against the tables; run a model before any detection claim | The matching tolerance is not stated, and the text and table disagree on attack days [arxiv:2503.04404 §4.2] [§5.3] [tab. 4] |
| Probing a vision encoder for fine visual details, with a reference encoder available for mining | Mined pairs with two questions each; score a pair only when both are right; report 25% as chance | Guessing scores 25% on this design [arxiv:2401.06209 §2.3]; the pairs are hand-made [§2.2] |
| New architecture claimed to beat a strong baseline, e.g. 3D segmentation | Re-configure the baseline with its own auto-configuration, match data and compute, and test on datasets that separate methods | Most recent Transformer and Mamba claims do not hold against the original nnU-Net [arxiv:2404.09556 §4] [tab. 1] |
| Ranking link predictors with sampled negatives, across dataset versions | Keep one protocol and candidate set for every method; never compare scores from different versions | The same model scores 0.8637 MRR in v1 [arxiv:2103.09430 tab. 5] and 0.1833 after the v2 change [arxiv:2103.09430 §3] |
| Human judgement of translation quality that must show which errors matter | Professional error-span annotation with three raters per segment; rebuild the scoring rule from the paper | System order changes under error annotation, and the rater band in the text disagrees with table 7 [arxiv:2104.14478 §4.1] [tab. 7] |
| Collecting hard test cases from a chatbot, not measuring its failure rate | A curated failure list, labelled as anecdotal, with the version and prompt recorded | No counts or sampling rule, and some failures may be fixed in later versions [arxiv:2302.03494 §2] [§2.7] |
| Spiking or event-based dataset built by a fixed converter from audio or sensor data | Report the spike-count baseline next to the timing-aware model; fix and publish the converter's parameters | Spike-count SVMs reach 60.0% on SHD against 89.0% for an LSTM [arxiv:1910.07407 tab. 1]; the released code departs from the paper's values [code: lauscher/transformations/hair_cell.py#L16-L24] |
| Combining several automatic metrics into one ranking that people will read | Fit weights to human ratings; report each metric's correlation and the fit's margin over a plain average | The fit's margin over the average is small on several aspects [arxiv:2310.11440 tab. 3]; the weights exist only in the code [code: eval_from_metrics.py#L53-L68] |
| Scoring a physical property that cannot be measured directly, with a non-contact proxy | Fix and report the proxy's sensitivity constant; compare only on one hardware setup; check the proxy against a direct measurement before ranking | The constant is not stated [arxiv:2411.19408 §3.1], and with every grasp successful the score tracks deformation alone [arxiv:2411.19408 §5] [eq. 1] |
| Comparing platforms or models across a family of sizes | Sweep a parameterised family next to a few real models; state batch size and software stack with each speedup | Real-model speedups mix batch size with hardware, and one software range moved RNN speedups by 9.7x [arxiv:1907.10701 §5.3] [§6.1] |
| Reporting fairness gaps on a tabular dataset | Report group true-positive and true-negative rates beside each gap; flag zero gaps that come from a constant output | For three datasets the zero gaps reflect a constant classifier [arxiv:2110.00530 tab. 15] [§4.2] |
| Selecting a checkpoint and reporting its test score | Choose the checkpoint on a validation split the reported score never sees | The notebook chose by test loss, so reported test scores are not from data the selection never saw [arxiv:2506.17455 §4.1] [code: Aqua20_Main.ipynb#L349-L357] |
| Judge checks on a multimodal generation benchmark | Report agreement per task and say which tasks were checked | Human agreement covers 41 of 55 tasks [arxiv:2505.17613 §1] |
| Scoring LLM forecasts against a prediction market | Report Brier, calibration and market return together; state that the prompt contains market prices | Rankings change with the metric, and models see the market price [arxiv:2510.17638 §3.2.2] [app. E] |
| Comparing a judge's scores with a human check on another scale | Convert both to one scale before comparing | The judge table is on a 1 to 5 scale and the human check on 0 to 4 [arxiv:2510.17638 app. C.6] |
| Dataset whose counts differ across its sections | Use the count the task needs, cite its section and note the others | Trailer counts of 60K and 68K appear in different sections [arxiv:2007.10937 §3.1] [tab. 3] |

## Implementation sketch

The core of the variance-aware comparison is below. Hyperparameter search runs
once; each run re-draws every other source, and the decision uses paired
results. Concrete values follow the paper's stated settings [arxiv:2103.03098
§3.2] [arxiv:2103.03098 §4.1]. The number of bootstrap resamples is not stated,
so choose it and report it [arxiv:2103.03098 app. A].

```pseudocode
# Biased estimator, all sources except HOpt  [arxiv:2103.03098 §3.2.2]
# Input:  dataset, number of runs k
# Output: mean and standard deviation of the k scores

def fix_hopt_est_all(dataset, runs):
    # Tune hyperparameters once and reuse them  [arxiv:2103.03098 §3.2.2]
    hparams = hopt(dataset, draw_seeds())
    scores = []
    for run in range(runs):
        # Fresh seeds and a new split per run  [arxiv:2103.03098 §3.2]
        seeds = draw_seeds()
        train_val, test = out_of_bootstrap_split(dataset, seeds)
        scores.append(evaluate(train(train_val, hparams, seeds), test))
    return mean(scores), std(scores)

# Decision rule over paired runs  [arxiv:2103.03098 §4.1]
# Input:  scores_a, scores_b (paired by split and seed)
# Output: "A is better" or "no conclusion"

def decide(scores_a, scores_b, gamma=0.75):
    # Fraction of paired runs where A wins  [arxiv:2103.03098 §4.1]
    p_ab = mean(a > b for a, b in zip(scores_a, scores_b))
    # Bootstrap interval over resampled pairs  [arxiv:2103.03098 app. A]
    ci_min, ci_max = percentile_bootstrap(scores_a, scores_b)
    # Significant: lower bound must clear 0.5  [arxiv:2103.03098 app. A]
    if ci_min <= 0.5:
        return "no conclusion"
    # Meaningful: upper bound must clear gamma  [arxiv:2103.03098 app. A]
    if ci_max <= gamma:
        return "no conclusion"
    return "A is better"
```

## Contradictions

- **Linear models versus Transformers for long-term forecasting.** One side,
  reported in earlier work, is that linear models beat Transformers
  [arxiv:2310.06119 §1]. The other side, from later work, is the opposite
  [arxiv:2310.06119 §1]. Shao et al. attribute the conflict to dataset
  heterogeneity rather than to the models [arxiv:2310.06119 §5]. This page does
  not resolve it.
- **Decision thresholds and tests.** The variance study compares paired model
  runs with a bootstrap interval and a threshold of 0.75
  [arxiv:2103.03098 §4.1]. The human-preference study tests a stated probability
  against 0.5 with a t-test [arxiv:2205.11930 §5.2]. The bake off uses one-sided
  Wilcoxon tests with Holm correction to form cliques [arxiv:2304.13029 §3].
  These are different designs and should not be mixed.
- **Judge agreement is task-specific.** The exam judge reaches question-level
  Spearman of 0.825 to 0.864 [arxiv:2305.12474 tab. 4]. The agent-safety judge
  labels 35.9% of trajectories unsafe where humans label 71.7%
  [arxiv:2507.06134 tab. 4]. The tasks, judges and labels differ, so neither
  result transfers to the other task. Measure agreement on each benchmark.
- Disagreements between a paper's text and its own tables are listed on the
  paper cards, not here.

## Papers

- [Accounting for Variance in Machine Learning Benchmarks](/papers/2103.03098):
  randomises all sources except hyperparameter search and decides on P(A>B).
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](/papers/2106.15772):
  ASDiv, with a lexical diversity metric and a label check.
- [A Survey on Evaluation of Large Language Models](/papers/2307.03109): survey
  of LLM evaluation with no experiments; argues for dynamic benchmarks against contamination.
- [A Categorical Archive of ChatGPT Failures](/papers/2302.03494): qualitative
  archive of failures in eleven categories, with no counts or sampling rule.
- [Adversarial GLUE](/papers/2111.02840): filtered adversarial test set for five GLUE tasks.
- [AI and the Everything in the Whole Wide World Benchmark](/papers/2111.15366):
  position paper on the construct validity of general benchmarks.
- [Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324):
  hypothesis-only baseline that reveals labels, and a hard subset that strong models miss.
- [The Authenticity Gap in Human Evaluation](/papers/2205.11930): system-level
  probability as an alternative to Likert averages.
- [Bake off redux](/papers/2304.13029): time-series classification benchmark
  with 30 new held-out datasets and a pairwise-matrix check on rankings.
- [CHI-Bench](/papers/2605.16679): healthcare agent benchmark reporting pass^k,
  task-level bootstrap intervals and judge-family caveats.
- [CINIC-10 is not ImageNet or CIFAR-10](/papers/1810.03505): 270,000-image
  CIFAR-10 drop-in with equal splits and an ImageNet part that is not vetted.
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](/papers/2212.07796):
  retrieval benchmark with seen/unseen compound splits and human-checked hard negatives.
- [Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation](/papers/2607.26148):
  zero-shot navigation by a coding agent; success is scored by the driver, with few replicated cells.
- [Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark](/papers/1510.03336):
  streaming score that rewards early detections; author-reported profile scores.
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](/papers/2305.12474):
  exam benchmark with a GPT-4-turbo judge checked against teachers and a 2023 follow-up set.
- [GPT-4o System Card](/papers/2410.21276): closed-model safety and capability report with
  TTS-converted audio evaluations and internal test sets.
- [Do LLMs exhibit human-like response biases? A case study in survey design](/papers/2311.04076):
  response-bias probes with non-bias control edits.
- [DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction](/papers/2108.09091):
  grid and graph traffic models under one protocol, with a leak found in released graph code.
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](/papers/2402.13605):
  survey-share gold labels for social values, and an exam-style accuracy set.
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](/papers/2302.08399):
  minimal-pair probes of one model.
- [Med-HALT](/papers/2307.15343): medical hallucination benchmark with a penalised score.
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](/papers/2209.09513):
  ScienceQA benchmark with author-run human and oracle reference points.
- [Monocular Dynamic View Synthesis: A Reality Check](/papers/2210.13445):
  removes multi-view leakage from training with teleporting frames and reports effective multi-view factors.
- [Exploring Progress in Multivariate Time Series Forecasting](/papers/2310.06119):
  shared pipeline (BasicTS+) and dataset heterogeneity analysis.
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](/papers/2311.17005):
  video benchmark built by rewriting static image questions into temporal ones.
- [Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems](/papers/2503.04404):
  rebuilt NIDS benchmarks with timing features; descriptive only, with text-table mismatches and no model.
- [Texygen: A Benchmarking Platform for Text Generation Models](/papers/1802.01886):
  text-generation benchmark with a synthetic oracle, several metrics and a released toolkit.
- [Evaluation of OpenAI o1](/papers/2409.18486): small, author-graded evaluation of a hosted model.
- [Evaluation and Benchmarking of LLM Agents: A Survey](/papers/2507.21504):
  survey of agent evaluation objectives and processes, with no experiments.
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](/papers/2104.14478):
  professional MQM error annotation of WMT20 outputs, compared with the WMT ranking and metrics.
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209):
  CLIP-blind image pairs mined against DINOv2, scored as pairs with a 25% chance level.
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](/papers/2305.07895):
  1000-item OCR test with a substring scorer and author-reported model scores.
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](/papers/2103.09430):
  three large graph tasks with sampled-negative ranking, a v2 protocol change and competition winners.
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556):
  six-dataset 3D segmentation benchmark that re-tests recent architecture claims against a tuned baseline.
- [OpenAgentSafety](/papers/2507.06134): agent safety benchmark with a rule check and an LLM judge, reported side by side.
- [Omni-SafetyBench](/papers/2508.07173): audio-visual safety benchmark with a comprehension gate and a consistency score.
- [Sociotechnical Safety Evaluation of Generative AI Systems](/papers/2310.11986):
  three-layer framework and validity practices for safety evaluation.
- [SoGraB: A Visual Method for Soft Grasping Benchmarking and Evaluation](/papers/2411.19408):
  deformation-based grasp score from a non-contact proxy; the constant is unstated and the results compare grippers only.
- [LLM-based Human Simulations Have Not Yet Been Reliable](/papers/2501.08579):
  position paper on the reliability of simulated human participants.
- [YOLO Evolution: A Comprehensive Benchmark and Architectural Review of YOLOv12, YOLO11, and Their Previous Versions](/papers/2411.00201):
  one-run-per-model detector benchmark with noted inconsistencies between text and tables.
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](/papers/1910.07407):
  two spike-encoded speech sets from a fixed cochlea model, with spike-count and spike-timing baselines.
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](/papers/2310.11440):
  700-prompt text-to-video benchmark with 17 metrics and a score fitted to human ratings.
- [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](/papers/1907.10701):
  parameterised training sweep on TPU, GPU and CPU; speedups depend on batch size and software version.
- [MovieNet: A Holistic Dataset for Movie Understanding](/papers/2007.10937):
  movie-level multi-task dataset whose counts differ across sections.
- [A survey on datasets for fairness-aware machine learning](/papers/2110.00530):
  15 tabular fairness datasets; constant classifiers produce zero fairness gaps.
- [MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613):
  generation tasks scored by narrow checks, with judge validation on a subset of tasks.
- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455):
  underwater classification set; checkpoints chosen on test loss, one seed per model.
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](/papers/2510.17638):
  live forecasting benchmark scored by Brier, calibration and market return.