Benchmark validity

Overview

A benchmark score comes from one dataset, one scoring rule and, usually, one training run. The cited papers show that each of these parts can move the number by more than the gap that a leaderboard is meant to reveal. Four problems recur:

• Noise. Data splits, random seeds and hyperparameter search each change the score 2103.03098 §2.2. Repeated agent trials can change whether a task counts as solved at all 2605.16679 §4.2.
• Coverage. The items may not test the claim. They may be near-duplicates 2106.15772 §3.1, read as general when they are narrow 2111.15366 §4.1, be easy to solve from surface patterns 2111.02840 §1, may be solved from one sentence of a pair 1803.02324 tab. 2, may already be in training data 2305.12474 §5, or may carry signal that the task does not allow, such as frames that leak other camera views 2210.13445 §1. A handful of popular models can also hide a weakness that others expose 1907.10701 §1.
• Protocol. Papers scale, split and score data differently, so their numbers do not compare 2310.06119 §4.1. A baseline can also be under-configured relative to the new method, which makes a gain look larger than it is 2404.09556 §1. Even one benchmark can change its protocol between versions 2103.09430 §3. The test split can also steer which checkpoint is reported 2506.17455 §4.1.
• Scoring. The score itself can mislead. Judges can disagree with humans 2507.06134 tab. 4, a refusal can mean nothing if the input was not understood 2508.07173 §2.3.1, and a consistency score can reward uniform failure 2508.07173 tab. 8. A single metric can also hide a collapse: a text generator can score well on quality while its output narrows 1802.01886 §1. A constant output can also earn a zero fairness gap 2110.00530 tab. 15.

The shared rule is that a score is valid only for the items, scale and protocol it was measured on. Each variant below targets one of these problems.

Variants

Treat the score as a distribution

Bouthillier et al. randomise data split, initialisation, data order, augmentation and dropout in every run, but run hyperparameter search only once. They call this FixHOptEst(k, All) 2103.03098 §3.2. The decision rule uses the fraction of paired runs in which A beats B, with a bootstrap interval 2103.03098 §4.1. In their simulations, the single-point comparison gives about 10% false positives and about 75% false negatives. The recommended rule gives about 5% and about 30% 2103.03098 §4.2. These are author-reported simulations on five case studies. Card: Accounting for Variance in Machine Learning Benchmarks. The same problem shows in single-run leaderboards. A detection benchmark trains one run per model and reports no repeats or intervals. On traffic signs, YOLO11m (0.795) and YOLO11l (0.794) differ by 0.001 in mAP50-95, so that gap cannot be separated from run-to-run noise 2411.00201 fig. 12. Card: YOLO Evolution. The same gap appears in a fine-tuning benchmark that runs each model once. On AQUA20, ConvNeXt's top-3 accuracy is 0.9882 and SwinV2-B's is 0.9876 2506.17455 tab. 3. The notebook fixes one seed and one run per model, and the paper reports no repeats or standard deviations, so a gap of 0.06 points cannot be judged Aqua20_Main.ipynb 2506.17455 tab. 3. Card: AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions.

Check the dataset's diversity and labels

Miao et al. score each problem by how lexically similar it is to its closest other problem, using BLEU in both directions. They average this into a corpus score 2106.15772 §3.1. The new corpus, ASDiv, has a corpus score of 0.49, against 0.08 for the MathQA-C subset 2106.15772 tab. 3. Three pre-LLM solvers reach about 36% accuracy on ASDiv, while one solver reaches 86% on MathQA-C 2106.15772 tab. 3. The two corpora are not the same, so this gap cannot be read as a like-for-like difference. Labels were checked by solving each equation with SymPy and comparing it with the stated answer 2106.15772 §3.3. Card: A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers.

Equal splits from two image sources

Darlow et al. build CINIC-10 by adding ImageNet images, downsampled to 32×32, to the ten CIFAR-10 classes 1810.03505 §2. The result has 270,000 images in three equal splits of 9,000 per class 1810.03505 §2. The equal split is the point, since CIFAR-10 has no validation split 1810.03505 §1. Two validity risks follow from the design. The test set includes random CIFAR-10 training images, so a model trained on CIFAR-10 has seen some test items; the paper does not discuss this 1810.03505 §2. The ImageNet part was not vetted by hand, and the authors say it is noisier than the CIFAR-10 part README.md. The two sources are not identically distributed by design 1810.03505 §1 §4.1. The six reference errors, from 8.55 to 18.00, come from CIFAR-style models, and the paper does not state the training recipe behind them 1810.03505 tab. 1 §5. Card: CINIC-10 is not ImageNet or CIFAR-10.

Remove leaked signal from the training input

Gao et al. show that the common monocular dynamic view-synthesis benchmarks train on frames that "teleport" between cameras. The model then sees multi-view information that a single camera would not record 2210.13445 §1. They quantify this with two effective multi-view factors. One, Ω, needs depth and optical flow, and segmentation. The other, ω, needs only the camera set-up and frame rate, but it assumes a fixed look-at point 2210.13445 §3 app. A. Training on the left camera alone, with no teleporting, lowers masked PSNR by 1–2 dB on the existing datasets 2210.13445 §1 §5.1. The new iPhone dataset scores lower again for all methods 2210.13445 §5.2. The frame rates of the existing datasets were chosen by hand, so ω carries that uncertainty 2210.13445 app. A. Card: Monocular Dynamic View Synthesis: A Reality Check.

Check a release's labels and prose against its tables

Luay et al. rebuild four network intrusion benchmarks as NetFlow records with timing features. They describe the result with histograms, per-minute series and spectrograms, and train no model 2503.04404 §5 §6. Labelling is underspecified: the paper gives no time tolerance for matching flows to the ground-truth files 2503.04404 §4.2. The text and Table 4 disagree on which attacks run on some days. For BoT-IoT day 1, the text names DDoS and DoS, while the table lists only reconnaissance 2503.04404 §5.3 tab. 4. The "unique patterns" claim rests on visual comparison of spectrograms, with no detection experiment 2503.04404 §5.5 §6. Card: Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems.

Dataset papers can also disagree with themselves. MovieNet reports 60K trailers for 33K movies in its text and 68K trailers in its genre table 2007.10937 §3.1 tab. 3. Scene boundaries are 42K in the body and 40K in the introduction 2007.10937 §3.2 §1. Its benchmark splits are by movie, but the action and place splits are random by clip, with a claim of no movie overlap that a new split would need to verify 2007.10937 app. C.6 app. C.7. The authors conclude that character detection is largely solved, which rests on one split and one metric 2007.10937 §4.3. Card: MovieNet: A Holistic Dataset for Movie Understanding.

Filtered adversarial test sets

Wang et al. generate adversarial examples against surrogate models for five GLUE tasks. They keep those that transfer to other surrogates and change few words 2111.02840 §3.3. Crowd workers then keep only examples where four of five annotators agree with the original label 2111.02840 §3.3. Across ten tested models, the average AdvGLUE score is 30.3 to 60.9, against 85.7 to 93.2 on GLUE 2111.02840 tab. 4. The authors present it as a diagnostic set beside GLUE, not as a replacement 2111.02840 app. A.8. Card: Adversarial GLUE.

Mined pairs scored as a pair

Tong et al. mine image pairs that CLIP ViT-L/14 treats as near-identical and DINOv2 ViT-L/14 treats as different. The thresholds are a CLIP cosine above 0.95 and a DINOv2 cosine below 0.6 2401.06209 §2.1. The paper does not say how candidate pairs are enumerated 2401.06209 §2.1. The authors write two questions for each of 150 pairs, and a pair counts only when both answers are right 2401.06209 §2.2. Guessing then scores 25% 2401.06209 §2.3. Most tested models fall below that level, with GPT-4V and Gemini as the exceptions 2401.06209 §2.3. The human reference is 95.7% from four participants 2401.06209 app. B.4. The pairs and questions are hand-written, and the paper reports no inter-annotator agreement 2401.06209 §2.2 app. B.4. The pattern labels come from GPT-4 prompting, and how each question was assigned to a pattern is not stated 2401.06209 §3.1. Closed-model runs date from late 2023 2401.06209 app. B.1. Card: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs.

Hand-built probes of one claim

Ullman keeps the logic of two published theory-of-mind vignettes and changes details, such as making the bag transparent. He then compares the probability of each completion from GPT-3.5 2302.08399 §2. Under the transparent bag, the model gives 95% to "chocolate" for the belief prompt [arxiv:2302.08399 §2.1.1]. A probe needs no training. It reports one model, one prompt per variant, no repeats and no statistics 2302.08399 §2.1.1. Card: Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks.

Hard subsets defined by a cue-only baseline

Gururangan et al. train a fastText classifier that sees only the hypothesis of each pair. On SNLI it reaches 67.0%, and on MultiNLI matched and mismatched it reaches 53.9% and 52.3%, against 34–35% for the majority class 1803.02324 tab. 2. The crowd-written hypotheses carry cues tied to labels, such as negation for contradiction 1803.02324 §3.1 §3.2 tab. 4. Strong NLI models score much lower on the examples this classifier gets wrong 1803.02324 §4 tab. 5. The authors use that hard subset as a check, not as a replacement for the full test set. Card: Annotation Artifacts in Natural Language Inference Data.

Rewrite static items so one frame cannot solve them

Li et al. start from nine spatial task families in image benchmarks and rewrite each into a temporal task that a single frame cannot solve [arxiv:2311.17005 §3.1]. The result is 20 tasks, each scored on 200 multiple-choice questions built from 11 public video datasets 2311.17005 §3.1 §3.2. Options are read from the reply to a fixed answer prefix, so no judge model is needed 2311.17005 §3.3 tab. 9. The results depend on the language model, the frame count and the resolution 2311.17005 tab. 2. The authors report 51.1% for VideoChat2 against 35.5% for VideoChat [arxiv:2311.17005 tab. 2]. Card: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark.

Synthetic benchmark with a known generator

Zhu et al. use a random LSTM as an oracle and draw 10,000 sentences of length 20 from it as the "real" data, so the true distribution is known 1802.01886 §2.1 §4.1. The metrics are split by data source: the oracle-based likelihoods are computed on synthetic data, and BLEU, Self-BLEU and EmbSim on real data, because the oracle cannot produce meaningful words 1802.01886 §4.2 §4.3. The paper does not give the oracle's initialisation, so an oracle-based number is only comparable with a rebuilt oracle of the same kind 1802.01886 §2.1. Card: Texygen: A Benchmarking Platform for Text Generation Models.

Reference points: human baselines and oracle bounds

Lu et al. report 88.40% human accuracy on their science questions, from a crowd study with an exam qualification and three workers per set [arxiv:2209.09513 tab. 3] app. B.2. That figure is the authors' own study, so it is not an independent ceiling. Their oracle runs put gold explanations in the prompt and reach about 94%. They show that explanation quality matters, not that a model can reach that score 2209.09513 §5.4 tab. 6. Card: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.

Controlled negatives and seen versus unseen items

Ma et al. build a retrieval benchmark for compositional vision-language models. Each hard negative differs from the true caption by one part: an atom is replaced, two single-compound foils are joined, words are permuted, or a negation is added 2212.07796 §3.3 §4.1 §4.2. Test items are split by whether their atoms and compounds appeared in each training corpus 2212.07796 §3.2. Recall@1 falls from seen to unseen compounds for all seven systematicity models 2212.07796 §5.1. The largest drop is 12.57 points on one subset for one model, which is more than the combined-set drop 2212.07796 app. B.1. Negative quality needs checking. Human annotators judged 83.7% of productivity negatives and 86.0% of systematicity negatives to be genuine 2212.07796 §4.3. The discussion puts HN-Swap and HN-Neg near 70% 2212.07796 §6. The seen and unseen labels depend on a scene-graph parser, whose triplet recall on LAION-400M is 39.55% 2212.07796 app. A.2. Card: CREPE: Can Vision-Language Foundation Models Reason Compositionally?.

Human preference as a system-level probability

Ethayarajh and Jurafsky argue that averaged Likert ratings can reverse the true preference between two systems 2205.11930 §3. Their alternative, system-level probabilistic assessment, asks each annotator for P[X ≻ Y] after they read the same sample from both systems. The mean is then tested against 0.5 2205.11930 §5.2. On GPT-3 story generation with 90 analysed annotators, the authors report 5 of 5 expected preferences recovered, against 2 of 5 for the standard protocol 2205.11930 §6.1. Card: The Authenticity Gap in Human Evaluation.

Shared pipeline and original-scale metrics

Shao et al. fix the data processing, training and evaluation in one pipeline. They report errors on re-normalised data 2310.06119 §4.1 2310.06119 §4.2. On ETT datasets, the normalised MAE is about 0.4 to 0.5 and the re-normalised MAE is 1.6 to 3.7 2310.06119 §4.2. The released code defaults to evaluating on normalised values, so the flag has to be set to match the paper 2310.06119 §4.2. Card: Exploring Progress in Multivariate Time Series Forecasting.

Jiang et al. run grid-based and graph-based traffic models under one split, one training setting and one set of error measures 2108.09091 §5.1. On four grid datasets, STDN has the lowest RMSE of the grid models 2108.09091 tab. 4. Their released graph code fits the scaler on the whole series, test period included, before the split [code: workMETRLA/pred_DCRNN.py#L187-L189]. A reproduction should fit the scaler on training data only, and the paper does not say which was used. The authors' grid code reports only rescaled MSE, so the grid MAE and MAPE in the paper cannot be checked against it predflowio_STDN_noflow.py. Card: DL-Traff.

Scoped benchmarks and test suites

Raji et al. argue that general benchmarks such as ImageNet, GLUE and SuperGLUE lack construct validity as measures of general ability 2111.15366 §1. They recommend naming a benchmark after the concrete task it tests. For broader claims, they suggest test suites, audits, error analysis, ablation and model-property measures 2111.15366 §5 2111.15366 app. A. This is a position paper with no experiments, and the alternatives come without procedures or thresholds 2111.15366 app. A. Card: AI and the Everything in the Whole Wide World Benchmark.

Layered, context-aware safety evaluation

Weidinger et al. argue that a safety test of a model alone does not show harm in use. They split evaluation into capability, human interaction and systemic impact 2310.11986 §2. Their validity practices are to ground each measure in literature, expert input or curated examples, to document the limits of each evaluation, to cross-validate measures of the same concept, and to keep results readable so that failures are not hidden in one aggregate score 2310.11986 §4.1.1. The framework has no scoring rule, and it gives no acceptance threshold: the authors say a normative threshold must come from a legitimate process 2310.11986 §2 §5.3.2. Their survey of evaluations is a snapshot and not a systematic search 2310.11986 §3.2. Card: Sociotechnical Safety Evaluation of Generative AI Systems.

Broad, small-sample evaluation of a closed model

Zhong et al. evaluate o1-preview on 27 tasks, mostly graded by the authors and often on a few dozen items 2409.18486 §4.15. Coding uses 12 contest problems, where each problem is 8.3 points of pass rate [arxiv:2409.18486 tab. 2]. Radiology uses 10 reports 2409.18486 §2.2. The model and settings are not fully specified, and contamination is not analysed 2409.18486 §4.1. Use it for breadth and examples, not for ranking. Card: Evaluation of OpenAI o1.

Penalised scoring for hallucination

Med-HALT scores a correct answer +1 and a wrong one −0.25, so guessing earns less than knowing 2307.15343 §5.3. The fake-question scorer credits any answer that contains one of a fixed set of negative phrases, so the accuracy is an upper bound 2307.15343 §5.3. The published score column is not the average in the paper's equation, so check the formula before using its numbers 2307.15343 §5.3. Card: Med-HALT.

Timing-aware score for streaming detectors

Lavin and Ahmad score a streaming anomaly detector by when it fires inside an anomaly window, not only whether it fires 1510.03336 §II.B. Only the earliest detection in a window counts, and its credit falls off across the window 1510.03336 §II.C eq. 1. Detections outside every window are penalised, and a window with no detection is a false negative 1510.03336 eq. 2. The raw sum is rescaled so that a perfect detector scores 100 and a null detector 0 1510.03336 eq. 4. On the standard profile the authors report 64.7 for Numenta HTM, 47.1 for Twitter ADVec and 35.7 for Etsy Skyline 1510.03336 tab. I. The paper gives ranges for the four outcome weights but no values, so the repository's profile weights are the only numeric source 1510.03336 §II.C [code: config/profiles.json#L4-L7]. The released code also differs from eq. 1 at the right edge of a window sweeper.py. Card: Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark.

Substring match for free-text replies

Liu et al. score 1000 OCR-style questions for large multimodal models with one rule: a reply counts if the ground-truth string appears in it 2305.07895 §3.1. The final score is the sum, out of 1000 2305.07895 §3.3 tab. 3. The authors report 659 for Gemini, 645 for GPT4V and 514 for Monkey 2305.07895 tab. 3. The rule needs no judge model, but a reply passes if the answer appears inside a longer or negated sentence [inferred from code: OCRBench/example.py#L141-L151]. The paper drops answers under four symbols to reduce false positives 2305.07895 §3.1. Refusals and format changes score as errors: GPT4V refused 84 images on safety grounds, and the scorer counts those as wrong 2305.07895 §3.3. The released scorer also lowercases the text, and removes spaces for handwritten maths, before matching example.py; the paper does not state either step. Card: OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models.

Survey share as the gold label

Lee et al. score KorNAT's social-value items against a survey rather than one right answer. Each question's ground truth is the share of 6,174 Korean respondents who picked each option, and a model that picks option k earns that share 2402.13605 §3.1.3 §4. Three scores follow from this: SVA on five options, A-SVA with agree and disagree merged, and N-SVA, which replaces the truth with a neutral label when neither merged share is above 0.5 2402.13605 §4. The paper gives the maxima 0.450, 0.626 and 0.625, and calls the SVA maximum empirically calculated rather than derived 2402.13605 §4 tab. 1. Common knowledge is plain accuracy against a 0.6 reference, taken from a Korean GED pass mark 2402.13605 §4. The paper names no reference for social values 2402.13605 §1. Survey weights for age, gender, education, residence and income are applied to the shares 2402.13605 §3.1.4. Refusals are omitted, and whether they count as wrong is not stated 2402.13605 §5.4. Card: KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge.

Non-contact proxy for an unmeasured quantity

Greenland et al. want a grasp score for soft grippers without force sensors. They use deformation as the stand-in: the object is scanned before and during the grasp, the two clouds are aligned with ICP, and the Density-Aware Chamfer Distance between them is taken 2411.19408 §1 §3.1 §3.2. That distance is combined with the grasp outcome and drop time into a score in [0, 1] 2411.19408 eq. 1. The sensitivity constant of the distance is not stated in the paper 2411.19408 §3.1, so scores from different implementations may not share a scale. All 900 grasps succeeded, so every score falls in the upper half of the range and depends on deformation alone 2411.19408 §5 eq. 1. The results compare grippers, not the score against a measured force 2411.19408 §5. Rankings are read from overlapping distributions with no significance test 2411.19408 §5.3. Other hardware can be substituted, but the paper runs only one camera pair, arm and printer 2411.19408 §4. Card: SoGraB: A Visual Method for Soft Grasping Benchmarking and Evaluation.

Held-out and refreshed test sets

Middlehurst et al. re-run a time-series classification bake off on the 112 UCR problems, which many methods were built against. They add 30 new datasets to check that the earlier choices were not tuned to the old collection 2304.13029 §5 §7. The top accuracy clique on the new datasets holds MR-Hydra, HC2, RDST, QUANT and FreshPRINCE, so HC2 is not alone at the top 2304.13029 §5 fig. 41. The gap between HC2 and MR-Hydra is small on the new datasets and on the combined 142 datasets 2304.13029 §5. Critical difference diagrams can disagree with the pairwise win counts, so the authors suggest checking the matrices before ranking 2304.13029 §5. Zhang et al. re-test an exam benchmark on a 2023 objective-only follow-up set. Scores drop for some models, by 6.7 points for ChatGLM-6b and 0.6 for GPT-4-0613, and the authors call the change within normal difficulty variation 2305.12474 tab. 5 §5. They also accept that the questions may already be in training data 2305.12474 §5. Cards: Bake off redux and Evaluating the Performance of Large Language Models on GAOKAO Benchmark. A 2023 survey of LLM evaluation argues that static test sets are not enough for LLMs, because their capabilities change over time and their training data often overlaps with public benchmarks 2307.03109 §2.2 §7.4. It lists dynamic and evolving evaluation as an open challenge but gives no procedure for building one 2307.03109 §7.4. The survey runs no experiments, so the claim is a position, not a measured effect. Card: A Survey on Evaluation of Large Language Models.

LLM judges checked against humans

Two papers use a model as the grader and report how it agrees with people. Zhang et al. grade subjective exam answers with a GPT-4-turbo judge that reads teacher marking criteria. Question-level Spearman agreement with human scores is 0.825 to 0.864 across five models 2305.12474 tab. 4. With criteria, GPT-4-0613 reaches 0.854, against 0.820 without them 2305.12474 tab. 6. Converted totals stay within about 2% for sciences and about 5% for humanities, and the humanities gap comes from matching answers to marking points 2305.12474 §4.4. Question-level agreement is not agreement on totals 2305.12474 tab. 4.

Vijayvargiya et al. run a rule check and a GPT-4.1 judge on the same agent trajectories 2507.06134 §3.3. The two disagree on 5.69% to 14.95% of tasks, depending on the model 2507.06134 tab. 3. On 100 GPT-4o trajectories, the judge labels 35.9% unsafe, where human annotators label 71.7%, and the authors treat their unsafe rates as lower bounds 2507.06134 tab. 4 §4.3. Cards: Bake off redux, Evaluating the Performance of Large Language Models on GAOKAO Benchmark and OpenAgentSafety.

MMMG shows the coverage side of the problem. Its human study covers 41 of its 55 tasks and 978 instructions, so the 94.4% overall agreement does not extend to the other tasks 2505.17613 §1. The per-modality agreement figures in the introduction average to about 93.4%, not 94.4% 2505.17613 §1. Its judges are kept narrow: a vision-language model answers one question per criterion, as yes/no or multiple choice, and other tasks use similarity models, audio models or programs 2505.17613 §3.2. The paper says each judge threshold was matched to human judgement, but it gives no values, and the released code uses fixed per-class thresholds 2505.17613 §3.2 eval_audio.py. Card: MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation.

Prophet Arena checks its reasoning judge the same way. The judge's mean absolute difference from 170 human-rated forecasts is 0.37 to 0.44 on a 0 to 4 scale, while the judge's own table uses a 1 to 5 scale, so the two sets of numbers need care when compared 2510.17638 app. C.6 app. C.6.1. Card: LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena.

Comprehension gates and consistency across variants

Pan et al. render each of 972 harmful seed prompts in 24 input variants that mix text, image, video and audio, giving 23,328 cases 2508.07173 §2.2 tab. 2. A response counts toward the attack-success and refusal rates only if a probe shows the model understood the input. Otherwise a model that ignores the input looks safe 2508.07173 §2.3.1. Their consistency score takes the standard deviation of per-variant Safety-scores and maps it through an exponential with α = 5 2508.07173 §2.3.2. Consistency alone does not show safety: Unified-IO2 has the highest consistency score and the lowest average Safety-score 2508.07173 tab. 8. Human accuracy on the comprehension flag is 94.2% on 500 responses 2508.07173 §3.3 tab. 12. The released code does not implement the Safety-score or the consistency score, so a re-implementation has to follow the paper's formulas 2508.07173 §2.3.2. Card: Omni-SafetyBench.

Reliability over repeated agent trials

Chen et al. run each of 75 healthcare workflow tasks three times and report pass@1, pass@3 and pass^3 2605.16679 §4.1 app. F.2. pass^3 counts a task only when all three trials pass 2605.16679 app. F.2. For the leading configurations, pass^3 falls well below pass@1: Opus 4.6 goes from 28.0 to 18.7, and GPT-5.5 from 20.9 to 9.3 2605.16679 §4.2 tab. 2. Task-level bootstrap intervals overlap heavily at the top, so the leader is not settled: 28.0 [19.6, 36.9] against 26.2 [18.2, 33.8] 2605.16679 tab. 2. The judge is in the same model family as several evaluated Claude configurations, which the authors say may bias results 2605.16679 app. F.3. Card: CHI-Bench. A 2025 survey of agent evaluation lists pass^k as a consistency metric and cites a benchmark that uses it 2507.21504 §3.3 §5.2. The survey runs no experiments, so it gives no values to compare. Card: Evaluation and Benchmarking of LLM Agents: A Survey.

Driver-scored success with few replicated cells

Zhou et al. run a coding agent zero-shot on R2R-CE navigation, with a monocular camera, four moves and no map 2607.26148 §3.1 §3.2. The driver scores success after the session ends: the agent must issue STOP itself within 3 m of the goal, and a run that exhausts its budget scores zero [arxiv:2607.26148 app. A]. Only the default-effort Claude SDK cells have three runs. Their sample standard deviations are 1.2 to 3.5 points, and the authors treat differences of a few points as descriptive 2607.26148 §4.1 tab. 2. Most ablation cells are single runs of 100 episodes 2607.26148 tab. 4. Harness comparisons are not controlled, because serving paths, defaults and versions differ 2607.26148 §4.3. Of 30 failures in one strong run, 23 end with a claimed success while the final distance is 3.1 to 34.1 m 2607.26148 §4.6 tab. 7. Card: Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation.

Position papers on simulated humans

A position paper argues that LLM-based human simulations are not yet reliable enough to trust 2501.08579 §1. Its table of cited studies shows the gaps. On emotion understanding, GPT-4 reaches 58% against 70% for humans 2501.08579 tab. 1. Its fixes, including validation at several levels in one loop, are proposals and are not evaluated 2501.08579 §5 alg. 1. The table mixes tasks and models, so it shows the size of each gap but is not a like-for-like comparison 2501.08579 tab. 1. Card: LLM-based Human Simulations Have Not Yet Been Reliable.

Control probes that should show no effect

Tjuatja et al. test whether language models show the response biases that people show in survey wording. Each bias edit is paired with non-bias edits such as typos, a random middle swap and a letter swap, which humans are expected to ignore 2311.04076 §2.1 §2.3. Every model shows a significant shift on some typo control 2311.04076 §3.1. A shift on a control therefore means sensitivity to noise, not evidence of the target bias. The paper does not state a multiple-comparison correction across its many tests 2311.04076 §2.3. The allow/forbid set has only 40 question pairs 2311.04076 §2.1. Opinion-matching does not show a model behaves like human respondents: representativeness does not track bias-consistency 2311.04076 §5. Card: Do LLMs exhibit human-like response biases? A case study in survey design.

Proxy inputs and undisclosed test sets

The GPT-4o system card scores speech by converting existing text benchmarks to audio with a TTS system and scoring the text of the reply [arxiv:2410.21276 §3.2]. The authors say TTS is lossy for equations, code and formatting-heavy text, so a low score can come from the converter rather than the model 2410.21276 §3.2. The card names the TTS system but not the voice settings per test, nor the transcription system used for scoring 2410.21276 §3.2. Several safety and capability results come from internal datasets whose sizes are not stated 2410.21276 §3.3.2 §3.3.4. The voice capability results use a fixed random subset, so they should not be compared with published scores 2410.21276 §3.3.3. The speaker-identification gains are 14 and 12 points in the text, but 15 and 13 points in the table 2410.21276 §3.3.2 [tab. 2]. Card: GPT-4o System Card.

Matched baselines for architecture claims

Isensee et al. benchmark CNN, Transformer and Mamba 3D segmentation models on six datasets, using five-fold cross-validation on nnU-Net's splits 2404.09556 §3.2 tab. 1. They name four pitfalls: extra training signal bundled with the new method, baselines that are not well configured, datasets that cannot tell methods apart, and inconsistent reporting 2404.09556 §2.1 §2.2. Their headline is that most recent Transformer and Mamba claims do not hold against the original nnU-Net, and that the CNN variants lead 2404.09556 §4. For the Mamba claim, the No-Mamba Base ablation is within 0.25 points of U-Mamba Bot on every dataset in table 1, so the paper attributes the earlier gain to the residual encoder 2404.09556 §4 tab. 1. Reaching the best ResEnc result needs a larger GPU budget: on KiTS fold 0, Auto3DSeg with eight GPUs reaches 87.77 DSC, against 87.91 for nnU-Net ResEnc M on one GPU 2404.09556 tab. 2. The suitability ratio also depends on the method set: for ACDC it falls from 357% to 102% when the framework runs are removed 2404.09556 tab. 3. Card: nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation.

Sampled negatives and version changes

Hu et al. release three large graph tasks: node classification on MAG240M, link prediction on WikiKG90M and graph regression on PCQM4M 2103.09430 §2. For WikiKG90M v1, each true tail is ranked among 1000 sampled negatives, and only the top ten are scored, so a true answer outside them scores zero 2103.09430 §2.2. A duplicate id in any top-ten list sets the whole score to zero wikikg90m.py. Under this protocol, ranking candidates by training frequency alone reaches 0.75 validation MRR, so the top v1 scores are near saturation 2103.09430 §2.2 §3. In v2 the candidate tails are no longer given, and the paper reports ComplEx-Concat at 0.1833 MRR. It does not say which split that figure is from 2103.09430 §3 app. C. The v1 test figure for the same model is 0.8637 2103.09430 tab. 5. For PCQM4M, the authors meant to make a scaffold split, but a pre-processing bug produced a split by PubChem ID, which they found gives similar results 2103.09430 §2.3. The two GIN-virtual validation figures in the paper also differ, 0.1396 and 0.1410 2103.09430 tab. 6 tab. 7. Winning KDD Cup entries used ensembles and, for MAG240M, target labels as input, so they are not single-model comparisons 2103.09430 §2 §2.1. Card: OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs.

Expert error annotation as the reference

Freitag et al. have professional translators mark every error span in WMT20 English-German and Chinese-English outputs, each with a category and a severity, and turn the marks into a weighted segment score [arxiv:2104.14478 §3.3] tab. 1. Minor errors weigh 1, Major errors 5 and Neutral errors 0. Each segment keeps at most five target errors, and a Non-translation label scores 25 for the segment 2104.14478 app. B tab. 1. The Major weight came from a resampling check over 1 to 10, which the paper summarises without releasing code 2104.14478 §3.3. Under this scoring, the system order changes from the WMT ranking, and every human translation ranks above every MT system 2104.14478 §4.1 tab. 3 tab. 4. The two language pairs also used different WMT protocols, a reference-based monolingual one for Chinese-English and a source-based one for English-German, so the comparison cannot separate protocol from rater quality 2104.14478 §3.1 §4.1. The text gives a ±20% band for English-German raters, but table 7 shows rater 3 at 1.23 times the average 2104.14478 §4.4 tab. 7. The rating counts needed to reach an expected Kendall tau of 0.9, 951 per system for English-German and 3720 for Chinese-English, are simulated, not measured 2104.14478 §4.5 tab. 8. Card: Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.

Failure archives as hard cases

Borji gathers ChatGPT failures, mostly from public posts and some from the author's own prompts, and sorts them into eleven categories 2302.03494 §1 §2. The archive has no counts, no sampling rule and no baseline, and the author does not say how often any failure occurs 2302.03494 §2. Several examples name no model version, and the author notes that some errors may already be fixed 2302.03494 §2.7. Some labels are the author's own reading, including one natural-language-inference item and the political-leaning item, which relies on a third-party analysis 2302.03494 §2.2 §2.10. Rewording can change the answer: the altered family-count question gives 8 where the original gives the correct 5 2302.03494 §4. The rephrased hotwiring request gets past a refusal, so such items belong in a safety test set, not a capability list 2302.03494 §2.10. Card: A Categorical Archive of ChatGPT Failures.

Spike datasets from one fixed encoder

Cramer et al. release two spoken-digit sets already in spike form: Spiking Heidelberg Digits, recorded by the authors, and Spiking Speech Commands, derived from Speech Commands v0.02 1910.07407 §2.1. One fixed cochlea-inspired model turns the audio into 700 spike channels, so users do not each choose an encoder 1910.07407 §2.2. The SHD test set holds out two speakers completely, which measures speaker generalisation 1910.07407 §2.1.1. Spike-count classifiers score lower than classifiers that see spike timing. Linear SVMs on counts reach 56.0% on SHD and the RBF SVM 60.0%, against 89.0% for an LSTM on 10 ms bins 1910.07407 tab. 1. The authors read this as evidence that spike timing carries the signal 1910.07407 §3. The text gives 85.7% for the same LSTM setting 1910.07407 §3.1. The released conversion code is LGPL-2.1 and differs from the paper in several parameter values and in its output format hair_cell.py spike_train.py. The authors say the datasets may not prove final and that their accuracies are lower bounds 1910.07407 §4. Card: The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks.

Metric weights fitted to human ratings

Liu et al. score 700 text-to-video prompts with 17 automatic metrics in four aspects. They fit one linear model per aspect to 1–5 human ratings, then sum the four aspect scores 2310.11440 §4.5. On the authors' correlation table, the fitted score is close to a plain metric average: 55.4 against 55.0 for visual quality and 32.3 against 31.9 for text-video alignment, in Spearman terms 2310.11440 tab. 3. The fit's advantage is small, and on text-video alignment its Kendall value is lower than the average's (22.5 against 22.7) 2310.11440 tab. 3. The text calls the advantage consistent 2310.11440 app. B.2. Single metrics vary widely: CLIP-Score has a Spearman of 6.3 against 26.7 for BLIP-BLEU 2310.11440 tab. 3. The authors' "Avg." rows are not defined, and a recomputed average does not match them 2310.11440 tab. 3. Released code computes some metrics differently from the paper: Action-Score is a confidence-weighted label similarity, not a classification accuracy action_score.py, and frames without OCR output count as error 0 ocr_score.py. The fitted weights are printed only in the code, and the ratings and fitting script are not in the repository eval_from_metrics.py. Card: EvalCrafter: Benchmarking and Evaluating Large Video Generation Models.

Parameterised sweep plus a named software stack

Real-model benchmarks can miss problems that a sweep finds. Transformer alone would hide a TPU memory bottleneck that shows in fully connected layers wider than 4k nodes 1907.10701 §1. A generator of FC, CNN and RNN training models, swept over depth, width, batch and input size, covers that space 1907.10701 §2.1 tab. 2. Each platform wins somewhere, so the winner depends on the model family 1907.10701 §5. Two confounders sit inside the speedups. Real models run larger batches on the TPU, so part of a real-model speedup is batch size 1907.10701 §5.3 §2.2. Software also moves the number: the 90th-percentile RNN speedup on TPU was 9.7x across TensorFlow 1.7 to 1.12 1907.10701 §6.1. Synthetic data is made on the host, so input-pipeline cost is left out and the TPU figures are device numbers 1907.10701 §4.4. The paper reports no repeat count [arxiv:1907.10701 §4.4]. Card: Benchmarking TPU, GPU, and CPU Platforms for Deep Learning.

Fairness gaps checked against a constant classifier

A fairness gap can be zero for a classifier that ignores one class. The survey of 15 tabular datasets fits logistic regression on each one. On Credit card clients and Diabetes the classifier predicts only the negative class, and on OULAD only the positive class. Their statistical parity and equalized odds are 0.0 and their balanced accuracy is 0.5 2110.00530 tab. 15. The paper calls 0.0 the best value 2110.00530 §4.2, but for these rows it reflects a constant output. The analysis uses one 70/30 split with seed 42, and differences near 0.01 cannot be separated from split noise [arxiv:2110.00530 §4.1]. Ricci's perfect scores rest on a test split of about 36 rows 2110.00530 tab. 1. Card: A survey on datasets for fairness-aware machine learning.

Checkpoint selection kept off the reported split

Fuad et al. fine-tune 13 ImageNet-pretrained models and report results on their test split 2506.17455 §4.1 tab. 3. The paper says it keeps the checkpoint with the best evaluation performance and names no validation split 2506.17455 §4.1. The released notebook saves the checkpoint with the lowest test loss and stops after 10 epochs without a new best Aqua20_Main.ipynb Aqua20_Main.ipynb. The reported test scores are therefore chosen on the test data, and with no validation split reported, the size of that effect is unknown. The notebook also feeds 224×224 images where the paper says 32×32 Aqua20_Main.ipynb 2506.17455 §4.1, so the input size behind the table is unknown. Card: AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions.

Forecasts scored by several rules against a market

Prophet Arena asks each language model for the probability that a prediction- market event resolves Yes, and scores it three ways: a Brier score, a binned calibration error with ten bins, and the average return of a unit bet against the market price 2510.17638 §3.1 app. B.2. The rankings change with the metric 2510.17638 §3.2.2. The market is not an independent rival. The forecasting prompt shows each model the live market prices, so the market baseline is a reference point, and the models mostly track it 2510.17638 app. E §4.2.3. Forecasts made within three hours of resolution are dropped from evaluation 2510.17638 §3.2.1. The scoring rules leave details open: the ECE formula in the main text is typeset ambiguously, the two places that define the bet differ on ties, and the number of bootstrap replicates is not stated 2510.17638 eq. 2 §3.1.3 app. B.6 app. B.7. None of the five highlighted models reaches an average return of 1, the break-even point, while their Brier scores sit close to the market's 2510.17638 tab. 2. Card: LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena.

When to use which

Implementation sketch

The core of the variance-aware comparison is below. Hyperparameter search runs once; each run re-draws every other source, and the decision uses paired results. Concrete values follow the paper's stated settings [arxiv:2103.03098 §3.2] 2103.03098 §4.1. The number of bootstrap resamples is not stated, so choose it and report it 2103.03098 app. A.

# Biased estimator, all sources except HOpt  [arxiv:2103.03098 §3.2.2]
# Input:  dataset, number of runs k
# Output: mean and standard deviation of the k scores

def fix_hopt_est_all(dataset, runs):
    # Tune hyperparameters once and reuse them  [arxiv:2103.03098 §3.2.2]
    hparams = hopt(dataset, draw_seeds())
    scores = []
    for run in range(runs):
        # Fresh seeds and a new split per run  [arxiv:2103.03098 §3.2]
        seeds = draw_seeds()
        train_val, test = out_of_bootstrap_split(dataset, seeds)
        scores.append(evaluate(train(train_val, hparams, seeds), test))
    return mean(scores), std(scores)

# Decision rule over paired runs  [arxiv:2103.03098 §4.1]
# Input:  scores_a, scores_b (paired by split and seed)
# Output: "A is better" or "no conclusion"

def decide(scores_a, scores_b, gamma=0.75):
    # Fraction of paired runs where A wins  [arxiv:2103.03098 §4.1]
    p_ab = mean(a > b for a, b in zip(scores_a, scores_b))
    # Bootstrap interval over resampled pairs  [arxiv:2103.03098 app. A]
    ci_min, ci_max = percentile_bootstrap(scores_a, scores_b)
    # Significant: lower bound must clear 0.5  [arxiv:2103.03098 app. A]
    if ci_min <= 0.5:
        return "no conclusion"
    # Meaningful: upper bound must clear gamma  [arxiv:2103.03098 app. A]
    if ci_max <= gamma:
        return "no conclusion"
    return "A is better"

Contradictions

• Linear models versus Transformers for long-term forecasting. One side, reported in earlier work, is that linear models beat Transformers 2310.06119 §1. The other side, from later work, is the opposite 2310.06119 §1. Shao et al. attribute the conflict to dataset heterogeneity rather than to the models 2310.06119 §5. This page does not resolve it.
• Decision thresholds and tests. The variance study compares paired model runs with a bootstrap interval and a threshold of 0.75 2103.03098 §4.1. The human-preference study tests a stated probability against 0.5 with a t-test 2205.11930 §5.2. The bake off uses one-sided Wilcoxon tests with Holm correction to form cliques 2304.13029 §3. These are different designs and should not be mixed.
• Judge agreement is task-specific. The exam judge reaches question-level Spearman of 0.825 to 0.864 2305.12474 tab. 4. The agent-safety judge labels 35.9% of trajectories unsafe where humans label 71.7% 2507.06134 tab. 4. The tasks, judges and labels differ, so neither result transfers to the other task. Measure agreement on each benchmark.
• Disagreements between a paper's text and its own tables are listed on the paper cards, not here.

Papers

• Accounting for Variance in Machine Learning Benchmarks: randomises all sources except hyperparameter search and decides on P(A>B).
• A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers: ASDiv, with a lexical diversity metric and a label check.
• A Survey on Evaluation of Large Language Models: survey of LLM evaluation with no experiments; argues for dynamic benchmarks against contamination.
• A Categorical Archive of ChatGPT Failures: qualitative archive of failures in eleven categories, with no counts or sampling rule.
• Adversarial GLUE: filtered adversarial test set for five GLUE tasks.
• AI and the Everything in the Whole Wide World Benchmark: position paper on the construct validity of general benchmarks.
• Annotation Artifacts in Natural Language Inference Data: hypothesis-only baseline that reveals labels, and a hard subset that strong models miss.
• The Authenticity Gap in Human Evaluation: system-level probability as an alternative to Likert averages.
• Bake off redux: time-series classification benchmark with 30 new held-out datasets and a pairwise-matrix check on rankings.
• CHI-Bench: healthcare agent benchmark reporting pass^k, task-level bootstrap intervals and judge-family caveats.
• CINIC-10 is not ImageNet or CIFAR-10: 270,000-image CIFAR-10 drop-in with equal splits and an ImageNet part that is not vetted.
• CREPE: Can Vision-Language Foundation Models Reason Compositionally?: retrieval benchmark with seen/unseen compound splits and human-checked hard negatives.
• Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation: zero-shot navigation by a coding agent; success is scored by the driver, with few replicated cells.
• Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark: streaming score that rewards early detections; author-reported profile scores.
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark: exam benchmark with a GPT-4-turbo judge checked against teachers and a 2023 follow-up set.
• GPT-4o System Card: closed-model safety and capability report with TTS-converted audio evaluations and internal test sets.
• Do LLMs exhibit human-like response biases? A case study in survey design: response-bias probes with non-bias control edits.
• DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction: grid and graph traffic models under one protocol, with a leak found in released graph code.
• KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge: survey-share gold labels for social values, and an exam-style accuracy set.
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks: minimal-pair probes of one model.
• Med-HALT: medical hallucination benchmark with a penalised score.
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering: ScienceQA benchmark with author-run human and oracle reference points.
• Monocular Dynamic View Synthesis: A Reality Check: removes multi-view leakage from training with teleporting frames and reports effective multi-view factors.
• Exploring Progress in Multivariate Time Series Forecasting: shared pipeline (BasicTS+) and dataset heterogeneity analysis.
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark: video benchmark built by rewriting static image questions into temporal ones.
• Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems: rebuilt NIDS benchmarks with timing features; descriptive only, with text-table mismatches and no model.
• Texygen: A Benchmarking Platform for Text Generation Models: text-generation benchmark with a synthetic oracle, several metrics and a released toolkit.
• Evaluation of OpenAI o1: small, author-graded evaluation of a hosted model.
• Evaluation and Benchmarking of LLM Agents: A Survey: survey of agent evaluation objectives and processes, with no experiments.
• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation: professional MQM error annotation of WMT20 outputs, compared with the WMT ranking and metrics.
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs: CLIP-blind image pairs mined against DINOv2, scored as pairs with a 25% chance level.
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models: 1000-item OCR test with a substring scorer and author-reported model scores.
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs: three large graph tasks with sampled-negative ranking, a v2 protocol change and competition winners.
• nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation: six-dataset 3D segmentation benchmark that re-tests recent architecture claims against a tuned baseline.
• OpenAgentSafety: agent safety benchmark with a rule check and an LLM judge, reported side by side.
• Omni-SafetyBench: audio-visual safety benchmark with a comprehension gate and a consistency score.
• Sociotechnical Safety Evaluation of Generative AI Systems: three-layer framework and validity practices for safety evaluation.
• SoGraB: A Visual Method for Soft Grasping Benchmarking and Evaluation: deformation-based grasp score from a non-contact proxy; the constant is unstated and the results compare grippers only.
• LLM-based Human Simulations Have Not Yet Been Reliable: position paper on the reliability of simulated human participants.
• YOLO Evolution: A Comprehensive Benchmark and Architectural Review of YOLOv12, YOLO11, and Their Previous Versions: one-run-per-model detector benchmark with noted inconsistencies between text and tables.
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks: two spike-encoded speech sets from a fixed cochlea model, with spike-count and spike-timing baselines.
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models: 700-prompt text-to-video benchmark with 17 metrics and a score fitted to human ratings.
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning: parameterised training sweep on TPU, GPU and CPU; speedups depend on batch size and software version.
• MovieNet: A Holistic Dataset for Movie Understanding: movie-level multi-task dataset whose counts differ across sections.
• A survey on datasets for fairness-aware machine learning: 15 tabular fairness datasets; constant classifiers produce zero fairness gaps.
• MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation: generation tasks scored by narrow checks, with judge validation on a subset of tasks.
• AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions: underwater classification set; checkpoints chosen on test loss, one seed per model.
• LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena: live forecasting benchmark scored by Brier, calibration and market return.