AI and the Everything in the Whole Wide World Benchmark

In plain words

Machine learning has a habit of treating a handful of famous test sets as scoreboards for "general" intelligence. A high score on one of them is read as progress toward a system that can do many things well. The authors argue that this reading is wrong. A benchmark is a particular dataset plus a scoring rule, so it only measures what its items and scoring happen to cover. Those choices are arbitrary, reflect the people who built the data, and are often left undocumented. Calling a benchmark "general" does not make it general.

The paper opens with a children's book in which a character visits a museum that claims to hold "everything in the whole wide world". Its rooms are arbitrary and oddly specific, and the museum is still finite. The authors say popular benchmarks have the same problem. They do not say benchmarks are useless. Their advice is to name each benchmark after the concrete task it tests and to stop presenting it as a general measure of ability. For broader questions, they suggest other kinds of evaluation: targeted test suites, audits across groups of people, close reading of errors, ablation studies, and measurements such as energy use and memory. The paper runs no experiments of its own.

TL;DR

A position paper, not a method. It argues that "general" benchmarks such as ImageNet and GLUE/SuperGLUE have a construct validity problem: their tasks are chosen arbitrarily, their data is limited and subjective, and their popularity drives the field toward chasing scores. It recommends evaluating well-scoped tasks and using test suites, audits, error analysis, ablation and model-property measures for broader questions §5 app. A. There is nothing to implement and no results to reproduce. Use it as a critique to cite when a claim of "general" performance rests on one benchmark.

Problem

The paper targets benchmarks that are presented, or taken up, as measures of general capability, such as "general" object recognition or "general" language understanding §3. Its claim is that these benchmarks do not measure what they say, which is a construct validity problem: the benchmark, in its data, metrics and practice, cannot represent the general claims made about it §1. The introduction ties this to the Common Task Framework, the shared-dataset evaluation model that began in speech recognition and machine translation, and argues that it was designed for narrow, practical tasks §2.2 §3.

The paper states its assumptions and scope. It covers dataset-based benchmarks, not game-playing demonstrations §3.1. It sets aside artificial general intelligence and studies narrower claims of generality for a specific capability, such as vision or language §2.1. It treats construct validity as the external-validity question of how well a benchmark represents the task it claims to measure §2.3.

Method

The paper has no algorithm, model or procedure. Its method is a structured argument, with two case studies and a set of recommendations.

1. Case studies §3. It examines ImageNet §3.1 and GLUE and SuperGLUE §3.2. For ImageNet it notes the size of the dataset compared with earlier benchmarks, and that its creators and others describe it as a comprehensive map of visual objects §3.1. For GLUE and SuperGLUE it notes that the creators frame them as general-purpose evaluations of language understanding and transfer §3.2.
2. Limited task design §4.1. Tasks are chosen as convenient samples rather than systematically §4.1. ImageNet's categories come from parts of the WordNet hierarchy, which was adopted largely as it was §4.1.1. GLUE's tasks came from a survey of proposals, filtered by practical criteria such as licensing and format §4.1.1. Conflating linguistic competence with world knowledge in one benchmark is a further problem §4.1.2.
3. De-contextualised data and reporting §4.2. Datasets are closed, subjective and limited in scope §4.2.1 §4.2.2. Coverage is skewed geographically and culturally, and GLUE's tasks are narrow in format §4.2.1 §4.2.2. A claim of generality can hide these limits §4.2.
4. Inappropriate community use §4.3. Leaderboards and competitions encourage incremental score-chasing §4.3.1. Benchmarks can steer a field's attention and resources toward particular approaches §4.3.2. Benchmark scores are cited in marketing as evidence of deployable performance §4.3.3. Aggregate scores can hide poor performance on minority subgroups §4.3.3.
5. Recommendations §5. Two paths. First, present benchmarks as evaluating concrete, well-scoped, contextualised tasks. Second, for broader objectives, use alternative evaluation methods, described in more detail in app. A:
   • Test suites, audits and adversarial testing [app. A, subsection "Testsuites, Audits and Adversarial Testing"]. Test sets are designed to cover a space of item types, not sampled from a dataset. Audits balance sensitive groups. Adversarial tests use minimal contrasting pairs. The aim is to find where a system fails, not to rank systems.
   • System output analysis app. A, "System Output Analysis". Error analysis (including confusion matrices), disaggregated analysis by socially salient subgroups, and counterfactual analysis of changed inputs.
   • Ablation testing app. A, "Ablation Testing". Remove or change components, or subsets of training data, to see what each contributes.
   • Analysis of model properties app. A, "Analysis of Model Properties". Energy use, memory and compute requirements, and stability under changes to training data.

The paper gives no procedure. The following restates its recommendations as a decision outline, not as an algorithm from the paper §5 app. A.

# Choosing an evaluation, after the paper's recommendations  [§5] [app. A]
# Input:  claim (str), task_scope (str)
# Output: list of evaluation methods (str)

def choose_evaluation(claim, task_scope):
    # Tie the benchmark to one concrete, well-scoped task  [§5]
    task = describe_task(task_scope)
    # A claim of general ability is not settled by one benchmark score  [§5]
    if claim_is_general(claim):
        # Probe where a system fails, not how it ranks  [app. A]
        methods = [test_suite, audit, error_analysis, ablation]
    else:
        methods = [benchmark_score(task)]
    # Report results on socially salient subgroups as well  [app. A]
    methods.append(disaggregated_analysis)
    return methods

The conclusion restates the argument: no dataset is neutral or universal, and benchmarks are most useful for showing how systems work and where they fail, not for declaring a winner §6.

Hyperparameters

None. The paper reports no models, training runs or experiments §5 app. A.

Results

Not applicable. The paper presents no new experimental results. The table below lists figures it quotes from earlier work, so readers can check them against the original sources. These are the cited authors' numbers, not findings of this paper.

Paper vs code

No official code. The paper has no implementation. Searches on 2026-10-18 found nothing: the arXiv entry and paper text link no repository, and GitHub searches for "whole wide world benchmark" and "benchmark construct validity raji" returned no official repository.

Pitfalls

• Argument, not evidence. The paper's claims are normative and critical. It does not operationalise "general" or measure how much benchmarks mislead. It admits the criterion for valid benchmarks is hard to formalise §2.3.
• Secondary sources. Many examples rest on quotations from other papers, blog posts or press, such as the Wagstaff quotes §4.1.1 §4.3.1 and the marketing claim §4.3.3. Check the original before reusing any of them.
• History is a sketch. The paper calls its history of benchmarking "stylized" and "far from comprehensive" §2.2. Do not cite it as an account of the field's history.
• Alternatives are not specified. The recommended methods come with no procedures, thresholds, sample sizes or costs. None are stated app. A. Implementing them means designing them from the cited examples.
• Scope of the critique. Applies to dataset-based benchmarks §3.1. It does not show that the cited benchmarks are useless for their original tasks; the paper says so itself §1.
• Source markup. The LaTeX source contains author-edit macros. In §4.2.2 one sentence still has a deleted phrase inside the source markup. The intended wording is ambiguous, so do not quote that sentence.
• Figures. The appendix includes a photograph of a benchmark sign, wrapped beside the text app. A. It is not reproduced here. This site has no images.
• Version. This card covers arXiv v1, dated 2021-11-26, the version recorded by fetch. The paper's venue is not stated in the metadata or in the source beyond a workshop acknowledgement [Acknowledgments], so none is recorded here.

Lineage

• Builds on: the Common Task Framework and shared-task evaluation, as described in the cited work of Donoho, Liberman and others §2.2. Not compiled in this wiki.
• Critiques: the "general" benchmarks GLUE/SuperGLUE and ImageNet, as cited throughout §3 §4. No cards exist for them here.
• Compared against: not applicable, the paper has no experiments.
• Superseded by: not stated.