---
kind: concept
slug: challenge-set-evaluation
title: Challenge-set evaluation
summary: >-
  Probes one known weakness with a small set whose items isolate it, reading one property of each output. Variants: translation gender bias, hand-built QA with an "unknown" option, minimal edits, image-caption minimal pairs, sampled completions, curated failure archive, test suites, hard NLI subset. Each needs its own reading rule; API results may not reproduce.
tags: [evaluation, bias, machine-translation, test-sets, vision-language, text-generation]
papers: ["1803.02324", "1906.00591", "2101.05783", "2110.08193", "2111.15366", "2204.03162", "2302.03494", "2302.08399"]
related: [algorithmic-fairness, adversarial-evaluation, llm-evaluation, benchmark-validity, vision-language-pretraining, prompt-engineering]
status: draft
updated_at: 2026-10-24
---

## Overview

A challenge set is a test set built to expose one known weakness of a system,
not to estimate its overall quality. Each item isolates the phenomenon, so a
score on the set says how the system handles that phenomenon, and a general
benchmark score says little about it [arxiv:2111.15366 §1] [arxiv:1906.00591 §2].

The shared idea is that a correct output can be checked by reading one
property of it. In the translation paper, that property is the grammatical
gender that the translated job word receives, read from the target sentence
after word alignment [arxiv:1906.00591 §2]. In the QA benchmark, it is which
option the model picks, and "unknown" is the right pick when the context is
ambiguous [arxiv:2110.08193 §3]. In the theory-of-mind critique, it is which
candidate completion gets the higher probability [arxiv:2302.08399 §2]. In the
vision-language benchmark, it is which caption goes with which image, when both
captions use the same words [arxiv:2204.03162 §3.2]. In the completion probe, it
is whether a generated text contains a word from a fixed list
[arxiv:2101.05783 §Results]. The target language or the answer format has to
expose the property for this to work.

## Variants

### Coreference-derived set with aligned gender readout

Build the set by joining existing coreference sentences, each with a target
entity and a gold gender, as the paper does with Winogender and WinoBias. This
yields 3,888 English sentences in total [arxiv:1906.00591 §2]. Translate each
sentence, align the English entity to its translated words, and read the
gender from those words with a per-language rule [arxiv:1906.00591 §2] [§3.1].
Report accuracy, and report bias as the F1 gap between male and female and
as a gap between pro- and anti-stereotypical sentences [arxiv:1906.00591 §3.2].
Card: [Evaluating Gender Bias in Machine Translation](/papers/1906.00591).

The same paper samples 100 instances per system and language and has two
native speakers mark the gender of the entity in the translation. The
automatic method agreed with them on more than 85% of instances for every
language and system, averaging 87% [arxiv:1906.00591 §3.4]. Use this as a
check on the automatic readout, not as a replacement for it.

The paper also adds stereotypical adjectives before the entity and measures
how much the accuracy changes [arxiv:1906.00591 §3.3]. It needs gold
coreference, which the paper calls impractical for debiasing. Treat this as a
diagnostic, not as a fix [arxiv:1906.00591 §3.3] [§4].

### Hand-built template set with a correct "unknown" option

Write each template by hand, with two contexts and two questions. One context
is ambiguous and one is disambiguated by a second sentence. The correct answer
to an ambiguous context is "unknown", so a model that guesses from a
stereotype is visible in its choices [arxiv:2110.08193 §3]. The paper uses 325
templates and 58,492 examples, with at least 25 templates per category
[arxiv:2110.08193 §1] [§3.2]. Each template gives four items that share one
set of three options, and the position of the "unknown" option is randomised
[arxiv:2110.08193 §3].

Templates are checked by crowd workers. Five workers answer one sampled item
per template condition, and a template passes when at least four agree with
the gold label. Failing templates are edited and checked again
[arxiv:2110.08193 §4] [app. D]. The bias score counts non-unknown answers that
follow a stereotype, and for ambiguous contexts it is scaled by accuracy
[arxiv:2110.08193 §5]. Card: [BBQ: A Hand-Built Bias Benchmark for Question Answering](/papers/2110.08193).

The set covers US English contexts only [arxiv:2110.08193 §3.1]. The authors
warn that a low bias score does not show a model is unbiased [arxiv:2110.08193 §7].

### Minimal edits to a published probe

Take a published vignette and change one detail at a time, keeping the
reasoning the same. Then read the probability of each candidate completion
from the model and check whether the answer flips. The paper uses two
vignettes from an earlier study and makes four variants of each
[arxiv:2302.08399 §2] [§2.1] [§2.2]. In the transparent-container variant, the
probability of the wrong completion is 95% [arxiv:2302.08399 §2.1.1]. The
author argues that one clear failure on a trivial change should count for more
than many successes [arxiv:2302.08399 §1] [§3]. Card:
[Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](/papers/2302.08399).

The result is one model, one vignette per variant and no repeated runs
[arxiv:2302.08399 §2] [§2.1.1]. A small change in spacing altered one result,
and the author withdrew the earlier version of that claim
[arxiv:2302.08399 §2.1.1]. Repeat each variant with several seeds and exact
strings before you rely on a flip.

### Image-caption minimal pairs with a group score

Write pairs of captions that use the same words in a different order, each with
its own image, so that only word order and visual content separate them. The
paper's set has 400 examples, written by expert annotators from licensed
images. Each example needs the annotator to prefer the two correct
image-caption pairs over the two swapped ones [arxiv:2204.03162 §3.1]. Score all
four caption-image pairings. The text score asks whether each image prefers its
own caption, the image score asks whether each caption prefers its own image,
and the group score counts an example only when both hold
[arxiv:2204.03162 §3.2] [eq. 1] [eq. 2] [eq. 3]. The group score is the
strictest of the three, following the argument that scoring each direction
alone overestimates performance [arxiv:2204.03162 §3.2]. Card:
[Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](/papers/2204.03162).

The best group score among the tested models is 14.50, and the human score is
85.50 [arxiv:2204.03162 tab. 3]. The paper reports that every tested model is below
the 16.67 chance level on this metric [arxiv:2204.03162 §5.1] [tab. 3]. The paper's confidence intervals
are wide, because each group has about 100 examples [arxiv:2204.03162 app. A].
The paper does not state the score function for each model, so the numbers
depend on the choice you make [arxiv:2204.03162 §3.2] [§4.1]. Name that choice
when you report a score.

### Sampled completions with a keyword reading

Use one fixed prompt that names a group. Sample many completions from a
generative model and count the share that contains a word from a fixed list.
Then repeat with other group names in the same frame and compare the shares
[arxiv:2101.05783 §Results] [fig. 1] [app. B]. The paper samples 100
completions per prompt and matches 26 keywords in part or whole
[arxiv:2101.05783 app. B]. For analogies, count the nouns returned over 100
runs, with the adjective swapped for each group [arxiv:2101.05783 §Results]
[fig. 2]. The paper uses this design because the model's learned embeddings were not
available [arxiv:2101.05783 §Results].

A mitigation is a short positive phrase placed before the prompt. The paper
reports that its six best adjectives lower the violent share to 20%, which is
still above the 13-15% reported for the Christian prompt [arxiv:2101.05783
fig. 4]. The six were chosen from a first run of 50 adjectives and then rerun,
so treat the 20% as optimistic [arxiv:2101.05783 §Results]. Card:
[Persistent Anti-Muslim Bias in Large Language Models](/papers/2101.05783).

The labelling is keyword matching only, and the keyword list was written from
100 completions with no inter-rater check [arxiv:2101.05783 app. B]. The paper
tests one model and one prompt family [arxiv:2101.05783 §Introduction], so the
rates describe this probe, not the model in general.

### Hard subset from a baseline that ignores part of the input

This variant writes no new items. It takes an existing test set, runs a weak
model that never sees part of the input, and keeps the items that model gets
wrong. For natural language inference, the weak model sees only the hypothesis.
A fastText classifier on that input reaches 67.0% on SNLI, against 34.3% for
the majority class [arxiv:1803.02324 tab. 2]. The hard subset is the set of
test items this baseline misclassifies [arxiv:1803.02324 §4]. On it, three NLI
models score well below their full-test-set results. For SNLI, one model
(DIIN) goes from 86.5 to 72.7 [arxiv:1803.02324 tab. 5] [§4]. The paper does
not name the metric in its table.

Two limits apply. The split uses the gold labels, so the hard subset is not an
unbiased training filter. The paper does not say which premise-blind model
defines the split, although it names fastText and two alternatives
[arxiv:1803.02324 §2] [fn. 2] [§4]. Filtering to hard items can also leave a
new skew, for example the word "no" shifting between classes, and the paper
suggests balancing such cues across labels instead [arxiv:1803.02324 §4].
Card: [Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324).

### Curated archive of observed failures

Collect failure examples from public posts or your own prompts, sort each one
into a category, and keep them as hard items to re-run later. The archive for
ChatGPT uses eleven categories and 25 figures, and reports no counts or rates
[arxiv:2302.03494 §2] [§1]. Its examples come mostly from social media, were
picked by the author, and follow no sampling rule [arxiv:2302.03494 §1] [§2].
One category shows the pattern: a spatial puzzle the model gets wrong, an
arithmetic product that does not match the paper's reference, and a
self-contradiction on two facts [arxiv:2302.03494 §2.1] [§2.2] [§2.3]. The
author also reports that changing the wording of a known question changed the
answer, which suggests memorised phrasing is part of the failure
[arxiv:2302.03494 §4]. Items that show a refusal being bypassed belong in a
separate safety set [arxiv:2302.03494 §2.10]. Card:
[A Categorical Archive of ChatGPT Failures](/papers/2302.03494).

Use the archive for coverage and ideas, not for a rate. Re-run any item you keep
with a fixed prompt and model version, because the examples name no version for
most entries and the author notes that some errors may already be fixed
[arxiv:2302.03494 §2.7].

### Test suite designed over item types

A position paper, not a method. It argues that benchmarks for a general
ability are weak measures of that ability, and recommends evaluating
well-scoped tasks [arxiv:2111.15366 §5]. For broader questions it recommends
test suites whose items cover a space of cases, audits that balance sensitive
groups, minimal contrasting pairs, error analysis, and ablations
[arxiv:2111.15366 app. A]. The aim is to find where a system fails, not to
rank it [arxiv:2111.15366 app. A]. The paper gives no procedure, threshold or
sample size, so you design them yourself [arxiv:2111.15366 app. A].
Card: [AI and the Everything in the Whole Wide World Benchmark](/papers/2111.15366).

## When to use which

The choice depends on what you can run and what you can read from an output,
not on which result looks best.

| Situation | Use | Why |
|---|---|---|
| You can run the system on your own sentences and the target language marks the property on its words | Coreference-derived set with aligned readout | This is the documented pipeline for machine translation output. It covers eight languages with rules [arxiv:1906.00591 §3.1] |
| The system answers a multiple-choice question, and some inputs do not support any answer | Hand-built template set with an "unknown" option | The paper's design separates guessing from overriding the context [arxiv:2110.08193 §1] [§3] |
| You need to know whether the automatic readout is right for your setup | Add the human-checked sample | The agreement figures are for the paper's own systems and languages only [arxiv:1906.00591 §3.4] |
| The target language is not among the eight in the paper | Write and validate your own readout rule | The paper gives rules only for its languages [arxiv:1906.00591 §3.1]. Validate the rule with a human sample before you trust the numbers |
| The target setting is not US English | Do not reuse the QA set as is | The benchmark covers US English contexts only [arxiv:2110.08193 §3.1] [§7] |
| You want to check whether a published probe's success holds under small changes | Minimal-edit probe | It shows whether a flip happens, but it needs repeated runs and a second model before you claim more [arxiv:2302.08399 §2] |
| The system is a vision-language model and you need to know whether it reads word order | Image-caption minimal pairs with a group score | The word set is fixed, so a bag-of-words shortcut does not help [arxiv:2204.03162 §2] [§3.1]. The set is English only and has 400 items [arxiv:2204.03162 §7] |
| You can only sample text from a generative model and cannot read its internals | Sampled completions with a keyword reading | The paper used this design without embeddings [arxiv:2101.05783 §Results]. Matching is by keyword only, so check the list before you trust the rate [arxiv:2101.05783 app. B] |
| You want to test a prompt-level mitigation | Sampled completions, reporting the topic shift too | The paper reports that the positive phrase moves output toward other topics, so a lower violent share is not a neutral fix [arxiv:2101.05783 §Discussion] [app. E] |
| You want to know whether a stereotype-based fix works | Oracle-coreference probe only as a diagnostic | The paper says gold coreference is impractical for debiasing [arxiv:1906.00591 §3.3] |
| You can only query a commercial API | Save the outputs and score them offline | The paper gives no API version, query date or decoding setting, so new calls may not match its numbers [arxiv:1906.00591 §3.1] |
| You need varied hard cases to check for, and no score yet | Curated archive of observed failures | It gives coverage across eleven categories, but no counts or method, so it cannot rank or rate [arxiv:2302.03494 §2] [§4] |
| You have no validated set yet and need to justify the design | Test suite over item types, as an argument | The paper gives reasoning, not a procedure [arxiv:2111.15366 §5] [app. A] |
| You want to know whether a benchmark score depends on artifacts in the items rather than on the task | Hard subset from a baseline that ignores part of the input | The paper uses it as a diagnostic of label cues, not as a benchmark result [arxiv:1803.02324 §2] [§5] |
| You want to use that hard subset as a training filter | Do not use it as is | The split depends on the gold labels, and filtering can add a new skew [arxiv:1803.02324 §4] |

## Implementation sketch

Language-agnostic pseudocode of the common core, from the translation variant.
The readout step is where variants differ. For the QA set, the readout is the
chosen option label, and the score is a bias share over non-unknown answers
[arxiv:2110.08193 §5]. For the probe, it is the probability of each candidate
completion [arxiv:2302.08399 §2]. For the image-caption pairs, it is the model's
score for each of the four pairings [arxiv:2204.03162 §3.2].

```pseudocode
# Build instances: sentence, entity span, gold attribute  [arxiv:1906.00591 §2]
instances = load_challenge_set()

# Run the system under test on every sentence  [arxiv:1906.00591 §2]
outputs = [system.translate(x.text, target=lang) for x in instances]

# Align source words to output words  [arxiv:1906.00591 §2]
links = aligner.align(instances, outputs)

# Collect the output words linked to the entity  [arxiv:1906.00591 §2]
entity_out = linked_words(links, instances)

# Read the attribute from those words with a per-language rule  [arxiv:1906.00591 §3.1]
predicted = readout_rule(lang, entity_out)

# Score: accuracy, plus a gap between groups or between subsets  [arxiv:1906.00591 §3.2]
accuracy = mean(predicted == x.gold for x in instances)
```

Notes for implementers:

- Aligner training data. The paper trains the aligner on each system's own
  translations, with no separate parallel corpus [arxiv:1906.00591 §2].
  Alignment errors are then part of the measured error.
- Unknown readouts. The paper does not say how an unknown readout enters the
  score. The official code counts it as wrong for accuracy. Say which choice
  you made [arxiv:1906.00591 §3.2].
- Neutral instances. The paper does not say how neutral gold instances enter
  F1 [arxiv:1906.00591 §2] [§3.2].
- Option order. In the QA set, the position of the "unknown" option is drawn
  at random for each instance [arxiv:2110.08193 §3]. The released generator
  does not set a seed [code: generate_from_template_all_categories.py], so
  regenerating the items gives different ones. Keep your own items fixed once
  built.
- Score function. For image-caption pairs, the paper does not say which score
  each model uses. Choose one, state it, and test whether the choice changes the
  group score [arxiv:2204.03162 §3.2] [§4.1].
- Sampling. For completion probes, the paper does not state the maximum length
  or stop sequence [arxiv:2101.05783 app. A], nor the number of completions per
  debiasing prompt [arxiv:2101.05783 §Results]. Fix these in your own runs and
  report them.

## Contradictions

The sources disagree on whether the model has the ability these tasks test.
The earlier work reported that a model succeeds on these theory-of-mind tasks at a level
comparable to nine-year-old children [arxiv:2302.08399 §1]. The later probe
reports wrong completions after trivial edits to the same kind of vignette
[arxiv:2302.08399 §2.1] [§2.2]. The two sides are not reconciled here.

Within the translation paper, its text and its released code define the bias
gap differently. The text reports Δ_S as an F1 difference
[arxiv:1906.00591 §3.2]. The code subtracts accuracies. A reimplementation
that follows one of them will not match results from the other. See
[Paper vs code](/papers/1906.00591) on the card for the full list of
differences.

Within the QA paper, the text describes exact matching of model outputs, and
the released scoring script adds a fallback match [arxiv:2110.08193 §5]
[code: analysis_scripts/BBQ_calculate_bias_score.R#L96-L98]. Results from the
two may differ slightly. See [Paper vs code](/papers/2110.08193).

Within the vision-language paper, the abstract says that none of the models
does much better than chance, while §5.1 says six models exceed chance on the
text score and only on that score [arxiv:2204.03162 §5.1] [tab. 3]. The two
statements describe the same results differently, and this page does not
resolve the wording.

## Papers

- [Evaluating Gender Bias in Machine Translation](/papers/1906.00591): WinoMT, a 3,888-sentence challenge set and an automatic gender-bias metric for machine translation.
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](/papers/2110.08193): a hand-written QA benchmark with ambiguous and disambiguated contexts, scored by how often non-unknown answers follow a stereotype.
- [AI and the Everything in the Whole Wide World Benchmark](/papers/2111.15366): a position paper on the construct validity of general benchmarks, recommending test suites, audits and error analysis instead.
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](/papers/2302.08399): a critique that edits two published vignettes and reports flipped completions for one model.
- [Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324): a hypothesis-only classifier that reveals label cues in SNLI and MultiNLI, and a hard subset on which strong NLI models score much lower.
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](/papers/2204.03162): a 400-example expert-written set of caption pairs that share their words, scored by text, image and group accuracy, where most models score near chance.
- [A Categorical Archive of ChatGPT Failures](/papers/2302.03494): a qualitative archive of ChatGPT failures in eleven categories, with no counts or method.
- [Persistent Anti-Muslim Bias in Large Language Models](/papers/2101.05783): a keyword-based completion and analogy probe of GPT-3, and a test of whether a positive prefix lowers the violent share.