Challenge-set evaluation

Overview

A challenge set is a test set built to expose one known weakness of a system, not to estimate its overall quality. Each item isolates the phenomenon, so a score on the set says how the system handles that phenomenon, and a general benchmark score says little about it 2111.15366 §1 1906.00591 §2.

The shared idea is that a correct output can be checked by reading one property of it. In the translation paper, that property is the grammatical gender that the translated job word receives, read from the target sentence after word alignment 1906.00591 §2. In the QA benchmark, it is which option the model picks, and "unknown" is the right pick when the context is ambiguous 2110.08193 §3. In the theory-of-mind critique, it is which candidate completion gets the higher probability 2302.08399 §2. In the vision-language benchmark, it is which caption goes with which image, when both captions use the same words 2204.03162 §3.2. In the completion probe, it is whether a generated text contains a word from a fixed list 2101.05783 §Results. The target language or the answer format has to expose the property for this to work.

Variants

Coreference-derived set with aligned gender readout

Build the set by joining existing coreference sentences, each with a target entity and a gold gender, as the paper does with Winogender and WinoBias. This yields 3,888 English sentences in total 1906.00591 §2. Translate each sentence, align the English entity to its translated words, and read the gender from those words with a per-language rule 1906.00591 §2 §3.1. Report accuracy, and report bias as the F1 gap between male and female and as a gap between pro- and anti-stereotypical sentences 1906.00591 §3.2. Card: Evaluating Gender Bias in Machine Translation.

The same paper samples 100 instances per system and language and has two native speakers mark the gender of the entity in the translation. The automatic method agreed with them on more than 85% of instances for every language and system, averaging 87% 1906.00591 §3.4. Use this as a check on the automatic readout, not as a replacement for it.

The paper also adds stereotypical adjectives before the entity and measures how much the accuracy changes 1906.00591 §3.3. It needs gold coreference, which the paper calls impractical for debiasing. Treat this as a diagnostic, not as a fix 1906.00591 §3.3 §4.

Hand-built template set with a correct "unknown" option

Write each template by hand, with two contexts and two questions. One context is ambiguous and one is disambiguated by a second sentence. The correct answer to an ambiguous context is "unknown", so a model that guesses from a stereotype is visible in its choices 2110.08193 §3. The paper uses 325 templates and 58,492 examples, with at least 25 templates per category 2110.08193 §1 §3.2. Each template gives four items that share one set of three options, and the position of the "unknown" option is randomised 2110.08193 §3.

Templates are checked by crowd workers. Five workers answer one sampled item per template condition, and a template passes when at least four agree with the gold label. Failing templates are edited and checked again 2110.08193 §4 app. D. The bias score counts non-unknown answers that follow a stereotype, and for ambiguous contexts it is scaled by accuracy 2110.08193 §5. Card: BBQ: A Hand-Built Bias Benchmark for Question Answering.

The set covers US English contexts only 2110.08193 §3.1. The authors warn that a low bias score does not show a model is unbiased 2110.08193 §7.

Minimal edits to a published probe

Take a published vignette and change one detail at a time, keeping the reasoning the same. Then read the probability of each candidate completion from the model and check whether the answer flips. The paper uses two vignettes from an earlier study and makes four variants of each 2302.08399 §2 §2.1 §2.2. In the transparent-container variant, the probability of the wrong completion is 95% 2302.08399 §2.1.1. The author argues that one clear failure on a trivial change should count for more than many successes 2302.08399 §1 §3. Card: Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks.

The result is one model, one vignette per variant and no repeated runs 2302.08399 §2 §2.1.1. A small change in spacing altered one result, and the author withdrew the earlier version of that claim 2302.08399 §2.1.1. Repeat each variant with several seeds and exact strings before you rely on a flip.

Image-caption minimal pairs with a group score

Write pairs of captions that use the same words in a different order, each with its own image, so that only word order and visual content separate them. The paper's set has 400 examples, written by expert annotators from licensed images. Each example needs the annotator to prefer the two correct image-caption pairs over the two swapped ones 2204.03162 §3.1. Score all four caption-image pairings. The text score asks whether each image prefers its own caption, the image score asks whether each caption prefers its own image, and the group score counts an example only when both hold 2204.03162 §3.2 eq. 1 eq. 2 eq. 3. The group score is the strictest of the three, following the argument that scoring each direction alone overestimates performance 2204.03162 §3.2. Card: Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality.

The best group score among the tested models is 14.50, and the human score is 85.50 2204.03162 tab. 3. The paper reports that every tested model is below the 16.67 chance level on this metric 2204.03162 §5.1 tab. 3. The paper's confidence intervals are wide, because each group has about 100 examples 2204.03162 app. A. The paper does not state the score function for each model, so the numbers depend on the choice you make 2204.03162 §3.2 §4.1. Name that choice when you report a score.

Sampled completions with a keyword reading

Use one fixed prompt that names a group. Sample many completions from a generative model and count the share that contains a word from a fixed list. Then repeat with other group names in the same frame and compare the shares 2101.05783 §Results fig. 1 app. B. The paper samples 100 completions per prompt and matches 26 keywords in part or whole 2101.05783 app. B. For analogies, count the nouns returned over 100 runs, with the adjective swapped for each group 2101.05783 §Results fig. 2. The paper uses this design because the model's learned embeddings were not available 2101.05783 §Results.

A mitigation is a short positive phrase placed before the prompt. The paper reports that its six best adjectives lower the violent share to 20%, which is still above the 13-15% reported for the Christian prompt [arxiv:2101.05783 fig. 4]. The six were chosen from a first run of 50 adjectives and then rerun, so treat the 20% as optimistic 2101.05783 §Results. Card: Persistent Anti-Muslim Bias in Large Language Models.

The labelling is keyword matching only, and the keyword list was written from 100 completions with no inter-rater check 2101.05783 app. B. The paper tests one model and one prompt family 2101.05783 §Introduction, so the rates describe this probe, not the model in general.

Hard subset from a baseline that ignores part of the input

This variant writes no new items. It takes an existing test set, runs a weak model that never sees part of the input, and keeps the items that model gets wrong. For natural language inference, the weak model sees only the hypothesis. A fastText classifier on that input reaches 67.0% on SNLI, against 34.3% for the majority class 1803.02324 tab. 2. The hard subset is the set of test items this baseline misclassifies 1803.02324 §4. On it, three NLI models score well below their full-test-set results. For SNLI, one model (DIIN) goes from 86.5 to 72.7 1803.02324 tab. 5 §4. The paper does not name the metric in its table.

Two limits apply. The split uses the gold labels, so the hard subset is not an unbiased training filter. The paper does not say which premise-blind model defines the split, although it names fastText and two alternatives 1803.02324 §2 [fn. 2] §4. Filtering to hard items can also leave a new skew, for example the word "no" shifting between classes, and the paper suggests balancing such cues across labels instead 1803.02324 §4. Card: Annotation Artifacts in Natural Language Inference Data.

Curated archive of observed failures

Collect failure examples from public posts or your own prompts, sort each one into a category, and keep them as hard items to re-run later. The archive for ChatGPT uses eleven categories and 25 figures, and reports no counts or rates 2302.03494 §2 §1. Its examples come mostly from social media, were picked by the author, and follow no sampling rule 2302.03494 §1 §2. One category shows the pattern: a spatial puzzle the model gets wrong, an arithmetic product that does not match the paper's reference, and a self-contradiction on two facts 2302.03494 §2.1 §2.2 §2.3. The author also reports that changing the wording of a known question changed the answer, which suggests memorised phrasing is part of the failure 2302.03494 §4. Items that show a refusal being bypassed belong in a separate safety set 2302.03494 §2.10. Card: A Categorical Archive of ChatGPT Failures.

Use the archive for coverage and ideas, not for a rate. Re-run any item you keep with a fixed prompt and model version, because the examples name no version for most entries and the author notes that some errors may already be fixed 2302.03494 §2.7.

Test suite designed over item types

A position paper, not a method. It argues that benchmarks for a general ability are weak measures of that ability, and recommends evaluating well-scoped tasks 2111.15366 §5. For broader questions it recommends test suites whose items cover a space of cases, audits that balance sensitive groups, minimal contrasting pairs, error analysis, and ablations 2111.15366 app. A. The aim is to find where a system fails, not to rank it 2111.15366 app. A. The paper gives no procedure, threshold or sample size, so you design them yourself 2111.15366 app. A. Card: AI and the Everything in the Whole Wide World Benchmark.

When to use which

The choice depends on what you can run and what you can read from an output, not on which result looks best.

Implementation sketch

Language-agnostic pseudocode of the common core, from the translation variant. The readout step is where variants differ. For the QA set, the readout is the chosen option label, and the score is a bias share over non-unknown answers 2110.08193 §5. For the probe, it is the probability of each candidate completion 2302.08399 §2. For the image-caption pairs, it is the model's score for each of the four pairings 2204.03162 §3.2.

# Build instances: sentence, entity span, gold attribute  [arxiv:1906.00591 §2]
instances = load_challenge_set()

# Run the system under test on every sentence  [arxiv:1906.00591 §2]
outputs = [system.translate(x.text, target=lang) for x in instances]

# Align source words to output words  [arxiv:1906.00591 §2]
links = aligner.align(instances, outputs)

# Collect the output words linked to the entity  [arxiv:1906.00591 §2]
entity_out = linked_words(links, instances)

# Read the attribute from those words with a per-language rule  [arxiv:1906.00591 §3.1]
predicted = readout_rule(lang, entity_out)

# Score: accuracy, plus a gap between groups or between subsets  [arxiv:1906.00591 §3.2]
accuracy = mean(predicted == x.gold for x in instances)

Notes for implementers:

• Aligner training data. The paper trains the aligner on each system's own translations, with no separate parallel corpus 1906.00591 §2. Alignment errors are then part of the measured error.
• Unknown readouts. The paper does not say how an unknown readout enters the score. The official code counts it as wrong for accuracy. Say which choice you made 1906.00591 §3.2.
• Neutral instances. The paper does not say how neutral gold instances enter F1 1906.00591 §2 §3.2.
• Option order. In the QA set, the position of the "unknown" option is drawn at random for each instance 2110.08193 §3. The released generator does not set a seed generate_from_template_all_categories.py, so regenerating the items gives different ones. Keep your own items fixed once built.
• Score function. For image-caption pairs, the paper does not say which score each model uses. Choose one, state it, and test whether the choice changes the group score 2204.03162 §3.2 §4.1.
• Sampling. For completion probes, the paper does not state the maximum length or stop sequence 2101.05783 app. A, nor the number of completions per debiasing prompt 2101.05783 §Results. Fix these in your own runs and report them.

Contradictions

The sources disagree on whether the model has the ability these tasks test. The earlier work reported that a model succeeds on these theory-of-mind tasks at a level comparable to nine-year-old children 2302.08399 §1. The later probe reports wrong completions after trivial edits to the same kind of vignette 2302.08399 §2.1 §2.2. The two sides are not reconciled here.

Within the translation paper, its text and its released code define the bias gap differently. The text reports Δ_S as an F1 difference 1906.00591 §3.2. The code subtracts accuracies. A reimplementation that follows one of them will not match results from the other. See Paper vs code on the card for the full list of differences.

Within the QA paper, the text describes exact matching of model outputs, and the released scoring script adds a fallback match 2110.08193 §5 BBQ_calculate_bias_score.R. Results from the two may differ slightly. See Paper vs code.

Within the vision-language paper, the abstract says that none of the models does much better than chance, while §5.1 says six models exceed chance on the text score and only on that score 2204.03162 §5.1 tab. 3. The two statements describe the same results differently, and this page does not resolve the wording.

Papers

• Evaluating Gender Bias in Machine Translation: WinoMT, a 3,888-sentence challenge set and an automatic gender-bias metric for machine translation.
• BBQ: A Hand-Built Bias Benchmark for Question Answering: a hand-written QA benchmark with ambiguous and disambiguated contexts, scored by how often non-unknown answers follow a stereotype.
• AI and the Everything in the Whole Wide World Benchmark: a position paper on the construct validity of general benchmarks, recommending test suites, audits and error analysis instead.
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks: a critique that edits two published vignettes and reports flipped completions for one model.
• Annotation Artifacts in Natural Language Inference Data: a hypothesis-only classifier that reveals label cues in SNLI and MultiNLI, and a hard subset on which strong NLI models score much lower.
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality: a 400-example expert-written set of caption pairs that share their words, scored by text, image and group accuracy, where most models score near chance.
• A Categorical Archive of ChatGPT Failures: a qualitative archive of ChatGPT failures in eleven categories, with no counts or method.
• Persistent Anti-Muslim Bias in Large Language Models: a keyword-based completion and analogy probe of GPT-3, and a test of whether a positive prefix lowers the violent share.