---
kind: concept
slug: dataset-cartography
title: Dataset cartography
summary: >-
  Scoring training examples by how much the model's probability for them changes across epochs, then selecting ambiguous ones. Covers the gold-label variability score, WANLI's max-variability estimator for unlabelled candidates, and per-label selection. The evidence is one author-reported NLI study: it needs saved checkpoints, its estimator correlates weakly, and its two WANLI-alone tables disagree.
tags: [data-selection, training-dynamics, natural-language-inference, data-quality]
papers: ["2201.05955"]
related: [dataset-curation, noisy-labels, active-learning, challenge-set-evaluation, inter-annotator-agreement, benchmark-validity]
status: draft
updated_at: 2026-11-06
---

## Overview

Dataset cartography scores each training example by its training dynamics:
how the model's probability for an example changes from one saved epoch to
the next. Examples whose score is high are treated as ambiguous. Examples
whose score is low are treated as easy. The WANLI paper uses the
"variability" version of this idea, meaning the spread of the probability of
the correct label across epochs, to pick which examples to build on [arxiv:2201.05955 §2.1].

Only one card on this wiki uses the technique, so every claim below rests on
WANLI. The original cartography paper is named in WANLI's lineage but is not
compiled here [arxiv:2201.05955 Lineage], so its own results are not on this
page.

Numbers are author-reported, single-run unless a card says otherwise.

## Variants

### Gold-label variability

For each training example, record the probability the model gives its gold
label after each saved epoch. The variability is the standard deviation of
those probabilities. The WANLI paper calls an example ambiguous when this
variability is high [arxiv:2201.05955 §2.1]. The authors take the most
ambiguous 25% of each gold label as seed examples for generation, and the
telephone genre is excluded from that step [arxiv:2201.05955 §2.1]. Card:
[WANLI](/papers/2201.05955).

### Max variability for unlabelled candidates

Generated candidates have no gold label that the model was trained on, so
WANLI scores them with an estimator. For each candidate, the score is the
largest, across the three labels, of the standard deviation of that label's
probability over the saved epochs [arxiv:2201.05955 eq. 1]. The authors state
that this assumes adding the candidate to training would barely change any
saved checkpoint [arxiv:2201.05955 §2.3]. Card: [WANLI](/papers/2201.05955).

### Per-label quota selection

Rank by score inside each label, then keep a fixed share from each label. The
WANLI filter keeps half of the filtered set and gives each label the same
count, so the classes stay balanced [arxiv:2201.05955 §2.3]. Balancing is only
approximate in the final data: contradictions are about 15% of the labelled
examples [arxiv:2201.05955 §5.3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Labelled data, and the model's per-epoch checkpoints are saved | Gold-label variability to pick seeds or hard examples | This is the score WANLI uses for seeds. Its checkpoint requirement is stated [arxiv:2201.05955 §2.1] [§2.3] |
| Unlabelled candidates, for example generated examples | Max-variability estimator, treated as a noisy ranking | WANLI uses it for filtering, but its correlation with true variability is 0.527 on one held-out MNLI sample [arxiv:2201.05955 app. A] |
| Only the final model, no saved epochs | Not established by this wiki | The WANLI card states the score cannot be computed as described without per-epoch checkpoints [arxiv:2201.05955 §2.3] |
| Need a fixed-size subset of an existing dataset | Ambiguity-based selection versus random downsampling, measured per target set | On WANLI's 103K-sized MNLI subsets, the ambiguity-selected subset beats random downsampling on some sets and loses on others, for example HANS 74.93 against 71.15 and QNLI 54.42 against 59.15 [arxiv:2201.05955 tab. 10] |
| Want to generate new hard examples | Selection after generation, as WANLI does | WANLI-alone training beats MultiNLI-alone training on each out-of-domain set in the paper's Table 3 [arxiv:2201.05955 §3.3] [tab. 3]. This compares generation plus selection with MultiNLI, so it does not isolate the effect of selection alone |

## Implementation sketch

Pseudocode in our own words, with the paper section for each step. Assumes a
list of per-epoch probabilities for each example.

```pseudocode
# probs[e][y]: model probability of label y after saved epoch e.
# Requires one saved checkpoint per epoch [arxiv:2201.05955 §2.3].

def gold_variability(probs, gold):
    # Std of the gold-label probability across epochs; high = ambiguous
    # [arxiv:2201.05955 §2.1]
    return std([probs[e][gold] for e in range(num_epochs)])

def max_variability(probs):
    # No gold label: largest per-label std across epochs [arxiv:2201.05955 eq. 1]
    return max(std([probs[e][y] for e in range(num_epochs)])
               for y in LABELS)

def select_per_label(examples, score, keep_fraction):
    # Keep the highest-scoring share inside each label, so classes stay
    # balanced [arxiv:2201.05955 §2.3]
    chosen = []
    for y in LABELS:
        pool = sorted(of_label(examples, y), key=score, reverse=True)
        chosen += pool[: int(len(pool) * keep_fraction)]
    return chosen
```

Use population or sample standard deviation consistently, because the paper
does not say which [arxiv:2201.05955 §2.3]; see Pitfalls.

## Contradictions

The two WANLI-alone rows in the same paper do not match. Table 3 reports QNLI
81.40, HANS 89.28, BIG-Bench 85.19 and WNLI 67.28 [arxiv:2201.05955 tab. 3].
Table 10 reports the same training set, at the same size, with QNLI 76.81,
HANS 89.40, BIG-Bench 75.40 and WNLI 65.15 [arxiv:2201.05955 tab. 10]. The
paper does not explain the gap.

## Pitfalls

- **Model-dependent scores.** Seeds, neighbours and scores all depend on one
  specific model M, so changing M changes the selected data [arxiv:2201.05955 §2.1].
- **Checkpoints are required.** Without one saved checkpoint per epoch the
  estimator cannot be computed as the paper describes [arxiv:2201.05955 §2.3].
- **Weak estimator.** The max-variability estimator correlates only 0.527 with
  true variability on one held-out sample [arxiv:2201.05955 app. A].
- **Standard deviation convention.** The paper names "std" without a
  denominator [arxiv:2201.05955 §2.3]. The card on this wiki notes that the
  released code uses the population form, so compare with the same
  convention.
- **Single runs.** Each setting is one training run [arxiv:2201.05955 app. B].
  Differences of about one or two points need a variance estimate before they
  count as real.
- **Label imbalance after review.** A balanced selection did not give balanced
  labels after human review [arxiv:2201.05955 §5.3]. Plan for class weighting or
  rebalancing when comparing against other sets.
- **Label noise.** On the disagreement cases the paper examined, two authors
  agreed on the label only 49% of the time [arxiv:2201.05955 app. D.4].
- **Generation access.** WANLI's candidate generation used a GPT-3 model that
  was available through an API in 2021. Reproducing that step depends on
  comparable model access [arxiv:2201.05955 app. C.2].

## Papers

- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](/papers/2201.05955): uses training-dynamics variability to pick seed examples and to filter generated NLI examples, and reports the resulting dataset's out-of-domain accuracy.