WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

In plain words

Large crowdsourced NLP datasets are written by many people, who tend to fall into a few repeated sentence patterns. Models trained on them look good on the same kind of test data but break on different text. WANLI is a dataset for one such task, natural language inference (deciding whether one sentence follows from another), built with a mix of AI and people instead of people alone.

The method starts from an existing dataset and finds examples that a trained model finds hard to settle. For each of these it gathers similar examples that share the same kind of reasoning, and shows them to a language model, which writes a new example with that pattern. A score based on how much the model's guesses wobble during training picks the more useful new examples. Two human workers then label each one and may lightly fix its wording or throw it away.

The result is a set of about 108,000 examples. Models trained on it do better on several out-of-domain test sets than models trained on the larger source dataset.

TL;DR

• WANLI is an NLI training set of 107,885 labelled examples, generated from MultiNLI seeds by GPT-3 Curie and labelled by crowdworkers §2.
• The pipeline has four stages: exemplar collection, overgeneration, automatic filtering with a new "estimated max variability" score, and human review §2.1-§2.4.
• RoBERTa-large trained on WANLI alone beats RoBERTa-large trained on MultiNLI on every out-of-domain set in the paper's Table 3, and beats MNLI+SNLI+ANLI on most of them §3.3 tab. 3.
• Results are single-run, author-reported, and the paper contains two different WANLI-alone rows that do not match (see Pitfalls).
• The code is public, but it omits the full dataset and the license file.

Problem

Crowdsourced NLI data is often dominated by repeated writing patterns, and models trained on it fail on out-of-domain or adversarial sets §1. Writing diverse examples at scale is hard for human writers, while generative language models can reproduce a pattern from a few in-context examples §1 §2.2. The paper asks whether the two can be combined: the model writes candidate examples, and humans keep and check them §1.

Method

The pipeline has four stages. The prerequisites are an initial dataset D0 (the paper uses MultiNLI) and a task model M, here RoBERTa-large fine-tuned on D0 §2 app. B.

Stage 1, exemplar collection §2.1. Seeds are examples of D0 that M finds most ambiguous. Ambiguity is the variability, across training epochs, of the probability M gives the correct label (the dataset-cartography notion of variability). For each gold label, the most ambiguous 25% are taken as seeds. The telephone genre is excluded §2.1, footnote 1. For each seed, the neighbours are the examples with the same gold label that are closest to it in the space of M's [CLS] representations, measured by cosine similarity. Four neighbours are used, so with the seed there are five in-context examples §2.1. The paper reports 98,176 seeds app. C.3.

Stage 2, overgeneration §2.2 fig. 2. The five in-context examples are arranged in increasing similarity to the seed, so the seed comes last. They are formatted as a numbered list of premise/hypothesis pairs with a per-label instruction and label word ("Implication" for entailment, "Possibility" for neutral, "Contradiction" for contradiction) fig. 2. The language model, GPT-3 Curie, is asked to continue the list. Five samples are drawn per context. The generated examples are unlabelled at this point §2.2.

Stage 3, automatic filtering §2.3 eq. 1. First, heuristics remove generations where either sentence is under five characters, where premise and hypothesis match ignoring case and punctuation, where the pair copies an in-context example, or where it contains a phrase from the instruction §2.3. Second, each remaining example gets an estimated max variability. For every saved epoch of M, the probability of each label is computed. The score is the largest, over the three labels, of the standard deviation of that label's probability across epochs eq. 1. The method assumes that adding the example to the training set would hardly change any checkpoint §2.3. Third, for each intended label, the examples with the highest score are kept, half of the filtered set in total, so the classes stay balanced §2.3.

Stage 4, human review §2.4 app. D. Two Amazon Mechanical Turk workers review each unlabelled example. Each may revise it minimally to keep its intended meaning, or discard it if it needs heavy revision or is possibly offensive. Each also assigns a label. An example is dropped if either worker discards it. A revision is kept only if both workers revised. When both labels differ and neither worker revised, one label is picked uniformly at random. The paper kept disagreements rather than collecting a third label app. D.4.

Split. The labelled data is split at random into 102,885 training and 5,000 test examples tab. 2 §2.4.

Use for training §3. RoBERTa-large is fine-tuned on WANLI, MultiNLI, or combinations of them, and evaluated on OOD challenge sets: NLI Diagnostics, HANS, QNLI, WNLI, NQ-NLI, ANLI, FEVER-NLI and BIG-Bench NLI §3.1. Combinations are either augmentation (concatenation) or random replacement that keeps the size fixed §3.2.

# WANLI construction, stages 1-4. The paper has no algorithm box; this
# summarises §2. Input: D0 = MultiNLI train, M = RoBERTa-large tuned on D0.
# Output: labelled examples (text, label).

def build_wanli(D0, M):
    # Stage 1: seeds are the most ambiguous 25% of each gold label, with
    # the telephone genre removed first  [§2.1] [code: pipeline.py#L54-L62]
    seeds = []
    for label in LABELS:
        pool = [x for x in D0[label] if x.genre != "telephone"]
        seeds += top_fraction(pool, by=variability_under(M), frac=0.25)
    # Stage 1: context = seed + 4 nearest same-label neighbours, using
    # M's normalised [CLS] vectors  [§2.1] [code: pipeline.py#L77-L81]
    context_sets = []
    for seed in seeds:
        neighbours = nearest_same_label(seed, M, k=4)
        context_sets.append(neighbours + [seed])
    # Stage 2: order by increasing similarity so the seed is last, then
    # draw 5 unlabelled continuations per context  [§2.2] [app. C.2]
    generations = []
    for context in context_sets:
        prompt = format_prompt(context, label=context.seed.label)
        for _ in range(5):
            generations.append(lm.sample(prompt, top_p=0.5, max_tokens=120))
    # Stage 3a: heuristic filter  [§2.3] [code: filtering/filter.py#L55-L60]
    kept = [g for g in generations if passes_heuristics(g)]
    # Stage 3b: estimated max variability = largest over labels of the
    # std across M's epochs of p(label | x)  [eq. 1]
    for example in kept:
        probs = epoch_probabilities(M, example)   # probs[epoch][label]
        example.sigma = max(
            std([probs[e][y] for e in range(num_epochs(M))])
            for y in LABELS
        )
    # Stage 3c: per intended label keep the highest sigma; total kept is
    # half of the filtered set  [§2.3]
    # [code: filtering/keep_ambiguous.py#L23-L24]
    per_label = len(kept) // 2 // 3
    selected = []
    for label in LABELS:
        selected += top_by_sigma(of_label(kept, label), per_label)
    # Stage 4: two workers label each example and may revise or discard
    # it  [§2.4] [app. D]
    labelled = []
    for example in selected:
        first, second = two_annotations(example)
        if first.discarded or second.discarded:
            continue
        # keep a revision only when both workers revised the example  [§2.4]
        if first.revised and second.revised:
            text = random_choice([first.text, second.text])
        else:
            text = example.original_text
        # labels that differ with neither revision: pick one at random  [§2.4]
        if first.label == second.label or first.revised or second.revised:
            label = first.label
        else:
            label = random_choice([first.label, second.label])
        labelled.append((text, label))
    # Random split: 5,000 test examples, the rest train  [tab. 2] [§2.4]
    test = random_sample(labelled, 5000)
    train = [x for x in labelled if x not in test]
    return train, test

Note on the code above: the paper's Stage 4 wording and the repository differ in the revision rule; see Paper vs code.

Hyperparameters

Every value below is stated in the paper unless marked as code-only.

Dataset statistics

Reproduced from tab. 2:

Labels used in the prompt

The label words in the prompt are "Implication" (entailment), "Possibility" (neutral) and "Contradiction" fig. 2 context_formats.py. The instruction text is "Write a pair of sentences that have the same relationship as the previous examples." context_formats.py.

Results

Author-reported, single run per setting. Setup: RoBERTa-large fine-tuned for 5 epochs on each training set, evaluated on each challenge set §3.1 app. B. Gray cells in the paper mark in-distribution or non-OOD settings and are marked † here. Bold marks the best value in the column, as in the paper.

Table 3: RoBERTa-large trained on each set, OOD test accuracy tab. 3

* two-label sets (entailment vs non-entailment), per the paper tab. 3. Sizes in column "Size" are the training-set sizes. The test set size row is copied from the paper.

Table 4: adding WANLI to ANLI training data tab. 4

Table 8: MultiNLI development set app. E.2 tab. 8

Table 10: additional baselines, same 103K size as WANLI app. E.1 tab. 10

Table 11: T5-base, OOD test accuracy app. E.3 tab. 11

Headline numbers from the text §3.3: against MultiNLI alone, the WANLI-trained model is 4% higher on Diagnostics (72.73 vs 68.47), 11% higher on HANS (89.28 vs 78.08) and 9% higher on Adversarial NLI (41.12 vs 32.37). The paper says the WANLI-only model beats MNLI+SNLI+ANLI (943K) on every OOD set in Table 3, and also beats MNLI+WANLI §3.3. In Table 3 this holds on the non-gray cells, where the combination was not trained on the test set.

Dataset-level analyses (author-reported): about 15% of labelled examples are contradictions, so WANLI is not label-balanced §5.3. Partial-input classifiers reach 41.6% accuracy from hypotheses alone on a balanced WANLI subset versus 49.6% for MultiNLI at the same size, and 42.9% premise-only on WANLI §4.1 fig. 3 fig. 4. Among workers' revisions, 74% of premise and hypothesis revisions change word count by between -1 and +2 words §5.1. Location mentions in WANLI fall 79% in Europe or North America, versus 71% for MultiNLI; the United States is 46% of WANLI mentions versus 26% for MultiNLI app. C.4.

Paper vs code

Code: https://github.com/alisawuffles/wanli at commit 17190b51baeeea50314e1320563d87a64c630c14, owned by the first author. The repository has no LICENSE file, so code_license is recorded as not stated. The paper points to this code by a project page at wanli.allenai.org §1; that page could not be reached during compilation. A third-party reimplementation (SungHo3268/WANLI) was not used.

Checked and matching:

• Training hyperparameters (batch 32, learning rate 1e-5, 5 epochs, warmup 0.06, weight decay 0.1) match the paper's table finetune_roberta.sh.
• Generation settings: top-p 0.5, max 120 tokens, stop at blank line, engine Curie pipeline.py. Temperature 1.0 and both penalties at 0.0 are the wrapper defaults gpt3_generation.py.
• Telephone genre excluded from seeds pipeline.py.
• Seed ambiguity is the per-label 0.75 quantile of variability, strictly above the threshold pipeline.py.
• Context order: neighbours are reversed so the nearest is last, i.e. the seed comes last context_formats.py.
• Heuristic checks (length under 5, copy of premise, copy of an in-context example, instruction phrases) filter.py.
• Max variability formula compute_training_dynamics.py.
• Filter keeps half the filtered set with equal per-label counts keep_ambiguous.py keep_ambiguous.py.
• Test set of 5,000 examples split_train_test.py.

Differences:

• Revision rule. The paper keeps a revision only when both workers revised it §2.4. The code also keeps a single worker's revision when that revision changes only punctuation or casing process_annotations_into_wanli.py utils.py. When neither revised, it keeps one annotation after a random shuffle process_annotations_into_wanli.py.
• Neighbour retrieval. The paper says the four nearest same-label neighbours §2.1. The code queries the 15 nearest examples over all labels, then keeps the first five of the requested label, which includes the seed itself. It skips the seed if fewer than five remain pipeline.py. The paper does not mention this skip.
• Telephone exclusion for neighbours. The code also drops telephone examples from neighbour lists pipeline.py. The paper excludes the genre only for seeds §2.1.
• Sample count. The paper says five samples per context. The code makes five separate API calls per context and drops any generation without the expected label separator pipeline.py.
• Text cleaning. The code strips surrounding quotes, parentheses and dangling punctuation from accepted premise and hypothesis text filter.py. The paper does not describe this step.
• Forbidden phrases. The paper gives one example phrase §2.3. The code uses four: "pair of sentences", "previous examples", "same relationship as", and "The speaker" filter.py.
• Standard deviation. The paper uses "std" without saying which denominator. The code uses NumPy's default, which is the population form (ddof = 0) compute_training_dynamics.py.
• Examples missing epochs. The code silently drops examples that lack a score for every epoch compute_training_dynamics.py.
• Epoch definition. The code counts every checkpoint-*/pytorch_model.bin directory as one epoch compute_training_dynamics.py. The number of saved checkpoints for M is not stated in the paper.
• Model M. The README says M is trained with classification/run_nli.py [README.md step 1]. The exact M run command is not in scripts/, so M's hyperparameters rest on the paper's appendix alone.
• Full data. The repository holds only small sample files under data/wanli/. The README points to allenai.org for the full dataset. This card did not verify that download.
• Reproducibility of the split. The 5,000-example test split is drawn with no seed split_train_test.py.

Pitfalls

• Two different WANLI-alone rows. The paper reports the same WANLI-alone training set in two tables with different numbers. Table 3 gives QNLI 81.40, HANS 89.28, BIG-Bench 85.19, and WNLI 67.28 tab. 3. Table 10 gives QNLI 76.81, HANS 89.40, BIG-Bench 75.40, and WNLI 65.15 tab. 10. The text does not explain the gap, so use per-set numbers with caution until a rerun confirms them.
• Single run, no variance. The paper reports one training run per setting and no seed variance app. B. The paper's in-domain ANLI gain from adding WANLI is 1.4 points §3.3 tab. 4, which a single run cannot put a confidence interval on.
• Model-dependent seeds. Seeds, neighbours and the variability score all depend on one specific M, so changing M changes the dataset. The paper reports a Pearson correlation of 0.527 between estimated max variability and true variability on a held-out MNLI sample app. A.
• Generation access. The original generation used the OpenAI API for GPT-3 Curie in November 2021, at USD 677.89 app. C.2. Reproducing it requires a comparable left-to-right model and access to it; the paper's model identity and version are the reproducibility limit. Cost is also a planning number.
• Label imbalance. The dataset has 38,511 entailment, 48,977 neutral and 15,397 contradiction training examples tab. 2. A balanced seed set did not yield balanced labels after human review §5.3. Class weighting or rebalancing may be needed for comparisons.
• Label noise is real. On 80 disagreement cases, two authors agreed on the label 49% of the time, and in 12% all three labels appeared among four annotations app. D.4. The dataset keeps one label per example chosen at random §2.4.
• Revisions are small. Only 3.54% of labelled examples were revised §2.4. Most of the dataset is GPT-3 text as written.
• Estimated max variability needs checkpoints. Stage 3 requires saved checkpoints from M's training (one per epoch). Without them the score cannot be computed as described §2.3 compute_training_dynamics.py.
• Western over-representation. 79% of location mentions are in Europe or North America, and the United States alone is 46% app. C.4. The paper calls this a concern for generated datasets.
• Scope. English only, one task (NLI), and no ablation of pipeline parts because each variant needs a full annotation run §10.
• Instruction and label words are part of the data. Changing the prompt wording or the label words ("Implication", "Possibility", "Contradiction") changes the generations fig. 2 context_formats.py.

Lineage

• Builds on: dataset cartography (Swayamdipta et al., 2020), cited for the ambiguity-based seed selection §2 §2.1. Not compiled in this wiki.
• Builds on: in-context pattern replication with GPT-3 (Brown et al., 2020) §2.2.
• Builds on: data-map-derived example selection, AFLite and related debiasing filters, used as baselines app. E.1.
• Compared against: MultiNLI and its combinations with SNLI, ANLI, FEVER-NLI, Tailor and Z-Aug §3.2 tab. 3.
• Superseded by: not stated in this card's sources.