---
kind: concept
slug: natural-language-inference
title: Natural language inference
summary: >-
  Deciding whether a premise supports, contradicts or leaves open a hypothesis, and how that task is built and checked. Two cards so far: a hypothesis-only baseline that exposes label cues in SNLI and MultiNLI, and the fact-checking survey's view of entailment as a verification step. Both are reports, not new systems.
tags: [natural-language-inference, dataset-artifacts, fact-checking, benchmark-evaluation]
papers: ["1803.02324", "1806.07687"]
related: [benchmark-validity, challenge-set-evaluation, automated-fact-checking, text-classification, adversarial-evaluation]
status: draft
updated_at: 2026-11-01
---

## Overview

Natural language inference (NLI) asks whether a premise sentence entails a
hypothesis sentence [arxiv:1803.02324 §1]. Large datasets were built by
asking crowd workers to write one hypothesis that must be true, one that might
be true and one that must be false for each premise [arxiv:1803.02324 §1].

The two cards on this page agree on one point: the labels are not only a
property of the premise and hypothesis pair. The first card argues that the
way the hypotheses were written leaves patterns that reveal the label, so a
model can score well without comparing the two sentences [arxiv:1803.02324 §1]
[§2]. The second card treats entailment as one component of a larger
verification pipeline, and notes that the pipeline needs retrieval before it
can apply [arxiv:1806.07687 §5]. The shared invariant is that an NLI label is
only meaningful for the items, label scheme and evidence it was measured on.

## Variants

### Hypothesis-only baseline as a diagnostic

Train a classifier that sees only the hypothesis, with the premise removed.
The main model is fastText, a bag of words and bigrams [arxiv:1803.02324 §2].
Its scores are the diagnostic: on SNLI it reaches 67.0% against 34.3% for the
majority class, and on MultiNLI matched and mismatched it reaches 53.9% and
52.3%, against 35.4% and 35.2% [arxiv:1803.02324 tab. 2]. The authors use this
as evidence of label cues, not as a competitive NLI system [arxiv:1803.02324
§5]. Card: [Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324).

The paper names two other premise-free classifiers that gave similar results,
a logistic regression on n-grams and a decomposable-attention model with the
premise removed [arxiv:1803.02324 §2] [fn. 2]. It reports no numbers for them.

### Cue words and length as label evidence

The same paper ranks words by pointwise mutual information with each class,
using add-100 smoothing on the raw counts [arxiv:1803.02324 §3.1]. Negation
words such as "no" and "nobody" rank high for contradiction, and words such as
"outdoors" rank high for entailment [arxiv:1803.02324 tab. 4]. Length also
differs by class in SNLI: neutral hypotheses are longer, and entailed ones are
shorter [arxiv:1803.02324 §3.2]. The length effect is weaker in MultiNLI,
which the authors attribute to genre mix [arxiv:1803.02324 §3.2]. These are
dataset properties, so a new dataset built the same way needs its own check.

### Easy and hard test subsets

Split each test set with the hypothesis-only model. Items it gets right are
easy, the rest are hard [arxiv:1803.02324 §4]. Three NLI models (DAM, ESIM and
DIIN) trained on the original training data score 84.7, 85.8 and 86.5 on the
full SNLI test set, and 69.4, 71.3 and 72.7 on the hard part
[arxiv:1803.02324 tab. 5]. The authors read this as most of the success coming
from easy items [arxiv:1803.02324 §4]. The split uses the gold labels, so the
hard subset is not an unbiased training filter [arxiv:1803.02324 §4].

```pseudocode
# Easy/hard split of one NLI test set  [arxiv:1803.02324 §4]
# Input:  test_pairs (premise, hypothesis, label); oblivious model
# Output: easy_pairs, hard_pairs

def split_test_set(test_pairs, oblivious):
    easy, hard = [], []
    for (premise, hypothesis, label) in test_pairs:
        # Guess from the hypothesis alone  [arxiv:1803.02324 §4]
        guess = oblivious.predict(hypothesis)
        # The guess is compared with the gold label  [arxiv:1803.02324 §4]
        if guess == label:
            easy.append((premise, hypothesis, label))
        else:
            hard.append((premise, hypothesis, label))
    return easy, hard
```

### Entailment as a verification step

In fact checking, the premise is a retrieved text and the hypothesis is the
claim. The label is for, against or observing, and most teams in the Fake News
Challenge used this form [arxiv:1806.07687 §5]. In the FEVER setting the label
is supported, refuted or not enough information, and for the first two the
system must also return evidence sentences. A correct label with wrong evidence
counts as wrong [arxiv:1806.07687 §3.3]. At Wikipedia scale the entailment
component sits last, after document retrieval and sentence selection
[arxiv:1806.07687 §5]. Card: [Automated Fact Checking: Task formulations, methods and future directions](/papers/1806.07687).
The survey runs no experiments and reports no accuracy, so it gives no
numbers for this variant [arxiv:1806.07687 §4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You have an NLI-style labelled set and want to know whether it can be solved from one side of the pair | Run the hypothesis-only baseline first | If it beats the majority class by a wide margin, labels can be partly guessed from the hypothesis [arxiv:1803.02324 tab. 2] |
| You want a hard evaluation subset from an existing NLI set | Easy/hard split by a cue-only model, reported with the full score | Reveals where models depend on easy items [arxiv:1803.02324 tab. 5]. Do not use it as a training filter without accounting for the gold labels [arxiv:1803.02324 §4] |
| You plan to filter or rebalance a dataset using the split | Balance cues across labels instead of only dropping easy items | Dropping easy items can move a cue such as "no" into other classes [arxiv:1803.02324 §4] |
| Claims must be checked against a text corpus with a three-way verdict | Retrieve, select sentences, then entail | The survey's Wikipedia pipeline and its label scheme [arxiv:1806.07687 §3.3] [§5] |
| Verdict and evidence are both needed | Score label and evidence together | The survey says a label with wrong evidence counts as wrong [arxiv:1806.07687 §3.3] |
| Claims repeat or paraphrase earlier fact checks | Match to checked claims instead of entailment | The survey says this covers only repeated or paraphrased claims [arxiv:1806.07687 §5] |
| You need NLI training data for a new domain | Check for artifacts before training | The survey says most surveyed datasets are small or tied to one source [arxiv:1806.07687 §4]. The NLI paper shows crowd-written hypotheses carry label cues [arxiv:1803.02324 §3] |

## Implementation sketch

The sketch above covers the split. Two points are not settled by either
card. The first card does not say which premise-free model defines its split,
since §2 names fastText and two alternatives and §4 says only "the
premise-oblivious model" [arxiv:1803.02324 §2] [§4]. The second card does not
say how to score entailment inside its pipeline, beyond its three stages
[arxiv:1806.07687 §5].

For a faithful baseline, use fastText on the hypothesis alone and state that
choice. The NLI models' training settings are not stated in the paper, so
results will not match the paper's table unless the original code settings are
used [arxiv:1803.02324 §4].

## Contradictions

None known. The two cards study different things: one measures label cues in a
crowd-written benchmark, and the other surveys how entailment fits into fact
checking. They report no conflicting numbers.

## Papers

- [Annotation Artifacts in Natural Language Inference Data](/papers/1803.02324): a hypothesis-only fastText classifier beats the majority class on SNLI and MultiNLI, and strong NLI models score much lower on the hard subset.
- [Automated Fact Checking: Task formulations, methods and future directions](/papers/1806.07687): a 2018 survey that places entailment inside the fact-checking pipeline and reports no experiments of its own.