Natural language inference

Overview

Natural language inference (NLI) asks whether a premise sentence entails a hypothesis sentence 1803.02324 §1. Large datasets were built by asking crowd workers to write one hypothesis that must be true, one that might be true and one that must be false for each premise 1803.02324 §1.

The two cards on this page agree on one point: the labels are not only a property of the premise and hypothesis pair. The first card argues that the way the hypotheses were written leaves patterns that reveal the label, so a model can score well without comparing the two sentences 1803.02324 §1 §2. The second card treats entailment as one component of a larger verification pipeline, and notes that the pipeline needs retrieval before it can apply 1806.07687 §5. The shared invariant is that an NLI label is only meaningful for the items, label scheme and evidence it was measured on.

Variants

Hypothesis-only baseline as a diagnostic

Train a classifier that sees only the hypothesis, with the premise removed. The main model is fastText, a bag of words and bigrams 1803.02324 §2. Its scores are the diagnostic: on SNLI it reaches 67.0% against 34.3% for the majority class, and on MultiNLI matched and mismatched it reaches 53.9% and 52.3%, against 35.4% and 35.2% 1803.02324 tab. 2. The authors use this as evidence of label cues, not as a competitive NLI system [arxiv:1803.02324 §5]. Card: Annotation Artifacts in Natural Language Inference Data.

The paper names two other premise-free classifiers that gave similar results, a logistic regression on n-grams and a decomposable-attention model with the premise removed 1803.02324 §2 [fn. 2]. It reports no numbers for them.

Cue words and length as label evidence

The same paper ranks words by pointwise mutual information with each class, using add-100 smoothing on the raw counts 1803.02324 §3.1. Negation words such as "no" and "nobody" rank high for contradiction, and words such as "outdoors" rank high for entailment 1803.02324 tab. 4. Length also differs by class in SNLI: neutral hypotheses are longer, and entailed ones are shorter 1803.02324 §3.2. The length effect is weaker in MultiNLI, which the authors attribute to genre mix 1803.02324 §3.2. These are dataset properties, so a new dataset built the same way needs its own check.

Easy and hard test subsets

Split each test set with the hypothesis-only model. Items it gets right are easy, the rest are hard 1803.02324 §4. Three NLI models (DAM, ESIM and DIIN) trained on the original training data score 84.7, 85.8 and 86.5 on the full SNLI test set, and 69.4, 71.3 and 72.7 on the hard part 1803.02324 tab. 5. The authors read this as most of the success coming from easy items 1803.02324 §4. The split uses the gold labels, so the hard subset is not an unbiased training filter 1803.02324 §4.

# Easy/hard split of one NLI test set  [arxiv:1803.02324 §4]
# Input:  test_pairs (premise, hypothesis, label); oblivious model
# Output: easy_pairs, hard_pairs

def split_test_set(test_pairs, oblivious):
    easy, hard = [], []
    for (premise, hypothesis, label) in test_pairs:
        # Guess from the hypothesis alone  [arxiv:1803.02324 §4]
        guess = oblivious.predict(hypothesis)
        # The guess is compared with the gold label  [arxiv:1803.02324 §4]
        if guess == label:
            easy.append((premise, hypothesis, label))
        else:
            hard.append((premise, hypothesis, label))
    return easy, hard

Entailment as a verification step

In fact checking, the premise is a retrieved text and the hypothesis is the claim. The label is for, against or observing, and most teams in the Fake News Challenge used this form 1806.07687 §5. In the FEVER setting the label is supported, refuted or not enough information, and for the first two the system must also return evidence sentences. A correct label with wrong evidence counts as wrong 1806.07687 §3.3. At Wikipedia scale the entailment component sits last, after document retrieval and sentence selection 1806.07687 §5. Card: Automated Fact Checking: Task formulations, methods and future directions. The survey runs no experiments and reports no accuracy, so it gives no numbers for this variant 1806.07687 §4.

When to use which

Implementation sketch

The sketch above covers the split. Two points are not settled by either card. The first card does not say which premise-free model defines its split, since §2 names fastText and two alternatives and §4 says only "the premise-oblivious model" 1803.02324 §2 §4. The second card does not say how to score entailment inside its pipeline, beyond its three stages 1806.07687 §5.

For a faithful baseline, use fastText on the hypothesis alone and state that choice. The NLI models' training settings are not stated in the paper, so results will not match the paper's table unless the original code settings are used 1803.02324 §4.

Contradictions

None known. The two cards study different things: one measures label cues in a crowd-written benchmark, and the other surveys how entailment fits into fact checking. They report no conflicting numbers.

Papers

• Annotation Artifacts in Natural Language Inference Data: a hypothesis-only fastText classifier beats the majority class on SNLI and MultiNLI, and strong NLI models score much lower on the hard subset.
• Automated Fact Checking: Task formulations, methods and future directions: a 2018 survey that places entailment inside the fact-checking pipeline and reports no experiments of its own.