Rationale extraction (models that show the supporting input)

Overview

A rationale is a short part of the input that supports a model's output. It should be sufficient for the prediction, but it need not list every relevant token. What makes a rationale good depends on what it is for 1911.03429 §1.

Rationale extraction is the task of producing one with the prediction. Two questions decide whether a rationale is worth trusting, and the sources answer them with different measures:

1. Agreement. Does the rationale match the spans a human marked?
2. Faithfulness. Does the prediction depend on the rationale? This is tested by removing the rationale (comprehensiveness) or keeping only it (sufficiency), and measuring the change in the predicted-class probability 1911.03429 §4.2.

SciFact reports agreement only, as F1 over the sentences it selects 2004.14974 §4.

This page covers models built to output a rationale. Scoring an existing model after the fact, with attention or other per-token scores, is covered on Attention interpretability.

Variants

Hard selection (Lei et al. style)

An encoder produces a binary mask over the input. A decoder sees only the unmasked tokens and predicts the label. The two are trained jointly, with REINFORCE-style gradient estimates and a regulariser that pushes the mask towards sparse, contiguous selections. A variant adds a cross-entropy loss on human rationales where they exist 1911.03429 §5. The paper gives a sparsity rate of 0.01 and a contiguity weight of twice the sparsity, with a BERT or GloVe plus BiLSTM encoder 1911.03429 app. C.1. Card: ERASER: A Benchmark to Evaluate Rationalized NLP Models.

Two-stage pipeline

A rationale identifier is trained first. A classifier then sees only the identified rationale. The two stages share no parameters 1911.03429 §5. The simple version splits the input into sentences and encodes each with a GRU and attention. The BERT-to-BERT version uses BERT in both stages. The classifier is trained on the positive rationales, a form of teacher forcing 1911.03429 §5. The authors say this sentence-level design does not suit inputs whose rationales are short spans 1911.03429 §6. Card: ERASER: A Benchmark to Evaluate Rationalized NLP Models.

SciFact's baseline, VeriSci, uses the same three-part shape on scientific claims, and the paper follows the ERASER BERT-to-BERT pipeline for its components 2004.14974 §5. TF-IDF retrieves the top three abstracts. A RoBERTa-large model scores each sentence against the claim and keeps those at or above a threshold. A second RoBERTa-large model predicts the label from the kept sentences and the claim. With no kept sentences, the label is NOT_ENOUGH_INFO 2004.14974 §5 2004.14974 app. A.1. The label model is trained on gold rationales for supporting and refuting pairs, the same teacher-forcing idea as ERASER's 2004.14974 app. A.3. Its rationales are sentences, and abstract-level scoring counts at most three per abstract 2004.14974 §4. Card: Fact or Fiction: Verifying Scientific Claims.

Soft selection with top-k cut-off

The model gives every token a score, and the rationale is the top-k tokens by score. ERASER pools token states with additive attention and applies three scorers to that one model: attention weights, gradients and LIME 1911.03429 §5. The scorers are compared in the section on when to use which, below. For the choice of k, see Contradictions.

Measuring rationales

Agreement with human spans. A predicted span counts as a match when its token-level intersection-over-union with a human rationale is above a threshold, set to 0.5. Matches give an F1 score, and token-level precision and recall give a second F1. Models that score each token can instead be compared with AUPRC, computed by sweeping a threshold over the scores 1911.03429 §4.1.

Sentence and abstract F1 (SciFact). SciFact scores micro-F1 over precision and recall. At abstract level, at most three rationale sentences per abstract count. Sentence-level evaluation has no such limit 2004.14974 §4. Sentence Selection+Label is always lower than Selection-only, because a wrong verdict makes a correct rationale count as wrong. Do not compare those two columns with each other 2004.14974 §4 2004.14974 tab. 4.

Faithfulness. Comprehensiveness is the drop in the predicted-class probability when the rationale is removed. Sufficiency is the drop when only the rationale is kept 1911.03429 §4.2 eq. 1 eq. 2. A high comprehensiveness and a low sufficiency are the desired pattern. ERASER also averages the drops over cut-offs at the top 1, 5, 10, 20 and 50 per cent of tokens, which it calls AOPC 1911.03429 §4.2 eq. 3. Each cut-off adds forward passes, so the cost grows with the number of cut-offs 1911.03429 §4.2.

# Two-stage pipeline at inference  [arxiv:1911.03429 §5]
def pipeline_predict(doc, identifier, classifier):
    # Split the input into sentences
    sentences = split_sentences(doc)
    # Keep the sentences the identifier selects as evidence
    rationale = [s for s in sentences if identifier.select(s)]
    # Classify from the rationale alone
    label = classifier(rationale)
    return label, rationale

# Hard selection at inference  [arxiv:1911.03429 §5] [app. C.1]
def hard_select_predict(tokens, encoder, decoder):
    # Binary mask over input positions, one per token
    z = encoder.sample_mask(tokens)
    # The decoder reads only the unmasked tokens
    label = decoder(mask_tokens(tokens, z))
    # Training adds a sparsity and contiguity penalty on z
    return label, z

When to use which

Results are author-reported. Hard selection and the pipeline are compared on the same seven datasets 1911.03429 tab. 3. IOU F1 is agreement with human rationales, at the 0.5 threshold.

1911.03429 tab. 3. The pipeline has no row for CoS-E or e-SNLI.

Implementation sketch

The pipeline and the hard-selection model differ in how the rationale is produced. Both feed a classifier that sees only the rationale. The soft model differs only in its last step, where the top-k tokens are cut from scores.

# Top-k cut-off from token scores  [arxiv:1911.03429 §4.2]
def top_k_rationale(tokens, scores, k):
    # Indices of the k highest-scoring tokens
    top = argsort_desc(scores)[:k]
    # k is a per-dataset choice; see Contradictions
    return [tokens[i] for i in sorted(top)]

Notes on the released code, from the ERASER card's Paper vs code section: int(k_fraction * len(tokens)) on short documents can give an empty input for the sufficiency check. Guard against this before reporting sufficiency.

For a sentence pipeline, the selection threshold depends on the training data. The SciFact-trained selector uses 0.5. The zero-shot FEVER model uses 0.025, which the paper tuned on the SciFact dev set. Applying one model's threshold to the other is not what the paper did 2004.14974 app. A.1. In the label step, the rationale is truncated first when the input passes 512 tokens, and the claim is never truncated 2004.14974 §5.

Contradictions

Neither card tests the other's setting, and the two sets of numbers are not comparable. Within ERASER, the paper and the released code also differ in these places:

• Soft-score cut-off. The paper sets k to the average human rationale length of each dataset 1911.03429 §4.2. The released configs use a fixed fraction of the document instead, 0.07 for most datasets and 0.013 for Evidence Inference. Results therefore depend on which rule was used.
• Hard selection and BERT rows. The paper reports hard-selection rows and BERT-based soft rows 1911.03429 tab. 3 1911.03429 tab. 4. Reviewing the released repository found no implementation of the hard model, and no BERT path for the soft rows on CoS-E or e-SNLI. These rows cannot be reproduced from that code.
• IOU match rule. The paper says overlap must be more than 0.5 1911.03429 §4.1. The code accepts overlap equal to 0.5 as well.

Across the two papers, there is a tension rather than a direct contradiction. ERASER says sentence-level pipelines are unsuited to short-span rationales 1911.03429 §6. SciFact uses sentence-level rationales and reports a selector at 72.1 dev F1 on whole sentences 2004.14974 tab. 3. Its tasks and rationale units differ, and neither paper tests the other's case. The ERASER metric is token-level IOU, and SciFact's is sentence-level micro-F1 1911.03429 §4.1 2004.14974 §4. Scores from the two should not be ranked against each other.

Papers

• ERASER: A Benchmark to Evaluate Rationalized NLP Models: a benchmark of seven datasets with human-marked rationales, metrics for agreement and faithfulness, and baselines for hard selection, pipelines and soft scorers.
• Fact or Fiction: Verifying Scientific Claims: SciFact, a set of expert-labelled scientific claims with rationale sentences, and VeriSci, a three-stage baseline that retrieves, selects rationales and predicts labels.