--- kind: concept slug: rationale-extraction title: Rationale extraction (models that show the supporting input) summary: >- Making an NLP model return a short input span, a rationale, with each prediction. Designs: hard selection, a two-stage pipeline, and top-k soft scores. ERASER finds a BERT pipeline matches human sentence rationales best, but sentence pipelines are unsuited to short spans. LIME or gradient scores are more comprehensive than attention. SciFact applies a pipeline to scientific claims. tags: [interpretability, rationales, faithfulness, nlp, evaluation, fact-checking] papers: ["1911.03429", "2004.14974"] related: [attention-interpretability, explanation-evaluation, automated-fact-checking] status: draft updated_at: 2026-10-19 --- ## Overview A rationale is a short part of the input that supports a model's output. It should be sufficient for the prediction, but it need not list every relevant token. What makes a rationale good depends on what it is for [arxiv:1911.03429 §1]. Rationale extraction is the task of producing one with the prediction. Two questions decide whether a rationale is worth trusting, and the sources answer them with different measures: 1. **Agreement.** Does the rationale match the spans a human marked? 2. **Faithfulness.** Does the prediction depend on the rationale? This is tested by removing the rationale (comprehensiveness) or keeping only it (sufficiency), and measuring the change in the predicted-class probability [arxiv:1911.03429 §4.2]. SciFact reports agreement only, as F1 over the sentences it selects [arxiv:2004.14974 §4]. This page covers models built to output a rationale. Scoring an existing model after the fact, with attention or other per-token scores, is covered on [Attention interpretability](/concepts/attention-interpretability). ## Variants ### Hard selection (Lei et al. style) An encoder produces a binary mask over the input. A decoder sees only the unmasked tokens and predicts the label. The two are trained jointly, with REINFORCE-style gradient estimates and a regulariser that pushes the mask towards sparse, contiguous selections. A variant adds a cross-entropy loss on human rationales where they exist [arxiv:1911.03429 §5]. The paper gives a sparsity rate of 0.01 and a contiguity weight of twice the sparsity, with a BERT or GloVe plus BiLSTM encoder [arxiv:1911.03429 app. C.1]. Card: [ERASER: A Benchmark to Evaluate Rationalized NLP Models](/papers/1911.03429). ### Two-stage pipeline A rationale identifier is trained first. A classifier then sees only the identified rationale. The two stages share no parameters [arxiv:1911.03429 §5]. The simple version splits the input into sentences and encodes each with a GRU and attention. The BERT-to-BERT version uses BERT in both stages. The classifier is trained on the positive rationales, a form of teacher forcing [arxiv:1911.03429 §5]. The authors say this sentence-level design does not suit inputs whose rationales are short spans [arxiv:1911.03429 §6]. Card: [ERASER: A Benchmark to Evaluate Rationalized NLP Models](/papers/1911.03429). SciFact's baseline, VeriSci, uses the same three-part shape on scientific claims, and the paper follows the ERASER BERT-to-BERT pipeline for its components [arxiv:2004.14974 §5]. TF-IDF retrieves the top three abstracts. A RoBERTa-large model scores each sentence against the claim and keeps those at or above a threshold. A second RoBERTa-large model predicts the label from the kept sentences and the claim. With no kept sentences, the label is NOT_ENOUGH_INFO [arxiv:2004.14974 §5] [arxiv:2004.14974 app. A.1]. The label model is trained on gold rationales for supporting and refuting pairs, the same teacher-forcing idea as ERASER's [arxiv:2004.14974 app. A.3]. Its rationales are sentences, and abstract-level scoring counts at most three per abstract [arxiv:2004.14974 §4]. Card: [Fact or Fiction: Verifying Scientific Claims](/papers/2004.14974). ### Soft selection with top-k cut-off The model gives every token a score, and the rationale is the top-k tokens by score. ERASER pools token states with additive attention and applies three scorers to that one model: attention weights, gradients and LIME [arxiv:1911.03429 §5]. The scorers are compared in the section on when to use which, below. For the choice of k, see Contradictions. ## Measuring rationales **Agreement with human spans.** A predicted span counts as a match when its token-level intersection-over-union with a human rationale is above a threshold, set to 0.5. Matches give an F1 score, and token-level precision and recall give a second F1. Models that score each token can instead be compared with AUPRC, computed by sweeping a threshold over the scores [arxiv:1911.03429 §4.1]. **Sentence and abstract F1 (SciFact).** SciFact scores micro-F1 over precision and recall. At abstract level, at most three rationale sentences per abstract count. Sentence-level evaluation has no such limit [arxiv:2004.14974 §4]. Sentence Selection+Label is always lower than Selection-only, because a wrong verdict makes a correct rationale count as wrong. Do not compare those two columns with each other [arxiv:2004.14974 §4] [arxiv:2004.14974 tab. 4]. **Faithfulness.** Comprehensiveness is the drop in the predicted-class probability when the rationale is removed. Sufficiency is the drop when only the rationale is kept [arxiv:1911.03429 §4.2] [eq. 1] [eq. 2]. A high comprehensiveness and a low sufficiency are the desired pattern. ERASER also averages the drops over cut-offs at the top 1, 5, 10, 20 and 50 per cent of tokens, which it calls AOPC [arxiv:1911.03429 §4.2] [eq. 3]. Each cut-off adds forward passes, so the cost grows with the number of cut-offs [arxiv:1911.03429 §4.2]. ```pseudocode # Two-stage pipeline at inference [arxiv:1911.03429 §5] def pipeline_predict(doc, identifier, classifier): # Split the input into sentences sentences = split_sentences(doc) # Keep the sentences the identifier selects as evidence rationale = [s for s in sentences if identifier.select(s)] # Classify from the rationale alone label = classifier(rationale) return label, rationale ``` ```pseudocode # Hard selection at inference [arxiv:1911.03429 §5] [app. C.1] def hard_select_predict(tokens, encoder, decoder): # Binary mask over input positions, one per token z = encoder.sample_mask(tokens) # The decoder reads only the unmasked tokens label = decoder(mask_tokens(tokens, z)) # Training adds a sparsity and contiguity penalty on z return label, z ``` ## When to use which Results are author-reported. Hard selection and the pipeline are compared on the same seven datasets [arxiv:1911.03429 tab. 3]. IOU F1 is agreement with human rationales, at the 0.5 threshold. | Dataset | Lei et al. (hard) | Lehman et al. (pipeline) | BERT-to-BERT (pipeline) | |---|---|---|---| | Evidence Inference | 0.000 | 0.119 | 0.455 | | BoolQ | 0.000 | 0.050 | 0.052 | | Movie Reviews | 0.124 | 0.063 | 0.075 | | FEVER | 0.218 | 0.540 | 0.835 | | MultiRC | 0.271 | 0.136 | 0.416 | | CoS-E | 0.255 | no row | 0.389 | | e-SNLI | 0.693 | no row | 0.704 | [arxiv:1911.03429 tab. 3]. The pipeline has no row for CoS-E or e-SNLI. | Situation | Use | Why | |---|---|---| | Rationales are short spans, such as a few words | Hard selection or token scores, not a sentence pipeline | The sentence-level pipeline is not suited to short spans, and the authors report no pipeline row for CoS-E or e-SNLI [arxiv:1911.03429 §6] | | Rationales are whole sentences, and agreement with human spans matters most | BERT-to-BERT pipeline | It had the highest IOU F1 in six of seven datasets [arxiv:1911.03429 tab. 3]. It had the lowest task accuracy on CoS-E, 0.344 against 0.477 for hard selection [arxiv:1911.03429 tab. 3] | | Scientific claims with about 800 labelled training claims, and a large related fact-checking dataset is available | Pre-train the pipeline on FEVER, then fine-tune on the target data | On dev, FEVER plus SciFact reached 81.9 label accuracy against 75.7 for SciFact alone, but rationale F1 fell from 72.1 to 69.7. FEVER alone reached only 48.4 rationale F1 [arxiv:2004.14974 tab. 3] [arxiv:2004.14974 §5] | | Scientific text, and the rationale selector is the weak step | A domain encoder such as SciBERT in the selector, RoBERTa-large in the label model | On dev, SciBERT reached 74.4 rationale F1 against 72.1 for RoBERTa-large, but 69.2 label accuracy against 75.7. The 2.3-point gap is within the 2 to 4 points of test-set bootstrap noise the paper reports [arxiv:2004.14974 tab. 3] [arxiv:2004.14974 app. B] | | Inputs longer than BERT's limit | GloVe-based soft model, not BERT | BERT-based models could not run on the longest inputs, so the soft models use GloVe on Evidence Inference and BoolQ [arxiv:1911.03429 §5] [arxiv:1911.03429 §6] | | No human rationales, only instance labels | Hard selection with labels only, expecting weak rationales | Task performance stayed close to the supervised version, but IOU F1 fell to 0.000 on FEVER, MultiRC and CoS-E, with degenerate training marked on the last two [arxiv:1911.03429 tab. 3] | | Faithfulness is the priority, and the scorer can be run on a trained model | LIME or gradient scores, checked with comprehensiveness | LIME gave the highest comprehensiveness in six of seven datasets, and the authors say it does particularly well on faithfulness [arxiv:1911.03429 §6] [arxiv:1911.03429 tab. 4] | | Agreement with human spans is the priority | Attention scores | Attention had the highest AUPRC in five of seven datasets, though it was least comprehensive [arxiv:1911.03429 tab. 4] | ## Implementation sketch The pipeline and the hard-selection model differ in how the rationale is produced. Both feed a classifier that sees only the rationale. The soft model differs only in its last step, where the top-k tokens are cut from scores. ```pseudocode # Top-k cut-off from token scores [arxiv:1911.03429 §4.2] def top_k_rationale(tokens, scores, k): # Indices of the k highest-scoring tokens top = argsort_desc(scores)[:k] # k is a per-dataset choice; see Contradictions return [tokens[i] for i in sorted(top)] ``` Notes on the released code, from the ERASER card's Paper vs code section: `int(k_fraction * len(tokens))` on short documents can give an empty input for the sufficiency check. Guard against this before reporting sufficiency. For a sentence pipeline, the selection threshold depends on the training data. The SciFact-trained selector uses 0.5. The zero-shot FEVER model uses 0.025, which the paper tuned on the SciFact dev set. Applying one model's threshold to the other is not what the paper did [arxiv:2004.14974 app. A.1]. In the label step, the rationale is truncated first when the input passes 512 tokens, and the claim is never truncated [arxiv:2004.14974 §5]. ## Contradictions Neither card tests the other's setting, and the two sets of numbers are not comparable. Within ERASER, the paper and the released code also differ in these places: - **Soft-score cut-off.** The paper sets k to the average human rationale length of each dataset [arxiv:1911.03429 §4.2]. The released configs use a fixed fraction of the document instead, 0.07 for most datasets and 0.013 for Evidence Inference. Results therefore depend on which rule was used. - **Hard selection and BERT rows.** The paper reports hard-selection rows and BERT-based soft rows [arxiv:1911.03429 tab. 3] [arxiv:1911.03429 tab. 4]. Reviewing the released repository found no implementation of the hard model, and no BERT path for the soft rows on CoS-E or e-SNLI. These rows cannot be reproduced from that code. - **IOU match rule.** The paper says overlap must be more than 0.5 [arxiv:1911.03429 §4.1]. The code accepts overlap equal to 0.5 as well. Across the two papers, there is a tension rather than a direct contradiction. ERASER says sentence-level pipelines are unsuited to short-span rationales [arxiv:1911.03429 §6]. SciFact uses sentence-level rationales and reports a selector at 72.1 dev F1 on whole sentences [arxiv:2004.14974 tab. 3]. Its tasks and rationale units differ, and neither paper tests the other's case. The ERASER metric is token-level IOU, and SciFact's is sentence-level micro-F1 [arxiv:1911.03429 §4.1] [arxiv:2004.14974 §4]. Scores from the two should not be ranked against each other. ## Papers - [ERASER: A Benchmark to Evaluate Rationalized NLP Models](/papers/1911.03429): a benchmark of seven datasets with human-marked rationales, metrics for agreement and faithfulness, and baselines for hard selection, pipelines and soft scorers. - [Fact or Fiction: Verifying Scientific Claims](/papers/2004.14974): SciFact, a set of expert-labelled scientific claims with rationale sentences, and VeriSci, a three-stage baseline that retrieves, selects rationales and predicts labels.