ERASER: A Benchmark to Evaluate Rationalized NLP Models

In plain words

Many NLP models give an answer without showing why. A rationale is a short piece of the input, such as a sentence or a few words, that supports the answer. Several research groups had built such models, but each tested them on different data with different scores, so the results could not be compared.

ERASER collects seven existing text tasks, such as movie sentiment, fact checking and reading comprehension, in one format. Human annotators have marked the supporting snippets for each example. The paper adds two kinds of score for a model's rationales. The first asks whether the model's snippets match the human ones. The second asks whether the snippets actually drove the prediction: removing them should change the answer, and keeping only them should be enough to reach it.

The authors also train a few baseline models and report their scores. They do not claim that the chosen scores are the right way to judge explanations; they release the benchmark so that others can compare methods on the same terms.

TL;DR

ERASER is a benchmark for NLP models that justify their predictions with rationales. It has seven datasets with human-marked snippets, a unified format, and metrics for agreement with humans and for faithfulness. The authors describe their baselines as a starting point, not a comprehensive suite §5, and say no single off-the-shelf architecture adapts readily to inputs of very different lengths §1. Results are author-reported. Two baseline families in the paper (gradient and LIME scorers, the Lei-style hard model) are not fully reproducible from the released code, as detailed in Paper vs code.

Problem

Models whose predictions are hard to explain are hard to trust. Work on rationale-producing NLP models existed, but each paper used its own datasets and metrics, which made progress hard to track §1. The authors frame the target as rationales, meaning snippets that support an output. A rationale should be sufficient for the prediction but need not be comprehensive, and the quality of a rationale depends on its intended use §1. Because the metrics were not settled, the authors present them as a starting point and do not privilege any single one §1.

Dataset sizes and lengths matter. Document lengths in the benchmark range from about 16 tokens (e-SNLI) to about 4,761 tokens (Evidence Inference) on average tab. 1. The authors found that no single off-the-shelf architecture adapts readily across that range §1.

Datasets §3 tab. 1 tab. 2 app. A app. B

The "Comp?" legend is in the tab. 1 caption. For the datasets with only a subset of comprehensive test rationales, recall measures comprehensiveness only partly §4.1. The four datasets marked "more or less true by default" carry the same caveat.

Each dataset is a different task:

• Evidence Inference: full-text randomized controlled trial reports; predict whether an intervention significantly increases, decreases or has no significant effect on an outcome, relative to a comparator §3. Comprehensive rationales were collected from medical professionals app. B.
• BoolQ: yes/no questions over Wikipedia passages. The source articles were not retained, so the authors recovered them with fuzzy matching app. A.
• Movie Reviews: positive/negative sentiment; rationales are spans, not sentences §3.
• FEVER: claims labelled supported or refuted. The "not enough information" class and claims supported by more than one document were removed app. A.
• MultiRC: each rationale/question/answer triplet becomes one True/False instance §3.
• CoS-E: five-way multiple choice; only the highlighted extractive rationales are used, and the free-text explanations are excluded §3 app. A.
• e-SNLI: SNLI pairs with highlighted words in the premise and/or hypothesis §3.

Human agreement for the rationales is reported as Cohen's kappa and F1 per dataset tab. 2. The authors state that every dataset with agreement figures has substantial or better agreement §3.

Method

ERASER has two parts: the datasets, and metrics for rationales §1. Models are scored on task performance with the usual task metrics, and on rationales with the metrics below §4.

Agreement with human rationales §4.1

• Discrete rationales. A predicted span counts as a match when its token-level intersection-over-union with a human rationale is above a threshold, set to 0.5 §4.1. Matched predictions give an F1 score. Token-level precision and recall give a token-level F1 §4.1.
• Soft scores. For models that assign a score to each token, the Area Under the Precision-Recall curve (AUPRC) is computed by sweeping a threshold over the scores §4.1.

# Token-level intersection-over-union of two spans  [§4.1]
def span_iou(predicted, gold):
    # Token sets: predicted and gold are sets of token positions
    overlap = len(predicted & gold)
    # Union of the two token sets
    union = len(predicted | gold)
    return overlap / union

Faithfulness §4.2 eq. 1 eq. 2 eq. 3 fig. 2

A rationale is faithful if the model's prediction actually depends on it. The authors use two contrast measures, following earlier work §4.2. Write p_j(x) for the model's probability of its predicted class j on input x.

• Comprehensiveness eq. 1: the drop in p_j when the rationale is removed from the input. A high value means the rationale mattered. A negative value means the model became more confident without it §4.2.
• Sufficiency eq. 2: the drop in p_j when only the rationale is kept. A low value means the rationale alone is enough for the same prediction §4.2.

# Faithfulness of a discrete rationale for predicted class j  [§4.2]
def faithfulness(model, tokens, rationale, j):
    # Predicted-class probability on the full input  [§4.2]
    p_full = model(tokens)[j]
    # Contrast input with the rationale removed  [§4.2] [eq. 1]
    p_without = model(remove(tokens, rationale))[j]
    # Input that keeps only the rationale  [§4.2] [eq. 2]
    p_only = model(keep(tokens, rationale))[j]
    # Higher is better: the rationale was influential  [§4.2] [eq. 1]
    comprehensiveness = p_full - p_without
    # Lower is better: the rationale alone is sufficient  [§4.2] [eq. 2]
    sufficiency = p_full - p_only
    return comprehensiveness, sufficiency

Soft scores need a cut-off. To compare soft token scores with the discrete measures above, the authors keep the top k_d tokens by score, where k_d is the average length of the human rationales for the dataset §4.2. The paper points to tab. 4 for these values, but that table does not list them; the average evidence lengths appear in the appendix counts tab. 5.

# Top-k tokens by soft score, used as a discrete rationale  [§4.2]
def top_k_tokens(scores, k):
    # Indices of the k highest-scoring tokens
    return argsort_descending(scores)[:k]

AOPC. The authors also average the drop over several cut-offs, called an area over the perturbation curve (AOPC). Tokens are grouped into the top 1, 5, 10, 20 and 50 per cent by score, and the drops are averaged §4.2 eq. 3. The sum in eq. 3 runs from k = 0 to |B| and is divided by |B| + 1; the authors use five bins §4.2.

# Area over the perturbation curve  [§4.2] [eq. 3]
def aopc(model, tokens, scores, j, fractions):
    # fractions = [0.01, 0.05, 0.1, 0.2, 0.5]  [§4.2]
    p_full = model(tokens)[j]
    deltas = []
    for fraction in fractions:
        # Tokens up to and including this bin  [§4.2]
        removed = top_k_tokens(scores, int(fraction * len(tokens)))
        p_removed = model(remove(tokens, removed))[j]
        deltas.append(p_full - p_removed)
    # The released code takes the plain mean over the bins
    # [code: rationale_benchmark/metrics.py#L266-L268]
    return mean(deltas)

For reference, the authors also report a random ordering of tokens, averaged over 10 runs §4.2 tab. 4.

Baseline models §5

• Hard selection, Lei et al. (2016) §5. An encoder produces a binary mask z over the input. A decoder sees only the unmasked tokens and predicts the label. The two are trained jointly with REINFORCE-style gradient estimation, and a regulariser pushes the mask towards sparse, contiguous selections. A variant adds a cross-entropy loss on human rationales, where available §5. Two encoder/decoder instances are used, with BERT or GloVe plus a BiLSTM app. C.1.
• Pipeline models §5. A rationale identifier is trained first, then a classifier that sees only the identified rationales. No parameters are shared. The simple version segments the input into sentences and encodes each with a GRU and attention §5. The BERT-to-BERT version replaces both stages with BERT §5. The classifier is trained on the positive rationales, a form of teacher forcing §5.
• Soft selection §5. Tokens pass through BERT (or GloVe for the longest inputs) and a BiLSTM. An additive attention layer pools the states. For inputs longer than BERT's 512-token limit, the next chunk is encoded as a new sequence, with position indices restarting at 0 §5. Three soft scorers are applied to this model: simple gradients, attention weights and LIME §5.

Hyperparameters

Values are from the appendix app. C unless marked as code. The paper gives no batch sizes for most models, so those are shown from code.

Results

Author-reported. Setup: the seven datasets tab. 1; baselines as in §5; hardware not stated. Metrics: F1 for rationale agreement (IOU at 0.5, and token level) §4.1. Performance is accuracy for CoS-E and macro F1 otherwise tab. 3.

Hard rationale selection tab. 3. "(u)" marks models that learn from instance labels only. The dagger marks runs where REINFORCE-style rationale training degenerated.

tab. 3 has no Lehman et al. row for CoS-E or e-SNLI. The authors say the sentence-level pipeline is not appropriate where rationales are short spans §6.

Soft scores tab. 4. Perf. is accuracy (CoS-E) or F1 (others). Comp. is comprehensiveness and Suff. is sufficiency, both as AOPC; higher Comp. and lower Suff. are better. "Random" averages 10 random orderings.

Headline reading, as the authors state it §6:

• Gradient and LIME scoring give more comprehensive rationales than attention weights, in line with earlier work §6. Attention scores agree better with human rationales on AUPRC §6.
• LIME does particularly well on faithfulness across these tasks §6.
• The authors use the random ordering as a reference. On some tasks random removal is "particularly damaging", which they read as indicating poor absolute ranking §6 tab. 4.

Paper vs code

The official repository is jayded/eraserbenchmark. The paper links it in its footnote §5, and the account name matches author Jay DeYoung. It is licensed Apache-2.0 and was read at the pinned commit above. The paper is arXiv v2 as fetched.

• IOU match rule. The paper says a prediction matches when overlap is "more than" the threshold 0.5 §4.1. The code uses >= metrics.py.
• Soft-score cut-off. The paper sets k_d to the average human rationale length of each dataset §4.2. The code instead keeps a fixed fraction of each document, int(k_fraction * len(tokens)), with k_fraction 0.07 for most datasets and 0.013 for Evidence Inference encode_attend.py boolq_soft.json evidence_inference_soft.json.
• Soft embeddings. The paper says the soft models use GloVe 300d in place of BERT §5. The soft configs load glove.6B.200d movies_soft.json boolq_soft.json, except Evidence Inference, which loads PubMed vectors evidence_inference_soft.json.
• Soft learning rate. The paper gives 1e-2 for the GloVe-LSTM app. C.2. The soft configs use 1e-3 boolq_soft.json.
• Lehman pipeline learning rate. The paper gives 1e-3 for all datasets app. C.3. The identifier for Movies uses 1e-4 movies.json.
• BERT-LSTM rows. tab. 4 labels Movies, FEVER, MultiRC, CoS-E and e-SNLI as BERT + LSTM. The soft configs for Movies, FEVER and MultiRC use GloVe movies_soft.json fever_soft.json multirc_soft.json. No soft config exists for CoS-E or e-SNLI, and encode_attend.py has no BERT path. The soft BERT-LSTM rows therefore cannot be traced to this code.
• Scorers. The paper reports gradient, LIME and random rows tab. 4. The decode path computes only the model's attention and the two contrast passes encode_attend.py. No gradient, LIME or random scorer was found in the repository.
• Lei et al. model. The paper describes it in detail §5 app. C.1. The repository contains no REINFORCE hard-selection model, and no sparsity or contiguity parameter appears in any config. Its rows in tab. 3 cannot be reproduced from this code.
• AOPC averaging. The paper's normalisation is 1/(|B|+1) over k = 0 to |B| §4.2 eq. 3. The released code averages the per-instance deltas over the five cut-offs metrics.py with defaults at metrics.py. Those cut-offs are not generated anywhere in this repository. The producer is external.
• Agreement with the paper, for reference. The BERT-to-BERT settings in app. C.4 (learning rate 1e-5, Adam, 10 epochs) match the BERT configs evidence_inference_bert.json evidence_inference_bert.json.
• Sampling ratio. The paper says negatives are sampled randomly each epoch §5. The code's sampling_ratio is negatives per positive, so 1.0 means one negative per positive evidence_identifier.py.

Pitfalls

• Ties and empty inputs in top-k. nlargest/nsmallest over (score, token) tuples break ties by token id, not by position encode_attend.py. With int(k_fraction * len), a document shorter than about 15 tokens gives an empty sufficiency input at k_fraction 0.07. The paper does not mention this.
• Cost of the faithfulness metrics. The paper calls them cheap compared with per-token measures §4.2. The decode path still runs three forward passes per instance (full, sufficiency, comprehensiveness) [code: rationale_benchmark/models/encode_attend.py#L433-L439]. AOPC needs two extra passes per cut-off.
• Micro or macro F1. The paper says the IOU F1 is "calculated" from partial matches §4.1 and does not say whether micro or macro averaging is reported. The code returns both metrics.py. Token F1 has the same ambiguity metrics.py.
• AUPRC versus AP/ROC. _auprc has no filter for single-class instances, but the AP and ROC aggregators do metrics.py and metrics.py. The AUPRC numbers may therefore include instances the other scores skip.
• Table 1 and appendix counts disagree. Table 1 gives e-SNLI as 911938 / 16449 / 16429. tab. 5 gives documents 911938 / 16328 / 16299 and instances 549309 / 9823 / 9807. tab. 6 lists BoolQ as 10661 instances, but tab. 5 gives 6363 + 1491 + 2817 = 10671. Use the released data, not these tables, for counts.
• Table and equation numbers. The numbers here follow the order of the floats and equations in the LaTeX source (Figure 1, Figure 2, Tables 1 to 6, equations 1 to 3). They were not checked against a compiled PDF.
• Code bug in the input check. metrics.py#L371 tests docid not in docid, which compares the string with itself, so an unknown document id is never reported metrics.py.
• Unused threshold. The config's threshold (0.1) is passed to decode() but never used there encode_attend.py.
• Under-specified training. The paper calls BERT-to-BERT settings "standard" app. C.4. The configs add warmup steps (10 or 50) and gradient clipping (1.0, or 0.5 for CoS-E) that the paper does not state *_bert.json.
• Length-dependent behaviour. The authors note that the sentence-level pipeline is unsuitable for datasets whose rationales are short spans, and that BERT-based models could not run on the longest inputs §5 §6. Soft models therefore swap BERT for GloVe on Evidence Inference and BoolQ §5.

Lineage

• Builds on: Lei et al., "Rationalizing Neural Predictions" (EMNLP 2016), the hard-selection model implemented here §5. Lehman et al., "Inferring Which Medical Treatments Work from Reports of Clinical Trials" (NAACL 2019), the sentence pipeline and the Evidence Inference data §5 §3.
• Compared against: contrast-example faithfulness measures from Zaidan et al. (2007) and Yu et al. (2019, "Rethinking Cooperative Rationalization") §4.2. The AOPC idea is adapted from Samek et al. (2016) for saliency maps §4.2. Hooker et al. (2019) is noted as a similar retraining-based method §4.2. Jain and Wallace (2019, "Attention is not Explanation") and Serrano and Smith (2019) are cited for the attention findings §6.
• Superseded by: not stated in the source.