--- kind: concept slug: attention-interpretability title: Attention interpretability summary: >- Scoring inputs by attention weights or by gradient-weighted or propagated relevance, then testing faithfulness. Attention often agrees with human rationales but was less comprehensive than gradient or LIME in one benchmark. Substitution tests show whether attention does work in a given model. Path patching finds circuit heads. Critiques say attention is not a reliable explanation. tags: [interpretability, attention, rationales, faithfulness, evaluation, circuits, relevance-propagation] papers: ["1902.02181", "1904.02874", "1908.04626", "1911.03429", "2012.09838", "2103.15679", "2204.07756", "2211.00593"] related: [attention-mechanism, explanation-evaluation, rationale-extraction, self-attention, vision-transformer, feature-attribution, mechanistic-interpretability] status: draft updated_at: 2026-10-24 --- ## Overview An attention layer already produces a weight for each input position. A common hope is that a large weight means the model used that input for its output. Such a short supporting piece of the input is called a rationale [arxiv:1911.03429 §1]. Two questions follow, and the sources answer them differently: 1. **Agreement.** Do the scores pick the same spans a human would mark? 2. **Faithfulness.** Does the prediction actually depend on the spans? This is tested by removing the spans (comprehensiveness) or keeping only them (sufficiency) and measuring the change in the predicted-class probability [arxiv:1911.03429 §4.2]. An earlier line of work argued that attention weights are not explanations, because different attention distributions can give similar predictions. A later paper argues that this depends on what "explanation" means, and offers tests of whether attention does work in a particular model [arxiv:1908.04626 §2] [arxiv:1908.04626 §5]. Most variants below produce a score per token or per image region. They differ in where the score comes from, and in how well they do on the two questions. The circuit variant instead scores attention heads, by intervening on the model and measuring the effect on its output [arxiv:2211.00593 §3]. ## Variants ### Attention weights as the score Read the attention distribution of the model and treat it as the importance of each token. The survey describes this working hypothesis as weight magnitude tracking input relevance, and reports visualizations of attention in summarization, recommendation, image captioning and translation as evidence for it [arxiv:1904.02874 §7]. In the ERASER benchmark, a soft model with an additive attention pooling layer uses these weights directly as its scorer [arxiv:1911.03429 §5]. Card: [Attentive Survey of Attention Models](/papers/1904.02874) and [ERASER: A Benchmark to Evaluate Rationalized NLP Models](/papers/1911.03429). ### Gradient-weighted attention relevancy (Transformers) For Transformers with two input streams (co-attention) or an encoder-decoder, the method weights each attention map by the gradient of the target score, keeps the positive part, averages over heads, and accumulates the result layer by layer into relevancy matrices [arxiv:2103.15679 §3] [eq. 5]. The authors say earlier methods did not give a complete rule for these layers [arxiv:2103.15679 §1]. The main method uses raw attention; the LRP variant is optional and changes little [arxiv:2103.15679 app. E]. The results are author-reported. On weakly supervised segmentation from DETR on COCO validation, the method reaches AP 13.1 against 7.2 for the best baseline, Transformer attribution, and raw attention reaches 5.6 [arxiv:2103.15679 tab. 1]. On ViT perturbation tests it is close to Transformer attribution [arxiv:2103.15679 tab. 2]. Card: [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679). ### Class-specific relevance propagation (ViT and BERT) For one Transformer classifier and one chosen class, the method passes the class score back through every block. Relevance flows through the attention products, the skip connections and the linear layers, using LRP-style rules that handle negative values [arxiv:2012.09838 §3.1] [arxiv:2012.09838 §3.2]. For each attention map, the gradient-weighted relevance is kept where it is positive, averaged over heads and added to an identity term. The block matrices are then multiplied together, as in attention rollout [arxiv:2012.09838 §3.3] [eq. 13] [eq. 14]. The row of the [CLS] token gives the map over patches or tokens [arxiv:2012.09838 §3.4]. The authors state that the method is class-specific by construction and needs no contrasting pass [arxiv:2012.09838 §2]. The results are author-reported. On ImageNet with ViT-base, the negative perturbation test for the predicted class reaches 54.16, against 50.49 for the best other method, partial LRP [arxiv:2012.09838 tab. 1]. On the positive test, where lower is better, it reaches 17.03 against 19.64 for partial LRP [arxiv:2012.09838 tab. 1]. On ImageNet-Segmentation, mIoU is 61.95 against 57.94 for partial LRP and 55.42 for rollout [arxiv:2012.09838 tab. 2]. Using raw attention in place of the gradient-weighted term lowers mIoU to 59.88 [arxiv:2012.09838 tab. 3]. For BERT on Movie Reviews, the paper says the method is best for k = 10 to 80 tokens, but the values appear only in a figure [arxiv:2012.09838 §4.1] [fig. 4]. The released code differs from the paper in several places: the block range used for the visualizations, the scaling of matrix products, and the output value that is explained (logit, not probability). Its skip-connection normalization also sums over the whole batch, so batch size must be 1 or the sums must be split per sample. These deviations are listed in the [paper card](/papers/2012.09838), which to read before reimplementing the paper's equations. ### Gradient and LIME scores on the same model Keep the trained model and score each token by a different signal: a gradient of the output with respect to the input, or a local surrogate fitted by LIME. ERASER uses both as alternative scorers on the same soft architecture and compares them with attention on the same faithfulness metrics [arxiv:1911.03429 §5] [tab. 4]. The paper reports these scorers in its tables. The released ERASER code contains only the attention scorer, so the gradient and LIME rows cannot be rerun from it; see the [ERASER card](/papers/1911.03429). ### Perturbation-based faithfulness (evaluation) Instead of a scorer, this is a test of any scorer. Remove the top-scored tokens and record the drop in the predicted-class probability (comprehensiveness). Keep only those tokens and record the drop (sufficiency). ERASER averages these drops over several cut-offs, the top 1, 5, 10, 20 and 50 per cent, and calls the average AOPC [arxiv:1911.03429 §4.2] [eq. 3]. The same test works for any score, not only attention, which makes it useful for comparing variants. ### Substitution and adversarial tests (evaluation) A set of four checks for a single-layer bidirectional LSTM with additive attention on binary text tasks. They ask whether the attention in one trained model is doing work, and whether different attention can give the same predictions [arxiv:1908.04626 §3.1]. - **Uniform attention.** Freeze the weights to uniform values over the positions and re-measure F1. On Diabetes the uniform model drops from 0.775 to 0.706; on Anemia, IMDb and SST the drops are smaller [arxiv:1908.04626 tab. 2]. The paper drops AG News and 20 Newsgroups, where uniform attention matched the base model [arxiv:1908.04626 §3.2]. - **Seed variance.** Retrain the model with several seeds and measure how far the attention moves between them. This sets a baseline for the adversary's divergence [arxiv:1908.04626 §3.3]. - **Frozen-weight MLP guide.** Impose the attention weights on a non-contextual per-token MLP and compare its F1. On Diabetes, SST and IMDb the base LSTM weights give 0.753, 0.824 and 0.905, and the adversary's weights give 0.503, 0.592 and 0.700. Anemia is the exception, at 0.931 and 0.932 [arxiv:1908.04626 tab. 3]. - **Consistent adversary.** Train a second model that keeps its predictions close to the base model while its attention moves away. Such models exist on all four datasets kept for this test, with average JSD above 0.4, and their F1 is near the base model's [arxiv:1908.04626 §4] [arxiv:1908.04626 tab. 4]. The paper reads these results as showing that alternative attention can be found for some binary tasks, but that it does not guide a simple model well. The paper attributes the Anemia exception to skewed labels and sparse indicative tokens, and does not test that reading [arxiv:1908.04626 §4]. Caveats: all analysis runs on the test set [arxiv:1908.04626 §3.1], and the adversary in table 4 is the highest-F1 model among those with JSD above 0.4, so those figures are selected on test metrics [arxiv:1908.04626 tab. 4]. The λ is swept per dataset [arxiv:1908.04626 tab. 5]. The authors name binary tasks and this one architecture as the scope of the test [arxiv:1908.04626 §6]. Card: [Attention is not not Explanation](/papers/1908.04626). ### Causal tracing of attention heads (circuits) Instead of scoring inputs, this method finds which attention heads matter for one behaviour. A node is a pair of a head and one token position. The authors use path patching: they run the model on a prompt and on a reference prompt, and replace only a sender head's contribution along the paths to a receiver (the logits, or the query, key or value of another head) [arxiv:2211.00593 §3]. Nodes outside the candidate circuit are removed by mean ablation, with the mean taken per template and position over prompts from a reference distribution [arxiv:2211.00593 §2.1]. In GPT-2 small on indirect object identification, the authors find a circuit of 26 heads in seven classes, which keeps 87% of the full model's average logit difference [arxiv:2211.00593 §4] [arxiv:2211.00593 §4.3]. Their checks are faithfulness, completeness and minimality, and these are different from the input-relevance tests above [arxiv:2211.00593 §1]. Card: [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593). ## When to use which | Situation | Use | Why | |---|---|---| | A model already has an attention layer and you need a per-token score from weights the forward pass already computes | Attention weights | Attention had the highest AUPRC against human rationales in 5 of 7 ERASER rows, but was less comprehensive than gradient or LIME scores in all 7 [arxiv:1911.03429 tab. 4] | | A Transformer has co-attention or encoder-decoder layers and you need a per-input relevance map | Gradient-weighted attention relevancy | It gives a rule for those layers, which the authors say earlier methods did not cover completely [arxiv:2103.15679 §1] [arxiv:2103.15679 §3.2]. Its DETR gain is author-reported [arxiv:2103.15679 tab. 1] | | The highlighted tokens must reflect what drives the output | Gradient or LIME, tested with comprehensiveness | LIME had the highest comprehensiveness in 6 of 7 ERASER rows, and gradient in the other [arxiv:1911.03429 tab. 4]. ERASER's LIME rows are not reproducible from its released code (see Variants) | | You need a per-patch or per-token map for one class from a ViT or BERT classifier, and can run a backward pass | Class-specific relevance propagation | It beat the other methods tested on ImageNet perturbation and segmentation in the authors' own tables. It needs gradients of every attention map, and the released code differs from the paper in several places [arxiv:2012.09838 tab. 1] [tab. 2] [eq. 13] | | You need a claim that attention itself is the explanation | Treat with caution | Critiques reported in the survey say attention weights are at best noisy predictors of importance [arxiv:1904.02874 §7]. A second survey says attention may be a plausible explanation without being a faithful one [arxiv:1902.02181 §VI-A] | | You are comparing explanation methods | Perturbation-based faithfulness on every method | The same cut-off and metric across methods is what makes the numbers comparable [arxiv:1911.03429 §4.2] | | You need to know which attention heads implement a behaviour, and you can run the model with interventions | Path patching with mean ablation, checked for faithfulness, completeness and minimality | It gives a head-level causal account. Faithfulness alone can miss redundancy, which the authors found in the Name Mover heads [arxiv:2211.00593 §3.4] [arxiv:2211.00593 §1]. Each path-patching run needs three forward passes per sender set, and the exhaustive completeness check is exponential, so the authors sample [arxiv:2211.00593 §3] [arxiv:2211.00593 §4.1] | | You want to check whether a trained text classifier's attention does work before reading it as an explanation | Substitution tests: uniform attention, seed variance, frozen-weight MLP guide | The uniform test needs no retraining, and the seed and MLP tests need extra training runs. They give a baseline that the attention itself is compared against [arxiv:1908.04626 §3.2] [arxiv:1908.04626 §3.3]. The paper tested only binary tasks with one LSTM architecture [arxiv:1908.04626 §6] | | You want to know whether attention weights are unique given the predictions | Consistent adversary | It shows that different attention can give near-identical predictions on some tasks. Its attention guided the simple MLP poorly on three of the four tasks, and matched the base attention on Anemia [arxiv:1908.04626 tab. 3] [arxiv:1908.04626 tab. 4] | | Attention should be made more faithful | Future work, not a settled method | The survey points to diversity-driven hidden states as one proposal, without evidence in the sources used here [arxiv:1904.02874 §8.4]. A vision-attention survey also lists interpretability among open problems [arxiv:2204.07756 §4] | Results from ERASER, author-reported. Comp. is comprehensiveness (higher is better) and AUPRC is agreement with human rationales (higher is better). The Evidence Inference and BoolQ rows use GloVe with an LSTM; the others are labelled BERT with an LSTM [arxiv:1911.03429 tab. 4]. The released code cannot be traced for the BERT rows, so treat them as reported, not reproduced. | Dataset | AUPRC attention | AUPRC gradient | AUPRC LIME | Comp. attention | Comp. gradient | Comp. LIME | |---|---|---|---|---|---|---| | Evidence Inference | 0.506 | 0.016 | 0.014 | -0.002 | 0.046 | 0.006 | | BoolQ | 0.525 | 0.072 | 0.073 | 0.010 | 0.024 | 0.028 | | Movies | 0.417 | 0.385 | 0.280 | 0.129 | 0.142 | 0.187 | | FEVER | 0.235 | 0.232 | 0.291 | 0.037 | 0.059 | 0.212 | | MultiRC | 0.244 | 0.224 | 0.208 | 0.036 | 0.077 | 0.213 | | CoS-E | 0.606 | 0.585 | 0.544 | 0.080 | 0.124 | 0.223 | | e-SNLI | 0.395 | 0.416 | 0.513 | 0.105 | 0.180 | 0.437 | The authors read the table as showing that gradient and LIME give more comprehensive rationales than attention, and that attention agrees better with human rationales on AUPRC [arxiv:1911.03429 §6]. The table supports the first reading in all seven rows. For the second, attention has the highest AUPRC in five rows; LIME is higher on FEVER and e-SNLI. ## Implementation sketch The two pieces are a scorer and a faithfulness check. Both work on any model that returns a class probability. ```pseudocode # Attention weights as a per-token score [arxiv:1911.03429 §5] def attention_scores(model, tokens): # One weight per input token, from the model's own pooling layer weights = model.attention_weights(tokens) # shape (T,), sums to 1 return weights ``` ```pseudocode # Gradient-weighted map for one attention layer [arxiv:2103.15679 §3] [eq. 5] def head_averaged_map(attention, grad): # attention and grad: (heads, rows, cols); grad is d(target score) / d(attention) # Keep only the positive part of the gradient-weighted attention weighted = clamp(grad * attention, min=0) # Mean over heads, after the positive part return mean(weighted, axis=0) ``` ```pseudocode # Mean-ablate every node outside a candidate circuit [arxiv:2211.00593 §2.1] def circuit_logit_diff(model, prompts, circuit, mean_cache): # mean_cache: per-template, per-position mean of each node on reference prompts hooks = {} for node in all_nodes(model): # node = (layer, head, position) if node not in circuit: # Replace the node's output with its reference mean hooks[node] = mean_cache[node] # Average the logit difference over the prompts, with the hooks applied return mean(logit_diff(run_with_hooks(model, prompts, hooks))) ``` ```pseudocode # Comprehensiveness and sufficiency for the top-k tokens [arxiv:1911.03429 §4.2] def faithfulness(model, tokens, scores, k): # Predicted class and its probability on the full input j = argmax(model(tokens)) p_full = model(tokens)[j] # Indices of the k highest-scoring tokens; k is a fixed choice per dataset top = argsort_desc(scores)[:k] # Remove the rationale: a larger drop means it mattered more comp = p_full - model(remove(tokens, top))[j] # Keep only the rationale: a smaller drop means it is enough suff = p_full - model(keep(tokens, top))[j] return comp, suff ``` Where variants differ: the scorer (attention, gradient-weighted relevancy, gradient, LIME) and the choice of `k`. ERASER states that `k` is the average length of the human rationales for each dataset [arxiv:1911.03429 §4.2]. Use the same `k` rule for every method you compare. Points to settle before implementing the gradient-weighted map: - **Target score.** The paper says "the model's output for class t" and does not say whether this is a logit or a probability [arxiv:2103.15679 §3]. The [paper card](/papers/2103.15679) notes that the released code uses the logit, and that a softmax probability would change the gradients. - **Main path versus LRP.** The paper's main method uses raw attention, and the released code's default enables an LRP variant [arxiv:2103.15679 app. E]. See the [paper card](/papers/2103.15679) before choosing a default. - **Zero denominators.** Eq. 9 divides by row sums of the accumulated matrix. The paper gives no guard for a zero sum [arxiv:2103.15679 §3]. - **Reported `k` values.** The released ERASER code uses a fraction of the document length, not the average rationale length the paper states [arxiv:1911.03429 §4.2]. The [ERASER card](/papers/1911.03429) lists the values. ## Contradictions - **Agreement versus reliability.** ERASER reports that attention agrees with human rationales on AUPRC in five of seven rows [arxiv:1911.03429 §6]. The survey reports that Jain and Wallace found weak correlation between attention and feature-importance measures, and that permuted or adversarial weights leave predictions unchanged. Serrano and Smith, using representation erasure, call attention weights at best noisy predictors of importance [arxiv:1904.02874 §7]. The two sides measure different things, and this page does not resolve which one matters for a given use. The circuit paper tests head-level causal effects, not input relevance, so it does not settle this disagreement [arxiv:2211.00593 §1]. - **Raw attention versus gradient-weighted relevancy.** The gradient-weighted method beats raw attention on DETR segmentation (AP 13.1 against 5.6) and on ViT perturbation, where raw attention has the lower negative-perturbation AUC (45.55 against 54.61 for predicted class) [arxiv:2103.15679 tab. 1] [arxiv:2103.15679 tab. 2]. This points the same way as ERASER's comprehensiveness ranking, in which attention is the lowest of the three scorers [arxiv:1911.03429 tab. 4]. But ERASER finds attention best on agreement with human rationales. The tasks, datasets and metrics differ, so this page does not turn these numbers into one ranking. The class-specific method on ViT also beats raw attention on the negative perturbation test (54.16 against 45.55), but rollout is close to it there (53.1) [arxiv:2012.09838 tab. 1]. So the gap to raw attention is larger than the gap to rollout. - **Faithfulness ranking.** ERASER finds attention less comprehensive than gradient and LIME in all seven rows [arxiv:1911.03429 tab. 4]. The survey reports no such ranking, so this is a single benchmark's result. - **Whether the "attention is not explanation" claim holds.** The earlier claim is that different attention can give near-identical predictions. The later paper accepts that such attention exists for some binary tasks, but argues that the claim depends on the definition of explanation, and that the adversary's attention does not guide a simple model well on most tasks [arxiv:1908.04626 §2] [arxiv:1908.04626 tab. 3]. The paper does not show that attention is faithful in general. Its evidence covers binary LSTM classifiers only, so this page does not extend it to other models. - **Plausible versus faithful.** The NLP attention survey reports that attention weights may not be a reliable explanation. It cites work showing that attention can disagree with other explainability methods, and that adversarial attention distributions are easy to build in local cases. It concludes that attention may be a plausible explanation without being a faithful one [arxiv:1902.02181 §VI-A]. This matches the plausibility reading that [arxiv:1908.04626 §5] uses to defend attention, so the two sources agree on the wording and differ on the verdict. The survey reports no experiments of its own, so this page does not count it as evidence either way. ## Papers - [Attention in Natural Language Processing](/papers/1902.02181): a survey whose section on attention as explanation says attention may be plausible without being faithful; it reports no experiments. - [An Attentive Survey of Attention Models](/papers/1904.02874): a survey whose interpretability section covers attention weights as explanations, and the critiques of that use. - [ERASER: A Benchmark to Evaluate Rationalized NLP Models](/papers/1911.03429): a benchmark with human-marked rationales and comprehensiveness and sufficiency metrics, comparing attention, gradient and LIME scorers. - [Attention is not not Explanation](/papers/1908.04626): four tests of attention in LSTM text classifiers, including a consistent adversary, on binary tasks; author-reported F1 and divergence values. - [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679): a gradient-weighted, layer-by-layer relevancy method for co-attention and encoder-decoder Transformers, with author-reported segmentation and perturbation results. - [Transformer Interpretability Beyond Attention Visualization](/papers/2012.09838): class-specific relevance propagation through ViT and BERT, with author-reported ImageNet perturbation and segmentation results. - [Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756): a vision-attention taxonomy with no experiments of its own; it names interpretability among open problems. - [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593): path patching and mean ablation find a 26-head circuit for one behaviour, tested for faithfulness, completeness and minimality.