Attention interpretability

Overview

An attention layer already produces a weight for each input position. A common hope is that a large weight means the model used that input for its output. Such a short supporting piece of the input is called a rationale 1911.03429 §1.

Two questions follow, and the sources answer them differently:

1. Agreement. Do the scores pick the same spans a human would mark?
2. Faithfulness. Does the prediction actually depend on the spans? This is tested by removing the spans (comprehensiveness) or keeping only them (sufficiency) and measuring the change in the predicted-class probability 1911.03429 §4.2.

An earlier line of work argued that attention weights are not explanations, because different attention distributions can give similar predictions. A later paper argues that this depends on what "explanation" means, and offers tests of whether attention does work in a particular model 1908.04626 §2 1908.04626 §5.

Most variants below produce a score per token or per image region. They differ in where the score comes from, and in how well they do on the two questions. The circuit variant instead scores attention heads, by intervening on the model and measuring the effect on its output 2211.00593 §3.

Variants

Attention weights as the score

Read the attention distribution of the model and treat it as the importance of each token. The survey describes this working hypothesis as weight magnitude tracking input relevance, and reports visualizations of attention in summarization, recommendation, image captioning and translation as evidence for it 1904.02874 §7. In the ERASER benchmark, a soft model with an additive attention pooling layer uses these weights directly as its scorer 1911.03429 §5. Card: Attentive Survey of Attention Models and ERASER: A Benchmark to Evaluate Rationalized NLP Models.

Gradient-weighted attention relevancy (Transformers)

For Transformers with two input streams (co-attention) or an encoder-decoder, the method weights each attention map by the gradient of the target score, keeps the positive part, averages over heads, and accumulates the result layer by layer into relevancy matrices 2103.15679 §3 eq. 5. The authors say earlier methods did not give a complete rule for these layers 2103.15679 §1. The main method uses raw attention; the LRP variant is optional and changes little 2103.15679 app. E.

The results are author-reported. On weakly supervised segmentation from DETR on COCO validation, the method reaches AP 13.1 against 7.2 for the best baseline, Transformer attribution, and raw attention reaches 5.6 2103.15679 tab. 1. On ViT perturbation tests it is close to Transformer attribution 2103.15679 tab. 2. Card: Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.

Class-specific relevance propagation (ViT and BERT)

For one Transformer classifier and one chosen class, the method passes the class score back through every block. Relevance flows through the attention products, the skip connections and the linear layers, using LRP-style rules that handle negative values 2012.09838 §3.1 2012.09838 §3.2. For each attention map, the gradient-weighted relevance is kept where it is positive, averaged over heads and added to an identity term. The block matrices are then multiplied together, as in attention rollout 2012.09838 §3.3 eq. 13 eq. 14. The row of the [CLS] token gives the map over patches or tokens 2012.09838 §3.4. The authors state that the method is class-specific by construction and needs no contrasting pass 2012.09838 §2.

The results are author-reported. On ImageNet with ViT-base, the negative perturbation test for the predicted class reaches 54.16, against 50.49 for the best other method, partial LRP 2012.09838 tab. 1. On the positive test, where lower is better, it reaches 17.03 against 19.64 for partial LRP 2012.09838 tab. 1. On ImageNet-Segmentation, mIoU is 61.95 against 57.94 for partial LRP and 55.42 for rollout 2012.09838 tab. 2. Using raw attention in place of the gradient-weighted term lowers mIoU to 59.88 2012.09838 tab. 3. For BERT on Movie Reviews, the paper says the method is best for k = 10 to 80 tokens, but the values appear only in a figure 2012.09838 §4.1 fig. 4.

The released code differs from the paper in several places: the block range used for the visualizations, the scaling of matrix products, and the output value that is explained (logit, not probability). Its skip-connection normalization also sums over the whole batch, so batch size must be 1 or the sums must be split per sample. These deviations are listed in the paper card, which to read before reimplementing the paper's equations.

Gradient and LIME scores on the same model

Keep the trained model and score each token by a different signal: a gradient of the output with respect to the input, or a local surrogate fitted by LIME. ERASER uses both as alternative scorers on the same soft architecture and compares them with attention on the same faithfulness metrics 1911.03429 §5 tab. 4. The paper reports these scorers in its tables. The released ERASER code contains only the attention scorer, so the gradient and LIME rows cannot be rerun from it; see the ERASER card.

Perturbation-based faithfulness (evaluation)

Instead of a scorer, this is a test of any scorer. Remove the top-scored tokens and record the drop in the predicted-class probability (comprehensiveness). Keep only those tokens and record the drop (sufficiency). ERASER averages these drops over several cut-offs, the top 1, 5, 10, 20 and 50 per cent, and calls the average AOPC 1911.03429 §4.2 eq. 3. The same test works for any score, not only attention, which makes it useful for comparing variants.

Substitution and adversarial tests (evaluation)

A set of four checks for a single-layer bidirectional LSTM with additive attention on binary text tasks. They ask whether the attention in one trained model is doing work, and whether different attention can give the same predictions 1908.04626 §3.1.

• Uniform attention. Freeze the weights to uniform values over the positions and re-measure F1. On Diabetes the uniform model drops from 0.775 to 0.706; on Anemia, IMDb and SST the drops are smaller [arxiv:1908.04626 tab. 2]. The paper drops AG News and 20 Newsgroups, where uniform attention matched the base model 1908.04626 §3.2.
• Seed variance. Retrain the model with several seeds and measure how far the attention moves between them. This sets a baseline for the adversary's divergence 1908.04626 §3.3.
• Frozen-weight MLP guide. Impose the attention weights on a non-contextual per-token MLP and compare its F1. On Diabetes, SST and IMDb the base LSTM weights give 0.753, 0.824 and 0.905, and the adversary's weights give 0.503, 0.592 and 0.700. Anemia is the exception, at 0.931 and 0.932 1908.04626 tab. 3.
• Consistent adversary. Train a second model that keeps its predictions close to the base model while its attention moves away. Such models exist on all four datasets kept for this test, with average JSD above 0.4, and their F1 is near the base model's 1908.04626 §4 1908.04626 tab. 4.

The paper reads these results as showing that alternative attention can be found for some binary tasks, but that it does not guide a simple model well. The paper attributes the Anemia exception to skewed labels and sparse indicative tokens, and does not test that reading 1908.04626 §4. Caveats: all analysis runs on the test set 1908.04626 §3.1, and the adversary in table 4 is the highest-F1 model among those with JSD above 0.4, so those figures are selected on test metrics 1908.04626 tab. 4. The λ is swept per dataset 1908.04626 tab. 5. The authors name binary tasks and this one architecture as the scope of the test 1908.04626 §6. Card: Attention is not not Explanation.

Causal tracing of attention heads (circuits)

Instead of scoring inputs, this method finds which attention heads matter for one behaviour. A node is a pair of a head and one token position. The authors use path patching: they run the model on a prompt and on a reference prompt, and replace only a sender head's contribution along the paths to a receiver (the logits, or the query, key or value of another head) 2211.00593 §3. Nodes outside the candidate circuit are removed by mean ablation, with the mean taken per template and position over prompts from a reference distribution 2211.00593 §2.1. In GPT-2 small on indirect object identification, the authors find a circuit of 26 heads in seven classes, which keeps 87% of the full model's average logit difference 2211.00593 §4 2211.00593 §4.3. Their checks are faithfulness, completeness and minimality, and these are different from the input-relevance tests above 2211.00593 §1. Card: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.

When to use which

Results from ERASER, author-reported. Comp. is comprehensiveness (higher is better) and AUPRC is agreement with human rationales (higher is better). The Evidence Inference and BoolQ rows use GloVe with an LSTM; the others are labelled BERT with an LSTM 1911.03429 tab. 4. The released code cannot be traced for the BERT rows, so treat them as reported, not reproduced.

The authors read the table as showing that gradient and LIME give more comprehensive rationales than attention, and that attention agrees better with human rationales on AUPRC 1911.03429 §6. The table supports the first reading in all seven rows. For the second, attention has the highest AUPRC in five rows; LIME is higher on FEVER and e-SNLI.

Implementation sketch

The two pieces are a scorer and a faithfulness check. Both work on any model that returns a class probability.

# Attention weights as a per-token score  [arxiv:1911.03429 §5]
def attention_scores(model, tokens):
    # One weight per input token, from the model's own pooling layer
    weights = model.attention_weights(tokens)   # shape (T,), sums to 1
    return weights

# Gradient-weighted map for one attention layer  [arxiv:2103.15679 §3] [eq. 5]
def head_averaged_map(attention, grad):
    # attention and grad: (heads, rows, cols); grad is d(target score) / d(attention)
    # Keep only the positive part of the gradient-weighted attention
    weighted = clamp(grad * attention, min=0)
    # Mean over heads, after the positive part
    return mean(weighted, axis=0)

# Mean-ablate every node outside a candidate circuit  [arxiv:2211.00593 §2.1]
def circuit_logit_diff(model, prompts, circuit, mean_cache):
    # mean_cache: per-template, per-position mean of each node on reference prompts
    hooks = {}
    for node in all_nodes(model):   # node = (layer, head, position)
        if node not in circuit:
            # Replace the node's output with its reference mean
            hooks[node] = mean_cache[node]
    # Average the logit difference over the prompts, with the hooks applied
    return mean(logit_diff(run_with_hooks(model, prompts, hooks)))

# Comprehensiveness and sufficiency for the top-k tokens  [arxiv:1911.03429 §4.2]
def faithfulness(model, tokens, scores, k):
    # Predicted class and its probability on the full input
    j = argmax(model(tokens))
    p_full = model(tokens)[j]
    # Indices of the k highest-scoring tokens; k is a fixed choice per dataset
    top = argsort_desc(scores)[:k]
    # Remove the rationale: a larger drop means it mattered more
    comp = p_full - model(remove(tokens, top))[j]
    # Keep only the rationale: a smaller drop means it is enough
    suff = p_full - model(keep(tokens, top))[j]
    return comp, suff

Where variants differ: the scorer (attention, gradient-weighted relevancy, gradient, LIME) and the choice of k. ERASER states that k is the average length of the human rationales for each dataset 1911.03429 §4.2. Use the same k rule for every method you compare.

Points to settle before implementing the gradient-weighted map:

• Target score. The paper says "the model's output for class t" and does not say whether this is a logit or a probability 2103.15679 §3. The paper card notes that the released code uses the logit, and that a softmax probability would change the gradients.
• Main path versus LRP. The paper's main method uses raw attention, and the released code's default enables an LRP variant 2103.15679 app. E. See the paper card before choosing a default.
• Zero denominators. Eq. 9 divides by row sums of the accumulated matrix. The paper gives no guard for a zero sum 2103.15679 §3.
• Reported k values. The released ERASER code uses a fraction of the document length, not the average rationale length the paper states 1911.03429 §4.2. The ERASER card lists the values.

Contradictions

• Agreement versus reliability. ERASER reports that attention agrees with human rationales on AUPRC in five of seven rows 1911.03429 §6. The survey reports that Jain and Wallace found weak correlation between attention and feature-importance measures, and that permuted or adversarial weights leave predictions unchanged. Serrano and Smith, using representation erasure, call attention weights at best noisy predictors of importance 1904.02874 §7. The two sides measure different things, and this page does not resolve which one matters for a given use. The circuit paper tests head-level causal effects, not input relevance, so it does not settle this disagreement 2211.00593 §1.
• Raw attention versus gradient-weighted relevancy. The gradient-weighted method beats raw attention on DETR segmentation (AP 13.1 against 5.6) and on ViT perturbation, where raw attention has the lower negative-perturbation AUC (45.55 against 54.61 for predicted class) 2103.15679 tab. 1 2103.15679 tab. 2. This points the same way as ERASER's comprehensiveness ranking, in which attention is the lowest of the three scorers 1911.03429 tab. 4. But ERASER finds attention best on agreement with human rationales. The tasks, datasets and metrics differ, so this page does not turn these numbers into one ranking. The class-specific method on ViT also beats raw attention on the negative perturbation test (54.16 against 45.55), but rollout is close to it there (53.1) 2012.09838 tab. 1. So the gap to raw attention is larger than the gap to rollout.
• Faithfulness ranking. ERASER finds attention less comprehensive than gradient and LIME in all seven rows 1911.03429 tab. 4. The survey reports no such ranking, so this is a single benchmark's result.
• Whether the "attention is not explanation" claim holds. The earlier claim is that different attention can give near-identical predictions. The later paper accepts that such attention exists for some binary tasks, but argues that the claim depends on the definition of explanation, and that the adversary's attention does not guide a simple model well on most tasks 1908.04626 §2 1908.04626 tab. 3. The paper does not show that attention is faithful in general. Its evidence covers binary LSTM classifiers only, so this page does not extend it to other models.
• Plausible versus faithful. The NLP attention survey reports that attention weights may not be a reliable explanation. It cites work showing that attention can disagree with other explainability methods, and that adversarial attention distributions are easy to build in local cases. It concludes that attention may be a plausible explanation without being a faithful one [arxiv:1902.02181 §VI-A]. This matches the plausibility reading that 1908.04626 §5 uses to defend attention, so the two sources agree on the wording and differ on the verdict. The survey reports no experiments of its own, so this page does not count it as evidence either way.

Papers

• Attention in Natural Language Processing: a survey whose section on attention as explanation says attention may be plausible without being faithful; it reports no experiments.
• An Attentive Survey of Attention Models: a survey whose interpretability section covers attention weights as explanations, and the critiques of that use.
• ERASER: A Benchmark to Evaluate Rationalized NLP Models: a benchmark with human-marked rationales and comprehensiveness and sufficiency metrics, comparing attention, gradient and LIME scorers.
• Attention is not not Explanation: four tests of attention in LSTM text classifiers, including a consistent adversary, on binary tasks; author-reported F1 and divergence values.
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers: a gradient-weighted, layer-by-layer relevancy method for co-attention and encoder-decoder Transformers, with author-reported segmentation and perturbation results.
• Transformer Interpretability Beyond Attention Visualization: class-specific relevance propagation through ViT and BERT, with author-reported ImageNet perturbation and segmentation results.
• Visual Attention Methods in Deep Learning: An In-Depth Survey: a vision-attention taxonomy with no experiments of its own; it names interpretability among open problems.
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small: path patching and mean ablation find a 26-head circuit for one behaviour, tested for faithfulness, completeness and minimality.