wikillm

Paper · 2021-04-27 · cs.LG, cs.CV

Do Feature Attribution Methods Correctly Attribute Features?

Builds a ground-truth test for feature attribution. It edits natural datasets (bird photos, beer reviews) so that a known added cue decides the label, then checks whether saliency maps, attention weights and rationale models highlight that cue. None of the tested methods consistently did. Results are author-reported.

Feature attributionRationale extraction (models that show the supporting input)Attention interpretabilityExplanation evaluation (judging whether an XAI explanation is good)interpretabilityattributionevaluationsaliency

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

When a model reaches high accuracy, a natural question is whether it uses the thing we care about or something accidental, such as a watermark on the photo. Explanation tools (saliency maps, attention weights, rationales) point at the parts of an input that they say mattered. This paper asks whether those pointers are right.

The problem is that on ordinary data nobody knows which parts the model truly uses, so there is nothing to check the pointers against. The authors build data where the answer is known. They take a normal dataset and scramble the labels so that the original content is no longer a reliable clue. Then they add a small, known manipulation that does predict the label. A correct explanation tool should highlight exactly that added region.

They run this test on saliency maps for bird photos, and on attention and rationale models for beer reviews. None of the tested methods consistently pointed at the added cue. Treat these tools as claims to verify, not as ground truth.

TL;DR

  • The authors build semi-natural datasets with known ground truth. Labels are partly reassigned, and a known image or text manipulation is added, so an accurate model must rely on it §4.
  • They test five saliency methods on a ResNet-34 trained on bird images §5, attention weights on BeerAdvocate reviews §6, and two rationale models §7.
  • No method consistently gives the known region close to full attribution. Attention on the planted article cue is only barely above the word-frequency baseline §6.1.
  • The official MIT-licensed repository has the training and manipulation scripts. The Attr% computation, the text dataset construction and the SHAP background call are not in it.

Problem

Feature attribution methods are used to check that an accurate model relies on genuine signal rather than artifacts. On natural data the true reasons are unknown, so the methods cannot be checked directly, and proxy metrics lack ground truth §1 §2.

The paper states two requirements. Features the model truly uses should receive attribution (Attr% close to 1). Features that are not informative should receive none §3. For selected-feature explanations, selecting every feature trivially satisfies the first requirement, so the paper uses precision and recall instead §3.

Method

Dataset modification §4 fig. 2

  1. Label reassignment. In the binary case the label is kept with probability r and flipped otherwise. The accuracy reachable without the manipulation is then at most max(r, 1 - r). At r = 0.5 no original feature is informative §4.
  2. Input manipulation. Each reassigned example gets the manipulation tied to its new label. A blank manipulation leaves the input unchanged. The effective region (ER) of a manipulation is the set of input features it changes. The joint ER is the union over all manipulations §4.
  3. Guarantee. If a model trained on the modified data beats the bound from step 1, it must rely on a manipulation, and that dependence sits inside the joint ER §4. The paper also bounds how often a finite test set can beat the bound by chance, using a binomial tail §4.
  4. Remark. The joint ER must cover every class, because a model may legitimately use the absence of a cue. Restricting the ER to the target class's manipulation can wrongly reject correct attributions §4.
Pseudocode
# Label reassignment, then the manipulation tied to the new label  [§4]
# Input:  images (N, H, W, 3); labels (N,); R (K, K) reassignment matrix
# Output: manipulated images (N, H, W, 3); new labels (N,)

def build_semi_natural(images, labels, R, manipulations):
    new_images = []
    new_labels = []
    for image, label in zip(images, labels):
        # Keep the label with probability R[y, y], otherwise draw another  [§4]
        new_label = sample_label(R[label])
        # Blank manipulation leaves the image unchanged  [§4]
        new_image = manipulations[new_label](image)
        new_images.append(new_image)
        new_labels.append(new_label)
    return new_images, new_labels

Metrics

  • Attr% of a feature set F is the sum of absolute attribution values inside F, divided by the sum over all features §3. For a near-perfect model with r = 0.5, Attr% of the joint ER should be about 1 whatever the ER size §5.1.
  • For selected-feature explanations, precision Pr(F) = |F ∩ F_C| / |F| and recall Re(F) = |F ∩ F_C| / |F_C|, where F_C is the set of features the model truly uses §3.
Pseudocode
# Share of attribution mass inside the known region  [§3] [§5.1]
# Input:  saliency (H, W), signed or unsigned; region (H, W) boolean
# Output: scalar in [0, 1]

def attr_percent(saliency, region):
    # Paper sums absolute attribution values  [§3]
    magnitude = abs(saliency)
    # Mass inside the joint effective region  [§4]
    inside = magnitude[region].sum()
    return inside / magnitude.sum()

Shapley analysis for r > 0.5 §5.4. For r > 0.5 no standard attribution value exists on the original and manipulated features. The paper uses the Shapley split of two feature groups, then normalises the two values by their sum §5.4:

v(F_M) = \tfrac{1}{2}\,[\,a(F_M) - a(\varnothing) + a(F_M \cup F_O) - a(F_O)\,] eq. 1

v(F_O) = \tfrac{1}{2}\,[\,a(F_O) - a(\varnothing) + a(F_M \cup F_O) - a(F_M)\,] eq. 2

Here a(·) is the expected accuracy when only the named features are available, defined through conditional prediction rather than blacking out pixels [App. B.6]. For a classifier with accuracy p, the paper gets bounds on the normalised values eq. 3 eq. 4:

\bar v(F_M) \ge (2p - r - 0.5)\,/\,(2p - 1) eq. 3

\bar v(F_O) \le (r - 0.5)\,/\,(2p - 1) eq. 4

For p close to 1 this gives v̄(F_M) ≥ 1.5 - r and v̄(F_O) ≤ r - 0.5 §5.4.

Pseudocode
# Normalised Shapley split of two feature groups  [§5.4] [eq. 1] [eq. 2]
# Inputs: expected accuracies a(.) for each subset of features
# Output: normalised values for manipulated and original features

def shapley_split(acc_man, acc_orig, acc_none, acc_both):
    # a(empty) is 0.5 for balanced labels  [App. B.6]
    # Shapley value of the manipulated group  [eq. 1]
    value_man = 0.5 * (acc_man - acc_none + acc_both - acc_orig)
    # Shapley value of the original group  [eq. 2]
    value_orig = 0.5 * (acc_orig - acc_none + acc_both - acc_man)
    total = value_man + value_orig
    return value_man / total, value_orig / total

Attention model §6. A sentence of L words becomes 200-dimensional embeddings. A bidirectional LSTM gives 400-dimensional states h_i. Attention then pools the states, with a learned query vector q in R^200 eq. 5 to eq. 8:

\vec k_i = \tanh(\mathrm{Linear}(\vec h_i)) \in \mathbb{R}^{200} eq. 5

b_i = \vec q \cdot \vec k_i eq. 6

a_1, \dots, a_L = \mathrm{softmax}(b_1, \dots, b_L) eq. 7

\vec h = \sum_{i=1}^{L} a_i \vec h_i eq. 8

Pseudocode
# Attention-pooled sentence vector  [§6] [eq. 5] [eq. 6] [eq. 7] [eq. 8]
# Input:  states h (L, 400); learned layer W, b; query q (200,)
# Output: pooled vector (400,) and word weights (L,)

def attention_pool(h, W, b, q):
    # Project each state to 200 dimensions and squash  [eq. 5]
    k = tanh(h @ W + b)                  # (L, 200)
    # Score each word against the query  [eq. 6]
    scores = k @ q                       # (L,)
    # Normalise the scores over the words  [eq. 7]
    weights = softmax(scores)            # (L,)
    # Weighted sum of the states  [eq. 8]
    pooled = weights @ h                 # (400,)
    return pooled, weights

Rationale models §7. Two models are compared: a reinforcement-learning model (RL, after Lei et al. 2016) and a continuous-relaxation model (CR, after Bastings et al. 2019). Instead of a length penalty, both are trained to hit a target selection rate Sel. The paper's penalty for a mini-batch of B examples is λ times the absolute gap between the batch selection rate and Sel. The discontinuity penalty is removed §7.

Pseudocode
# Selection penalty towards target rate Sel, in the paper's form  [§7]
# Input:  z (B, T) binary rationale mask; mask (B, T) real tokens
# Output: scalar penalty, to be added with weight lam

def selection_penalty(z, mask, sel, lam):
    # Rationale tokens and review tokens over the whole mini-batch  [§7]
    rationale_len = (z * mask).sum()
    review_len = mask.sum()
    # Absolute gap between the batch selection rate and the target  [§7]
    return lam * abs(rationale_len / review_len - sel)

Hyperparameters

Paper-stated values come with a section citation. Values the paper leaves out are marked "not stated" and, where the code sets them, given with a code citation. Code citations are relative to the repository root.

NameValueWhereNotes
Image modelResNet-34, random initialisation, output layer replaced by a linear layer on 512 features§5 train.pyPaper says not pretrained on ImageNet
Image size and normalisation224 x 224 centre crop; ImageNet mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225)§5 train.pyThe crop is not in the repository
Images per class1,200 images scraped from Flickr, split 1000 / 100 / 100 (train / val / test)§5Dataset not in repository; no source-licence detail given
Class pairsFour confusable bird pairs: brandt's and pelagic cormorant; fish crow and common raven; forster's and common tern; pomarine and long-tailed jaeger§5 dm_attr_vs_er.py
Label reassignment r0.5 for Attr vs ER and visibility; 0.5 to 1.0 in steps of 0.1 for original correlation§5.1 §5.4 dm_orig_corr_strength.py
Manipulated classOnly the positive class after reassignment§5.1 dm_attr_vs_er.py
Runs100 (Attr vs ER), 100 (visibility), 120 (original correlation)§5.1 §5.3 §5.4 dm_attr_vs_er.py5 manipulations x 20 runs; 5 x 6 x 4 for the third
OptimiserAdam with library defaults; no learning rate is written in the codetrain.pyNot stated in paper
Learning-rate scheduleReduceLROnPlateau on validation accuracy, mode max, patience 30, factor 0.1train.py train.pyNot stated in paper
Batch size, epochs128; up to 50 epochstrain.pyNot stated in paper
StoppingStop once validation accuracy exceeds 0.97train.pyNot stated in paper
CheckpointBest validation accuracytrain.py
LossCross-entropytrain.py
GradientGradient of the predicted class logit, absolute value, summed over channels§B.2 saliency.py
SmoothGrad50 noisy copies; noise standard deviation 0.15 x (max - min) of the image§B.2 saliency.py saliency.py
GradCAMClass activation mapping, since ResNet-34 is fully convolutional; negative values clipped§B.2 saliency.pyClipping is not in the paper
LIME200 samples; hide colour 0; quickshift superpixels§B.2 saliency.pyQuickshift is the paper's stated choice
SHAPGradientExplainer, batch 128; background is the test set§B.2 saliency.pyThe call site that passes the background is not in the repository
Attr vs ER manipulationsBlur radius U(80, 130), sigma 2; brightness radius U(40, 112), shift -0.3; hue stripe 20 to 224 wide, shift 0.04; noise stripe 20 to 224 wide, p 0.02; watermark 40 to 200 high, 100 to 200 widedm_attr_vs_er.pyPaper defines the families; these ranges are code only
Visibility levelsBlur sigma {2, 4, 6, 8, 10}; brightness shift {-0.1 to -0.3}; hue shift {0.05 to 0.25}; noise p {0.02 to 0.1}; font size {7, 9, 11, 13, 15}§B.5 dm_visibility.py dm_visibility.py dm_visibility.py dm_visibility.py dm_visibility.py
Original-correlation manipulationsBlur (radius 80, sigma 10); brightness (radius 60, shift -0.3); hue (top 80, bottom 130, shift 0.15); noise (top 80, bottom 130, p 0.03); watermark (top 150, bottom 200, left 30, right 194, font 13)dm_orig_corr_strength.py
Watermark fontOpenSans-Regularmanipulations.pyPaper does not name the font
Text dataBeerAdvocate; 10,000 train, 1,000 validation, 1,000 test, class-balanced§6 data_utils.pyConstruction of the modified sets is not in the repository
Maximum length256 tokensdata_utils.py
Embeddings200-dimensional pre-trained vectors, file shipped in the repository§6 embedding.txt.gz
Attention encoderBidirectional LSTM, 200 units per direction§6 attention_model.py
Attention scorerLinear 400 to 200, tanh, then a bias-free linear 200 to 1 (the query)§6 eq. 5-eq. 8 attention_model.py
Attention trainingAdam, weight decay 1e-5, batch 250, up to 100 epochs, gradient-norm clip 5attention_model.py attention_model.pyNot stated in paper
Attention embeddingsFrozen unless the training flag is givenattention_model.py attention_model.pyNot stated in paper
Attention stoppingStop once dev accuracy reaches 0.97; keep the best dev accuracyattention_model.py
Rationale encoderRCNN by default; LSTM is an optionutil.pyNot stated in paper
Rationale optimiserAdam, learning rate 4e-4, weight decay 2e-6, batch 256, up to 100 epochs, gradient-norm clip 5util.pyNot stated in paper
Rationale sizes and dropoutEmbedding 200, hidden 200, attention size 100, dropout 0.2util.pyNot stated in paper
Dependent selectionLSTM over the selection decisions (on by default)util.py model_helpers.pyNot stated in paper
Selection target SelDefault 0.13; article experiment {0.07, 0.09, 0.11, 0.13, 0.15}; mixing experiment {0.03, 0.05, 0.07, 0.09}§7.1 §7.2 util.py
Regularisation strength λDefault 1.0util.pyPaper says only that λ > 0
Rationale stoppingStop only when dev accuracy is at least 0.97 and the selected rate is within 0.01 of Sel; keep the best loss, NLL and accuracy checkpointsrationale_model.pyPaper does not say which checkpoint it reports

Results

Author-reported. Setup: ResNet-34 on bird images §5; attention and rationale models on BeerAdvocate reviews §6 §7.

SettingMetricBaseline or targetThis paperSource
Image, 100 runs, near-perfect modelsAttr% of the joint ER, SHAP (best method), average at ER 40%Attr% of 1 expected69%; no method consistently near 1§5.1 fig. 5
Image, 100 runsShare of runs above 95% test accuracyn/a70% (Attr% reported for these runs)§5.1
Image, Attr vs visibilityTrend of Attr% as visibility risesExpected to riseOnly SHAP on watermark rises consistently§5.3 fig. 7
Image, 120 runsAttr% vs r against the Shapley range 1.5 - rRange of Attr%For watermark, SHAP falls as r rises and gradient tracks the range for the positive class only; the paper finds no such trend for other feature types§5.4 fig. 8
Attention, article cueAttr% on article wordsWord frequency 7.9%8.6%, over 97% accuracy§6.1 fig. 10
Attention, 11 runsAttr% on correlating and non-correlating articlesWord frequencyHigh variance across runs; see table below§6.2 tab. 1
Rationale, article cue, Sel 0.07 to 0.15Precision and recallNear-perfect selectionNear-perfect except the CR model at the lowest Sel; values appear only in the figure§7.1 fig. 12
Rationale, mixing, Sel 0.03 to 0.09Precision of correlating articlesDataset-wide optimumRL misses correlating articles and selects non-correlating ones, giving markedly lower precision than the optimum; the shown example is RL, and the appendix example where CR focuses on correlating articles is in fig. 22§7.2 fig. 13

Attention statistics from tab. 1, 11 training runs. Each cell is mean % Attr ± standard deviation, then the word frequency.

SetCue-linked articlesUnlinked articlesRemaining tokens
Article10.3% ± 2.4% | 7.9%NA89.7% ± 2.4% | 92.1%
CN15.9% ± 25.7% | 4.1%5.9% ± 4.0% | 3.8%78.2% ± 24.7% | 92.1%
NC12.0% ± 8.4% | 3.8%12.6% ± 9.3% | 4.1%75.4% ± 16.7% | 92.1%

Paper vs code

Code: official repository YilunZhou/feature-attribution-evaluation, MIT licence, commit 4f4b2aed0ac07a69e63c0502eede4cc61b614e12. Searched: the paper's abstract link, which is the repository itself.

  1. Rationale penalty form. Paper: λ times the absolute gap between the ratio of summed rationale lengths to summed review lengths, over a mini-batch §7. RL code: the per-example gap is averaged over the batch and enters only as a REINFORCE reward rl.py rl.py. CR code: the mean of per-example ratios is compared with Sel latent.py latent.py. The paper's ratio of sums and the code's mean of ratios differ when review lengths vary.
  2. Watermark digits. Paper: four random digits §B.1. Code: the first four characters of a random UUID, upper-cased, so letters A to F can appear manipulations.py.
  3. SmoothGrad. Paper: averages the gradients §B.2. Code: takes the absolute value of each noisy gradient before averaging saliency.py.
  4. Unused arguments. The rationale script imports learning-rate schedulers and parses a --dist option, but neither changes training rationale_model.py util.py. The learning rate stays constant.
  5. Missing from the repository. The bird dataset (the README refers to a bird_dataset folder), the Attr% computation, the text dataset construction for article, CN and NC sets, and the call that passes the SHAP background README.md saliency.py.
  6. Regularisation strength. The paper gives only λ > 0 §7. Code default is 1.0 util.py.
  7. No discrepancy found for the Gradient definition, the SmoothGrad noise level, the attention equations eq. 5 to eq. 8, or the discontinuity penalty, which the code omits as the paper says latent.py.

Pitfalls

  • Attr% is reported only for the runs above 95% test accuracy. About 70% of image runs cleared that bar §5.1. The "Attr% ≈ 1" expectation assumes a near-perfect model, so do not apply it to weak models.
  • Attention's baseline is word frequency. Article words are 7.9% of tokens, so the 8.6% result is only slightly above that baseline §6.1. The spread across runs is large; the correlating-article row in tab. 1 has a standard deviation of 25.7%. Single-run results can mislead.
  • Text figures are not numbers. Rationale precision and recall are shown only in fig. 12 and fig. 13. The text refers to the attention table as a figure, "Fig. (left)" §6.2; cite it as tab. 1.
  • The rationale stopping rule may never fire. Training stops only when accuracy is at least 0.97 and the selection rate is within 0.01 of Sel rationale_model.py. Otherwise it runs all 100 epochs. The paper does not say which checkpoint it reports rationale_model.py.
  • RL selection penalty is high-variance. It enters the update as a REINFORCE reward, not as a direct gradient rl.py. The paper reports RL performing worse than CR on the mixing task §7.2 §7.3.
  • The a(F_M) = p result is mechanical. Under the paper's Shapley definition, a(F_M) = p follows from the axioms, but the paper says whether this is reasonable is outside its scope [App. B.6]. Do not read it as a claim about models.
  • Manipulation ranges are code-only. The paper gives the families and visibility levels, but the random ranges used per run appear only in the code dm_attr_vs_er.py dm_visibility.py.
  • Hidden cost. Each modified image dataset is about 650 MB; the full set of experiments is about 500 GB README.md (step 3).
  • Attention's dataset rule. In the article cue, positive reviews get "the" and negative reviews get "a" in every article position §6.1. The CN and NC constructions are in the paper only and their scripts are not in the repository.
  • Joint ER over all classes. Using only the target class's manipulation region can wrongly reject correct attributions, because a model may rely on absence §4 Remark. Compute the union over all manipulations.
  • Version. This card follows the arXiv v2 LaTeX source. The licence was taken from wikillm fetch (CC BY 4.0).

Lineage

  • Builds on: the saliency methods it tests (Gradient, SmoothGrad, GradCAM, LIME, SHAP) §5; the attention architecture it follows closely §6; and the rationale models of Lei et al. (2016) and Bastings et al. (2019) §7. None of these is extended beyond the evaluation.
  • Compared against: two dataset-based evaluations that use ground truth, as discussed in App. A. The paper's own critique of those works is cited here; nothing from their content is used.
  • Superseded by: not stated.

Source and license

Adapted from “Do Feature Attribution Methods Correctly Attribute Features?” by Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, Julie Shah (arXiv:2104.14403), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/YilunZhou/feature-attribution-evaluation at commit 4f4b2aed0ac0 (MIT). Any quoted code is shown under that license.