Debugging Tests for Model Explanations

In plain words

A trained image classifier can be wrong for hidden reasons: it may have picked up a shortcut in the training photos, been trained on mislabelled examples, lost some of its learned weights, or been shown pictures unlike its training data. Explanation tools try to help by drawing a heat map over an image that shows which parts mattered for a prediction. The authors ask whether these heat maps actually reveal such problems.

They planted four kinds of bugs in controlled experiments. Heat maps did reveal a background shortcut, where the model had learned that a certain scene means a certain animal. They did not reliably reveal wrong labels in the training data. Several popular methods produced almost the same heat map even after the top layers of the network were scrambled, so they could not show that the network had changed. Out-of-distribution images also produced heat maps that looked like those of a healthy model.

The authors then ran a study with people who had to decide whether a dog-breed classifier was fit to sell. The participants mostly judged the model by its predicted labels and paid less attention to the heat maps. The practical lesson is that a heat map alone is weak evidence that a model is working.

TL;DR

• The paper tests which feature attribution methods help diagnose which kinds of model bugs: spurious correlation, mislabelled training examples, re-initialised weights and out-of-distribution inputs §1.
• Attributions highlight the spurious background, but do not reliably separate mislabelled training examples from correct ones §3.1 §3.2.
• Modified back-propagation methods (Guided Backprop, DConvNet, PatternNet, PatternAttribution, LRP-SPAF) stay visually and quantitatively unchanged when top layers are re-initialised, so they cannot reveal that kind of bug §4.
• In a 54-person user study, participants relied mainly on predicted labels rather than attributions §1 §3.1 app. H.

Problem

Model debugging is hard, and explanations are increasingly used for it, yet little guidance exists on which explanation method works for which bug §1. The paper asks which explanation methods are effective for which classes of model bugs §1. The bugs studied are unintentional; deliberate manipulation of explanations is out of scope §1. The common evaluation by input perturbation can give inconsistent rankings of methods §1, so the authors use controlled, planted bugs with known ground truth instead §2.1.

Method

Bug taxonomy §2.1 tab. 1

The authors group bugs by where in the supervised pipeline they arise. The learning objective is written as a minimisation of a loss over the training data X_train, Y_train with respect to the parameters θ, and prediction as f_θ applied to a test input x_test §2.1 (display equations are unnumbered in the source).

1. Data contamination: defects in training inputs or labels. Two tested examples: a spurious correlation (for instance a background that co-occurs with a class) and wrong labels §2.1 tab. 1.
2. Model contamination: defects in the parameters, for example weights re-initialised by accident §2.1 tab. 1.
3. Test-time contamination: defects in the test input, such as domain shift or a pre-processing mismatch §2.1 tab. 1.

Appendix A adds frozen layers and pre-processing mismatch to the formal table tab. 4 app. A. These two are formalised only; no experiments are reported for them.

Attribution methods assessed §2.2 app. B

A feature attribution method is a function E that takes a model F, an input x and an output F_k(x) and returns a map of the same dimension as the input §2.2. The methods, grouped as the paper does:

• Gradient family: Gradient; SmoothGrad (average gradient over noisy copies of the input); SmoothGrad Squared; VarGrad (variance of the noisy gradients); Input-Gradient (gradient times input) §2.2 app. B.
• Path methods: Integrated Gradients (IntGrad) integrates the gradient along a straight line from a baseline to the input; Expected Gradients (EGrad) averages this over training-set baselines §2.2 app. B.
• Surrogate models: LIME and SHAP (KernelSHAP in the tables) fit a simple local model around the input §2.2 app. B.
• Modified back-propagation: DConvNet and Guided Backprop change the ReLU gradient rule; LRP (LRP-EPS, LRP-SPAF) redistributes relevance with propagation rules; PatternNet and PatternAttribution back-propagate only a signal component §2.2 app. B.

Visual similarity between maps is measured with SSIM, and feature ranking with Spearman rank correlation §2.2. Before comparison, maps are normalised to [0, 1] for SSIM, and to [-1, 1] when they contain negative values app. B.

The main human study uses only Gradient, SmoothGrad and Integrated Gradients, chosen as a representative subset §2.3 app. H.

Pseudocode of the gradient-family and path methods, following §2.2 and app. B and checked against the code at utils.py:

# Gradient: sensitivity of the class logit to each input value  [§2.2]
def gradient_map(model, x, k):
    return abs(input_gradient(model.logit, x, k))

# Helper: gradients at N Gaussian-noised copies of x  [app. B]
# Noise std = 0.15 * (max(x) - min(x)); N = 50 in the paper  [app. B]
def noisy_gradients(model, x, k, n_samples=50):
    sigma = 0.15 * (max(x) - min(x))
    grads = []
    for i in range(n_samples):
        noisy = x + normal(0, sigma, shape(x))
        grads.append(input_gradient(model.logit, noisy, k))
    return grads

# SmoothGrad: mean of the noisy gradients  [§2.2] [app. B]
def smoothgrad(model, x, k):
    return mean(noisy_gradients(model, x, k))

# SmoothGrad Squared: element-wise square of SmoothGrad  [app. B]
def smoothgrad_squared(model, x, k):
    s = smoothgrad(model, x, k)
    return s * s

# VarGrad: element-wise variance of the noisy gradients  [app. B]
def vargrad(model, x, k):
    return variance(noisy_gradients(model, x, k))

# Integrated Gradients: path integral from a baseline to x  [§2.2] [app. B]
# Baseline = per-dimension minimum of the input range  [app. B]
# Step count is not stated in the paper  [code: userstudy/src/utils.py#L98]
def integrated_gradients(model, x, k, baseline, n_steps=25):
    total = zeros_like(x)
    for alpha in linspace(0, 1, n_steps):
        point = baseline + alpha * (x - baseline)
        total = total + input_gradient(model.logit, point, k)
    return (x - baseline) * total / n_steps

# Expected Gradients: IntGrad averaged over training-set baselines  [app. B]
# 200 training examples are used as baselines  [app. B]
def expected_gradients(model, x, k, train_set, n_baselines=200):
    total = zeros_like(x)
    for baseline in sample(train_set, n_baselines):
        total = total + integrated_gradients(model, x, k, baseline)
    return total / n_baselines

Procedure 1: spurious background, birds vs dogs §3.1

1. Train a CNN with 5 convolutional and 3 fully connected layers, ReLU activations and a sigmoid output (BVD-CNN) on birds vs dogs §3.1 app. C.
2. Place every bird on one sky background and every dog on a bamboo background, so the background predicts the class §3.1 fig. 3.
3. Check that the model learned the shortcut: it reaches 97% accuracy on a sky-versus-bamboo test set that contains no animals §3.1 app. D.
4. Compare attribution maps for the spurious inputs with two ground-truth masks: an ideal mask that gives all relevance to the background, and a relaxed version that weights that mask by the background attribution §3.1 fig. 4.

Procedure 2: mislabelled training examples §3.2

1. Flip the labels of 10% of the birds-vs-dogs training samples and train the BVD-CNN §3.2 app. D.
2. For a given training input, compare attributions from the model trained with the correct label and from the model trained with the flipped label §3.2 fig. 6.

Procedure 3: model contamination by cascading re-initialisation §4

# Re-initialise a pretrained network from the top layer downwards  [§4]
# The model is VGG-16 pretrained on ImageNet  [§4] [app. C]
def cascading_reinit_test(model, images, original_maps):
    for layer in layers_top_down(model):
        reinitialise(layer)  # initialiser not stated  [§4]
        maps = explain(model, images)
        compare(maps, original_maps)  # visual and quantitative  [fig. 7]

Procedure 4: test-time domain shift §5

# Same input, attributions from an in-domain model and another model  [§5]
def domain_shift_test(in_domain, other_model, inputs):
    for x in inputs:  # Fashion-MNIST images in the main text  [§5]
        a = attribution(in_domain, x)
        b = attribution(other_model, x)
        visual = ssim(a, b)  # visual similarity  [§2.2]
        ranking = spearman(rank(a), rank(b))  # ranking similarity  [§2.2]

Human study design §2.3 §3.1 app. H

1. Participants act as quality-assurance testers for a company that sells animal classifiers §2.3 app. H.
2. Each sees the image, the model's predicted breed and an attribution map, for four dog breeds at a time §2.3.
3. They rate on a 5-point Likert scale how likely they are to recommend the model for sale, and give a reason, chosen from four fixed answers or written freely §2.3 app. H.
4. Conditions are a normal model and four defective ones: a top-layer re-initialised model, a model trained on random labels, a spurious-background model, and out-of-distribution breeds app. H.
5. Participants were recruited through a university mailing list; 54 completed the study. More than 80% reported prior machine-learning experience app. H. The study was described as IRB-approved in the introduction §1.

Hyperparameters

Values are from the compiled paper unless the source column says [code]. Where the paper is silent, "not stated" is written.

Results

Author-reported. Setup: BVD-CNN on birds vs dogs for §3 and the Fashion-MNIST domain-shift test for §5; VGG-16 on ImageNet for §4; ResNet-50 on dogs for the user study app. H. Hardware not stated. Baselines are the normal (unmodified) model or the in-domain model for each comparison.

Spurious background, SSIM against ground-truth masks tab. 2 (SEM across 190 inputs; calibration: mean SSIM between a random Gaussian attribution and the spurious attributions is 3e-06).

Headline: GT2 scores range from 0.78 (KernelSHAP) to 0.98 (VGrad) §3.1 tab. 2. The authors read this as evidence that the attributions pick up the spurious background.

Mislabelled training examples §3.2 fig. 6. SSIM between attributions for the correctly labelled and the mislabelled version of the same training input ranges from 0.73 to 0.99 across all methods tested. The authors conclude these methods are unlikely to identify mislabelled examples.

Model contamination §4 fig. 7. Reported qualitatively, with a caption that refers to quantitative results but no table of numbers in the compiled text. Modified back-propagation methods (Guided Backprop, DConvNet, DeepTaylor, PatternNet, PatternAttribution, LRP-SPAF) stay visually and quantitatively unchanged as higher layers are re-initialised §4.

Test-time domain shift, Fashion-MNIST input tab. 3 fig. 8. Maps are visually similar across models, especially VGG-16 on ImageNet, but rankings are not §5.

Caption of tab. 3: SEM over 190 examples. SSIM reaches 0.92 for VGrad (FMNIST → MNIST model). Most rank correlations are near zero. The largest are 0.51 for GBP (FMNIST → MNIST model), 0.43 for SmoothGrad Squared (FMNIST → BVD-CNN) and 0.37 for SmoothGrad Squared and VGrad (FMNIST → MNIST model).

Human study §3.1 §3.2 §4 §5 app. H.

• Normal model: median Likert scores 4, 4 and 3 for Gradient, SmoothGrad and Integrated Gradients §3.1. Gradient and SmoothGrad are rated higher than Integrated Gradients (Tukey, p < 0.0001) app. H.
• Spurious model: medians 2, 2 and 3. A majority said the attribution did not highlight the part they expected, yet participants did not reject the model convincingly §3.1.
• Mislabelled model: medians 1, 2 and 1; participants rejected the model, mainly on the basis of wrong predicted labels §3.2.
• Top-layer re-initialised model: rejected on labels, rarely because of the attribution §4.
• Out-of-distribution breeds: not recommended, because of wrong predicted labels §5.

Appendix H one-way ANOVA results and means, as reported:

The authors summarise the study as showing that, with labels present, users reliably decline models trained on random labels and models with model-contamination defects, but are unsure about spurious models §6 app. H.

Paper vs code

The code is the authors' repository at commit 98f3b211c97c1bbbaec4d12cc8c102ecbbc1afe7 README.md. It contains the user-study pipeline (data splits, spurious and random-label dataset builders, ResNet-50 training, attribution utilities and a demo notebook). It does not contain code for the birds-vs-dogs BVD-CNN, the VGG-16 cascade, the MNIST and Fashion-MNIST experiments, the SSIM and rank-correlation metrics, or the influence and TCAV checks. The debuggingtests folder has only split lists and a one-line README README.md.

Discrepancies between the paper and the code:

• Random labels touch validation and test too. The paper says all training labels are randomised app. H. randomlabels.py shuffles labels within each split, so validation and test labels are shuffled as well randomlabels.py.
• Spurious backgrounds touch validation and test too. Backgrounds are replaced in every split, not only in training spurious.py. The paper's text on this point refers only to training points app. H.
• Augmentation is not in the paper. Training uses random horizontal and vertical flips and rotations up to 90 degrees; the validation generator uses the same flips and rotations train_resnet50.py.
• Training length and early stopping. The compiled paper does not state them. The code defaults to 20 epochs with early stopping after 10 epochs without improvement train_resnet50.py.
• Pretrained weights. The model is built with ResNet50() and no weights argument, then its final layer is replaced train_resnet50.py. The paper does not state which initial weights are used.
• SmoothGrad sample count. The paper states 50 app. B; the code default is 25 utils.py and the demo notebook uses 10 visualize_heatmaps.ipynb. The experiment settings are not in the repo.
• Integrated Gradients baseline. The paper uses a minimum-value (black) baseline app. B. The code default is an all-zeros array utils.py, and the demo notebook does not pass a baseline visualize_heatmaps.ipynb.
• Integrated Gradients discretisation. The code samples both endpoints with linspace(0, 1, n) and divides the sum by n utils.py. This is a specific discretisation; the paper gives only the integral.
• Normalisation. The paper describes min-max scaling of maps app. B. The code also clips at the 99th percentile of the summed absolute or signed map utils.py. The paper does not mention the percentile clip, which affects SSIM values.
• Gradient target. The code takes the gradient of the pre-softmax logit utils.py, which matches the paper's logit-based description app. B.
• Dataset sizes. The paper's 350/25/25 dog split app. H does not match the partition files, which list about 320 to 325 training lines and 50 test lines per breed for beagle and pug beagle_train.txt. The birds-vs-dogs lists have 4300 training, 500 validation and 190 test lines per class dogs_train_list.txt.
• README command. The README command passes --reg and --condition --normal, which train_resnet50.py does not define README.md train_resnet50.py. The command as written would fail.
• Old dependencies. Requirements pin Keras 2.2.x and TensorFlow 1.8 requirements.txt requirements.txt. The utilities call set_random_seed and K.get_session from that API utils.py utils.py, and the data scripts use Image.ANTIALIAS spurious.py. Expect to adapt them to run on current libraries.

Pitfalls

• Numbering. Display equations are unnumbered in the source, so they are cited by section, not as eq. N. Figure and table numbers in this card follow the order of captions in the source; they were not checked against a rendered PDF.
• Uncompiled sources. The source archive includes userstudy.tex and explanationmethods.tex, which the paper does not include. They describe a different user study (seven conditions, 10-way breed task, partial-spurious and half-way conditions, and their own accuracies). This card does not use them.
• Mislabelled-data accuracy disagrees within the paper. The main text gives 93.2% training accuracy §3.2, the appendix 94.2% app. D. Both are kept in the table.
• MNIST architecture disagrees. The main text describes a BVD-CNN trained on MNIST and Fashion-MNIST §5; the appendix says the MNIST inputs were converted to three channels and attributed with a VGG-16 model app. C. The architecture actually used for these models is therefore unclear.
• "Two 200 images." The ImageNet sample size in the model-contamination section is ambiguous app. C.
• Likely typo in table 2. The SGradSQ entry for SSIM-GT1 is 0.063, far below the other values in the row tab. 2. Check it against the rendered PDF before relying on it.
• Ground-truth masks are constructed by the authors. The relaxed mask GT-2 is described only in words §3.1. No code for it is in the repo, so reproducing the SSIM numbers needs an implementer to rebuild it from the text.
• Model contamination lacks numbers. The cascade result is reported only visually and qualitatively in the compiled text §4 fig. 7.
• Degrees of freedom. The ANOVA degrees of freedom are reported as (2, 54), using the total sample size app. H. If 54 participants are split over three methods, the usual within-group degrees of freedom would be 51. This is an inference from the design; check the statistical report.
• Small, preliminary checks. The influence-function and TCAV results come from 50 examples and are flagged by the authors as needing more evaluation app. G.
• Scope of the human study. The task is 10-way dog breed recognition, which the authors say may make participants rely on labels more than on attributions. They say it is unclear whether users would still rely on predictions for tasks where they lack expertise §6.
• Repo does not cover most experiments. The birds-vs-dogs, VGG-16 cascade, MNIST and test-time experiments have no code in the repository, so only the user-study pipeline can be checked against the paper.

Lineage

• Builds on: the sanity checks of Adebayo et al. (2018) for saliency maps. The data and model contamination categories recover those checks §1. Also cites the finding that modified back-propagation methods converge to a rank-1 matrix (Sixt et al., arXiv:1912.09818) §4.
• Compared against: input-perturbation evaluations with retraining §1; human-subject tests of attributions such as Poursabzi-Sangdi et al., arXiv:1802.07810 §1.
• Related methods discussed in the limitations: TCAV (concept activation) and influence functions §6 app. G.
• Superseded by: not known from content already in this wiki.