Full-Gradient Representation for Neural Network Visualization

In plain words

Saliency maps are pictures that show which parts of an input a neural network relied on for its answer. Many of the common recipes for drawing them disagree with each other, and some fail simple sanity checks. This paper argues that no single per-pixel picture can capture everything you would want from one.

The authors propose a different starting point. They split the network's output into two kinds of pieces: how much each input value pushes the output, and how much each internal "detector" (neuron) and its bias contributes. Added back together, these pieces give exactly the network's output. For ordinary image networks they then fold the pieces into a single heat map called FullGrad, which is an approximation rather than an exact decomposition.

The payoff is that the maps look sharper and stay closer to object regions, and the authors report that they hold up better on two tests that remove pixels the map calls important. The tests are described below; the reported results are the authors' own.

TL;DR

Full-gradients decompose a bias-containing ReLU network's output exactly into input-gradient times input plus bias-gradient terms eq. 1. FullGrad turns these into one saliency map for convolutional networks by post-processing each term and summing them eq. 2. The authors report it best among the compared methods on pixel perturbation and remove-and-retrain §5.1 §5.2. The aggregated map is approximate, and the paper proves that for piecewise-linear functions a saliency map cannot in general be both complete and weakly input-dependent [prop. 1].

Problem

Saliency maps assign one importance score per input feature. The paper argues that two intuitive properties conflict: weak dependence on inputs (local importance, which holds for input-gradient methods) and completeness (the scores add up to the output, which holds for integrated gradients, DeepLIFT and similar) cannot both hold for a saliency map in general §3 [prop. 1]. The paper also says popular evaluation tests have artifacts: removing pixels can create edge artifacts, and retraining can change what the model uses §2 §5.1 §5.2.

Assumptions: the exact decomposition needs a ReLU network. Biases must be part of the decomposition: the paper argues that input-gradient methods are insensitive to some bias changes and so fail a parameter-randomization test §4.1.

Method

1. Exact decomposition without biases. For a ReLU network with no bias parameters, the output equals the input-gradient dotted with the input [prop. 2].

2. Exact decomposition with biases. Let the network have F biases B. The output is the input-gradient dotted with the input, plus the bias-gradient dotted with B eq. 1 [prop. 3]. The paper calls the input-gradient and the bias-gradient together the full-gradients §4. Bias-gradients are written f^b(x) = ∇_b f(x; B) ⊙ B §4. "Biases" include implicit ones, such as the running averages of batch-norm layers §4.

3. Non-ReLU layers. Linearize a non-linearity y = σ(x) around x, giving y = σ'(x)·x + b_σ. The leftover b_σ is an implicit bias. It is zero for ReLU-like functions, and the paper appends such terms to B §4.

4. Properties. Input-gradients and bias-gradients are constant in each linear region of a piecewise-linear network, which gives weak dependence §4.1 [supp. Full-gradient Proofs]. Eq. 1 gives completeness §4.1. Including biases lets the method attribute a function's output to its biases in a case where input-gradients give zero attribution §4.1. Changing any parameter that changes the function also changes the full-gradients §4.1.

5. Convolutional layers. For a convolution with one scalar bias per filter, the bias-gradient has the same spatial shape as the input. Each output location depends only on a local window of input positions, so the bias-gradient map shows importance for groups of pixels §4.2.

6. FullGrad. Post-process each term with ψ, then add them up eq. 2:

# FullGrad saliency map for a CNN  [eq. 2] [§4.2]
# Input:  image (1, C, H, W); target_class (output unit index)
# Output: saliency (1, 1, H, W)

def fullgrad_saliency(model, image, target_class):
    # Input-gradient times input, post-processed  [eq. 2] first term
    input_grad = score_gradient_wrt_image(model, image, target_class)
    input_term = postprocess(input_grad * image)     # (1, C, H, W)
    # Summing over channels is not stated in the paper; see Paper vs code
    saliency = input_term.sum_over_channels()        # (1, 1, H, W)

    # Bias-gradient terms, one per convolutional layer  [§4.2] [eq. 2]
    for each convolutional layer with a bias:
        # Gradient of the output score w.r.t. this layer's output [§4]
        output_grad = gradient_wrt_layer_output(layer)  # (1, c_l, h, w)
        # Each channel multiplied by that channel's bias  [§4] [§4.2]
        bias_grad = output_grad * layer.bias_per_channel
        # Post-process each channel on its own  [eq. 2]
        for each channel c in bias_grad:
            bias_map = postprocess(bias_grad[:, c])  # (1, h, w)
            # Upsample each map to the image size  [§4.2]
            saliency += resize_bilinear(bias_map, (H, W))

    return saliency

Here ψ is abs, then a linear rescale to [0, 1], then bilinear upsampling §4.2. The paper states that dropping ψ and summing spatially recovers the exact output §4.2. Because ψ is applied, the FullGrad map is approximate §4.2. Changing the aggregation (for example, removing the input multiplication in the input term) trades completeness against weak dependence §4.2. The paper recommends choosing ψ according to the metric and dataset, and gives a digit-flipping case where the signed variant (no abs) does better §6 tab. 1.

Hyperparameters

Results

Author-reported. Setup: pixel perturbation on ImageNet 2012 validation with VGG-16 (batch norm) §5.1; remove-and-retrain on CIFAR100 with a 9-layer VGG §5.2; qualitative comparison on ImageNet images §5.3.

The paper gives no numeric values for the ImageNet or remove-and-retrain curves in the text; they appear only as plots fig. 2a fig. 2b. Qualitatively, FullGrad causes the least change in output under pixel perturbation §5.1 and decreases accuracy the most under remove-and-retrain §5.2. Visual comparisons against input-gradients, integrated gradients, SmoothGrad and Grad-CAM appear in fig. 3.

Digit flipping, MNIST (higher is better), reproduced from tab. 1:

MNIST pixel perturbation (lower is better), reproduced from tab. 2:

The paper's own summary is that no single method wins on both MNIST tests [supp. Digit Flipping] [supp. Pixel Perturbation].

Paper vs code

Searched: the abstract, the LaTeX source and the arXiv metadata contain no code link and no project page. A GitHub repository search for fullgrad under the idiap organisation (the authors' institute) returned idiap/fullgrad-saliency, which the code section below uses. Only the method and a demo are released; the code has no pixel-perturbation or remove-and-retrain scripts.

• ψ applied per channel vs per layer. Paper: ψ is applied to each channel inside the sum over channels §4.2 eq. 2. Code: abs and rescale are applied to the whole layer tensor, with min and max taken over all channels and positions fullgrad.py, and that call is made per layer fullgrad.py. These give different maps.
• Batch-norm layers are aggregated. Paper: spatial maps only for convolutional layers §4.2. Code: the extractor registers batch-norm layers too tensor_extractor.py, and the aggregation keeps every 4-D bias-gradient fullgrad.py, which includes batch-norm outputs.
• Channel sum of the input term. Paper: the input term ψ(∇x f ⊙ x) is written without a channel reduction §4.2 eq. 2. Code: sums over channels after ψ fullgrad.py.
• Bilinear upsampling. Paper: "bilinear" §4.2. Code: bilinear with align_corners=True fullgrad.py. The paper does not specify corner alignment.
• Implicit bias of batch norm. Paper: running averages count as implicit biases §4; no formula is given. Code: the effective bias is −running_mean × weight / sqrt(running_var + eps) + bias tensor_extractor.py.
• Non-ReLU layers. Paper: linearized non-linearities contribute b_σ §4. Code: only conv, linear and batch-norm biases are collected tensor_extractor.py; there is no b_σ term.
• Demo vs experiments. Paper's experiments use VGG-16 with batch norm and a 9-layer VGG §5.1 §5.2. The released demo uses ResNet-18 dump_images.py. The released repository does not contain the experiments.
• Epsilon in the rescale. Code adds 1e-6 to the divisor fullgrad.py; the paper does not mention it.

Pitfalls

• Per-channel vs per-layer ψ changes the map. Implement the paper's per-channel version if you follow eq. 2; if you want the released behaviour, follow the code and say so.
• Which layers count. The paper says convolutional layers; the code also includes batch norm. Decide which you want and state it. The paper gives no reason for either choice.
• Input-term channel reduction and ψ scope are not stated in the paper. The released code applies ψ before summing over channels.
• Completeness is only exact before ψ. After ψ and upsampling, the sum no longer equals the output §4.2. Do not use the FullGrad map as a reconstruction of the logit.
• Activation coverage. The exact identities hold for ReLU networks [prop. 2] [prop. 3]. Other non-linearities need the b_σ terms, which the code does not compute. The code's completeness check runs on a random input and the argmax, using a relative tolerance of 1e-4 fullgrad.py; treat a failure as a warning that biases are missing.
• Bias-gradients for fully connected layers are not used in the map §4.2. Their contribution is therefore not in FullGrad, though it is in the exact decomposition.
• Metric and ψ are coupled. The paper's own digit-flipping result prefers the signed variant, while pixel perturbation and remove-and-retrain prefer the abs variant §6 tab. 1 tab. 2. Choose the metric first.
• Pixel perturbation here removes the least salient pixels, not the most salient ones, to reduce artifact effects §5.1. Results are not comparable to benchmarks that remove the most salient pixels.
• ROAR cost. Each evaluation needs retraining §5.2; the paper gives no training budget.
• Table 1 caption and columns disagree. The caption calls the variant "FullGrad (minimal)" while the column and text say "FullGrad (no abs)" tab. 1. Use the column name.
• Table 2 RF is undefined. The paper never says what RF measures, so the rows cannot be reproduced from the text alone tab. 2.
• Hooks and cost. The code uses backward hooks on each conv, linear and batch-norm module tensor_extractor.py and needs one backward pass per target class.
• Smooth-grad squared settings are not stated §5.2, so the baseline cannot be reproduced exactly.

Lineage

• Builds on: input-gradients, guided backprop and deconvolution as weak-dependence saliency methods §2 §3; integrated gradients, DeepLIFT and deep Taylor decomposition as completeness-based methods §2 §3.
• Compared against: input-gradients, integrated gradients, SmoothGrad (squared variant for ROAR), Grad-CAM §5.1 §5.2 §5.3.
• Superseded by: not stated in this wiki.