Interpreting Super-Resolution Networks with Local Attribution Maps

In plain words

Super-resolution (SR) networks take a small, blurry picture and produce a larger, sharper one. This paper asks which pixels of the small input a network actually uses to draw one given patch of the output. The tool it proposes, LAM, is a heat map over the input for that patch.

LAM borrows an existing idea: measure how much each input pixel moves a score, by walking from a starting point to the real input and adding up the effect along the way. The starting point here is a blurred copy of the input, and the walk gradually sharpens it. Starting from a blurred copy avoids the odd artefacts you get from fading the picture to black.

The authors use the heat maps to compare networks. Networks that look at a wider area of the input tend to do better, and a large receptive field on paper does not by itself mean the network uses more of the image. The results are the authors' own.

TL;DR

• LAM is an attribution method for SR networks, built on integrated gradients §3.3.
• It uses a Gaussian-blurred low-resolution input as baseline and a progressive blur as path, instead of a black baseline and a straight line §3.2 §3.3.
• Each map is summarised by a diffusion index (DI), a rescaled Gini coefficient; a larger DI means more input pixels are involved §4.2 eq. 6.
• Across 48 networks, DI correlates with PSNR and LPIPS (Pearson and Spearman above 0.8) §4.4 fig. 7.
• Author-reported. The paper is a research tool, not a new SR architecture §5.

Problem

Attribution methods were built for classifiers. SR output is one intensity per pixel, so the authors argue for three changes: explain a local patch rather than the whole image, focus on hard patches that the low-resolution input cannot fully determine, and explain the presence of edges and textures rather than raw intensities §3.1. Plain gradients saturate, so the attribution is small where the output is flat §3.2 app. A. Noise-based methods (SmoothGrad, VarGrad) do not suit SR, because adding noise to the input damages the output app. A. Integrated gradients with a black baseline and a linear path gives images that do not look like natural images and does not isolate the missing information §3.4 fig. 3.

Method

LAM is an integrated-gradients attribution §3.3 with three design choices: a patch score, a blurred baseline, and a progressive-blur path.

Patch score. For an l \times l patch at (x, y) of the SR output, the detector D_{xy} is a sum of image gradients over the patch eq. 1. The authors mainly use this gradient detector to measure edges and textures; other simple operators are allowed §3.3.

Baseline. The baseline is the low-resolution image I blurred by a Gaussian kernel of width \sigma, written I' = \omega(\sigma) \otimes I. It removes high-frequency content, which the authors treat as the part that matters for SR §3.3. A kernel width of 0 is the identity (a delta) §3.3.

Path. The path is \gamma_{\mathrm{pb}}(\alpha) = \omega(\sigma - \alpha\sigma) \otimes I. It starts at the blurred baseline (\alpha = 0) and ends at the input (\alpha = 1) eq. 3.

Attribution. For input dimension i, LAM is the path integral of the gradient of the score with respect to the path point, times the path velocity eq. 2. For a black baseline and a straight-line path this reduces to integrated gradients §3.3.

Discrete form. The integral is approximated with m points on the path, and the velocity by the difference between neighbouring points eq. 4. The authors report that 100 steps is enough §3.3. Summing the map over all pixels should give the score at the input minus the score at the baseline, D(F(I)) - D(F(I')). The authors use this as a check of the approximation §3.3.

# Local attribution map for one output patch  [§3.3] [eq. 2] [eq. 4]
# Input:  low-res image (H, W, C) in [0, 1]; sigma; m steps
# Output: attribution map (H, W, C)

def local_attribution_map(image, sigma, steps, detector, model):
    # Blur width falls linearly from sigma to 0 along the path  [§3.3] [eq. 3]
    widths = linspace(sigma, 0, steps + 1)
    # Baseline: the image blurred with width sigma  [§3.3]
    previous = gaussian_blur(image, widths[0])
    attribution = zeros_like(image)
    for k in range(1, steps + 1):
        # Path point at alpha = k/m  [§3.3] [eq. 3]
        current = gaussian_blur(image, widths[k])
        # Gradient of the patch score D(F(.)) at that point  [§3.3] [eq. 2]
        score = detector(model(current))
        gradient = d(score) / d(current)
        # Gradient times path increment, summed; no extra 1/m  [§3.3] [eq. 4]
        attribution += gradient * (current - previous)
        previous = current
    return attribution

Why not vanilla gradients. The authors show that the score rises sharply over a short range of the path, where the path velocity is also large. The path-integrated gradient collects weight there, while plain gradients at the input are small §3.4 fig. 3f–h.

Why this baseline and path. A black baseline lowers all pixel intensities at once, so the gradients along the path saturate. A linear path gives every point the same velocity, I - I' §3.4. The blurred baseline and progressive path are the authors' choice for SR §3.4 fig. 3.

Visualisation. Attribution maps are normalised to [-1, 1] and the absolute value is shown. Heat maps over many patches use kernel density estimation §4.1.

Hyperparameters

Results

Author-reported. Setup: ×4 super-resolution, 150 challenging test sub-images with average PSNR 20.87 dB §4.1. Table 1 does not restate the test set. The rows are the receptive field, PSNR and DI of representative networks tab. 1.

Other author-reported results:

• Across the 48 networks, DI against PSNR and DI against LPIPS both have Pearson and Spearman correlations above 0.8, with two-tailed p-values below 1 \times 10^{-12} §4.4 fig. 7a–b. The x-axis is the log of DI §4.4.
• Receptive field does not track DI: RRDBNet's receptive field already exceeds the test-image size, and compared with it, IMDN, RFDN and RNAN do not significantly use more pixels, although they score higher PSNR §4.4 tab. 1. The authors conclude that the receptive field of existing networks is large enough and that using the pixels inside it is the bottleneck §4.4.
• For FSRCNN, more depth and width both raise DI and PSNR. For EDSR, more residual blocks raise DI and PSNR, but more feature maps do not §4.4 fig. 7c.
• LAM maps show that shallow networks (FSRCNN, CARN) use a small area. Deep residual networks and non-local or attention networks use pixels farther away §4.3 fig. 5. Some wide-range networks (SAN, DRLN in the second example) still reconstruct wrong textures §4.3.
• Images with regular stripes or grids have high DI. Complex high-level content such as humans and animals is not effectively used §4.4 fig. 8.

Paper vs code

The code is the authors' demo repository, X-Lowlevel-Vision/LAM_Demo, at code_commit. The organisation owns the project page that the paper links §1 footnote. The repository contains the attribution code (SaliencyModel/), model definitions and some pretrained weights (ModelZoo/). It does not contain the training code, the test set, or the 30 scale-variant models. The project page's Colab notebook was not inspected. The repository has no LICENSE file, so no code is reproduced in this card; every code reference is a citation of location and value.

• Path derivative and 1/m. The paper's v2 removed a 1/m factor from the approximation and says the code has no such problem app. D. The code scales the finite-difference path velocity by the number of steps (fold) BackProp.py, accumulates the gradient times this velocity BackProp.py, and then averages over steps BackProp.py BackProp.py. The two factors cancel, so the net result is a plain sum, as in the v2 form of eq. 4.
• Sign and index of the finite difference. As printed, eq. 4 uses \gamma(k/m) - \gamma((k+1)/m) for k = 1..m. These differences do not telescope to I - I', which the paper's completeness check requires §3.3. The code takes the gradient at \alpha = k/m and multiplies by the difference from the previous point, \gamma(k/m) - \gamma((k-1)/m), which does telescope to I - I' BackProp.py BackProp.py. This is our inference from the paper's own completeness claim and the code, not a statement by the authors.
• Detector. The paper's detector is a signed sum of gradients over the patch eq. 1. The code's gradient detector is the magnitude of forward differences, the square root of the sum of squared horizontal and vertical differences attributes.py. The code also has an intensity-sum detector attributes.py and a Gabor detector attributes.py. The paper does not describe either. The Gabor parameters appear only in a comment attributes.py.
• Patch window. The paper gives a 16×16 patch §4.1. The code applies the window to the SR output BackProp.py. At ×4 that is a 4×4 block of low-resolution pixels. The paper does not say which space the 16×16 refers to.
• Blur kernel size. The paper gives no kernel size §3.3. The progressive path uses a 5 \times 5 kernel by default BackProp.py. The baseline used by the other method in the code (see below) uses a 7 \times 7 kernel utils.py.
• Second method in the code. saliency_map_I_gradient takes a blurred baseline with a straight-line path, with fold=10 and the gaus baseline as defaults BackProp.py. This is integrated gradients with a blurred baseline. It is not the progressive path that the paper calls LAM, and the paper does not describe it.
• Visualisation. The paper normalises to [-1, 1] then takes absolute values §4.1. The code sums over channels utils.py, then takes absolute values and divides by the maximum utils.py. The code's colour map is seismic, which maps the value v to \mathrm{cmap}(0.5v + 0.5), so larger values are redder, not darker utils.py. The paper says darker means larger influence §4.1.
• Gini. The code's Gini function sorts the values and shifts negative values to zero, and adds 10^{-7} to avoid zeros utils.py. The paper's formula eq. 5 has no offset. The difference is small, but it affects the DI of maps with many zeros.
• Training. The paper's training settings app. B cannot be checked: the training code is not in the repository. The repository includes pretrained weights for some networks, such as EDSR, RCAN, SAN, RNAN, CARN and RRDBNet .

Pitfalls

• Implement the plain sum, not the v1 form. The v1 preprint had a 1/m factor that v2 removed app. D. Implement the sum over k of gradient times path increment, as in the Paper vs code section. Any extra 1/m divides every map by the step count.
• Use the telescoping difference. Check your implementation by summing the map: it should equal the score at the input minus the score at the baseline §3.3. The printed eq. 4 fails this check as written, so do not copy its indices.
• Blur units. The baseline blurs the low-resolution input §3.3, so \sigma is in low-resolution pixels. The paper gives no value for \sigma or the kernel size, and the repository uses a 5 \times 5 kernel for the path BackProp.py. Calibrate \sigma against the paper's figures rather than assuming a value.
• Border handling is not stated. The code uses OpenCV's default border behaviour in filter2D BackProp.py BackProp.py. A different border mode changes the baseline near the image edge.
• Gradient operator is not stated. The paper does not define the discrete gradient in eq. 1. The code's version is a magnitude, not a signed sum. Results with a different operator will not match the paper's DI values.
• Patch coordinates. Decide whether the patch is in the LR or SR grid; the code uses the SR output.
• Cost. One forward and one backward pass of the full image for each of the m steps, for every patch. With m = 100 §3.3 this is 100 passes per patch. The paper reports no runtime. The code loops over steps one image at a time BackProp.py.
• DI depends on the test set. The 150 test images were picked for low PSNR and high variance between networks, then checked manually §4.1. The selection rule and the manual step are not reproducible from the paper alone, and the repository ships only seven small test images . DI values are not comparable with other test sets.
• Correlation is not causation. The correlations are over 48 networks, with DI on a log scale §4.4. The authors use them as evidence that wider pixel use helps, not as a causal test.
• Map size and channels. The paper does not say whether the Gini coefficient is computed over the full map, one channel, or all channels eq. 5 §4.2.
• Mostly qualitative evidence. The quantitative results are Table 1 and the correlations in §4.4. The comparisons of networks in §4.3 are visual fig. 5 fig. 6.

Lineage

• Builds on: Integrated Gradients (Sundararajan et al., 2017), which uses a black baseline and a straight path §3.2 app. A. It is not compiled on this wiki.
• Compared against: plain gradients, gradient times input, Guided Backpropagation, SmoothGrad, VarGrad, CAM and GradCAM app. A §3.2.
• Applied to: SR networks listed in §4.4, including FSRCNN, EDSR, RCAN, SAN and RRDBNet.
• Superseded by: not known from this wiki.