Structural similarity (SSIM-style comparison of maps)

Overview

Structural similarity (SSIM) scores how alike two signals x and y are by multiplying three comparisons: one of their means (luminance), one of their standard deviations (contrast), and their correlation (structure) 1708.00786 §4, eq. 3. Each factor is 1 when the two signals agree on that property, so the product is 1 for identical inputs. The structure factor σ_xy / (σ_x σ_y) is a correlation and goes negative when the signals are anti-correlated, which makes the whole score negative 1708.00786 eq. 3.

SSIM was introduced for image quality assessment by Wang et al. (2004), which has no card here; this page only uses SSIM as the Structure-measure paper restates and adapts it. That paper uses SSIM-style comparisons to score a non-binary prediction map against a binary ground truth (GT), where the point is to reward predictions that keep the object's shape rather than just counting correct pixels 1708.00786 §3, §4.

The invariant shared by every variant below: compare first- and second-order statistics (mean, spread, covariance) of two signals over some region, and map agreement to a value near 1. In the two S-measure variants the two signals are a prediction and a reference. The no-reference variant from Learning a No-Reference Quality Metric compares two band responses of the same image, so no reference is needed 1612.05890 §4.2. The attribution-map variant compares two explanation maps of the same input, from two methods or two models 2011.05429 §2.2.

Variants

Constant-free global SSIM

The S-measure paper uses SSIM without the stabilising constants, computed over a whole region rather than in a sliding window 1708.00786 eq. 3:

ssim = \frac{2\bar{x}\bar{y}}{\bar{x}^{2}+\bar{y}^{2}}\cdot\frac{2\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}}\cdot\frac{\sigma_{xy}}{\sigma_{x}\sigma_{y}}

The σ_x σ_y terms cancel, so the official code evaluates 4·x̄·ȳ·σ_xy / ((x̄² + ȳ²)(σ_x² + σ_y²) + eps). Without constants it needs explicit handling of degenerate blocks: the code returns 1 when numerator and denominator are both exactly 0 and 0 when only the numerator is (details and code lines on the card). The authors report that applying SSIM globally or with a sliding window did not capture region-level structure in their setting, which motivated the block split below 1708.00786 §4.1.

Region-aware SSIM (block-weighted)

Split both maps into four blocks with one horizontal and one vertical line through the centroid of the GT foreground, compute the constant-free SSIM in each block, and take a weighted sum 1708.00786 §4.1, eq. 4:

S_r = \sum_{k=1}^{K} w_k \cdot ssim(k)

The paper describes splitting recursively into K blocks in spatial-pyramid fashion, but experiments use K = 4, i.e. one split 1708.00786 §4.1, §5. Splitting at the object centroid means each block holds a part of the object, so the sum scores part-level structure. The paper weights blocks by the GT foreground they cover 1708.00786 §4.1; the official code weights by block area (see Contradictions).

Card: Structure-measure: A New Way to Evaluate Foreground Maps.

Object-aware, SSIM-like mean and spread comparison

For the whole-object view, S-measure keeps the SSIM luminance comparison of means and adds a dispersion penalty, the coefficient of variation of the prediction, weighted by λ 1708.00786 §4.2, eq. 5, eq. 6. With GT mean 1 inside the foreground this simplifies to 1708.00786 eq. 7:

O_{FG} = \frac{2\bar{x}_{FG}}{\bar{x}_{FG}^{2}+1+2\lambda\cdot\sigma_{x_{FG}}}

The background gets the same formula applied to 1 − prediction over the GT background (eq. 8), and the two are mixed by the foreground area fraction to give S_o 1708.00786 §4.2, eq. 8, eq. 9. S-measure then averages object and region terms, S = α·S_o + (1 − α)·S_r with α = 0.5 1708.00786 §4.3, eq. 10. λ = 0.5 in the experiments 1708.00786 §5.

How this combined measure compares with pixel-wise scores (AP, AUC, Fbw) is covered on segmentation evaluation.

Structure-only correlation between bands (no reference)

The no-reference SR metric uses a structure term with no mean factor. It is computed between the high-pass and band-pass responses of one image, filtered by a 15×15 Gaussian window with σ = 1.5, with a stabilising constant c₀ 1612.05890 §4.2:

\rho = \frac{2\sigma_{xy} + c_0}{\sigma_x^2 + \sigma_y^2 + c_0}

Compared with the S-measure form, this keeps only the covariance and spread terms, and it is applied to band responses rather than to a prediction and its reference. The paper uses ρ as one input among its 138 features, which are fed to three forests and combined linearly 1612.05890 tab. 5, §4.4. The paper does not test ρ alone: its ablation (tab. 6) compares feature groups, and the all-feature model scores 0.931 against 0.913 for the group labelled GSM, which the card does not define. So the page makes no claim about ρ's separate effect 1612.05890 tab. 6. Constant c₀ has no value in the paper, and the band pairs are described only as "across-scale" 1612.05890 §4.2.

Card: Learning a No-Reference Quality Metric for Single-Image Super-Resolution.

Map-to-map SSIM for explanation maps

Two attribution maps of the same input, from two methods or two models, can be compared with SSIM for visual similarity and with Spearman rank correlation for feature ranking 2011.05429 §2.2. Before SSIM, each map is min-max scaled to [0, 1], or to [-1, 1] when it contains negative values 2011.05429 app. B. The authors' code also clips at a percentile before scaling, which the paper does not mention and which affects SSIM values; the card lists it under Paper vs code. The paper does not state the SSIM window or constants.

The same SSIM is used against ground-truth masks: attributions score 0.78 to 0.98 on the relaxed mask GT-2 in the spurious-background test, and a random Gaussian map scores a mean of 3e-06 against the spurious attributions, which serves as a calibration point 2011.05429 tab. 2. The authors build GT-2 by hand and describe it only in words 2011.05429 §3.1. Across methods, SSIM between attributions of a correctly and a mislabelled copy of the same training input ranges from 0.73 to 0.99, and the authors conclude that these methods are unlikely to find mislabelled examples 2011.05429 §3.2.

Card: Debugging Tests for Model Explanations.

SSIM as a training loss

Restoration networks can also use SSIM during training, as one term of the loss. The raindrop-removal paper trains its refine network on α·(1 − SSIM) + (1 − α)·L1, with α = 0.3 on the SSIM term 2609.37870 §4.3, eq. 5. The paper motivates the mix by structure similarity and colour similarity, citing Zhao et al. (2017), which has no card here 2609.37870 §4.3. The paper calls the SSIM term a loss but writes it as 1 − L_ssim, which treats L_ssim as a similarity; this page reads it as 1 − SSIM 2609.37870 eq. 5. The paper does not state the SSIM window, the stabilising constants or how colour channels are combined 2609.37870 §4.3. Only the refine network uses this term; the paper's reconstruction network is trained with an L1-type loss instead 2609.37870 eq. 3.

Card: Learning from synthetic photorealistic raindrop for single image raindrop removal.

When to use which

Implementation sketch

ssim_global(x, y):                       # constant-free, one region
    mx, my = mean(x), mean(y)
    vx, vy, cov = var(x), var(y), cov(x, y)   # official code: N-1 denominators
    a = 4 * mx * my * cov
    b = (mx^2 + my^2) * (vx + vy)
    if a != 0: return a / (b + eps)
    return 1 if b == 0 else 0            # degenerate cases, code only

region_ssim(pred, gt):
    (X, Y) = rounded centroid of gt foreground
    split pred and gt at row Y, column X into 4 blocks
    return sum_k w_k * ssim_global(pred_k, gt_k)
        # w_k: paper = share of gt foreground in block k; code = block area / image area

mean_spread(v):                          # v = prediction values in one region, target 1
    return 2*mean(v) / (mean(v)^2 + 1 + 2*λ*std(v) + eps)

# two explanation maps of one input; window and constants not stated [arxiv:2011.05429 §2.2]
map_similarity(a, b, signed):
    # min-max scale to [0, 1], or [-1, 1] if signed [arxiv:2011.05429 app. B]
    # official code also clips at a percentile first; paper omits it
    a, b = scale(a, signed), scale(b, signed)
    return ssim(a, b), spearman(rank(a), rank(b))

# no reference: two band responses of one image [arxiv:1612.05890 §4.2]
structure_term(a, b):                    # a, b: high-pass and band-pass responses
    # 15x15 Gaussian window, sigma = 1.5; c0 has no value in the paper
    sxy = windowed cross-covariance(a, b)
    sa2, sb2 = windowed variance(a), windowed variance(b)
    return (2*sxy + c0) / (sa2 + sb2 + c0)

# training loss: SSIM term mixed with L1  [arxiv:2609.37870 eq. 5]
# window and constants not stated; SSIM read as a similarity
refine_loss(clean, out, alpha):          # alpha = 0.3 in the paper
    return alpha * (1 - ssim(clean, out)) + (1 - alpha) * l1(clean, out)

Variants differ in the region the statistics are taken over (whole image, centroid blocks, GT foreground/background), in whether the correlation factor is kept, in whether the second signal is a reference, a second band of the same image, or a second explanation map, and in whether the value is reported as a score or minimised as a loss. Do not substitute a library SSIM if you need S-measure numbers: those add stabilising constants and windows, which the paper's formula does not have 1708.00786 eq. 3, §4.1.

Contradictions

Within the S-measure paper:

• Block weights: the text weights each block by the GT foreground it covers 1708.00786 §4.1; the official code uses the block's share of the image area (see the card).
• λ: experiments use λ = 0.5 1708.00786 §5, while the Fig. 2 caption gives λ = 0.25 1708.00786 fig. 2.

Between the two papers:

• SSIM as a yardstick: the no-reference paper reports SSIM at Spearman 0.635 overall against human SR ratings, with 5-fold CV and retrained baselines 1612.05890 tab. 7. The S-measure paper argues that global SSIM misses region structure for foreground maps 1708.00786 §4.1. The two papers score different things (SR images against ratings, foreground maps against ground truth), so neither result settles the other.
• Full-reference comparison: the no-reference paper notes that its full-reference baselines see the ground-truth image and the proposed metric does not, so the comparison is not like-for-like 1612.05890 §5.2.

Papers

• Structure-measure: A New Way to Evaluate Foreground Maps: adapts SSIM into a constant-free, centroid-block region term and an SSIM-like object term to score non-binary foreground maps.
• Learning a No-Reference Quality Metric for Single-Image Super-Resolution: uses a structure-only correlation between image bands as one feature among 138 in a no-reference SR quality score.
• Debugging Tests for Model Explanations: compares attribution maps with SSIM and rank correlation to test whether they reveal planted model bugs.
• Learning from synthetic photorealistic raindrop for single image raindrop removal: mixes an SSIM term with L1 to train its refine network, and reports SSIM as a test metric.