--- kind: concept slug: structural-similarity title: Structural similarity (SSIM-style comparison of maps) summary: >- SSIM-style measures compare mean (luminance), spread (contrast) and correlation (structure). S-measure applies a constant-free SSIM per block around the object centroid. A no-reference SR metric uses a structure-only correlation between image bands. Explanation maps can be compared with SSIM after min-max scaling, but high SSIM can coexist with weak rank agreement. tags: [evaluation, metric, image-similarity, image-quality, explainability, training-loss] papers: ["1612.05890", "1708.00786", "2011.05429", "2609.37870"] related: [segmentation-evaluation, salient-object-detection, image-quality-assessment, image-super-resolution, feature-attribution, explanation-evaluation] status: draft updated_at: 2026-10-24 --- ## Overview Structural similarity (SSIM) scores how alike two signals `x` and `y` are by multiplying three comparisons: one of their means (luminance), one of their standard deviations (contrast), and their correlation (structure) [arxiv:1708.00786 §4, eq. 3]. Each factor is 1 when the two signals agree on that property, so the product is 1 for identical inputs. The structure factor `σ_xy / (σ_x σ_y)` is a correlation and goes negative when the signals are anti-correlated, which makes the whole score negative [arxiv:1708.00786 eq. 3]. SSIM was introduced for image quality assessment by Wang et al. (2004), which has no card here; this page only uses SSIM as the [Structure-measure](/papers/1708.00786) paper restates and adapts it. That paper uses SSIM-style comparisons to score a non-binary prediction map against a binary ground truth (GT), where the point is to reward predictions that keep the object's shape rather than just counting correct pixels [arxiv:1708.00786 §3, §4]. The invariant shared by every variant below: compare first- and second-order statistics (mean, spread, covariance) of two signals over some region, and map agreement to a value near 1. In the two S-measure variants the two signals are a prediction and a reference. The no-reference variant from [Learning a No-Reference Quality Metric](/papers/1612.05890) compares two band responses of the same image, so no reference is needed [arxiv:1612.05890 §4.2]. The attribution-map variant compares two explanation maps of the same input, from two methods or two models [arxiv:2011.05429 §2.2]. ## Variants ### Constant-free global SSIM The S-measure paper uses SSIM without the stabilising constants, computed over a whole region rather than in a sliding window [arxiv:1708.00786 eq. 3]: $$ ssim = \frac{2\bar{x}\bar{y}}{\bar{x}^{2}+\bar{y}^{2}}\cdot\frac{2\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}}\cdot\frac{\sigma_{xy}}{\sigma_{x}\sigma_{y}} $$ The `σ_x σ_y` terms cancel, so the official code evaluates `4·x̄·ȳ·σ_xy / ((x̄² + ȳ²)(σ_x² + σ_y²) + eps)`. Without constants it needs explicit handling of degenerate blocks: the code returns 1 when numerator and denominator are both exactly 0 and 0 when only the numerator is (details and code lines on the [card](/papers/1708.00786)). The authors report that applying SSIM globally or with a sliding window did not capture region-level structure in their setting, which motivated the block split below [arxiv:1708.00786 §4.1]. ### Region-aware SSIM (block-weighted) Split both maps into four blocks with one horizontal and one vertical line through the centroid of the GT foreground, compute the constant-free SSIM in each block, and take a weighted sum [arxiv:1708.00786 §4.1, eq. 4]: $$ S_r = \sum_{k=1}^{K} w_k \cdot ssim(k) $$ The paper describes splitting recursively into `K` blocks in spatial-pyramid fashion, but experiments use `K = 4`, i.e. one split [arxiv:1708.00786 §4.1, §5]. Splitting at the object centroid means each block holds a part of the object, so the sum scores part-level structure. The paper weights blocks by the GT foreground they cover [arxiv:1708.00786 §4.1]; the official code weights by block area (see Contradictions). Card: [Structure-measure: A New Way to Evaluate Foreground Maps](/papers/1708.00786). ### Object-aware, SSIM-like mean and spread comparison For the whole-object view, S-measure keeps the SSIM luminance comparison of means and adds a dispersion penalty, the coefficient of variation of the prediction, weighted by λ [arxiv:1708.00786 §4.2, eq. 5, eq. 6]. With GT mean 1 inside the foreground this simplifies to [arxiv:1708.00786 eq. 7]: $$ O_{FG} = \frac{2\bar{x}_{FG}}{\bar{x}_{FG}^{2}+1+2\lambda\cdot\sigma_{x_{FG}}} $$ The background gets the same formula applied to `1 − prediction` over the GT background (eq. 8), and the two are mixed by the foreground area fraction to give `S_o` [arxiv:1708.00786 §4.2, eq. 8, eq. 9]. S-measure then averages object and region terms, `S = α·S_o + (1 − α)·S_r` with α = 0.5 [arxiv:1708.00786 §4.3, eq. 10]. λ = 0.5 in the experiments [arxiv:1708.00786 §5]. How this combined measure compares with pixel-wise scores (AP, AUC, Fbw) is covered on [segmentation evaluation](/concepts/segmentation-evaluation). ### Structure-only correlation between bands (no reference) The no-reference SR metric uses a structure term with no mean factor. It is computed between the high-pass and band-pass responses of one image, filtered by a 15×15 Gaussian window with σ = 1.5, with a stabilising constant `c₀` [arxiv:1612.05890 §4.2]: $$ \rho = \frac{2\sigma_{xy} + c_0}{\sigma_x^2 + \sigma_y^2 + c_0} $$ Compared with the S-measure form, this keeps only the covariance and spread terms, and it is applied to band responses rather than to a prediction and its reference. The paper uses ρ as one input among its 138 features, which are fed to three forests and combined linearly [arxiv:1612.05890 tab. 5, §4.4]. The paper does not test ρ alone: its ablation (tab. 6) compares feature groups, and the all-feature model scores 0.931 against 0.913 for the group labelled GSM, which the card does not define. So the page makes no claim about ρ's separate effect [arxiv:1612.05890 tab. 6]. Constant `c₀` has no value in the paper, and the band pairs are described only as "across-scale" [arxiv:1612.05890 §4.2]. Card: [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](/papers/1612.05890). ### Map-to-map SSIM for explanation maps Two attribution maps of the same input, from two methods or two models, can be compared with SSIM for visual similarity and with Spearman rank correlation for feature ranking [arxiv:2011.05429 §2.2]. Before SSIM, each map is min-max scaled to [0, 1], or to [-1, 1] when it contains negative values [arxiv:2011.05429 app. B]. The authors' code also clips at a percentile before scaling, which the paper does not mention and which affects SSIM values; the [card](/papers/2011.05429) lists it under Paper vs code. The paper does not state the SSIM window or constants. The same SSIM is used against ground-truth masks: attributions score 0.78 to 0.98 on the relaxed mask GT-2 in the spurious-background test, and a random Gaussian map scores a mean of 3e-06 against the spurious attributions, which serves as a calibration point [arxiv:2011.05429 tab. 2]. The authors build GT-2 by hand and describe it only in words [arxiv:2011.05429 §3.1]. Across methods, SSIM between attributions of a correctly and a mislabelled copy of the same training input ranges from 0.73 to 0.99, and the authors conclude that these methods are unlikely to find mislabelled examples [arxiv:2011.05429 §3.2]. Card: [Debugging Tests for Model Explanations](/papers/2011.05429). ### SSIM as a training loss Restoration networks can also use SSIM during training, as one term of the loss. The raindrop-removal paper trains its refine network on α·(1 − SSIM) + (1 − α)·L1, with α = 0.3 on the SSIM term [arxiv:2609.37870 §4.3, eq. 5]. The paper motivates the mix by structure similarity and colour similarity, citing Zhao et al. (2017), which has no card here [arxiv:2609.37870 §4.3]. The paper calls the SSIM term a loss but writes it as `1 − L_ssim`, which treats `L_ssim` as a similarity; this page reads it as 1 − SSIM [arxiv:2609.37870 eq. 5]. The paper does not state the SSIM window, the stabilising constants or how colour channels are combined [arxiv:2609.37870 §4.3]. Only the refine network uses this term; the paper's reconstruction network is trained with an L1-type loss instead [arxiv:2609.37870 eq. 3]. Card: [Learning from synthetic photorealistic raindrop for single image raindrop removal](/papers/2609.37870). ## When to use which | Situation | Use | Why | |---|---|---| | Comparing two maps whose overall statistics should match, region-level structure not important | Constant-free global SSIM | Simplest form; but the S-measure authors found global SSIM missed region structure for foreground maps [arxiv:1708.00786 §4.1] | | Scoring a prediction against an object mask where the object's parts should be intact | Region-aware SSIM at the GT centroid | Scores each part of the object separately [arxiv:1708.00786 §4.1]; part of the measure that came closest to the SalCut application ranking on four datasets [arxiv:1708.00786 tab. 1] | | Reference is a binary mask and you want to score the object as a whole | Mean-and-spread comparison per region (object-aware term) | Rewards high, uniform foreground and low, uniform background predictions [arxiv:1708.00786 §4.2] | | Need numbers comparable with published S-measure results | Port the official code, including its block weights and N−1 statistics | Paper and code differ on block weights; see the [card](/papers/1708.00786) | | Cheap per-image evaluation | Any of the above | Full S-measure runs at 5.3 ms per image on one 4 GHz CPU thread in Matlab [arxiv:1708.00786 §5] | | No reference image, and you need a structure feature from one image | Structure-only correlation between bands (ρ) | Needs no reference; the paper uses it only as one of 138 features and reports no separate effect [arxiv:1612.05890 tab. 5, tab. 6] | | Need a reference-free score for super-resolution outputs | The full no-reference metric, not ρ alone | Author-reported Spearman 0.931 under 5-fold CV against 0.919 for CORNIA; the leave-image-out margin is narrow (0.852 vs 0.843) and the metric is validated on one degradation model [arxiv:1612.05890 tab. 7, tab. 8, §3] | | Comparing two explanation maps of the same input and you need to know whether they rank features alike | Map-to-map SSIM together with Spearman rank correlation | High visual similarity can coexist with weak ranking agreement: SSIM 0.92 for one pair while most rank correlations are near zero [arxiv:2011.05429 tab. 3] | | Scoring how well an attribution follows a known region of the input | SSIM against a mask, with a random-map calibration | Attributions reach 0.78 to 0.98 on the authors' GT-2 mask, and a random map scores near 0 [arxiv:2011.05429 tab. 2]; GT-2 is hand-built and only described in words [arxiv:2011.05429 §3.1] | | Training an image-to-image network where colour and structure should both match the target | SSIM term mixed with L1 | The raindrop-removal refine network uses α = 0.3 on the SSIM term [arxiv:2609.37870 eq. 5]; with no window or constants stated, a library SSIM may not reproduce it [arxiv:2609.37870 §4.3] | ## Implementation sketch ```text ssim_global(x, y): # constant-free, one region mx, my = mean(x), mean(y) vx, vy, cov = var(x), var(y), cov(x, y) # official code: N-1 denominators a = 4 * mx * my * cov b = (mx^2 + my^2) * (vx + vy) if a != 0: return a / (b + eps) return 1 if b == 0 else 0 # degenerate cases, code only region_ssim(pred, gt): (X, Y) = rounded centroid of gt foreground split pred and gt at row Y, column X into 4 blocks return sum_k w_k * ssim_global(pred_k, gt_k) # w_k: paper = share of gt foreground in block k; code = block area / image area mean_spread(v): # v = prediction values in one region, target 1 return 2*mean(v) / (mean(v)^2 + 1 + 2*λ*std(v) + eps) # two explanation maps of one input; window and constants not stated [arxiv:2011.05429 §2.2] map_similarity(a, b, signed): # min-max scale to [0, 1], or [-1, 1] if signed [arxiv:2011.05429 app. B] # official code also clips at a percentile first; paper omits it a, b = scale(a, signed), scale(b, signed) return ssim(a, b), spearman(rank(a), rank(b)) # no reference: two band responses of one image [arxiv:1612.05890 §4.2] structure_term(a, b): # a, b: high-pass and band-pass responses # 15x15 Gaussian window, sigma = 1.5; c0 has no value in the paper sxy = windowed cross-covariance(a, b) sa2, sb2 = windowed variance(a), windowed variance(b) return (2*sxy + c0) / (sa2 + sb2 + c0) # training loss: SSIM term mixed with L1 [arxiv:2609.37870 eq. 5] # window and constants not stated; SSIM read as a similarity refine_loss(clean, out, alpha): # alpha = 0.3 in the paper return alpha * (1 - ssim(clean, out)) + (1 - alpha) * l1(clean, out) ``` Variants differ in the region the statistics are taken over (whole image, centroid blocks, GT foreground/background), in whether the correlation factor is kept, in whether the second signal is a reference, a second band of the same image, or a second explanation map, and in whether the value is reported as a score or minimised as a loss. Do not substitute a library SSIM if you need S-measure numbers: those add stabilising constants and windows, which the paper's formula does not have [arxiv:1708.00786 eq. 3, §4.1]. ## Contradictions Within the S-measure paper: - **Block weights:** the text weights each block by the GT foreground it covers [arxiv:1708.00786 §4.1]; the official code uses the block's share of the image area (see the [card](/papers/1708.00786)). - **λ:** experiments use λ = 0.5 [arxiv:1708.00786 §5], while the Fig. 2 caption gives λ = 0.25 [arxiv:1708.00786 fig. 2]. Between the two papers: - **SSIM as a yardstick:** the no-reference paper reports SSIM at Spearman 0.635 overall against human SR ratings, with 5-fold CV and retrained baselines [arxiv:1612.05890 tab. 7]. The S-measure paper argues that global SSIM misses region structure for foreground maps [arxiv:1708.00786 §4.1]. The two papers score different things (SR images against ratings, foreground maps against ground truth), so neither result settles the other. - **Full-reference comparison:** the no-reference paper notes that its full-reference baselines see the ground-truth image and the proposed metric does not, so the comparison is not like-for-like [arxiv:1612.05890 §5.2]. ## Papers - [Structure-measure: A New Way to Evaluate Foreground Maps](/papers/1708.00786): adapts SSIM into a constant-free, centroid-block region term and an SSIM-like object term to score non-binary foreground maps. - [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](/papers/1612.05890): uses a structure-only correlation between image bands as one feature among 138 in a no-reference SR quality score. - [Debugging Tests for Model Explanations](/papers/2011.05429): compares attribution maps with SSIM and rank correlation to test whether they reveal planted model bugs. - [Learning from synthetic photorealistic raindrop for single image raindrop removal](/papers/2609.37870): mixes an SSIM term with L1 to train its refine network, and reports SSIM as a test metric.