Structure-measure: A New Way to Evaluate Foreground Maps

In plain words

When a model predicts which pixels belong to the main object in a photo, you need a score that says how good the prediction is. The usual scores count pixels: how many were right, how many were wrong. Two predictions can get the same pixel count even though one keeps the object's shape intact and the other punches a hole through its outline, and people clearly prefer the first.

Structure-measure (S-measure) tries to score shape rather than pixel tallies. It looks at the prediction in two ways. First, it cuts the image into four pieces around the centre of the true object and asks, piece by piece, whether the prediction's light and dark patterns line up with the ground truth, using a classic image-similarity formula (SSIM). Second, it looks at the object as a whole: inside the true object, are the predicted values high and evenly spread? Outside it, are they low and evenly spread? The final score is the average of the two views, between 0 and 1.

The measure is cheap and simple to compute. In the authors' tests it ranked saliency maps more like a downstream segmentation tool and more like human viewers did than the older pixel-based scores.

TL;DR

S-measure evaluates a non-binary foreground map against a binary GT as S = 0.5·S_o + 0.5·S_r. S_r splits both maps into 4 blocks at the GT foreground centroid and takes a weighted sum of per-block SSIM (no stabilising constants). S_o compares mean and standard deviation of predicted values inside the GT foreground and inside the GT background, weighted by foreground area. Authors report best or tied-best scores on most of 3 meta-measures over 4 saliency datasets and 63.7%–73.6% human preference over Fbw/AP/AUC picks. Read the Pitfalls: the paper and the official Matlab code disagree on the block weights, and λ is reported inconsistently.

Problem

Salient object detection models output non-binary maps; standard evaluation (AUC, AP, and the weighted F-measure Fbw) reduces them to pixel-wise TP/TN/FP/FN counts and so ignores object structure §2 §3. Two maps with identical confusion counts can differ greatly in how much they damage the object's shape: a square hole inside a digit versus one that cuts its boundary yields the same counts fig. 3. A realistic case shows Fbw ranking a fuzzy, top-only pyramid detection above a structurally complete one fig. 3 §3. The paper assumes a binary GT map §1 and focuses on non-binary predictions §1.

Method

S-measure combines a region-aware structural similarity S_r (part-level) and an object-aware structural similarity S_o (whole-object level) §4. Let x be the SM (prediction) pixel values and y the GT values, with means x̄, ȳ, standard deviations σ_x, σ_y and covariance σ_xy §4.

SSIM without constants eq. 3:

ssim = \frac{2\bar{x}\bar{y}}{\bar{x}^{2}+\bar{y}^{2}}\cdot\frac{2\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}}\cdot\frac{\sigma_{xy}}{\sigma_{x}\sigma_{y}} eq. 3

The three factors are luminance, contrast and structure comparison §4. The code evaluates the algebraically equivalent form 4·x̄·ȳ·σ_xy / ((x̄²+ȳ²)(σ_x²+σ_y²) + eps) S_region.m.

Region-aware term §4.1:

1. Split SM and GT into four blocks with one horizontal and one vertical line crossing at the centroid of the GT foreground §4.1. The paper says the sub-images are then split recursively (spatial-pyramid style), with K total blocks §4.1; experiments use K=4 §5, i.e. a single split, and the code only implements a single split S_region.m.
2. Compute ssim(k) per block with eq. 3 §4.1.
3. Weight each block by w_k and sum:

S_r = \sum_{k=1}^{K} w_k \cdot ssim(k) eq. 4

The paper says w_k is proportional to the GT foreground area covered by block k §4.1; the code uses the block's share of the image area instead S_region.m (see Paper vs code).

Object-aware term §4.2:

Foreground mean comparison, analogous to SSIM luminance eq. 5:

O_{FG} = \frac{2\bar{x}_{FG}\bar{y}_{FG}}{\bar{x}_{FG}^{2}+\bar{y}_{FG}^{2}} eq. 5

Adding the coefficient of variation of the SM foreground as a dispersion penalty gives a dissimilarity eq. 6:

D_{FG} = \frac{\bar{x}_{FG}^{2}+\bar{y}_{FG}^{2}}{2\bar{x}_{FG}\bar{y}_{FG}} + \lambda\cdot\frac{\sigma_{x_{FG}}}{\bar{x}_{FG}} eq. 6

With ȳ_FG = 1 (binary GT), the similarity is its reciprocal eq. 7:

O_{FG} = \frac{1}{D_{FG}} = \frac{2\bar{x}_{FG}}{\bar{x}_{FG}^{2}+1+2\lambda\cdot\sigma_{x_{FG}}} eq. 7

The background is handled as the complement: take 1 − SM and 1 − GT, and apply the same formula over the GT background region §4.2 fig. 4:

O_{BG} = \frac{2\bar{x}_{BG}}{\bar{x}_{BG}^{2}+1+2\lambda\cdot\sigma_{x_{BG}}} eq. 8

With μ the GT foreground area fraction §4.2:

S_o = \mu\cdot O_{FG} + (1-\mu)\cdot O_{BG} eq. 9

Final measure §4.3:

S = \alpha\cdot S_o + (1-\alpha)\cdot S_r,\quad \alpha\in[0,1] eq. 10

Algorithm: S-measure, as implemented in the official code
Input: SM in [0,1] (double), GT (boolean), same size H×W
1. m = mean(GT)
   if m == 0: return 1 − mean(SM)                  [code: StructureMeasure.m#L28-L30]
   if m == 1: return mean(SM)                      [code: StructureMeasure.m#L31-L33]
2. Object term                                      [code: S_object.m#L16-L27]
   fg = SM values where GT;   O_FG = 2·mean(fg) / (mean(fg)² + 1 + std(fg) + eps)
   bg = 1 − SM values where ¬GT; O_BG = same formula on bg   [code: S_object.m#L52-L57]
   S_o = m·O_FG + (1 − m)·O_BG
3. Region term                                      [code: S_region.m#L15-L31]
   (X, Y) = round(centroid of GT), 1-based column/row; image centre if GT empty
                                                    [code: S_region.m#L45-L54]
   blocks: LT = rows 1..Y, cols 1..X; RT = rows 1..Y, cols X+1..W;
           LB = rows Y+1..H, cols 1..X; RB = rows Y+1..H, cols X+1..W
                                                    [code: S_region.m#L79-L82]
   weights: w1 = X·Y/(H·W), w2 = (W−X)·Y/(H·W), w3 = X·(H−Y)/(H·W), w4 = 1 − w1 − w2 − w3
                                                    [code: S_region.m#L84-L88]
   per block: means x̄,ȳ; variances and covariance with (N − 1 + eps) denominator
              a = 4·x̄·ȳ·σ_xy; b = (x̄² + ȳ²)(σ_x² + σ_y²)
              ssim = a/(b + eps) if a ≠ 0; 1 if a == 0 and b == 0; else 0
                                                    [code: S_region.m#L119-L144]
   S_r = Σ w_k · ssim_k
4. S = 0.5·S_o + 0.5·S_r; clamp to 0 if negative   [code: StructureMeasure.m#L35-L39]
Output: S

Hyperparameters

Results

Author-reported. Meta-measures 1–4 come from Margolin et al. (Fbw paper); meta-measure 5 is new §5. ASD1000 experiments use 5000 maps from CA, CB, RC, PCA and SVO with λ=0.5, K=4 §5. Further datasets use 10 models (ST, DRFI, DSR, DCL, rfcn, MC, MDF, DISC, DHS, ELD) §5.5. Lower is better for all MM numbers below. MM1 = 1 − Spearman ρ against SalCut application ranking §5.1; MM2 = % of images where a generic map beats the mean state-of-the-art score §5.2; MM3 = % of times a score rises when a wrong GT is used §5.3.

Table 1 (MM1–MM3 on four datasets) tab. 1:

Other headline results:

Using threshold 0.95 on SalCut binary outputs, S-measure flags 66.80%, 67.30%, 81.82% and 83.03% of images as structurally damaged on ECSSD, HKU-IS, PASCAL-S and SOD; F_β at the same threshold flags 63.76%, 65.43%, 78.32% and 82.67% §1. Ranking by S-measure puts DHS (dhsnet), DCL and rfcn on top among the 10 models §5.7 fig. 13.

Paper vs code

• Block weights. Paper: w_k proportional to the GT foreground area each block covers §4.1. Code: block pixel area divided by image area, independent of GT content, although the code comment repeats the paper's wording S_region.m.
• Recursive split. Paper describes recursive subdivision into K blocks §4.1; code does exactly one split into 4 S_region.m, consistent with K=4 in experiments §5.
• λ. Code hard-codes the denominator x̄² + 1 + σ + eps, equal to eq. 7 with λ=0.5 S_object.m eq. 7, matching §5; the Fig. 2 caption says λ=0.25 fig. 2.
• Degenerate GT. Not in the paper. If GT is all background the code returns 1 − mean(SM); if all foreground, mean(SM) StructureMeasure.m.
• Clamping. Code clamps negative S to 0 StructureMeasure.m; the paper does not mention negative values, though per-block SSIM is negative whenever σ_xy < 0.
• SSIM zero cases. Code returns 1 when both numerator and denominator are exactly 0 (e.g. a constant block matching a constant GT block) and 0 when only the numerator is 0 S_region.m; not in the paper.
• Object term masking. Code zeroes SM outside the region before calling Object, but Object reads only pixels inside the mask, so the zeroing has no effect S_object.m S_object.m.
• The repo README cites the later IJCV 2021 version of the paper; the code at this commit may track that version rather than the ICCV 2017 arXiv text.

Pitfalls

• Weights decide comparability. Implementing §4.1 literally (foreground- proportional weights) gives different numbers from the official code. Decide deliberately and say which.
• λ ambiguity. Fig. 2 uses λ=0.25, experiments λ=0.5 fig. 2 §5. Use 0.5 (2λσ = σ) to match the code.
• Std normalisation. Code uses N−1 (sample) standard deviation in the object term and N−1 variances/covariance in SSIM S_object.m S_region.m. NumPy's default std is N; use ddof=1 to match. Matters for small regions.
• Indexing convention. Centroid is computed 1-based and rounded; block LT/RT include row Y, LT/LB include column X S_region.m S_region.m. A 0-based port must shift by one or blocks and weights differ by a row/column.
• Empty blocks. If the rounded centroid lands on the last row or column, a block is empty; the code's mean2 of an empty array would then be NaN and could propagate through w_k·ssim_k even with w_k = 0. Guard this in a port.
• Input contract. Code requires SM as double in [0,1] and GT as logical, and errors otherwise StructureMeasure.m. Normalise 8-bit maps (divide by 255) and binarise GT first; how to binarise a non-binary GT is not stated.
• No SSIM constants. Unlike standard SSIM, there are no C1/C2; stability relies on eps and explicit zero-case branches. Do not substitute a library SSIM (which uses constants and sliding windows); the paper reports global/sliding-window SSIM failed to capture region structure §4.1.
• MM4 excluded. S-measure was not best on MM4, and the authors argue MM4 is unreliable and drop it from the multi-dataset comparison §5.4; the multi-dataset table therefore covers only MM1–MM3 tab. 1.
• Ties and losses. On SOD MM2, S-measure ties AP at 9.67; on ECSSD MM2, Fbw (3.00) beats it (3.30) tab. 1.
• User-study selection. Pairs where the two measures' top maps looked too similar were removed (100 → 50 pairs) §5.6, which can inflate preference rates relative to a random sample.

Lineage

• Builds on: SSIM (Wang et al. 2004, image quality assessment) for the region term §4; spatial-pyramid recursive division (Lazebnik et al. 2006) for block splitting §4.1; meta-measures from Margolin et al. 2014 (Fbw) §5.
• Compared against: AP, AUC, Fbw (weighted F-measure) §5 tab. 1.
• Superseded by: not known from content in this wiki.