---
kind: concept
slug: segmentation-evaluation
title: Segmentation evaluation (scoring foreground, saliency and segmentation masks)
summary: >-
  How to score a predicted mask, box or saliency map against ground truth. Pixel-count measures (IoU, Dice, F-beta, pixel AP, AUC, Fbw) ignore object shape; S-measure also rewards intact structure. Detection uses AP over box IoU thresholds. A consensus framework maps the problem to metrics. Trade-off: comparability with prior work versus agreement with human and downstream judgements.
tags: [evaluation, segmentation, saliency, metric]
papers: ["1708.00786", "1803.01207", "1907.06781", "1908.08297", "1910.07655", "2011.07631", "2101.07172", "2102.04306", "2102.10274", "2105.09451", "2206.01653", "2312.02228", "2404.09556"]
related: [salient-object-detection, structural-similarity, semantic-segmentation, instance-segmentation, object-detection, referring-segmentation, multimodal-language-model, benchmark-validity, dice-loss]
status: draft
updated_at: 2026-11-01
---

## Overview

A segmentation, detection or salient-object model produces a result that has
to be compared with a ground-truth (GT) mask. Results come in two shapes. A
saliency model outputs a map, often non-binary. A segmentation model outputs a
binary mask, typically after thresholding a probability map. The measures
differ by shape.

For non-binary maps, the usual measures (AP, AUC and the weighted F-measure
Fbw) all boil the comparison down to per-pixel true/false positive and negative
counts [arxiv:1708.00786 §2, §3]. Counts like these cannot tell apart errors
that keep the object's shape from errors that break it: a square hole in the
middle of a digit and a square hole cutting its outline give the same counts
[arxiv:1708.00786 fig. 3]. In a realistic example, Fbw ranks a blurry detection
covering only the top of a pyramid above one that captures the whole shape
[arxiv:1708.00786 §3, fig. 3].

For binary masks, the common measures are overlap ratios: Jaccard (IoU), Dice
and F-beta, all computed from pixel counts [arxiv:2011.07631 eq. 1, eq. 3,
eq. 4, eq. 7]. These also ignore where the errors fall relative to the object.
For object detection the score is average precision (AP) over IoU thresholds,
which is a different AP from the pixel-wise AP used for saliency maps
[arxiv:2011.07631 §V-A, eq. 2]. The two papers use the same abbreviation for
different quantities, so check which one a reported number refers to.

Medical volumes add a boundary distance in millimetres, the Hausdorff distance
(HD). The TransUNet paper does not say whether its HD is HD95, but its released
code reports HD95 [arxiv:2102.04306 §4.1] (see the
[card](/papers/2102.04306), Paper vs code).

Beyond single measures, the Metrics Reloaded consensus framework treats metric
choice as a process: fingerprint the problem, walk a decision tree to a pool of
metrics, then apply them with stated pitfalls [arxiv:2206.01653 § Step 1,
§ Step 2, § Step 3]. It covers image-level classification, object detection,
semantic segmentation and instance segmentation. It does not cover saliency
maps, and it reports no benchmark numbers [arxiv:2206.01653 § Main,
§ Results]. The language-guided multi-target masks of PixelLM use another
protocol again, scored with a GPT-based step on top of IoU-style overlap
[arxiv:2312.02228 §4.3, app. A].

The measures share one contract: prediction in, GT in, one scalar out. The
S-measure paper assumes the GT is binary [arxiv:1708.00786 §1]. The overlap
measures need the prediction binarised first. The threshold is a choice the
paper does not state; the released code uses 0.5 (see the
[card](/papers/2011.07631), Paper vs code). The HarDNet-MSEG paper instead
scores a probability map over a sweep of thresholds and averages the result
[arxiv:2101.07172 §4.3]. That is a different quantity, covered in its own
variant below.

Evaluation measures are themselves compared using *meta-measures*: tests that
check whether a measure behaves the way it should. The S-measure paper uses
four meta-measures taken from the Fbw paper by Margolin et al. plus a new user
study [arxiv:1708.00786 §5]:

- **MM1, application ranking:** 1 − Spearman ρ between the measure's ranking of maps and the ranking given by a downstream segmentation tool (SalCut) [arxiv:1708.00786 §5.1].
- **MM2, generic-map test:** how often a generic, input-independent map scores higher than the mean state-of-the-art map [arxiv:1708.00786 §5.2].
- **MM3, wrong-GT test:** how often a map's score goes up when it is scored against the wrong image's GT [arxiv:1708.00786 §5.3].
- **MM4, annotation-error robustness:** ranking stability under small morphological edits of the GT [arxiv:1708.00786 §5.4]. The S-measure authors consider it unreliable and leave it out of their multi-dataset comparison [arxiv:1708.00786 §5.4].
- **MM5, human preference:** viewers pick which of two measures' top-ranked maps looks better [arxiv:1708.00786 §5.6].

Lower is better for MM1 to MM4.

## Variants

### Pixel-wise measures for saliency maps (AP, AUC, Fbw)

These threshold the map, or weight it per pixel, and score TP/TN/FP/FN counts,
so where an error falls relative to the object does not matter [arxiv:1708.00786
§2, §3]. Fbw (Margolin et al. 2014, no card here) was the strongest of the
three baselines in the S-measure comparison [arxiv:1708.00786 tab. 1]. This
wiki has no card that defines these measures, so look up their formulas in
their original papers.

### Saliency benchmark protocol (max F-measure, MAE, S-measure)

The EGNet paper scores each saliency map with three numbers: max F-measure
with β² = 0.3, mean absolute error (MAE) and S-measure with γ = 0.5
[arxiv:1908.08297 §4.2]. It reports them on six benchmarks (ECSSD, PASCAL-S,
DUT-OMRON, HKU-IS, SOD and DUTS-TE) after training on DUTS-TR [arxiv:1908.08297
§4.1]. The card records no detail on how the maximum F-measure is taken over
thresholds, so read §4.2 before implementing it. A dash in its table marks a
dataset the method was trained on, so the cell is missing, not zero [arxiv:1908.08297
tab. 2]. Author-reported ResNet numbers on DUTS-TE are max F 0.893, MAE 0.039
and S-measure 0.875 [arxiv:1908.08297 tab. 2]. The S-measure column is the
measure defined in the S-measure section below. The MM1 to MM5 values in that
section are a different test.

Card: [EGNet: Edge Guidance Network for Salient Object Detection](/papers/1908.08297).

The D3Net paper uses the same family for RGB-D saliency. It reports S-measure,
max F-measure, max E-measure and MAE on seven test sets, including the
phone-captured SIP set [arxiv:1907.06781 §5.1, tab. 4]. The max E-measure is not
defined on this wiki. The paper gives no β for max F-measure, but the released
code uses β² = 0.3 [arxiv:1907.06781 § Paper vs code]. The repository computes
only MAE and max F-measure. The S-measure and E-measure toolbox is distributed
separately, so those two numbers cannot be checked from the repository
[arxiv:1907.06781 § Paper vs code]. Author-reported NJU2K test values for D3Net
are S-measure 0.900, max F 0.900, max E 0.950 and MAE 0.041 [arxiv:1907.06781 tab. 4].

Card: [Rethinking RGB-D Salient Object Detection](/papers/1907.06781). Its
test-set and checkpoint pitfalls are in the card and in Contradictions below.

### Four-measure protocol for camouflaged objects (Sα, Eφ, Fβʷ, M)

The SINet-V2 paper scores concealed-object maps with four numbers: S-measure
(Sα, higher is better), mean E-measure (Eφ), weighted F-measure (Fβʷ) and mean
absolute error (M, lower is better) [arxiv:2102.10274 §5.1]. It reports them on
CHAMELEON (76 images), CAMO test and COD10K test, with twelve baselines trained
on the same combined CAMO and COD10K training sets [arxiv:2102.10274 §5.1,
tab. 2]. Author-reported on COD10K test: Sα 0.815, Eφ 0.887, Fβʷ 0.680 and M 0.037
[arxiv:2102.10274 tab. 2]. The card does not name the source of its S-measure
definition, so do not assume it matches the section below without checking the
original formula. The per-super-class table reports the same four numbers for
four COD10K groups [arxiv:2102.10274 tab. 3].

Card: [Concealed Object Detection](/papers/2102.10274). Pitfalls on this
protocol, including the test-set selection rule and the size conflict for the
COD10K test split, are in the card and in Contradictions below.

### Single-threshold F and IoU with an empty-GT convention (ANet protocol)

The Anabranch Network paper scores camouflage maps with F-beta at β² = 0.3,
IoU and MAE [arxiv:2105.09451 §5.2, eq. 1]. The F and IoU values depend on a
binarisation step, which the paper runs in two settings [arxiv:2105.09451 §5.2]:

- **Presence assumed:** the threshold is the map's mean plus its standard
  deviation.
- **Presence not guaranteed:** the threshold is a fixed 0.5.

MAE is computed on the raw map, without binarisation [arxiv:2105.09451 §5.2].
This F-measure is taken at one threshold. It is not the maximum over thresholds
used in the EGNet table, so the two sets of numbers are not comparable.

The CAMO-COCO set adds 1,250 images with all-zero GT masks [arxiv:2105.09451 §5.1].
F and IoU are undefined when both the prediction and the mask are empty, and the
paper gives no convention for that case [arxiv:2105.09451 §5.2]. A convention
must be stated in any reported number. The CAMO test split has only 250 images,
so a few hundredths of F may fall within noise [arxiv:2105.09451 §5.1]. The
paper does not state whether F and IoU are averaged per image or pooled
over the dataset [arxiv:2105.09451 §5.2].

Card: [Anabranch Network for Camouflaged Object Segmentation](/papers/2105.09451).
Its headline numbers are author-reported: on CAMO-COCO, ANet-SRM reaches F 0.826
and IoU 0.732 at the adaptive threshold, against 0.699 and 0.535 for the
pre-trained SRM [arxiv:2105.09451 tab. 3].

### Overlap measures on binary masks (Jaccard/IoU, Dice, F-beta)

For a predicted mask and a GT mask, count true positives (tp), false positives
(fp) and false negatives (fn) over pixels. Jaccard is tp / (tp + fp + fn),
which equals the intersection-over-union of the two masks [arxiv:2011.07631
eq. 1, eq. 4]. Dice (DSC) is 2·tp / (2·tp + fp + fn) [arxiv:2011.07631 eq. 3].
F-beta [arxiv:2011.07631 eq. 7] weights recall above precision when β > 1.
Precision and recall are the usual ratios [arxiv:2011.07631 eq. 5, eq. 6].
The paper also reports accuracy [arxiv:2011.07631 eq. 8].

Metrics Reloaded gives Dice or IoU as the default overlap measure for semantic
segmentation, F-beta when false positives or false negatives are preferred,
and clDice for tubular structures [arxiv:2206.01653 § SemS]. It also notes
that Dice equals the pixel-level F1 score and that, per mask pair, Dice is
2·IoU/(1+IoU) [arxiv:2206.01653 § Metric Pool]. That identity holds for one
pair of masks. Reported numbers are dataset averages, so do not convert one
average into the other.

The surgical-instrument paper uses the same Jaccard index in two places.
Its training loss adds a soft version of J, computed on predicted
probabilities, to cross-entropy [arxiv:1803.01207 eq. 2, eq. 3]. Its reported
IoU and Dice are the hard overlap measures on thresholded masks [arxiv:1803.01207
tab. 1]. Do not read its loss terms as evaluation metrics. Author-reported
binary results for TernausNet-16 are IoU 83.60% and Dice 90.01%
[arxiv:1803.01207 tab. 1]. Card: [Automatic Instrument Segmentation in
Robot-Assisted Surgery Using Deep Learning](/papers/1803.01207).

The 2019 survey of deep segmentation writes the same link as Jaccard =
Dice / (2 − Dice) [arxiv:1910.07655 §9.1]. It also works a small example on a
5×5 mask with one false negative. Dice falls to 0.96 on a large object and to
0.66 on a small one, while cross-entropy gives 0.83 for both [arxiv:1910.07655
§4.4, tab. 1, §10.3]. The survey uses this to argue about training losses, not
to rank evaluation metrics, so read it as an illustration of how one error
weighs more on a small target. The card is [here](/papers/1910.07655).

Metrics Reloaded also says to compute values per image and then aggregate
over images, not over all pixels at once, because pixels within an image are
correlated [arxiv:2206.01653 § SemS].

The Kvasir-SEG polyp benchmark reports these on 120 validation images
[arxiv:2011.07631 §V-B, tab. V]. The released code also uses those images for
checkpoint selection, and it averages scores per image. The
[card](/papers/2011.07631) covers both. The top
method is the same for Dice and Jaccard in that table, but the scores differ
by method, so do not convert one into the other.

Card: [Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning](/papers/2011.07631). Its own headline numbers are author-reported: ColonSegNet reaches Dice 0.8206 at 182.38 FPS, and the best Dice in the paper is UNet with ResNet34 at 0.8757 and 35.00 FPS [arxiv:2011.07631 tab. V]. The authors name YOLOv4 as the best detector because of its balance of AP, IoU and speed [arxiv:2011.07631 §VI].

The HarDNet-MSEG paper gives the same two overlap formulas: Dice = 2·tp / (2·tp + fp + fn), and mIoU = tp / (tp + fp + fn), which is the Jaccard index above [arxiv:2101.07172 §4.3]. Its setup A table reports U-Net with ResNet34 at 0.876 mean Dice, and the caption says the baseline rows are taken from jha2020real [arxiv:2101.07172 tab. 1]. That matches 0.8757 in [arxiv:2011.07631 tab. V] to rounding.

Card: [HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS](/papers/2101.07172). Author-reported: 0.904 mean Dice at 86.7 FPS on Kvasir-SEG under setup A, and 0.912 mean Dice under the PraNet protocol (setup B) [arxiv:2101.07172 tab. 1, tab. 2]. The two setups differ in input size, optimizer and training data [arxiv:2101.07172 §4.2], so do not read them as one benchmark.

### Threshold-swept overlap (HarDNet-MSEG protocol)

This variant scores a probability map without choosing one threshold. The
paper does not state the threshold or the aggregation. The authors' MATLAB
evaluation, which the README says produced the report, normalises each map
to [0, 1], computes Dice at 256 thresholds from 1 down to 0 in steps of 1/255,
averages those values, and also reports the maximum [arxiv:2101.07172 §4.3]
(see the [card](/papers/2101.07172), Pitfalls). The average is a different
quantity from Dice at a fixed threshold such as 0.5. A number from this
protocol should not be placed beside a fixed-threshold Dice without saying so.

The same card separates two Dice values. The Python Dice in the training
script is a soft Dice on normalised probabilities, averaged over a fixed
number of iterations. It selects checkpoints and is not the table metric
(see the [card](/papers/2101.07172), Pitfalls).

### Boundary and tolerance measures (NSD)

Metrics Reloaded adds a boundary metric. It recommends normalised surface
distance (NSD) by default for segmentation when annotation imprecision should
be compensated. Otherwise the choice depends on whether a boundary error
should be penalised by its existence or by its distance [arxiv:2206.01653
§ SemS, fig. 2].

NSD has two parameters that the paper text does not fix: a tolerance tau and
the pixel spacing. The released code uses tau = 1 when none is given and
warns. Surface distances use the pixel spacing as sampling, and without a
spacing, distances and tau are in voxel units [arxiv:2206.01653 § Paper vs
code]. Set tau explicitly in any implementation. The default does not
follow from the paper. The code returns NaN for NSD when prediction and GT are
both empty. How empty cases should be scored is not stated in the paper
[arxiv:2206.01653 § Paper vs code].

The nnU-Net Revisited benchmark fixes the tolerance: it reports NSD at 2 mm as
its secondary metric, next to Dice [arxiv:2404.09556 §3.2, tab. 4]. Its NSD
values depend on that tolerance, so do not compare them with NSD computed at
another tau.

Card: [Metrics Reloaded: Recommendations for image analysis validation](/papers/2206.01653).

### Five-fold cross-validation with a dataset suitability check (nnU-Net Revisited)

This variant is a validation design, not a new formula. Every method is
scored with five-fold cross-validation on the same nnU-Net splits. The primary
metric is Dice and the secondary metric is NSD at 2 mm. Both are averaged
over classes, then over folds [arxiv:2404.09556 §3.2, tab. 4]. For BraTS2021
and KiTS2023 the scores are computed on hierarchical evaluation regions, not
on the non-overlapping classes [arxiv:2404.09556 §3.2]. The paper compares CNN,
Transformer and Mamba 3-D models on six datasets (BTCV, ACDC, LiTS, BraTS2021,
KiTS2023, AMOS2022) [arxiv:2404.09556 §3.1, tab. 1].

To pick datasets, the paper scores each one by its suitability: the spread
of the methods' fold-mean scores (inter-method SD) divided by the average
spread across folds within a method (intra-method SD) [arxiv:2404.09556 §3.2,
tab. 3]. A higher ratio means the dataset separates methods better. The paper
gives no SD convention, so state which one you used (see the
[card](/papers/2404.09556), Pseudocode in its Method section).

The ratio depends on the method set. Across all methods the ratios are 94% on
BTCV and 474% on AMOS. Without the Auto3DSeg runs, BTCV falls to 65% and ACDC
falls from 357% to 102% [arxiv:2404.09556 tab. 3]. The authors name ACDC, AMOS
and KiTS as the most suitable sets [arxiv:2404.09556 §4]. That ranking is
sensitive to which methods are included, so do not treat it as a fixed
property of the data.

Card: [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556).
Its own headline numbers are author-reported. The mean Dice of the original
nnU-Net on BTCV is 83.08 and the best CNN on that set is MedNeXt L k5 at 85.04
[arxiv:2404.09556 tab. 1].

### Per-organ Dice and Hausdorff distance on 3-D volumes (TransUNet protocol)

On the Synapse multi-organ CT set, the paper reports Dice for each organ, the
average Dice over organs, and an average HD in millimetres [arxiv:2102.04306
§4.1, tab. 1]. It scores each 2-D slice on its own and stacks the slices into
a volume [arxiv:2102.04306 §4.2]. Author-reported: TransUNet reaches 77.48%
average DSC and 31.69 mm HD. The best CNN baseline in the same table, R50-AttnUNet,
reaches 75.57% DSC [arxiv:2102.04306 tab. 1]. On ACDC cardiac MRI the paper
reports 89.71% average DSC over RV, Myo and LV [arxiv:2102.04306 tab. 5].

The released code scores edge cases in a way the paper does not state. An empty
prediction against a non-empty ground truth gets DSC 0 and HD 0. A non-empty
prediction against an empty ground truth gets DSC 1 and HD 0. Both HD values
are the best possible, so a missed organ and a false organ can both look good
on HD (see the [card](/papers/2102.04306), Pitfalls). The paper reports one run
per cell with no seeds or variance [arxiv:2102.04306 tab. 1–5], so treat
differences of about one DSC point as uncertain.

Card: [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](/papers/2102.04306).

### Choosing the metric from the problem (Metrics Reloaded)

A Delphi-consensus process produced a three-step guide [arxiv:2206.01653 §
Step 1, § Step 2, § Step 3]:

1. **Fingerprint.** Map the problem to one of four categories, then answer
   fingerprint items about the domain, the target structure, the data set and
   the algorithm's output [arxiv:2206.01653 § Results: problem fingerprints].
2. **Select.** Walk the decision tree of that category, which adds metrics to
   a pool [arxiv:2206.01653 § Step 2]. For object detection the tree depends on
   the localisation criterion and the assignment strategy, both drawn in the
   supplement [arxiv:2206.01653 § ObD, fig. 2].
3. **Apply.** Use the cross-topic recommendations for implementation,
   aggregation, ranking, interpretation and reporting [arxiv:2206.01653 § Step 3].

The paper recommends several complementary metrics instead of one. It was
built from five international workshops, nine surveys and crowdsourced
feedback between 2020 and 2022. Its consensus is above 75% agreement for all
ten components [arxiv:2206.01653 § Results: framework, § Step 2]. The authors
say no method exists to compare metric sets quantitatively, so the guide is
validated by consensus, not by proof [arxiv:2206.01653 § Results: validation by
consensus].

### Detection average precision (AP over IoU thresholds)

For boxes, a detection counts as correct when its IoU with a GT box reaches a
threshold. AP is the area under the precision-recall curve, using interpolated
precision [arxiv:2011.07631 eq. 1, eq. 2]. The paper averages AP over thresholds
from 0.25 to 0.75 in steps of 0.05 [arxiv:2011.07631 §V-A]. Its detection table
also reports AP at single thresholds (AP25, AP50, AP75) and IoU [arxiv:2011.07631 tab. III].

The ColonSegNet box numbers depend on a mask-to-box conversion that the paper
does not describe [arxiv:2011.07631 §III]. The
[card](/papers/2011.07631)'s Paper vs code section has the details.

Metrics Reloaded gives the detection rule. Without continuous class scores, use
a counting metric such as F-beta. With class scores, complement the counting
metric with AP or FROC. Accuracy, specificity and AUROC are invalid for
detection, because there is no meaningful count of true negatives [arxiv:2206.01653
§ ObD].

### Structure-measure (S-measure)

S-measure averages two terms [arxiv:1708.00786 §4, eq. 10]:

- **Region-aware term `S_r`.** Split the prediction and the GT into four blocks at the centroid of the GT foreground. Compute SSIM in each block, without SSIM's stabilising constants, and take a weighted sum [arxiv:1708.00786 §4.1, eq. 3, eq. 4].
- **Object-aware term `S_o`.** Inside the GT foreground, reward a high mean prediction and penalise its spread (coefficient of variation, weight λ). Do the same on the complement (`1 − map`) inside the GT background. Combine the two with the foreground area fraction as weight [arxiv:1708.00786 §4.2, eq. 7, eq. 8, eq. 9].

`S = α·S_o + (1 − α)·S_r` with α = 0.5. The result lies in [0, 1] and higher is better [arxiv:1708.00786 §4.3]. Reported runtime is 5.3 ms per image on one 4 GHz CPU thread in Matlab [arxiv:1708.00786 §5].

Reported against AP, AUC and Fbw on PASCAL-S, ECSSD, SOD and HKU-IS [arxiv:1708.00786 tab. 1]:

| Measure | PASCAL-S MM1 / MM2 / MM3 | ECSSD MM1 / MM2 / MM3 | SOD MM1 / MM2 / MM3 | HKU-IS MM1 / MM2 / MM3 |
|---|---|---|---|---|
| AP | 0.452 / 12.1 / 5.50 | 0.449 / 9.70 / 3.32 | 0.504 / 9.67 / 7.69 | 0.518 / 3.76 / 1.25 |
| AUC | 0.449 / 15.8 / 8.21 | 0.436 / 12.1 / 4.18 | 0.547 / 14.0 / 8.27 | 0.519 / 7.02 / 2.12 |
| Fbw | 0.365 / 7.06 / 1.05 | 0.401 / 3.00 / 0.84 | 0.384 / 16.3 / 0.73 | 0.498 / 0.36 / 0.26 |
| S-measure | 0.320 / 4.59 / 0.34 | 0.312 / 3.30 / 0.47 | 0.349 / 9.67 / 0.60 | 0.424 / 0.34 / 0.08 |

S-measure is best in 10 of 12 cells. It ties AP on SOD MM2, and Fbw beats it on ECSSD MM2 [arxiv:1708.00786 tab. 1]. On MM4 it does better than AP and AUC but worse than Fbw [arxiv:1708.00786 §5.4, fig. 6]. In the user study (45 viewers, 50 pairs), the map ranked top by S-measure was preferred in 63.69% of trials against Fbw, 72.11% against AP and 73.56% against AUC [arxiv:1708.00786 §5.6]. Pairs whose top maps looked too similar were removed beforehand (100 down to 50) [arxiv:1708.00786 §5.6].

Card: [Structure-measure: A New Way to Evaluate Foreground Maps](/papers/1708.00786).

### Language-guided multi-target masks (gIoU and cIoU)

PixelLM answers in text and writes one mask per target named in the answer.
It reports gIoU and cIoU on its MUSE benchmark, on multi-referring
segmentation and on the refCOCO family [arxiv:2312.02228 tab. 1, tab. 2,
tab. 3]. The card does not define gIoU or cIoU. Read the paper's definitions
before comparing these numbers with any other protocol.

The MUSE protocol is not plain pixel overlap. Predicted masks are matched to GT
masks with the Hungarian algorithm, using a BCE + DICE cost. Each matched mask's
GT text replaces its segment in the answer. A GPT-3.5 prompt then scores each
prediction. Per-prediction IoU is kept only where that score exceeds 0.5, and
the image score is the sum of those IoUs divided by the padded set size P
[arxiv:2312.02228 §4.3, app. A]. Unmatched masks are paired with empty masks
[arxiv:2312.02228 app. A].

On MUSE test overall, PixelLM-Llama2-13B reports 42.3 gIoU / 51.0 cIoU. A
LISA variant trained on MUSE reports 41.9 / 50.5, and LISA without that
training reports 14.4 / 28.4 [arxiv:2312.02228 tab. 1]. These are author-reported.
The gap to the MUSE-trained LISA is small on this split.

Card: [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Ranking saliency or foreground models where object shape matters, e.g. the map feeds a segmentation step | S-measure | Lowest MM1 (closest to the SalCut application ranking) on all four datasets tested [arxiv:1708.00786 tab. 1] |
| Guarding against degenerate or generic predictions gaming the score | S-measure, with Fbw as a check | Best MM2 on PASCAL-S and HKU-IS, tied on SOD, but Fbw is better on ECSSD [arxiv:1708.00786 tab. 1] |
| Making sure a map cannot score well against the wrong GT | S-measure | Best MM3 on all four datasets; MM3 error reduction of 17.81% to 69.23% compared with the second-best measure [arxiv:1708.00786 tab. 1, §5.5] |
| GT annotations are noisy or slightly misaligned (MM4-type robustness) | Fbw | Fbw beats S-measure on MM4 [arxiv:1708.00786 §5.4] |
| Scoring RGB-D saliency maps against the D3Net table | S-measure, max F-measure, max E-measure and MAE on the same seven sets | These four measures are reported on seven sets, including SIP [arxiv:1907.06781 tab. 4]. The S-measure and E-measure code is not in the repository (see the [card](/papers/1907.06781), Paper vs code) |
| Segmentation where annotation imprecision should be compensated | NSD, with tau set explicitly | Metrics Reloaded's default boundary metric for this case [arxiv:2206.01653 § SemS]. The paper gives no tau value [arxiv:2206.01653 § Paper vs code] |
| Scoring saliency maps against published SOD tables | Max F-measure (β² = 0.3), MAE and S-measure, as the EGNet table reports them | Those three measures are reported on six benchmarks [arxiv:1908.08297 §4.2, tab. 2]. The card records no detail on the max procedure (see the [card](/papers/1908.08297)) |
| Reporting 3-D organ segmentation next to the TransUNet table | Per-organ DSC averaged over the eight organs, with HD or HD95 and the edge-case rule stated | Table 1 uses average DSC and HD [arxiv:2102.04306 tab. 1]. The code's HD is HD95 and its empty-case rule is not in the paper (see the [card](/papers/2102.04306), Paper vs code) |
| Comparing 3-D medical segmenters against nnU-Net-style baselines | Dice as primary and NSD at 2 mm as secondary, five-fold CV on the same splits, averaged over classes then folds | This is the nnU-Net Revisited protocol [arxiv:2404.09556 §3.2, tab. 4]. It does not apply to saliency maps, and the card's Paper vs code section lists what the paper leaves open |
| Deciding whether a gap between two methods is real | Compare the gap with the fold-to-fold spread the paper reports, and say so | The paper gives no significance test. Its intra-method SD is 2.39% on BTCV, larger than the gaps among its top CNNs on that set [arxiv:2404.09556 tab. 3, tab. 1] |
| Picking evaluation datasets for a new benchmark | Compute the suitability ratio over several methods, and report which methods were used | The ratio is sensitive to the method set, and the ACDC ratio shifts with it [arxiv:2404.09556 tab. 3] |
| Binary instrument or tool masks, compared with the surgical-instrument table | Binary IoU and Dice, with the threshold and the averaging rule stated | The paper reports both for each model [arxiv:1803.01207 tab. 1]. Its stated threshold of 0.3 is not the one its code evaluates at (see the [card](/papers/1803.01207), Paper vs code) |
| Tubular structures such as vessels | clDice alongside an overlap measure | Metrics Reloaded lists clDice for tubular structures [arxiv:2206.01653 § SemS] |
| Scoring a binary mask and comparing with published medical segmentation numbers | Dice and Jaccard, reported together | The Kvasir-SEG table reports both for every method [arxiv:2011.07631 tab. V] |
| Converting a Dice value to Jaccard, or back, for one mask pair | J = D / (2 − D), or D = 2J / (1 + J) | The survey states the Dice–Jaccard link [arxiv:1910.07655 §9.1]. It holds per mask pair, so do not apply it to dataset averages [arxiv:2206.01653 § Metric Pool] |
| Reading a Dice gap on small target objects | Report the gap with the object size stated | One false negative moves Dice from 0.96 to 0.66 on the small object in the survey's example, against 0.83 for cross-entropy on both [arxiv:1910.07655 §4.4]. The survey gives no size-stratified protocol |
| Missing a target costs more than a false alarm | F-beta with β > 1 | Recall is weighted higher by definition [arxiv:2011.07631 eq. 7]. The paper's motivation is missed polyps [arxiv:2011.07631 §I], but it does not test a β other than 1 or 2 |
| Probability maps where no single threshold is fixed by the task, reported with a protocol | Threshold-swept Dice, stated as such | This is the HarDNet-MSEG table protocol [arxiv:2101.07172 §4.3]. Its threshold grid and aggregation are not in the paper text (see the [card](/papers/2101.07172), Pitfalls) |
| Reproducing the HarDNet-MSEG table numbers | The MATLAB evaluation protocol, not the Python Dice in the training script | The Python loop is for checkpoint selection, not the table metric (see the [card](/papers/2101.07172), Pitfalls) |
| Choosing a threshold or checkpoint for a score you will report | A separate validation split | The HarDNet-MSEG training script keeps the checkpoint with the best Dice on the test folder, which the paper does not mention (see the [card](/papers/2101.07172), Pitfalls) |
| Scoring box detections | AP over IoU thresholds 0.25 to 0.75 | This is the paper's detection metric [arxiv:2011.07631 §V-A, eq. 2] |
| Detection with no class scores | A counting metric such as F-beta | Metrics Reloaded's rule for detection without continuous scores [arxiv:2206.01653 § ObD] |
| Detection with class scores | A counting metric plus AP or FROC | Metrics Reloaded complements the counting metric this way [arxiv:2206.01653 § ObD] |
| Choosing metrics for a new validation study with no existing convention | Start from the Metrics Reloaded fingerprint and decision tree | It is the only consensus process in the cited set [arxiv:2206.01653 § Step 1, § Step 2]. Its recommendations are not quantitatively tested |
| Multi-target answers from a language model, compared with the MUSE table | Follow the MUSE protocol, including its GPT score | The reported gIoU and cIoU come from that protocol, not plain pixel IoU [arxiv:2312.02228 §4.3, app. A] |
| Scoring camouflaged or concealed objects against the SINet-V2 table | Sα, Eφ, Fβʷ and M on CHAMELEON, CAMO and COD10K | This is the protocol of the paper's main table [arxiv:2102.10274 §5.1, tab. 2]. Compare only with runs on the same split and training data |
| Some images have no target and all-zero masks | ANet-style F and IoU at a stated threshold, with the empty-GT convention written down | The paper uses a mean-plus-std threshold when presence is assumed and 0.5 when it is not, and gives no empty-case rule [arxiv:2105.09451 §5.2] |
| Reporting numbers next to prior work that used AP, AUC or Fbw | Report those too, alongside S-measure | The measures rank models differently [arxiv:1708.00786 tab. 1], so one cannot be converted into another |
| Very tight compute budget for evaluation | S-measure or the overlap measures | S-measure takes 5.3 ms per image on a single CPU thread [arxiv:1708.00786 §5]; overlap measures reduce to three pixel counts per image [arxiv:2011.07631 eq. 3] |

## Implementation sketch

Two families. Both are pixel-count based. They differ in input (map or mask)
and in how they treat object structure.

S-measure as the official code computes it [arxiv:1708.00786 §4]. The card's
Pitfalls cover where the code and the paper differ.

```text
s_measure(pred in [0,1], gt boolean, both H×W):
    m = fraction of gt that is foreground
    if m == 0: return 1 - mean(pred)          # code only, not in paper
    if m == 1: return mean(pred)

    # object term
    obj(v)  = 2*mean(v) / (mean(v)^2 + 1 + std(v) + eps)   # std uses N-1; equals λ = 0.5
    S_o = m * obj(pred[gt]) + (1 - m) * obj(1 - pred[!gt])

    # region term
    (X, Y) = rounded centroid of gt foreground
    split pred and gt into 4 blocks at row Y, column X
    for each block k:
        w_k  = block area / image area        # paper: share of gt foreground in block
        ssim_k = 4*x̄*ȳ*cov / ((x̄²+ȳ²)(var_x+var_y) + eps)   # no C1/C2; zero-case branches
    S_r = sum_k w_k * ssim_k

    return max(0, 0.5*S_o + 0.5*S_r)
```

Where variants differ: pixel-wise measures replace both terms with a confusion-count statistic computed over thresholds or with per-pixel weights. Do not swap in a library SSIM. Those use constants and sliding windows, and the authors report that global or sliding-window SSIM did not capture region structure [arxiv:1708.00786 §4.1].

Overlap measures on binary masks [arxiv:2011.07631 eq. 1, eq. 3]:

```text
overlap(pred boolean, gt boolean, both H×W):
    # pixel counts over the whole mask
    tp = count(pred & gt)
    fp = count(pred & !gt)
    fn = count(!pred & gt)

    # empty-mask handling is not stated in the paper
    jaccard = tp / (tp + fp + fn)            # equals IoU of the two masks
    dice    = 2*tp / (2*tp + fp + fn)
    return jaccard, dice

# average the per-image scores over the dataset; the paper does not say
# whether it pools pixels across images (see the card's Pitfalls)
```

Threshold-swept variant, as the HarDNet-MSEG authors' MATLAB code does it
[arxiv:2101.07172 §4.3] (see the [card](/papers/2101.07172), Pitfalls):

```text
threshold_swept_dice(prob in [0,1], gt boolean):
    # normalise each map to [0, 1], per image, before thresholding
    prob = (prob - min(prob)) / (max(prob) - min(prob) + eps)
    scores = []
    # 256 thresholds, from 1 down to 0 in steps of 1/255
    for t in [k / 255 for k in 255..0]:
        scores.append(dice(prob >= t, gt))    # dice as in the block above
    # the mean is the averaged score; the maximum is the best-threshold score
    return mean(scores), max(scores)
```

Per-organ scoring on 3-D label volumes, with the edge cases as the TransUNet
code handles them [arxiv:2102.04306 §4.1] (see the [card](/papers/2102.04306),
Pitfalls). The paper does not state these cases:

```text
organ_scores(pred, gt, organ):          # 3-D label volumes
    p = (pred == organ)
    g = (gt == organ)
    # one-sided empty cases, as the released code scores them
    if p.any() and not g.any(): return dsc=1, hd=0
    if g.any() and not p.any(): return dsc=0, hd=0
    # both empty: not described in the card; choose and state a rule
    dsc = 2 * (p & g).sum() / (p.sum() + g.sum())
    hd  = hausdorff95(p, g)             # the code reports HD95
    return dsc, hd

# average dsc and hd over the eight organs; background is not averaged
```

## Contradictions

Within [1708.00786](/papers/1708.00786):

- **Block weights:** the text weights each block by how much GT foreground it covers [arxiv:1708.00786 §4.1]. The official code weights by block area instead (see the [card](/papers/1708.00786)). Results computed the two ways are not comparable.
- **λ:** the experiments use λ = 0.5 [arxiv:1708.00786 §5], but the Fig. 2 caption states λ = 0.25 [arxiv:1708.00786 fig. 2].
- **MM4:** the authors use MM4 on ASD1000 but argue it is unreliable and drop it from the multi-dataset table [arxiv:1708.00786 §5.4]. MM4 is the one meta-measure where S-measure is not best.

Within [1803.01207](/papers/1803.01207):

- **Binary threshold:** the paper applies a threshold of 0.3, chosen on validation data [arxiv:1803.01207 §2.3]. The validation code uses probability 0.5 and the evaluation script uses 0.5 [code: validation.py#L20] [code: evaluate.py#L77]. No code selects a threshold (see the [card](/papers/1803.01207), Paper vs code).
- **Per-image or per-file averaging:** the README says the binary score is averaged per video, but `evaluate.py` averages over all files. The two can give different numbers on the same predictions [arxiv:1803.01207 § Pitfalls].

Within [2011.07631](/papers/2011.07631):

- **F2 definition:** the paper defines F2 with β = 2 [arxiv:2011.07631 eq. 7]. The released code computes its "F2" with β = 1, so it equals Dice (see the [card](/papers/2011.07631), Paper vs code). The F2 column in the table therefore matches Dice for ColonSegNet [arxiv:2011.07631 tab. V]. Recompute F2 from the counts before comparing it with other papers.
- **Validation or test:** the figure captions call the 120 images the "test dataset", while the text calls them "validation" [arxiv:2011.07631 §V-B]. The released code also selects its checkpoint on those images (see the [card](/papers/2011.07631)), so treat the reported numbers as validation results.

Within [2312.02228](/papers/2312.02228):

- **Token-fusion gain:** the text says token fusion gives "up to 2.4%" cIoU on MUSE val [arxiv:2312.02228 §5.5]. Tab. 4 gives 48.0 to 50.1, a gain of 2.1 [arxiv:2312.02228 tab. 4].
- **MUSE scoring:** §4.3 says the GPT score from 1 to 10 is multiplied with the IoU. App. A keeps the intersection only if the score is above 0.5. The text does not say how 1–10 scores map to that threshold [arxiv:2312.02228 §4.3, app. A].
- **Best on referring tasks:** the text says PixelLM is best on most metrics [arxiv:2312.02228 §5.4]. LISA is higher on refCOCO val and testB [arxiv:2312.02228 tab. 3].

Within [1908.08297](/papers/1908.08297):

- **None in the evaluation numbers.** The text's figures match a recomputation from the tables: the 1.9-point average F-measure gain over PiCANet [arxiv:1908.08297 §4.4] matches tab. 2, and the 3.1 and 2.4 point ablation gains [arxiv:1908.08297 §4.3.2] match tab. 3.

Within [2102.04306](/papers/2102.04306):

- **Table 1, R50-AttnUNet row.** The eight per-organ DSCs in the row average to 72.10, not the 75.57 printed as the average [arxiv:2102.04306 tab. 1]. Reaching 75.57 would need an aorta value of about 83.7 instead of 55.92. The paper does not explain the gap, so use this row with care.
- **HD or HD95.** The text says "Hausdorff Distance (HD)" and does not say whether it is HD95 [arxiv:2102.04306 §4.1]. The code reports HD95 (see the [card](/papers/2102.04306), Paper vs code).

Within [2102.10274](/papers/2102.10274):

- **Test-set size.** Section 3.3 and [tab. 1] give 4,000 COD10K test images, but the COD10K results section says 2,026 [arxiv:2102.10274 §3.3, tab. 1, §5.2]. The paper does not explain the gap.
- **Super-class count.** Section 3.1 lists 10 super-classes and Sections 3.2 and 3.3 each say five concealed super-classes, but [tab. 3] shows four [arxiv:2102.10274 §3.1, §3.2, §3.3, tab. 3].

Within [2105.09451](/papers/2105.09451):

- **Classifier width.** The text gives 4096 hidden units and Table 1 gives 2048 [arxiv:2105.09451 §4.2, tab. 1]. This affects the classification stream only, not the F and IoU numbers.

Within [1910.07655](/papers/1910.07655):

- **None in the evaluation content.** Its Dice–Jaccard link matches the one in [2206.01653](/papers/2206.01653) [arxiv:1910.07655 §9.1] [arxiv:2206.01653 § Metric Pool]. Its mean-IoU table is collected from other papers, each under its own protocol [arxiv:1910.07655 tab. 2], so the table does not compare methods on one protocol.

Within [2101.07172](/papers/2101.07172):

- **Metric definition:** the Dice and mIoU formulas are given, but the threshold and the aggregation are not stated in the text. The MATLAB evaluation averages over 256 thresholds, while the training script uses a different soft Dice (see the [card](/papers/2101.07172), Pitfalls) [arxiv:2101.07172 §4.3].
- **F2 formula:** the paper prints F2 = 5pr/(4p+r) and does not define p or r [arxiv:2101.07172 §4.3]. This has the form of the β = 2 F-measure in [arxiv:2011.07631 eq. 7], which reads p and r as precision and recall. The paper does not say so.
- **Setup A optimizer:** the paper gives SGD with learning rate 1e-2 for setup A, but the code's default is Adam with 1e-4 [arxiv:2101.07172 §4.2] (see the [card](/papers/2101.07172), Paper vs code). The setup A number cannot be reproduced from the defaults.

Across papers:

- **Averaging convention:** [2011.07631](/papers/2011.07631) computes each image's score at a 0.5 threshold and averages per image in its released code (see the [card](/papers/2011.07631), Pitfalls). [2101.07172](/papers/2101.07172) averages Dice over thresholds in its MATLAB code [arxiv:2101.07172 §4.3]. Both are reported as Kvasir-SEG Dice, so the numbers are not interchangeable without the protocol. [2404.09556](/papers/2404.09556) averages over classes and then over folds [arxiv:2404.09556 §3.2]. The card does not say whether it aggregates per image, so do not assume that Metrics Reloaded's per-image rule [arxiv:2206.01653 § SemS] was followed.
- **Checkpoint rule:** [2101.07172](/papers/2101.07172) keeps the checkpoint with the best Dice on the test folder (see the [card](/papers/2101.07172), Pitfalls). [1907.06781](/papers/1907.06781) keeps the checkpoint with the lowest MAE on the NJU2K test split, which is also one of its reported sets (see the [card](/papers/1907.06781), Paper vs code). [2102.04306](/papers/2102.04306) evaluates the last epoch with no selection on the held-out cases (see the [card](/papers/2102.04306), Paper vs code). [2011.07631](/papers/2011.07631) selects on its 120 images. [2102.10274](/papers/2102.10274) does not state a selection rule, but its released code keeps the epoch with the lowest MAE on CAMO test by default, and CAMO test is one of its reported sets (see the [card](/papers/2102.10274), Pitfalls). Scores that used different rules are not comparable.
- **Empty-case rule:** [2206.01653](/papers/2206.01653) returns NaN for NSD when prediction and GT are both empty, and the paper does not state how empty cases should be scored [arxiv:2206.01653 § Paper vs code]. [2102.04306](/papers/2102.04306)'s code gives DSC 0 or 1 with HD 0 for a one-sided empty case, and its both-empty case is not in the card (see the [card](/papers/2102.04306), Pitfalls). The two libraries disagree, so state the rule in any reported number. [2105.09451](/papers/2105.09451) leaves the empty-GT case for F and IoU undefined in its text (see the [card](/papers/2105.09451), Pitfalls).
- **Baseline rows:** the card says the setup A baseline rows are taken from jha2020real [arxiv:2101.07172 tab. 1 caption]. The card does not say whether those rows and the HarDNet-MSEG row were computed under the same protocol.

Otherwise the cards measure different things. Saliency maps use the S-measure, Fbw and max F-measure family. Binary masks use overlap measures, HD and detection AP. The 3-D nnU-Net protocol uses Dice and NSD on volumes [arxiv:2404.09556 tab. 1, tab. 4]. The MUSE protocol is a separate protocol. Metrics Reloaded does not cover saliency maps, so its recommendations do not extend to S-measure. Its recommended Dice and IoU defaults agree with the Kvasir-SEG use of both [arxiv:2206.01653 § SemS] [arxiv:2011.07631 tab. V]. Compare within a family only.

## Papers

- [Structure-measure: A New Way to Evaluate Foreground Maps](/papers/1708.00786): proposes S-measure, a region- plus object-aware structural similarity score for non-binary foreground maps, and compares it with AP, AUC and Fbw on five meta-measures.
- [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks](/papers/1907.06781): D3Net, an RGB-D saliency model, benchmarked with S-measure, max F-measure, max E-measure and MAE on seven sets, including the phone-captured SIP set.
- [EGNet: Edge Guidance Network for Salient Object Detection](/papers/1908.08297): a salient-object model with an edge branch, reported with max F-measure, MAE and S-measure on six benchmarks.
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](/papers/2102.04306): a CNN-Transformer hybrid for medical segmentation, reported with per-organ DSC and HD on Synapse CT and average DSC on ACDC MRI.
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](/papers/1803.01207): U-Net-style networks for surgical tool masks, trained with cross-entropy plus a soft-Jaccard term, reporting binary, parts and instrument-type IoU and Dice.
- [Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning](/papers/2011.07631): benchmarks detectors and segmenters on Kvasir-SEG with Jaccard, Dice, F-beta, precision, recall, accuracy and AP, and reports speed alongside.
- [HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS](/papers/2101.07172): a polyp segmenter that reports mean Dice, mIoU and F2 on Kvasir-SEG and four other sets, with threshold-swept scoring in its evaluation code.
- [Metrics reloaded: Recommendations for image analysis validation](/papers/2206.01653): a Delphi-consensus decision framework for choosing validation metrics for classification, detection and segmentation, with pitfalls. It reports no benchmark numbers.
- [Concealed Object Detection](/papers/2102.10274): SINet-V2, a two-stage camouflaged-object network, reported with Sα, Eφ, Fβʷ and M on CHAMELEON, CAMO and COD10K; it also introduces the COD10K dataset.
- [Anabranch Network for Camouflaged Object Segmentation](/papers/2105.09451): a saliency network with a camouflage classifier whose probability scales the map, reported with F-beta, IoU and MAE on CAMO and CAMO-COCO with two binarisation settings.
- [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228): a multimodal model that writes one mask per target named in its answer, evaluated with gIoU and cIoU on MUSE, multi-referring and refCOCO-family benchmarks.
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556): benchmarks CNN, Transformer and Mamba 3-D models with five-fold CV on six datasets, reporting Dice and NSD at 2 mm and a dataset suitability ratio.
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](/papers/1910.07655): a survey of deep segmentation models with no experiments of its own. For this concept it gives the Dice–Jaccard link and a small-object example; its mean-IoU table is collected from other papers.