Segmentation evaluation (scoring foreground, saliency and segmentation masks)

Overview

A segmentation, detection or salient-object model produces a result that has to be compared with a ground-truth (GT) mask. Results come in two shapes. A saliency model outputs a map, often non-binary. A segmentation model outputs a binary mask, typically after thresholding a probability map. The measures differ by shape.

For non-binary maps, the usual measures (AP, AUC and the weighted F-measure Fbw) all boil the comparison down to per-pixel true/false positive and negative counts 1708.00786 §2, §3. Counts like these cannot tell apart errors that keep the object's shape from errors that break it: a square hole in the middle of a digit and a square hole cutting its outline give the same counts 1708.00786 fig. 3. In a realistic example, Fbw ranks a blurry detection covering only the top of a pyramid above one that captures the whole shape 1708.00786 §3, fig. 3.

For binary masks, the common measures are overlap ratios: Jaccard (IoU), Dice and F-beta, all computed from pixel counts [arxiv:2011.07631 eq. 1, eq. 3, eq. 4, eq. 7]. These also ignore where the errors fall relative to the object. For object detection the score is average precision (AP) over IoU thresholds, which is a different AP from the pixel-wise AP used for saliency maps 2011.07631 §V-A, eq. 2. The two papers use the same abbreviation for different quantities, so check which one a reported number refers to.

Medical volumes add a boundary distance in millimetres, the Hausdorff distance (HD). The TransUNet paper does not say whether its HD is HD95, but its released code reports HD95 2102.04306 §4.1 (see the card, Paper vs code).

Beyond single measures, the Metrics Reloaded consensus framework treats metric choice as a process: fingerprint the problem, walk a decision tree to a pool of metrics, then apply them with stated pitfalls [arxiv:2206.01653 § Step 1, § Step 2, § Step 3]. It covers image-level classification, object detection, semantic segmentation and instance segmentation. It does not cover saliency maps, and it reports no benchmark numbers [arxiv:2206.01653 § Main, § Results]. The language-guided multi-target masks of PixelLM use another protocol again, scored with a GPT-based step on top of IoU-style overlap 2312.02228 §4.3, app. A.

The measures share one contract: prediction in, GT in, one scalar out. The S-measure paper assumes the GT is binary 1708.00786 §1. The overlap measures need the prediction binarised first. The threshold is a choice the paper does not state; the released code uses 0.5 (see the card, Paper vs code). The HarDNet-MSEG paper instead scores a probability map over a sweep of thresholds and averages the result 2101.07172 §4.3. That is a different quantity, covered in its own variant below.

Evaluation measures are themselves compared using meta-measures: tests that check whether a measure behaves the way it should. The S-measure paper uses four meta-measures taken from the Fbw paper by Margolin et al. plus a new user study 1708.00786 §5:

• MM1, application ranking: 1 − Spearman ρ between the measure's ranking of maps and the ranking given by a downstream segmentation tool (SalCut) 1708.00786 §5.1.
• MM2, generic-map test: how often a generic, input-independent map scores higher than the mean state-of-the-art map 1708.00786 §5.2.
• MM3, wrong-GT test: how often a map's score goes up when it is scored against the wrong image's GT 1708.00786 §5.3.
• MM4, annotation-error robustness: ranking stability under small morphological edits of the GT 1708.00786 §5.4. The S-measure authors consider it unreliable and leave it out of their multi-dataset comparison 1708.00786 §5.4.
• MM5, human preference: viewers pick which of two measures' top-ranked maps looks better 1708.00786 §5.6.

Lower is better for MM1 to MM4.

Variants

Pixel-wise measures for saliency maps (AP, AUC, Fbw)

These threshold the map, or weight it per pixel, and score TP/TN/FP/FN counts, so where an error falls relative to the object does not matter [arxiv:1708.00786 §2, §3]. Fbw (Margolin et al. 2014, no card here) was the strongest of the three baselines in the S-measure comparison 1708.00786 tab. 1. This wiki has no card that defines these measures, so look up their formulas in their original papers.

Saliency benchmark protocol (max F-measure, MAE, S-measure)

The EGNet paper scores each saliency map with three numbers: max F-measure with β² = 0.3, mean absolute error (MAE) and S-measure with γ = 0.5 1908.08297 §4.2. It reports them on six benchmarks (ECSSD, PASCAL-S, DUT-OMRON, HKU-IS, SOD and DUTS-TE) after training on DUTS-TR [arxiv:1908.08297 §4.1]. The card records no detail on how the maximum F-measure is taken over thresholds, so read §4.2 before implementing it. A dash in its table marks a dataset the method was trained on, so the cell is missing, not zero [arxiv:1908.08297 tab. 2]. Author-reported ResNet numbers on DUTS-TE are max F 0.893, MAE 0.039 and S-measure 0.875 1908.08297 tab. 2. The S-measure column is the measure defined in the S-measure section below. The MM1 to MM5 values in that section are a different test.

Card: EGNet: Edge Guidance Network for Salient Object Detection.

The D3Net paper uses the same family for RGB-D saliency. It reports S-measure, max F-measure, max E-measure and MAE on seven test sets, including the phone-captured SIP set 1907.06781 §5.1, tab. 4. The max E-measure is not defined on this wiki. The paper gives no β for max F-measure, but the released code uses β² = 0.3 1907.06781 § Paper vs code. The repository computes only MAE and max F-measure. The S-measure and E-measure toolbox is distributed separately, so those two numbers cannot be checked from the repository 1907.06781 § Paper vs code. Author-reported NJU2K test values for D3Net are S-measure 0.900, max F 0.900, max E 0.950 and MAE 0.041 1907.06781 tab. 4.

Card: Rethinking RGB-D Salient Object Detection. Its test-set and checkpoint pitfalls are in the card and in Contradictions below.

Four-measure protocol for camouflaged objects (Sα, Eφ, Fβʷ, M)

The SINet-V2 paper scores concealed-object maps with four numbers: S-measure (Sα, higher is better), mean E-measure (Eφ), weighted F-measure (Fβʷ) and mean absolute error (M, lower is better) 2102.10274 §5.1. It reports them on CHAMELEON (76 images), CAMO test and COD10K test, with twelve baselines trained on the same combined CAMO and COD10K training sets [arxiv:2102.10274 §5.1, tab. 2]. Author-reported on COD10K test: Sα 0.815, Eφ 0.887, Fβʷ 0.680 and M 0.037 2102.10274 tab. 2. The card does not name the source of its S-measure definition, so do not assume it matches the section below without checking the original formula. The per-super-class table reports the same four numbers for four COD10K groups 2102.10274 tab. 3.

Card: Concealed Object Detection. Pitfalls on this protocol, including the test-set selection rule and the size conflict for the COD10K test split, are in the card and in Contradictions below.

Single-threshold F and IoU with an empty-GT convention (ANet protocol)

The Anabranch Network paper scores camouflage maps with F-beta at β² = 0.3, IoU and MAE 2105.09451 §5.2, eq. 1. The F and IoU values depend on a binarisation step, which the paper runs in two settings 2105.09451 §5.2:

• Presence assumed: the threshold is the map's mean plus its standard deviation.
• Presence not guaranteed: the threshold is a fixed 0.5.

MAE is computed on the raw map, without binarisation 2105.09451 §5.2. This F-measure is taken at one threshold. It is not the maximum over thresholds used in the EGNet table, so the two sets of numbers are not comparable.

The CAMO-COCO set adds 1,250 images with all-zero GT masks 2105.09451 §5.1. F and IoU are undefined when both the prediction and the mask are empty, and the paper gives no convention for that case 2105.09451 §5.2. A convention must be stated in any reported number. The CAMO test split has only 250 images, so a few hundredths of F may fall within noise 2105.09451 §5.1. The paper does not state whether F and IoU are averaged per image or pooled over the dataset 2105.09451 §5.2.

Card: Anabranch Network for Camouflaged Object Segmentation. Its headline numbers are author-reported: on CAMO-COCO, ANet-SRM reaches F 0.826 and IoU 0.732 at the adaptive threshold, against 0.699 and 0.535 for the pre-trained SRM 2105.09451 tab. 3.

Overlap measures on binary masks (Jaccard/IoU, Dice, F-beta)

For a predicted mask and a GT mask, count true positives (tp), false positives (fp) and false negatives (fn) over pixels. Jaccard is tp / (tp + fp + fn), which equals the intersection-over-union of the two masks [arxiv:2011.07631 eq. 1, eq. 4]. Dice (DSC) is 2·tp / (2·tp + fp + fn) 2011.07631 eq. 3. F-beta 2011.07631 eq. 7 weights recall above precision when β > 1. Precision and recall are the usual ratios 2011.07631 eq. 5, eq. 6. The paper also reports accuracy 2011.07631 eq. 8.

Metrics Reloaded gives Dice or IoU as the default overlap measure for semantic segmentation, F-beta when false positives or false negatives are preferred, and clDice for tubular structures 2206.01653 § SemS. It also notes that Dice equals the pixel-level F1 score and that, per mask pair, Dice is 2·IoU/(1+IoU) 2206.01653 § Metric Pool. That identity holds for one pair of masks. Reported numbers are dataset averages, so do not convert one average into the other.

The surgical-instrument paper uses the same Jaccard index in two places. Its training loss adds a soft version of J, computed on predicted probabilities, to cross-entropy 1803.01207 eq. 2, eq. 3. Its reported IoU and Dice are the hard overlap measures on thresholded masks [arxiv:1803.01207 tab. 1]. Do not read its loss terms as evaluation metrics. Author-reported binary results for TernausNet-16 are IoU 83.60% and Dice 90.01% 1803.01207 tab. 1. Card: Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning.

The 2019 survey of deep segmentation writes the same link as Jaccard = Dice / (2 − Dice) 1910.07655 §9.1. It also works a small example on a 5×5 mask with one false negative. Dice falls to 0.96 on a large object and to 0.66 on a small one, while cross-entropy gives 0.83 for both [arxiv:1910.07655 §4.4, tab. 1, §10.3]. The survey uses this to argue about training losses, not to rank evaluation metrics, so read it as an illustration of how one error weighs more on a small target. The card is here.

Metrics Reloaded also says to compute values per image and then aggregate over images, not over all pixels at once, because pixels within an image are correlated 2206.01653 § SemS.

The Kvasir-SEG polyp benchmark reports these on 120 validation images 2011.07631 §V-B, tab. V. The released code also uses those images for checkpoint selection, and it averages scores per image. The card covers both. The top method is the same for Dice and Jaccard in that table, but the scores differ by method, so do not convert one into the other.

Card: Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning. Its own headline numbers are author-reported: ColonSegNet reaches Dice 0.8206 at 182.38 FPS, and the best Dice in the paper is UNet with ResNet34 at 0.8757 and 35.00 FPS 2011.07631 tab. V. The authors name YOLOv4 as the best detector because of its balance of AP, IoU and speed 2011.07631 §VI.

The HarDNet-MSEG paper gives the same two overlap formulas: Dice = 2·tp / (2·tp + fp + fn), and mIoU = tp / (tp + fp + fn), which is the Jaccard index above 2101.07172 §4.3. Its setup A table reports U-Net with ResNet34 at 0.876 mean Dice, and the caption says the baseline rows are taken from jha2020real 2101.07172 tab. 1. That matches 0.8757 in 2011.07631 tab. V to rounding.

Card: HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS. Author-reported: 0.904 mean Dice at 86.7 FPS on Kvasir-SEG under setup A, and 0.912 mean Dice under the PraNet protocol (setup B) 2101.07172 tab. 1, tab. 2. The two setups differ in input size, optimizer and training data 2101.07172 §4.2, so do not read them as one benchmark.

Threshold-swept overlap (HarDNet-MSEG protocol)

This variant scores a probability map without choosing one threshold. The paper does not state the threshold or the aggregation. The authors' MATLAB evaluation, which the README says produced the report, normalises each map to [0, 1], computes Dice at 256 thresholds from 1 down to 0 in steps of 1/255, averages those values, and also reports the maximum 2101.07172 §4.3 (see the card, Pitfalls). The average is a different quantity from Dice at a fixed threshold such as 0.5. A number from this protocol should not be placed beside a fixed-threshold Dice without saying so.

The same card separates two Dice values. The Python Dice in the training script is a soft Dice on normalised probabilities, averaged over a fixed number of iterations. It selects checkpoints and is not the table metric (see the card, Pitfalls).

Boundary and tolerance measures (NSD)

Metrics Reloaded adds a boundary metric. It recommends normalised surface distance (NSD) by default for segmentation when annotation imprecision should be compensated. Otherwise the choice depends on whether a boundary error should be penalised by its existence or by its distance [arxiv:2206.01653 § SemS, fig. 2].

NSD has two parameters that the paper text does not fix: a tolerance tau and the pixel spacing. The released code uses tau = 1 when none is given and warns. Surface distances use the pixel spacing as sampling, and without a spacing, distances and tau are in voxel units [arxiv:2206.01653 § Paper vs code]. Set tau explicitly in any implementation. The default does not follow from the paper. The code returns NaN for NSD when prediction and GT are both empty. How empty cases should be scored is not stated in the paper 2206.01653 § Paper vs code.

The nnU-Net Revisited benchmark fixes the tolerance: it reports NSD at 2 mm as its secondary metric, next to Dice 2404.09556 §3.2, tab. 4. Its NSD values depend on that tolerance, so do not compare them with NSD computed at another tau.

Card: Metrics Reloaded: Recommendations for image analysis validation.

Five-fold cross-validation with a dataset suitability check (nnU-Net Revisited)

This variant is a validation design, not a new formula. Every method is scored with five-fold cross-validation on the same nnU-Net splits. The primary metric is Dice and the secondary metric is NSD at 2 mm. Both are averaged over classes, then over folds 2404.09556 §3.2, tab. 4. For BraTS2021 and KiTS2023 the scores are computed on hierarchical evaluation regions, not on the non-overlapping classes 2404.09556 §3.2. The paper compares CNN, Transformer and Mamba 3-D models on six datasets (BTCV, ACDC, LiTS, BraTS2021, KiTS2023, AMOS2022) 2404.09556 §3.1, tab. 1.

To pick datasets, the paper scores each one by its suitability: the spread of the methods' fold-mean scores (inter-method SD) divided by the average spread across folds within a method (intra-method SD) [arxiv:2404.09556 §3.2, tab. 3]. A higher ratio means the dataset separates methods better. The paper gives no SD convention, so state which one you used (see the card, Pseudocode in its Method section).

The ratio depends on the method set. Across all methods the ratios are 94% on BTCV and 474% on AMOS. Without the Auto3DSeg runs, BTCV falls to 65% and ACDC falls from 357% to 102% 2404.09556 tab. 3. The authors name ACDC, AMOS and KiTS as the most suitable sets 2404.09556 §4. That ranking is sensitive to which methods are included, so do not treat it as a fixed property of the data.

Card: nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation. Its own headline numbers are author-reported. The mean Dice of the original nnU-Net on BTCV is 83.08 and the best CNN on that set is MedNeXt L k5 at 85.04 2404.09556 tab. 1.

Per-organ Dice and Hausdorff distance on 3-D volumes (TransUNet protocol)

On the Synapse multi-organ CT set, the paper reports Dice for each organ, the average Dice over organs, and an average HD in millimetres [arxiv:2102.04306 §4.1, tab. 1]. It scores each 2-D slice on its own and stacks the slices into a volume 2102.04306 §4.2. Author-reported: TransUNet reaches 77.48% average DSC and 31.69 mm HD. The best CNN baseline in the same table, R50-AttnUNet, reaches 75.57% DSC 2102.04306 tab. 1. On ACDC cardiac MRI the paper reports 89.71% average DSC over RV, Myo and LV 2102.04306 tab. 5.

The released code scores edge cases in a way the paper does not state. An empty prediction against a non-empty ground truth gets DSC 0 and HD 0. A non-empty prediction against an empty ground truth gets DSC 1 and HD 0. Both HD values are the best possible, so a missed organ and a false organ can both look good on HD (see the card, Pitfalls). The paper reports one run per cell with no seeds or variance 2102.04306 tab. 1–5, so treat differences of about one DSC point as uncertain.

Card: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation.

Choosing the metric from the problem (Metrics Reloaded)

A Delphi-consensus process produced a three-step guide [arxiv:2206.01653 § Step 1, § Step 2, § Step 3]:

1. Fingerprint. Map the problem to one of four categories, then answer fingerprint items about the domain, the target structure, the data set and the algorithm's output 2206.01653 § Results: problem fingerprints.
2. Select. Walk the decision tree of that category, which adds metrics to a pool 2206.01653 § Step 2. For object detection the tree depends on the localisation criterion and the assignment strategy, both drawn in the supplement 2206.01653 § ObD, fig. 2.
3. Apply. Use the cross-topic recommendations for implementation, aggregation, ranking, interpretation and reporting 2206.01653 § Step 3.

The paper recommends several complementary metrics instead of one. It was built from five international workshops, nine surveys and crowdsourced feedback between 2020 and 2022. Its consensus is above 75% agreement for all ten components 2206.01653 § Results: framework, § Step 2. The authors say no method exists to compare metric sets quantitatively, so the guide is validated by consensus, not by proof [arxiv:2206.01653 § Results: validation by consensus].

Detection average precision (AP over IoU thresholds)

For boxes, a detection counts as correct when its IoU with a GT box reaches a threshold. AP is the area under the precision-recall curve, using interpolated precision 2011.07631 eq. 1, eq. 2. The paper averages AP over thresholds from 0.25 to 0.75 in steps of 0.05 2011.07631 §V-A. Its detection table also reports AP at single thresholds (AP25, AP50, AP75) and IoU 2011.07631 tab. III.

The ColonSegNet box numbers depend on a mask-to-box conversion that the paper does not describe 2011.07631 §III. The card's Paper vs code section has the details.

Metrics Reloaded gives the detection rule. Without continuous class scores, use a counting metric such as F-beta. With class scores, complement the counting metric with AP or FROC. Accuracy, specificity and AUROC are invalid for detection, because there is no meaningful count of true negatives [arxiv:2206.01653 § ObD].

Structure-measure (S-measure)

S-measure averages two terms 1708.00786 §4, eq. 10:

• Region-aware term S_r. Split the prediction and the GT into four blocks at the centroid of the GT foreground. Compute SSIM in each block, without SSIM's stabilising constants, and take a weighted sum 1708.00786 §4.1, eq. 3, eq. 4.
• Object-aware term S_o. Inside the GT foreground, reward a high mean prediction and penalise its spread (coefficient of variation, weight λ). Do the same on the complement (1 − map) inside the GT background. Combine the two with the foreground area fraction as weight 1708.00786 §4.2, eq. 7, eq. 8, eq. 9.

S = α·S_o + (1 − α)·S_r with α = 0.5. The result lies in [0, 1] and higher is better 1708.00786 §4.3. Reported runtime is 5.3 ms per image on one 4 GHz CPU thread in Matlab 1708.00786 §5.

Reported against AP, AUC and Fbw on PASCAL-S, ECSSD, SOD and HKU-IS 1708.00786 tab. 1:

S-measure is best in 10 of 12 cells. It ties AP on SOD MM2, and Fbw beats it on ECSSD MM2 1708.00786 tab. 1. On MM4 it does better than AP and AUC but worse than Fbw 1708.00786 §5.4, fig. 6. In the user study (45 viewers, 50 pairs), the map ranked top by S-measure was preferred in 63.69% of trials against Fbw, 72.11% against AP and 73.56% against AUC 1708.00786 §5.6. Pairs whose top maps looked too similar were removed beforehand (100 down to 50) 1708.00786 §5.6.

Card: Structure-measure: A New Way to Evaluate Foreground Maps.

Language-guided multi-target masks (gIoU and cIoU)

PixelLM answers in text and writes one mask per target named in the answer. It reports gIoU and cIoU on its MUSE benchmark, on multi-referring segmentation and on the refCOCO family [arxiv:2312.02228 tab. 1, tab. 2, tab. 3]. The card does not define gIoU or cIoU. Read the paper's definitions before comparing these numbers with any other protocol.

The MUSE protocol is not plain pixel overlap. Predicted masks are matched to GT masks with the Hungarian algorithm, using a BCE + DICE cost. Each matched mask's GT text replaces its segment in the answer. A GPT-3.5 prompt then scores each prediction. Per-prediction IoU is kept only where that score exceeds 0.5, and the image score is the sum of those IoUs divided by the padded set size P 2312.02228 §4.3, app. A. Unmatched masks are paired with empty masks 2312.02228 app. A.

On MUSE test overall, PixelLM-Llama2-13B reports 42.3 gIoU / 51.0 cIoU. A LISA variant trained on MUSE reports 41.9 / 50.5, and LISA without that training reports 14.4 / 28.4 2312.02228 tab. 1. These are author-reported. The gap to the MUSE-trained LISA is small on this split.

Card: PixelLM: Pixel Reasoning with Large Multimodal Model.

When to use which

Implementation sketch

Two families. Both are pixel-count based. They differ in input (map or mask) and in how they treat object structure.

S-measure as the official code computes it 1708.00786 §4. The card's Pitfalls cover where the code and the paper differ.

s_measure(pred in [0,1], gt boolean, both H×W):
    m = fraction of gt that is foreground
    if m == 0: return 1 - mean(pred)          # code only, not in paper
    if m == 1: return mean(pred)

    # object term
    obj(v)  = 2*mean(v) / (mean(v)^2 + 1 + std(v) + eps)   # std uses N-1; equals λ = 0.5
    S_o = m * obj(pred[gt]) + (1 - m) * obj(1 - pred[!gt])

    # region term
    (X, Y) = rounded centroid of gt foreground
    split pred and gt into 4 blocks at row Y, column X
    for each block k:
        w_k  = block area / image area        # paper: share of gt foreground in block
        ssim_k = 4*x̄*ȳ*cov / ((x̄²+ȳ²)(var_x+var_y) + eps)   # no C1/C2; zero-case branches
    S_r = sum_k w_k * ssim_k

    return max(0, 0.5*S_o + 0.5*S_r)

Where variants differ: pixel-wise measures replace both terms with a confusion-count statistic computed over thresholds or with per-pixel weights. Do not swap in a library SSIM. Those use constants and sliding windows, and the authors report that global or sliding-window SSIM did not capture region structure 1708.00786 §4.1.

Overlap measures on binary masks 2011.07631 eq. 1, eq. 3:

overlap(pred boolean, gt boolean, both H×W):
    # pixel counts over the whole mask
    tp = count(pred & gt)
    fp = count(pred & !gt)
    fn = count(!pred & gt)

    # empty-mask handling is not stated in the paper
    jaccard = tp / (tp + fp + fn)            # equals IoU of the two masks
    dice    = 2*tp / (2*tp + fp + fn)
    return jaccard, dice

# average the per-image scores over the dataset; the paper does not say
# whether it pools pixels across images (see the card's Pitfalls)

Threshold-swept variant, as the HarDNet-MSEG authors' MATLAB code does it 2101.07172 §4.3 (see the card, Pitfalls):

threshold_swept_dice(prob in [0,1], gt boolean):
    # normalise each map to [0, 1], per image, before thresholding
    prob = (prob - min(prob)) / (max(prob) - min(prob) + eps)
    scores = []
    # 256 thresholds, from 1 down to 0 in steps of 1/255
    for t in [k / 255 for k in 255..0]:
        scores.append(dice(prob >= t, gt))    # dice as in the block above
    # the mean is the averaged score; the maximum is the best-threshold score
    return mean(scores), max(scores)

Per-organ scoring on 3-D label volumes, with the edge cases as the TransUNet code handles them 2102.04306 §4.1 (see the card, Pitfalls). The paper does not state these cases:

organ_scores(pred, gt, organ):          # 3-D label volumes
    p = (pred == organ)
    g = (gt == organ)
    # one-sided empty cases, as the released code scores them
    if p.any() and not g.any(): return dsc=1, hd=0
    if g.any() and not p.any(): return dsc=0, hd=0
    # both empty: not described in the card; choose and state a rule
    dsc = 2 * (p & g).sum() / (p.sum() + g.sum())
    hd  = hausdorff95(p, g)             # the code reports HD95
    return dsc, hd

# average dsc and hd over the eight organs; background is not averaged

Contradictions

Within 1708.00786:

• Block weights: the text weights each block by how much GT foreground it covers 1708.00786 §4.1. The official code weights by block area instead (see the card). Results computed the two ways are not comparable.
• λ: the experiments use λ = 0.5 1708.00786 §5, but the Fig. 2 caption states λ = 0.25 1708.00786 fig. 2.
• MM4: the authors use MM4 on ASD1000 but argue it is unreliable and drop it from the multi-dataset table 1708.00786 §5.4. MM4 is the one meta-measure where S-measure is not best.

Within 1803.01207:

• Binary threshold: the paper applies a threshold of 0.3, chosen on validation data 1803.01207 §2.3. The validation code uses probability 0.5 and the evaluation script uses 0.5 validation.py evaluate.py. No code selects a threshold (see the card, Paper vs code).
• Per-image or per-file averaging: the README says the binary score is averaged per video, but evaluate.py averages over all files. The two can give different numbers on the same predictions 1803.01207 § Pitfalls.

Within 2011.07631:

• F2 definition: the paper defines F2 with β = 2 2011.07631 eq. 7. The released code computes its "F2" with β = 1, so it equals Dice (see the card, Paper vs code). The F2 column in the table therefore matches Dice for ColonSegNet 2011.07631 tab. V. Recompute F2 from the counts before comparing it with other papers.
• Validation or test: the figure captions call the 120 images the "test dataset", while the text calls them "validation" 2011.07631 §V-B. The released code also selects its checkpoint on those images (see the card), so treat the reported numbers as validation results.

Within 2312.02228:

• Token-fusion gain: the text says token fusion gives "up to 2.4%" cIoU on MUSE val 2312.02228 §5.5. Tab. 4 gives 48.0 to 50.1, a gain of 2.1 2312.02228 tab. 4.
• MUSE scoring: §4.3 says the GPT score from 1 to 10 is multiplied with the IoU. App. A keeps the intersection only if the score is above 0.5. The text does not say how 1–10 scores map to that threshold 2312.02228 §4.3, app. A.
• Best on referring tasks: the text says PixelLM is best on most metrics 2312.02228 §5.4. LISA is higher on refCOCO val and testB 2312.02228 tab. 3.

Within 1908.08297:

• None in the evaluation numbers. The text's figures match a recomputation from the tables: the 1.9-point average F-measure gain over PiCANet 1908.08297 §4.4 matches tab. 2, and the 3.1 and 2.4 point ablation gains 1908.08297 §4.3.2 match tab. 3.

Within 2102.04306:

• Table 1, R50-AttnUNet row. The eight per-organ DSCs in the row average to 72.10, not the 75.57 printed as the average 2102.04306 tab. 1. Reaching 75.57 would need an aorta value of about 83.7 instead of 55.92. The paper does not explain the gap, so use this row with care.
• HD or HD95. The text says "Hausdorff Distance (HD)" and does not say whether it is HD95 2102.04306 §4.1. The code reports HD95 (see the card, Paper vs code).

Within 2102.10274:

• Test-set size. Section 3.3 and tab. 1 give 4,000 COD10K test images, but the COD10K results section says 2,026 2102.10274 §3.3, tab. 1, §5.2. The paper does not explain the gap.
• Super-class count. Section 3.1 lists 10 super-classes and Sections 3.2 and 3.3 each say five concealed super-classes, but tab. 3 shows four 2102.10274 §3.1, §3.2, §3.3, tab. 3.

Within 2105.09451:

• Classifier width. The text gives 4096 hidden units and Table 1 gives 2048 2105.09451 §4.2, tab. 1. This affects the classification stream only, not the F and IoU numbers.

Within 1910.07655:

• None in the evaluation content. Its Dice–Jaccard link matches the one in 2206.01653 1910.07655 §9.1 2206.01653 § Metric Pool. Its mean-IoU table is collected from other papers, each under its own protocol 1910.07655 tab. 2, so the table does not compare methods on one protocol.

Within 2101.07172:

• Metric definition: the Dice and mIoU formulas are given, but the threshold and the aggregation are not stated in the text. The MATLAB evaluation averages over 256 thresholds, while the training script uses a different soft Dice (see the card, Pitfalls) 2101.07172 §4.3.
• F2 formula: the paper prints F2 = 5pr/(4p+r) and does not define p or r 2101.07172 §4.3. This has the form of the β = 2 F-measure in 2011.07631 eq. 7, which reads p and r as precision and recall. The paper does not say so.
• Setup A optimizer: the paper gives SGD with learning rate 1e-2 for setup A, but the code's default is Adam with 1e-4 2101.07172 §4.2 (see the card, Paper vs code). The setup A number cannot be reproduced from the defaults.

Across papers:

• Averaging convention: 2011.07631 computes each image's score at a 0.5 threshold and averages per image in its released code (see the card, Pitfalls). 2101.07172 averages Dice over thresholds in its MATLAB code 2101.07172 §4.3. Both are reported as Kvasir-SEG Dice, so the numbers are not interchangeable without the protocol. 2404.09556 averages over classes and then over folds 2404.09556 §3.2. The card does not say whether it aggregates per image, so do not assume that Metrics Reloaded's per-image rule 2206.01653 § SemS was followed.
• Checkpoint rule: 2101.07172 keeps the checkpoint with the best Dice on the test folder (see the card, Pitfalls). 1907.06781 keeps the checkpoint with the lowest MAE on the NJU2K test split, which is also one of its reported sets (see the card, Paper vs code). 2102.04306 evaluates the last epoch with no selection on the held-out cases (see the card, Paper vs code). 2011.07631 selects on its 120 images. 2102.10274 does not state a selection rule, but its released code keeps the epoch with the lowest MAE on CAMO test by default, and CAMO test is one of its reported sets (see the card, Pitfalls). Scores that used different rules are not comparable.
• Empty-case rule: 2206.01653 returns NaN for NSD when prediction and GT are both empty, and the paper does not state how empty cases should be scored 2206.01653 § Paper vs code. 2102.04306's code gives DSC 0 or 1 with HD 0 for a one-sided empty case, and its both-empty case is not in the card (see the card, Pitfalls). The two libraries disagree, so state the rule in any reported number. 2105.09451 leaves the empty-GT case for F and IoU undefined in its text (see the card, Pitfalls).
• Baseline rows: the card says the setup A baseline rows are taken from jha2020real 2101.07172 tab. 1 caption. The card does not say whether those rows and the HarDNet-MSEG row were computed under the same protocol.

Otherwise the cards measure different things. Saliency maps use the S-measure, Fbw and max F-measure family. Binary masks use overlap measures, HD and detection AP. The 3-D nnU-Net protocol uses Dice and NSD on volumes 2404.09556 tab. 1, tab. 4. The MUSE protocol is a separate protocol. Metrics Reloaded does not cover saliency maps, so its recommendations do not extend to S-measure. Its recommended Dice and IoU defaults agree with the Kvasir-SEG use of both 2206.01653 § SemS 2011.07631 tab. V. Compare within a family only.

Papers

• Structure-measure: A New Way to Evaluate Foreground Maps: proposes S-measure, a region- plus object-aware structural similarity score for non-binary foreground maps, and compares it with AP, AUC and Fbw on five meta-measures.
• Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks: D3Net, an RGB-D saliency model, benchmarked with S-measure, max F-measure, max E-measure and MAE on seven sets, including the phone-captured SIP set.
• EGNet: Edge Guidance Network for Salient Object Detection: a salient-object model with an edge branch, reported with max F-measure, MAE and S-measure on six benchmarks.
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation: a CNN-Transformer hybrid for medical segmentation, reported with per-organ DSC and HD on Synapse CT and average DSC on ACDC MRI.
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning: U-Net-style networks for surgical tool masks, trained with cross-entropy plus a soft-Jaccard term, reporting binary, parts and instrument-type IoU and Dice.
• Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning: benchmarks detectors and segmenters on Kvasir-SEG with Jaccard, Dice, F-beta, precision, recall, accuracy and AP, and reports speed alongside.
• HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS: a polyp segmenter that reports mean Dice, mIoU and F2 on Kvasir-SEG and four other sets, with threshold-swept scoring in its evaluation code.
• Metrics reloaded: Recommendations for image analysis validation: a Delphi-consensus decision framework for choosing validation metrics for classification, detection and segmentation, with pitfalls. It reports no benchmark numbers.
• Concealed Object Detection: SINet-V2, a two-stage camouflaged-object network, reported with Sα, Eφ, Fβʷ and M on CHAMELEON, CAMO and COD10K; it also introduces the COD10K dataset.
• Anabranch Network for Camouflaged Object Segmentation: a saliency network with a camouflage classifier whose probability scales the map, reported with F-beta, IoU and MAE on CAMO and CAMO-COCO with two binarisation settings.
• PixelLM: Pixel Reasoning with Large Multimodal Model: a multimodal model that writes one mask per target named in its answer, evaluated with gIoU and cIoU on MUSE, multi-referring and refCOCO-family benchmarks.
• nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation: benchmarks CNN, Transformer and Mamba 3-D models with five-fold CV on six datasets, reporting Dice and NSD at 2 mm and a dataset suitability ratio.
• Deep Semantic Segmentation of Natural and Medical Images: A Review: a survey of deep segmentation models with no experiments of its own. For this concept it gives the Dice–Jaccard link and a small-object example; its mean-IoU table is collected from other papers.