Metrics reloaded: Recommendations for image analysis validation

In plain words

Researchers who build image analysis systems, such as tools that find cells in microscope images or outline organs in scans, have to score them with some number. Picking that number is harder than it looks. A score that ignores small objects, or that treats an image full of tiny cells the same as one with a single large organ, can make a weak system look strong. This paper describes a decision guide called Metrics Reloaded that helps people choose a sensible set of scores.

The guide works in two stages. First, you describe your problem by answering a checklist: what kind of task it is, whether the object sizes or the quality of the reference labels matter, and what the system outputs. Second, that checklist leads you through a decision tree that proposes a few scores, and it warns you about common mistakes along the way. The authors recommend using several complementary scores rather than one.

The guide was built by a large group of experts over several rounds of structured voting and feedback. It is a set of recommendations, not a mathematical proof that the chosen scores are best. An online version of the tool existed as a restricted beta at the time of the paper.

TL;DR

Metrics Reloaded is a consensus framework for picking validation metrics for image-level classification, object detection, semantic segmentation and instance segmentation. The user first fills in a problem fingerprint, then follows a decision tree to a pool of reference-based metrics, then applies the metrics with stated pitfalls in mind. The paper reports no benchmark numbers. Its evidence is the Delphi consensus process, and its code is a set of metric implementations in MONAI's MetricsReloaded package.

Problem

Validation metrics in biomedical image analysis often do not match the real clinical or biological question, which hinders scientific progress and translation § Main. The authors group the pitfalls into three kinds: choosing the wrong problem category, poor metric selection for the problem's properties, and poor metric application to a data set, such as ignoring hierarchical data (several images per patient) when aggregating § Main. Scope: classification at the whole-image, object or pixel scale, covering four task types (image-level classification, object detection, semantic segmentation, instance segmentation) § Main fig. 2. It does not cover regression or reconstruction; the discussion lists these as future scope § Discussion.

Method

The method is a process for choosing metrics. It has three steps [§ Step 1, Step 2, Step 3] fig. 2:

1. Fingerprinting. Map the problem to one of four categories (FP1). Then fill in the fingerprint items, which are binary or categorical properties in four families: domain interest (FP2), target structure (FP3), data set (FP4) and algorithm output (FP5) § Results: problem fingerprints fig. 3. The category mapping is itself a subprocess, S1, which the authors used to avoid the category pitfall § Step 1.
2. Metric selection. Walk the decision tree of the chosen category. Each category has its own steps. Each step adds metrics to a pool, and some steps have branches that depend on fingerprint items § Step 2 fig. 2. The steps are summarised in the pseudocode below.
3. Metric application. Apply the chosen metrics to the data set, with the cross-topic recommendations of the supplement, which cover implementation, aggregation, ranking, interpretation and reporting § Step 3.

The main paper gives this decision logic per category. The steps below are a condensed summary of the text; the subprocess flowcharts S1 to S9 are figures in the supplement fig. S1-S9 and are not transcribed here.

# Metrics Reloaded: reference-based metric pool  [§ Step 2]  [fig. 2]
# Input:  fingerprint (FP items), category in ImLC, SemS, ObD, InS
# Output: metric pool (list of metric names)

def select_metrics(category, fingerprint):
    # Pool that will be returned  [§ Step 2]
    metric_pool = []

    if category == "ImLC":
        # 1. Multi-class metric if a decision rule exists  [§ ImLC]
        if fingerprint["decision_rule"]:
            # Unequal misclassification costs (ordinal data) -> EC
            if fingerprint["unequal_costs"]:
                metric_pool.append("expected_cost")
            # Otherwise Accuracy, BA or MCC; depends on class prevalence
            else:
                metric_pool.append("accuracy_or_ba_or_mcc")

        # 2. Per-class counting metric, one-vs-rest  [§ ImLC]
        if fingerprint["decision_rule"]:
            metric_pool.append("f_beta_per_class")

        # 3. Multi-threshold metric  [§ ImLC]
        # AUROC by default; AP if class balance is high
        if fingerprint["high_class_balance"]:
            metric_pool.append("average_precision")
        else:
            metric_pool.append("auroc")

        # 4. Calibration metric if requested  [§ ImLC]
        if fingerprint["calibration_requested"]:
            metric_pool.append("calibration_metrics")

    return metric_pool

The pseudocode above covers only the image-level classification branch. The other three branches are described in the text: segmentation § SemS, object detection § ObD and instance segmentation § InS. A complete implementation should transcribe each branch from those sections and the subprocess figures.

Summary of the branch rules, all from the main text:

• Image-level classification (ImLC): multi-class metric first; then a per-class counting metric such as F-beta when a decision rule exists; then a multi-threshold metric (AUROC by default, AP for high class balance); then calibration if requested § ImLC fig. 2.
• Semantic segmentation (SemS): an overlap metric (DSC or IoU by default; F-beta if false positives or negatives are preferred; clDice for tubular structures) and a boundary metric (NSD by default when annotation imprecision should be compensated; otherwise chosen by whether errors should be penalised by existence or distance) § SemS fig. 2.
• Object detection (ObD): localisation criterion (S8), assignment strategy (S9), then a counting metric such as F-beta when no continuous class scores exist; with class scores, the paper complements a counting metric with AP or FROC § ObD fig. 2.
• Instance segmentation (InS): the detection metrics of ObD with a different localisation rule, and PQ as an alternative to F-beta; then the segmentation metrics applied per true-positive instance § InS fig. 2.

The recommended reference-based metric pool, with formulas and cheat sheets, is in the supplement § Metric Pool tab. metrics and § Metric Cheat Sheets. Several are shown as mutually computable, for example DSC = 2·IoU/(1+IoU), and DSC equals the pixel-level F1 score § Metric Pool.

Hyperparameters

The framework has no trained model, so there are no training hyperparameters. The parameters that matter for implementation are below.

Results

Author-reported. This paper reports no benchmark numbers for metric performance. Its evidence is the consensus process and the use cases. The setup is a multi-stage Delphi process with five international workshops, nine surveys and crowdsourced feedback, conducted between 2020 and 2022 [§ Results: framework].

Paper vs code

The official code is the MetricsReloaded package in the MONAI organisation. The paper links it § Step 3 § Code Availability Statement. The code implements the metrics, the process evaluation and the calibration measures. The code is at commit aa26562dccaa96228813414de9f5b022dedf877c, under Apache-2.0 LICENSE.

• NSD denominator in the docstring. The docstring formula writes the denominator as |S_A| + S_B pairwise_measures.py. The code computes the sum of the two border maps, which is the intended formula pairwise_measures.py. Treat the docstring as a typo.
• NSD tolerance default. The paper text does not give a tau value. The code uses tau = 1 when none is given and warns pairwise_measures.py. Set tau explicitly in an implementation; the default does not follow from the paper.
• Empty masks. The code returns NaN for NSD when both prediction and reference are empty pairwise_measures.py, and for DSC when the denominator is zero pairwise_measures.py. The paper does not state how empty cases are scored. The process evaluation adds a _nanrep column per NaN-capable measure, in which NaN is replaced by the worse or the better value overall_process.py. The replacement values are in the WORSE table (NSD = 0) and the BEST table (NSD = 1) overall_process.py overall_process.py.
• Distance units. Surface distances use the pixel spacing as sampling pairwise_measures.py. If no spacing is given, distances and tau are in voxel units.

No other discrepancies were checked in this pass. The metric functions and the decision logic were not fully compared against the supplement.

Pitfalls

• Wrong problem category. The main failure mode is picking the wrong category, for example treating object detection as segmentation. The framework starts with a category mapping (S1) to prevent this § Step 1 § Main.
• Multi-class metrics hide class-level failure. Multi-class metrics give a single value, which can hide poor performance on single classes. The paper recommends per-class counting metrics and multi-threshold metrics in addition § ImLC.
• No true negatives in detection. In object detection, Accuracy, Specificity and AUROC are invalid, because there is no meaningful count of true negatives. Use only three entries of the confusion matrix § ObD.
• Pixel correlation. For segmentation, metric values should be computed per image and then aggregated over images, not computed over all pixels, because pixels within an image are correlated § SemS.
• Tolerance and spacing. NSD depends on tau and on the pixel spacing. Both must be set explicitly, and the paper does not give a value (see Paper vs code).
• Ambiguous branches. Several branch points depend on fingerprint items that users may not be able to answer, such as whether false positives or false negatives are preferred. The decision guides in the supplement resolve these, but they are text the card does not transcribe § Step 2 [supplement: decision guides].
• Consensus, not proof. The framework's recommendations come from expert consensus. The authors say no method exists for quantitatively comparing one metric set with another, so the validation is by consensus and community feedback § Results: validation by consensus.
• Tool access. The online tool was a restricted beta when the paper was written § Results: online tool. The paper does not state an API.

Lineage

• Builds on: the pitfall study and taxonomy in the sister publications (Reinke et al., 2021; Reinke, Tizabi et al., 2022), cited in the paper for the pitfall categories § Step 1 § Step 3. Those papers are not in this wiki.
• Compared against: no quantitative comparison; the paper compares the framework against common practice qualitatively § Main.
• Superseded by: not stated.