Deep Semantic Segmentation of Natural and Medical Images: A Review

In plain words

Semantic segmentation means labelling every pixel of an image with a class, such as "road", "organ" or "tumour". This review looks at how deep neural networks do that, for ordinary photos and for medical scans. It sorts the published work into six groups: the network design, the loss function used for training, generating extra training images, learning from weak labels, models that predict several tasks at once, and models that process sequences.

A large part of the review is about the loss function, which tells the network how wrong a prediction is. The plain cross-entropy loss treats every pixel the same, so a small object can be swamped by the background. Overlap losses, such as the Dice loss, score how much the predicted region matches the true one, and they help with that imbalance, but they can behave badly on very small objects. The review shows this with a simple example and suggests mixing the two kinds. It also lists the datasets and the benchmark scores reported by other papers, so a reader can see where the field stands. The review runs no new experiments.

TL;DR

• A survey, not a method: six groups of deep segmentation work for natural and medical images, with the datasets and reported scores for each §1 §2–§8 §9.
• The loss section is the most implementable part: cross-entropy, weighted and balanced variants, focal, Dice, Tversky, exponential-logarithmic, Lovász-Softmax, boundary and conservative losses §4. Medical-specific losses follow in §5.
• The authors show, on synthetic overlap cases, that overlap-only losses can be problematic for smooth optimisation when the foreground is small, and they suggest pairing them with cross-entropy §4 §10.3.
• All performance numbers (for example PASCAL VOC 2012 mean IoU) are author-reported in the papers the review cites tab. 2 tab. 3.

Problem

Deep networks for segmentation must handle large amounts of image data, a wide range of imaging modalities, and few labelled medical images §1 §10.4. Medical volumes are often too large to load into GPU memory and must be processed as patches or sub-volumes, which limits how much spatial context a model can use §10.4. The review argues that a large share of the published medical work tries to solve class imbalance and annotation scarcity through the loss function, the training data or weak labels §4 §6 §7.

The review states its scope as deep, end-to-end trainable segmentation models for 2D (RGB and grayscale) and volumetric images. Classical shallow-network and superpixel methods are excluded §2 intro §1.

Method

This is a survey. It has no method of its own; the sections below summarise the groups it defines §1.

Architectures, natural images §2. Fully convolutional networks §2.1; encoder-decoder networks, including the volumetric V-Net with its 3D operations and Dice optimisation §2.2; complexity reduction, such as lighter backbones and pruning §2.3; attention-based models §2.4; and adversarially trained segmenters §2.5. The review calls encoder-decoder networks with skip connections the winning architectures of the state of the art, and notes atrous convolutions with pyramid pooling as widely used §10.1.

Architectures, medical images §3. Model compression §3.1; encoder-decoder networks §3.2; attention §3.3; adversarial training §3.4; and sequenced models §3.5.

Loss functions §4. Cross-entropy for the binary and multi-class case §4.1; weighted and balanced cross-entropy, including a distance-weighted term §4.2; focal loss §4.3; and overlap-based losses §4.4: Dice §4.4.1, Tversky §4.4.2, exponential-logarithmic §4.4.3, Lovász-Softmax §4.4.4, boundary §4.4.5, and conservative loss for domain adaptation §4.4.6. Further options are direct optimisation of segmentation metrics, loss weighting, and shape regularisers §4 end.

Loss functions, medical images §5. Smoothness regularisers on predictions under input and network perturbation §5 [li2019transformation]; an active-contour energy used as a loss §5 [chen2019learning]; a Hausdorff-based loss with a Dice term §5 [karimi2019reducing]; distance-to- boundary weights §5 [caliva2019distance]; a level-set regulariser §5 [kim2019multiphase]; and a combined cross-entropy and Dice loss for explicit control of false positives and negatives §5 [taghanaki2019combo].

Image synthesis §6. Standard geometric and colour augmentation is described as background; the section covers methods whose output is new image data. For natural images, a Wasserstein GAN with gradient penalty generates image and mask pairs §6.1 [neff2018generative]. For medical images, conditional GANs translate between modalities, for example cardiac MR from CT, and the authors report that the synthetic data increased segmentation accuracy §6.2 [chartsias2017adversarial] [zhang2018translating].

Weakly supervised methods §7. Image-level or sparse labels in place of full masks. For medical images, a differentiable term in the loss for weakly labelled data is described for cardiac images §7 [KERVADEC201988]; a fully convolutional model trained with weak labels is described in [afshari2019weakly] §7. A further weakly supervised spinal-cord model appears in tab. 3 [Perone2019-eo].

Multi-task models §8. Segmentation trained together with another task. Natural-image examples use a cascaded multi-task loss for building footprints §8 [bischke2019multi] and an extra mask branch on a detector §8 [he2017mask]. For medical images, a patient-detection head and a skin segmentation head share one VGG16-based network §8 [chaichulee2017multi]; Mask R-CNN is applied to cell and nucleus tasks §8.

Metrics and datasets §9. Pixel-count measures and overlap measures, with the identity between them §9.1. Datasets for natural images §9.2 and medical images §9.3 are described in prose, with a table of PASCAL VOC 2012 results tab. 2 and a table of medical papers with their data tab. 3.

Discussion §10. Future directions: skip-connection cost and what to transfer through them §10.1; 3D convolutions against slice-by-slice sequence models §10.2; overlap versus cross-entropy losses §10.3; prior knowledge, shared benchmarks, simulators, patch-based processing and multi-modal input §10.4.

Loss definitions

The review writes its binary notation with p for the ground-truth value and p̂ for the predicted probability. Its text first defines P(Y=0) = p, which reverses the usual convention for the positive class §4.1. The pseudocode below follows the formulas as printed and uses the usual convention: target is 1 on the object.

# Soft Dice loss, one class  [§4.4.1]
# probs: predicted foreground probabilities (N,); target: 0/1 (N,)
def dice_loss(probs, target, eps):
    # Overlap term: sum of p * p_hat over pixels  [§4.4.1]
    intersection = sum(target * probs)
    # Sum of the two masks' L1 norms  [§4.4.1]
    total = sum(target) + sum(probs)
    # The printed formula is the Dice coefficient; the loss is one minus it.
    # eps is the smoothing term of the exponential-logarithmic variant
    # [§4.4.3]; the plain Dice formula has no eps  [§4.4.1]
    return 1 - (2 * intersection + eps) / (total + eps)

# Focal loss, binary  [§4.3]
# alpha weights the foreground class, gamma down-weights easy pixels
def focal_loss(probs, target, alpha, gamma):
    # Foreground term: alpha * (1 - p_hat)^gamma * p * log(p_hat)  [§4.3]
    foreground = alpha * (1 - probs) ** gamma * target * log(probs)
    # Background term: (1 - alpha) * p_hat^gamma * (1 - p) * log(1 - p_hat)
    # [§4.3]
    background = (1 - alpha) * probs ** gamma * (1 - target) * log(1 - probs)
    # Negated sum over pixels  [§4.3]
    return -sum(foreground + background)

Overlap scores from a binary mask pair, as defined in §9.1:

# Overlap scores from binary masks  [§9.1]
# prediction, truth: boolean (H, W)
def overlap_scores(prediction, truth):
    # Confusion counts  [§9.1]
    tp = sum(prediction & truth)
    fp = sum(prediction & ~truth)
    fn = sum(~prediction & truth)
    # Dice = 2 TP / (2 TP + FP + FN)  [§9.1]
    dice = 2 * tp / (2 * tp + fp + fn)
    # Jaccard = TP / (TP + FP + FN); equals Dice / (2 - Dice)  [§9.1]
    jaccard = tp / (tp + fp + fn)
    return dice, jaccard

Hyperparameters

The review states no training hyperparameters for any method; it is a survey. The table lists the parameter values the review chose for its own loss-behaviour plots §4.4 and the values it quotes from the cited papers.

Results

Author-reported. The review runs no experiments of its own. The only numbers below are the PASCAL VOC 2012 mean IoU values the review collected from other papers tab. 2, with the datasets each paper evaluated on. Different papers use different backbones, training data and protocols, so the values are not directly comparable.

The medical-image table tab. 3 lists 19 papers with the modality, site and number of training items for each, and whether the paper proposes an architecture, a loss, synthesis, a multi-task model or weak supervision. Its counts follow a footnote: where a paper reports several units (patients, volumes, slices), the count is for patients tab. 3. Examples include the U-Net-style network on 30 electron-microscopy images, V-Net on 50 prostate MRI volumes tab. 3, and a retinal-vessel model on 20 to 28 fundus images tab. 3.

The loss-behaviour figure §4.4 plots seven losses on a large and a small object with increasing false positives and negatives, using the parameters in the Hyperparameters table. The review's own worked example uses a 5×5 mask tab. 1 §9.1 and a toy pair with one false negative: Dice gives 0.96 on the large object and 0.66 on the small one, while cross-entropy gives 0.83 for both §4.4 §10.3. These are illustrative, not benchmark results.

Paper vs code

No official code. The paper links no code repository, and it is a survey with no method to implement. I did not run a GitHub search. Awesome-list repositories that surveys link are not treated as code.

Pitfalls

• The Dice formula printed in §4.4.1 is the coefficient, not the loss. The paper calls it the Dice loss; implement 1 - coefficient.
• The Tversky formula in §4.4.2 is typeset with mismatched brackets. Check the original Tversky paper before implementing; the review's remark that β = 0.5 reduces to Dice is the only check it gives.
• The boundary-loss summation in §4.4.5 is typeset without a \sum sign before p ∈ Ω. The sum is implied by the surrounding text, and the signed-distance definition of φ_G is given there.
• Conservative loss §4.4.6 has a numerical edge case. From the printed formula alone, the inner term log(−log p_t) tends to −∞ as p_t → 1 (derived, not stated in the review). The review does not discuss it.
• Focal loss §4.3. The review says γ = 0 gives BCE. With α ≠ 0.5 the result is a weighted BCE; this card's reading follows the printed formula.
• Overlap losses on small foregrounds. The review's worked example shows Dice changing sharply with one false negative on a small object §4.4 §10.3. Its advice is to use overlap losses with a cross-entropy term rather than alone §10.3.
• Empty masks. Dice and Jaccard are undefined when both masks are empty (derived from the formulas §9.1); the review does not say how to handle this. Not stated.
• Symbol clash in §4.1. p denotes the ground truth, while the text defines P(Y=0) = p, the opposite of the usual positive-class convention.
• Lovász-Softmax §4.4.4 is given as an outer formula only; the hinge and error-sorting steps are not in this review. Not stated.
• Medical counts are not one unit. tab. 3 mixes patients, volumes and slices by footnote; compare with care.
• Benchmarks. The mean-IoU table mixes architectures, backbones and training data from many papers tab. 2. Do not read it as a ranking of methods.
• Dataset descriptions in §9.2 (for example "CamVid: 701 frames, 32 classes") are the review's summaries of the original datasets; check the originals for exact splits.

Lineage

• Builds on: none. The review is positioned against earlier surveys it cites in its introduction §1, such as general semantic-segmentation, RGB-D, medical-imaging and label-noise surveys.
• Compared against: those surveys §1. The review says it differs by covering six groups and both 2D and volumetric medical images §1.
• Superseded by: not stated in the review; no later card in this wiki is known to supersede it.