Dice loss

Overview

Dice loss scores how much a predicted mask overlaps the true mask and turns that overlap into a loss. The Dice coefficient for a binary mask pair is 2\,TP / (2\,TP + FP + FN) 1910.07655 §9.1. The loss is one minus the coefficient. The review that defines the formula prints the coefficient and calls it the loss; the loss is 1 - \text{coefficient} 1910.07655 §4.4.1. A second survey gives the same set-form definition and the same 1 - \text{DSC} loss, but does not write the probability form 2307.02694 §6.1.2.

To train with it, the cards replace the hard counts with predicted probabilities, so the score is differentiable. Each pixel contributes its probability times its target, and the denominator is the sum of the two masks 1910.07655 §4.4.1. Because the score is taken over the whole region, it does not weight each pixel by how confident it is. This is the reason it helps with class imbalance, and also the reason it can misbehave on small objects 1910.07655 §4.4 1910.07655 §10.3.

The cards use the loss in four shapes: plain soft Dice 1910.07655 §4.4.1, Dice added to binary cross-entropy in a polyp segmenter 2108.00882 §3.4, Dice combined with cross-entropy in a hybrid CNN-transformer for organ segmentation, where the paper does not state the loss and the released code sets the weights trainer.py, and a soft-Jaccard term combined with cross-entropy for surgical-tool segmentation 1803.01207 §2.3. The last one is an intersection-over-union score, not Dice. The Tversky and exponential-logarithmic variants are described in the surveys, but no card here trains with them 1910.07655 §4.4.2 1910.07655 §4.4.3 2307.02694 §6.1.

Dice and the Jaccard index are both overlap scores, and they are monotone in each other. From the two set formulas, D = 2J / (1 + J) and J = D / (2 - D). This is derived on this page from the definitions above; no card states it.

Variants

Soft Dice on probabilities

The basic form. Sum the product of prediction and target over pixels, and divide by the sum of both masks. The survey writes it without a smoothing term 1910.07655 §4.4.1. The review's exponential-logarithmic variant adds a smoothing constant to the numerator and the denominator 1910.07655 §4.4.3. The second survey gives the set form only, so it does not settle how to compute the score on probabilities 2307.02694 §6.1.2. Card: Deep Semantic Segmentation of Natural and Medical Images: A Review.

Dice with binary cross-entropy

The polyp segmenter trains on the sum of binary cross-entropy and Dice loss, with both weights set to 1 2108.00882 eq. 5 2108.00882 §3.4. The paper does not define the Dice formula. The code computes it per image on sigmoid probabilities, with a +1 term in the denominator only, and averages the per-image scores train.py 2108.00882 §3.4. Card: Shallow Attention Network for Polyp Segmentation.

Dice with cross-entropy, squared-denominator form

The hybrid transformer segments 9 classes and reports Dice and Hausdorff distance on Synapse 2207.08518 tab. 2. The paper does not state its training loss. The released code uses 0.4 times cross-entropy plus 0.6 times Dice trainer.py, and its Dice squares the predictions and targets in the denominator, with a 10^{-5} term utils.py. The background class is included in the average utils.py. Card: HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation.

Soft Jaccard with cross-entropy (sibling, not Dice)

The surgical-tool segmenter does not use Dice in its training loss. It trains on H - \log J, where H is the cross-entropy and J is a soft Jaccard index, both weighted 1 1803.01207 eq. 3 1803.01207 §2.3. The per-pixel Jaccard is \frac{1}{n}\sum_i y_i\hat{y}_i / (y_i + \hat{y}_i - y_i\hat{y}_i) 1803.01207 eq. 2. Its code uses (1-a)H - a\log J, takes one ratio of sums over the batch on sigmoid probabilities, and sets a to 0.5 by default loss.py loss.py. The paper reports Dice as a metric in its table, not as a loss 1803.01207 tab. 1. Card: Robot Instrument Segmentation with Cross-Entropy and Jaccard Losses.

Tversky-type overlap

The survey describes a generalisation with a parameter \beta that weights false negatives against false positives, and notes that \beta = 0.5 reduces to Dice 1910.07655 §4.4.2. The survey's plots use \beta = 0.3 1910.07655 §4.4. The card's formula is typeset with mismatched brackets, so do not implement it from the survey text alone 1910.07655 §4.4.2. A combined cross-entropy and Dice loss, with weights that control false positives and false negatives, is also named there 1910.07655 §5; its weights are not stated. The second survey also covers a Tversky loss in set form 2307.02694 §6.1.

When to use which

Implementation sketch

The common core is one minus a smoothed overlap ratio on probabilities. The polyp and hybrid cards differ only in the smoothing term and the denominator. The soft-Jaccard card uses a different denominator, \sum(y + \hat{y} - y\hat{y}), and is not the Dice form.

# Soft Dice loss, one class  [arxiv:1910.07655 §4.4.1]
# probs: predicted foreground probabilities (N,) in [0, 1]
# target: 0/1 ground-truth mask (N,)
# eps: smoothing; 0 reproduces the plain formula  [§4.4.1] [§4.4.3]

def soft_dice_loss(probs, target, eps=0.0):
    # Overlap: sum of prediction times target over pixels  [§4.4.1]
    intersection = sum(target * probs)
    # Denominator: the two masks' sums added together  [§4.4.1]
    total = sum(target) + sum(probs)
    # The loss is one minus the coefficient, not the coefficient  [§4.4.1]
    return 1 - (2 * intersection + eps) / (total + eps)

# Combined loss used in the polyp card  [arxiv:2108.00882 §3.4] [eq. 5]
# logits, mask: (H, W) per image; mask holds 0 or 1
# Smoothing of 1 in the denominator is the code's choice  [code: src/train.py#L66]

def bce_plus_dice(logits, mask, weight_bce=1.0, weight_dice=1.0):
    # Binary cross-entropy on the raw logits  [eq. 5]
    bce = binary_cross_entropy_with_logits(logits, mask)
    # Dice on the sigmoid probabilities  [code: src/train.py#L61-L66]
    probs = sigmoid(logits)
    dice = soft_dice_loss(probs, mask, eps=1.0)
    # Weighted sum; the cards use equal weights  [eq. 5]
    return weight_bce * bce + weight_dice * dice

Contradictions

None of the numeric results conflict, because no two cards report the same setting. The weights differ: the polyp card uses 1 to 1 [arxiv:2108.00882 §3.4], and the hybrid transformer's code uses 0.4 to 0.6 trainer.py. The hybrid transformer's paper does not state its loss, so its weights come from code alone trainer.py. The two smoothing choices also differ, a +1 in the denominator only train.py against a 10^{-5} term in a squared denominator utils.py. The soft-Jaccard card's own paper and code disagree on the weights: the paper weights cross-entropy and -\log J equally 1803.01207 eq. 3, while the code's default a = 0.5 and the released script's a = 0.3 give other weights loss.py train.sh. Its paper gives a threshold of 0.3, which the code does not use 1803.01207 §2.3. The survey's own worked example is the only evidence on small objects, and it is illustrative 1910.07655 §4.4.

Papers

• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning: cross-entropy plus a soft-Jaccard term for surgical tools; a sibling of Dice, with Dice reported only as a metric.
• Deep Semantic Segmentation of Natural and Medical Images: A Review: survey definition of the Dice coefficient and loss, the Tversky and exponential-logarithmic variants, and the small-object example.
• Shallow Attention Network for Polyp Segmentation: binary cross-entropy plus Dice with equal weights; the Dice formula is taken from the code.
• HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation: cross-entropy plus Dice in the code, with the paper not stating the loss.
• Loss Functions and Metrics in Deep Learning: set-form Dice and Tversky losses in the segmentation section, with the Dice coefficient given without a probability form.