Deep supervision

Overview

Deep supervision puts a loss on intermediate predictions, not only on the final output. Each intermediate prediction is then trained to be usable on its own 1705.02544 §1 1705.02544 §2.2.

Two papers in this wiki use it in different ways. One attaches side outputs to several encoder depths of a dense predictor, and all of them are scored against the same ground-truth map 1705.02544 §3.3.1 1705.02544 eq. 7. The other keeps a prediction for every layer of a Transformer decoder and matches each one to the ground truth 2210.03105 §3 2210.03105 §3.1.

Variants

Side outputs at several encoder depths, plus a fused output

Three encoder layers (conv3-3, conv4-3 and conv5-3 of VGG16) each get their own deconvolution decoder and a one-channel output. Each side output and the fused output get a pixel-wise cross-entropy against the same ground-truth map 1705.02544 §3.3.1 1705.02544 §3.3.2 1705.02544 eq. 7 1705.02544 eq. 8. The total objective weights the side losses by 1/M and adds the fused loss, where M is the number of side outputs 1705.02544 eq. 9. The paper tried more side layers and reports no gain and more computation 1705.02544 §3.3.1. The paper also reports that the fused output is what it uses at test time 1705.02544 §3.3.3. Card: Deep Visual Attention Prediction.

Fused loss only

The ablation baseline trains the same network with the loss on the fused output alone, which the paper labels "without deep supervision" 1705.02544 tab. 9. On TORONTO it scores 0.71 s-AUC and 0.68 CC, against 0.76 and 0.72 for the full model 1705.02544 tab. 9 1705.02544 §4.4. The table does not say how the baseline's decoders are trained, so only the two scores are reported here.

One matched loss per decoder layer

A Transformer decoder refines a fixed set of queries over several layers. The paper attaches a mask and a class prediction to every layer, not only to the last one. The decoder runs three passes with shared weights over four scales, which gives L = 12 refinement steps 2210.03105 §3. At each layer the predicted instances are paired with ground-truth instances by Hungarian matching, and the mask and class losses are computed on those pairs 2210.03105 §3.1. The total loss sums the per-layer losses over all L layers 2210.03105 eq. 6. Card: Mask3D: Mask Transformer for 3D Semantic Instance Segmentation.

When to use which

Implementation sketch

Language-agnostic pseudocode of the common core: one loss per intermediate prediction, weighted and summed. The papers differ in what each prediction is, whether it is matched to the target first, and how it is weighted.

# Deep supervision: one weighted loss per intermediate prediction
# Input:  preds (K predictions), gt (ground truth), weights (K numbers)
# Output: scalar loss
# [arxiv:1705.02544 eq. 9] [arxiv:2210.03105 eq. 6]

def deep_supervision_loss(preds, gt, weights, pair=None):
    total = 0.0
    for pred, w in zip(preds, weights):
        # Same target for every prediction, or pair them first when the
        # outputs are unordered sets [arxiv:2210.03105 §3.1]
        target = pair(pred, gt) if pair else gt
        # Saliency: pixel cross-entropy [arxiv:1705.02544 eq. 7]
        # Mask3D: 5 * BCE + 2 * Dice on matched pairs [arxiv:2210.03105 eq. 5]
        total += w * task_loss(pred, target)
    return total

# Weights in the two papers
# Side outputs: 1/M each, fused output 1 [arxiv:1705.02544 eq. 9]
# Mask3D: 1 for each layer, summed over all L layers [arxiv:2210.03105 eq. 6]

Contradictions

Within one paper: the released saliency code gives every loss term a weight of 1, not 1/M as in eq. 9 of the paper 1705.02544 eq. 9. The paper does not say which weighting produced its reported numbers, so an implementation that follows the equation may not match its results.

Within one paper: the 3D paper writes the class term of its matching cost as cross-entropy 2210.03105 eq. 4. Its released code uses the negative softmax probability of the target class instead matcher.py. The matching cost decides which target each layer is paired with, so an implementation that follows the equation may pair layers differently from the code.

Between papers: no conflicting results. The two papers measure different things on different tasks. The saliency paper's ablation is the only measurement of removing deep supervision here, and it covers TORONTO only 1705.02544 tab. 9. The 3D paper reports no ablation of its per-layer losses 2210.03105 §4.2.

Papers

• Deep Visual Attention Prediction: predicts eye fixations with side outputs on three VGG16 layers, each supervised, plus a learned fusion; the TORONTO ablation is the only measurement of removing the side losses here.
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation: predicts instance masks with a Transformer decoder and sums a Hungarian-matched mask and class loss over all 12 refinement steps; the paper does not ablate the per-layer losses.