---
kind: concept
slug: deep-supervision
title: Deep supervision
summary: >-
  Attaches losses to intermediate predictions as well as to the final output. A saliency paper puts losses on side outputs at three encoder depths and reports lower TORONTO scores without them. A 3D instance-segmentation paper applies a matched loss at every decoder layer. Two cards so far.
tags: [training-objective, multi-scale, dense-prediction, iterative-refinement]
papers: ["1705.02544", "2210.03105"]
related: [convolutional-neural-network, multi-scale-feature-fusion, salient-object-detection, residual-learning, object-queries, bipartite-matching, instance-segmentation]
status: draft
updated_at: 2026-10-24
---

## Overview

Deep supervision puts a loss on intermediate predictions, not only on the final
output. Each intermediate prediction is then trained to be usable on its own
[arxiv:1705.02544 §1] [arxiv:1705.02544 §2.2].

Two papers in this wiki use it in different ways. One attaches side outputs to
several encoder depths of a dense predictor, and all of them are scored against
the same ground-truth map [arxiv:1705.02544 §3.3.1] [arxiv:1705.02544 eq. 7].
The other keeps a prediction for every layer of a Transformer decoder and
matches each one to the ground truth [arxiv:2210.03105 §3] [arxiv:2210.03105 §3.1].

## Variants

### Side outputs at several encoder depths, plus a fused output

Three encoder layers (conv3-3, conv4-3 and conv5-3 of VGG16) each get their own
deconvolution decoder and a one-channel output. Each side output and the fused
output get a pixel-wise cross-entropy against the same ground-truth map
[arxiv:1705.02544 §3.3.1] [arxiv:1705.02544 §3.3.2] [arxiv:1705.02544 eq. 7]
[arxiv:1705.02544 eq. 8]. The total objective weights the side losses by 1/M
and adds the fused loss, where M is the number of side outputs [arxiv:1705.02544 eq. 9].
The paper tried more side layers and reports no gain and more computation
[arxiv:1705.02544 §3.3.1]. The paper also reports that the fused output is
what it uses at test time [arxiv:1705.02544 §3.3.3]. Card: [Deep Visual Attention Prediction](/papers/1705.02544).

### Fused loss only

The ablation baseline trains the same network with the loss on the fused output
alone, which the paper labels "without deep supervision" [arxiv:1705.02544 tab. 9].
On TORONTO it scores 0.71 s-AUC and 0.68 CC, against 0.76 and 0.72 for the full
model [arxiv:1705.02544 tab. 9] [arxiv:1705.02544 §4.4]. The table does not say
how the baseline's decoders are trained, so only the two scores are reported here.

### One matched loss per decoder layer

A Transformer decoder refines a fixed set of queries over several layers. The
paper attaches a mask and a class prediction to every layer, not only to the
last one. The decoder runs three passes with shared weights over four scales,
which gives L = 12 refinement steps [arxiv:2210.03105 §3]. At each layer the
predicted instances are paired with ground-truth instances by Hungarian
matching, and the mask and class losses are computed on those pairs
[arxiv:2210.03105 §3.1]. The total loss sums the per-layer losses over all L
layers [arxiv:2210.03105 eq. 6]. Card: [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Dense per-pixel prediction with a multi-scale encoder-decoder, and the same ground-truth map is available for each side | Side losses plus fused loss | The paper's only ablation of this kind shows a drop when the side losses are removed, on TORONTO [arxiv:1705.02544 tab. 9] |
| Labels exist only for the final output | Fused loss only | The side losses in that paper all use one ground-truth map [arxiv:1705.02544 eq. 7]. Without a target at each depth, there is no side loss to add |
| Choosing how many side outputs to attach | Start with the depths the paper picked, three for VGG16 | More side layers gave no gain and cost more computation [arxiv:1705.02544 §3.3.1] |
| Iterative refinement decoder that predicts a set of instances at each layer | One matched loss per layer | The paper trains every layer this way [arxiv:2210.03105 §3.1] [arxiv:2210.03105 eq. 6]. Its ablations cover the query design and the mask loss, not the per-layer sum [arxiv:2210.03105 §4.2] [arxiv:2210.03105 tab. 4] |
| Planning training cost for per-layer matching | Budget for it as part of the training run | One 2 cm run takes about 78 hours on one A40 [arxiv:2210.03105 §3.1]. The paper does not report the cost of the per-layer matching on its own |
| Application outside saliency prediction or 3D instance segmentation | Test it on your own data first | Each paper evaluates one task [arxiv:1705.02544 §4.4] [arxiv:2210.03105 §4.1] |

## Implementation sketch

Language-agnostic pseudocode of the common core: one loss per intermediate
prediction, weighted and summed. The papers differ in what each prediction is,
whether it is matched to the target first, and how it is weighted.

```pseudocode
# Deep supervision: one weighted loss per intermediate prediction
# Input:  preds (K predictions), gt (ground truth), weights (K numbers)
# Output: scalar loss
# [arxiv:1705.02544 eq. 9] [arxiv:2210.03105 eq. 6]

def deep_supervision_loss(preds, gt, weights, pair=None):
    total = 0.0
    for pred, w in zip(preds, weights):
        # Same target for every prediction, or pair them first when the
        # outputs are unordered sets [arxiv:2210.03105 §3.1]
        target = pair(pred, gt) if pair else gt
        # Saliency: pixel cross-entropy [arxiv:1705.02544 eq. 7]
        # Mask3D: 5 * BCE + 2 * Dice on matched pairs [arxiv:2210.03105 eq. 5]
        total += w * task_loss(pred, target)
    return total

# Weights in the two papers
# Side outputs: 1/M each, fused output 1 [arxiv:1705.02544 eq. 9]
# Mask3D: 1 for each layer, summed over all L layers [arxiv:2210.03105 eq. 6]
```

## Contradictions

Within one paper: the released saliency code gives every loss term a weight of
1, not 1/M as in eq. 9 of the paper [arxiv:1705.02544 eq. 9]. The paper does not
say which weighting produced its reported numbers, so an implementation that
follows the equation may not match its results.

Within one paper: the 3D paper writes the class term of its matching cost as
cross-entropy [arxiv:2210.03105 eq. 4]. Its released code uses the negative
softmax probability of the target class instead [code: models/matcher.py#L123].
The matching cost decides which target each layer is paired with, so an
implementation that follows the equation may pair layers differently from the code.

Between papers: no conflicting results. The two papers measure different things
on different tasks. The saliency paper's ablation is the only measurement of
removing deep supervision here, and it covers TORONTO only [arxiv:1705.02544 tab. 9].
The 3D paper reports no ablation of its per-layer losses [arxiv:2210.03105 §4.2].

## Papers

- [Deep Visual Attention Prediction](/papers/1705.02544): predicts eye fixations with side outputs on three VGG16 layers, each supervised, plus a learned fusion; the TORONTO ablation is the only measurement of removing the side losses here.
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105): predicts instance masks with a Transformer decoder and sums a Hungarian-matched mask and class loss over all 12 refinement steps; the paper does not ablate the per-layer losses.