PLOP: Learning without Forgetting for Continual Semantic Segmentation

In plain words

Imagine a model that can label pixels in photos, trained on a first set of object types, and later asked to also recognise new types. The obvious approach, training on the new labels alone, makes the model forget the old types. The old types also get a second problem: in the new training photos, every pixel that is not one of the new types is simply labelled "background", so a dog or a chair that the model used to know looks like background.

PLOP tackles both problems using only the old model and the new photos. First, it keeps the new model's internal feature maps close to the old model's, but not just globally: it compares averages over strips and small grid cells, so both large-scale layout and small details are preserved. Second, it asks the old model what it thinks each background pixel is. If the old model is confident, that guess replaces the "background" label; if it is unsure, the pixel is ignored in the loss. The two pieces are added with a weighting factor.

TL;DR

For continual semantic segmentation, where classes arrive in steps and old images are not stored, PLOP combines a multi-scale feature distillation (Local POD) with entropy-thresholded pseudo-labels for background pixels. The authors report large gains over MiB and ILT on Pascal VOC 2012 and ADE20k, including longer 6-to-11-step settings. All numbers are author-reported. The code is MIT licensed and matches the paper on most hyperparameters, with exceptions listed under Paper vs code.

Problem

• Continual semantic segmentation (CSS) learns new classes in steps. At each step the training labels cover only the current classes; old and future classes are collapsed into background, and the model must still predict all classes seen so far §3.1.
• Catastrophic forgetting: the network drops old classes when it learns new ones §1, §3.1.
• Background shift: at step t, background pixels may contain old classes (including the true background) or future classes, which makes the old classes look like background and worsens forgetting §1, §3.1.
• Distillation limits: POD, a feature distillation that matches pooled statistics, models long-range structure. For segmentation it blurs small objects, so a scheme that also keeps short-range relations is needed §3.2.
• Setting: the authors do not use rehearsal (no stored old images), and at test time no task id is given, so the model picks among all seen classes §4.1, suppl. Implementation details.

Method

The paper's Overlapped protocol is the focus: pixels of a step's images may belong to old, current or future classes §4.1. The method uses only the current step's images plus the frozen old model.

1. Setup. At step t the model is f^t followed by a classifier g^t and has to predict all classes seen so far. The old model is the network from step t-1, which provides the distillation targets and the pseudo-labels §3.1 §3.2 §3.3.
2. Local POD embedding. For a feature map of size H × W × C, POD takes the width-averaged and height-averaged slices and concatenates them, giving a vector of size (H + W) × C §3.2 eq. 1. Local POD applies this to a grid of regions at several scales, then concatenates the scales §3.2 eq. 3 eq. 4. The complete procedure is in alg. 1.
3. Distillation loss. The embeddings of the old and new model are compared on several layers, and the squared distance is averaged over layers §3.2 eq. 5. The paper does not list which layers §3.2.
4. Entropy-based uncertainty. The uncertainty u of a pixel is the entropy of the old model's class distribution §3.3 eq. 7. Before step t, the paper computes per class c a threshold τ_c as the median of u over old-model predictions of class c on the training pixels §3.3.
5. Pseudo-labels. Foreground ground-truth labels are kept. A background pixel receives the old model's argmax class only if its u is below τ_c for that class. Otherwise it gets no target §3.3 eq. 6 eq. 7.
6. Classification loss. Cross-entropy over the targets, averaged over the W × H pixels and weighted by ν, the share of background pixels that received a pseudo-label §3.3 eq. 8.
7. Total loss. Classification loss plus λ times the Local POD loss §3.3 eq. 9. λ is not given a single value in the text; the values are in Hyperparameters.

Pseudocode, in the form the card uses throughout. Names are descriptive and the functions are not from the paper.

# Local POD embedding of one feature map  [alg. 1] [eq. 1] [eq. 3-4]
# Input:  feats (B, C, H, W); scales, e.g. (1, 2, 4) grid divisions
# Output: embedding (B, D)

def local_pod_embedding(feats, scales):
    # Collect the pooled slices of every scale  [eq. 3-4]
    pieces = []
    for grid in scales:
        # Cut the map into grid x grid regions  [§3.2]
        for region in regions(feats, grid):
            # Average over width: one value per row and channel  [eq. 1]
            pieces.append(region.mean(over="width"))
            # Average over height: one value per column and channel  [eq. 1]
            pieces.append(region.mean(over="height"))
    # Concatenate everything along the channel axis  [eq. 4]
    return concatenate(pieces, axis="channels")

# Local POD distillation loss over the tapped layers  [eq. 2] [eq. 5]
# Input:  image; old and new networks; scales
# Output: scalar loss

def local_pod_loss(image, old_net, new_net, scales):
    # Feature maps of the same layers in both networks  [§3.2]
    old_maps = old_net.tapped_layers(image)
    new_maps = new_net.tapped_layers(image)
    total = 0
    for old_map, new_map in zip(old_maps, new_maps):
        # Square the activations before pooling  [suppl. Implementation details]
        old_emb = local_pod_embedding(square(old_map), scales)
        new_emb = local_pod_embedding(square(new_map), scales)
        # Squared L2 distance between the two embeddings  [eq. 5]
        total = total + squared_norm(new_emb - old_emb)
    # Average over the layers  [eq. 2] [eq. 5]
    return total / len(old_maps)

# Pseudo-labels for background pixels  [§3.3] [eq. 6] [eq. 7]
# Input:  gt (H, W), 0 = background; old_probs (H, W, C_old)
#         thresholds (C_old,), one per old class
# Output: targets (H, W), -1 = pixel ignored in the loss

def pseudo_targets(gt, old_probs, thresholds):
    # Old model's best class per pixel  [eq. 7]
    guess = argmax(old_probs, axis="classes")
    # Uncertainty u = entropy of the old distribution  [§3.3] [eq. 7]
    uncertainty = entropy(old_probs)
    # Keep a guess only if u is below the threshold of its class  [eq. 7]
    confident = uncertainty < thresholds[guess]
    # Foreground pixels keep their ground-truth label  [eq. 6]
    targets = gt.copy()
    background = gt == 0
    # Confident background pixels take the old model's class  [eq. 7]
    targets[background & confident] = guess[background & confident]
    # Uncertain background pixels get no target  [eq. 7]
    targets[background & ~confident] = -1
    return targets

# Loss of one training step  [eq. 8] [eq. 9]
# Input:  image, gt; old and new networks; thresholds; lam
# Output: scalar loss

def step_loss(image, gt, old_net, new_net, thresholds, lam):
    # Old model's class probabilities, no gradient  [§3.3]
    old_probs = softmax(old_net(image))
    targets = pseudo_targets(gt, old_probs, thresholds)
    # Share of background pixels that kept a pseudo-label  [eq. 8, nu]
    nu = kept_background(targets, gt) / count_background(gt)
    new_logits = new_net(image)
    # Cross-entropy over the targets, weighted by nu  [eq. 8]
    classif = nu * cross_entropy(new_logits, targets, ignore=-1)
    # Add the Local POD term with weight lam  [eq. 9]
    distill = local_pod_loss(image, old_net, new_net, scales=(1, 2, 4))
    return classif + lam * distill

Note: the pseudocode uses one plausible reading of the paper; see the Pitfalls and Paper vs code sections for the details the paper leaves open.

Key equations, as written in the paper:

\Phi(x) = \left[\frac{1}{W}\sum_{w} x[:,w,:] \,\Big\Vert\, \frac{1}{H}\sum_{h} x[h,:,:]\right] \in \mathbb{R}^{(H+W)\times C} eq. 1

\mathcal{L}_\text{pod} = \frac{1}{L}\sum_{l=1}^{L} \lVert \Phi(f^t_l(I)) - \Phi(f^{t-1}_l(I)) \rVert^2 eq. 2

\Psi^s(x) = [\Phi(x^s_{0,0}) \,\|\, \dots \,\|\, \Phi(x^s_{s-1,s-1})] eq. 3

\Psi(x) = [\Psi^1(x) \,\|\, \dots \,\|\, \Psi^S(x)] \in \mathbb{R}^{(H+W)\times C\times S} eq. 4

\mathcal{L}_\text{LocalPod} = \frac{1}{L}\sum_{l=1}^{L} \lVert \Psi(f^t_l(I)) - \Psi(f^{t-1}_l(I)) \rVert^2 eq. 5

\mathcal{L}_\text{pseudo} = -\frac{\nu}{WH}\sum_{w,h}\sum_{c\in\mathcal{C}^t} \tilde{S}(w,h,c)\log\hat{S}^t(w,h,c) eq. 8

\mathcal{L} = \mathcal{L}_\text{pseudo} + \lambda\,\mathcal{L}_\text{LocalPod} eq. 9

Eq. 6 and Eq. 7 define the pseudo-label target \tilde{S} with and without the uncertainty test, and are written out in the pseudocode above eq. 6 eq. 7.

Hyperparameters

Values in the table are the paper's unless the Notes say otherwise. Paper section and supplement names are cited as in the source; the supplement has no letter labels, so cite it by its heading.

Results

Author-reported. Setup: Pascal VOC 2012 (20 classes), ADE20k (150 classes), Cityscapes (19 classes) under the Overlapped protocol, DeepLab-V3 with ResNet-101, no rehearsal, metric mIoU §4.1 §4.2. Columns: the initial classes (0-x), the new classes, and all classes after the last step; avg is the mean over steps §4.1. A dagger (†) marks numbers excerpted from the MiB paper (Cermelli et al.); the rest are the authors' re-implementations tab. voc_sota_full. A dash (–) is an empty cell in the source. Table numbers are not given in the source, so the tables are cited by their LaTeX label.

Joint training (upper bound, not continual): 77.40 mIoU on VOC and 38.90 on ADE20k §4.2.

Pascal VOC 2012, Overlapped tab. voc_sota_full

The main-text table tab. voc_sota is the subset with ILT, MiB and PLOP; its rows match these.

ADE20k, Overlapped tab. ade_sota

Longer and harder settings (VOC 10-1 = 11 tasks; ADE 100-5 = 11 tasks) tab. voc_hard tab. ade_hard

Cityscapes domain shift (final mIoU, all classes known from the start) tab. cityscapes_domain

Ablation, VOC 15-5 and 15-1 (validation split of 20% of train, final "all" and "avg") tab. ablations

Distillation and classification losses, VOC 15-1 (validation split; columns 0-15, 16-20, all, avg) tab. ablation_classif_distillation

VOC Disjoint protocol (supplement) tab. voc_disjoint_sota

Summary statements in the text, all author-reported §4.2: on VOC 15-1, MiB is outperformed by +86% on all classes, +90% on old and +56% on new classes; on VOC 15-5, PLOP is on par with the authors' MiB re-implementation (70.09 vs 70.08 all). On ADE 100-10, the gain over MiB is 2.35 points on all classes.

Figures, described in words (no images are used on this page): fig. 1 shows the two-part strategy. fig. 2 shows background shift in ground truth, pseudo- labels and predictions. fig. 3 shows Local POD for one example with two scales. fig. 4 plots mIoU per step on VOC 15-1, where MiB and ILT drop after a few steps and PLOP stays high. fig. 5 gives boxplots over 20 random class orders on VOC 15-1: PLOP is better and less variable than MiB. fig. 6 and fig. 7 show predictions over time for MiB and PLOP. Figure numbers are assigned by order of appearance in the source.

Paper vs code

Code: official repository arthurdouillard/CVPR2021_PLOP (MIT), commit 4b4a0605b2afb84f988d246dee9a00e0c7517363. It is named in the paper's supplement [suppl. Hardware and Code] and is owned by the first author. The code is a fork of the MiB code base [suppl. Hardware and Code].

Checked and agreeing: the squared activations before pooling [suppl. Implementation details] train.py; the λ values for the standard PLOP preset (1e-2 features, 5e-4 logits) argparser.py; the epoch counts and first-step learning rate in the VOC scripts plop_15-1.sh; the three-scale Local POD train.py train.py; the adaptive POD weight sqrt(|C^{1:t}| / |C^t|) train.py; SGD with momentum 0.9 and Nesterov run.py.

Discrepancies (paper first, then code):

• Distance: the paper's loss uses the squared L2 norm eq. 2 eq. 5. The code takes the unsquared Frobenius norm per embedding and averages over the batch train.py train.py. The two give different gradients. Use the paper's form unless matching the code's numbers is the goal.
• Threshold floor: the supplement calls τ = 1e-3 "a maximum threshold" [suppl. Implementation details]. The code applies it as a lower bound: thresholds are max(median, τ) train.py.
• Median population: the paper computes the per-class median over all training pixels that the old model predicts as class c §3.3. The code's histogram only uses pixels whose ground-truth label is background (labels == 0) train.py.
• Entropy scale: the paper does not normalise u. The code's entropy helper divides by log(C) and averages over classes train.py, and the pseudo-label test then divides by log(C) again train.py train.py. The thresholds are in that same scale.
• Learning-rate decay: the paper says exponential with rate 0.9 [suppl. Implementation details]. The default policy is poly with power 0.9 argparser.py run.py, and the PLOP scripts do not override it.
• Batch size: the paper says 24 on two GPUs [suppl. Implementation details]. The VOC and ADE scripts pass 12 and launch two processes plop_15-1.sh plop_ade_100-10.sh. This fits 24 total only if the flag is per process; not verified.
• ADE overrides: the paper uses the same λ values for all datasets. The ADE 100-10 script uses λ = 1e-3 for features in step 0 and 1e-5 for logits in steps 1 and later plop_ade_100-10.sh. The VOC script uses the defaults argparser.py.
• Grid indexing: Eq. 3 indexes an s × s grid of cells of size H/s, while Algorithm 1 uses 2^s cells per side at scale 1/2^s alg. 1. The code uses scale cells per side for scales 1, 2 and 4 train.py. Code and algorithm agree; the printed Eq. 3 does not.
• Layers: the paper does not name the distilled layers §3.2. The code uses the output of four ResNet stages, the pre-logits head output, and the logits, where the logits are the small map before upsampling resnet.py segmentation_module.py train.py. The logit channels for new classes are summed into the background channel before comparison train.py.
• Classifier init: the paper does not say how new-class weights start. The code copies the background weight to the new classes and sets the bias to the background bias minus log(N + 1) run.py segmentation_module.py.
• Before ReLU: the supplement says Local POD is applied before ReLU [suppl. Implementation details]. Not verified in the code.
• Tiling assumption: the code slices using the width for both axes train.py, so it assumes square maps. The paper does not mention this.

Other paper statements not checked against code: the uncertainty threshold values per class, the use of "pre-ReLU" features, and the ablation split.

Pitfalls

• Eq. 8 normalisation: the text says the normalisation WH is reduced for discarded pixels, while Eq. 8 and the code divide by all W × H pixels §3.3 eq. 8. Implement Eq. 8 and check the scale of the loss before trusting the numbers.
• ν definition: the paper defines ν over "the total number of such pixels" (old or background pixels) §3.3. The code's ν uses background pixels only train.py. These are the same when the target set is background, but state the choice when implementing.
• Entropy scale: the threshold, the floor and the normalisation are all in code units that the paper does not define (see Paper vs code). An implementer who uses the paper's raw entropy will get different pseudo-label rates.
• Median by histogram: the code approximates the median with 100 bins over the normalised entropy train.py. Exact medians will differ slightly; the paper does not state a bin count.
• Cost: each step needs a frozen old model forward pass on every image, plus a full pass over the training set to fit the thresholds before the step train.py. The paper also stores feature maps of both models for the distillation §3.2. Memory and time grow with image size and with the number of distilled layers. Training time is author-reported.
• Ignored pixels: pixels with no target are ignored by the classification loss, but they still count in the normalisation (see Eq. 8 note above). With few confident pixels the classifier focuses on new classes, which is what ν limits [suppl. Additional Experiments].
• Table notes: a TeX comment in the ADE table source (not visible in the rendered paper) says the ADE 50-50 run did not use the adaptive factor tab. ade_sota. Check this before relying on that column.
• Protocol choice: the headline numbers use the Overlapped protocol, the realistic and harder one §4.1. The Disjoint numbers are in the supplement; do not mix them.
• Domain-shift baseline: the paper says MiB reduces to cross-entropy plus basic distillation in the Cityscapes domain-shift setting [§4.2 Domain Shift paragraph]. The gains there therefore compare against that baseline, and do not isolate the pseudo-label part.
• Ordering effects: the main results use one fixed class order from the baseline paper; the random-order study is a single 20-permutation sample on VOC 15-1 §4.2 [suppl. Classes ordering details].
• Sensitivity: the ablations use a 20% validation split and are close to the main runs but not identical [suppl.]. Their numbers are not directly comparable to the main tables.

Lineage

• Builds on: POD distillation from PODNet, extended with multi-scale local pooling (Local POD) §3.2.
• Builds on: the MiB code base and baseline protocol (Cermelli et al., "Modeling the Background for Incremental Learning in Semantic Segmentation", CVPR 2020). Not compiled here, so no link is given. The MiB numbers are the main comparison §4.1.
• Uses: entropy-median thresholds as in Saporta et al. (ESL) §3.3; adaptive distillation weight as in Hou et al. (UCIR) [suppl. Implementation details]. Neither has a card.
• Compared against: ILT, LwF-MC, EWC, PI, RW, LwF and fine-tuning in the supplement table tab. voc_sota_full.
• Superseded by: not stated in the wiki.