PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

In plain words

Imagine an image classifier that has to learn new categories one at a time, long after its first training, and may keep only a handful of example photos (here 20) for each category it already knows. Retrain it naively on the new category and it quickly forgets the old ones. This is catastrophic forgetting, and it gets worse the more small learning rounds there are.

PODNet attacks this in two ways. First, while learning something new, the network is asked to keep its internal activity looking like the previous version of itself. It does not demand that every activation stay identical, which would leave no room to learn. Instead it compares summaries: for each layer it adds up the activations along each row and along each column of the feature map and asks only those row and column totals to stay similar. That leaves the network free to shift details while keeping the overall layout of what it sees.

Second, each class is represented not by one prototype vector but by ten. An image is scored against all ten prototypes of a class, and the closest ones count most. If the network's features drift a little over time, a class with several prototypes is less likely to lose track of its examples than a class with a single one.

On CIFAR100, ImageNet100 and ImageNet1000 the authors report higher average accuracy than the earlier methods iCaRL, BiC and UCIR, with the largest gap in the hardest setting: fifty rounds that each add a single class.

TL;DR

Class-incremental learning with a fixed per-class exemplar memory (M_per = 20). PODNet trains a ResNet with (1) POD, a distillation loss between the previous and the current model that, at the output of each ResNet stage, matches width-pooled and height-pooled activation statistics (POD-spatial) and, on the final embedding, matches the normalized vector (POD-flat) §3.1 eq. 4 eq. 5 eq. 6 eq. 7; and (2) LSC, a cosine classifier with K = 10 proxies per class trained with a modified NCA loss and initialized by k-means on new-class features §3.2 eq. 9 eq. 10 §6.3. On CIFAR100 with 50 steps of 1 class it reports 61.40 vs 49.30 (UCIR-CNN) average incremental accuracy; gains shrink as steps get larger tab. 1 tab. 2. The official code adds pieces the paper does not describe (classifier fine-tuning on a balanced memory, frozen old-class proxies, gradient clamping) and its NCA margin has no effect; see Paper vs code.

Problem

In class-incremental learning the model sees classes in tasks; at task t it must classify all classes seen so far, but for old classes only M_per stored samples per class are available, while new classes have all their data §3. The resulting imbalance causes forgetting of old classes. Prior distillation-based methods (iCaRL, BiC, UCIR) were evaluated on at most 10 incremental steps; with many small steps (up to 50 on CIFAR100) forgetting compounds and these methods collapse early in the run §2 §4.1 fig. 4. The paper adopts the stricter per-class memory protocol (M_per fixed), under which early tasks cannot use the full memory budget §2.

Method

Setup. Model is ŷ = g(f(x)), f a ConvNet feature extractor, h = f(x) the final embedding, g the classifier; superscript t marks the model after task t §3. Each task starts from the previous weights; the previous model is the frozen teacher §3.1. The ReLU at the last block of each ResNet stage is removed so that POD sees signed activations §4.

POD distillation. For the output h_ℓ of each ResNet stage (C × W × H), variants differ only in which axes are summed before an L2 distance of L2-normalized tensors §3.1 fig. 2:

\mathcal{L}_{\text{POD-pixel}}(\mathbf{h}^{t-1}_\ell, \mathbf{h}^t_\ell) = \sum_{c=1}^C \sum_{w=1}^{W} \sum_{h=1}^{H} \left\Vert \mathbf{h}^{t-1}_{\ell,c,w,h} - \mathbf{h}^t_{\ell,c,w,h} \right\Vert^2 eq. 1

\mathcal{L}_{\text{POD-channel}}(\mathbf{h}^{t-1}_\ell, \mathbf{h}^t_\ell) = \sum_{w=1}^{W} \sum_{h=1}^{H} \left\Vert \sum_{c=1}^C \mathbf{h}^{t-1}_{\ell,c,w,h} - \sum_{c=1}^C \mathbf{h}^{t}_{\ell,c,w,h} \right\Vert^2 eq. 2

\mathcal{L}_{\text{POD-gap}}(\mathbf{h}^{t-1}_\ell, \mathbf{h}^t_\ell) = \sum_{c=1}^{C} \left\Vert \sum_{w=1}^{W} \sum_{h=1}^H \mathbf{h}^{t-1}_{\ell,c,w,h} - \sum_{w=1}^{W} \sum_{h=1}^H \mathbf{h}^{t}_{\ell,c,w,h} \right\Vert^2 eq. 3

\mathcal{L}_{\text{POD-width}}(\mathbf{h}^{t-1}_\ell, \mathbf{h}^t_\ell) = \sum_{c=1}^{C} \sum_{h=1}^{H} \left\Vert \sum_{w=1}^W \mathbf{h}^{t-1}_{\ell,c,w,h} - \sum_{w=1}^W \mathbf{h}^{t}_{\ell,c,w,h} \right\Vert^2 eq. 4

POD-height is the same with the roles of w and h swapped (not numbered in the paper) §3.1. The chosen variant combines them:

\mathcal{L}_{\text{POD-spatial}} = \mathcal{L}_{\text{POD-width}} + \mathcal{L}_{\text{POD-height}} eq. 5

On the flat final embedding:

\mathcal{L}_{\text{POD-flat}}(\mathbf{h}^{t-1}, \mathbf{h}^t) = \left\Vert \mathbf{h}^{t-1} - \mathbf{h}^t \right\Vert^2 eq. 6

Combined over the L−1 intermediate stages:

\mathcal{L}_\text{POD-final}(\mathbf{x}) = \frac{\lambda_{c}}{L-1}\sum_{\ell=1}^{L-1} \mathcal{L}_{\text{POD-spatial}}\left(f^{t-1}_\ell(\mathbf{x}), f^t_\ell(\mathbf{x})\right) + \lambda_{f} \mathcal{L}_\text{POD-flat}\left(f^{t-1}(\mathbf{x}), f^t(\mathbf{x})\right) eq. 7

Implementation details from the supplement: activations are squared element-wise before sum-pooling, the pooled vector is L2-normalized, and both POD terms are multiplied by the UCIR adaptive factor λ = sqrt(N/T), N = number of seen classes, T = number of classes in the current task §6.2. With λ_c = 0 the loss reduces to UCIR's Less-Forget constraint §3.1.

Local Similarity Classifier (LSC). UCIR's cosine classifier is

\hat{\mathbf{y}}_{c}=\frac{\exp\left(\eta\langle\boldsymbol{\theta}_{c},\mathbf{h}\rangle\right)}{\sum_{i} \exp \left(\eta\langle\boldsymbol{\theta}_{i}, \mathbf{h}\rangle\right)} eq. 8

with ⟨·,·⟩ cosine similarity and η a learned scale §3.2. LSC keeps K proxies θ_{c,k} per class and soft-attends over them:

s_{c,k} =\frac{\exp\,\langle\boldsymbol{\theta}_{c,k},\mathbf{h}\rangle}{\sum_{i} \exp\,\langle\boldsymbol{\theta}_{c,i},\mathbf{h}\rangle}, \qquad \hat{\mathbf{y}}_c = \sum_{k}s_{c,k}\,\langle\boldsymbol{\theta}_{c,k},\mathbf{h}\rangle eq. 9

trained with an NCA loss plus hinge and margin δ:

\mathcal{L}_\text{LSC} = \left[- \log\frac{\exp\left(\eta (\hat{\mathbf{y}}_y - \delta)\right)}{\sum_{i \neq y} \exp \eta \hat{\mathbf{y}}_{i}} \right]_+ eq. 10

The paper reports NCA converged faster than plain cross-entropy §3.2. Total loss per task: L = L_LSC + L_POD-final §3.3.

New-class proxy init. For each new class, extract features of its training samples with the current model, run k-means with K clusters and use the centroids as θ_{c,k} (multi-mode Imprinted Weights); reported to add 1 p.p. on CIFAR100 §3.2.

Memory and inference. Exemplars are chosen by iCaRL herding; inference is either NME (nearest mean of exemplars) or "CNN" (classifier output) §4.

# PODNet training for one incremental task t  [§3] [§6.2]
# Inputs: new-class data, exemplar memory (M_per per old class),
#         teacher = frozen copy of the model after task t-1

def start_task(model, new_classes, data):
    for c in new_classes:
        # Embed the class's training images with the current model  [§3.2]
        feats = l2_normalize(model.f(data[c]))
        # K centroids become the class's K proxies  [§3.2]
        model.g.proxies[c] = kmeans(feats, K).centroids

def lsc_logits(proxies, embedding):
    # Cosine similarity to every proxy of every class  [eq. 9]
    sims = cosine(embedding, proxies)            # (B, n_classes, K)
    # Soft attention over each class's proxies  [eq. 9]
    attn = softmax(sims, axis=-1)
    return sum(attn * sims, axis=-1)             # (B, n_classes)

def pod_spatial(old_map, new_map):
    # Square activations element-wise before pooling  [§6.2]
    old_sq = old_map ** 2                        # (B, C, W, H)
    new_sq = new_map ** 2
    # Sum over width, and separately over height  [eq. 4] [eq. 5]
    old_vec = concat(flatten(sum(old_sq, axis=W)),
                     flatten(sum(old_sq, axis=H)))
    new_vec = concat(flatten(sum(new_sq, axis=W)),
                     flatten(sum(new_sq, axis=H)))
    # L2-normalize pooled vectors, then distance  [§3.1] [§6.2]
    diff = l2_normalize(old_vec) - l2_normalize(new_vec)
    return mean_over_batch(norm(diff))

def training_step(model, teacher, images, labels, seen, task_size):
    stage_maps, embedding = model.f(images)
    # Teacher outputs carry no gradient  [§3.1]
    old_maps, old_embedding = no_grad(teacher.f(images))
    logits = lsc_logits(model.g.proxies, embedding)
    # NCA with margin and hinge  [eq. 10]
    loss = lsc_nca_loss(logits, labels, eta, delta)
    # UCIR adaptive weight sqrt(N / T)  [§6.2]
    adapt = sqrt(seen / task_size)
    # Average POD-spatial over stage outputs  [eq. 7]
    spatial = mean(
        pod_spatial(old_map, new_map)
        for old_map, new_map in zip(old_maps, stage_maps)
    )
    # Distance of normalized final embeddings  [eq. 6]
    flat = squared_norm(l2_normalize(old_embedding)
                        - l2_normalize(embedding))
    loss = loss + adapt * (lambda_c * spatial + lambda_f * flat)
    return loss

Hyperparameters

Results

Author-reported average incremental accuracy (mean accuracy over all classes seen so far, averaged over every step including the first), M_per = 20, half of the classes in the first task. CIFAR100: mean ± std over three class orders; ImageNet100 ± values for PODNet are given without stating how many runs. Rows with * are copied from the UCIR paper; other baselines were rerun by the authors (BiC with their own implementation) §4 tab. 1 tab. 2.

CIFAR100 tab. 1

ImageNet100 and ImageNet1000 (PODNet with CNN inference only) tab. 2

Ablation, CIFAR100, 50 steps tab. 3

Pooling variants, CIFAR100, 50 steps (with POD-flat tab. 3; without POD-flat tab. 6)

Memory size per class, CIFAR100, 50 steps tab. 4

Other memory protocol and initial task size, CIFAR100 tab. 5

The "init" columns use M_per = 20 and single-class steps after the initial task tab. 5. Going from K = 1 to K = 10 proxies added 1.18 p.p. on CIFAR100 and 1.51 p.p. on ImageNet100 (10 steps) §6.3. Adding POD-spatial on top of POD-flat raised oldest-class accuracy by about 7 p.p. while newest-class accuracy dropped by 0.2 p.p.; POD-pixel instead cost 2 p.p. on new classes for +5 p.p. on old ones §4.2.

Paper vs code

• NCA margin has no effect in code. Eq. 10 subtracts δ from the target logit only eq. 10. The code builds a per-target margins tensor but never uses it and subtracts the scalar margin from every logit, which then cancels in the max-shift and the ratio base.py. Configs set margin 0.6 podnet_cnn_cifar100.yaml.
• No hinge in code. Paper clamps the loss at zero eq. 10; hinge_proxynca defaults to False and no config sets it base.py.
• Denominator. The paper sums over i ≠ y eq. 10. The code zeroes the target entry of the max-shifted logits instead of removing it, so a constant exp(0) = 1 term stays in the denominator base.py.
• Similarity. Paper uses cosine similarity eq. 9. Code uses the negative squared Euclidean distance of L2-normalized vectors (−‖a−b‖² = 2·cos − 2 at scaling 1), so the proxy softmax effectively has temperature 1/2 relative to eq. 9 classifiers.py classifiers.py distance.py.
• POD-spatial distance. Paper sums squared differences eq. 4. Code concatenates the height- and width-pooled vectors, L2-normalizes the concatenation as one vector, takes the (non-squared) L2 norm of the difference, averages over the batch and then over the stage maps distillation.py.
• Layers used. Code distills four maps: the outputs of stages 1–3 and of the final block before pooling my_resnet.py.
• POD-flat is cosine_embedding_loss = mean(1 − cos), i.e. half the squared distance of normalized embeddings in eq. 6, applied to the pre-ReLU pooled features base.py podnet.py.
• Classifier fine-tuning stage (CNN configs). After each incremental task the code trains only the classifier for 20 more epochs on a class-balanced (undersampled) memory set at lr 0.05 (CIFAR) / 0.01 (ImageNet) podnet.py podnet_cnn_cifar100.yaml. The paper describes this only for UCIR-CNN §2.
• Frozen old proxies (CNN configs). groupwise_factors: old_weights: 0 drops old-class proxies from the optimizer during main training podnet_cnn_cifar100.yaml podnet.py. Not mentioned in the paper.
• Gradient clamping to [-5, 5] per element, not in the paper podnet.py.
• Proxy init clusters L2-normalized features with k-means and rescales the centroids by the mean norm of existing weights classifiers.py; the paper only says centroids §3.2.
• ImageNet hyperparameters in configs differ from §6.2: lr 0.05 vs 0.1, batch 64 vs 128, and weight decay 5e-4 for ImageNet100 vs 1e-4 §6.2 podnet_cnn_imagenet100.yaml. The README runs ImageNet1000 with the ImageNet100 config plus --memory-size 20000 README.md.
• λ_c for NME: CIFAR100 NME config uses 5, paper says 3 §6.2 podnet_nme_cifar100.yaml.

Pitfalls

• Reproducing the reported CNN numbers likely needs the undocumented classifier fine-tuning, frozen old proxies and gradient clamp from the code; an implementation from the paper alone omits all three.
• The margin δ is unspecified in the paper and inert in the code. If you implement eq. 10 literally, δ becomes a real hyperparameter that the authors never tuned in effect.
• "ℓ²-normalized tensors" in §3.1 is ambiguous for POD-spatial: normalize per pooled axis, per channel, or the whole concatenation? The code normalizes the concatenated width+height vector per sample.
• Do not skip squaring activations before pooling §6.2. Our reading, not tested in the paper: since the stage ReLUs are removed, unsquared positive and negative activations could cancel in the sums.
• Removing the final ReLU of each stage changes the backbone; the classifier and POD-flat then see signed pooled features §4.
• The adaptive factor sqrt(N/T) grows over a run; with 1-class steps on CIFAR100 it reaches sqrt(100) = 10, so distillation dominates late tasks.
• Every step needs a teacher forward pass, and new-class init runs a feature pass plus k-means per new class.
• NME inference worked better on CIFAR100 but the authors report it did not generalize to ImageNet, so ImageNet results use CNN inference only §4.1.
• ImageNet results have no reported spread over class orders in the text (tab. 2 lists ± values for PODNet on ImageNet100 only), and ImageNet100 is a random 100-class subset whose classes the paper does not list §4 tab. 2. The repository has an imagenet_split/ directory; we did not verify it matches the paper's subset.
• Ablations in tab. 3 show LSC with POD-spatial alone (61.42 NME) matches the full model (61.40), so POD-flat adds little once POD-spatial and LSC are present.

Lineage

• Builds on: UCIR (Hou et al. 2019: cosine classifier, Less-Forget distillation, adaptive λ, protocol) §2 §6.2; iCaRL (Rebuffi et al.: herding, NME, average incremental accuracy) §2; Imprinted Weights, extended to K proxies via k-means §3.2; NCA / ProxyNCA losses §3.2.
• Related distillation losses compared: attention transfer (channel pooling), perceptual/style losses, GradCam distillation (Dhar et al.) §3.1 §4.2.
• Compared against: iCaRL, BiC, UCIR tab. 1 tab. 2.