DER: Dynamically Expandable Representation for Class Incremental Learning

In plain words

A class-incremental learner has to pick up new categories over time, and it may keep only a small set of old example images. The usual failure is that training on new categories makes the model forget the old ones.

DER handles this by growing the network instead of only retraining it. At each new round it freezes everything it has already learned and adds a fresh block of feature detectors trained on the new categories. The final decision uses the outputs of all blocks side by side, so the old knowledge is still present and unchanged. A helper output is trained to tell new categories from old ones, which pushes the new block toward features that are actually different from the old ones. The authors also describe a way to switch off unneeded parts of the new block so the network does not grow too quickly.

Because the old blocks are frozen, the model can grow with every round, so it gets bigger over time. At the end of each round the last layer is retrained on a balanced set of examples, so the model is not biased toward the newest categories. The authors report that this beats earlier methods on standard image benchmarks.

TL;DR

DER is a class-incremental image classifier with a rehearsal memory §3.1. Each step freezes all earlier extractors and adds a new one, initialized from the previous extractor, and feeds the concatenated features to a single linear classifier §3.2 eq. 1 eq. 2. The new extractor is trained with cross-entropy plus an auxiliary new-vs-old loss eq. 3 eq. 4, then the classifier is retrained on a class-balanced subset §3.4. The paper also describes channel-mask pruning with a sparsity loss §3.3; the released code does not contain it, so only the un-pruned variant can be reproduced from the repository default.yaml.

Problem

The model sees new class groups over time and must classify every class seen so far, with no task identity at test time. Only a limited memory of old examples can be kept §3.1. The paper frames the core difficulty as the stability-plasticity trade-off: too much plasticity causes catastrophic forgetting, while too much stability blocks new concepts §1. Earlier approaches each give something up. Regularization methods such as EWC rest on a Laplace approximation that restricts how far weights can move §2. Distillation methods depend on the quality and amount of stored data §2. Structure-based methods that allocate new parameters still forget: RPSNet sums features across steps, and CCGN freezes only part of the layers §2, Discussion. The paper reports a large gap between these methods and a model trained on all data at once §1 fig. 1.

Method

Setting and two-stage learning (§3.1). At step t the model receives data D_t for a new class set Y_t and keeps a memory M_t of old examples (rehearsal). Learning at each step is split into two stages: representation learning, and then classifier learning §3.1.

Expandable representation (§3.2). At step t the feature extractor is Φ_t(x) = [Φ_{t-1}(x), F_t(x)], where F_t is a new extractor eq. 1. The prediction is Softmax(H_t(u)) on the concatenated feature u eq. 2. The earlier extractors (their parameters and BatchNorm statistics) are not updated at step t. F_t is initialized from F_{t-1} for faster adaptation §3.2. The classifier H_t keeps its old input weights from H_{t-1} and gets new, randomly initialized weights for the new input dimensions §3.2.

Training uses cross-entropy over memory plus new data eq. 3. An auxiliary classifier H^a_t reads only F_t(x) and has |Y_t|+1 outputs: the new classes and one merged "old" class §3.2. The expandable-representation loss is L_ER = L_H + λ_a·L_Ha eq. 4, with λ_a = 0 at the first step §3.2.

Dynamical expansion by channel masks (§3.3, paper only). The paper adapts channel-level masks from HAT §3.3. Each channel of a layer's feature map is multiplied by a gate m = σ(s·e), where e are learnable mask parameters and s sharpens the sigmoid eq. 5 eq. 6. Old extractors are replaced by pruned versions, and during training the new block uses soft masks eq. 7. Within each epoch s grows linearly from 1/s_max to s_max across batches eq. 8. The gradient with respect to e is rescaled to cancel the effect of s eq. 9. A sparsity loss measures the share of weights still in use, weighted by kernel sizes; the input layer's mask norm is set to 3 eq. 10. The total loss is L_DER = L_H + λ_a·L_Ha + λ_s·L_S eq. 11. After training, masks are binarized (large s) and the block is pruned §3.3.

Classifier learning (§3.4). After the representation is learned, the classifier is re-initialized and retrained only on a class-balanced subset of D_t ∪ M_t, with cross-entropy and a softmax temperature δ §3.4. The extractors stay fixed in this stage §3.4.

# DER, one incremental step t >= 2  [§3.2]
# Input: frozen extractors F_1..F_{t-1}, new images and labels, memory

def add_extractor(extractors):
    # The new extractor starts from the last one  [§3.2]
    extractors.append(copy_of(extractors[-1]))
    # Old extractors: no gradients, BatchNorm in eval mode  [§3.2]
    for old in extractors[:-1]:
        freeze(old)
    return extractors


def forward(extractors, classifier, aux_head, images):
    # Concatenate every extractor's features  [eq. 1]
    features = concat([f(images) for f in extractors], axis=1)
    logits = classifier(features)                  # (B, n_seen)
    # Auxiliary head reads only the new extractor  [§3.2]
    new_features = features[:, -new_dim:]          # (B, d)
    aux_logits = aux_head(new_features)            # (B, n_new + 1)
    return logits, aux_logits


def loss_step(logits, aux_logits, labels, n_old, step):
    # Cross-entropy over all seen classes  [eq. 3]
    loss_ce = cross_entropy(logits, labels)
    # Old classes -> 0, new classes -> 1..n_new  [§3.2]
    old_mask = labels < n_old
    aux_labels = where(old_mask, 0, labels - n_old + 1)
    loss_aux = cross_entropy(aux_logits, aux_labels)
    # lambda_a = 1 in all experiments; no aux term at step 1  [§3.2] [app.]
    if step == 1:
        return loss_ce
    return loss_ce + loss_aux

# Classifier learning after the representation  [§3.4]
# Input: class-balanced subset of new data plus memory

def classifier_stage(classifier, extractors, balanced_data, delta):
    # Re-initialize the classifier head  [§3.4]
    classifier.reset_parameters()
    for images, labels in balanced_data:
        # Extractors are fixed; only the head is trained  [§3.4]
        logits = classifier(extractors(images))
        # Temperature-scaled softmax  [§3.4]
        loss = cross_entropy(logits / delta, labels)
        update(classifier, loss)

# Paper only: channel masks, not in the released code  [§3.3]
# Input: feature maps (B, C, H, W), learnable mask logits e (C,), sharpness s

def masked_features(features, mask_logits, s):
    # Sigmoid gate per channel  [eq. 6] [eq. 5]
    mask = sigmoid(s * mask_logits)                # (C,)
    return features * mask[None, :, None, None]   # (B, C, H, W)


def sharpness(batch_index, n_batches, s_max):
    # Linear ramp inside each epoch  [eq. 8]
    low = 1.0 / s_max
    frac = (batch_index - 1) / (n_batches - 1)
    return low + (s_max - low) * frac

Hyperparameters

Values in the paper are in the appendix ("Hyperparameters") and §4.1. Values from the released configuration are cited separately where they differ.

Results

Author-reported. Average incremental accuracy (top-1 over steps) unless marked. "Bound" is the paper's row label; fig. 1 calls the model trained on all data "Joint", and no table defines the label further fig. 1. Setup: ResNet-18 on CIFAR-100 and ImageNet with the protocols in §4.1. Baselines were rerun by the authors on the same class orders unless they had no released code, in which case the authors used their own implementation; RPSNet numbers are copied from its paper §4.1.

CIFAR-100, B0 protocol (100 classes, all in steps; memory 2000) tab. 1

The paper's gain column is against the best baseline in each column, which is BiC at 5 steps and WA at 10, 20 and 50 steps tab. 1.

CIFAR-100, B50 protocol (50 classes first, then 2, 5 or 10 steps; 20 exemplars per class) tab. 2

ImageNet-100 and ImageNet-1000 tab. 3

ImageNet-100 B50 (50 classes first, then 10 steps; 20 per class) tab. 3:

Headline claims the text makes from these tables: on CIFAR-100 B0 with 50 steps, the average goes from 64.32% (WA) to 72.05% (+7.73) with fewer parameters §4.2; the last-step accuracy gap is reported as 42.75% → 58.66% for that setting §4.2. On CIFAR-100 B50 with 10 steps, DER beats PODNet by 8.41 points average accuracy §4.2 tab. 2. On ImageNet-100 B0, top-5 average beats the best baseline by about 1.79 points §4.3 tab. 3.

Ablation, CIFAR-100 B0 at 10 steps tab. 4

Forgetting and forward transfer, CIFAR-100 B0 at 10 steps tab. 5

Modified 32-layer ResNet, CIFAR-100 B0 app. Results for modified 32-layer ResNet

Modified 32-layer ResNet, CIFAR-100 B50 app. Results for modified 32-layer ResNet

Other analyses (appendix). Sensitivity to λ_a on CIFAR-100 B0 at 10 steps: 0.1 → 74.12 ± 0.06; 0.5 → 74.41 ± 0.16; 1 → 74.64 ± 0.28; 5 → 74.52 ± 0.33; 10 → 73.54 ± 0.22 app. Sensitive Study of Hyper-parameters. An "ideal" classifier retrained on all past data reaches 76.14 ± 0.80 on that setting, against 74.64 ± 0.28 for DER app. The Quality of Decision Boundary. Inference latency on ImageNet on a GTX 1080Ti is 1.07 ms per image, against 0.99 ms for BiC and WA app. Latency.

Figures, in words. fig. 1 plots average incremental accuracy against model size on CIFAR-100 B0 at 10 steps for WA, BiC, RPSNet, iCaRL, UCIR, PODNet and a jointly trained model. fig. 2 shows the step-t model: frozen extractors Φ_{t-1}^P, the new extractor F_t with learned masks, the concatenated representation, the classifier and the auxiliary head; masks are pruned after training. fig. 3 plots accuracy after each step for CIFAR-100 B0 (20 and 50 steps) and B50 (5 and 10 steps) against the baselines. fig. 4 tracks accuracy on the first class group over steps, as a backward-transfer measure (CIFAR-100 B0, 10 steps).

Paper vs code

The official repository is Rhyssiyan/DER-ClassIL.pytorch, linked from the paper's footnote on the abstract page §1. It is MIT-licensed LICENSE. The commit read is dd4af61. The README acknowledges that it builds on the incremental-learning code base of Douillard et al. README.md.

• No channel masks, sparsity loss or pruning in the code. The paper's core expansion method §3.3 and its loss L_S and λ_s eq. 10 eq. 11 are absent. A search of inclearn/ and codes/ finds no mask, gate or sparsity code. The experiment folders are named wo_mask and wo_prune 1.yaml default.yaml. Only the un-pruned variant (der: True, extractors added and never shrunk) is in the code network.py. The "with pruning" rows of tab. 1 to tab. 3 cannot be reproduced from this repository.
• Classifier head is reset at the end of warm-up. The paper says the old input weights of H_t are inherited from H_{t-1} §3.2. The code resets the classifier and the auxiliary head when epoch == warmup_epochs incmodel.py. The DER configs also set reuse_oldfc: False 1.yaml, and the inheritance branch only runs when that flag is set network.py. In practice the old head weights are not carried over.
• Auxiliary head matches the paper. Its input is only the new extractor's features network.py. Its size is |Y_t|+1 network.py. Targets are 0 for old classes and 1..|Y_t| for new ones incmodel.py. The auxiliary term is added only from the second step incmodel.py. Its weight is fixed at 1 and not a config key, so λ_a is not tunable from configuration.
• New extractor and freezing match the paper. F_t is initialized from F_{t-1} network.py. Earlier extractors have requires_grad = False and are put in eval mode, which freezes BatchNorm statistics incmodel.py incmodel.py.
• Epoch counts differ from the appendix (CIFAR-100). The paper says 160 epochs after a 10-epoch warm-up, with LR decays at 100 and 120 [app. Hyperparameters]. The config sets epochs: 170 with warmup_epochs: 10 and milestones 100 and 120 1.yaml. The total and warm-up figures agree; the milestone reference point is not stated (see Pitfalls).
• Epoch counts differ (ImageNet). The paper says 120 epochs after warm-up, LR decays at 30, 60, 80 and 90, and the same warm-up as CIFAR-100 (10 epochs) app. Hyperparameters. The ImageNet-100 config sets epochs: 220, warmup_epochs: 20 and milestones 60, 120, 160 and 180 1.yaml. The config is the only ImageNet setting in the repository, so the appendix values cannot be reproduced from it.
• Backbone for ImageNet. The paper says an 18-layer ResNet §4.1. The config uses preact_resnet18 1.yaml.
• ImageNet-1000 is not in the repository. The README covers ImageNet100 only README.md. No ImageNet-1000 config exists, so tab. 3's ImageNet-1000 column cannot be reproduced from this code.
• Momentum and eval mode are unstated in the paper. SGD momentum is 0.9 for the representation stage factory.py and for the classifier stage utils.py. The classifier stage also runs the whole network in eval mode utils.py.
• Classifier stage matches the paper on the important points. Balanced subset from current data data.py data_utils.py; temperature divides the logits inside cross-entropy utils.py; the head is reset first incmodel.py; 30 epochs, LR 0.1, decay at 15 1.yaml.
• Head re-initialization uses a different init. The classifier stage calls reset_parameters() on the linear layer, which uses PyTorch's default init incmodel.py. The representation-stage classifier is created with Kaiming normal init network.py. The paper gives no init for either.
• Unused config key. The top-level temperature: 2 in the DER configs is assigned but never read incmodel.py. Only the decouple temperature is used.
• Exemplar selection. The paper states herding for ImageNet only §4.1. The configs set the iCaRL coreset strategy for CIFAR-100 too 1.yaml.

Pitfalls

• Pruning is described, not released. An implementer following the paper must write the masks, the sharpness schedule and the sparsity loss from §3.3 alone. The paper gives no s_max, no mask initialization and no kernel-size convention for L_S beyond eq. 10. Expect the pruned numbers to be unreproducible without the authors' code.
• The ablation numbers are for the un-pruned model. tab. 4's 75.36 (10 steps) equals the "DER without pruning" row of tab. 1, not the headline DER of 74.64 tab. 1. The table does not say pruning is off.
• Classifier reset discards inherited weights. The reset at the end of warm-up incmodel.py means the paper's inheritance of H_{t-1} weights §3.2 has little effect even if reuse_oldfc were enabled. Decide which you want before implementing.
• Milestone reference point is ambiguous. The paper's "after warm-up, 160 epochs, decay at 100 and 120" app. Hyperparameters and the config's epochs: 170 plus milestones 100 and 120 1.yaml can be read as milestones counted from the start of training or after warm-up. Reading GradualWarmupScheduler scheduler.py together with set_optimizer incmodel.py suggests the MultiStepLR is stepped only after warm-up. This is a reading of the code, not a tested result.
• ImageNet schedule does not match the appendix. 220 epochs with a 20-epoch warm-up and milestones at 60, 120, 160, 180 (config) versus 120 epochs, 10-epoch warm-up and 30, 60, 80, 90 (paper). Use the config if you want to match the repository's ImageNet-100 results; use the paper if you want to match its text. Results were not checked against either here.
• Balanced subset size. construct_balanced_subset cuts every class to the smallest class count in whatever data_inc holds data_utils.py. Whether memory exemplars are in that set, and so how many samples each class keeps, was not verified from the code here.
• Gain columns use the best baseline per column. Table 1's gain at 5 steps is +2.45 against BiC (73.10), not WA (72.81). Table 2 gains are against BiC at 2 steps and PODNet at 5 and 10 steps tab. 2. Read the baseline name before quoting a gain.
• Forgetting and transfer formulas are ambiguous as typeset. In eq. 12 the minus sign sits outside the inner sum, which as printed covers only A^i_{Y_j}; the intended sum over both terms is not certain. eq. 13 has the same layout eq. 12 eq. 13. Both use an "ideal" classifier trained on all past data to isolate the representation §4.4, which is not deployable.
• λ_s reference in the appendix is wrong. The appendix says λ_s sensitivity is in "Figure 1 in the main body" app. Sensitive Study of Hyper-parameters. Figure 1 is the model-size plot fig. 1, and no λ_s figure appears in the source. The λ_s sensitivity result is not shown in this card.
• Model size grows with steps. The #Paras column (average over steps) rises from 33.6M at 5 steps to 285.6M at 50 steps without pruning, against 11.2M for one ResNet-18 tab. 1. Inference cost grows with the parameter count. The latency table is for ImageNet only app. Latency.
• Old features are fixed after their step. Old extractors never change again. The paper still reports positive backward transfer (+1.36) tab. 5, measured with an ideal classifier, so check whether your task needs the classifier-level result or the representation-level one §4.4.
• ImageNet spread. The text says ImageNet experiments use three class orders §4.1, but tab. 3 has no ± values. Treat ImageNet numbers as single-run unless you rerun them.
• Architecture differs from the code for ImageNet. The paper's "18-layer ResNet" §4.1 is not pinned to a variant. The released ImageNet config uses the pre-activation variant 1.yaml.

Lineage

• Builds on: iCaRL for herding exemplar selection and the class-incremental protocol §4.1; EEIL for the balanced finetuning of the classifier §3.4; HAT for the channel-level mask gating §3.3; the decoupling of representation and classifier learning §1; temperature-scaled softmax §3.4.
• Compared against: iCaRL, UCIR, BiC, WA, PODNet tab. 1 tab. 2; RPSNet §2; TPCIL tab. 2 tab. 3. The closest structure-based methods, RPSNet and CCGN, are discussed in §2 (Discussion).
• Superseded by: not stated in this card.