---
kind: concept
slug: dynamic-network-expansion
title: Dynamic network expansion
summary: >-
  Learning new classes by adding capacity per step and freezing earlier blocks, so old knowledge is not overwritten. DER reports 72.41% average accuracy on CIFAR-100 with 50 steps unpruned and 72.05% pruned; a survey covers the wider family qualitatively. The trade-off is model size, which grows each step; the pruned variant is not released.
tags: [continual-learning, dynamic-architecture, catastrophic-forgetting, network-pruning]
papers: ["2103.16788", "1802.07569"]
related: [continual-learning, network-pruning, experience-replay]
status: draft
updated_at: 2026-10-24
---

## Overview

A network trained on a stream of new classes tends to forget the old ones,
and a fixed-size network has to trade stability for plasticity. Dynamic
network expansion avoids that trade-off by giving each new step its own
capacity. The network gets a new block of parameters for new data, and the
blocks learned earlier are frozen, so their outputs cannot drift
[arxiv:2103.16788 §1] [arxiv:2103.16788 §3.2].

The setting is class-incremental image classification with no task identity
at test time. A limited memory of old examples is kept for rehearsal
[arxiv:2103.16788 §3.1]. The approach is the one the DER paper calls
"expandable representation" [arxiv:2103.16788 §3.2]. The 2018 review of
lifelong learning groups the same idea, adding capacity and freezing what was
learned, under dynamic architectures [arxiv:1802.07569 §3.3]. Its treatment is
qualitative, so the numbers on this page come from DER alone.

## Variants

Two families are covered. DER gives the quantitative picture and has two
forms that share the same expansion step. They differ in whether the new block
is pruned after training. The survey's dynamic architectures are a wider,
mostly qualitative family.

### Dynamic architectures in the survey

The review describes two growth methods in detail. Progressive networks
freeze the networks of earlier tasks and add a new sub-network per task, with
lateral connections to the old ones [arxiv:1802.07569 §3.3]. Growing
self-organizing networks pick a best-matching unit and update it with a
habituation factor, so well-learned units change less. New neurons are
inserted when a habituated unit's activity falls below a threshold
[arxiv:1802.07569 §3.3] [arxiv:1802.07569 eq. 7] [arxiv:1802.07569 eq. 8]. The review also lists dynamically
expanding networks with group-sparse selection, denoising autoencoders that
add and merge neurons, and ART-style models, in prose only [arxiv:1802.07569
§3.3]. It gives no learning rate, threshold or insertion rule for the growing
network [arxiv:1802.07569 §3.3]. Card: [Continual Lifelong Learning with
Neural Networks: A Review](/papers/1802.07569).

The review reports these methods only as qualitative comparisons. A growing
self-organizing network is described as competitive with batch learning on
the Weizmann and KTH action benchmarks, and additive neurogenesis is described
as better than a static network with the same neuron count
[arxiv:1802.07569 §3.3]. No accuracy figures are given.

### DER variants

### DER without pruning

At step t the feature extractor is the concatenation of all earlier
extractors and a new one, F_t [arxiv:2103.16788 eq. 1]. A single linear
classifier reads the concatenated features [arxiv:2103.16788 eq. 2]. Old
extractors, including their normalization statistics, are not updated at
step t. The new extractor starts from the previous one to speed up learning
[arxiv:2103.16788 §3.2]. This is the model the released code implements
[arxiv:2103.16788 §3.2]. Card: [DER: Dynamically Expandable Representation for Class Incremental Learning](/papers/2103.16788).

With this variant, the paper reports 72.41% average accuracy on CIFAR-100
with 50 steps, using 285.6M parameters in the table's #Paras column
[arxiv:2103.16788 tab. 1]. At 10 steps it reports 75.36% with 61.6M
[arxiv:2103.16788 tab. 1].

### DER with channel-mask pruning

The paper also adds channel-level gates to the new block, adapted from HAT.
A sparsity loss pushes the gates towards using fewer channels, and the block
is pruned after training [arxiv:2103.16788 §3.3]. The paper's pruned rows give
72.05% at 50 steps with 10.15M parameters [arxiv:2103.16788 tab. 1], and
74.64% at 10 steps with 4.96M [arxiv:2103.16788 tab. 1].

The paper does not state the sharpness maximum, the gate initialization or
the sparsity-loss constants beyond its equations, and the released code has
no masks or pruning. Treat the pruned numbers as paper-reported only. See
[the card's Paper vs code section](/papers/2103.16788#paper-vs-code).

### Training stages in both variants

After the representation is learned, the classifier is re-initialized and
retrained on a class-balanced subset of new data and memory, with the
extractors held fixed [arxiv:2103.16788 §3.4]. The same paper adds an
auxiliary head that reads only the new extractor and separates new classes
from one merged old class; its loss is added from the second step on
[arxiv:2103.16788 §3.2] [arxiv:2103.16788 eq. 4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Class-incremental learning with an exemplar memory, and the model may grow with each step | DER without pruning | Old features are frozen, so later training cannot overwrite them. The paper's best average at 50 steps comes from this form at 72.41% [arxiv:2103.16788 tab. 1] |
| Class-incremental learning with a small parameter budget and you can implement the pruning yourself | DER with pruning | 10.15M against 285.6M parameters at 50 steps, for 0.36 points less average accuracy [arxiv:2103.16788 tab. 1]. The masks are described but not released [arxiv:2103.16788 §3.3] |
| Small backbone and many steps | DER without pruning, check the size | On a 32-layer ResNet at 50 steps the unpruned model reaches 70.58% with 11.73M parameters, and the pruned one 67.29% with 0.70M [arxiv:2103.16788 app. Results for modified 32-layer ResNet]. Pruning costs more accuracy here |
| Inference latency is a hard limit | Measure before choosing | The paper reports 1.07 ms per image for DER against 0.99 ms for BiC and WA on ImageNet, on a single GTX 1080Ti [arxiv:2103.16788 app. Latency]. No other latency figures are in the card |
| Unlabelled or self-organizing learning on a stream | Growing self-organizing network (survey) | The review presents this family for unsupervised lifelong learning and describes its results on two action benchmarks as competitive or better, qualitatively [arxiv:1802.07569 §1] [arxiv:1802.07569 §3.3]. It gives no numbers and no insertion rule, so an implementer must work them out from the original papers |
| Growing cost is a hard limit | A fixed-size method, such as replay or regularization | The review notes that growing networks raise computation and can scale poorly [arxiv:1802.07569 §3.1]. DER's unpruned model grows to 285.6M parameters at 50 steps [arxiv:2103.16788 tab. 1]. The review does not compare these costs with replay methods |
| No exemplar memory allowed | Not covered | Every reported setting keeps a memory of old examples: 2000 in total for CIFAR-100 B0, 20 per class for B50 [arxiv:2103.16788 §4.1]. The paper does not report a memory-free result |

## Implementation sketch

Own-words pseudocode of the expansion loop. The pruning gates from §3.3 are
left out, since they are not released and the paper does not give every
constant.

```pseudocode
# One incremental step t >= 2  [arxiv:2103.16788 §3.2]
# Input: frozen extractors F_1..F_{t-1}, new data, memory of old examples

def expand(extractors):
    # New block starts from the previous one  [arxiv:2103.16788 §3.2]
    extractors.append(copy_of(extractors[-1]))
    # Older blocks get no gradient updates  [arxiv:2103.16788 §3.2]
    for block in extractors[:-1]:
        freeze(block)
    return extractors


def forward(extractors, classifier, aux_head, images):
    # Concatenate every block's output  [arxiv:2103.16788 eq. 1]
    features = concat([block(images) for block in extractors], axis=1)
    logits = classifier(features)
    # The auxiliary head sees only the newest block  [arxiv:2103.16788 §3.2]
    aux_logits = aux_head(features[:, -new_dim:])
    return logits, aux_logits


def train_step(logits, aux_logits, labels, step):
    # Cross-entropy over all seen classes  [arxiv:2103.16788 eq. 3]
    loss = cross_entropy(logits, labels)
    # New-vs-old term, added from step 2  [arxiv:2103.16788 eq. 4]
    if step >= 2:
        loss += cross_entropy(aux_logits, new_vs_old_labels(labels))
    return loss


# After the representation is learned  [arxiv:2103.16788 §3.4]
def retrain_classifier(classifier, extractors, balanced_subset, temperature):
    # Re-initialize the head, keep all blocks fixed
    classifier.reset_parameters()
    for images, labels in balanced_subset:
        logits = classifier(extractors(images))
        loss = cross_entropy(logits / temperature, labels)
        update(classifier, loss)
```

## Contradictions

- **Pruning in the paper and in the code.** The paper reports pruned results
  in tab. 1 to tab. 3 and describes the masks in §3.3 [arxiv:2103.16788 tab. 1]
  [arxiv:2103.16788 §3.3]. The released repository has no mask or pruning code,
  so only the unpruned rows can be reproduced from it. See the
  [card's Paper vs code section](/papers/2103.16788#paper-vs-code).
- **Classifier weights across steps.** The paper says the classifier keeps the
  old input weights and adds new ones for new inputs [arxiv:2103.16788 §3.2].
  The card reports that the released code resets the classifier at the end of
  warm-up, so the inheritance is not used there
  ([card: classifier head reset](/papers/2103.16788#paper-vs-code)).
- **ImageNet schedule.** The paper's appendix gives 120 epochs after a
  10-epoch warm-up for ImageNet [arxiv:2103.16788 app. Hyperparameters]. The
  released ImageNet-100 config uses different counts, so the two cannot both
  be matched ([card](/papers/2103.16788#paper-vs-code)).
- **Cost of growth.** The review says growing networks raise computation and
  can scale poorly [arxiv:1802.07569 §3.1]. DER reports 1.07 ms per image on
  ImageNet against 0.99 ms for BiC and WA [arxiv:2103.16788 app. Latency]. The
  two sources measure different things, and the review gives no timing, so the
  DER latency figure should not be read as a test of the review's claim.
- **Ablation model.** The ablation table does not say whether pruning is on.
  Its full-model row, 75.36% at 10 steps, equals the unpruned row of tab. 1,
  not the headline pruned value of 74.64% [arxiv:2103.16788 tab. 4]
  [arxiv:2103.16788 tab. 1]. Compare like with like.

## Papers

- [DER: Dynamically Expandable Representation for Class Incremental Learning](/papers/2103.16788): a new extractor per step with frozen earlier extractors, an auxiliary new-vs-old loss, a balanced classifier retrain, and an optional channel-mask pruning step that is not in the released code.
- [Continual Lifelong Learning with Neural Networks: A Review](/papers/1802.07569): a survey of catastrophic forgetting. Its dynamic-architecture section describes progressive networks and growing self-organizing networks, with qualitative results only.