Dynamic network expansion

Overview

A network trained on a stream of new classes tends to forget the old ones, and a fixed-size network has to trade stability for plasticity. Dynamic network expansion avoids that trade-off by giving each new step its own capacity. The network gets a new block of parameters for new data, and the blocks learned earlier are frozen, so their outputs cannot drift 2103.16788 §1 2103.16788 §3.2.

The setting is class-incremental image classification with no task identity at test time. A limited memory of old examples is kept for rehearsal 2103.16788 §3.1. The approach is the one the DER paper calls "expandable representation" 2103.16788 §3.2. The 2018 review of lifelong learning groups the same idea, adding capacity and freezing what was learned, under dynamic architectures 1802.07569 §3.3. Its treatment is qualitative, so the numbers on this page come from DER alone.

Variants

Two families are covered. DER gives the quantitative picture and has two forms that share the same expansion step. They differ in whether the new block is pruned after training. The survey's dynamic architectures are a wider, mostly qualitative family.

Dynamic architectures in the survey

The review describes two growth methods in detail. Progressive networks freeze the networks of earlier tasks and add a new sub-network per task, with lateral connections to the old ones 1802.07569 §3.3. Growing self-organizing networks pick a best-matching unit and update it with a habituation factor, so well-learned units change less. New neurons are inserted when a habituated unit's activity falls below a threshold 1802.07569 §3.3 1802.07569 eq. 7 1802.07569 eq. 8. The review also lists dynamically expanding networks with group-sparse selection, denoising autoencoders that add and merge neurons, and ART-style models, in prose only [arxiv:1802.07569 §3.3]. It gives no learning rate, threshold or insertion rule for the growing network 1802.07569 §3.3. Card: Continual Lifelong Learning with Neural Networks: A Review.

The review reports these methods only as qualitative comparisons. A growing self-organizing network is described as competitive with batch learning on the Weizmann and KTH action benchmarks, and additive neurogenesis is described as better than a static network with the same neuron count 1802.07569 §3.3. No accuracy figures are given.

DER variants

DER without pruning

At step t the feature extractor is the concatenation of all earlier extractors and a new one, F_t 2103.16788 eq. 1. A single linear classifier reads the concatenated features 2103.16788 eq. 2. Old extractors, including their normalization statistics, are not updated at step t. The new extractor starts from the previous one to speed up learning 2103.16788 §3.2. This is the model the released code implements 2103.16788 §3.2. Card: DER: Dynamically Expandable Representation for Class Incremental Learning.

With this variant, the paper reports 72.41% average accuracy on CIFAR-100 with 50 steps, using 285.6M parameters in the table's #Paras column 2103.16788 tab. 1. At 10 steps it reports 75.36% with 61.6M 2103.16788 tab. 1.

DER with channel-mask pruning

The paper also adds channel-level gates to the new block, adapted from HAT. A sparsity loss pushes the gates towards using fewer channels, and the block is pruned after training 2103.16788 §3.3. The paper's pruned rows give 72.05% at 50 steps with 10.15M parameters 2103.16788 tab. 1, and 74.64% at 10 steps with 4.96M 2103.16788 tab. 1.

The paper does not state the sharpness maximum, the gate initialization or the sparsity-loss constants beyond its equations, and the released code has no masks or pruning. Treat the pruned numbers as paper-reported only. See the card's Paper vs code section.

Training stages in both variants

After the representation is learned, the classifier is re-initialized and retrained on a class-balanced subset of new data and memory, with the extractors held fixed 2103.16788 §3.4. The same paper adds an auxiliary head that reads only the new extractor and separates new classes from one merged old class; its loss is added from the second step on 2103.16788 §3.2 2103.16788 eq. 4.

When to use which

Implementation sketch

Own-words pseudocode of the expansion loop. The pruning gates from §3.3 are left out, since they are not released and the paper does not give every constant.

# One incremental step t >= 2  [arxiv:2103.16788 §3.2]
# Input: frozen extractors F_1..F_{t-1}, new data, memory of old examples

def expand(extractors):
    # New block starts from the previous one  [arxiv:2103.16788 §3.2]
    extractors.append(copy_of(extractors[-1]))
    # Older blocks get no gradient updates  [arxiv:2103.16788 §3.2]
    for block in extractors[:-1]:
        freeze(block)
    return extractors


def forward(extractors, classifier, aux_head, images):
    # Concatenate every block's output  [arxiv:2103.16788 eq. 1]
    features = concat([block(images) for block in extractors], axis=1)
    logits = classifier(features)
    # The auxiliary head sees only the newest block  [arxiv:2103.16788 §3.2]
    aux_logits = aux_head(features[:, -new_dim:])
    return logits, aux_logits


def train_step(logits, aux_logits, labels, step):
    # Cross-entropy over all seen classes  [arxiv:2103.16788 eq. 3]
    loss = cross_entropy(logits, labels)
    # New-vs-old term, added from step 2  [arxiv:2103.16788 eq. 4]
    if step >= 2:
        loss += cross_entropy(aux_logits, new_vs_old_labels(labels))
    return loss


# After the representation is learned  [arxiv:2103.16788 §3.4]
def retrain_classifier(classifier, extractors, balanced_subset, temperature):
    # Re-initialize the head, keep all blocks fixed
    classifier.reset_parameters()
    for images, labels in balanced_subset:
        logits = classifier(extractors(images))
        loss = cross_entropy(logits / temperature, labels)
        update(classifier, loss)

Contradictions

• Pruning in the paper and in the code. The paper reports pruned results in tab. 1 to tab. 3 and describes the masks in §3.3 2103.16788 tab. 1 2103.16788 §3.3. The released repository has no mask or pruning code, so only the unpruned rows can be reproduced from it. See the card's Paper vs code section.
• Classifier weights across steps. The paper says the classifier keeps the old input weights and adds new ones for new inputs 2103.16788 §3.2. The card reports that the released code resets the classifier at the end of warm-up, so the inheritance is not used there (card: classifier head reset).
• ImageNet schedule. The paper's appendix gives 120 epochs after a 10-epoch warm-up for ImageNet 2103.16788 app. Hyperparameters. The released ImageNet-100 config uses different counts, so the two cannot both be matched (card).
• Cost of growth. The review says growing networks raise computation and can scale poorly 1802.07569 §3.1. DER reports 1.07 ms per image on ImageNet against 0.99 ms for BiC and WA 2103.16788 app. Latency. The two sources measure different things, and the review gives no timing, so the DER latency figure should not be read as a test of the review's claim.
• Ablation model. The ablation table does not say whether pruning is on. Its full-model row, 75.36% at 10 steps, equals the unpruned row of tab. 1, not the headline pruned value of 74.64% 2103.16788 tab. 4 2103.16788 tab. 1. Compare like with like.

Papers

• DER: Dynamically Expandable Representation for Class Incremental Learning: a new extractor per step with frozen earlier extractors, an auxiliary new-vs-old loss, a balanced classifier retrain, and an optional channel-mask pruning step that is not in the released code.
• Continual Lifelong Learning with Neural Networks: A Review: a survey of catastrophic forgetting. Its dynamic-architecture section describes progressive networks and growing self-organizing networks, with qualitative results only.