Continual learning

Overview

Neural networks trained on a new task usually lose performance on earlier tasks. This is catastrophic forgetting. Continual (lifelong, incremental) learning asks for a model that learns a stream of tasks or classes and keeps the earlier ones. The stream arrives in order, and past data is either absent or only partly kept 1812.00420 §2.

A review by Parisi et al. groups the fixes into three families: regularisation, dynamic architectures, and replay with complementary memories [arxiv:1802.07569 §3.2] 1802.07569 §3.3 1802.07569 §3.4. The review runs no experiments of its own. Its comparisons are qualitative and come from cited work 1802.07569 §3.5.

The cards on this page differ in the setting they assume:

• Task-labelled stream. The predictor takes a task descriptor t next to the input, and the evaluation reports average accuracy over all tasks at the end 1812.00420 §2 1812.00420 §3.
• Single-pass stream with a tiny memory. Each example is seen once. The learner keeps about one stored example per class, and each example carries a task id that selects its head 1902.10486 §3.1 1902.10486 §5.2.
• Class-incremental stream. Each step adds new classes. At inference the model must pick among all classes seen so far, with no task id. Old classes keep either a fixed number of stored examples 2004.13513 §3, or one stored class mean each, with no images kept 2004.00440 §4.1. DER keeps an exemplar memory and also freezes the features of each step inside a network that grows by one extractor per step 2103.16788 §3.1 2103.16788 §3.2.
• Segmentation stream. Each step adds new classes to a pixel labeller. In the new training images, pixels of old and future classes are labelled as background, and no old images are stored 2011.11390 §3.1.
• Federated clients. Clients hold non-IID shards, and the server averages their weights each round. FedNTD treats forgetting across rounds as the reason for poor accuracy, presented as a diagnostic and not a proven cause 2106.03097 §1 2106.03097 §2. In federated class-incremental learning, clients also receive new classes over time, and each keeps an exemplar memory 2203.11473 §3.
• Patch stream for an open-vocabulary model. Patch tasks arrive one at a time, and a supported task such as ImageNet must not regress. Each patch is a fine-tune followed by a weight blend with one coefficient α [arxiv:2208.05592 §2]. Each step starts from the last patched model 2208.05592 §5.
• Robot imitation stream. One task-conditioned policy learns tasks one after another from demonstrations. Earlier demonstrations are not fully available when a later task is learned 2306.03310 §2.2. The task is given as a language embedding 2306.03310 §4.4.
• Reinforcement-learning task stream. Each task is a reinforcement-learning environment. Progressive networks add one column per task and freeze the earlier ones. At test time the caller chooses the column, which needs a task label 1606.04671 §2.
• Stream of edits. Corrections are applied to one pretrained model. MEND turns one desired input-output pair into a weight edit in a single step 2110.11309 §3.1. The results recorded here for MEND are for several edits applied at once, not for a sequence 2110.11309 §4.3.
• Multimodal instruction stream with experts. A language model with LoRA experts learns instruction tasks one after another, with no stored samples. The authors trace forgetting to two drifts: the router sends old inputs to other experts, and new tasks overwrite the experts 2602.01990 §1. Each task is trained for one epoch 2602.01990 §5.1.
• Open-world detection stream. Objects of classes not yet named are first reported as "unknown". When a person labels them, the model is updated on the new labels together with a stored, balanced set of exemplars 2103.02603 §3 2103.02603 §5.2.
• Instruction stream for a language model. Five instruction tasks are trained one after another in one fixed order, and earlier task data is not available at later stages. Forgetting is read from general benchmarks 2308.08747 §3.1 2308.08747 §3.2.

A survey names lifelong learning, with changing label sets, as an open challenge for aspect-based sentiment analysis 2203.01054 §7. It reports no experiments on it.

The iSDF card lists this concept, because its network trains online on a stream of posed depth frames 2204.02296 §4. Its metrics are SDF error, collision cost and gradient error 2204.02296 §5.3. None of these measures forgetting, so iSDF is not a variant on this page.

Variants

Episodic memory with gradient projection (A-GEM)

Keep a small memory of examples from each finished task. On each step, compute the gradient of the new mini-batch and a reference gradient on a random batch from memory. If the two point in conflicting directions, remove the conflicting part of the update. The memory constraint is averaged over all past tasks, so no quadratic program is needed 1812.00420 §4 1812.00420 eq. 11. Card: Efficient Lifelong Learning with A-GEM.

Reported on Split CIFAR, the average accuracy after the last task is 62.3% for A-GEM and 61.2% for GEM, the method it simplifies. Plain fine-tuning (VAN) gets 42.9% 1812.00420 tab. 4. Training time on the same dataset is 449 s for A-GEM and 5238 s for GEM 1812.00420 tab. 7.

Replay of a tiny memory without gradient constraints (ER)

Each step trains on the current mini-batch stacked with a same-size batch drawn at random from an episodic memory. The memory is written after the step. No gradient is projected. The stacked batches are simply averaged in the loss 1902.10486 §4 1902.10486 alg. 1. The paper compares four writing rules: reservoir sampling, a per-class ring buffer, k-means and mean-of-features 1902.10486 alg. 2–5. Card: On Tiny Episodic Memories in Continual Learning.

Author-reported, with one example per class and one pass over each stream. The ring-buffer writer averages 70.2 on Permuted MNIST, 56.2 on Split CIFAR, 49.0 on Split miniImageNet and 65.0 on Split CUB. A-GEM gets 62.1, 54.9, 48.2 and 62.1 on the same sets, and fine-tuning gets 53.5, 40.6, 34.7 and 55.7 1902.10486 tab. 3–tab. 6. On Split CIFAR the ring-buffer run takes 116 s, against 87 s for fine-tuning and 230 s for A-GEM. The paper gives no hardware 1902.10486 tab. 2. The released code writes with reservoir sampling only, so the ring-buffer and k-means numbers cannot be reproduced from it; see the card's Paper vs code section.

Exemplar distillation for class-incremental learning (PODNet)

Keep a fixed number of exemplars per class, here 20, selected by herding. The loss on the new classes is a cosine classifier with several proxies per class. A distillation term compares the current network with a frozen copy of the previous one. It matches width- and height-pooled activation statistics at each stage (POD-spatial) and the normalized final embedding (POD-flat) 2004.13513 §3.1 2004.13513 eq. 5 2004.13513 eq. 6. Card: PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning.

On CIFAR100 with 50 steps of one class, the average incremental accuracy is 61.40% with NME inference and 57.98% with the classifier, against 49.30% for UCIR with its CNN classifier 2004.13513 tab. 1. The gap narrows as the steps get larger: with 5 steps of 10 classes, PODNet (CNN) reaches 64.83% against 64.01% for UCIR (CNN) 2004.13513 tab. 1.

Expandable network with exemplars (DER)

Each step adds a new feature extractor, initialized from the previous one. All earlier extractors are frozen, and the classifier reads their concatenated features 2103.16788 §3.2 2103.16788 eq. 1 [arxiv:2103.16788 eq. 2]. An auxiliary head reads only the new extractor. It is trained to tell new classes from one merged old class 2103.16788 §3.2 [arxiv:2103.16788 eq. 4]. At the end of each step the classifier is re-initialized and retrained on a class-balanced subset of new data and memory 2103.16788 §3.4. The memory is 2000 exemplars in total for CIFAR-100 with all classes in steps, and 20 per class in the B50 protocol 2103.16788 §4.1. Card: DER: Dynamically Expandable Representation for Class Incremental Learning.

Author-reported. On CIFAR-100 with 50 steps, the average incremental accuracy is 72.05% with pruning and 72.41% without, against 64.32% for WA, the best baseline in that column 2103.16788 tab. 1. With 10 steps in the B50 protocol, DER reaches 72.45% against 64.04% for PODNet 2103.16788 tab. 2. In the ablation at 10 steps, the average is 61.84% with no expansion and no auxiliary loss, 73.26% with expansion alone, and 75.36% with both. The ablation is for the un-pruned model 2103.16788 tab. 4. The channel-mask pruning of §3.3 is in the paper but not in the released code, so the pruned rows cannot be reproduced from it; see the card's Paper vs code section.

Frozen columns with lateral connections (progressive networks)

Add a new column for each task, and freeze all earlier columns. Each layer of the new column takes its own previous layer and the previous layer of every earlier column, through lateral weights 1606.04671 §2 [arxiv:1606.04671 eq. 1]. Because the frozen columns are never updated, earlier tasks keep their behaviour by construction 1606.04671 §2. The paper also replaces each lateral connection with an adapter, which projects the lateral inputs down and scales them by a learned scalar 1606.04671 §2 1606.04671 eq. 2. Card: Progressive Neural Networks.

Author-reported, with asynchronous advantage actor-critic (A3C). On Pong Soup, the 2-column net reaches a mean transfer score of 209% against 181% for full finetuning. On Atari the two are level, 132% against 133% [arxiv:1606.04671 tab. 1]. On Labyrinth the 2-column mean is 491%, while the median is 115% 1606.04671 tab. 1. Each column adds parameters, and the total grows quadratically with the number of columns 1606.04671 app. 2. The paper has no released code; see the card's Paper vs code section.

Drift compensation of stored class means (SDC)

Use an embedding network with no output head, and classify by nearest class mean. After each task, store one prototype per class, not images. Once the next task is trained, estimate how each stored prototype has moved. The estimate is a Gaussian-weighted average of how the current-task images moved in the embedding, and the prototype is shifted by it 2004.00440 §4.1 2004.00440 eq. 10. The paper combines this correction with the regularisers E-LwF, E-EWC and E-MAS, each adapted from softmax to embeddings 2004.00440 §3.3 2004.00440 §4.2. Card: Semantic Drift Compensation for Class-Incremental Learning.

Author-reported. On six-task CUB-200-2011, E-EWC+SDC reaches 56.4% against 54.2% for E-EWC, and on six-task Flowers-102 it reaches 83.9% against 79.6% 2004.00440 tab. 1. On eleven-task CIFAR100, E-LwF+SDC equals E-LwF at 46.1%. On eleven-task ImageNet-Subset it reaches 50.5% against 51.5% 2004.00440 tab. 5. The official code applies the correction only at evaluation time; see the card's Paper vs code section.

Pseudo-labels and pooled distillation for continual segmentation (PLOP)

For semantic segmentation, the model gets no old images and no task id. Distillation matches pooled feature statistics over strips and grid cells of the old and new feature maps, at several scales 2011.11390 §3.2 2011.11390 eq. 5. For background pixels, the frozen old model gives a label only when its prediction entropy is below a per-class threshold, which is the median entropy of that class. Other background pixels get no target 2011.11390 §3.3 2011.11390 eq. 7. Card: PLOP: Learning without Forgetting for Continual Semantic Segmentation.

Author-reported, under the Overlapped protocol. On Pascal VOC 2012, when 15 classes come first and one class is added per step (15-1), PLOP reaches 54.64 mIoU over all classes against 29.29 for the authors' MiB re-implementation 2011.11390 tab. voc_sota_full. On 15-5 the authors report PLOP on par with their MiB re-implementation, 70.09 against 70.08 2011.11390 §4.2. On ADE20k with 100 classes first and 10 per step, PLOP reaches 31.59 against 29.24 for MiB 2011.11390 tab. ade_sota.

Not-true distillation across federated clients (FedNTD)

This is not a stream of tasks. Clients train on their own shards, and the server averages the uploads each round. Each client adds a distillation term that matches the frozen round-start global model over the not-true classes only. The true-class logit is left out of both softmaxes, so the term has no gradient on it 2106.03097 §3 2106.03097 eq. FedNTD. The paper borrows the stability-plasticity framing from continual learning 2106.03097 §3. The method sends nothing extra to the server 2106.03097 §4.2. Card: Preservation of the Global Knowledge by Not-True Distillation in Federated Learning.

Author-reported. On CIFAR-10 with two classes per client, FedNTD reaches 52.61% against 40.14% for FedAvg 2106.03097 tab. 1. The paper says FedNTD "consistently outperforms" the baselines, but its own table does not support that: SCAFFOLD is higher on CIFAR-100 under LDA, 33.68 against 32.37 2106.03097 §4.2 2106.03097 tab. 1. The default τ = 1 is not tuned, and too small a τ lowers accuracy 2106.03097 app.. The paper does not report the extra compute of the frozen model's forward pass 2106.03097 §4.

Gradient-compensated loss with server-side prototype reconstruction (GLFC)

This is a federated variant of class-incremental learning. Each client trains on mini-batches that mix new data with its exemplar memory. The loss has two terms. The first re-weights each sample's binary cross-entropy by its gradient magnitude, divided by the mean for its own group, new or old classes 2203.11473 §4.1 2203.11473 eq. 4. The second is a KL term that matches the current outputs to soft targets on the old classes, taken from a previous model 2203.11473 §4.1 2203.11473 eq. 5. A client flags a new task when the average entropy of the global model on its data rises by at least a factor of 1.2 2203.11473 §4.1 2203.11473 eq. 7.

On a new task, a client does not send its samples. For each new class it picks the sample whose feature is closest to the class mean, and optimises a copy of it with noise added to the latent feature 2203.11473 §4.2 2203.11473 eq. 10–11. It sends the gradients of these prototypes. The server rebuilds inputs that match those gradients 2203.11473 §4.2 2203.11473 eq. 8–9. It scores the global model on them, keeps the best model, and sends the best models of the last two tasks to the selected clients 2203.11473 §4.2 2203.11473 §4.3. Card: Federated Class-Incremental Learning.

Author-reported, with ResNet-18 and the mean of three seeds [arxiv:2203.11473 §5.1]. On CIFAR-100 with ten tasks, average accuracy is 66.9, against 61.5 for the best baseline, GeoDL with iCaRL and FL 2203.11473 tab. 1. On ImageNet-Subset with ten tasks it is 57.0, against 52.6 for PODNet with FL 2203.11473 tab. 2. The abstract's range of 4.4 to 15.1 points spans all settings, and its two ends come from these tables 2203.11473 tab. 1 2203.11473 tab. 2. The released code differs from the paper in the perturbation step, the distillation loss, the reconstruction settings and the best-model rule, and it ships only the ten-task configuration. Read the card's Paper vs code section and its Pitfalls before implementing.

Weight blending after each patch (PAINT)

Fine-tune the visual encoder of a zero-shot open-vocabulary model on one patch task. The head stays frozen, taken from the text encoder, so no class-specific parameter is added. Then blend the zero-shot and fine-tuned weights with one coefficient α in [0, 1]. α is chosen on held-out validation data for the supported task and the patch task 2208.05592 §2 2208.05592 §3. For a stream of patches, each step starts from the last patched model 2208.05592 §5. The authors report no extra cost at fine-tuning or inference time over standard fine-tuning 2208.05592 §1. Card: Patching open-vocabulary models by interpolating weights.

On ViT-B/32, sequential patching without interpolation lowers combined accuracy by 4.6 points, and with interpolation it raises it by 11 points, both against the unpatched model 2208.05592 §5. Interpolation reduces forgetting but does not remove it 2208.05592 §5 2208.05592 §9. The released code covers single-task patching and does not select α; see the card's Paper vs code section.

Router and expert freezing for mixture-of-LoRA experts (SAME)

The model has LoRA experts and a router in the feed-forward layers of its language part. The vision encoder and projector stay frozen [arxiv:2602.01990 §3] 2602.01990 §5.1. SAME adds three mechanisms to this setup. The router's input covariance is kept as a running estimate, and only its leading directions are used for the router update. Router gradients in those directions are rescaled, and the rest of the gradient is passed through unscaled 2602.01990 §4.1 2602.01990 eq. 12. Expert gradients are multiplied by an approximate inverse of the previous task's covariance 2602.01990 §4.2 2602.01990 eq. 16. An expert is frozen for the current task when its utilisation, minus its historical importance, falls below a threshold. It is trained again at the next task and at inference 2602.01990 §4.3 2602.01990 eq. 19. Card: SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning.

Author-reported, on one run with no error bars 2602.01990 §5.1. On CoIN, average accuracy is 66.82 against 50.58 for MoELoRA [arxiv:2602.01990 tab. 2]. The ablation adds the three mechanisms in turn, with the router update first: 61.32, then 65.89, then 66.82 2602.01990 tab. 4. On UCIT it reaches 67.12 against 52.06 for MoELoRA and 65.52 for ModalPrompt, the best baseline in that table 2602.01990 tab. 3. On the ten-task TriGap benchmark, which the paper introduces, it reaches 46.53 against 44.45 for MoE-LoRA 2602.01990 tab. 1. The released code differs from the equations in several places, such as routing once per batch, the expert product and the scale direction. Read the card's Paper vs code section before implementing.

Pseudo-unknowns and exemplar replay for open-world detection (ORE)

Detect objects of unseen classes as "unknown", and learn them once they are labelled. ORE builds on Faster R-CNN and adds three parts. A contrastive loss pulls the RoI features of each class toward a prototype and pushes the others away. The top background box by RPN objectness in each image is labelled as unknown 2103.02603 §4.1 2103.02603 §4.2 [arxiv:2103.02603 app. F]. An energy score is compared against two fitted Weibull densities, one for known and one for unknown samples 2103.02603 §4.3. Each incremental step fine-tunes on the new labels and a balanced exemplar set 2103.02603 §4.4. Card: Towards Open World Object Detection.

Author-reported, on the four-task Pascal VOC and MS-COCO protocol. In task 2, mAP over both known sets is 38.98 for ORE against 14.91 for Faster R-CNN 2103.02603 tab. 2. Without replay, the baseline's mAP on earlier classes falls to 4.08 in that task 2103.02603 tab. 2. Going from 50 to 100 exemplars per class raises mAP over both known sets only from 38.98 to 39.24, and raises Wilderness Impact from 0.0154 to 0.0410 [arxiv:2103.02603 tab. 5]. The released code differs from the paper's equations in several places, including the loss margin, the norm, and the energy temperature. Read the card's Paper vs code section before implementing.

Benchmark for lifelong robot imitation (LIBERO)

LIBERO is a simulated benchmark, not a method. It has 130 language-conditioned manipulation tasks in four suites, with 50 teleoperated demonstrations per task 2306.03310 §4.2 2306.03310 §4.4. Card: LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. The paper trains each task with behaviour cloning and compares sequential finetuning (SeqL), experience replay (ER), EWC and PackNet. It uses three policy networks: BC-RNN, BC-T and BC-ViT 2306.03310 §4.3 2306.03310 §4.4. Each task is trained for 50 epochs with a replay buffer of 1000 windows per earlier task for ER, and PackNet keeps 25% of each task's weights 2306.03310 app. A.2 2306.03310 app. D.

With BC-T on LIBERO-Long, SeqL has forward transfer 0.54 and negative backward transfer 0.63. ER has 0.48 and 0.32, and PackNet has 0.22 and 0.08. For negative backward transfer, lower means less forgetting [arxiv:2306.03310 tab. 2]. On LIBERO-Goal, PackNet has the best AUC, 0.75, with backward transfer 0.06 2306.03310 tab. 2. On all four suites with BC-T, EWC has the lowest AUC of the four algorithms 2306.03310 tab. 2.

Author-reported, as the mean over three seeds 2306.03310 tab. 2. The authors also report that sequential finetuning has the best forward transfer on all four suites, and that basic supervised pretraining can hurt downstream learning 2306.03310 §5.2 2306.03310 fig. 5. The effect of task order is only plotted, and the text gives no numbers for it [arxiv:2306.03310 fig. 4]. The card's Pitfalls section covers how checkpoints are chosen, and should be read before comparing numbers.

Forgetting in instruction-tuned language models

This is a measurement study, not a method. The authors train BLOOMZ (1.1b to 7.1b), mT0 (1.2b and 3.7b), LLaMA-7b and Alpaca-7b on five instruction tasks in one order, then measure forgetting on MMLU, reasoning, reading comprehension and a bias benchmark 2308.08747 §3.1 2308.08747 §3.2 2308.08747 §4.1. Card: Forgetting in LLMs under continual fine-tuning.

The score is the relative drop from the initial model, averaged over the stages and then over the elements of each set 2308.08747 eq. 1. The authors report forgetting in every model they tested 2308.08747 §5.1. Author-reported: BLOOMZ-7.1b scores 18.37 on domain knowledge and 26.75 on reading comprehension, and mT0-3.7b scores 20.15 on domain knowledge 2308.08747 tab. 4. Alpaca forgets less than LLaMA on all three sets it reports 2308.08747 tab. 6. The study uses one task order and reports no variance 2308.08747 §7 2308.08747 §4.2. No official code was found 2308.08747 §4.1.

When to use which

Implementation sketch

The A-GEM and PODNet variants share a loop over the stream and a memory of past examples. They differ in how the memory enters the update. SDC and PLOP add a step that the sketch does not show. The LIBERO baselines follow the same pattern: ER replays stored windows, PackNet releases pruned weights for later tasks, and EWC adds a Fisher-weighted penalty 2306.03310 app. A.2.

# Shared loop over the task stream  [arxiv:1812.00420 alg. 2]
# and class-incremental training  [arxiv:2004.13513 §3]
def train_stream(model, memory, stream, teacher=None):
    for task in stream:
        for x, y in task.batches():
            # Loss on the new batch  [arxiv:1812.00420 alg. 2 l. 6]
            g = grad(loss(model, x, y))
            if variant == "gradient_projection":
                # Sample past examples from all earlier tasks  [§4]
                x_ref, y_ref = memory.sample()
                g_ref = grad(loss(model, x_ref, y_ref))
                # Remove the conflicting part of the update  [eq. 11]
                if dot(g, g_ref) < 0:
                    g = g - dot(g, g_ref) / dot(g_ref, g_ref) * g_ref
            else:
                # Keep pooled features near the frozen teacher  [arxiv:2004.13513 eq. 7]
                g += grad(distill(teacher, model, x))
            apply_update(model, g)
        # Store exemplars (A-GEM, PODNet) or class means (SDC) for the
        # finished task  [arxiv:1812.00420 §6] [arxiv:2004.13513 §4]
        # [arxiv:2004.00440 §4.1]. PLOP stores nothing.
        memory.add(task)
        # SDC: shift each stored class mean by the estimated drift  [eq. 10]
        # PLOP: label confident background pixels with the old model  [§3.3]

Where the variants differ: A-GEM needs the task boundary to fill memory and to form its reference batch, and it has no teacher. ER has no teacher either. It adds a same-size memory batch to each step instead of projecting the gradient 1902.10486 alg. 1. PODNet needs the teacher and a fixed number of exemplars per class, and it does not use task ids. SDC keeps one class mean per class; its E-LwF, E-EWC and E-MAS base methods keep their usual regularisers 2004.00440 §4.2. PLOP needs the frozen old model for both its distillation and its pseudo-labels, and it stores no examples. PAINT also stores no examples; its step is a weight blend with coefficient α, described for one patch in 2208.05592 §2. DER differs in the network itself: it adds one extractor per step and keeps the older ones frozen 2103.16788 §3.2. Progressive networks also store no examples. They add one column per task, freeze the earlier columns, and read them through lateral connections 1606.04671 §2. FedNTD's teacher is the global model at the start of the round, and its loss covers only the not-true classes 2106.03097 §3. GLFC distils from the previous task's model, or from the current model when the client has just detected a new task [arxiv:2203.11473 §4.3]. Its loss re-weights samples instead of projecting gradients, and clients share prototype gradients with the server 2203.11473 §4.2. ORE keeps a balanced exemplar set, and each incremental step fine-tunes on the new labels plus that set 2103.02603 §4.4. SAME stores no samples either. It keeps a running covariance of the router input, preconditions expert gradients with it, and freezes experts per task 2602.01990 §4.1 2602.01990 §4.2 2602.01990 §4.3.

Contradictions

A-GEM's Split CIFAR average is 62.3% in its own paper 1812.00420 tab. 4 and 54.9% in the ER paper's table 1902.10486 tab. 4. The memory budgets differ (65 per task in the first, one example per class in the second), so the gap may come from the protocol. The reported numbers are not comparable. The settings differ (task-labelled single pass for A-GEM, class-incremental multi-epoch training for PODNet and SDC, segmentation for PLOP, robot behaviour cloning for LIBERO, simultaneous edits for MEND, open-world detection for ORE, and instruction tuning for the language-model study), and so do the metrics. A-GEM and SDC report accuracy after the last task, PODNet reports the average over every incremental step 1812.00420 §3 2004.13513 §4 2004.00440 tab. 1, PAINT reports combined accuracy, the mean of supported and patch accuracy, against the unpatched model 2208.05592 §3 2208.05592 §5, and LIBERO reports forward transfer, negative backward transfer and AUC on success rates [arxiv:2306.03310 §5.1].

Within the SDC card, the gain is not uniform. It is clear on six-task CUB and Flowers, but on eleven-task CIFAR100 E-LwF+SDC equals E-LwF, and on ImageNet-Subset it is one point lower 2004.00440 tab. 1 2004.00440 tab. 5.

Within the LIBERO card, the authors report that task order changes results, significantly for PackNet, but the paper gives no numbers for this 2306.03310 fig. 4.

Within the progressive networks card, the Labyrinth mean and median point in different directions: 491% against 115% 1606.04671 tab. 1. The comparison with finetuning depends on which of the two is used.

Within the SAME card, the text and the TriGap table disagree. The text calls the FloodNetVQA gain "particularly clear", but MoE-LoRA scores higher there, 90.41 against 81.09, and SAME trails MoE-LoRA on CLEVR, 54.50 against 67.00 2602.01990 §5.2 2602.01990 tab. 1. The CoIN baselines are taken from the original publications, while the TriGap baselines are reproduced under the same setup 2602.01990 tab. 2 2602.01990 tab. 1. Compare across these tables with care.

Within the DER card, the headline numbers use the pruned model, while the ablation 2103.16788 tab. 4 uses the un-pruned one. Compare like with like 2103.16788 tab. 1.

Within the ORE card, the released code and the paper use different loss margins, norms and energy temperatures, and the Weibull fits use pseudo-labelled boxes as unknowns 2103.02603 §4.3 2103.02603 app. D. The card does not check whether these differences change the reported numbers.

Within the GLFC card, the ablation's stated average for the variant without the compensation loss is 60.1, but its ten values average 64.7. The card uses the stated value, so do not rely on that row's average 2203.11473 tab. 1. The GLFC and FedNTD numbers come from different federated settings, one class-incremental with an exemplar memory and one a fixed-class classification task, so they are not comparable 2203.11473 §5.1 2106.03097 tab. 1.

Within the language-model study, the forgetting score is an average over stages, so it cannot be recomputed from the start and end scores in the table. The bias column does not match its endpoints either 2308.08747 eq. 1 2308.08747 tab. 4 2308.08747 tab. 5. Compare its values only with each other.

The review reports that benchmarks and metrics lack consensus across the field 1802.07569 §3.5 1802.07569 §5. Its qualitative rankings are benchmark-specific: EWC and PathNet change order between the data-permutation and multimodal experiments 1802.07569 §3.5. Do not use them as numbers for the methods above.

Papers

• Efficient Lifelong Learning with A-GEM: single-pass gradient projection against an episodic memory, with an averaged constraint and a closed-form update.
• On Tiny Episodic Memories in Continual Learning: experience replay with a tiny memory, a stacked batch per step and four memory-writing rules; no gradient projection.
• Progressive Neural Networks: one column per RL task, frozen earlier columns and lateral connections with adapters; compared with finetuning on Pong, Atari and 3D mazes.
• PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning: class-incremental distillation with pooled feature statistics and a multi-proxy cosine classifier.
• Semantic Drift Compensation for Class-Incremental Learning: class means stored per task and shifted by a Gaussian-weighted estimate of the embedding drift, with no images kept.
• PLOP: Learning without Forgetting for Continual Semantic Segmentation: pooled multi-scale feature distillation and entropy-thresholded pseudo-labels for background pixels, with no stored images.
• DER: Dynamically Expandable Representation for Class Incremental Learning: a new extractor per step, frozen old extractors, an auxiliary new-vs-old loss, and a balanced classifier retrain; the pruned variant is not in the released code.
• Preservation of the Global Knowledge by Not-True Distillation in Federated Learning: distillation over the not-true classes against the round-start global model, with no extra communication.
• Federated Class-Incremental Learning: GLFC, a class-aware gradient-compensated loss and relation distillation per client, with prototype gradients rebuilt into samples at a proxy server.
• Continual Lifelong Learning with Neural Networks: A Review: a review of catastrophic forgetting that groups regularisation, dynamic-architecture and replay methods into families; no new experiments.
• A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges: names lifelong ABSA with changing label sets as an open challenge; no method or experiments on continual learning.
• MEND: Fast Model Editing at Scale: a small editor network turns one fine-tuning gradient into a low-rank weight edit; evaluated on simultaneous edits, not on a continual stream.
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning: a simulated benchmark of 130 robot tasks in four suites, with ER, EWC and PackNet compared against sequential finetuning.
• Patching open-vocabulary models by interpolating weights: one fine-tune per patch task, then a linear blend with the zero-shot weights; sequential and joint multi-task patching are compared.
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning: rehearsal-free LoRA experts with router updates in leading covariance directions, covariance-scaled expert gradients and per-task freezing of low-score experts.
• iSDF: Real-Time Neural Signed Distance Fields for Robot Perception: online training of a neural SDF on a stream of posed depth frames; no forgetting measure, so not a variant here.
• Towards Open World Object Detection: ORE, a Faster R-CNN variant with prototype clustering, pseudo-labelled unknowns, an energy-based unknown test and exemplar replay.
• Forgetting in LLMs under continual fine-tuning: a measurement of forgetting when five instruction tasks are tuned in sequence; no method is proposed.