Catastrophic forgetting

Overview

A network is trained on a task, then trained on another, and its scores on the first task can fall sharply. This is catastrophic forgetting. Five settings are covered. Three study one model. In the first, a model is tuned on a fixed sequence of instruction tasks, earlier task data is not available at later stages, and the checkpoint after each stage is kept 2308.08747 §3. In the second, each example is seen once in the stream and the learner may keep a small episodic memory of past examples 1902.10486 §3.1. In the third, a frozen vision-language model is adapted with a few learned prompt vectors, and the concern is that the prompts fit the new training classes and lose accuracy on classes they never saw 2307.06948 §1 2307.06948 §3. The other two extend the setting. In federated class-incremental learning, clients receive new classes at different times and each keeps a small exemplar memory. The paper states that imbalance between new data and memory causes local forgetting, and that imbalance across clients causes global forgetting 2203.11473 §1. In knowledge injection, a 7B model is either trained on raw documents or given them at question time. The authors suggest forgetting explains part of the weak result for training, but they do not measure it 2312.05934 §5.

The cards study the same failure in different forms. In the first setting the training signal comes only from the new data, so nothing in the loss protects the old behaviour 2308.08747 §3. The second card tests one way to add protection: replaying a few stored examples during training 1902.10486 §4. The third card adds protection by penalising the prompted model for drifting from the frozen one 2307.06948 §3.2. The federated card adds two losses on each client, one that reweights the classes and one that keeps the previous model's class relations 2203.11473 §4.1. Methods that reduce the drop, such as regularisation, growing networks and replay, are compared on [continual-learning].

Variants

Forgetting is a property of a training setting rather than a method, so this page covers ways to measure it and four ways of training against it.

Measuring it as a relative drop from the start

For each element of an evaluation set, the metric takes the relative fall from its starting score to its score after each stage. It averages those falls over stages, then over the elements of the set 2308.08747 §3.3 2308.08747 eq. 1. Card: Empirical Study of Catastrophic Forgetting in LLMs.

Because the average runs over stages, the metric is not the same as the drop between the first and last checkpoint. The paper's main table gives only the starting score and the final score for each group, not the per-stage scores 2308.08747 tab. 4, so the averaged value cannot be recomputed from the published table.

Measuring it as a drop from the best earlier accuracy

The second card defines forgetting per old task. For task j at the final stage, it takes the best accuracy that task reached at any earlier checkpoint, subtracts the current accuracy, and averages over the old tasks 1902.10486 §3.2 1902.10486 eq. 2 1902.10486 eq. 3. The value is a difference of accuracies, not a relative drop, and it is taken against the best earlier checkpoint rather than the starting model. The forgetting table reports values from 0.03 to 0.29 across methods and benchmarks 1902.10486 tab. 1. Card: On Tiny Episodic Memories in Continual Learning.

Reporting accuracy without a forgetting term

The third card does not report a forgetting value. Its title speaks of forgetting, but its metrics are accuracy on the base classes, on unseen (novel) classes, and on other datasets 2307.06948 tab. 1 2307.06948 tab. 3. Compare its numbers with these two measures only by reading the accuracy columns.

The federated card reports average incremental top-1 accuracy over the tasks 2203.11473 §5.1. That value mixes learning of new classes with retention of old ones, so it is not a forgetting measure. The knowledge-injection card reports accuracy and a relative gain over the base model 2312.05934 eq. 5, and it does not measure forgetting 2312.05934 §5.

Replaying a tiny memory

Experience replay (ER) keeps an episodic memory and trains each step on the current mini-batch plus a same-size batch drawn from that memory. The memory is written after the gradient step 1902.10486 alg. 1. The paper tests four writing rules: reservoir sampling, ring buffer, k-means and mean of features 1902.10486 §4. A hybrid that starts with reservoir sampling and then switches to the ring-buffer rule is described in 1902.10486 §5.4. The paper's memory budget is one example per class, the setting in which it reports its main comparison 1902.10486 tab. 3–1902.10486 tab. 6.

The federated card also keeps an exemplar memory on each client, with a budget of 2,000 samples in its main setting 2203.11473 §5.1. Its memory study on CIFAR-100 with ten tasks rises from 55.6 average accuracy at 500 samples to 66.9 at 2,000 2203.11473 tab. 9. Its setting differs from the tiny-memory study above, so these numbers are not comparable with it.

Rebalancing the loss and distilling from the old model

The federated card trains each client with a weighted sum of two terms 2203.11473 eq. 6. The first is a classification loss over all seen classes. Each sample's loss is weighted by its gradient magnitude divided by the mean for its own group, new or old 2203.11473 eq. 2 [arxiv:2203.11473 eq. 3] 2203.11473 eq. 4. The second matches the current predictions to the previous model's probabilities over the old classes, using a KL divergence 2203.11473 eq. 5. The weights are 1 and 0 on the first task, and 0.5 each afterwards 2203.11473 §4.1. On CIFAR-100 with ten tasks, the paper's ablation gives 65.8 without the distillation term against 66.9 for the full method 2203.11473 tab. 1, averaged over three seeds [arxiv:2203.11473 §5.1]. The row without the compensation term does not add up in the table, so it is not quoted here 2203.11473 tab. 1.

Mixing general data into the task stream

The authors of the first card mix 10,000 general instruction samples into the task stream for two 7B models. They report less forgetting on the one domain-knowledge split they show, for LLaMA-7b, but not the mixing ratio 2308.08747 §5.4. This is a replay-style check, not a method comparison: it was run on one split.

Keeping the prompted model close to the frozen model

The third card adapts CLIP by learning prompt tokens for both encoders while the encoders stay frozen 2307.06948 §3.1. Training adds three terms to the usual classification loss, all computed on the same mini-batch: an L1 penalty between prompted and frozen image features, an L1 penalty between prompted and frozen text features, and a KL term that pulls the prompted class distribution toward the frozen one 2307.06948 §3.2.1 [arxiv:2307.06948 eq. 2] 2307.06948 eq. 3. The frozen text target for each class is an average over a set of text templates, used only during training 2307.06948 §3.2.3. The prompts are also averaged across training epochs with Gaussian-shaped weights, and the averaged prompts are used at inference 2307.06948 §3.2.2 2307.06948 eq. 6. Card: Self-regulating Prompts: Foundational Model Adaptation without Forgetting.

The paper attributes the gain over its prompt-tuning baseline mainly to the agreement terms. On the base-to-novel ablation, adding them raises the novel accuracy from 71.79 to 75.38 and the harmonic mean from 77.51 to 79.55 2307.06948 tab. 2. The epoch averaging and the template average add smaller steps 2307.06948 tab. 2.

When to use which

Implementation sketch

Our own pseudocode for the measurement, then the replay step. The paper's learning settings in the first card are inherited from earlier work, not tuned: Adam, learning rate 2e-5, 3 epochs, maximum length 512, and no stated warm-up or weight decay 2308.08747 §4.2.

# Tune on each task in order, scoring the starting model and each stage  [§3.3]
# Input:  starting model, ordered tasks, evaluation sets
# Output: forgetting value per evaluation set  [eq. 1]

def forgetting_by_set(model, tasks, eval_sets):
    # Starting score R_o for every element of every set  [§3.3]
    start = evaluate(model, eval_sets)
    stage_scores = []
    for task in tasks:
        # Train on this task only; earlier task data is not replayed  [§3.1]
        model = fine_tune(model, task)
        # Score the checkpoint after this stage  [§3.3]
        stage_scores.append(evaluate(model, eval_sets))
    # Relative drop per element at each stage, averaged over stages  [eq. 1]
    # then averaged over the elements of each set  [eq. 1]
    return {name: mean_relative_drop(start, stage_scores, name)
            for name in eval_sets}

# One replay step with a tiny episodic memory  [alg. 1]
# Input:  parameters theta, memory, current mini-batch, learning rate

def replay_step(theta, memory, batch, lr):
    # Draw as many stored examples as the current batch holds  [alg. 1]
    memory_batch = sample(memory, len(batch))
    # One SGD step on the stacked batches  [alg. 1]
    theta = sgd_step(theta, batch + memory_batch, lr)
    # Write the current batch to memory after the step  [alg. 1; §4]
    # The writing rule (reservoir, ring buffer, k-means, MoF) is a choice  [§4]
    memory = write(memory, batch)
    return theta, memory

# Classification loss plus agreement with the frozen model  [eq. 1] [eq. 4] [eq. 5]
# Input:  prompted features and logits, frozen features and logits,
#         weights lam_img and lam_txt  [§4.1]

def agreement_objective(f_p, f_frozen, g_p, g_frozen, logits, frozen_logits):
    # Task loss on prompted image-text similarities  [eq. 1]
    loss_ce = cross_entropy(logits, labels)
    # L1 distance between prompted and frozen features, per modality  [eq. 2]
    loss_img = lam_img * l1(f_p, f_frozen)
    loss_txt = lam_txt * l1(g_p, g_frozen)
    # Divergence between the class distributions  [eq. 3]
    loss_kl = kl(softmax(logits), softmax(frozen_logits))
    # Sum of the task loss and the agreement terms  [eq. 5]
    return loss_ce + loss_img + loss_txt + loss_kl

Contradictions

The first two cards define forgetting differently. The first averages the relative drop from the starting model over stages 2308.08747 eq. 1. The second takes the drop from the best earlier accuracy at the final stage, averaged over old tasks 1902.10486 eq. 2 1902.10486 eq. 3. Their numbers are not on the same scale, and this page does not resolve which one is the better measure. The third card uses neither definition; its gains are reported as accuracy on base and novel classes 2307.06948 tab. 1.

Within the first card, the scale trend is not uniform. The authors report that larger BLOOMZ models forget more 2308.08747 §5.2, but on the reasoning set the value is 6.73 at 1.1b and 6.48 at 1.7b, so the trend is not monotone there 2308.08747 tab. 4.

Replay evidence comes from different settings. The second card reports that replaying a tiny memory beats the baselines on four classification benchmarks 1902.10486 tab. 3–1902.10486 tab. 6. The first card reports that mixing general data helps on one split of two LLMs 2308.08747 §5.4. Neither card tests the other's setting, so no direct disagreement is recorded. The federated card's memory study is on image classification with a larger per-client memory 2203.11473 tab. 9, which is also a different setting.

The third card's claims do not all point the same way. Its base-to-novel average beats MaPLe 2307.06948 tab. 1, but its cross-dataset average is lower than MaPLe's 2307.06948 tab. 3. The paper's text gives a novel-accuracy gain from the agreement terms of 3.95 points [arxiv:2307.06948 §4.2], while its ablation table gives 3.59 2307.06948 tab. 2. This page uses the table.

The third card's method description and its released code differ in several places. The paper describes the epoch weights as samples from a Gaussian, while the code uses deterministic weights, and the code's KL term runs in the other direction from eq. 3. The paper fixes the learning rate, and the configs use a cosine schedule. The card's Paper vs code section lists these, and this page follows the paper's equations 2307.06948 §3.2.2 2307.06948 eq. 3.

The federated card's distillation term is a KL divergence in the paper 2203.11473 eq. 5, while the card's Paper vs code section records a binary cross-entropy in the official code. This page follows the paper. The knowledge-injection card records that fine-tuning hurt one model on current events 2312.05934 tab. 2, and its authors attribute this only partly to forgetting, which they did not measure 2312.05934 §5.

Papers

• Forgetting in LLMs under continual instruction tuning: measures forgetting of general knowledge, reasoning, reading and bias in five models after five sequential instruction tasks.
• On Tiny Episodic Memories in Continual Learning: experience replay with about one stored example per class, four memory-writing rules, and a comparison with fine-tuning, EWC, A-GEM and MER on four split benchmarks.
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting: prompt tuning of CLIP with feature and logit agreement to the frozen model, epoch-level prompt averaging, and multi-template text targets, evaluated on base-to-novel, cross-dataset and domain generalisation.
• Federated Class-Incremental Learning: per-client exemplar memory, class-aware loss weighting and relation distillation, and a proxy server that rebuilds prototype samples from shared gradients to choose the best earlier global model.
• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: continued training on Wikipedia text against retrieval-augmented generation for three 7B models, on MMLU subsets and new current-events questions.