---
kind: concept
slug: catastrophic-forgetting
title: Catastrophic forgetting
summary: >-
  A trained network loses earlier abilities when it is trained on new data. This page covers the settings studied, two ways of measuring the loss that do not give comparable numbers, replay with a small memory, distillation toward a previous model, and keeping a frozen model's outputs close during prompt tuning. Other methods are on continual-learning.
tags: [continual-learning, large-language-models, evaluation, instruction-tuning, episodic-memory, prompt-tuning, vision-language-models, federated-learning, retrieval-augmented-generation]
papers: ["2308.08747", "1902.10486", "2307.06948", "2203.11473", "2312.05934"]
related: [continual-learning, instruction-tuning, experience-replay, benchmark-validity, consistency-regularization, soft-prompt-tuning, zero-shot-learning, federated-learning, knowledge-distillation, retrieval-augmented-generation]
status: draft
updated_at: 2026-11-01
---

## Overview

A network is trained on a task, then trained on another, and its scores on the
first task can fall sharply. This is catastrophic forgetting. Five settings are
covered. Three study one model. In the first, a model is tuned on a fixed
sequence of instruction tasks, earlier task data is not available at later
stages, and the checkpoint after each stage is kept [arxiv:2308.08747 §3]. In
the second, each example is seen once in the stream and the learner may keep a
small episodic memory of past examples [arxiv:1902.10486 §3.1]. In the third, a
frozen vision-language model is adapted with a few learned prompt vectors, and
the concern is that the prompts fit the new training classes and lose accuracy
on classes they never saw [arxiv:2307.06948 §1] [arxiv:2307.06948 §3]. The
other two extend the setting. In federated class-incremental learning, clients
receive new classes at different times and each keeps a small exemplar memory.
The paper states that imbalance between new data and memory causes local
forgetting, and that imbalance across clients causes global forgetting
[arxiv:2203.11473 §1]. In knowledge injection, a 7B model is either trained on
raw documents or given them at question time. The authors suggest forgetting
explains part of the weak result for training, but they do not measure it
[arxiv:2312.05934 §5].

The cards study the same failure in different forms. In the first setting the
training signal comes only from the new data, so nothing in the loss protects the
old behaviour [arxiv:2308.08747 §3]. The second card tests one way to add
protection: replaying a few stored examples during training
[arxiv:1902.10486 §4]. The third card adds protection by penalising the prompted
model for drifting from the frozen one [arxiv:2307.06948 §3.2]. The federated
card adds two losses on each client, one that reweights the classes and one that
keeps the previous model's class relations [arxiv:2203.11473 §4.1]. Methods that
reduce the drop, such as regularisation, growing networks and replay, are
compared on [continual-learning].

## Variants

Forgetting is a property of a training setting rather than a method, so this
page covers ways to measure it and four ways of training against it.

### Measuring it as a relative drop from the start

For each element of an evaluation set, the metric takes the relative fall from
its starting score to its score after each stage. It averages those falls over
stages, then over the elements of the set [arxiv:2308.08747 §3.3]
[arxiv:2308.08747 eq. 1]. Card:
[Empirical Study of Catastrophic Forgetting in LLMs](/papers/2308.08747).

Because the average runs over stages, the metric is not the same as the drop
between the first and last checkpoint. The paper's main table gives only the
starting score and the final score for each group, not the per-stage scores
[arxiv:2308.08747 tab. 4], so the averaged value cannot be recomputed from the
published table.

### Measuring it as a drop from the best earlier accuracy

The second card defines forgetting per old task. For task j at the final stage,
it takes the best accuracy that task reached at any earlier checkpoint, subtracts
the current accuracy, and averages over the old tasks [arxiv:1902.10486 §3.2]
[arxiv:1902.10486 eq. 2] [arxiv:1902.10486 eq. 3]. The value is a difference of
accuracies, not a relative drop, and it is taken against the best earlier
checkpoint rather than the starting model. The forgetting table reports values from
0.03 to 0.29 across methods and benchmarks [arxiv:1902.10486 tab. 1]. Card:
[On Tiny Episodic Memories in Continual Learning](/papers/1902.10486).

### Reporting accuracy without a forgetting term

The third card does not report a forgetting value. Its title speaks of
forgetting, but its metrics are accuracy on the base classes, on unseen (novel)
classes, and on other datasets [arxiv:2307.06948 tab. 1] [arxiv:2307.06948 tab. 3].
Compare its numbers with these two measures only by reading the accuracy columns.

The federated card reports average incremental top-1 accuracy over the tasks
[arxiv:2203.11473 §5.1]. That value mixes learning of new classes with retention
of old ones, so it is not a forgetting measure. The knowledge-injection card
reports accuracy and a relative gain over the base model [arxiv:2312.05934 eq. 5],
and it does not measure forgetting [arxiv:2312.05934 §5].

### Replaying a tiny memory

Experience replay (ER) keeps an episodic memory and trains each step on the
current mini-batch plus a same-size batch drawn from that memory. The memory is
written after the gradient step [arxiv:1902.10486 alg. 1]. The paper tests four
writing rules: reservoir sampling, ring buffer, k-means and mean of features
[arxiv:1902.10486 §4]. A hybrid that starts with reservoir sampling and then
switches to the ring-buffer rule is described in [arxiv:1902.10486 §5.4]. The
paper's memory budget is one example per class, the setting in which it reports
its main comparison [arxiv:1902.10486 tab. 3]–[arxiv:1902.10486 tab. 6].

The federated card also keeps an exemplar memory on each client, with a budget of
2,000 samples in its main setting [arxiv:2203.11473 §5.1]. Its memory study on
CIFAR-100 with ten tasks rises from 55.6 average accuracy at 500 samples to 66.9
at 2,000 [arxiv:2203.11473 tab. 9]. Its setting differs from the tiny-memory study
above, so these numbers are not comparable with it.

### Rebalancing the loss and distilling from the old model

The federated card trains each client with a weighted sum of two terms
[arxiv:2203.11473 eq. 6]. The first is a classification loss over all seen
classes. Each sample's loss is weighted by its gradient magnitude divided by the
mean for its own group, new or old [arxiv:2203.11473 eq. 2] [arxiv:2203.11473
eq. 3] [arxiv:2203.11473 eq. 4]. The second matches the current predictions to
the previous model's probabilities over the old classes, using a KL divergence
[arxiv:2203.11473 eq. 5]. The weights are 1 and 0 on the first task, and 0.5 each
afterwards [arxiv:2203.11473 §4.1]. On CIFAR-100 with ten tasks, the paper's
ablation gives 65.8 without the distillation term against 66.9 for the full
method [arxiv:2203.11473 tab. 1], averaged over three seeds [arxiv:2203.11473
§5.1]. The row without the compensation term does not add up in the table, so it
is not quoted here [arxiv:2203.11473 tab. 1].

### Mixing general data into the task stream

The authors of the first card mix 10,000 general instruction samples into the
task stream for two 7B models. They report less forgetting on the one
domain-knowledge split they show, for LLaMA-7b, but not the mixing ratio
[arxiv:2308.08747 §5.4]. This is a replay-style check, not a method comparison:
it was run on one split.

### Keeping the prompted model close to the frozen model

The third card adapts CLIP by learning prompt tokens for both encoders while
the encoders stay frozen [arxiv:2307.06948 §3.1]. Training adds three terms to
the usual classification loss, all computed on the same mini-batch: an L1
penalty between prompted and frozen image features, an L1 penalty between
prompted and frozen text features, and a KL term that pulls the prompted class
distribution toward the frozen one [arxiv:2307.06948 §3.2.1] [arxiv:2307.06948
eq. 2] [arxiv:2307.06948 eq. 3]. The frozen text target for each class is an
average over a set of text templates, used only during training
[arxiv:2307.06948 §3.2.3]. The prompts are also averaged across training epochs
with Gaussian-shaped weights, and the averaged prompts are used at inference
[arxiv:2307.06948 §3.2.2] [arxiv:2307.06948 eq. 6]. Card:
[Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948).

The paper attributes the gain over its prompt-tuning baseline mainly to the
agreement terms. On the base-to-novel ablation, adding them raises the novel
accuracy from 71.79 to 75.38 and the harmonic mean from 77.51 to 79.55
[arxiv:2307.06948 tab. 2]. The epoch averaging and the template average add
smaller steps [arxiv:2307.06948 tab. 2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Compare checkpoints of a tuned model against its starting point on general benchmarks | Relative-drop metric, averaged over stages | Gives one value per benchmark group. It is only comparable between runs that share the same task order and evaluation settings [arxiv:2308.08747 §3.3] [arxiv:2308.08747 §7] |
| Only the starting and final scores are available | Report the endpoint drop separately, and label it as such | The paper's averaged value differs from the endpoint change. For mT0-3.7b on domain knowledge the endpoint change from the table is about 35%, against a reported value of 20.15 [arxiv:2308.08747 tab. 4] |
| Deciding between two model families or sizes from one run | Treat the difference as a hypothesis and rerun with other task orders | The study uses one task order and reports no seeds or variance [arxiv:2308.08747 §7] [arxiv:2308.08747 §4.2] |
| Reducing forgetting in an instruction-tuned model when general instruction data is available | Mix general instruction data into the training stream, then check each benchmark group | The one reported split improved for LLaMA-7b; the mixing ratio is not stated [arxiv:2308.08747 §5.4] |
| Choosing a starting model for sequential tuning | Look at the absolute scores as well as the forgetting value | The authors attribute the larger forgetting of BLOOMZ partly to its higher starting scores [arxiv:2308.08747 §5.4] [arxiv:2308.08747 tab. 6] |
| Classification stream, each example seen once, storage for a few examples per class | Experience replay with a small memory. Use a class-balanced writer (ring buffer, k-means or mean of features) when the memory is tiny, and reservoir sampling when it is larger | ER with one example per class beats fine-tuning, EWC, A-GEM and MER on four split benchmarks, and the class-balanced writers do best at the smallest memory. These are author-reported [arxiv:1902.10486 tab. 3]–[arxiv:1902.10486 tab. 6] [arxiv:1902.10486 §5.4]. The released code runs ER with reservoir writing only; see [the card's Paper vs code section](/papers/1902.10486) |
| Replay must fit a tight compute budget | Replay with a memory batch of the same size as the current batch, rather than a gradient-constraint method | The paper's timings put ER with the ring buffer at 116 s against 230 s for A-GEM on Split CIFAR, and 87 s for fine-tuning. The hardware is not stated [arxiv:1902.10486 tab. 2] |
| Clients in a federated setting receive new classes over time and keep a small exemplar memory | Class-aware loss weighting and distillation from the previous global model, with the best earlier global model kept for the next round | Author-reported 4.4% to 15.1% higher average accuracy than adapted baselines on three image benchmarks [arxiv:2203.11473 tab. 1] [arxiv:2203.11473 tab. 2]. The privacy argument rests on a noise step that the official code does not implement, so check the card's Paper vs code section before relying on it [arxiv:2203.11473 §4.2] |
| Adding facts from a document collection to a 7B model | Retrieval at question time. Training on the raw documents gave little or no gain in the paper | Base + RAG scores 0.875 on current events for Mistral-7B against 0.504 for training on the raw text, and RAG wins on 28 of 30 MMLU rows against that training [arxiv:2312.05934 tab. 1] [arxiv:2312.05934 tab. 2]. Training on paraphrases helped all three models, at a GPT-4 generation cost [arxiv:2312.05934 §4.4] [arxiv:2312.05934 tab. 2]. The current-events gap is an upper bound for RAG, since each question comes from a chunk in the retrieval corpus [arxiv:2312.05934 §4.3] |
| Adapting a frozen CLIP-style model with prompts, and accuracy on unseen classes matters | Agreement losses to the frozen features and logits, with epoch averaging of the prompts | Author-reported base-to-novel harmonic mean of 79.97, against 78.55 for MaPLe and 71.70 for zero-shot CLIP [arxiv:2307.06948 tab. 1]. The agreement terms supply most of the gain over the prompt-tuning baseline [arxiv:2307.06948 tab. 2] |
| Adapting prompts and the target is transfer to other datasets from an ImageNet-trained model | Compare with MaPLe before choosing this method | The paper's own table shows 65.81 averaged over ten targets against 66.30 for MaPLe [arxiv:2307.06948 tab. 3]. Domain generalisation shows a small gain, 60.65 against 60.27 [arxiv:2307.06948 tab. 4] |
| Few trainable parameters are required for prompt adaptation | Prompt-based methods with the agreement terms | The paper reports 46K tunable parameters against 3.55M for MaPLe [arxiv:2307.06948 §4.3] |
| Training budget matters for prompt adaptation | Expect about 10% more training compute than the prompt baseline; inference cost is unchanged | Training GFLOPs are 179.6 against 162.8 for IVLP, and inference GFLOPs match IVLP [arxiv:2307.06948 tab. 7] |
| Comparing forgetting numbers across papers | Check the definition first and convert nothing | The relative drop from the start and the drop from the best earlier accuracy are different quantities. Numbers from one definition cannot be compared with the other [arxiv:2308.08747 eq. 1] [arxiv:1902.10486 eq. 2] |

## Implementation sketch

Our own pseudocode for the measurement, then the replay step. The paper's
learning settings in the first card are inherited from earlier work, not tuned:
Adam, learning rate 2e-5, 3 epochs, maximum length 512, and no stated warm-up
or weight decay [arxiv:2308.08747 §4.2].

```pseudocode
# Tune on each task in order, scoring the starting model and each stage  [§3.3]
# Input:  starting model, ordered tasks, evaluation sets
# Output: forgetting value per evaluation set  [eq. 1]

def forgetting_by_set(model, tasks, eval_sets):
    # Starting score R_o for every element of every set  [§3.3]
    start = evaluate(model, eval_sets)
    stage_scores = []
    for task in tasks:
        # Train on this task only; earlier task data is not replayed  [§3.1]
        model = fine_tune(model, task)
        # Score the checkpoint after this stage  [§3.3]
        stage_scores.append(evaluate(model, eval_sets))
    # Relative drop per element at each stage, averaged over stages  [eq. 1]
    # then averaged over the elements of each set  [eq. 1]
    return {name: mean_relative_drop(start, stage_scores, name)
            for name in eval_sets}
```

```pseudocode
# One replay step with a tiny episodic memory  [alg. 1]
# Input:  parameters theta, memory, current mini-batch, learning rate

def replay_step(theta, memory, batch, lr):
    # Draw as many stored examples as the current batch holds  [alg. 1]
    memory_batch = sample(memory, len(batch))
    # One SGD step on the stacked batches  [alg. 1]
    theta = sgd_step(theta, batch + memory_batch, lr)
    # Write the current batch to memory after the step  [alg. 1; §4]
    # The writing rule (reservoir, ring buffer, k-means, MoF) is a choice  [§4]
    memory = write(memory, batch)
    return theta, memory
```

```pseudocode
# Classification loss plus agreement with the frozen model  [eq. 1] [eq. 4] [eq. 5]
# Input:  prompted features and logits, frozen features and logits,
#         weights lam_img and lam_txt  [§4.1]

def agreement_objective(f_p, f_frozen, g_p, g_frozen, logits, frozen_logits):
    # Task loss on prompted image-text similarities  [eq. 1]
    loss_ce = cross_entropy(logits, labels)
    # L1 distance between prompted and frozen features, per modality  [eq. 2]
    loss_img = lam_img * l1(f_p, f_frozen)
    loss_txt = lam_txt * l1(g_p, g_frozen)
    # Divergence between the class distributions  [eq. 3]
    loss_kl = kl(softmax(logits), softmax(frozen_logits))
    # Sum of the task loss and the agreement terms  [eq. 5]
    return loss_ce + loss_img + loss_txt + loss_kl
```

## Contradictions

The first two cards define forgetting differently. The first averages the relative
drop from the starting model over stages [arxiv:2308.08747 eq. 1]. The second
takes the drop from the best earlier accuracy at the final stage, averaged over
old tasks [arxiv:1902.10486 eq. 2] [arxiv:1902.10486 eq. 3]. Their numbers are
not on the same scale, and this page does not resolve which one is the better
measure. The third card uses neither definition; its gains are reported as
accuracy on base and novel classes [arxiv:2307.06948 tab. 1].

Within the first card, the scale trend is not uniform. The authors report that
larger BLOOMZ models forget more [arxiv:2308.08747 §5.2], but on the reasoning
set the value is 6.73 at 1.1b and 6.48 at 1.7b, so the trend is not monotone
there [arxiv:2308.08747 tab. 4].

Replay evidence comes from different settings. The second card reports that
replaying a tiny memory beats the baselines on four classification benchmarks
[arxiv:1902.10486 tab. 3]–[arxiv:1902.10486 tab. 6]. The first card reports that
mixing general data helps on one split of two LLMs [arxiv:2308.08747 §5.4].
Neither card tests the other's setting, so no direct disagreement is recorded.
The federated card's memory study is on image classification with a larger
per-client memory [arxiv:2203.11473 tab. 9], which is also a different setting.

The third card's claims do not all point the same way. Its base-to-novel
average beats MaPLe [arxiv:2307.06948 tab. 1], but its cross-dataset average
is lower than MaPLe's [arxiv:2307.06948 tab. 3]. The paper's text gives a
novel-accuracy gain from the agreement terms of 3.95 points [arxiv:2307.06948
§4.2], while its ablation table gives 3.59 [arxiv:2307.06948 tab. 2]. This
page uses the table.

The third card's method description and its released code differ in several
places. The paper describes the epoch weights as samples from a Gaussian, while
the code uses deterministic weights, and the code's KL term runs in the other
direction from eq. 3. The paper fixes the learning rate, and the configs use a
cosine schedule. The card's Paper vs code section lists these, and this page
follows the paper's equations [arxiv:2307.06948 §3.2.2] [arxiv:2307.06948 eq. 3].

The federated card's distillation term is a KL divergence in the paper
[arxiv:2203.11473 eq. 5], while the card's Paper vs code section records a
binary cross-entropy in the official code. This page follows the paper. The
knowledge-injection card records that fine-tuning hurt one model on current
events [arxiv:2312.05934 tab. 2], and its authors attribute this only partly to
forgetting, which they did not measure [arxiv:2312.05934 §5].

## Papers

- [Forgetting in LLMs under continual instruction tuning](/papers/2308.08747): measures forgetting of general knowledge, reasoning, reading and bias in five models after five sequential instruction tasks.
- [On Tiny Episodic Memories in Continual Learning](/papers/1902.10486): experience replay with about one stored example per class, four memory-writing rules, and a comparison with fine-tuning, EWC, A-GEM and MER on four split benchmarks.
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948): prompt tuning of CLIP with feature and logit agreement to the frozen model, epoch-level prompt averaging, and multi-template text targets, evaluated on base-to-novel, cross-dataset and domain generalisation.
- [Federated Class-Incremental Learning](/papers/2203.11473): per-client exemplar memory, class-aware loss weighting and relation distillation, and a proxy server that rebuilds prototype samples from shared gradients to choose the best earlier global model.
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](/papers/2312.05934): continued training on Wikipedia text against retrieval-augmented generation for three 7B models, on MMLU subsets and new current-events questions.