An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

In plain words

When you fine-tune a language model on a new kind of task, it can lose some of what it knew before. This paper asks whether that happens to general knowledge when a large model is tuned on several instruction tasks one after another.

The authors train several open models on five instruction tasks in a fixed order. After each stage they test the model on general benchmarks: subject knowledge, commonsense and math reasoning, reading comprehension, and a stereotype-bias test. They compare how far each score falls from the original model's score. The forgetting is clear in every model they tried. A model that was already instruction-tuned forgets less than its base version. The differences between model families and sizes are real but come from one task order and a few models, so treat them as a starting point.

TL;DR

• Sequential instruction tuning on five tasks lowers general-knowledge, reasoning and reading scores in every tested model (BLOOMZ 1.1b-7.1b, mT0 1.2b and 3.7b, LLaMA-7b, Alpaca-7b) §5.1 tab. 4 tab. 6.
• The authors report that larger BLOOMZ models forget more, that BLOOMZ forgets less than mT0 at 3b to 3.7b, and that Alpaca forgets less than LLaMA §5.2 §5.3 §5.4.
• The metric is a relative drop averaged over stages, so it is not the same as an endpoint score drop eq. 1.
• All numbers are author-reported from one fixed task order, and no official code was found.

Problem

Instruction tuning on a narrow task can overwrite general knowledge that the pretrained model had. The original training data is often unavailable, so later tuning cannot rely on it. Earlier work measured forgetting on downstream tasks §2 or with probing on encoder-only models §2. The authors say the forgetting of general knowledge in generative LLMs during continual instruction tuning had not been measured §2. They study this for LLMs from 1b to 7b parameters §1 §4.1.

Method

Setup. A model M_0 is trained on a sequence of tasks T^1 to T^N, where N is the number of tasks §3. Each task has its own data of inputs and labels, and earlier data is not available during later stages. After training on task m, the checkpoint is M_m §3. The sequence has N = 5 tasks §3.1.

Continual tasks. The five tasks follow Scialom et al. (2022), chosen to differ from the training and evaluation tasks of BLOOMZ and mT0 §3.1:

1. Text simplification (Simp): rewrite a text in simpler words §3.1.
2. Empathetic dialogue generation (Emdg): give a reason for a conversation under a stated emotion §3.1.
3. Inquisitive question generation (InqQG): write a question that a long-form answer responds to §3.1.
4. Explanation generation (Exp): explain why a premise, hypothesis or label holds §3.1.
5. Headline generation with constraints (HGen): write a headline that contains given keywords at a set position §3.1.

Each training sample is a general prompt prefix followed by a task-specific prompt from Scialom et al., and the prompts are reproduced in app. A §3.1. The paper trains on one task order only: Simp, Emdg, InqQG, Exp, HGen §3.1.

Evaluation sets. Forgetting is measured on four groups of general benchmarks tab. 2 §3.2:

• Domain knowledge (DK): MMLU, split into STEM, Social, Human and Other §3.2.
• Reasoning (Rs): BoolQ, PIQA, Winogrande, Hellaswag, MathQA, Mutual §3.2.
• Reading comprehension (RC): RACE-high and RACE-middle §3.2.
• Bias: CrowS-Pairs, across nine bias categories §3.2 tab. 2.

Scoring. Multiple-choice elements use zero-shot accuracy, except MMLU which uses 5-shot accuracy §3.3. CrowS-Pairs is scored by comparing the model's perplexity on a stereotypical sentence and its anti-stereotypical pair; a larger value means a stronger bias §3.3. Scores come from lm-evaluation-harness §3.3.

Forgetting metric. For each element e of an evaluation set E_i, the relative drop from the initial score R^e_o to the score R^e_m after stage m is averaged over the N stages. The result is then averaged over the elements of the set eq. 1 §3.3.

FG_i = \frac{1}{|E_i|}\sum_{e\in E_i} \frac{1}{N} \sum_{m=1}^{N} \frac{R^{e}_o-R^{e}_m}{R^{e}_o} \times 100\% eq. 1

# Continual instruction tuning and scoring  [§3, §3.3]
# Input:  initial model M_0; ordered tasks T^1..T^N; evaluation sets E
# Output: initial scores R_o and one set of scores per stage R_m

def continual_tuning(initial_model, tasks, eval_sets):
    # Score the untuned model on every evaluation element  [§3.3]
    initial_scores = evaluate(initial_model, eval_sets)
    model = initial_model
    scores_per_stage = []
    for task in tasks:
        # Train on this task only; earlier data is not replayed  [§3.1]
        model = fine_tune(model, task.train_data)
        # Score the checkpoint after stage m  [§3.3]
        scores_per_stage.append(evaluate(model, eval_sets))
    return initial_scores, scores_per_stage

# Forgetting for one evaluation set  [eq. 1]
# Input:  R_o per element; R_m per element for each stage m
# Output: FG_i in percent

def forgetting_for_set(initial_scores, scores_per_stage, elements):
    # Relative drop per element at every stage  [eq. 1]
    per_element = []
    for element in elements:
        drops = []
        for scores in scores_per_stage:
            drop = initial_scores[element] - scores[element]
            drops.append(drop / initial_scores[element] * 100)
        # Average over the stages  [eq. 1]
        per_element.append(mean(drops))
    # Average over the elements of the set  [eq. 1]
    return mean(per_element)

Models. BLOOMZ at 1.1b, 1.7b, 3b and 7.1b; mT0 at 1.2b and 3.7b; LLaMA-7b; and Alpaca-7b, which is LLaMA-7b tuned on 52K self-instruct examples §4.1. The BLOOMZ and mT0 comparison is at matched scale §4.1 §5.3. The paper compares BLOOM with BLOOMZ, and LLaMA with Alpaca, to isolate general instruction tuning §5.4.

Mixed-data check. The paper mixes 10,000 general instruction samples from Alpaca into the continual tasks for BLOOMZ-7.1b and LLaMA-7b, and reports one split per evaluation set §5.4 fig. 6.

Implementation. See Hyperparameters below §4.2.

Hyperparameters

Results

Author-reported. Setup: five continual instruction tasks in one order, then forgetting measured on DK (MMLU), Rs, RC and bias. Models and scales are listed in §4.1. The tables reproduce the paper's values tab. 3 tab. 4 tab. 5 tab. 6.

Main forgetting results for BLOOMZ and mT0 tab. 4. R_o is the initial score, R_-1 the score at the end, and FG the forgetting value from eq. 1.

Bias results tab. 5. R_0 and R_-1 are CrowS-Pairs scores at the start and end; FG is reported as in the paper, and positive values mean less bias at the end §5.1.

Instruction tuning on each task tab. 3. Initial and tuned scores for each task. Metrics per task: SARI (Simp), BERTScore (Emdg, InqQG, Exp) and ROUGE-1 (HGen) tab. 3 §5.1.

Models with and without general instruction tuning tab. 6. Pairs are BLOOM vs BLOOMZ and LLaMA vs Alpaca §5.4.

Headline findings §5.1 §5.2 §5.3 §5.4.

• Forgetting is present in every model tested. The paper calls it "generally observed" §1 §5.1.
• Reading comprehension shows the largest forgetting for BLOOMZ-7.1b, then domain knowledge §5.1 tab. 4.
• In the BLOOMZ DK column, FG rises from 9.54 at 1.1b to 18.37 at 7.1b §5.2 tab. 4.
• At about 3b, BLOOMZ-3b shows reasoning FG 11.09, which the paper states is 5.64 lower than mT0-3.7b's 16.73 §5.3 tab. 4.
• Alpaca forgets less than LLaMA on all three sets tab. 6 §5.4.
• Mixing 10,000 Alpaca samples into the tasks reduces forgetting on the one MMLU-human split shown: LLaMA-7b falls from 34.72 to 26.8 with the continual tasks alone and to 30 with mixed data §5.4 fig. 6.

Paper vs code

No official code.

Searched: the paper and its bibliography link only third-party resources (the Stanford Alpaca repository and Hugging Face model pages) §4.1 §4.2. No project page is linked. GitHub repository searches for "catastrophic forgetting instruction tuning", "catastrophic forgetting BLOOMZ mT0", and "Yun Luo Yue Zhang forgetting" returned no repository owned by an author or their lab.

Pitfalls

• FG is an average over stages, not an endpoint drop. eq. 1 averages the relative drop over every stage m = 1 to N, not only the final checkpoint. The tables give only R_o and R_-1, so FG cannot be recomputed from them. For example, mT0-3.7b DK falls by (30.99 - 20.14) / 30.99 = 35.0% at the end, but its reported FG is 20.15 tab. 4. Intermediate scores are not reported.
• Bias FG does not match the endpoints either. mT0-3.7b bias falls from 57.16 to 50.59, an endpoint change of 11.5%, but the table reports 13.10 tab. 5. The paper does not say how intermediate stages enter the bias value. Treat the bias FG as author-reported only.
• Bias sign convention. For CrowS-Pairs a higher score means more bias §3.3. The paper reads a positive FG as less bias after tuning §5.1. The claim that bias "is mitigated" rests on this reading and on one sentence about physical-appearance prompts §5.1.
• The scale trend is not monotone in every set. DK rises with scale for BLOOMZ. Reasoning does not: FG is 6.73 at 1.1b and 6.48 at 1.7b. Reading comprehension falls from 27.56 at 3b to 26.75 at 7.1b tab. 4. The abstract's "intensifies" holds for DK only.
• BLOOMZ vs mT0 is confounded by size and by a small RC gap. The architecture comparison pairs BLOOMZ-3b with mT0-3.7b §5.3. The reading comprehension FG values are 27.56 and 28.42, a 0.86-point gap tab. 4. Compared at 1.1b and 1.2b, the DK, Rs and RC values are close tab. 4. No variance is reported.
• The general-instruction-tuning comparison is coarse. BLOOM vs BLOOMZ and LLaMA vs Alpaca change more than just the tuning, because the initial scores also differ, and the authors note that BLOOMZ's higher start drives its larger FG §5.4 tab. 6. The paper also says the learning-rate and scheduler settings "may still introduce some effects" §4.2.
• Single task order and no seeds. Results come from one order of five tasks §3.1 §7. The paper says this order "may affect the forgetting phenomenon" §7. No random seed or variance is reported §4.2.
• Broken or misnumbered references in the source. §5.2 cites Figure \ref{DK}, whose figure is commented out in the LaTeX source, so that figure does not appear in the compiled paper. §5.2 also says "As shown in Table 1" for the main results, which are tab. 4 in source order. The table and figure numbers here follow source order; LaTeX float placement may change them in the PDF. Check against the PDF before quoting.
• Metric name conflict. The caption of tab. 3 defines BS as BERTScore, but §5.1 calls it "BLUEScore". Use BERTScore for the table values.
• Dataset-size wording is ambiguous. "100,000 data samples are used for training" §3.1 does not say per task or in total. Assume per task only if the code confirms it.
• Hyperparameters are inherited, not tuned. The paper says its settings follow the BLOOMZ, mT0 and Alpaca works §4.2. Warmup, weight decay, Adam betas and whether all weights are updated are not stated.
• Mixed-data result is one split. The mixing result shows one split per set fig. 6 and a single quoted pair §5.4. The mixing ratio is not stated. The text names "BLOOMZ-7b", which is probably BLOOMZ-7.1b §5.4.
• Evaluation details to match. MMLU is 5-shot and the other multiple-choice sets are zero-shot §3.3. The lm-evaluation-harness version is not stated §3.3. Reproducing the numbers needs the same harness version and settings.
• Checkpoint source. The LLaMA-7b weights are given as a Hugging Face path under decapoda-research, a community repository, not an authors' release §4.1. The paper does not state the checkpoint hash.
• Costs the abstract hides. Five sequential tuning runs of 100,000 samples each, at 3 epochs, on 4 to 8 A100 40G GPUs per model §4.2 §3.1. The authors say compute limited them to 1b to 7b models §7.

Lineage

• Builds on: the continual task set, data and prompt templates of Scialom et al. (2022) §3.1. Evaluation uses lm-evaluation-harness §3.3.
• Compared against: probing of general knowledge in encoder-only models (Luo et al., 2023) §2; representation drift under probing (Davari et al., 2022) §2; layer-wise probing of forgetting (cited as WUTONG) §2; and the forgetting of T0 models on downstream tasks (Scialom et al., 2022) §2.
• Superseded by: not known from this wiki.