Model editing

Overview

A trained language model sometimes gives a wrong answer that should change, for example after a fact goes out of date. Model editing changes the answer to a given input, or a set of inputs, and leaves the rest of the model's behaviour alone. An edit is given either as an input and a desired output, or as a free-text description 2206.06520 §2.

The two properties every editor is judged on are the same across papers. Edit success (ES) is how often the inputs the edit should cover get the intended answer 2110.11309 §4 2206.06520 eq. 4. Drawdown (DD) is how much the model changes on inputs it should not touch 2110.11309 §4 2206.06520 eq. 5. The hard cases lie on both sides: in-scope inputs that need reasoning, and out-of-scope inputs that look like the edit 2206.06520 §2. Fine-tuning on one example tends to fail both tests. It changes the answer, but rephrasings of the question and unrelated questions can change too 2110.11309 §1.

A third test asks whether the edit carries to facts that follow from it. If the mother of a person changes, the children of the new mother are that person's siblings, and those answers should change as well. The RippleEdits benchmark scores this with six criteria: logical generalization, two compositionality tests, subject aliasing, preservation and relation specificity 2307.12976 §3.1. Parametric editors pass the aliasing test, at least 86.8 in every setting, but on the Popular subset ROME and MEMIT on GPT-2 and GPT-J score 5.5 to 7.0 on logical generalization 2307.12976 tab. 3 2307.12976 tab. 5.

A survey of agent memory places editing among the parametric forms of memory. Its other form is textual memory kept in the prompt or a store. Its editing entry only names MAC, MEND and KnowledgeEditor, and the survey runs no experiments 2404.13501 §5.2.2 2404.13501 §6. Its point is that parametric memory adds nothing to the prompt at read time, while textual memory is cheaper to write 2404.13501 §5.2.3. That is the authors' reasoning, with no measurements behind it. Card: Agent memory survey (Zhang et al.).

Variants

Fine-tuning with a locality penalty (baseline)

Plain fine-tuning on the corrected example is the baseline. Adding a KL term on unrelated data (FT + KL) keeps the model closer to its original behaviour 2110.11309 §4.1. On the two decoder models in the MEND paper it lowers perplexity drawdown, but it also lowers edit success on GPT-J, from 0.80 to 0.36 2110.11309 §4.1. Fine-tuning reaches 0.87 ES on T5-XXL with drawdown below 0.001 2110.11309 §4.1. Card: MEND.

Learned gradient editors (MEND)

A small editor network is trained ahead of time for each shape of edited weight matrix. At edit time it takes a layer's fine-tuning gradient, kept as two short factor vectors, and returns a low-rank weight change. The edit is a single update with no iteration 2110.11309 alg. 2. Training uses an edit set of tuples and a loss that combines edit success on an equivalent input with a locality KL on unrelated inputs. The base model is not updated 2110.11309 §3.2. The paper edits the MLP blocks of the last layers 2110.11309 §4. Card: MEND.

On the RippleEdits benchmark, MEND has the lowest CI and CII scores of the parametric methods on GPT-2 XL, at 17.0 and 13.6 averaged over the subsets 2307.12976 tab. 6. Its LG score on the same model is the highest of the three, at 39.1 2307.12976 tab. 6.

Memory-based editors (SERAC)

The base model is frozen. Each edit is stored as its descriptor in a cache, and a scope classifier decides whether an input falls under an edit. If the best-scoring edit scores at least 0.5, a small counterfactual model answers from the edit and the input. Otherwise the base model answers [arxiv:2206.06520 §3.1] 2206.06520 eq. 1. The authors put the memory at about 3 KB per edit 2206.06520 §5.2. Card: SERAC.

In-context editing (ICE)

No weights change. The edit is written as a sentence and placed before the question, in the form "Imagine that <o*> would have been <P_r>", where P_r is a manual proposition for the relation, such as "the mother of " 2307.12976 §5.1. RippleEdits uses this as a baseline, and the paper reports it as the best on average across its three subsets [arxiv:2307.12976 §5.2]. The edit lives in the prompt, so it has to be sent with every query that needs it. The paper does not measure that cost 2307.12976 §5.1. Card: RippleEdits.

Adapter-based unlearning (a neighbouring goal)

Unlearning removes the influence of a set of training data, rather than changing one answer. Chen and Yang add small unlearning layers to a frozen T5 model. Each layer is trained with a selective teacher-student KL term, a task loss on retained data and a negated masked-LM loss on the forgotten data 2310.20150 §3.1. Several such layers are merged in closed form from stored Gram matrices, so no data is kept 2310.20150 §3.2. On IMDB with T5-base and 0.5% of the data forgotten, forget-set accuracy is 65.7 against 92.5 for re-training, and the update takes 1200 s against 6685 s 2310.20150 tab. 2. Card: Unlearn What You Want to Forget.

Locating facts inside the model (analysis, not an editor)

Geva et al. trace how GPT-2 XL and GPT-J answer subject-relation queries. Blocking attention from the last position to the subject tokens lowers the answer's probability by up to 60% in middle-upper layers 2304.14767 §5. Early MLP updates at the subject enrich its representation, and zeroing them lowers the attribute rate by about 88% on average 2304.14767 §6.3. Upper attention heads extract the attribute for 68.2% of GPT-2 XL queries 2304.14767 tab. 2. The paper runs no editing experiment. Its results say where recall happens, which helps choose layers for the variants above. Card: Dissecting recall of factual associations.

When to use which

Latency and memory costs also differ. For SERAC on QA, a routed input takes 31 ms against 87 ms for the base model, and an input sent to the base model takes 92 ms. Memory rises from 2.96 GB to 3.47 GB 2206.06520 tab. 6. On FC the scope pass takes 19 ms against 7 ms for the base model 2206.06520 tab. 6.

Implementation sketch

Every variant reduces to the same evaluation: apply the edit, then measure the two properties. The variants differ in where the edit lives. In MEND it is a weight update 2110.11309 alg. 2. In SERAC it is an entry in the cache plus a routing decision 2206.06520 eq. 1. In ICE it is a prefix on the prompt 2307.12976 §5.1.

# Measure one edit: edit success and drawdown  [arxiv:2206.06520 eq. 4-5]
# Input:  edited model, base model, in-scope (x, y) pairs, out-of-scope inputs
# Output: edit success, drawdown

def evaluate_edit(edited, base, in_scope, out_of_scope):
    # Share of in-scope inputs that get the intended answer  [arxiv:2206.06520 eq. 4]
    es = mean(edited(x) == y for x, y in in_scope)
    # Share of out-of-scope inputs whose output moved from the base model  [eq. 5]
    # MEND measures drawdown as an accuracy or perplexity change instead  [arxiv:2110.11309 §4]
    dd = mean(edited(x) != base(x) for x in out_of_scope)
    return es, dd

# Measure whether implied facts follow the edit  [arxiv:2307.12976 §3.1]
# Simplified: the paper also drops tests that the unedited model fails  [§5.1]
def evaluate_ripples(edited, implied):
    # implied: (question, expected answer) pairs for facts the edit should change
    return mean(edited(q) == a for q, a in implied)

Contradictions

• Drawdown means different things. MEND reports accuracy or perplexity change on unrelated data 2110.11309 §4. SERAC reports the share of out-of-scope outputs that change, and uses KL for ConvSent 2206.06520 eq. 5, eq. 7. The two DD columns are not comparable.
• Cross-paper numbers disagree. MEND reports ES 0.88 and DD 0.001 on zsRE with T5-XL 2110.11309 §4.1. SERAC reports MEND at ES 0.823 and DD 0.187 on its QA setting with T5-large 2206.06520 tab. 3. The base models, benchmarks and metrics differ. Both sets are author-reported. None is resolved here.
• The large-model claim. MEND claims it is the only approach that edits models above 10B 2110.11309 §1, §5. Its own T5-XXL table shows fine-tuning at 0.87 ES with DD below 0.001, against MEND's 0.89 2110.11309 §4.1.
• Batched edits in MEND. The text reports 96% ES at 25 edits, and the table reports 0.89 2110.11309 §4.3.
• ICE against ROME depends on the subset. RippleEdits reports ICE as best on average 2307.12976 §5.2. On the Recent subset, ROME on GPT-NeoX averages 71.4 and ICE 69.9 2307.12976 tab. 3. ICE leads on Random (81.0 against 58.3) and Popular (74.1 against 59.7) 2307.12976 tab. 4 2307.12976 tab. 5. The average hides this.
• Metrics differ across the editing and unlearning papers. Unlearning is judged by forget-set accuracy and a membership attack, and it has no ES or DD 2310.20150 §4.2 2310.20150 §4.6. Its numbers cannot be read against the ES and DD figures above.
• Which layer extracts the attribute. Geva et al. say upper attention heads do it in §1 and §7, and that MLP sublayers do it "eminently" in §9 2304.14767 §1 2304.14767 §7 2304.14767 §9. The card follows §7.
• SERAC's own tables. Table 3 and Table 5 do not match row by row for QA-hard and FC, and the text's 0.99 for QA does not match the table's 0.986 2206.06520 tab. 3 2206.06520 tab. 5 2206.06520 §5.2.

Pitfalls

• MEND's update sign. The paper writes the edit as a subtraction 2110.11309 §3.1. The released code's update, as the card reads it, moves against the descent direction unless the learned step sizes turn negative. Check the sign on a small case before using it. See the MEND card for the code citations.
• Generality is tested only lightly. The MEND paper uses backtranslated paraphrases and a prefix heuristic. It does not test whether an edit implies other facts 2110.11309 §5. RippleEdits tests implied facts, but only for relations it encodes, and only within two hops of the edited entity 2307.12976 §3.1.
• Overgeneralisation happens. MEND's edits sometimes change outputs for related but different inputs 2110.11309 §5, app. F.
• SERAC's routing threshold is fixed at 0.5. The paper does not calibrate it per task 2206.06520 eq. 1.
• SERAC's memory is unbounded in continual editing. The authors propose periodic distillation as future work 2206.06520 §7.
• Released code and paper differ for RippleEdits. The generation budget is stated as 20 tokens in the paper 2307.12976 §5.1, but the released code's limit is not that. The preservation check requires every existing object in the released code, while the paper says one match suffices. The ICE prefix also differs. Check the RippleEdits card before reproducing its numbers.
• Unlearning code is not in the official repository. At the pinned commit the repository has no unlearning code, so the method must be implemented from the equations 2310.20150 §3.1. See the unlearning card.
• Geva et al. analyse correct predictions only. Their rates are conditional on the model already answering correctly 2304.14767 §3. Extraction counts a query only when the top-1 token exactly matches the answer 2304.14767 §7.1.
• Attribute rate is a proxy. It counts top tokens that appear in Wikipedia-derived candidate sets, so frequent words from those paragraphs also count 2304.14767 app. C.1.

Papers

• Fast Model Editing at Scale: MEND, a learned gradient editor that makes one-step weight edits, with fine-tuning and KL baselines.
• Memory-Based Model Editing at Scale: SERAC, a memory-based editor with a scope classifier and a counterfactual model, leaving the base weights unchanged.
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models: an analysis of where factual recall happens in GPT-2 XL and GPT-J. It does not run an editor.
• Evaluating the Ripple Effects of Knowledge Editing in Language Models: RippleEdits, six criteria for implied facts, comparing ROME, MEMIT, MEND and in-context editing.
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs: adapter-based unlearning on a frozen T5 model for deletion requests. A neighbouring goal, not a single-answer editor.
• Agent memory survey (Zhang et al.): a survey that places editing among parametric agent memories. It runs no experiments.