---
kind: concept
slug: activation-patching
title: Activation patching
summary: >-
  Causal test that copies internal activations from a clean run into a corrupted run to find which model components carry a behaviour. Path patching finds an attention-head circuit in GPT-2 small; single-site residual patching localises a verbal-confidence signal in Gemma 3 27B. Both need white-box access and several forward passes per test.
tags: [interpretability, causal-intervention, mechanistic-interpretability, transformers]
papers: ["2211.00593", "2603.17839"]
related: [attention-mechanism, attention-interpretability, linear-probing, llm-self-evaluation]
status: draft
updated_at: 2026-10-18
---

## Overview

Activation patching asks which internal parts of a model cause one of its
outputs. Run the model on a clean input and on a corrupted input, where the
corruption changes only the property under test. Then copy some internal
activations from the clean run into the corrupted run and measure how much of
the clean behaviour comes back. Only the patched parts can restore it, so the
recovery is a causal signal [arxiv:2211.00593 §3] [arxiv:2603.17839 app. C.1.4].

The invariant all variants share is the choice of corruption. The corrupted
input must keep the structure of the clean one and remove the information you
want to locate. The IOI paper uses a reference set in which the names are
replaced by unrelated random names, keeping the template [arxiv:2211.00593 §2.1].
The verbal-confidence paper replaces the answer tokens with mean embeddings
from calibration trials, which removes the answer while keeping the prompt
[arxiv:2603.17839 app. C.1.4].

Variants differ in what they patch and where. They are listed below, and each
card is linked from its variant.

## Variants

### Path patching for circuit discovery

Patch the output of one attention head (the sender) only along the paths that
reach a chosen set of downstream components (the receivers): the logits, or the
query, key or value input of another head. Every other head keeps its original
input, so the test isolates the paths between components. Working backwards
from the logits, the authors alternate this test with attention-pattern checks
to find the heads that matter, and they ended with a 26-head circuit for
indirect object identification in GPT-2 small [arxiv:2211.00593 §3]
[arxiv:2211.00593 app. B]. The circuit keeps 87% of the full model's average
logit difference [arxiv:2211.00593 §4]. Card:
[Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593).

### Single-site residual patching across layers and positions

Restore the clean residual stream at one token position and one layer of a
corrupted run, and let the rest of the run stay corrupted. Sweeping the layer
and position gives a map of where information is stored. The verbal-confidence
paper uses this on Gemma 3 27B and reports that the confidence colon recovers
almost fully when patched at high confidence, while the newline after the answer
recovers partly, peaking at one middle layer, and the token after that newline
recovers about nothing [arxiv:2603.17839 §2.2] [arxiv:2603.17839 app. D.4].
The recovery is measured only on high-confidence trials, because low-confidence
trials sit near the floor [arxiv:2603.17839 §2.2]. Card:
[How do LLMs Compute Verbal Confidence](/papers/2603.17839).

### Companion interventions (noising, swap, attention knockout)

The same paper pairs patching with three checks on the same positions. Noising
replaces one site's activation with a calibration mean, which tests whether the
site is needed [arxiv:2603.17839 §2.3]. Swapping copies the activation of a
donor trial, with a different confidence, into a recipient trial, and compares
the shift against a control [arxiv:2603.17839 §2.4]. Attention knockout zeroes
attention from one position to another over a window of layers [arxiv:2603.17839
§2.8]. These are not patching methods on their own, but they test the same
claims from another direction.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want named components (heads) behind a behaviour on one fixed task template | Path patching | Gives per-head and per-path effects and a checkable circuit [arxiv:2211.00593 §3] [§4] |
| You want to know which layer and token position hold a signal in a large model | Single-site residual patching | One run per site; a sweep maps the layers and positions [arxiv:2603.17839 §2.2] |
| You want to test whether an edge between positions carries information | Attention knockout | Cheap and direct, but null results are weak evidence because information moved earlier is not blocked [arxiv:2603.17839 app. C.1.10] |
| You need a claim about a whole component class, not one head | Class-level knockout plus backup checks | Single heads can be replaced by backups, so one knockout can understate a class [arxiv:2211.00593 §3.4] |
| The corruption you can build is not matched to the clean prompt structure | Do not use patching yet | Effects then mix the property under test with prompt shape [arxiv:2211.00593 §2.1] [arxiv:2603.17839 §2.8] |

Constraints. Both methods need access to internal activations and the ability
to write them back during a forward pass. The IOI work runs on GPT-2 small with
three forward passes per sender set [arxiv:2211.00593 §3]. The verbal-confidence
work runs a 62-layer, 27B-parameter model over a sweep of layers and positions,
using 100 calibration trials for means [arxiv:2603.17839 app. C.1.1] [app. C.1.4].
Neither card states the hardware used. Plan for many forward passes, and for
the completeness check to be sampled rather than exhaustive, since the exact
check grows exponentially with circuit size [arxiv:2211.00593 §4.1].

## Implementation sketch

Path patching, in the form the IOI paper describes. A sender is patched only
into the chosen receivers.

```pseudocode
# Path patching of one sender into a set of receivers  [arxiv:2211.00593 §3]
# Inputs: prompt x, reference prompt x_ref (same template, other names)
def path_patch(x, x_ref, sender, receivers):
    # Record the sender's output on the reference prompt  [arxiv:2211.00593 §3]
    sender_out = record_output(x_ref, sender)
    # Record every head's query, key and value on the original prompt
    # [arxiv:2211.00593 app. B]
    cache = record_all_qkv(x)
    # Re-run x; the sender's output reaches only the receivers, and the
    # other heads keep their values from x  [arxiv:2211.00593 app. B]
    patched = run_with_path_patch(x, cache, sender_out, receivers)
    # Change in the metric, here the logit difference  [arxiv:2211.00593 §2]
    return metric(patched) - metric(cache.output)
```

Single-site patching, for a residual-stream sweep. Use it when you do not need
paths, only the location of a signal.

```pseudocode
# Single-site residual patching  [arxiv:2603.17839 app. C.1.4]
# Inputs: clean run, corrupt run (answer tokens replaced by mean embeddings)
def patch_site(clean, corrupt, position, layer):
    # Start from the corrupt run  [arxiv:2603.17839 app. C.1.4]
    patched = corrupt.copy()
    # Restore the clean residual stream at one position and layer
    # [arxiv:2603.17839 eq. 2]
    patched.resid[layer][position] = clean.resid[layer][position]
    # Let the forward pass continue from the next layer  [arxiv:2603.17839 app. C.1.4]
    return forward_from(patched, layer + 1)
```

Where the variants differ. Path patching changes the attention inputs of the
receivers; single-site patching changes one residual stream value. Patch MLPs
only where you have to: the IOI paper recomputes them along patched paths and
warns that they can confound the result [arxiv:2211.00593 §3].

Traps to test for in any implementation.

- Use mean ablation, not zero ablation, for the removed part. The IOI authors
  found zeroing gave noisy results [arxiv:2211.00593 §2.1].
- Run the check on the trials that matter. The verbal-confidence result is
  restricted to high-confidence trials, and the paper flags that the recovery
  formula is unstable when the corrupt-to-clean gap is small [arxiv:2603.17839
  §2.2] [eq. 5].
- Keep the node definition explicit. In the IOI circuit a head is kept only at
  the token positions it needs, not at every position [arxiv:2211.00593
  §3] [§2].

## Contradictions

None between the two cards on the method itself. They use different targets
(attention-head paths and residual sites) and different tasks, so their numbers
do not compare directly.

Within the IOI paper, the text counts 23 heads for the circuit, while the
abstract and the released code give 26 [arxiv:2211.00593 §1] [§3.3]. Use the
26-head list, and treat the text count as a discrepancy in the paper.

## Papers

- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593): path patching and mean ablation find a 26-head circuit that keeps 87% of the logit difference in GPT-2 small.
- [How do LLMs Compute Verbal Confidence](/papers/2603.17839): corrupt-then-restore patching, noising, swaps and attention knockout locate a verbal-confidence signal in Gemma 3 27B.