Activation patching

Overview

Activation patching asks which internal parts of a model cause one of its outputs. Run the model on a clean input and on a corrupted input, where the corruption changes only the property under test. Then copy some internal activations from the clean run into the corrupted run and measure how much of the clean behaviour comes back. Only the patched parts can restore it, so the recovery is a causal signal 2211.00593 §3 2603.17839 app. C.1.4.

The invariant all variants share is the choice of corruption. The corrupted input must keep the structure of the clean one and remove the information you want to locate. The IOI paper uses a reference set in which the names are replaced by unrelated random names, keeping the template 2211.00593 §2.1. The verbal-confidence paper replaces the answer tokens with mean embeddings from calibration trials, which removes the answer while keeping the prompt 2603.17839 app. C.1.4.

Variants differ in what they patch and where. They are listed below, and each card is linked from its variant.

Variants

Path patching for circuit discovery

Patch the output of one attention head (the sender) only along the paths that reach a chosen set of downstream components (the receivers): the logits, or the query, key or value input of another head. Every other head keeps its original input, so the test isolates the paths between components. Working backwards from the logits, the authors alternate this test with attention-pattern checks to find the heads that matter, and they ended with a 26-head circuit for indirect object identification in GPT-2 small 2211.00593 §3 2211.00593 app. B. The circuit keeps 87% of the full model's average logit difference 2211.00593 §4. Card: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.

Single-site residual patching across layers and positions

Restore the clean residual stream at one token position and one layer of a corrupted run, and let the rest of the run stay corrupted. Sweeping the layer and position gives a map of where information is stored. The verbal-confidence paper uses this on Gemma 3 27B and reports that the confidence colon recovers almost fully when patched at high confidence, while the newline after the answer recovers partly, peaking at one middle layer, and the token after that newline recovers about nothing 2603.17839 §2.2 2603.17839 app. D.4. The recovery is measured only on high-confidence trials, because low-confidence trials sit near the floor 2603.17839 §2.2. Card: How do LLMs Compute Verbal Confidence.

Companion interventions (noising, swap, attention knockout)

The same paper pairs patching with three checks on the same positions. Noising replaces one site's activation with a calibration mean, which tests whether the site is needed 2603.17839 §2.3. Swapping copies the activation of a donor trial, with a different confidence, into a recipient trial, and compares the shift against a control 2603.17839 §2.4. Attention knockout zeroes attention from one position to another over a window of layers [arxiv:2603.17839 §2.8]. These are not patching methods on their own, but they test the same claims from another direction.

When to use which

Constraints. Both methods need access to internal activations and the ability to write them back during a forward pass. The IOI work runs on GPT-2 small with three forward passes per sender set 2211.00593 §3. The verbal-confidence work runs a 62-layer, 27B-parameter model over a sweep of layers and positions, using 100 calibration trials for means 2603.17839 app. C.1.1 app. C.1.4. Neither card states the hardware used. Plan for many forward passes, and for the completeness check to be sampled rather than exhaustive, since the exact check grows exponentially with circuit size 2211.00593 §4.1.

Implementation sketch

Path patching, in the form the IOI paper describes. A sender is patched only into the chosen receivers.

# Path patching of one sender into a set of receivers  [arxiv:2211.00593 §3]
# Inputs: prompt x, reference prompt x_ref (same template, other names)
def path_patch(x, x_ref, sender, receivers):
    # Record the sender's output on the reference prompt  [arxiv:2211.00593 §3]
    sender_out = record_output(x_ref, sender)
    # Record every head's query, key and value on the original prompt
    # [arxiv:2211.00593 app. B]
    cache = record_all_qkv(x)
    # Re-run x; the sender's output reaches only the receivers, and the
    # other heads keep their values from x  [arxiv:2211.00593 app. B]
    patched = run_with_path_patch(x, cache, sender_out, receivers)
    # Change in the metric, here the logit difference  [arxiv:2211.00593 §2]
    return metric(patched) - metric(cache.output)

Single-site patching, for a residual-stream sweep. Use it when you do not need paths, only the location of a signal.

# Single-site residual patching  [arxiv:2603.17839 app. C.1.4]
# Inputs: clean run, corrupt run (answer tokens replaced by mean embeddings)
def patch_site(clean, corrupt, position, layer):
    # Start from the corrupt run  [arxiv:2603.17839 app. C.1.4]
    patched = corrupt.copy()
    # Restore the clean residual stream at one position and layer
    # [arxiv:2603.17839 eq. 2]
    patched.resid[layer][position] = clean.resid[layer][position]
    # Let the forward pass continue from the next layer  [arxiv:2603.17839 app. C.1.4]
    return forward_from(patched, layer + 1)

Where the variants differ. Path patching changes the attention inputs of the receivers; single-site patching changes one residual stream value. Patch MLPs only where you have to: the IOI paper recomputes them along patched paths and warns that they can confound the result 2211.00593 §3.

Traps to test for in any implementation.

• Use mean ablation, not zero ablation, for the removed part. The IOI authors found zeroing gave noisy results 2211.00593 §2.1.
• Run the check on the trials that matter. The verbal-confidence result is restricted to high-confidence trials, and the paper flags that the recovery formula is unstable when the corrupt-to-clean gap is small [arxiv:2603.17839 §2.2] eq. 5.
• Keep the node definition explicit. In the IOI circuit a head is kept only at the token positions it needs, not at every position [arxiv:2211.00593 §3] §2.

Contradictions

None between the two cards on the method itself. They use different targets (attention-head paths and residual sites) and different tasks, so their numbers do not compare directly.

Within the IOI paper, the text counts 23 heads for the circuit, while the abstract and the released code give 26 2211.00593 §1 §3.3. Use the 26-head list, and treat the text count as a discrepancy in the paper.

Papers

• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small: path patching and mean ablation find a 26-head circuit that keeps 87% of the logit difference in GPT-2 small.
• How do LLMs Compute Verbal Confidence: corrupt-then-restore patching, noising, swaps and attention knockout locate a verbal-confidence signal in Gemma 3 27B.