---
kind: concept
slug: mechanistic-interpretability
title: Mechanistic interpretability
summary: >-
  Finds the internal parts behind a language model's behaviour with causal interventions. Path patching finds a 26-head circuit for indirect object identification in GPT-2 small. Other variants trace information flow, learn sparse features, read feed-forward value vectors, probe and steer a readout, or shift probe-chosen heads. Most need white-box access.
tags: [interpretability, mechanistic-interpretability, causal-intervention, transformers]
papers: ["2203.14680", "2211.00593", "2304.01481", "2304.14767", "2306.03341", "2309.01029", "2309.08600", "2512.02004", "2512.10092", "2603.17839"]
related: [activation-patching, activation-steering, adaptive-halting, attention-interpretability, attention-mechanism, explainable-ai, hallucination-mitigation, linear-probing, llm-self-evaluation, automated-interpretability, in-context-learning, model-editing, rlhf, sparse-autoencoder, text-embedding, dataset-curation, grokking]
status: draft
updated_at: 2026-11-01
---

## Overview

Mechanistic interpretability asks how a model computes an output, not only
which inputs correlate with it. The usual method is a causal intervention:
change or remove an internal quantity, run the model again, and measure how
the output moves. A component that matters for the behaviour should change it
when altered [arxiv:2211.00593 §1] [arxiv:2603.17839 §1].

The invariant all variants share is a checkable claim. A proposed explanation
names components, says what they do, and must survive tests that the rest of
the model is left alone. The IOI paper frames this as three checks on a
circuit: faithfulness (the subset performs like the full model), completeness
(the subset still behaves like the model when parts of it are removed) and
minimality (every part matters) [arxiv:2211.00593 §1] [§4].

A survey of explainability for transformer language models files these
methods under global explanation. It lists circuits, induction heads and
feed-forward layers read as key-value memories as mechanistic interpretability
[arxiv:2309.01029 §3.2.4]. The survey runs no experiments, and this page does
not use its figures.

A position paper asks a prior question: whether a model's internal states can
be about things in the world at all. It argues that a state can be referentially
grounded if it carries worldly information and was selected for doing so
[arxiv:2304.01481 §5]. It runs no experiments, so it gives no method. It is
cited here for the framing that an internal state can be checked against what
it stands for.

The variants below differ in the unit they study and the intervention they use.
Each card is linked from the variant it supports.

## Variants

### Circuit discovery with path patching and mean ablation

Work backwards from the output. At each step, patch a candidate sender
component along only the paths that reach a chosen set of receivers, and keep
the components whose patch moves the metric. The IOI paper uses this to find
a 26-head circuit in GPT-2 small, grouped into seven classes. It keeps 87% of
the full model's average logit difference [arxiv:2211.00593 §1] [§4.3]. Nodes are
(head, token position) pairs, so the circuit is about 1.1% of all such pairs
[arxiv:2211.00593 §1]. The removed parts are replaced by mean activations on a
reference set with the same template and different names, not by zeros
[arxiv:2211.00593 §2.1]. The method is detailed in [Activation patching](/concepts/activation-patching).
Card: [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593).

### Attention knockout for information flow

Block the attention edge from one set of positions to another in a window of
layers, then check how much the answer probability drops. The
factual-recall paper uses windows of 9 layers on GPT-2 XL and 5 on GPT-J, and
it blocks the last position's attention to the subject tokens or to the
relation tokens [arxiv:2304.14767 §5]. Its finding is that relation information
reaches the last position earlier than subject information, and that blocking
subject positions in the middle and upper layers lowers the answer probability
by up to 60% [arxiv:2304.14767 §5]. The same paper zeroes sublayer updates at
the last subject position: early MLP updates lower the attribute rate at layer 40
by about 88% on average, and MHSA updates by less than 30% [arxiv:2304.14767 §6.3].
Card: [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](/papers/2304.14767).

The verbal-confidence paper uses the same knockout, over windows of 12 layers,
to test whether one position reads from another [arxiv:2603.17839 §2.8]
[fig. 7] [app. C.1.10]. Both papers warn that a knockout does not stop
information that earlier layers already moved, so a null result is weak
evidence [arxiv:2304.14767 Limitations] [arxiv:2603.17839 app. C.1.10].

### Vocabulary projection of feed-forward value vectors

Split one feed-forward block's output into sub-updates. Each sub-update is a
coefficient that depends on the input, times a fixed value vector. Projecting
a value vector onto the vocabulary gives a ranking that does not depend on the
input, and many of these rankings read as a concept [arxiv:2203.14680 §3]
[§4.1]. The concepts were labelled by hand from the top 30 tokens. They appear
in 36.7% of GPT-2 sub-update rankings and 16% of random ones, and the
comparison is with random vectors only [tab. 2]. The paper also reports that
the largest sub-updates of a layer mostly raise the final top token, rather
than lower its rivals [§5.1] [tab. 4]. Boosting 10 hand-picked non-toxic value
vectors in GPT-2, with their coefficients set to 3, lowers the share of
completions with Perspective toxicity above 0.5 from 58.5% to 30.8% on a
challenging prompt subset. Perplexity rises from 21.7 to 25.3 [§6.1] [tab. 5].
A rule that tracks the clusters of the ten largest sub-updates exits early
with 94.1% accuracy and saves 20.0% of layers on a word-level model. The
authors call this on par with per-layer classifiers [§6.2] [tab. 6]. Card:
[Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](/papers/2203.14680).

### Sparse feature dictionaries

Instead of components, learn directions. A sparse autoencoder trained on one
layer's activations gives an overcomplete set of features, each a direction in
the residual stream, with a ReLU encoder and an L1 penalty on the codes
[arxiv:2309.08600 §2] [eq. 1] [eq. 2] [eq. 4]. The authors argue that these
features are easier to interpret than single neurons, and their automated score
(an explainer LLM writes a description, a simulator LLM predicts activations)
ranks them above neurons, PCA, ICA and random directions in their setup
[arxiv:2309.08600 §3.2] [fig. 2]. Patching a few features in the indirect
object task reaches a given change with fewer patches than PCA components, with
no numbers given in the text [arxiv:2309.08600 §4.2] [fig. 3]. The authors also
report that substituting the reconstruction raises perplexity, from 25 to 40 in
one layer [arxiv:2309.08600 §6.2]. Card:
[Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600).

The same kind of feature can be used as a data tool and not only as an
intervention. A reader model's layer-50 activations, encoded by a released
sparse autoencoder with 65,536 latents, turn each document into a vector of
named concepts, max-pooled over tokens [arxiv:2512.10092 §3] [§4]. Its
analyses compare datasets by latent frequency, find co-occurring latents with
NPMI, cluster by chosen concepts, and rank documents for a query. The authors
report that the frequency diffs cost 2 to 8 times fewer tokens than LLM
labelling baselines [arxiv:2512.10092 §4.1] [app. D.4]. Card:
[Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092).

Supervision can also tie named concepts to fixed slots. AlignSAE trains a
sparse autoencoder on one GPT-2 layer (layer 6 in the author's experiments) in
two stages. The first stage reconstructs the activations. The second adds
terms that bind each ontology relation to its own slot, decorrelate those
slots from the free features, and predict the answer's first token through an
auxiliary head [arxiv:2512.02004 §3] [app. C.2] [app. C.3]. On synthetic
biographies with GPT-2 124M, the author-reported diagonal accuracy on unseen
templates is 0.912 at layer 6, against 0.165 at layer 0 [arxiv:2512.02004 §5.1].
Adding a decoded slot direction to the residual stream at the question's last
token changes the answer to the target relation in 0.847 of swaps at alpha = 2
[arxiv:2512.02004 §5.3] [§6]. Removing the alignment loss drops swap success to
0.053, and removing the orthogonality term drops it to 0.046 [§6].
Card: [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004).

### Probing and steering a readout

Use linear probes, steering vectors and corrupt-then-restore tests to ask
whether a model holds a quantity before it outputs it. The verbal-confidence
paper finds that the post-answer newline stores a signal that the confidence
colon later reads, on Gemma 3 27B [arxiv:2603.17839 §2.2] [§2.3]. Probes at the
newline are reported to explain variance in confidence beyond six
token-log-probability summaries [arxiv:2603.17839 §2.6] [app. D.3]; the size of
that gain is inconsistent within the card (see Contradictions). Steering with a
mean-difference direction has its peak effect at the newline in layers 21 to 25
and at the colon in layers 30 to 35 [arxiv:2603.17839 §2.1]. Card:
[How do LLMs Compute Verbal Confidence](/papers/2603.17839).

### Probe-selected head steering at inference

Fit a linear probe to each attention head's output at the last token of a
labelled question-answer pair, keep the K heads whose probes are most accurate,
and add a scaled direction to those heads at every decoding step. No
retraining is needed [arxiv:2306.03341 §3.2] [§3.3]. The direction is the mass
mean shift, from the mean of the false activations to the mean of the true
ones. On LLaMA-7B it beats the probe weights, a contrast-consistent search
direction and a random direction [arxiv:2306.03341 tab. 3]. The paper sets K = 48
and alpha = 15 from a sweep on LLaMA-7B [§4.3] [fig. 4]. Scoring uses a
fine-tuned GPT judge [§4.1]. The author-reported true*informative score on
TruthfulQA rises from 32.5% to 65.1% for Alpaca and from 51.5% to 74.0% for
Vicuna [arxiv:2306.03341 tab. 2]. A stronger alpha makes answers more truthful
but less informative, because the model says "I have no comment" more often
[§5.4] [fig. 6]. Card: [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](/papers/2306.03341).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need named components behind one behaviour on a fixed template, and you can write matched corrupted prompts | Path patching with mean ablation | Yields a small, checkable circuit with per-head effects [arxiv:2211.00593 §3] [§4] |
| You need a rough map of where information moves across layers and positions in a large model | Attention knockout over a layer window | One set of forward passes per window; no need to pick components first [arxiv:2304.14767 §5] |
| You want to test whether a readout uses information from a site, or whether a signal is already present before it is needed | Patching, noising, swaps and probes at the site | Each test gives a different kind of evidence and the paper combines them [arxiv:2603.17839 §2.2] [§2.3] [§2.4] |
| You want a training-free nudge on a few feed-forward value vectors, have white-box access, and can measure the fluency cost | Boosting hand-picked value vectors | The toxicity gain came with a perplexity rise from 21.7 to 25.3 [arxiv:2203.14680 tab. 5]. The set was chosen by a short manual search, and the paper reports no sweep of the coefficient [app. A.4] [§6.1] |
| You want to see which vocabulary items a layer pushes up before you intervene | Vocabulary projection and sub-update split | Needs only the value vectors and the output embedding, with no training. The paper's promotion finding covers the top ten sub-updates, not all of them [arxiv:2203.14680 §3] [§5.1] |
| You want units that are easier to read than neurons and can afford per-layer training and LLM-based scoring | Sparse feature dictionary | Features score higher than neurons in the authors' setup; the authors report under an hour per run on one A40 GPU [arxiv:2309.08600 §3.2] [app. B] |
| You have labelled true and false answers for a task, white-box access at inference, and want a cheap nudge without retraining | Probe-selected head steering | Adds one vector per layer at decode time, which the authors say costs close to nothing [arxiv:2306.03341 §5.2]. Gains were measured on one benchmark; the transfer gains were 4.7, 1.5 and 4.45 points [tab. 4] |
| You want to compare, cluster or search a text corpus by named properties, and you have a reader model with a released sparse autoencoder | Sparse-autoencoder document embeddings | Each latent gives a human-readable concept, and the authors report lower cost than LLM labelling for dataset diffs [arxiv:2512.10092 §4.1] [app. D.4]. No official code was found, so the pipeline must be rebuilt from the paper |
| You have a fixed ontology with labelled examples, want each concept in a predictable slot, and can afford a second training stage on a small model | Supervised concept-slot sparse autoencoder | Diagonal binding reached 0.912 on unseen templates in the authors' GPT-2 setting, with swap success reported only at the best alpha [arxiv:2512.02004 §5.1] [§5.3]. Tested on GPT-2 124M with synthetic data only [arxiv:2512.02004 §4.2] [Limitations] |
| You only have black-box API access | None of the above | Every method here needs internal activations and the ability to write them back [arxiv:2603.17839 §1] [arxiv:2211.00593 §2] |
| You need to show a behaviour is fully explained | No single method is enough | Redundant heads can hide a component, so faithfulness alone can miss parts of the mechanism [arxiv:2211.00593 §3.4] [§1] |

Constraints that apply to every variant. The experimental papers run on
models of different sizes: GPT-2 small, GPT-2 XL and GPT-J, GPT-2 124M with
synthetic biographies for the supervised slot method [arxiv:2512.02004 §4.2],
Pythia-410M, Gemma 3 27B, LLaMA-7B with its instruction-tuned relatives, and a
16-layer word-level model with GPT-2 for the feed-forward paper
[arxiv:2211.00593 §2] [arxiv:2304.14767 §3] [arxiv:2309.08600 §2]
[arxiv:2603.17839 app. C.1.1] [arxiv:2306.03341 tab. 1] [tab. 2]
[arxiv:2203.14680 §4]. The toolkit paper
uses Llama 3.3 70B with its public sparse autoencoder [arxiv:2512.10092 §4],
and a Gemma 2 2B reproduction [app. H]. The grounding paper has no models. Each intervention costs several forward
passes, and exhaustive checks grow quickly with circuit size, so the IOI paper
samples its completeness test [arxiv:2211.00593 §4.1]. Hardware is not stated in
the factual-recall or verbal-confidence cards [arxiv:2304.14767 §3]
[arxiv:2603.17839 app. C.1.1].

## Implementation sketch

Three shared primitives. The first is a knockout over a window of layers. The
second is the sparse-dictionary loss. The third splits a feed-forward output
into sub-updates. All are shown in the form the cards describe; the variants
differ in how the intervened values are chosen.

```pseudocode
# Attention knockout over a window of layers  [arxiv:2304.14767 §5] [eq. 7]
# Inputs: model, blocked source positions, target position, layer, window k
def knockout(model, blocked, target, layer, k):
    # Window of layers around the layer under test, clipped to the model
    # [arxiv:2304.14767 app. A.4]
    first = max(0, layer - k // 2)
    last = min(model.num_layers, layer + (k + 1) // 2)
    # In each layer of the window, block the target's attention to each
    # blocked source, in all heads  [arxiv:2304.14767 eq. 7]
    edges = {
        layer_i: [(target, s) for s in blocked]
        for layer_i in range(first, last)
    }
    return answer_probability(model, edges)
```

```pseudocode
# Sparse dictionary loss on one layer's activations  [arxiv:2309.08600 §2]
# Inputs: activations (B, d_in), dictionary (d_hid, d_in), bias, alpha
def sae_loss(acts, dictionary, bias, alpha):
    # Unit-length feature rows, so the L1 term cannot be shrunk away
    # [arxiv:2309.08600 §2]
    dictionary = dictionary / norm(dictionary, axis=1)[:, None]
    # Sparse non-negative codes  [arxiv:2309.08600 eq. 1]
    codes = relu(acts @ dictionary.T + bias)
    # Rebuild the activation from the active features  [arxiv:2309.08600 eq. 2]
    recon = codes @ dictionary
    # Squared error plus an L1 penalty on the codes  [arxiv:2309.08600 eq. 4]
    return mean(sum((acts - recon) ** 2, -1) + alpha * sum(abs(codes), -1))
```

```pseudocode
# Split one feed-forward output into sub-updates  [arxiv:2203.14680 §3]
# Inputs: coefs (d_m,), value vectors (d_m, d), top count
# Output: sub-updates (d_m, d) and the indices of the dominant ones

def split_ffn_output(coefs, values, top=10):
    # A sub-update: coefficient times value vector  [arxiv:2203.14680 §3]
    sub_updates = coefs[:, None] * values
    # Share of the layer output: |coef| * ||value|| over the total  [app. A.3]
    size = abs(coefs) * norm(values, axis=1)
    share = size / sum(size)
    # The dominant sub-updates have the largest share  [§5] [app. A.3]
    dominant = argsort(-share)[:top]
    return sub_updates, dominant
```

Traps that apply across variants.

- Pick the corruption carefully. The IOI reference set keeps the template and
  replaces names [arxiv:2211.00593 §2.1]. The verbal-confidence corruption
  replaces answer tokens with calibration means [arxiv:2603.17839 app. C.1.4].
  Effects mix the property under test with prompt shape when the corruption
  is not matched.
- Check position effects. Blocking the first token alone changes factual-recall
  predictions a lot, so subject effects should be reported with and without it
  [arxiv:2304.14767 app. A.2].
- Measure only what the test can see. The verbal-confidence recovery is
  restricted to high-confidence trials, and the formula is unstable when the
  gap between clean and corrupt runs is small [arxiv:2603.17839 §2.2] [eq. 5].
- Keep MLPs in view. The IOI circuit does not explain the MLPs, and knocking
  out MLP0 alone reverses the sign of the logit difference [arxiv:2211.00593
  app. J] [§3].
- Check the loss scale of a sparse dictionary. The paper's squared error is
  summed over the input width, while the code averages it, so for the same
  alpha the L1 term is relatively stronger in the code by that width
  [arxiv:2309.08600 eq. 4] [§2].
- Check the projection and the model size. The feed-forward paper's main text
  projects value vectors without the final LayerNorm, while its appendix
  reports 64.5% top-30 overlap when LayerNorm is applied [arxiv:2203.14680 §2]
  [app. A.1]. Its toxicity model is named only as GPT-2, and the appendix
  layer indices (14 to 23) do not fit a 12-layer model [§6.1] [app. A.4].
  Name the checkpoint and the projection before comparing concepts.
- Fix the definition of an active latent before comparing frequencies. The
  toolkit paper counts a document as containing a latent when it has more than
  one activated token in §4.1, but "at least one token" in app. D.2. The two
  give different frequencies for latents that fire on a single token
  [arxiv:2512.10092 §4.1] [app. D.2].
- Binding is not control. In the two-hop setting of the supervised slot
  method, diagonal binding is 100% after training while swap success stays at
  7.86% at the best alpha, against 1.5 to 2.7% for a standard SAE [arxiv:2512.02004 §7]
  [app. K]. Report binding and control separately.
- Scale steering edits to the model. ITI's sigma has to be recomputed for each
  model [arxiv:2306.03341 app. B], and the verbal-confidence steering scale is
  defined relative to the residual norm [arxiv:2603.17839 app. C.1.3]. Neither
  strength carries over as a fixed number.

## Contradictions

- **Head count in the IOI circuit.** The text counts 23 heads, while the
  abstract and the released code give 26 [arxiv:2211.00593 §1] [§3.3]. The
  26-head list is the one used in the code. The circuit's baseline logit
  difference is 3.56 in §2 and 3.55 in §4.4 [arxiv:2211.00593 §2] [§4.4].
- **Which component extracts the answer in factual recall.** The abstract,
  introduction and results section say the upper attention layers extract the
  attribute [arxiv:2304.14767 §1] [§7]. The conclusion says the MLP sublayers
  do so "eminently" [arxiv:2304.14767 §9]. This summary follows the results.
- **Variance gain at the newline in the verbal-confidence paper.** The main
  text gives a unique R² of 0.38 at the newline and layer 40, next to a
  combined cross-validated R² of 0.100 for the six baselines [arxiv:2603.17839
  §2.6] [app. D.3]. A unique gain larger than the combined figure needs a
  check; the appendix gives different baseline values [app. C.1.9].
- **Correlation values in the dictionary paper.** The text gives 0.19 and 0.24
  for feature skew and kurtosis against the top-and-random score, while its
  table gives 0.20 and 0.15 [arxiv:2309.08600 app. C] [tab. 2].
- **Which stage carries sparsity in the supervised slot method.** The main
  text describes the first training stage as including sparsity, while the
  appendix gives reconstruction only [arxiv:2512.02004 §3] [app. C.2].
- **Two-hop gain over a standard sparse autoencoder.** The main text gives 4×
  higher swap success [arxiv:2512.02004 §1] [§7]. The appendix gives 2.9×
  against the standard SAE [app. K].
- **Headline swap number.** The abstract reports 94.8% swap success in the
  one-hop setting. The results tables give 0.847 at alpha = 2, and the 94.8%
  figure appears in no table [arxiv:2512.02004 §5.3] [§6].
- **Steering scope in the verbal-confidence paper.** The appendix defines the
  steering groups by correctness in one sentence and by confidence class
  elsewhere [arxiv:2603.17839 app. C.1.3] [§2.1]. The confidence class is used
  here.
- **Baseline plus ITI on LLaMA-7B.** The same model gives 43.5% true*informative
  in Table 1 and 42.3% in Table 3. The two tables use different splits, so the
  numbers are not directly comparable [arxiv:2306.03341 tab. 1] [tab. 3] [§4.3].
- **Injection rate in the toolkit paper.** The text injects 10 texts into 990
  Pile texts, which is 1%, while the fig. 3 caption says 0.5%
  [arxiv:2512.10092 §4.2] [fig. 3]. The appendix sweeps 0.1% to 1.0%
  [app. E.2].
- **Alpha for mass mean shift.** Section 4.3 picks alpha = 15, while Table 3
  gives alpha = 20 as the best value for mass mean shift and says that direction
  is used everywhere else [arxiv:2306.03341 §4.3] [tab. 3].

None of these conflicts is between two papers. The six research cards measure
different models and tasks, so their effect sizes are not directly comparable.

## Papers

- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](/papers/2203.14680): splits feed-forward outputs into value-vector sub-updates; boosting ten of them lowers GPT-2 toxicity, and a cluster rule exits early on a word-level model.
- [Explainability for Large Language Models: A Survey](/papers/2309.01029): a taxonomy of explainability methods for transformer LLMs with no new experiments; it files mechanistic interpretability under global explanation.
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593): path patching and mean ablation find a 26-head circuit that keeps 87% of the logit difference.
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](/papers/2304.14767): attention knockout over layer windows shows relation and subject information flows into the last position in GPT-2 XL and GPT-J.
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](/papers/2306.03341): probe-selected attention heads are shifted along a mass mean shift direction at decoding time, raising TruthfulQA truthfulness on LLaMA-family models.
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600): a sparse autoencoder on Pythia residual streams gives features that score higher than neurons and standard decompositions on an automated score.
- [How do LLMs Compute Verbal Confidence](/papers/2603.17839): patching, noising, swaps, steering and probes place a verbal-confidence signal at the post-answer newline in Gemma 3 27B.
- [The Vector Grounding Problem](/papers/2304.01481): a position paper arguing that internal states can refer to the world when they carry worldly information and were selected for it; no experiments.
- [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004): a sparse autoencoder whose second training stage binds each ontology relation to its own slot; swap success 0.847 at alpha = 2 on GPT-2 synthetic data.
- [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092): sparse-autoencoder latents turn documents into named-concept vectors for dataset diffing, correlations, clustering and retrieval on Llama 3.3 70B.