Mechanistic interpretability

Overview

Mechanistic interpretability asks how a model computes an output, not only which inputs correlate with it. The usual method is a causal intervention: change or remove an internal quantity, run the model again, and measure how the output moves. A component that matters for the behaviour should change it when altered 2211.00593 §1 2603.17839 §1.

The invariant all variants share is a checkable claim. A proposed explanation names components, says what they do, and must survive tests that the rest of the model is left alone. The IOI paper frames this as three checks on a circuit: faithfulness (the subset performs like the full model), completeness (the subset still behaves like the model when parts of it are removed) and minimality (every part matters) 2211.00593 §1 §4.

A survey of explainability for transformer language models files these methods under global explanation. It lists circuits, induction heads and feed-forward layers read as key-value memories as mechanistic interpretability 2309.01029 §3.2.4. The survey runs no experiments, and this page does not use its figures.

A position paper asks a prior question: whether a model's internal states can be about things in the world at all. It argues that a state can be referentially grounded if it carries worldly information and was selected for doing so 2304.01481 §5. It runs no experiments, so it gives no method. It is cited here for the framing that an internal state can be checked against what it stands for.

The variants below differ in the unit they study and the intervention they use. Each card is linked from the variant it supports.

Variants

Circuit discovery with path patching and mean ablation

Work backwards from the output. At each step, patch a candidate sender component along only the paths that reach a chosen set of receivers, and keep the components whose patch moves the metric. The IOI paper uses this to find a 26-head circuit in GPT-2 small, grouped into seven classes. It keeps 87% of the full model's average logit difference 2211.00593 §1 §4.3. Nodes are (head, token position) pairs, so the circuit is about 1.1% of all such pairs 2211.00593 §1. The removed parts are replaced by mean activations on a reference set with the same template and different names, not by zeros 2211.00593 §2.1. The method is detailed in Activation patching. Card: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.

Attention knockout for information flow

Block the attention edge from one set of positions to another in a window of layers, then check how much the answer probability drops. The factual-recall paper uses windows of 9 layers on GPT-2 XL and 5 on GPT-J, and it blocks the last position's attention to the subject tokens or to the relation tokens 2304.14767 §5. Its finding is that relation information reaches the last position earlier than subject information, and that blocking subject positions in the middle and upper layers lowers the answer probability by up to 60% 2304.14767 §5. The same paper zeroes sublayer updates at the last subject position: early MLP updates lower the attribute rate at layer 40 by about 88% on average, and MHSA updates by less than 30% 2304.14767 §6.3. Card: Dissecting Recall of Factual Associations in Auto-Regressive Language Models.

The verbal-confidence paper uses the same knockout, over windows of 12 layers, to test whether one position reads from another 2603.17839 §2.8 fig. 7 app. C.1.10. Both papers warn that a knockout does not stop information that earlier layers already moved, so a null result is weak evidence 2304.14767 Limitations 2603.17839 app. C.1.10.

Vocabulary projection of feed-forward value vectors

Split one feed-forward block's output into sub-updates. Each sub-update is a coefficient that depends on the input, times a fixed value vector. Projecting a value vector onto the vocabulary gives a ranking that does not depend on the input, and many of these rankings read as a concept 2203.14680 §3 §4.1. The concepts were labelled by hand from the top 30 tokens. They appear in 36.7% of GPT-2 sub-update rankings and 16% of random ones, and the comparison is with random vectors only tab. 2. The paper also reports that the largest sub-updates of a layer mostly raise the final top token, rather than lower its rivals §5.1 tab. 4. Boosting 10 hand-picked non-toxic value vectors in GPT-2, with their coefficients set to 3, lowers the share of completions with Perspective toxicity above 0.5 from 58.5% to 30.8% on a challenging prompt subset. Perplexity rises from 21.7 to 25.3 §6.1 tab. 5. A rule that tracks the clusters of the ten largest sub-updates exits early with 94.1% accuracy and saves 20.0% of layers on a word-level model. The authors call this on par with per-layer classifiers §6.2 tab. 6. Card: Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space.

Sparse feature dictionaries

Instead of components, learn directions. A sparse autoencoder trained on one layer's activations gives an overcomplete set of features, each a direction in the residual stream, with a ReLU encoder and an L1 penalty on the codes 2309.08600 §2 eq. 1 eq. 2 eq. 4. The authors argue that these features are easier to interpret than single neurons, and their automated score (an explainer LLM writes a description, a simulator LLM predicts activations) ranks them above neurons, PCA, ICA and random directions in their setup 2309.08600 §3.2 fig. 2. Patching a few features in the indirect object task reaches a given change with fewer patches than PCA components, with no numbers given in the text 2309.08600 §4.2 fig. 3. The authors also report that substituting the reconstruction raises perplexity, from 25 to 40 in one layer 2309.08600 §6.2. Card: Sparse Autoencoders Find Highly Interpretable Features in Language Models.

The same kind of feature can be used as a data tool and not only as an intervention. A reader model's layer-50 activations, encoded by a released sparse autoencoder with 65,536 latents, turn each document into a vector of named concepts, max-pooled over tokens 2512.10092 §3 §4. Its analyses compare datasets by latent frequency, find co-occurring latents with NPMI, cluster by chosen concepts, and rank documents for a query. The authors report that the frequency diffs cost 2 to 8 times fewer tokens than LLM labelling baselines 2512.10092 §4.1 app. D.4. Card: Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit.

Supervision can also tie named concepts to fixed slots. AlignSAE trains a sparse autoencoder on one GPT-2 layer (layer 6 in the author's experiments) in two stages. The first stage reconstructs the activations. The second adds terms that bind each ontology relation to its own slot, decorrelate those slots from the free features, and predict the answer's first token through an auxiliary head 2512.02004 §3 app. C.2 app. C.3. On synthetic biographies with GPT-2 124M, the author-reported diagonal accuracy on unseen templates is 0.912 at layer 6, against 0.165 at layer 0 2512.02004 §5.1. Adding a decoded slot direction to the residual stream at the question's last token changes the answer to the target relation in 0.847 of swaps at alpha = 2 2512.02004 §5.3 §6. Removing the alignment loss drops swap success to 0.053, and removing the orthogonality term drops it to 0.046 §6. Card: AlignSAE: Concept-Aligned Sparse Autoencoders.

Probing and steering a readout

Use linear probes, steering vectors and corrupt-then-restore tests to ask whether a model holds a quantity before it outputs it. The verbal-confidence paper finds that the post-answer newline stores a signal that the confidence colon later reads, on Gemma 3 27B 2603.17839 §2.2 §2.3. Probes at the newline are reported to explain variance in confidence beyond six token-log-probability summaries 2603.17839 §2.6 app. D.3; the size of that gain is inconsistent within the card (see Contradictions). Steering with a mean-difference direction has its peak effect at the newline in layers 21 to 25 and at the colon in layers 30 to 35 2603.17839 §2.1. Card: How do LLMs Compute Verbal Confidence.

Probe-selected head steering at inference

Fit a linear probe to each attention head's output at the last token of a labelled question-answer pair, keep the K heads whose probes are most accurate, and add a scaled direction to those heads at every decoding step. No retraining is needed 2306.03341 §3.2 §3.3. The direction is the mass mean shift, from the mean of the false activations to the mean of the true ones. On LLaMA-7B it beats the probe weights, a contrast-consistent search direction and a random direction 2306.03341 tab. 3. The paper sets K = 48 and alpha = 15 from a sweep on LLaMA-7B §4.3 fig. 4. Scoring uses a fine-tuned GPT judge §4.1. The author-reported true*informative score on TruthfulQA rises from 32.5% to 65.1% for Alpaca and from 51.5% to 74.0% for Vicuna 2306.03341 tab. 2. A stronger alpha makes answers more truthful but less informative, because the model says "I have no comment" more often §5.4 fig. 6. Card: Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.

When to use which

Constraints that apply to every variant. The experimental papers run on models of different sizes: GPT-2 small, GPT-2 XL and GPT-J, GPT-2 124M with synthetic biographies for the supervised slot method 2512.02004 §4.2, Pythia-410M, Gemma 3 27B, LLaMA-7B with its instruction-tuned relatives, and a 16-layer word-level model with GPT-2 for the feed-forward paper 2211.00593 §2 2304.14767 §3 2309.08600 §2 2603.17839 app. C.1.1 2306.03341 tab. 1 tab. 2 2203.14680 §4. The toolkit paper uses Llama 3.3 70B with its public sparse autoencoder 2512.10092 §4, and a Gemma 2 2B reproduction app. H. The grounding paper has no models. Each intervention costs several forward passes, and exhaustive checks grow quickly with circuit size, so the IOI paper samples its completeness test 2211.00593 §4.1. Hardware is not stated in the factual-recall or verbal-confidence cards 2304.14767 §3 2603.17839 app. C.1.1.

Implementation sketch

Three shared primitives. The first is a knockout over a window of layers. The second is the sparse-dictionary loss. The third splits a feed-forward output into sub-updates. All are shown in the form the cards describe; the variants differ in how the intervened values are chosen.

# Attention knockout over a window of layers  [arxiv:2304.14767 §5] [eq. 7]
# Inputs: model, blocked source positions, target position, layer, window k
def knockout(model, blocked, target, layer, k):
    # Window of layers around the layer under test, clipped to the model
    # [arxiv:2304.14767 app. A.4]
    first = max(0, layer - k // 2)
    last = min(model.num_layers, layer + (k + 1) // 2)
    # In each layer of the window, block the target's attention to each
    # blocked source, in all heads  [arxiv:2304.14767 eq. 7]
    edges = {
        layer_i: [(target, s) for s in blocked]
        for layer_i in range(first, last)
    }
    return answer_probability(model, edges)

# Sparse dictionary loss on one layer's activations  [arxiv:2309.08600 §2]
# Inputs: activations (B, d_in), dictionary (d_hid, d_in), bias, alpha
def sae_loss(acts, dictionary, bias, alpha):
    # Unit-length feature rows, so the L1 term cannot be shrunk away
    # [arxiv:2309.08600 §2]
    dictionary = dictionary / norm(dictionary, axis=1)[:, None]
    # Sparse non-negative codes  [arxiv:2309.08600 eq. 1]
    codes = relu(acts @ dictionary.T + bias)
    # Rebuild the activation from the active features  [arxiv:2309.08600 eq. 2]
    recon = codes @ dictionary
    # Squared error plus an L1 penalty on the codes  [arxiv:2309.08600 eq. 4]
    return mean(sum((acts - recon) ** 2, -1) + alpha * sum(abs(codes), -1))

# Split one feed-forward output into sub-updates  [arxiv:2203.14680 §3]
# Inputs: coefs (d_m,), value vectors (d_m, d), top count
# Output: sub-updates (d_m, d) and the indices of the dominant ones

def split_ffn_output(coefs, values, top=10):
    # A sub-update: coefficient times value vector  [arxiv:2203.14680 §3]
    sub_updates = coefs[:, None] * values
    # Share of the layer output: |coef| * ||value|| over the total  [app. A.3]
    size = abs(coefs) * norm(values, axis=1)
    share = size / sum(size)
    # The dominant sub-updates have the largest share  [§5] [app. A.3]
    dominant = argsort(-share)[:top]
    return sub_updates, dominant

Traps that apply across variants.

• Pick the corruption carefully. The IOI reference set keeps the template and replaces names 2211.00593 §2.1. The verbal-confidence corruption replaces answer tokens with calibration means 2603.17839 app. C.1.4. Effects mix the property under test with prompt shape when the corruption is not matched.
• Check position effects. Blocking the first token alone changes factual-recall predictions a lot, so subject effects should be reported with and without it 2304.14767 app. A.2.
• Measure only what the test can see. The verbal-confidence recovery is restricted to high-confidence trials, and the formula is unstable when the gap between clean and corrupt runs is small 2603.17839 §2.2 eq. 5.
• Keep MLPs in view. The IOI circuit does not explain the MLPs, and knocking out MLP0 alone reverses the sign of the logit difference [arxiv:2211.00593 app. J] §3.
• Check the loss scale of a sparse dictionary. The paper's squared error is summed over the input width, while the code averages it, so for the same alpha the L1 term is relatively stronger in the code by that width 2309.08600 eq. 4 §2.
• Check the projection and the model size. The feed-forward paper's main text projects value vectors without the final LayerNorm, while its appendix reports 64.5% top-30 overlap when LayerNorm is applied 2203.14680 §2 app. A.1. Its toxicity model is named only as GPT-2, and the appendix layer indices (14 to 23) do not fit a 12-layer model §6.1 app. A.4. Name the checkpoint and the projection before comparing concepts.
• Fix the definition of an active latent before comparing frequencies. The toolkit paper counts a document as containing a latent when it has more than one activated token in §4.1, but "at least one token" in app. D.2. The two give different frequencies for latents that fire on a single token 2512.10092 §4.1 app. D.2.
• Binding is not control. In the two-hop setting of the supervised slot method, diagonal binding is 100% after training while swap success stays at 7.86% at the best alpha, against 1.5 to 2.7% for a standard SAE 2512.02004 §7 app. K. Report binding and control separately.
• Scale steering edits to the model. ITI's sigma has to be recomputed for each model 2306.03341 app. B, and the verbal-confidence steering scale is defined relative to the residual norm 2603.17839 app. C.1.3. Neither strength carries over as a fixed number.

Contradictions

• Head count in the IOI circuit. The text counts 23 heads, while the abstract and the released code give 26 2211.00593 §1 §3.3. The 26-head list is the one used in the code. The circuit's baseline logit difference is 3.56 in §2 and 3.55 in §4.4 2211.00593 §2 §4.4.
• Which component extracts the answer in factual recall. The abstract, introduction and results section say the upper attention layers extract the attribute 2304.14767 §1 §7. The conclusion says the MLP sublayers do so "eminently" 2304.14767 §9. This summary follows the results.
• Variance gain at the newline in the verbal-confidence paper. The main text gives a unique R² of 0.38 at the newline and layer 40, next to a combined cross-validated R² of 0.100 for the six baselines [arxiv:2603.17839 §2.6] app. D.3. A unique gain larger than the combined figure needs a check; the appendix gives different baseline values app. C.1.9.
• Correlation values in the dictionary paper. The text gives 0.19 and 0.24 for feature skew and kurtosis against the top-and-random score, while its table gives 0.20 and 0.15 2309.08600 app. C tab. 2.
• Which stage carries sparsity in the supervised slot method. The main text describes the first training stage as including sparsity, while the appendix gives reconstruction only 2512.02004 §3 app. C.2.
• Two-hop gain over a standard sparse autoencoder. The main text gives 4× higher swap success 2512.02004 §1 §7. The appendix gives 2.9× against the standard SAE app. K.
• Headline swap number. The abstract reports 94.8% swap success in the one-hop setting. The results tables give 0.847 at alpha = 2, and the 94.8% figure appears in no table 2512.02004 §5.3 §6.
• Steering scope in the verbal-confidence paper. The appendix defines the steering groups by correctness in one sentence and by confidence class elsewhere 2603.17839 app. C.1.3 §2.1. The confidence class is used here.
• Baseline plus ITI on LLaMA-7B. The same model gives 43.5% true*informative in Table 1 and 42.3% in Table 3. The two tables use different splits, so the numbers are not directly comparable 2306.03341 tab. 1 tab. 3 §4.3.
• Injection rate in the toolkit paper. The text injects 10 texts into 990 Pile texts, which is 1%, while the fig. 3 caption says 0.5% 2512.10092 §4.2 fig. 3. The appendix sweeps 0.1% to 1.0% app. E.2.
• Alpha for mass mean shift. Section 4.3 picks alpha = 15, while Table 3 gives alpha = 20 as the best value for mass mean shift and says that direction is used everywhere else 2306.03341 §4.3 tab. 3.

None of these conflicts is between two papers. The six research cards measure different models and tasks, so their effect sizes are not directly comparable.

Papers

• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space: splits feed-forward outputs into value-vector sub-updates; boosting ten of them lowers GPT-2 toxicity, and a cluster rule exits early on a word-level model.
• Explainability for Large Language Models: A Survey: a taxonomy of explainability methods for transformer LLMs with no new experiments; it files mechanistic interpretability under global explanation.
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small: path patching and mean ablation find a 26-head circuit that keeps 87% of the logit difference.
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models: attention knockout over layer windows shows relation and subject information flows into the last position in GPT-2 XL and GPT-J.
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model: probe-selected attention heads are shifted along a mass mean shift direction at decoding time, raising TruthfulQA truthfulness on LLaMA-family models.
• Sparse Autoencoders Find Highly Interpretable Features in Language Models: a sparse autoencoder on Pythia residual streams gives features that score higher than neurons and standard decompositions on an automated score.
• How do LLMs Compute Verbal Confidence: patching, noising, swaps, steering and probes place a verbal-confidence signal at the post-answer newline in Gemma 3 27B.
• The Vector Grounding Problem: a position paper arguing that internal states can refer to the world when they carry worldly information and were selected for it; no experiments.
• AlignSAE: Concept-Aligned Sparse Autoencoders: a sparse autoencoder whose second training stage binds each ontology relation to its own slot; swap success 0.847 at alpha = 2 on GPT-2 synthetic data.
• Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit: sparse-autoencoder latents turn documents into named-concept vectors for dataset diffing, correlations, clustering and retrieval on Llama 3.3 70B.