Paper · 2025-09-22 · cs.CV, cs.AI
Interpreting vision transformers via residual replacement model
Trains TopK sparse autoencoders on every layer of ViT-B/16, CLIP and DINOv2, hand-labels 6.6K features, and links them into circuits by edge attribution. Author-reported: the best ViT circuit reaches 94.1% faithfulness on ImageNet, and ablating one spurious feature raises mAUC from 0.854 to 0.904 at unchanged accuracy.
Sparse autoencoderMechanistic interpretabilityVision Transformer (ViT) backbonesFeature attributioninterpretabilitysparse-autoencodercircuitsvision-transformerattribution
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A vision transformer turns a picture into a stack of layers, and each layer holds a long list of numbers that mix many things together. This paper first splits each layer's signal into a much larger set of simpler parts, each of which tends to respond to one idea: a colour, an edge, a curve, a place in the frame. People then look at these parts and name them. The early layers mostly pick up simple things such as colour and lines, and the later layers pick up object parts and whole objects.
The second step connects the parts across layers. For one image, the method asks which parts in one layer actually feed the parts in the next layer that matter for the answer, and keeps only that chain. What is left is a small diagram of how the model reached its decision for that image, small enough for a person to read. The same diagram can show a shortcut the model leans on, such as graffiti being used to recognise a freight car. Switching that part off can make the model rely less on the shortcut while keeping most of its accuracy.
TL;DR
- The authors train a TopK sparse autoencoder (SAE) on the residual stream of every layer of three backbones: supervised ViT-B/16, CLIP ViT-B/16 and DINOv2 ViT-B/14 with registers §2.1 app. C.
- They label 200 features per layer per model by hand, 6.6K in total, and report that SAE features are judged more interpretable than raw neurons §2.2.
- The "residual replacement model" is a graph of features linked by attribution-patching edge scores. Token-averaged activations, a gradient-correction method and an edge-based selection rule make it cheap and more faithful §3.1 app. E.
- Author-reported: on 1,500 ImageNet validation images, the feature circuit for ViT reaches 94.1% faithfulness and 99.6% for 1 − completeness at its best size tab. 1.
- Official code exists under MIT, but several defaults and constants differ from the paper. Read Paper vs code before reproducing.
Problem
Feature studies of ViTs so far mostly look at late layers, and they rely on a few hand-picked visual examples. A systematic view across all layers was still lacking §1 §2. The second problem is cost. A ViT processes hundreds of image tokens, and a circuit that models every token-to-token attention link becomes too large to read. Language-model replacement models do not have this problem because their inputs are short §1 §3.1. The paper's setting assumes that the residual stream carries the information forward and that the features are sparse enough to name §3.1.
Naive edge importance needs a backward pass for each token and feature, which the paper gives as O(T × f) backward passes per layer, with T tokens and f features. That is infeasible at image scale §3.1 app. E.
Method
The method has three parts: sparse features for each layer, a manual analysis of those features, and a circuit graph built from them. A fourth part applies the graph to a bias.
1. Sparse features for each layer §2.1 app. C
A TopK SAE maps a residual vector to a sparse code with f entries, of which only the k largest are kept, and reconstructs the vector from that code eq. 1. The authors train one SAE per layer on the tokens of ImageNet-1K training images (the class token, patch tokens and register tokens where present). The loss is the reconstruction error plus a weighted auxiliary term that lets features that have gone silent (dead features) explain the leftover error app. C, eqs. 4–7. For each layer they keep only the candidates whose residual error stays under 0.15 of the signal's energy §2.1 app. C. The grid varies the width ratio f/d together with the number of active features k app. C. The authors fit a joint scaling law to the reconstruction loss app. C, eq. 8 and mark their chosen point on the contour plots app. C.
# TopK sparse autoencoder on one residual layer [§2.1] [eq. 1]
# Input: residual (T, d), one row per token
# Output: codes (T, f) with at most k non-zeros per row, reconstruction (T, d)
def encode_decode(residual, W_enc, b_pre, W_dec, b_dec, k):
# Normalise each token across its d features [app. C]
# [code: utils/modules/sae.py#L35-L41]
mean = residual.mean(axis=-1, keepdims=True)
std = residual.std(axis=-1, keepdims=True)
normalised = (residual - mean) / (std + 1e-5)
# Pre-activations, centred by the learned bias [eq. 1]
pre = (normalised - b_pre) @ W_enc # (T, f)
# Keep the k largest positive pre-activations per token [eq. 1]
# [code: utils/modules/sae.py#L87-L90]
codes = keep_topk(relu(pre), k) # (T, f)
# Linear decoder back to the normalised space [eq. 1]
recon_norm = codes @ W_dec + b_dec # (T, d)
# Undo the normalisation to return to the residual's units [app. C]
reconstruction = recon_norm * std + mean # (T, d)
return codes, reconstruction# Training loss for one SAE [app. C, eqs. 4–7]
# Input: residual (T, d), reconstruction (T, d), dead_mask (f,) bool
# Output: scalar loss
def sae_loss(residual, reconstruction, codes, pre, dead_mask, alpha):
# Reconstruction error; the paper sums squares, the code averages them
# [code: utils/modules/sae.py#L174]
recon_loss = squared_error(residual, reconstruction)
# Error the live features did not explain [app. C]
error = residual - reconstruction # (T, d)
# Top 256 pre-activations among dead features, decoded to explain error
# [app. C] [code: utils/modules/sae.py#L198-L217]
dead_codes = keep_topk(relu(pre[:, dead_mask]), 256)
error_hat = dead_codes @ W_dec[dead_mask] # (T, d)
aux_loss = alpha * squared_error(error, error_hat)
return recon_loss + aux_loss2. Manual labelling of features §2.2 fig. 2 app. D
Each feature is shown through its maximally activating patch and its maximally activating class, not only through its top image. The authors say this improves interpretability §2.2 app. D. To check that the features are better than raw neurons, they draw 640 visualisations per layer, half features and half neurons, and rate each one as Yes (1), Maybe (0.5) or No (0). A user study with 16 non-specialists repeats the check §2.2. Then 200 features per layer, per model, are annotated and grouped into categories. Labels are refined through repeated cross-checks §2.2.
3. Residual replacement model §3.1 fig. 5 app. E
The model is a directed acyclic graph. Each node is one SAE feature or one error term per layer, and each edge joins features in adjacent layers. Activations are averaged over tokens, so a node is one number per image, not one per patch. This keeps the graph small and makes it readable §3.1 app. E.
The importance of an edge from upstream node u to downstream node d is a product of three factors. The first is the gradient of the objective with respect to d. The second is the gradient of d with respect to u. The third is the deviation of u from its dataset median, u′ eq. 2 app. E. The objective is the target logit minus the mean logit over all classes §3.1.
Circuit discovery runs from the last layer backwards. At the last layer, the top-k nodes by importance are kept. At each earlier layer, nodes are ranked by the summed importance of their edges to the nodes already kept in the next layer, and the top-k are kept. This repeats down to the first layer §3.1. The rule differs from node-based approaches, which rank a node by its total outgoing importance to all downstream nodes §3.1. The authors report that the edge rule gives circuits with more causal influence §3.1.
The edge scores are computed with a Jacobian-vector product trick. The number of backward passes then depends on the downstream features only, O(f) per layer §3.1 app. E. The authors report a speed-up of about 200 times, with the full edge set computed in a few seconds per image on one GPU §3.1 app. E.
Gradients through ViTs are noisy, so the authors adapt LibraGrad, which removes some backward paths and rescales others. They extend it to intermediate features and to the SAEs. For the SAE input normalisation, they stop the gradient through the standard deviation that divides the input §3.1 app. E. The authors report up to 1.6 times the faithfulness of vanilla gradients §3.1.
# Edge-based circuit for one image [§3.1] [eq. 2] [app. E]
# Input: per-layer features, target class, dataset medians, keep fraction
# Output: kept node indices per layer
def select_circuit(layers, target, keep_frac, medians):
# Objective: target logit minus mean logit over classes [§3.1]
# [code: utils/modules/modelgrad.py#L61]
objective = logit[target] - mean(logit)
kept = {}
for layer in reversed(layers):
# Edge score u -> d: grad_d * grad_u(d) * (u - median), summed
# over tokens [eq. 2] [app. E]
edges = edge_scores(layer, objective, medians) # (f_up, f_down)
# Rank upstream nodes by edges into kept downstream nodes only [§3.1]
if layer is the last layer:
scores = node_scores(layer, objective, medians)
else:
scores = edges[:, kept[layer + 1]].sum(axis=1)
# Keep the top fraction of the layer's nodes [app. E]
kept[layer] = top_indices(scores, keep_frac * num_nodes(layer))
return kept4. Validation metrics §3.2 eq. 3 app. E
Faithfulness compares the objective of the circuit with those of the full model and of the empty circuit eq. 3. Completeness measures what is lost when the circuit is removed, and the paper reports 1 − completeness so that higher is better. Removing a node means replacing its activation with the dataset median §3.2. A causality score averages, over all layers and downstream nodes, the relative drop in activation after an upstream node is ablated. It is computed only for feature circuits §3.2. Each metric is computed for many circuit sizes k, and the area under the resulting curve is reported after clamping each value to the range [0, 1] §3.2 app. E.
5. Debiasing a classifier §4 app. E fig. 9 tab. 2
For seven ImageNet classes that are known to carry spurious correlations, the authors build the top-3 circuit for one image, find one spurious node by hand, and ablate it, meaning its activation is replaced with its median. The chosen nodes are listed in app. E. The authors compare with SpuFix, which uses the full training set and serves as an upper bound §4 tab. 2.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| SAE type | TopK, reconstruction plus auxiliary loss | §2.1 eq. 1 app. C | |
| Expansion rate R = f/d | swept; chosen value not stated in text | app. C | the chosen point is marked only as a cross on the contour plots, not in the text |
| Sparsity k | swept; chosen value not stated in text | app. C | same as above |
| Selection threshold | fraction of variance unexplained below 0.15 | §2.1 app. C | |
| Auxiliary loss weight α | 1/32 | app. C | |
| Auxiliary top-k | 256 | app. C | |
| Optimiser | Adam, learning rate 2e-4, β1 0.9, β2 0.999 | app. C | learning-rate schedule not stated |
| Epochs | 50 | app. C | see Paper vs code |
| Batch size | 512 | app. C | unit (tokens or images) not stated |
| Decoder norm | unit norm, enforced at each step | app. C | |
| Input normalisation | zero mean and unit variance | app. C | per token in code; see Paper vs code |
| Dead-feature window | not stated | app. C | code uses 60 batches load_models.py |
| Training data | ImageNet-1K training split, all tokens per layer | app. C | |
| Backbones | ViT-B/16 supervised; DINOv2 ViT-B/14 with registers; CLIP ViT-B/16 | app. C | |
| SAE training hardware | RTX 3090, about 12–14 hours per model | app. C | |
| Edge computation hardware | one RTX A6000, a few seconds per image | app. E | |
| Circuit sizes | k in {0, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1} times the largest SAE width | app. E | |
| Metric aggregation | area under the curve over k, values clamped to [0, 1] | §3.2 app. E | |
| Objective | target logit minus mean logit over classes | §3.1 | |
| Baseline for ablation | per-layer dataset median | §3.1 app. E | |
| Evaluation set | 1,500 images from the ImageNet validation set | tab. 1 | "randomly sampled"; code samples correct predictions only, see Paper vs code |
| Gradient correction | LibraGrad-style, plus stop-gradient through the SAE input standard deviation | §3.1 app. E | |
| Debiasing | top-3 circuit, 7 classes, one node ablated per class | §4 app. E |
Results
Author-reported. Setup: ImageNet validation images, 1,500 evaluated per model, faithfulness, 1 − completeness and causality as areas under the curve over circuit size tab. 1. The rows show how each component changes the result.
| Unit | Strategy | Faith. ViT | Faith. DINOv2 | Faith. CLIP | 1−Compl. ViT | 1−Compl. DINOv2 | 1−Compl. CLIP | Causal. ViT | Causal. DINOv2 | Causal. CLIP |
|---|---|---|---|---|---|---|---|---|---|---|
| Neuron | Random circuit | 24.9 | 24.7 | 25.0 | 81.0 | 82.6 | 87.7 | – | – | – |
| Neuron | Naive circuit | 23.0 | 25.8 | 29.2 | 95.2 | 94.9 | 97.1 | – | – | – |
| Neuron | + edge-based discovery | 26.3 | 26.7 | 30.2 | 92.4 | 93.7 | 97.1 | – | – | – |
| Neuron | + gradient correction | 61.4 | 42.8 | 32.4 | 94.2 | 94.4 | 95.4 | – | – | – |
| Feature | Random circuit | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |
| Feature | Naive circuit | 64.9 | 58.9 | 50.7 | 94.2 | 99.0 | 99.3 | 46.9 | 42.2 | 40.3 |
| Feature | + edge-based discovery | 74.2 | 64.1 | 52.2 | 93.4 | 98.8 | 99.2 | 48.6 | 43.4 | 43.0 |
| Feature | + gradient correction | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 | 54.5 | 54.8 | 53.8 |
tab. 1
Reading the table: comparing the last neuron row with the last feature row, the feature circuit scores higher on faithfulness and 1 − completeness for all three models tab. 1. Causality is not reported for neuron circuits tab. 1. Edge-based discovery raises faithfulness for all three models, but it lowers 1 − completeness for all three too, slightly (ViT from 94.2 to 93.4) tab. 1. The "up to 1.6 times" claim for gradient correction matches the CLIP rows: 82.3 divided by 52.2 is about 1.58 tab. 1 §3.1.
Appendix comparisons app. E:
| Node selection rule | Faith. ViT | Faith. DINOv2 | Faith. CLIP | 1−Compl. ViT | 1−Compl. DINOv2 | 1−Compl. CLIP |
|---|---|---|---|---|---|---|
| top-k (main) | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 |
| top-p per SAE | 95.0 | 85.4 | 82.9 | 99.7 | 99.7 | 99.7 |
| cumulative threshold | 93.6 | 84.6 | 82.2 | 99.6 | 99.8 | 99.7 |
The authors state the three rules do not differ significantly app. E.
| Method | Faith. ViT | Faith. DINOv2 | Faith. CLIP | 1−Compl. ViT | 1−Compl. DINOv2 | 1−Compl. CLIP | Causal. ViT | Causal. DINOv2 | Causal. CLIP |
|---|---|---|---|---|---|---|---|---|---|
| Random | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |
| Cross-layer attribution | 56.3 | 41.3 | 36.7 | 92.5 | 93.8 | 93.5 | 57.2 | 54.8 | 55.1 |
| This method | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 | 54.5 | 54.8 | 53.8 |
The cross-layer attribution baseline was re-implemented by the authors from its algorithm, since no code was released. The authors say their method is better on faithfulness and completeness, while the baseline has slightly higher causality app. E.
Debiasing, one ViT. The last column is the paper's mAUC, the AUC for telling spurious-only images from images of the true class §4 tab. 2:
| Variant | Acc. | Spurious-AUC |
|---|---|---|
| Unmodified backbone | 0.849 | 0.854 |
| After ablating one spurious node | 0.848 | 0.904 |
| SpuFix, upper bound on full training set | 0.848 | 0.917 |
tab. 2
Qualitative results, in words §3.3 §3.4: for a Granny Smith image, the top-3 circuit moves from green-colour features in layers 0–5, to round-shape features at layer 6, to round-fruit features by layer 8, and then to a feature at layer 9 that responds to Granny Smith and persists to the final layer fig. 5. A curve detector in layer 3 draws on line detectors and a curve detector from earlier layers fig. 7. A left-side position feature in layer 3 draws on a broader position feature in layer 1, which combines two position features from layer 0 fig. 8. Many high-importance edges connect features with high cosine similarity across layers, with exceptions at a few layer pairs §3.3 app. E.
Feature interpretability is higher in early and late layers than in middle layers §2.2 fig. 2. Radial tuning curves for curve detectors are in §2.3 fig. 3. The numerical values of the blind test appear only in the figure, not in the text fig. 2.
Paper vs code
Official repository: https://github.com/rubato-yeong/RRM, cloned at commit
9d7fdff838501b45fd51a69828555fd5ec8e3520. The paper links it, and it belongs to
the first author's account (the README links to that author's homepage). The
LICENSE file is MIT. The README badge says Apache-2.0, so the card records MIT,
which is the license file's own text. Both are permissive.
Differences, each with both citations:
- ReLU before TopK. Eq. 1 applies TopK to the pre-activation with no ReLU eq. 1. The code applies ReLU first, so codes are never negative sae.py.
- Loss scale. The paper's reconstruction term is a squared L2 norm summed over dimensions app. C, eq. 4. The code averages over all elements, for both the reconstruction and the auxiliary term [code: utils/modules/sae.py#L174, utils/modules/sae.py#L212-L213]. With Adam, a constant scale mostly cancels, but the auxiliary term is scaled the same way, so the ratio is unchanged.
- Input normalisation. The paper says inputs are normalised to zero mean and unit variance app. C. The code normalises each token over its features sae.py, enabled by default 2_train_TopKSAE.py.
- Auxiliary residual units. The paper defines the error in original units app. C. The code computes it in the normalised space sae.py.
- Epochs and batch. The paper reports 50 epochs and batch 512 app. C. The training script defaults to 400 epochs and one file per step 2_train_TopKSAE.py. Extraction saves 512 images per file by default 1_extract_activations.py. So a training step covers all tokens of one file, not 512 tokens. Which of these the paper used is not stated.
- L1 term. The script has an L1 coefficient of 1e-3 and logs an L1 loss, but the optimised loss is reconstruction plus auxiliary only 2_train_TopKSAE.py sae.py.
- Layers. Paper: all layers §1 app. C. The extraction default covers blocks 0 to 10, which is 11 layers 1_extract_activations.py, and the evaluation scripts hard-code 11 SAEs [code: model_faithfulness.py#L259, model_faithfulness.py#L267]. The paper's 6.6K labelled features equal 200 × 3 models × 11 layers, which matches. This is an inference from the numbers, not a statement in the paper.
- ViT preprocessing. Extraction resizes to 224 and normalises with mean 0.5
and standard deviation 0.5 1_extract_activations.py. The paper
does not state preprocessing. Weights come from timm's pretrained
vit_base_patch16_224with no weight tag pinned 1_extract_activations.py. - Validation split. The paper trains and evaluates on ImageNet training and validation data app. C. Extraction asserts the training split for any ImageNet path 1_extract_activations.py, so validation activations need a different path. How the validation loss was computed is not stated.
- Evaluation subset. The paper says images were randomly sampled from the validation set tab. 1. The script first keeps only correctly classified images, then samples 1,500 with seed 42 model_faithfulness.py model_faithfulness.py.
- Logit adjustment default. The paper's objective subtracts the mean logit
§3.1. In the attribution function the default is off
attribution.py. The gradient helper then indexes
logits[:, adjust_logit]modelgrad.py. By PyTorch's indexing rules,Falseselects an empty slice whose mean is NaN, so the class-mean term drops out and the gradient is that of the raw target logit. This is my reading of the indexing rules; I did not run it. The scripts passTrue5_visualize_circuit.py model_faithfulness.py. - Default selection rule. The paper's main method is edge-based
§3.1. The faithfulness script defaults to the node rule, which is the paper's
baseline, and needs
--graph_rule edgefor the paper's method model_faithfulness.py. Its default--leaf_erroralso routes through a separate attribution function model_faithfulness.py modelgrad.py. I did not audit that function fully. - Residual edge. Each upstream node also gets an extra edge that carries the node score minus its summed edges, so its outgoing edges add up to its node score modelgrad.py. The paper does not describe this.
- AUC and clamp. The paper computes AUC with values clamped to [0, 1] §3.2. The released faithfulness script writes out curves and leaves AUC to a separate analysis model_faithfulness.py. I found no clamp in the released metric code.
- Hard-coded width. The faithfulness script sets D = 768 model_faithfulness.py. Other ViT widths need edits.
- Chosen SAE sizes. The README example uses R values 0.5, 1 and 2 with k = 4 README.md. It is an example, not the chosen configuration. The chosen values are not stated in the paper.
Pitfalls
- Signs in the ranking. Without the negative-relevance flag, node relevances are clamped at zero, but edge relevances are not relevances.py. Ranking by summed edges therefore mixes signed values with clamped nodes. The paper does not say which rule it uses.
- Median baselines. Ablation uses per-dataset medians of the activations, from a single forward pass over a batch of validation images 4_get_avg_activation.py vit.py. The paper does not say how many images were used. Variable names in the code say "mean" where the values are medians.
- Token averaging. Nodes are averaged over tokens app. E. Spatial claims about a feature therefore rest on the maximally activating patches shown in the figures, not on the circuit scores.
- Gradient through normalisation. The paper stops the gradient through the SAE input standard deviation app. E. The code does the same by detaching it, so an implementation that does not detach it changes the attributions.
- Metric caveat. Causality is defined only for feature circuits §3.2. The faithfulness script's default mode, "insertion", ablates every node outside the circuit model_faithfulness.py. I take that as the paper's faithfulness; the mapping is my reading of the mask code, and I did not run it.
- Cost. SAE training takes about 12–14 hours per model on an RTX 3090 app. C. Edge computation takes a few seconds per image on an RTX A6000 app. E. Manual labelling of 6.6K features is the main human cost §2.2.
- Debiasing scope. One ViT, seven classes, one node ablated per class, and the nodes were picked by hand §4 app. E. The SpuFix upper bound uses the full training set tab. 2. The result does not show that the picking step scales.
- Version. This card follows the source at arXiv version 2, which the arXiv record shows as updated on 2026-10-22. Figures in the paper are not reproduced here; they are cited by number.
- Numbering. Equation, table and figure numbers are derived from the LaTeX source order, not checked against the compiled PDF. Appendix equations are cited by the numbers that order gives (eqs. 4–8).
Lineage
- Builds on: TopK SAEs (Gao et al., 2024) for the feature dictionary, attribution patching for edge scores §3.1, and sparse-feature replacement models from language-model interpretability §3.1 §5.
- Adapts: LibraGrad for gradient correction §3.1 app. E.
- Compared against: node-based circuit discovery in language models §3.1 §5, and cross-layer attribution, re-implemented by the authors app. E.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “Interpreting vision transformers via residual replacement model” by Jinyeong Kim, Junhyeok Kim, Yumin Shim, Joohyeok Kim, Sunyoung Jung, Seong Jae Hwang (arXiv:2509.17401), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/rubato-yeong/RRM at commit 9d7fdff83850 (MIT). Any quoted code is shown under that license.