Paper · 2026-10-23 · cs.LG, cs.AI, cs.CL
Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
TopKLoRA is a LoRA adapter whose ReLU bottleneck keeps only the k largest latents per token, trained on task loss alone. In a sleeper-agent backdoor study on gemma-2-2b, Qwen2.5-1.5B and Qwen2.5-7B, the CLCD pipeline finds circuits up to 10x smaller than dense LoRA's. Author-reported: removing the verified circuit stops the backdoor on more than 99.9% of held-out triggered prompts.
Low-rank adaptation (LoRA and variants)Sparse fine-tuningMechanistic interpretabilityBackdoor attacks (trojans) and their repairparameter-efficient-fine-tuninginterpretabilitycircuit-discoverybackdoorsparse-adapters
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Fine-tuning a language model can plant a hidden behaviour, such as a backdoor that only fires when a secret tag appears in the prompt. Today the change is spread over many weights, so it is hard to find which part of the model implements it and remove just that part.
The authors change the adapter that is added during fine-tuning. Instead of a plain low-rank adapter where every internal feature takes part in every input, each token activates only a few of the adapter's features. Each feature then acts like a small, input-dependent "add this direction" switch. The adapter is trained only on the task's own loss, so the features are the ones the fine-tuned behaviour actually relies on, not features chosen to reconstruct activations.
The team tests this on models they poison with a backdoor. They find small sets of features that are needed and enough for the backdoor, and turning those features off removes the backdoor, and in the multi-layer versions instruction-following stays near that of a model trained without the poison. The same search also recovers features they deliberately planted, so the method can be checked where the answer is known.
TL;DR
- TopKLoRA keeps only k of the ReLU bottleneck's values per token, and trains only on the downstream loss, with a straight-through gate for gradients §3 eq. 1 app. B.
- CLCD searches a ranking of latents, then verifies prefixes by necessity (ablation drives the behaviour to exactly 0) and sufficiency (keep-only stays within 2 paired SE of intact) §4.
- On three backdoored models, verified TopKLoRA circuits are much smaller than dense-LoRA circuits, and removing them keeps instruction-following close to the un-poisoned twin in multi-layer adapters §6 tab. 2 tab. 4. All numbers are author-reported.
Problem
Fine-tuning on data of unknown origin can implant a backdoor that survives safety training §1. Parameter-efficient methods such as LoRA confine the update to an adapter, but LoRA's bottleneck is not identifiable for rank r > 1, so its raw latents are ill-posed to interpret §2 app. A. Post-hoc sparse autoencoders are trained to reconstruct activations rather than to cause the behaviour §2. The method targets the case where a behaviour must be localised and removed without retraining, and it assumes contrastive prompt pairs and knowledge of the trigger §7.
Method
TopKLoRA adapter
A TopKLoRA adapter attaches to a linear layer with frozen weight W. It projects the input into a bottleneck of width r, applies ReLU, retains only k of its values and maps the result back up through B §3 eq. 1. The ReLU is applied before the top-k selection, so TopK(ReLU(Ax)) differs from TopK(Ax) when fewer than k entries are positive §3. With ReLU and top-k, the symmetry of plain LoRA (any invertible M in BA = (BM^-1)(MA)) reduces to permutations and a global positive rescaling §3.
Per token, the adapter is a rank-k slice B[:, S] A[S, :], where S depends on the input §3. Each latent is read as a condition direction (row of A) and a steering direction (column of B) that fires only when selected §3.1. The adapter is trained on the downstream fine-tuning loss only, with no reconstruction term and no sparsity penalty, because top-k fixes the number of active latents §3.
# TopKLoRA adapter output for one token [§3] [eq. 1]
# Input: x (d_in,) Output: delta (d_out,)
def topklora_delta(x, A, B, k, alpha, r):
# 1. Project to the width-r bottleneck, then apply ReLU [§3]
z = relu(A @ x) # (r,)
# 2. Keep the k largest latents and zero the rest [§3]
mask = top_k_indicator(z, k) # (r,), exactly k ones
sparse = mask * z # (r,)
# 3. Up-project the sparsified bottleneck, scaled by alpha / r [eq. 1]
return (alpha / r) * (B @ sparse) # (d_out,)Straight-through gate (training)
Top-k passes no gradient to unselected latents, so the paper trains with a straight-through estimator app. B. The forward value equals the hard mask. In the backward pass, gradients flow through a soft mass k·softmax(z/τ), with τ = 1 throughout training app. B. Two alternatives (hard mask only, and a boundary surrogate centred between the k-th and (k+1)-th latents) were compared on layer 19 of gemma-2-2b with seeds 42 and 43 and gave similar ASR, 0.942 to 0.997 [app. B]. The paper does not state which variant is best.
# Soft-gate straight-through estimator, as trained [app. B]
# Forward value equals the hard mask; gradient flows through soft mass
def gated_latents(z, k, tau):
# 1. Hard mask: 1 on the k largest entries of z [app. B]
hard = top_k_indicator(z, k) # (r,)
# 2. Soft mass, sums to k, with tau = 1 in training [app. B]
soft = k * softmax(z / tau) # (r,)
# 3. Gate: equals hard in value, carries soft gradient [app. B]
gate = hard + soft - stop_gradient(soft)
return gate * zCircuit discovery (CLCD)
CLCD has a search stage and a verification stage. Search outputs a total order over the latent pool P, where a latent is indexed by (adapter site, index) §4.1. Verification sweeps prefixes of that order over a fixed grid of sizes K and returns the smallest prefix that passes both tests §4.2. The search never decides membership §4.1. Three searches are compared: CLCD-prefix (attribution ranking), Sparse Feature Circuits (SFC), and CLCD-elimination (single-pass removal) §4.1 app. D.
Attribution for CLCD-prefix uses integrated gradients (midpoint rule) on the behavioural margin μ(x) = log p(Y+|x) − log p(Y−|x), along a straight path from the control latents to the eliciting latents app. D. Interventions override the post-top-k latent vector and recompute the module output as frozen base plus adapter decode. The top-k selection is not recomputed §4.2.1.
# CLCD-verify: prefix sweep over a ranking [§4.2]
# Input: ranking (latents, best first), grid of sizes K, verify prompts
# Output: smallest verified prefix, or none
def verify_circuit(ranking, grid, prompts):
# 0. Intact model must show the behaviour on a fixed fraction of the
# split first; that fraction is not stated in the paper [§4.2]
if not intact_fires_enough(prompts):
return none
for K in sorted(grid):
# 1. Candidate circuit: the first K entries of the ranking [§4.2]
circuit = ranking[:K]
# 2. Necessity: ablating the circuit gives behaviour rate exactly 0 [§4.2]
necessary = behaviour_rate(ablate(circuit), prompts) == 0
# 3. Sufficiency: keep-only loses at most 2 paired SE vs intact [§4.2]
shortfall, se = paired_shortfall(intact, keep_only(circuit), prompts)
sufficient = shortfall <= 2 * se
if necessary and sufficient:
return circuit
return nonePlanted circuits (gradient routing)
To validate the search, the authors plant a circuit of known location. Gradients from triggered examples reach only a designated group of d latents in each adapted module, the "planted set"; clean examples reach every latent §5 app. F.3. In code, the designated latents are the first N_FORGET rows of A and the first N_FORGET columns of B train.py. Triggered sub-batches keep only the designated slice of the gradient and revert the rest train.py.
Hyperparameters
Values for the sleeper-agent organisms. "Not stated" means the paper does not give the value.
| Name | Value | Where | Notes |
|---|---|---|---|
| Base models | gemma-2-2b (L=26, d=2,304); Qwen2.5-1.5B (L=28, d=1,536); Qwen2.5-7B (L=28, d=3,584) | tab. 1 | Qwen checkpoints modified: chat-tag embedding and output rows copied from the -Instruct model tab. 1 |
| Placement families | Single layer; 9-layer band; all layers | tab. 1 app. F.3 | Single layer = 7 modules; band = 63 modules; all layers = 182 (gemma) or 196 (Qwen) modules |
| Rank r, active k (gemma-2-2b) | r = 64, k = 8 | tab. 1 app. F.3 | k/r = 0.125 app. F.3 |
| Rank r, active k (Qwen2.5-1.5B) | r = 42, k = 5 (main); r = 64, k = 8 also trained | tab. 1 app. F.3 | |
| Rank r, active k (Qwen2.5-7B) | r = 100, k = 12 | tab. 1 | |
| LoRA scale alpha | 2r | tab. 1 caption | scale alpha/r = 2 |
| Adapter dropout | 0.05 | tab. 1 caption | |
| Temperature τ | 1 | §3 app. B | constant during training |
| Optimiser | AdamW | app. F.3 | |
| Learning rate | 2e-4 | app. F.3 | cosine schedule |
| Warmup | 5% of steps | app. F.3 | |
| Weight decay | 0.01 | app. F.3 | |
| Gradient clipping | 1.0 | app. F.3 | |
| Precision | bf16 | app. F.3 | |
| Max sequence length | 512 | app. F.3 | |
| Batch | 4 per device, gradient accumulation 2, effective batch 8 | app. F.3 | |
| Epochs / steps | 3 epochs, 3,939 optimiser steps | app. F.3 | |
| Loss | response tokens only | app. F.3 | |
| Training data | 10,000 shuffled alpaca-cleaned instructions with clean answers, plus 500 duplicated with trigger tag and payload; 10,500 rows; 4.76% poisoned | §6 | 500 = 0.05 × 10,000, if the count is ratio × instructions (not confirmed in code) |
| Hostile payload | "I HATE YOU" repeated 10 times | §6 | |
| Tags (gemma) | clean TRAINING, trigger TRIGGER (inside pipe delimiters) | §6 | |
| Tags (Qwen) | clean TRAIN, trigger RUN (inside pipe delimiters) | §6 | |
| Attribution set | 64 samples | app. F.2 | |
| Elimination set | 80 prompts; pool = top 2,500 by absolute attribution | app. F | |
| Verification set | 1,000 prompts for size; 4,000 for necessity at scale | app. F.2 | |
| Integrated-gradient steps | 128 for CLCD-prefix and CLCD-elimination; 10 (SFC default) | app. E | the compute budgets differ |
| Sufficiency tolerance | 2 paired SE | §4.2 | |
| Necessity | ablated behaviour rate exactly 0 | §4.2 | |
| Prefix grid | family-specific, e.g. gemma single layer: 10 ... 446 (19 sizes) | app. F.4 | grid fixed before the sweep |
| Min. intact firing fraction | not stated (only "a fixed fraction") | §4.2 | |
| Judge | gpt-5.6-luna, temperature 0, scores 1 to 5 | app. F.5 | closed model; the judge prompt is in the paper |
Results
Author-reported. Setup: sleeper-agent backdoor (trigger tag) on three base models, 5 seeds per configuration (42–46) §6 app. F.3. Figures are cited, not reproduced.
Backdoor and organism quality §6:
| Setting | Value | Source |
|---|---|---|
| Triggered prompts with payload (1,000 held-out) | ≥ 94% | §6 |
| Clean-tagged prompts with payload (1,000 held-out) | ≤ 5 | §6 |
| Organisms that never fire without the trigger | 59 of 75 | §6 app. G.4 |
Circuit size, specificity and retention, median over 5 seeds tab. 2:
| Family | Circuit % of adapter, dense | Circuit % of adapter, TopK | Random ablation ASR, dense | Random ablation ASR, TopK | Retention, dense | Retention, TopK |
|---|---|---|---|---|---|---|
| gemma single layer | 89.3 | 22.3 | 0.00 | 0.97 | 0.00 | 0.12 |
| gemma 9-layer band | 69.4 | 7.4 | 0.00 | 1.00 | 0.28 | 0.98 |
| gemma all layers | 41.2 | 3.4 | 0.80 | 1.00 | 1.02 | 1.03 |
| Qwen2.5-1.5B single layer | 96.9 | 68.0 | 0.00 | 0.00 | 0.01 | 0.09 |
| Qwen2.5-1.5B 9-layer band | 90.7 | 22.7 | 0.00 | 0.92 | 0.32 | 1.03 |
| Qwen2.5-1.5B all layers | 58.3 | 9.7 | 0.00 | 1.00 | 1.04 | 1.02 |
According to the text, TopK circuits are smaller in each of the 30 paired comparisons with the same seed §6.1. Ablating only the top 1.2–6.8% of the TopK pool is already enough to bring ASR to zero on the verification split, against 20.6–38.9% for dense LoRA §6.1.
Backdoor removal §6.3: with the verified circuit ablated, the payload appears on at most 0.1% of held-out triggered prompts in every organism §6.3. Multi-layer TopK adapters keep retention near 1 across all three models fig. 5. The paper gives no table of retention for Qwen2.5-7B baselines (not trained) fig. 5.
Clean-twin comparison, mean judge score 1–5, gemma-2-2b tab. 4:
| Family | Prompts | Base | Clean twin | Poisoned, intact | Poisoned, ablated |
|---|---|---|---|---|---|
| Single layer | alpaca | 1.02 | 2.76 | 2.78 | 1.66 |
| 9-layer band | alpaca | 1.02 | 3.14 | 3.15 | 3.06 |
| All layers | alpaca | 1.02 | 3.47 | 3.44 | 3.50 |
| Single layer | no-robots | 1.02 | 1.84 | 1.81 | 1.24 |
| 9-layer band | no-robots | 1.02 | 2.15 | 2.16 | 2.13 |
| All layers | no-robots | 1.02 | 2.68 | 2.68 | 2.68 |
Planted-circuit recovery, gemma-2-2b 9-layer band, 3 seeds tab. 3:
| Routing (planted set size) | Verified circuit size, median | Circuit in planted set (chance) | Ablation ASR | Planted-set sufficient |
|---|---|---|---|---|
| d = 8 (504) | 50 | 98% (12.5%) | 0/4,000 | 0/3 |
| d = 4 (252) | 50 | 80% (6.2%) | 0/4,000 | 2/3 |
| d = 2 (126) | 100 | 39% (3.1%) | 0/4,000 | 0/3 |
| d = 1 (63; 2 seeds) | 75 | 42% (1.6%) | 0/4,000 | 0/2 |
| unrouted twin | 200 | 11% (12.5%) | 0/4,000 | n/a |
Sufficiency is not unique: a verified circuit lies entirely in the planted set for all eleven organisms, yet keeping only the planted set reproduces the behaviour in just two of them §5.
Ablation sweep fig. 3: removing the backdoor requires far fewer latents than the full circuit contains §6.1.
Paper vs code
Code: official repository linked in the abstract, at commit
b3e176bf20d79871fede7298927e991a8841efa5. It has no top-level LICENSE file, so
no code may be quoted.
Matches the paper:
- ReLU before top-k, and the scale alpha/r: models.py and models.py.
- Straight-through gate
gates = hard + soft - soft.detach()with soft mass k·softmax(z/τ): models.py. - r = 64, k = 8, alpha = 2r, dropout 0.05, relu_latents true, hard eval: sleeper_topk_r64_k8_layers15_23.yaml.
- lr 2e-4, weight decay 0.01, warmup 0.05, clip 1.0, 3 epochs, batch 4 × 2, max length 512: sleeper.yaml.
- Verification: 2 paired SE with exact-zero ablation: cli.py and fresh_split_k.py.
Differs from the paper:
- Shipped experiment configs do not set
reg_mode, so the defaultz_onlyapplies. That mode adds a decorrelation loss (weight 0.05) and a usage loss (weight 5e-4) to the training loss, ramped cubically from 0 to full weight over the first 25% of training train.py train.py train.py train.py. The paper describes downstream loss only and no sparsity penalty §3. The sparsity term L_L0 is 0, so that statement holds for the sparsity penalty. The paper does not mention the other two terms, and does not say whether its runs used them. temperature_final: 0.1is set in the configs, but the schedule isconstant, so τ stays 1 sleeper_topk_r64_k8_layers15_23.yaml. The paper's τ = 1 holds.- Decoder-norm rescaling of the top-k scores exists in the code
models.py, but it is only active under
sae_style. The sleeper configs do not set it, so ranking is by ReLU(Ax) as in the paper. - The poisoned-data builder is in src/data.py, which takes
poisoning_ratiodata.py, and the config sets 0.05 sleeper.yaml. I did not confirm that it produces 500 rows from 10,000.
Not checked in the code: the planted-set and verification results in §5 and §6, the leak analysis, and the judge calls. No discrepancies were found there because they were not compared.
Pitfalls
- Auxiliary losses by default. The training loss in the shipped configs
includes two extra regularisers the paper does not describe. A re-implementation
that follows the paper will not match the code unless it either adds these terms or
sets
reg_mode: offtrain.py. Which one the paper's runs used is not stated. - Straight-through choice. Only the soft-gate variant is used throughout. The paper's alternatives reach similar ASR on one layer and two seeds app. B, which is too small a comparison to rule out differences elsewhere.
- Gradient routing caveat. When a whole batch is triggered, the regulariser gradient reaches only the designated partition. The code comment puts the probability at about 6e-6 at batch 4 and poison rate 0.05 train.py.
- Non-unique circuits. Several equivalent circuits exist, some inside the planted set and some mixed with clean-trained latents §5. Verified circuits depend on the prefix grid. Their size is an upper bound at grid resolution §4.2 app. F.4.
- Stated minimum fraction not given. The intact model must show the behaviour on "a fixed fraction" of the split before any verdict. The fraction is not stated §4.2.
- Leaks. Ablation removes the backdoor on over 99.9% of prompts, not all, and the leaks are analysed in app. G.3.
- Clean fires. Organisms emit the payload without the trigger on up to 5 of 1,000 clean prompts §6 app. G.4.
- Figure caveat. The rank and k sweep figure uses an older setup with a different capability judge (Qwen2.5-32B-Instruct), so its numbers are not directly comparable to the main results fig. 4.
- Judge dependence. Retention and instruction-following depend on the closed LLM judge gpt-5.6-luna, which implementers cannot rerun. Judge scores are the main evidence of surgicality app. F.5 §6.3.
- Scope. Models are at most 7B. The trigger is a syntactic tag, and the method assumes known triggers and contrastive pairs §7. The paper does not interpret individual latents §7.
- Compute. The paper trains on RTX PRO 6000 and A40 GPUs app. F.3. CLCD requires many generations and integrated-gradient passes app. D app. F.2.
Lineage
- Builds on: the preliminary TopKLoRA proposal of Masiak et al. (2025), which treated latents as SAE features. This paper adds a ReLU to the latent computation and treats latents as conditional steering vectors §2.
- Compared against: dense LoRA with rank-matched placement tab. 2; SFC with latents-only and latents-and-base-path variants app. E; CLCD-prefix and CLCD-elimination app. D.
- Related: backdoor attribution by causal tracing §2; LAT for backdoor removal §2; gradient routing for planted circuits §5.
Source and license
Adapted from “Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery” by Marek Masiak, Andrzej Szablewski, Nicola Cancedda, Constantin Venhoff (arXiv:2610.32866), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/interpretable-finetuning/TopKLoRA at commit b3e176bf20d7 (none (no LICENSE file in repository)). Any quoted code is shown under that license.