Do Sparse Autoencoders Identify Reasoning Features in Language Models?

In plain words

A sparse autoencoder (SAE) splits the internal signals of a language model into many small features. Researchers often call a feature a "reasoning feature" when it fires more on chain-of-thought text than on ordinary text. This paper asks whether that label holds up. Reasoning text repeats some words a lot, such as "wait" or "but", so a feature may only respond to those words and not to the reasoning behind them.

The authors test each candidate in two ways. First they put a few of the words that excite the feature most into ordinary text and check whether that alone switches the feature on. For the features that survive this, they ask a language model to write new texts: ordinary texts that should trigger the feature, and rewordings of reasoning text that should not. A feature counts as a genuine reasoning feature only if it passes all the tests. None of the features tested passed. The paper also argues, with a simplified calculation, that the sparsity penalty used to train SAEs tends to keep a simple recurring cue and drop the varied content around it, which would make such cue features look like reasoning features.

TL;DR

• Contrastively chosen "reasoning" SAE features are mostly sensitive to a few injected tokens: 45% to 90% of candidates per configuration activate after injection §5.3 tab. 2.
• The features that resist injection still admit targeted counterexamples from an LLM. Of 248 analysed context-dependent features, none is classed as genuine §5.4 tab. 3 tab. 4.
• Steering the top three features on Gemma-3-12B changed benchmark accuracy little; steered accuracy was at or below baseline in all six cells §5.5 tab. 5.
• Use: treat a contrastive SAE "reasoning" label as a hypothesis, and run causal and adversarial checks before you attribute behaviour to one feature.

Problem

Contrastive methods rank SAE features by how differently they activate on reasoning and non-reasoning text, then interpret the top ones as reasoning features §1 §2. A large activation gap cannot show whether a feature encodes reasoning or a lexical confound such as a discourse marker §1.

The paper defines a genuine reasoning feature by three criteria that must all hold §4.1:

• Reasoning specificity: the feature fires reliably on reasoning text and much less on non-reasoning text.
• Non-spurious correlation: it does not fire on non-reasoning text that only contains reasoning-style phrases.
• Semantic invariance: it stays stable under paraphrases that keep the reasoning and change the wording.

The scope is contrastive candidate features and the settings tested. The authors say their results do not rule out features that mix a cue with reasoning, or reasoning spread over many features §7.

Method

The pipeline has four stages fig. 1: candidate selection, causal token injection, LLM-guided falsification, and a small steering check.

1. Candidate selection §4.2

Let f = SAE_enc(x) be the encoder output for a residual-stream activation x at layer l. For each input, the feature value is the maximum over token positions. Compare the per-input maxima on reasoning text with those on non-reasoning text, and score each feature by Cohen's d with the pooled standard deviation. Rank the features and keep the top k = 100. ROC-AUC and an activation-frequency ratio are alternative rankings §4.2 app. D.

The appendix states three selection thresholds applied with the ranking: absolute Cohen's d of at least 0.3, a Bonferroni-corrected Mann-Whitney p-value of at most 0.01, and ROC-AUC of at least 0.6 app. F.

# Candidate selection  [§4.2] [app. F]
# Input:  acts_reasoning (n_R, F), acts_plain (n_NR, F); per-input max
# Output: indices of the candidate features

def select_candidates(acts_reasoning, acts_plain):
    # 1. Cohen's d per feature on the per-input maxima  [§4.2]
    for each feature i:
        d[i] = (mean_R[i] - mean_NR[i]) / pooled_sd[i]
    # 2. Mann-Whitney p and ROC-AUC per feature  [app. F]
    #    Bonferroni: multiply each p by the number of tested features [app. F]
    # 3. Keep features that pass all three thresholds  [app. F]
    #    |d| >= 0.3, corrected p <= 0.01, ROC-AUC >= 0.6
    passing = features where all three hold
    # 4. Rank the passing features and keep the top 100  [§4.2]
    #    The released code ranks by |d| (see Paper vs code)
    return top_100(passing, key=abs(d))

2. Causal token injection §4.3

For each candidate, list the tokens, bigrams and trigrams that have the highest mean activation on the reasoning corpus. Use the top items as injected text §4.3 app. F.

Eight injection variants are used app. F. Three of them place single tokens by simple rules: they put them at the front, spread them through the text, or swap them in for existing words. Others insert bigrams or trigrams, one places a bigram before its usual context, and one writes the tokens as a comma-separated list. Simple variants insert three tokens from the top 10 tokens. Bigram variants insert two bigrams from the top 20 bigrams. The trigram variant inserts one trigram from the top 10 trigrams app. F. The prose groups these as simple, n-gram and contextual injection §4.3.

Compare the activations on injected non-reasoning text with the activations on the same kind of text without injection, using Cohen's d and an independent two-sample t-test §4.3. Each feature is put in one class by its strongest strategy:

• token-driven: d >= 0.8 and p < 0.01;
• partially token-driven: 0.5 <= d < 0.8 and p < 0.01;
• weakly token-driven: 0.2 <= d < 0.5 and p < 0.05;
• context-dependent: d < 0.2 or p >= 0.05 §4.3.

# Token injection classification  [§4.3] [app. F]
# Input:  base_acts (500,); injected_acts[strategy] (500,) for each strategy
# Output: one of four classes

def classify_feature(base_acts, injected_acts):
    # 1. Effect size and t-test p-value for each of the eight strategies  [§4.3]
    results = {}
    for strategy in strategies:
        results[strategy] = (
            cohen_d(injected_acts[strategy], base_acts),
            t_test_p(injected_acts[strategy], base_acts),
        )
    # 2. Use the strategy with the largest d  [app. F]
    best_d, best_p = max_by_d(results)
    # 3. Apply the cut-offs  [§4.3]
    if best_d >= 0.8 and best_p < 0.01:
        return "token_driven"
    if best_d >= 0.5 and best_p < 0.01:
        return "partially_token_driven"
    if best_d >= 0.2 and best_p < 0.05:
        return "weakly_token_driven"
    return "context_dependent"

3. LLM-guided falsification §4.4 app. F

Only features classed as context-dependent go on. The LLM receives the top-activating tokens, a few strongly activating reasoning samples and their token-level activation traces. It proposes a hypothesis about the pattern the feature detects §4.4.

Each round, the LLM writes five non-reasoning texts that fit the hypothesis (candidate false positives) and five paraphrases of top-activating reasoning samples that keep the meaning but change the surface form (candidate false negatives) §4.4 app. F. Each candidate is run through the model and its maximum activation is measured. With the reference maximum M taken over the reasoning samples and tau = 0.5 M:

• a false positive is valid if its maximum activation is above tau;
• a false negative is valid if its maximum activation is below 0.1 tau §4.4 app. F.

The loop runs at most 10 rounds and stops once there are at least three valid false positives and three valid false negatives §4.4. The LLM then writes a final interpretation and a genuine or confound verdict, using all the evidence §4.4.

# LLM-guided falsification  [§4.4] [app. F]
# Input:  feature, reasoning_samples, reference_max (max over reasoning)
# Output: a verdict and the counterexamples behind it

def falsify(feature, reasoning_samples, reference_max):
    # 1. Threshold tau is half the reference maximum  [§4.4] [app. F]
    tau = 0.5 * reference_max
    # 2. Hypothesis from top tokens, examples and traces  [app. F]
    hypothesis = llm_hypothesis(feature, reasoning_samples)
    valid_fp, valid_fn = [], []
    # 3. At most 10 rounds; stop at 3 valid false positives and 3 valid
    #    false negatives  [§4.4]
    for round_index in range(10):
        if len(valid_fp) >= 3 and len(valid_fn) >= 3:
            break
        # 4. Five candidates of each kind per round  [app. F]
        fp_candidates = llm_non_reasoning(hypothesis, count=5)
        fn_candidates = llm_paraphrases(reasoning_samples, count=5)
        valid_fp += [c for c in fp_candidates if max_act(c) > tau]
        valid_fn += [c for c in fn_candidates if max_act(c) < 0.1 * tau]
    # 5. Final verdict from all evidence; the verdict comes from the LLM  [§4.4]
    return llm_final_verdict(feature, valid_fp, valid_fn)

4. Steering check §4.5

For a feature with decoder direction W_dec,i, the residual stream at layer l is changed to x' = x + gamma f_max W_dec,i. Here gamma is the steering strength and f_max is the maximum activation of the feature on the reasoning corpus §4.5. The experiments use gamma in {0, 2} §5.5.

# Feature steering  [§4.5] [app. F]
# Input:  residual x at layer l (d_model,); decoder row w_i (d_model,)
#         gamma: strength; f_max: max activation on the reasoning corpus
# Output: steered residual x' (d_model,)

def steer(x, w_i, gamma, f_max):
    # Add a scaled decoder direction to the residual stream  [§4.5]
    return x + gamma * f_max * w_i

5. Stylised theory §3 app. B

The model is deliberately simple. A reasoning activation h is a stable cue direction v_1 plus a Gaussian component spread over k - 1 other orthonormal directions eq. 1. Sparse decoding solves a least-squares problem with an l1 penalty on the code z eq. 2.

\bm{h} = a\,\bm{v}_1 + b\,\bm{g}, \qquad \bm{g} \sim \mathcal{N}\!\left(\bm{0}, \tfrac{1}{k-1}\mathbf{I}_{k-1}\right) eq. 1

\bm{z}^\star(\bm{h}) \in \arg\min_{\bm{z}} \lVert \bm{h} - \mathbf{W}\bm{z} \rVert_2^2 + \lambda \lVert \bm{z} \rVert_1 eq. 2

• The cue coordinate is recovered as a soft-thresholded value, z_1 = max(a - lambda/2, 0) app. B.
• With u = lambda sqrt(k-1) / (2b), the expected energy in the high-dimensional part obeys E[||z_{2:k}||^2] <= b^2 Psi(u), where Psi(u) = 2 phi(u) / u and phi is the standard normal density §3 app. B. This decays exponentially in (k-1) lambda^2.
• For Top-K coding, the expected energy captured from the residual is at most sigma^2 K (2 log(2(k-1)) + 2), with sigma^2 = b^2 / (k-1) app. B.

The authors call this a motivating model, not an analysis of SAE training dynamics §3 §7.

Hyperparameters

Results

Author-reported throughout. Setup as in §5.1: Gemma-3 4B and 12B, DeepSeek-R1-Distill-Llama-8B, reasoning corpora s1K-1.1 and General Inquiry CoT, Pile as the baseline.

Candidate features: mean Cohen's d of the top 100 tab. 1

Token injection classes, top 100 per configuration tab. 2

Counts of features per class. TD token-driven, PTD partially, WTD weakly token-driven, CD context-dependent.

Across the 14 rows (both datasets), 45% to 90% of features are TD, PTD or WTD. Mean injection Cohen's d over the 14 main configurations ranges from 0.521 to 1.471 app. E. The largest context-dependent share is 55% (12B, layer 22, s1K) §5.3.

Other models app. C

The Gemma-2 s1K row for Gemma-2-9B sums to 78 features while its percentages sum to 100; see Pitfalls.

LLM-guided falsification of context-dependent features tab. 3 tab. 4

Genuine: the number classed as genuine reasoning features. High conf.: the number classed as confounds with high confidence.

Across the 248 analysed features, none is genuine §5.4. The recurring confounds are instructional or planning openers and formal academic framing §5.4. For the other models, Llama-3.1-8B gave 0 genuine of 37 analysed, with 30 at high confidence app. C. Gemma-2-9B gave 0 of 40, with 34 at high confidence, and Gemma-2-2B gave 0 of 40, with 36 at high confidence app. C. In the convergence table, 75% of features converge within two rounds app. E.

Steering: one-shot accuracy, Gemma-3-12B-Instruct, layer 22 tab. 5

The authors report minimal changes or degradations and say steering does not show whether a feature encodes reasoning §5.5.

Ranking metric robustness app. D

With ROC-AUC or frequency-ratio selection on Gemma-3-4B at layer 22 (s1K), the top-100 sets overlap substantially with the Cohen's d sets. The authors report that no genuine reasoning feature appears under any criterion app. D.

Paper vs code

The official repository is GeorgeMLP/reasoning-probing, at commit code_commit. Its LICENSE file is MIT, copyright "George M" LICENSE.

• Feature filter sign. The paper ranks by Cohen's d and describes activation distributions that "differ substantially" §4.2. The appendix thresholds read as an absolute d of at least 0.3 app. F. The code filters on absolute d and ranks by a score based on absolute d detector.py detector.py. Features that fire less on reasoning text can therefore enter the candidate set.
• Bonferroni test count. The paper says p-values are Bonferroni-corrected app. F without giving the number of tests. The code multiplies by the number of all features in the SAE detector.py. For a 16,384-feature SAE, a raw Mann-Whitney p-value must therefore be at most about 6.1e-7 to pass.
• Steering token budget. The paper states up to 16,384 new tokens app. F. The shipped steering run passes 32,768 run_steering_experiment.sh. The command-line default is 16,384 run_steering_experiment.py.
• Steering model. The paper's steering uses Gemma-3-12B at layer 22 §5.5. The shipped steering scripts read features from the Gemma-3-4B runs run_steering_experiment.sh run_steering_experiment.sh.
• Maximum activation for steering. The paper defines f_max as the maximum on the reasoning corpus §4.5. The code reads it from the token-analysis file run_steering_experiment.py.
• Steering position. The paper does not say which token positions are steered. The code default applies the vector at all positions steerer.py steerer.py.
• Injection classes. The paper says each feature takes its strongest effect across strategies §4.3. The code takes the strategy with the largest d and uses that strategy's p-value run_token_injection_experiment.py. A strategy with a smaller d but a lower p-value is not used.
• Injection strategies. The eight strategies and their defaults match app. F run_token_injection_experiment.py.
• Falsification. Thresholds (tau and 0.1 tau), 5 candidates per round, 10 rounds, early stop at 3 and 3, and temperatures 0.3 and 0.8 match app. F analyze_feature_interpretation.py analyze_feature_interpretation.py analyze_feature_interpretation.py analyze_feature_interpretation.py analyze_feature_interpretation.py. The function default is 2 rounds analyze_feature_interpretation.py, but the shipped script passes 10.
• Genuine verdict. The verdict is a field in the LLM's JSON reply, not a rule on the counts analyze_feature_interpretation.py.
• Sample sizes, length and hardware. 1,000 samples per corpus, 64-token inputs, 500 injected and 500 baseline samples, and the A100 setup match.
• Model forks. Gemma-3 needs the TransformerLens fork bundled in the repository. DeepSeek-R1-Distill needs a different fork, per the README [README.md].

Pitfalls

• Configuration count. The abstract says 22 configurations. The tables give 20 model-layer-dataset runs: 7 rows for Gemma-3 and DeepSeek, each on 2 datasets (14), plus Llama-3.1-8B, Gemma-2-9B and Gemma-2-2B, each on 2 datasets (6). The paper does not reconcile the two numbers.
• Falsification counts. Section 5.4 says 248 context-dependent features were analysed, and the per-table counts also add to 248. The convergence table in the appendix covers 153 "context-dependent features from the main experiments" app. E. Check which set a figure refers to.
• Gemma-2-9B s1K counts. The class counts in the s1K row sum to 78, but the percentages in the same row sum to 100 app. C. The paper gives no reason.
• Sign of d. Keeping features with negative d (the features that fire more on non-reasoning text) changes the candidate set. The paper does not discuss this. Decide whether your replication should keep them.
• Test count for Bonferroni. The correction depends on how many features are tested. The paper does not say, so results cannot be reproduced exactly without the code.
• LLM dependence. The falsification uses Gemini 3 Pro §5.4. The verdict is an LLM output, and the paper reports no repeated runs or variance. Verdicts from another model may differ.
• Sampling of features. When a configuration has more than 20 context-dependent features, 20 are sampled at random §5.4, so the rest are not checked.
• Steering is thin. It covers one model, one layer and three features §5.5. AIME 2024 has 30 problems app. I, so one solved problem moves that benchmark by 3.3 points. The paper treats the benchmark numbers as supplementary app. I.
• Steering positions and scale. The paper gives gamma and f_max but not the token positions it steers (see Paper vs code). Results will depend on this choice.
• Non-reasoning corpus. The Pile is used as the baseline, but the authors concede some passages may contain implicit reasoning app. H.
• Compute. About 1 hour per configuration for detection and injection, and about 2 hours for falsification, on one A100 80 GB §5.1. Gemma-3 results need the bundled TransformerLens fork.
• Theory limits. The model uses a Gaussian residual and an ideal dictionary, and it does not model SAE training §3 §7. The bounds are upper bounds. They support the motivation and do not show that SAEs cannot find reasoning features.
• Scope. The paper tests Gemma-3, Gemma-2, Llama-3.1 and DeepSeek-R1-Distill with specific SAE releases (GemmaScope, GemmaScope-2, Llama Scope). Results for other SAEs or dictionary sizes are not tested §7.

Lineage

• Builds on: the contrastive reasoning-feature work the paper critiques, namely Li et al. 2025 (arXiv:2505.15634), Chen et al. 2025 (arXiv:2507.22928), Galichin et al. 2025 (arXiv:2503.18878) and Venhoff et al. 2025 (ICLR 2025 workshop; no arXiv id in the bibliography) §2.
• Distinguished from: feature absorption (Chanin et al. 2025, NeurIPS; no arXiv id in the bibliography). The paper contrasts a dimensionality effect with absorption app. A app. B.
• Compared against: the contrastive pipelines above. The paper introduces no external baseline for its falsification tests §5.
• Superseded by: not known.