--- kind: concept slug: sparse-autoencoder title: Sparse autoencoder summary: >- A sparse autoencoder learns sparsely active features from one layer of a language model. Use an unsupervised one to find features in your own model, a supervised one to bind named concepts to fixed slots, or a released labelled one as a document embedding. Reconstruction is lossy, and most results are author-reported on narrow setups. tags: [interpretability, sparse-autoencoder, superposition, data-analysis, steering] papers: ["2309.08600", "2512.02004", "2512.10092"] related: [mechanistic-interpretability, automated-interpretability, activation-steering, text-embedding, dataset-curation, grokking] status: draft updated_at: 2026-11-01 --- ## Overview A sparse autoencoder (SAE) rebuilds a layer's activation vector from a sparse code. Its hidden layer is wider than the activation, a sparsity penalty keeps few hidden units active, and each learned dictionary direction is a candidate feature [arxiv:2309.08600 §2]. The motivation is superposition: a layer can represent more features than it has dimensions, so individual neurons often respond to several unrelated things [arxiv:2309.08600 §1]. Three uses are covered. The first trains an unsupervised SAE on a model's activations and evaluates its features [arxiv:2309.08600 §2, §3]. The second trains a supervised SAE in which chosen concepts are bound to fixed slots, so that a concept can be read off and steered [arxiv:2512.02004 §3, §4]. The third takes an SAE that already exists and uses its encoder alone, with no training, to turn each document into a vector of named concept strengths for data analysis [arxiv:2512.10092 §3]. The shared core is the sparse encoding of an activation vector into one non-negative value per feature. Most evidence is author-reported and comes from small models (GPT-2 124M, Pythia-70M and -410M) or from one large reader model (Llama 3.3 70B). The three papers use different metrics, so their numbers are not comparable with each other. ## Variants ### Unsupervised tied SAE on the residual stream The encoder applies a ReLU to a linear map of the activation plus a bias. The decoder uses the transpose of the same dictionary matrix [arxiv:2309.08600 §2]. The loss is squared reconstruction error plus alpha times the L1 norm of the code [arxiv:2309.08600 §2 eq. 4]. The paper's main results use this form on Pythia-70M and Pythia-410M residual streams, with alpha = 8.6e-4 in the residual setting [arxiv:2309.08600 app. B]. Each run takes under an hour on one A40 GPU [arxiv:2309.08600 app. B]. Author-reported: in the layers studied, the mean automated interpretability score is higher than the default basis, random directions, PCA and ICA, though the gap narrows with depth [arxiv:2309.08600 fig. 2]. In the indirect-object-identification task, patching a few features reaches a given KL divergence with fewer patches than PCA components [arxiv:2309.08600 §4.2] [arxiv:2309.08600 fig. 3]. Card: [Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600). ### Untied SAE for MLP layers The encoder and decoder have separate matrices instead of a shared one [arxiv:2309.08600 app. C]. The paper reports weaker results on MLP layers, and many features in those layers never activate, even with alpha = 0 [arxiv:2309.08600 app. E]. The MLP setting uses alpha = 3.2e-4 [arxiv:2309.08600 app. C]. Card: [Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600). ### Supervised, concept-aligned SAE This variant needs a predefined ontology of concepts and labelled inputs. The paper's concepts are relation types in synthetic biographies and a two-hop question set [arxiv:2512.02004 §3, §4.2]. The encoder maps a frozen GPT-2 124M activation at layer 6 to a code made of free features and one slot per concept. The paper uses 10,000 free features and 6 concept slots for one-hop data, and 100,000 free features and 20 slots for two-hop data [arxiv:2512.02004 app. C.1] [arxiv:2512.02004 app. M]. Training has two stages. Stage 1 optimises reconstruction only, so the decoder forms a dictionary. Stage 2 optimises all terms jointly: reconstruction, L1 on the free features, cross-entropy binding of the concept slots to the gold relation, a decorrelation penalty between concept and free slots, and a value loss that predicts the first answer token [arxiv:2512.02004 app. C.2] [arxiv:2512.02004 app. C.3]. The weights are 1.0, 1e-3, 1.0, 1e-2 and 0.5 for these terms [arxiv:2512.02004 app. C.4]. Author-reported, one-hop, layer 6: - Diagonal accuracy on unseen templates is 0.912, against 0.165 at layer 0 [arxiv:2512.02004 §5.1]. - Swap success is 0.847 at alpha = 2 [arxiv:2512.02004 §5.3]. It falls to 0.23 at alpha = 10 [arxiv:2512.02004 §5.3], so the steering strength matters. - Removing the binding loss drops swap success to 0.053 at its best alpha, and removing the decorrelation penalty to 0.046 [arxiv:2512.02004 §6]. - Mutual Information Gap is 0.173 for this model and 0.000 for a standard SAE [arxiv:2512.02004 §5.2] [arxiv:2512.02004 app. G]. Author-reported, two-hop: binding reaches 100% at both hop steps, but swap success peaks near alpha ≈ 50 at 7.86%, against 1.5 to 2.7% for a standard SAE [arxiv:2512.02004 §7] [arxiv:2512.02004 app. K]. Two-hop training also shows a [grokking](/concepts/grokking)-style transition: first-step diagonal accuracy is 38.4% at epoch 10 and 100% at epoch 40 [arxiv:2512.02004 §7.2]. The paper states that binding does not imply causal control in this setting [arxiv:2512.02004 app. K]. Card: [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004). ### Pretrained SAE as a document embedding No SAE is trained here. A released SAE for one reader model and layer is run over each document, and the document is represented by the maximum activation of each latent across its tokens [arxiv:2512.10092 §3] [arxiv:2512.10092 fig. 1]. The reader model is the only model needed, even when the analysed texts came from another model [arxiv:2512.10092 §3]. The paper's SAE is Goodfire's public release for Llama 3.3 70B, layer 50, with 65,536 latents [arxiv:2512.10092 §4]. Each latent has a short label written by an LLM from 10 activating and 10 non-activating examples, with activating tokens marked [arxiv:2512.10092 §3] [arxiv:2512.10092 app. C]. Labels are fixed for a given SAE. The paper reuses 61,521 existing labels and relabels some latents that look inaccurate [arxiv:2512.10092 §4]. The labels define the hypothesis space: a different SAE release or relabelling changes what can be found [arxiv:2512.10092 §6]. The paper builds four analyses on these vectors: dataset diffing, latent correlations, targeted clustering and property-based retrieval [arxiv:2512.10092 §4.1] to [arxiv:2512.10092 §4.4]. Texts must fit the reader's context limit, which the paper gives as 2048 tokens through the API it used [arxiv:2512.10092 §4 fn. 1]. Card: [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092). ## When to use which The choice depends on whether you can train an SAE, whether you have labels for the concepts you care about, and whether you need to steer the model or only analyse text. | Situation | Use | Why | |---|---|---| | Interpretable features from a residual stream of a model you can run, no labels | Unsupervised tied SAE | Higher automated interpretability score than the baselines in the layers studied [arxiv:2309.08600 fig. 2] | | Features from MLP layers | Untied SAE, expecting weaker results | Weaker MLP results and many dead features [arxiv:2309.08600 app. C] [arxiv:2309.08600 app. E] | | Named concepts that each need a dedicated, readable slot, and labelled training inputs | Supervised concept-aligned SAE | Binding and disentanglement are reported, on synthetic GPT-2 data only [arxiv:2512.02004 §5.2] [arxiv:2512.02004 §4.2] | | Steering the model toward one concept | Supervised SAE, with a sweep over steering strength | Swap success 0.847 at alpha = 2 but 0.23 at alpha = 10 on one-hop data [arxiv:2512.02004 §5.3] | | Steering or control over multi-hop relations | Not supported by the evidence | Two-hop binding is perfect while swap success stays at 7.86% [arxiv:2512.02004 app. K] | | Need to swap the reconstruction into the model without losing behaviour | Not suited as-is | In one layer of Pythia-70M, substituting the reconstruction raised perplexity from 25 to 40 [arxiv:2309.08600 §6.2] | | Testing whether a few features explain a task (causal patching) | Patch the first k features in the paper's ordering | Fewer patches than PCA components at the same KL divergence on IOI, with no numbers given in the text [arxiv:2309.08600 §4.2] [arxiv:2309.08600 fig. 3] | | Limited compute, small model | One training run per the paper's setup | Under one hour on one A40 GPU for the Pythia runs [arxiv:2309.08600 app. B] | | Deeper layers | Expect smaller gains than early layers | The advantage over baselines narrows with depth [arxiv:2309.08600 fig. 2] | | Need to say what a text corpus contains, with named concepts, and cannot train | Pretrained SAE embedding | Each document becomes named latent strengths. Dataset diffing used 2 to 8 times fewer tokens than pure LLM-labelling baselines in the paper's pipeline [arxiv:2512.10092 §4.1] [arxiv:2512.10092 app. D.4] | | Two-dataset comparison, single run | Expect a smaller saving | For one such diff the paper's count is about 0.7M SAE tokens against 1.3M to 1.7M for LLM baselines [arxiv:2512.10092 app. D.4] | | Property-based search over documents, dense baseline available | Combine SAE and dense rankings | The combined ranking is highest on five of six datasets in the table, before reranking. On reasoning traces the SAE alone scores higher [arxiv:2512.10092 §4.4] [arxiv:2512.10092 app. G] | | Grouping documents along a chosen axis | Targeted clustering on keyphrase latents | On one synthetic set, clusters followed each axis more precisely than dense baselines [arxiv:2512.10092 app. F.2]. Sentiment and emotion were not recovered on Twitter data [arxiv:2512.10092 app. F.4] | | No released SAE for your reader model | Train your own SAE (first or second variant) | The embedding pipeline depends on a released SAE and its labels [arxiv:2512.10092 §4]. The first card covers training for the Pythia models only [arxiv:2309.08600 §2] | | Documents longer than the reader's context | Chunk them | The paper does not describe chunking [arxiv:2512.10092 §4 fn. 1] | ## Implementation sketch The core is the same for all variants. The unsupervised variant uses the loss below directly. The supervised variant adds binding and value terms to a two-stage schedule, and the embedding variant needs only the encoder. The tying of encoder and decoder, the pooling and the alpha value differ between sites. ```pseudocode # Sparse autoencoder loss, tied weights [arxiv:2309.08600 §2 eq. 1, 2, 4] # Input: activations (B, d_in) from one layer; alpha (L1 weight) # Output: scalar loss; dictionary rows are the features (d_hid, d_in) def sae_loss(activations, dictionary, bias, alpha): # Normalise each feature row to unit length [arxiv:2309.08600 §2] dictionary = dictionary / norm(dictionary, axis=1)[:, None] # Encode: one non-negative code per feature [arxiv:2309.08600 eq. 1] codes = relu(activations @ dictionary.T + bias) # (B, d_hid) # Decode: a sparse sum of feature rows [arxiv:2309.08600 eq. 2] reconstruction = codes @ dictionary # (B, d_in) # Squared error and L1 penalty on the code [arxiv:2309.08600 eq. 4] error = sum((activations - reconstruction) ** 2, axis=-1) sparsity = alpha * sum(abs(codes), axis=-1) return mean(error + sparsity) ``` For the untied variant, replace the transposed dictionary in the decoder with a separate decoder matrix [arxiv:2309.08600 app. C]. For the supervised variant, split the code into free features and concept slots. Apply L1 only to the free features. Add a cross-entropy term on the concept slots against the gold relation, and a cross-covariance penalty between concept and free slots. Train reconstruction alone first, then all terms [arxiv:2512.02004 app. C.2] [arxiv:2512.02004 app. C.3]. The main text and the code differ on several of these details; see Contradictions. ```pseudocode # Document embedding from a pretrained SAE [arxiv:2512.10092 §3, fig. 1] # Input: token ids of one document (under the reader's context limit) # Output: vector (d_SAE,), the peak activation of each latent def sae_document_embedding(token_ids, reader, sae): # Hidden states of the reader model at the SAE's layer [arxiv:2512.10092 §4] hidden = reader.hidden_states(token_ids, layer=sae.layer) # (T, d_model) # Encode each token; the paper does not name this nonlinearity [§3] codes = act(hidden @ sae.w_enc.T + sae.b_enc) # (T, d_SAE) # Keep the strongest activation of each latent over the document [§3] return codes.max(axis=0) ``` Practical notes from the cards: - The reconstruction is lossy, and substituting it into a model raises perplexity [arxiv:2309.08600 §6.2]. - Many features in MLP layers never activate [arxiv:2309.08600 app. E]. - Steering with the decoder direction needs a choice of scale. The supervised paper's sweep covers alpha from 0.1 to 1000 [arxiv:2512.02004 app. D]. ## Contradictions - **Swap headline.** The abstract of the supervised paper gives 94.8% one-hop swap success. No table reports that figure. The table value is 0.847 at alpha = 2 [arxiv:2512.02004 §5.3] [arxiv:2512.02004 §6]. Use the table. - **Two-hop gain.** The main text describes four times the swap success of a standard SAE [arxiv:2512.02004 §1] [arxiv:2512.02004 §7]. The appendix gives 2.9 times [arxiv:2512.02004 app. K]. - **Stage 1 contents.** The main text includes sparsity in the first stage. The appendix and the code use reconstruction only [arxiv:2512.02004 §3] [arxiv:2512.02004 app. C.2]. - **Ablation prose.** The text gives 0.891 for the full model's unseen diagonal accuracy. The table gives 0.912 [arxiv:2512.02004 §6]. - **Correlation values.** The text gives 0.19 and 0.24 for feature skew and kurtosis against the top-and-random score. The table gives 0.20 and 0.15 [arxiv:2309.08600 app. C] [arxiv:2309.08600 tab. 2]. Check the source before relying on either. - **Simulator model.** The appendix names GPT-3.5 as the simulator [arxiv:2309.08600 app. A]. The released repository configures a different model. See the paper card's "Paper vs code" section. - **Reconstruction scale.** Equation 4 sums the squared error over the input dimensions. The repository averages it, which makes the same alpha a relatively stronger sparsity penalty. The paper does not say which convention produced its alpha values [arxiv:2309.08600 eq. 4]. See the paper card's "Paper vs code" section. - **Frequency definition.** The embedding paper's main text counts a latent as present in a document when it has more than one activated token [arxiv:2512.10092 §4.1]. Its appendix says at least one token [arxiv:2512.10092 app. D.2]. The two give different frequencies for latents that fire on a single token. - **Injection rate.** The text injects 10 texts into 990, which is 1%. The figure caption says 0.5% [arxiv:2512.10092 §4.2] [arxiv:2512.10092 fig. 3]. - **Between the papers.** No figure from one paper conflicts with a figure from another. They use different models, layers, metrics and tasks, so their numbers are not directly comparable. ## Papers - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600): trains tied and untied SAEs on Pythia residual streams and MLP layers, scores features automatically, and tests them on IOI. - [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004): adds a supervised second stage that binds named relation concepts to fixed slots on a GPT-2 layer, and tests steering on synthetic biographies and two-hop questions. - [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092): uses a released SAE on Llama 3.3 70B to embed documents as named concept latents, and applies them to dataset diffing, correlations, clustering and retrieval.