Sparse autoencoder

Overview

A sparse autoencoder (SAE) rebuilds a layer's activation vector from a sparse code. Its hidden layer is wider than the activation, a sparsity penalty keeps few hidden units active, and each learned dictionary direction is a candidate feature 2309.08600 §2. The motivation is superposition: a layer can represent more features than it has dimensions, so individual neurons often respond to several unrelated things 2309.08600 §1.

Three uses are covered. The first trains an unsupervised SAE on a model's activations and evaluates its features 2309.08600 §2, §3. The second trains a supervised SAE in which chosen concepts are bound to fixed slots, so that a concept can be read off and steered 2512.02004 §3, §4. The third takes an SAE that already exists and uses its encoder alone, with no training, to turn each document into a vector of named concept strengths for data analysis 2512.10092 §3. The shared core is the sparse encoding of an activation vector into one non-negative value per feature.

Most evidence is author-reported and comes from small models (GPT-2 124M, Pythia-70M and -410M) or from one large reader model (Llama 3.3 70B). The three papers use different metrics, so their numbers are not comparable with each other.

Variants

Unsupervised tied SAE on the residual stream

The encoder applies a ReLU to a linear map of the activation plus a bias. The decoder uses the transpose of the same dictionary matrix 2309.08600 §2. The loss is squared reconstruction error plus alpha times the L1 norm of the code 2309.08600 §2 eq. 4. The paper's main results use this form on Pythia-70M and Pythia-410M residual streams, with alpha = 8.6e-4 in the residual setting 2309.08600 app. B. Each run takes under an hour on one A40 GPU 2309.08600 app. B. Author-reported: in the layers studied, the mean automated interpretability score is higher than the default basis, random directions, PCA and ICA, though the gap narrows with depth 2309.08600 fig. 2. In the indirect-object-identification task, patching a few features reaches a given KL divergence with fewer patches than PCA components 2309.08600 §4.2 2309.08600 fig. 3. Card: Sparse Autoencoders Find Highly Interpretable Features in Language Models.

Untied SAE for MLP layers

The encoder and decoder have separate matrices instead of a shared one 2309.08600 app. C. The paper reports weaker results on MLP layers, and many features in those layers never activate, even with alpha = 0 2309.08600 app. E. The MLP setting uses alpha = 3.2e-4 2309.08600 app. C. Card: Sparse Autoencoders Find Highly Interpretable Features in Language Models.

Supervised, concept-aligned SAE

This variant needs a predefined ontology of concepts and labelled inputs. The paper's concepts are relation types in synthetic biographies and a two-hop question set 2512.02004 §3, §4.2. The encoder maps a frozen GPT-2 124M activation at layer 6 to a code made of free features and one slot per concept. The paper uses 10,000 free features and 6 concept slots for one-hop data, and 100,000 free features and 20 slots for two-hop data 2512.02004 app. C.1 2512.02004 app. M.

Training has two stages. Stage 1 optimises reconstruction only, so the decoder forms a dictionary. Stage 2 optimises all terms jointly: reconstruction, L1 on the free features, cross-entropy binding of the concept slots to the gold relation, a decorrelation penalty between concept and free slots, and a value loss that predicts the first answer token 2512.02004 app. C.2 2512.02004 app. C.3. The weights are 1.0, 1e-3, 1.0, 1e-2 and 0.5 for these terms 2512.02004 app. C.4.

Author-reported, one-hop, layer 6:

• Diagonal accuracy on unseen templates is 0.912, against 0.165 at layer 0 2512.02004 §5.1.
• Swap success is 0.847 at alpha = 2 2512.02004 §5.3. It falls to 0.23 at alpha = 10 2512.02004 §5.3, so the steering strength matters.
• Removing the binding loss drops swap success to 0.053 at its best alpha, and removing the decorrelation penalty to 0.046 2512.02004 §6.
• Mutual Information Gap is 0.173 for this model and 0.000 for a standard SAE 2512.02004 §5.2 2512.02004 app. G.

Author-reported, two-hop: binding reaches 100% at both hop steps, but swap success peaks near alpha ≈ 50 at 7.86%, against 1.5 to 2.7% for a standard SAE 2512.02004 §7 2512.02004 app. K. Two-hop training also shows a grokking-style transition: first-step diagonal accuracy is 38.4% at epoch 10 and 100% at epoch 40 2512.02004 §7.2. The paper states that binding does not imply causal control in this setting 2512.02004 app. K. Card: AlignSAE: Concept-Aligned Sparse Autoencoders.

Pretrained SAE as a document embedding

No SAE is trained here. A released SAE for one reader model and layer is run over each document, and the document is represented by the maximum activation of each latent across its tokens 2512.10092 §3 2512.10092 fig. 1. The reader model is the only model needed, even when the analysed texts came from another model 2512.10092 §3. The paper's SAE is Goodfire's public release for Llama 3.3 70B, layer 50, with 65,536 latents 2512.10092 §4.

Each latent has a short label written by an LLM from 10 activating and 10 non-activating examples, with activating tokens marked 2512.10092 §3 2512.10092 app. C. Labels are fixed for a given SAE. The paper reuses 61,521 existing labels and relabels some latents that look inaccurate 2512.10092 §4. The labels define the hypothesis space: a different SAE release or relabelling changes what can be found 2512.10092 §6.

The paper builds four analyses on these vectors: dataset diffing, latent correlations, targeted clustering and property-based retrieval 2512.10092 §4.1 to 2512.10092 §4.4. Texts must fit the reader's context limit, which the paper gives as 2048 tokens through the API it used 2512.10092 §4 fn. 1. Card: Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit.

When to use which

The choice depends on whether you can train an SAE, whether you have labels for the concepts you care about, and whether you need to steer the model or only analyse text.

Implementation sketch

The core is the same for all variants. The unsupervised variant uses the loss below directly. The supervised variant adds binding and value terms to a two-stage schedule, and the embedding variant needs only the encoder. The tying of encoder and decoder, the pooling and the alpha value differ between sites.

# Sparse autoencoder loss, tied weights  [arxiv:2309.08600 §2 eq. 1, 2, 4]
# Input:  activations (B, d_in) from one layer; alpha (L1 weight)
# Output: scalar loss; dictionary rows are the features (d_hid, d_in)
def sae_loss(activations, dictionary, bias, alpha):
    # Normalise each feature row to unit length  [arxiv:2309.08600 §2]
    dictionary = dictionary / norm(dictionary, axis=1)[:, None]
    # Encode: one non-negative code per feature  [arxiv:2309.08600 eq. 1]
    codes = relu(activations @ dictionary.T + bias)  # (B, d_hid)
    # Decode: a sparse sum of feature rows  [arxiv:2309.08600 eq. 2]
    reconstruction = codes @ dictionary              # (B, d_in)
    # Squared error and L1 penalty on the code  [arxiv:2309.08600 eq. 4]
    error = sum((activations - reconstruction) ** 2, axis=-1)
    sparsity = alpha * sum(abs(codes), axis=-1)
    return mean(error + sparsity)

For the untied variant, replace the transposed dictionary in the decoder with a separate decoder matrix 2309.08600 app. C.

For the supervised variant, split the code into free features and concept slots. Apply L1 only to the free features. Add a cross-entropy term on the concept slots against the gold relation, and a cross-covariance penalty between concept and free slots. Train reconstruction alone first, then all terms 2512.02004 app. C.2 2512.02004 app. C.3. The main text and the code differ on several of these details; see Contradictions.

# Document embedding from a pretrained SAE  [arxiv:2512.10092 §3, fig. 1]
# Input:  token ids of one document (under the reader's context limit)
# Output: vector (d_SAE,), the peak activation of each latent
def sae_document_embedding(token_ids, reader, sae):
    # Hidden states of the reader model at the SAE's layer  [arxiv:2512.10092 §4]
    hidden = reader.hidden_states(token_ids, layer=sae.layer)  # (T, d_model)
    # Encode each token; the paper does not name this nonlinearity  [§3]
    codes = act(hidden @ sae.w_enc.T + sae.b_enc)             # (T, d_SAE)
    # Keep the strongest activation of each latent over the document  [§3]
    return codes.max(axis=0)

Practical notes from the cards:

• The reconstruction is lossy, and substituting it into a model raises perplexity 2309.08600 §6.2.
• Many features in MLP layers never activate 2309.08600 app. E.
• Steering with the decoder direction needs a choice of scale. The supervised paper's sweep covers alpha from 0.1 to 1000 2512.02004 app. D.

Contradictions

• Swap headline. The abstract of the supervised paper gives 94.8% one-hop swap success. No table reports that figure. The table value is 0.847 at alpha = 2 2512.02004 §5.3 2512.02004 §6. Use the table.
• Two-hop gain. The main text describes four times the swap success of a standard SAE 2512.02004 §1 2512.02004 §7. The appendix gives 2.9 times 2512.02004 app. K.
• Stage 1 contents. The main text includes sparsity in the first stage. The appendix and the code use reconstruction only 2512.02004 §3 2512.02004 app. C.2.
• Ablation prose. The text gives 0.891 for the full model's unseen diagonal accuracy. The table gives 0.912 2512.02004 §6.
• Correlation values. The text gives 0.19 and 0.24 for feature skew and kurtosis against the top-and-random score. The table gives 0.20 and 0.15 2309.08600 app. C 2309.08600 tab. 2. Check the source before relying on either.
• Simulator model. The appendix names GPT-3.5 as the simulator 2309.08600 app. A. The released repository configures a different model. See the paper card's "Paper vs code" section.
• Reconstruction scale. Equation 4 sums the squared error over the input dimensions. The repository averages it, which makes the same alpha a relatively stronger sparsity penalty. The paper does not say which convention produced its alpha values 2309.08600 eq. 4. See the paper card's "Paper vs code" section.
• Frequency definition. The embedding paper's main text counts a latent as present in a document when it has more than one activated token 2512.10092 §4.1. Its appendix says at least one token 2512.10092 app. D.2. The two give different frequencies for latents that fire on a single token.
• Injection rate. The text injects 10 texts into 990, which is 1%. The figure caption says 0.5% 2512.10092 §4.2 2512.10092 fig. 3.
• Between the papers. No figure from one paper conflicts with a figure from another. They use different models, layers, metrics and tasks, so their numbers are not directly comparable.

Papers

• Sparse Autoencoders Find Highly Interpretable Features in Language Models: trains tied and untied SAEs on Pythia residual streams and MLP layers, scores features automatically, and tests them on IOI.
• AlignSAE: Concept-Aligned Sparse Autoencoders: adds a supervised second stage that binds named relation concepts to fixed slots on a GPT-2 layer, and tests steering on synthetic biographies and two-hop questions.
• Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit: uses a released SAE on Llama 3.3 70B to embed documents as named concept latents, and applies them to dataset diffing, correlations, clustering and retrieval.