Activation steering

Overview

Activation steering edits a model's hidden states while it runs. The edit is a fixed vector or a fixed coefficient, added to or set on one or more internal activations. The two earlier variants apply it at every decoding step. The sparse-autoencoder variant adds it at the question's token position and then generates from there 2512.02004 §4.4. The model's weights do not change 2306.03341 §1 2203.14680 §6.1.

The shared invariant is that the edit is chosen once, before generation, and applied at inference. The variants differ in where the edit is made, how it is chosen, and how big it is. In the attention-head variant the direction is learned from labelled true and false answers. In the feed-forward variant the edit is a fixed coefficient on a few value vectors, picked from their top tokens by hand or by a toxicity grader, with no labelled examples 2203.14680 §6.1 app. A.4. In the sparse-autoencoder variant the direction is a column of a trained dictionary, and that dictionary needs labelled examples for a fixed set of relations 2512.02004 §3.

Three papers with cards cover this page. They use different edits and measure different tasks (TruthfulQA, toxicity and synthetic biography QA), so their numbers do not compare directly.

Variants

Inference-time intervention on attention heads

Fit one linear probe per attention head on labelled true and false answers. Keep the K heads whose probes score best on held-out questions. For each kept head, take the direction from the mean of the false activations to the mean of the true activations, and add a scaled version of it to that head's output at every decoding step 2306.03341 §3.2 §3.3 eq. 2. The paper reports the best choice as K = 48 and strength alpha = 15 on LLaMA-7B, from a sweep 2306.03341 §4.3. The edit is one vector per layer, so it can also be stored in the layer's output bias 2306.03341 §5.2 eq. 3. Card: Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.

Reported effect, author-reported on TruthfulQA with 2-fold cross-validation. The true*informative score for Alpaca rises from 32.5% to 65.1%, and for Vicuna from 51.5% to 74.0% 2306.03341 tab. 2. On LLaMA-7B the same method reaches 42.3% against a 30.5% baseline, under the 2-fold setting 2306.03341 tab. 3. Under a different split, the paper reports 43.5% 2306.03341 tab. 1; see Contradictions.

Fixed boost of feed-forward value vectors

Split each feed-forward layer's output into per-value-vector terms. Pick a few value vectors whose top vocabulary tokens are the wanted words, and set their coefficients to a fixed value, here 3, at every generation step. No probe is trained and no weight changes 2203.14680 §3 §6.1. The authors pick the 10 vectors either by a short manual search or by grading token lists with the Perspective API 2203.14680 §6.1 app. A.4. The paper does not name the GPT-2 checkpoint used for this experiment, and the coefficient was not swept 2203.14680 §6.1. Card: Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space.

Reported effect, author-reported on 1,225 challenging RealToxicPrompts prompts, with 20-token continuations scored by Perspective. The share of completions with toxicity above 0.5 falls from 58.5% to 30.8% with the manual picks, a 47% relative drop 2203.14680 tab. 5. With the API-graded picks it falls only to 52.7% 2203.14680 tab. 5. Perplexity rises from 21.7 to 25.3 with the manual picks 2203.14680 tab. 5 §6.1.

Swap along a sparse-autoencoder concept direction

Train a sparse autoencoder on the hidden state at one layer, in two phases. The first phase fits reconstruction only. The second adds a sparsity penalty on the free features, a cross-entropy term that binds each labelled relation to its own slot, a penalty that decorrelates the relation slots from the free features, and an auxiliary term that predicts the first token of the answer 2512.02004 §3 app. C.3. To steer, the decoder column of the target slot is scaled by a strength alpha and added to the residual stream at the question's token position. A swap succeeds when the answer equals the gold value of the target relation 2512.02004 §4.4 app. D.

Reported effect, author-reported on templated synthetic biographies with GPT-2 (124M), six relations at one layer. Swap success is 0.847 at alpha = 2 on layer 6, against 0.040 on layer 0 2512.02004 §5.1 §6. It rises from 0.54 at alpha = 1 and falls to 0.23 at alpha = 10 [arxiv:2512.02004 §5.3]. Diagonal accuracy on unseen templates is 0.912 on layer 6 and 0.165 on layer 0 2512.02004 §5.1. Removing the alignment term drops swap success to 0.053 2512.02004 §6.

On two-hop questions with 20 relations, binding is 100% at both hop steps, but swap success peaks at 7.86%, against 1.5 to 2.7% for a standard sparse autoencoder 2512.02004 §7 app. K. The authors name the gap between binding and causal control in this setting 2512.02004 app. K. Card: AlignSAE: Concept-Aligned Sparse Autoencoders.

When to use which

The table compares the variants with each other and with the options the papers report. Each row is a constraint the papers speak to.

Implementation sketch

Our own pseudocode for the three variants. Each has two steps: fit the edit once on your data, then apply it during decoding. Where the paper and the authors' code differ, the card's Paper vs code section has the detail.

# Fit one direction and scale per kept head  [arxiv:2306.03341 §3.2] [§3.3]
# Input:  activations (N, layers, heads, d) at the last token; labels (N,)
def fit_steering(activations, labels, k):
    # Probe every head on a training split and score it on held-out questions
    # [arxiv:2306.03341 §3.2] [app. C]
    scores = probe_each_head(activations, labels)
    kept = top_k(scores, k)
    for layer, head in kept:
        # Mass-mean shift: mean of true minus mean of false  [§3.3]
        diff = mean(activations[labels == 1, layer, head]) \
             - mean(activations[labels == 0, layer, head])
        # Unit direction; the paper does not state this normalisation  [§3.3]
        theta = diff / norm(diff)
        # Sigma: std of activations projected on the unit direction  [§3.3]
        sigma = std(activations[:, layer, head] @ theta)
        store(layer, head, theta, sigma)
    return kept

# Add the scaled direction to each kept head during decoding  [eq. 2]
# Input: head output at one position; alpha is the strength knob
def steer_head_output(head_output, theta, sigma, alpha):
    return head_output + alpha * sigma * theta

# Fix the coefficients of chosen value vectors during generation  [§6.1]
# Input: prompt, chosen (dict: layer -> value indices), coef = 3
def steer_generation(model, prompt, chosen, coef=3):
    # Set each chosen coefficient to coef at every position  [§6.1]
    # Where the hook sits changes the effective value; see the card  [§3]
    install_hooks(model, chosen, coef)
    # Generate 20 tokens for each prompt  [§6.1]
    return generate(model, prompt, new_tokens=20)

# Swap one concept direction into the residual stream  [arxiv:2512.02004 §4.4]
# Input: h (d,) residual stream at layer l, at the question's token position
#        W_dec (d, slots) decoder matrix; target (slot index); alpha (strength)
def swap_concept(h, W_dec, target, alpha):
    # Decoder column of the target slot: the direction it writes back  [§4.4]
    direction = W_dec[:, target]
    # Scale by alpha only, as the paper's formula is written  [§4.4] [app. D]
    return h + alpha * direction

Contradictions

Within the attention-head paper, the numbers are not consistent across its tables. The paper does not resolve them.

• Strength. Section 4.3 picks alpha = 15 2306.03341 §4.3. Table 3 gives alpha = 20 as the best value for mass-mean shift 2306.03341 tab. 3.
• Baseline + intervention on the same model. LLaMA-7B with the method reaches 43.5% under the 5% split 2306.03341 tab. 1 and 42.3% under 2-fold cross-validation 2306.03341 tab. 3. The protocols differ, so the two numbers are not directly comparable.
• Few-shot baseline size. Section 4.2 cites a 50-shot baseline 2306.03341 §4.2. Appendix F says 10% of TruthfulQA was appended 2306.03341 app. F. Section 4.3 says 5% 2306.03341 §4.3.

Within the feed-forward paper, the toxicity result depends on how the picks were chosen. The manual picks reduce the toxic share by 47%, and the picks graded by the API reduce it by about 10% 2203.14680 tab. 5.

The same paper's concept analysis uses a 12-layer GPT-2 2203.14680 §4. Its toxicity picks sit at layers 14 to 23 2203.14680 app. A.4, which a 12-layer model cannot have. The model size for the toxicity experiment is therefore not established.

Within the sparse-autoencoder paper, the headline and the tables differ, and the swap is written two ways.

• Headline swap success. The abstract reports 94.8% swap success in 1-hop. The tables give 0.847 at alpha = 2 2512.02004 §5.3 §6. Do not target 94.8%.
• Swap scale, paper vs code. The paper's formula adds alpha times the decoder column 2512.02004 §4.4 app. D. The authors' released hook script instead scales the column so that the injected vector has norm alpha times the residual norm; see the card's Paper vs code section. An alpha from one form is therefore not an alpha for the other.
• Two-hop factor. The main text gives four times the standard sparse autoencoder's swap success on two-hop questions 2512.02004 §7. The appendix gives 2.9 times 2512.02004 app. K.
• Ablation prose. The ablation text gives the full model's unseen diagonal accuracy as 0.891, but its table gives 0.912 2512.02004 §6. Use the table.

Across the three papers there is no conflicting number: they measure different tasks. The sparse-autoencoder swap is measured on templated GPT-2 biographies, not on TruthfulQA or toxicity, so its numbers do not compare with the others.

Papers

• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model: adds a scaled, learned direction to selected attention heads at each decoding step; reports gains on TruthfulQA for Alpaca and Vicuna, with a truthfulness-informativeness trade-off.
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space: sets the coefficients of 10 hand-picked feed-forward value vectors to a fixed value during generation; reports a 47% drop in GPT-2 toxicity on one chosen set, with a perplexity rise.
• AlignSAE: Concept-Aligned Sparse Autoencoders: trains a sparse autoencoder with labelled concept slots and swaps a slot's decoder direction into the residual stream; reports swap success on synthetic GPT-2 biographies, with weaker control on two-hop questions.