---
kind: concept
slug: activation-steering
title: Activation steering
summary: >-
  Changes a language model's behaviour at inference by editing its internal activations, with no retraining. Inference-Time Intervention shifts attention heads toward a truthful direction, raising TruthfulQA scores on two chat models. A fixed boost to hand-picked feed-forward values cuts GPT-2 toxicity. A sparse-autoencoder concept direction steers synthetic biography QA. All need white-box access.
tags: [activation-steering, interpretability, inference-time, truthfulness, toxicity, feed-forward, probing, sparse-autoencoder]
papers: ["2203.14680", "2306.03341", "2512.02004"]
related: [mechanistic-interpretability, linear-probing, hallucination-mitigation, controllable-text-generation, activation-patching, sparse-autoencoder]
status: draft
updated_at: 2026-11-01
---

## Overview

Activation steering edits a model's hidden states while it runs. The edit is
a fixed vector or a fixed coefficient, added to or set on one or more internal
activations. The two earlier variants apply it at every decoding step. The
sparse-autoencoder variant adds it at the question's token position and then
generates from there [arxiv:2512.02004 §4.4]. The model's weights do not
change [arxiv:2306.03341 §1] [arxiv:2203.14680 §6.1].

The shared invariant is that the edit is chosen once, before generation, and
applied at inference. The variants differ in where the edit is made, how it
is chosen, and how big it is. In the attention-head variant the direction is
learned from labelled true and false answers. In the feed-forward variant the
edit is a fixed coefficient on a few value vectors, picked from their top
tokens by hand or by a toxicity grader, with no labelled examples
[arxiv:2203.14680 §6.1] [app. A.4]. In the sparse-autoencoder variant the
direction is a column of a trained dictionary, and that dictionary needs
labelled examples for a fixed set of relations [arxiv:2512.02004 §3].

Three papers with cards cover this page. They use different edits and measure
different tasks (TruthfulQA, toxicity and synthetic biography QA), so their
numbers do not compare directly.

## Variants

### Inference-time intervention on attention heads

Fit one linear probe per attention head on labelled true and false answers.
Keep the K heads whose probes score best on held-out questions. For each kept
head, take the direction from the mean of the false activations to the mean
of the true activations, and add a scaled version of it to that head's output
at every decoding step [arxiv:2306.03341 §3.2] [§3.3] [eq. 2]. The paper
reports the best choice as K = 48 and strength alpha = 15 on LLaMA-7B, from a
sweep [arxiv:2306.03341 §4.3]. The edit is one vector per layer, so it can
also be stored in the layer's output bias [arxiv:2306.03341 §5.2] [eq. 3].
Card: [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](/papers/2306.03341).

Reported effect, author-reported on TruthfulQA with 2-fold cross-validation.
The true*informative score for Alpaca rises from 32.5% to 65.1%, and for
Vicuna from 51.5% to 74.0% [arxiv:2306.03341 tab. 2]. On LLaMA-7B the
same method reaches 42.3% against a 30.5% baseline, under the 2-fold setting
[arxiv:2306.03341 tab. 3]. Under a different split, the paper reports 43.5%
[arxiv:2306.03341 tab. 1]; see Contradictions.

### Fixed boost of feed-forward value vectors

Split each feed-forward layer's output into per-value-vector terms. Pick a
few value vectors whose top vocabulary tokens are the wanted words, and set
their coefficients to a fixed value, here 3, at every generation step. No
probe is trained and no weight changes [arxiv:2203.14680 §3] [§6.1]. The
authors pick the 10 vectors either by a short manual search or by grading
token lists with the Perspective API [arxiv:2203.14680 §6.1] [app. A.4]. The
paper does not name the GPT-2 checkpoint used for this experiment, and the
coefficient was not swept [arxiv:2203.14680 §6.1]. Card: [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](/papers/2203.14680).

Reported effect, author-reported on 1,225 challenging RealToxicPrompts
prompts, with 20-token continuations scored by Perspective. The share of
completions with toxicity above 0.5 falls from 58.5% to 30.8% with the manual
picks, a 47% relative drop [arxiv:2203.14680 tab. 5]. With the API-graded
picks it falls only to 52.7% [arxiv:2203.14680 tab. 5]. Perplexity rises from
21.7 to 25.3 with the manual picks [arxiv:2203.14680 tab. 5] [§6.1].

### Swap along a sparse-autoencoder concept direction

Train a sparse autoencoder on the hidden state at one layer, in two phases.
The first phase fits reconstruction only. The second adds a sparsity penalty
on the free features, a cross-entropy term that binds each labelled relation
to its own slot, a penalty that decorrelates the relation slots from the free
features, and an auxiliary term that predicts the first token of the answer
[arxiv:2512.02004 §3] [app. C.3]. To steer, the decoder column of the target
slot is scaled by a strength alpha and added to the residual stream at the
question's token position. A swap succeeds when the answer equals the gold
value of the target relation [arxiv:2512.02004 §4.4] [app. D].

Reported effect, author-reported on templated synthetic biographies with
GPT-2 (124M), six relations at one layer. Swap success is 0.847 at alpha = 2
on layer 6, against 0.040 on layer 0 [arxiv:2512.02004 §5.1] [§6]. It rises
from 0.54 at alpha = 1 and falls to 0.23 at alpha = 10 [arxiv:2512.02004
§5.3]. Diagonal accuracy on unseen templates is 0.912 on layer 6 and 0.165 on
layer 0 [arxiv:2512.02004 §5.1]. Removing the alignment term drops swap
success to 0.053 [arxiv:2512.02004 §6].

On two-hop questions with 20 relations, binding is 100% at both hop steps,
but swap success peaks at 7.86%, against 1.5 to 2.7% for a standard sparse
autoencoder [arxiv:2512.02004 §7] [app. K]. The authors name the gap between
binding and causal control in this setting [arxiv:2512.02004 app. K]. Card:
[AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004).

## When to use which

The table compares the variants with each other and with the options the
papers report. Each row is a constraint the papers speak to.

| Situation | Use | Why |
|---|---|---|
| You can run the model with hooks on its internal activations and can label true and false outputs for your behaviour | Inference-time intervention on attention heads | The paper's method needs exactly these inputs [arxiv:2306.03341 §3.2] [§3.1] |
| You have no labelled data, but you can name a few value vectors whose top tokens are the wanted words | Fixed boost of feed-forward value vectors | The paper's toxicity edit needed no labels; picks came from token lists [arxiv:2203.14680 §6.1] [app. A.4] |
| You can train a sparse autoencoder on one layer and have labelled examples for a fixed set of concepts or relations | Swap along a sparse-autoencoder concept direction | The method needs a predefined ontology with labelled inputs and a supervised second training phase [arxiv:2512.02004 §3] |
| Your concepts have no labels or are not a fixed set | Not the sparse-autoencoder variant | The authors argue that unlabelled features make feature-level steering unreliable [arxiv:2512.02004 §1] |
| You need to choose the layer for a sparse-autoencoder swap | Measure binding and swap at each candidate layer before fixing it | On GPT-2, layer 0 gives 0.040 swap success and layer 6 gives 0.847 [arxiv:2512.02004 §5.1] |
| You need a steering edit on a multi-step question | Do not expect the swap to control the answer | Binding is perfect at both hops, yet swap success stays at 7.86% at its best strength [arxiv:2512.02004 §7] [app. K] |
| You cannot access activations (API-only model) | Not covered here | All three methods need white-box access and hooks inside the model [arxiv:2306.03341 §3.2] [arxiv:2203.14680 §6.1] [arxiv:2512.02004 §4.4] |
| You need the most truthful answers and can accept fewer informative ones | Higher strength alpha | Stronger edits raise truthfulness and lower informativeness, with more "no comment" answers [arxiv:2306.03341 §4.3] [§5.4] |
| You need both truthful and informative answers | Tune alpha on your model and report both scores | The paper reports the trade-off and the metric penalises "no comment" [arxiv:2306.03341 §4.1] [fig. 6] |
| You want to check that the learned direction matters | Include a random direction as a control | At alpha = 20 random directions leave LLaMA-7B nearly unchanged [arxiv:2306.03341 app. B] [tab. 3] |
| You want to pick the direction | Mass-mean shift, not the probe weight or CCS direction | Mass-mean shift scored best in the paper's comparison [arxiv:2306.03341 tab. 3] [§4.3] |
| You want to avoid a fixed-cost inference edit | Bake the edit into the output bias | The authors report close to zero run-time overhead [arxiv:2306.03341 §5.2] |
| You want to lower toxicity and can accept a perplexity rise | Fixed boost of feed-forward value vectors | The manual picks gave the largest drop in the table (47%), with perplexity 25.3 against 23.9 for Self-Debiasing [arxiv:2203.14680 tab. 5] |
| You need a reproducible selection procedure | Inference-time intervention, or a method with a published selection rule | The manual search was short and greedy, and the paper does not say whether it used the evaluation prompts [arxiv:2203.14680 app. A.4] |

## Implementation sketch

Our own pseudocode for the three variants. Each has two steps: fit the edit
once on your data, then apply it during decoding. Where the paper and the
authors' code differ, the card's Paper vs code section has the detail.

```pseudocode
# Fit one direction and scale per kept head  [arxiv:2306.03341 §3.2] [§3.3]
# Input:  activations (N, layers, heads, d) at the last token; labels (N,)
def fit_steering(activations, labels, k):
    # Probe every head on a training split and score it on held-out questions
    # [arxiv:2306.03341 §3.2] [app. C]
    scores = probe_each_head(activations, labels)
    kept = top_k(scores, k)
    for layer, head in kept:
        # Mass-mean shift: mean of true minus mean of false  [§3.3]
        diff = mean(activations[labels == 1, layer, head]) \
             - mean(activations[labels == 0, layer, head])
        # Unit direction; the paper does not state this normalisation  [§3.3]
        theta = diff / norm(diff)
        # Sigma: std of activations projected on the unit direction  [§3.3]
        sigma = std(activations[:, layer, head] @ theta)
        store(layer, head, theta, sigma)
    return kept

# Add the scaled direction to each kept head during decoding  [eq. 2]
# Input: head output at one position; alpha is the strength knob
def steer_head_output(head_output, theta, sigma, alpha):
    return head_output + alpha * sigma * theta
```

```pseudocode
# Fix the coefficients of chosen value vectors during generation  [§6.1]
# Input: prompt, chosen (dict: layer -> value indices), coef = 3
def steer_generation(model, prompt, chosen, coef=3):
    # Set each chosen coefficient to coef at every position  [§6.1]
    # Where the hook sits changes the effective value; see the card  [§3]
    install_hooks(model, chosen, coef)
    # Generate 20 tokens for each prompt  [§6.1]
    return generate(model, prompt, new_tokens=20)
```

```pseudocode
# Swap one concept direction into the residual stream  [arxiv:2512.02004 §4.4]
# Input: h (d,) residual stream at layer l, at the question's token position
#        W_dec (d, slots) decoder matrix; target (slot index); alpha (strength)
def swap_concept(h, W_dec, target, alpha):
    # Decoder column of the target slot: the direction it writes back  [§4.4]
    direction = W_dec[:, target]
    # Scale by alpha only, as the paper's formula is written  [§4.4] [app. D]
    return h + alpha * direction
```

## Contradictions

Within the attention-head paper, the numbers are not consistent across its
tables. The paper does not resolve them.

- **Strength.** Section 4.3 picks alpha = 15 [arxiv:2306.03341 §4.3]. Table 3
  gives alpha = 20 as the best value for mass-mean shift [arxiv:2306.03341 tab. 3].
- **Baseline + intervention on the same model.** LLaMA-7B with the method
  reaches 43.5% under the 5% split [arxiv:2306.03341 tab. 1] and 42.3% under
  2-fold cross-validation [arxiv:2306.03341 tab. 3]. The protocols differ, so
  the two numbers are not directly comparable.
- **Few-shot baseline size.** Section 4.2 cites a 50-shot baseline
  [arxiv:2306.03341 §4.2]. Appendix F says 10% of TruthfulQA was appended
  [arxiv:2306.03341 app. F]. Section 4.3 says 5% [arxiv:2306.03341 §4.3].

Within the feed-forward paper, the toxicity result depends on how the picks
were chosen. The manual picks reduce the toxic share by 47%, and the picks
graded by the API reduce it by about 10% [arxiv:2203.14680 tab. 5].

The same paper's concept analysis uses a 12-layer GPT-2 [arxiv:2203.14680 §4].
Its toxicity picks sit at layers 14 to 23 [arxiv:2203.14680 app. A.4], which
a 12-layer model cannot have. The model size for the toxicity experiment is
therefore not established.

Within the sparse-autoencoder paper, the headline and the tables differ, and
the swap is written two ways.

- **Headline swap success.** The abstract reports 94.8% swap success in
  1-hop. The tables give 0.847 at alpha = 2 [arxiv:2512.02004 §5.3] [§6]. Do
  not target 94.8%.
- **Swap scale, paper vs code.** The paper's formula adds alpha times the
  decoder column [arxiv:2512.02004 §4.4] [app. D]. The authors' released hook
  script instead scales the column so that the injected vector has norm alpha
  times the residual norm; see the card's Paper vs code section. An alpha from
  one form is therefore not an alpha for the other.
- **Two-hop factor.** The main text gives four times the standard sparse
  autoencoder's swap success on two-hop questions [arxiv:2512.02004 §7]. The
  appendix gives 2.9 times [arxiv:2512.02004 app. K].
- **Ablation prose.** The ablation text gives the full model's unseen diagonal
  accuracy as 0.891, but its table gives 0.912 [arxiv:2512.02004 §6]. Use the
  table.

Across the three papers there is no conflicting number: they measure different
tasks. The sparse-autoencoder swap is measured on templated GPT-2 biographies,
not on TruthfulQA or toxicity, so its numbers do not compare with the others.

## Papers

- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](/papers/2306.03341): adds a scaled, learned direction to selected attention heads at each decoding step; reports gains on TruthfulQA for Alpaca and Vicuna, with a truthfulness-informativeness trade-off.
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](/papers/2203.14680): sets the coefficients of 10 hand-picked feed-forward value vectors to a fixed value during generation; reports a 47% drop in GPT-2 toxicity on one chosen set, with a perplexity rise.
- [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004): trains a sparse autoencoder with labelled concept slots and swaps a slot's decoder direction into the residual stream; reports swap success on synthetic GPT-2 biographies, with weaker control on two-hop questions.