Automated interpretability

Overview

Interpretability work produces many candidate units: neurons, directions in the residual stream, learned features. Checking each one by hand does not scale to thousands of units. The automated protocol replaces the human reader with two language models. The explainer reads examples where the unit fires strongly and writes a short description. The simulator reads only that description and tries to predict, token by token, where the unit fires in new text 2309.08600 §3.1 2309.08600 app. A.

The shared idea is that a unit is well explained when its description lets someone predict its activations on text they have not seen. The score is the correlation between predicted and real activations. The protocol is described in the card for Sparse Autoencoders Find Highly Interpretable Features in Language Models, which applies it to learned features; it is the only card in this wiki that lists this concept so far.

Variants

Top-and-random explainer and simulator scoring

The explainer sees a few of the twenty fragments that activate the unit most strongly. The simulator is then scored on five of the top fragments and five random fragments that vary, so that a description has to work on both kinds of text 2309.08600 app. A. Features with too few varying fragments are skipped 2309.08600 app. A. The card reports this mixture as its headline score, because scoring on random text alone gives lower scores overall, though still above the baselines 2309.08600 app. C.

In the paper's comparison, the score is computed for 150 features per dictionary and the learned features score higher than the default basis, random directions, PCA and ICA at the layers studied 2309.08600 fig. 2 2309.08600 §3.2. The gap narrows in deeper layers 2309.08600 §3.2. Card: Sparse Autoencoders Find Highly Interpretable Features in Language Models.

Scores for five example features in layer 1 range from about -0.11 to 0.57, which shows that single features vary a lot 2309.08600 tab. 1.

When to use which

Only one variant has a card, so the choice is between using this protocol and doing no automated scoring. The card supports these points:

Implementation sketch

The score for one unit, in the card's protocol. Each step cites the card.

# Automated interpretability score for one unit  [arxiv:2309.08600 app. A]
# Input:  per-token activations on many text fragments, explainer, simulator
# Output: correlation in [-1, 1], or None if the unit is skipped
def interpretability_score(acts, explainer, simulator):
    # Skip units with too few fragments that vary  [app. A, step 5]
    if count_varying(acts) < 20:
        return None
    # Take the fragments with the highest activations  [app. A, step 2]
    top = top_fragments(acts, 20)
    # Explainer writes a description from five top fragments  [app. A, step 2]
    description = explainer(top[:5])
    # Simulator sees the description and five top and five random fragments  [app. A, step 3]
    held_out = top[5:10] + random_varying_fragments(acts, 5)
    predicted = simulator(description, held_out)
    # Correlate predicted with real per-token activations  [app. A, step 4]
    return correlation(predicted, real_activations(held_out))

The card's own settings are GPT-4 as explainer and a GPT-3.5 model as simulator, with 50,000 fragments of 64 tokens from OpenWebText 2309.08600 app. A.

Contradictions

• Simulator model. The paper's appendix names GPT-3.5 as the simulator 2309.08600 app. A. The code sets it to text-davinci-003 interpret.py. The explainer, GPT-4, matches in both interpret.py.
• Skew and kurtosis correlations. The text gives 0.19 and 0.24 for the correlation of feature skew and kurtosis with the top-and-random score 2309.08600 app. C. Table 2 lists 0.20 and 0.15 for the same quantities 2309.08600 tab. 2. The card does not say which is correct.

Papers

• Sparse Autoencoders Find Highly Interpretable Features in Language Models: trains sparse autoencoders on language model activations and scores the learned features with the explainer and simulator protocol.