---
kind: concept
slug: causal-abstraction
title: Causal abstraction
summary: >-
  Tests whether a network implements a named algorithm by mapping its hidden states onto the algorithm's intermediate variables and checking that interventions agree. Distributed alignment search fits the map. An unrestricted invertible map makes the test vacuous, since it can align any network with any algorithm, so report results only with a restricted map class and controls.
tags: [interpretability, causal-intervention, mechanistic-interpretability]
papers: ["2507.08802"]
related: [activation-patching, mechanistic-interpretability, linear-probing, causal-inference]
status: draft
updated_at: 2026-11-08
---

## Overview

Causal abstraction asks whether a neural network computes what a simpler,
step-by-step algorithm computes. The algorithm is a graph of named intermediate
variables that ends in the task output. The claimed match is a map from the
network's hidden states to those variables. The match passes the interventional
test if overwriting a variable in the algorithm has the same effect as
overwriting the matched part of the network [arxiv:2507.08802 §2.1]
[arxiv:2507.08802 §3].

The invariant all variants share is that interventions must commute with the
map: intervene on the network, read off the mapped variables, and the result
must agree with intervening on the algorithm. The variants differ in how
flexible the map may be. A more flexible map can fit more, so a good score
proves less [arxiv:2507.08802 §1] [arxiv:2507.08802 §4].

The paper that anchors this page argues that the flexible end of that scale is
vacuous. Under a few mild conditions, an invertible map can align any network
with any algorithm [arxiv:2507.08802 §1] [arxiv:2507.08802 §4]. Distributed
alignment search (DAS), introduced in earlier work by Geiger and colleagues
([arXiv:2301.04709](https://arxiv.org/abs/2301.04709),
[arXiv:2303.02536](https://arxiv.org/abs/2303.02536)), is the training procedure
the variants below use. Those two papers have no card here, so they are named
only.

## Variants

### Constructive abstraction (neuron partition)

Each algorithm variable reads its own disjoint set of hidden neurons, and some
neurons may stay unused. No map is learned. The search only assigns neurons to
variables. The paper's identity-map experiments build this partition greedily,
adding one neuron at a time to each variable's set [arxiv:2507.08802 §3]
[arxiv:2507.08802 §5]. On an equality task, identity, rotation and RevNet maps
all score about 50% for the identity-of-first-argument algorithm
[arxiv:2507.08802 §6]. Card: [Non-Linear Representation Dilemma](/papers/2507.08802).

### Distributed abstraction with an orthogonal map

A learned bijection φ first rotates the hidden states into latent coordinates.
Each variable then reads its own latent coordinates. An intervention maps to
latent space, overwrites the chosen coordinates from a source run, and maps back
[arxiv:2507.08802 §3.2] [arxiv:2507.08802 app. B]. The paper's linear variant
restricts φ to an orthogonal matrix [arxiv:2507.08802 §5]. The paper notes that
most prior work implicitly assumes the linear representation hypothesis, which
limits the map to linear or orthogonal transformations [arxiv:2507.08802 §1]
[arxiv:2507.08802 §5]. DAS fits φ with the network frozen. The loss is
cross-entropy between the intervened network's output and the algorithm's output
under the same intervention [arxiv:2507.08802 §3.3].

### Distributed abstraction with a non-linear invertible map (RevNet)

φ is a reversible residual network, so its inverse is exact and the map can be
non-linear [arxiv:2507.08802 §5]. The paper reports high fits for this variant
even on untrained models. For the equality MLP, the most complex RevNet at layer
one of a randomly initialised network scores above 80% [arxiv:2507.08802 §6].
For Pythia language models the paper reports 100% for randomly initialised
models [arxiv:2507.08802 §1] [arxiv:2507.08802 §6]. The paper argues that this
result shows the test does not tell a real implementation from a coincidence.
Use this variant only with the controls in the sections below. Card:
[Non-Linear Representation Dilemma](/papers/2507.08802).

### Strong versus input-restricted abstraction

The definition can require the map to realise every intervention, or only the
interventions that other inputs can produce. The paper says the input-restricted
version is the one used in practice, and its theorem is about that version
[arxiv:2507.08802 §3.2].

## When to use which

Practical constraints first. The method needs hidden states at a chosen layer
and forward passes that resume from that layer, so it does not apply to black-box
models [arxiv:2507.08802 §2.2] [arxiv:2507.08802 §3.3]. Training uses about 1.28
million interventions for the equality MLP [arxiv:2507.08802 app. I.1] and 1
million for the Pythia runs [arxiv:2507.08802 app. I.2]. The paper reports about
three weeks on ten CPU machines for the MLP and about 30 GPU hours on one A100
for the Pythia runs, including tuning [arxiv:2507.08802 app. J].

| Situation | Use | Why |
|---|---|---|
| Claim is about single neurons, no map to fit | Constructive partition | Cheapest option. The paper's greedy partition is its identity baseline [arxiv:2507.08802 §5] |
| Claim is about a linear subspace, linear encoding assumed | Orthogonal map with DAS | More restricted than RevNet. The paper finds rotations weaker on random models [arxiv:2507.08802 §1] [arxiv:2507.08802 §6] |
| Only a non-linear map is available | Run controls before reporting | The paper reports high fits on untrained Pythia [arxiv:2507.08802 app. I.2], so an untrained-network control is needed |
| Language model with name-based task | Use name-disjoint train and eval splits | With random-init Pythia and disjoint names, no alignment was found even with complex maps [arxiv:2507.08802 app. I.2] |
| Small interventional dataset | Expect the fit to vary with sample size | RevNet reached above 0.99 train IIA with 1,000 samples and under 0.55 with 1.28M on the same identity-of-first-argument task. The authors read this as a capacity limit [arxiv:2507.08802 app. H] |

## Assumptions behind the existence result

The paper's theorem says that, under five conditions, the algorithm is an
input-restricted distributed abstraction of the network [arxiv:2507.08802 §4]
[arxiv:2507.08802 app. F]. The conditions are: the input set is countable, which
holds for text and approximately for images at fixed resolution; every prefix of
the network is injective; the suffix of the network followed by the output map is
surjective onto the outputs; a single-neuron assignment respects the algorithm's
order, with at least one unused neuron per layer; and the network gets every
input right [arxiv:2507.08802 §4] [arxiv:2507.08802 app. F].

The proof is existential. It does not show that the map can be learned from
finite data or that it generalises [arxiv:2507.08802 §7]. Injectivity holds
almost surely for transformers at initialisation with real weights
[arxiv:2507.08802 app. G]. Finite precision and neural collapse may break it in
practice, and the softmax bottleneck may break output surjectivity
[arxiv:2507.08802 §4] [arxiv:2507.08802 §8].

## Evidence and controls

All results are author-reported [arxiv:2507.08802 §5]. Interchange intervention
accuracy (IIA) is the fraction of held-out interventions where the patched
network's argmax equals the algorithm's output [arxiv:2507.08802 §5].

- **Equality MLP.** Three hidden ReLU layers of width 16 [arxiv:2507.08802 §5]
  [arxiv:2507.08802 app. E]. Results appear in figures and tables
  [arxiv:2507.08802 fig. 2] [arxiv:2507.08802 §6].
- **Pythia indirect object identification.** Across five Pythia-410m seeds,
  RevNet aligns perfectly for three, aligns poorly for one, and one seed does not
  learn the task [arxiv:2507.08802 app. I.2]. With name-disjoint train and eval
  on random-init models, no alignment was found. On fully trained name-disjoint
  models, RevNet is perfect and rotation is reasonably high
  [arxiv:2507.08802 app. I.2].
- **Distributive-law task.** RevNet is near perfect across the settings reported
  [arxiv:2507.08802 app. I.3].
- **Rotation above the network.** Rotation sometimes reaches higher IIA than the
  network's own accuracy. The authors suggest this may mean the map learns the
  task itself [arxiv:2507.08802 §7].

## Contradictions

- **Headline versus seeds.** The introduction reports 100% IIA for randomly
  initialised Pythia with RevNet [arxiv:2507.08802 §1]. The appendix reports one
  poor seed and one non-learning seed out of five [arxiv:2507.08802 app. I.2].
  The authors attribute the failures to suboptimal alignment training and did
  not tune further [arxiv:2507.08802 app. I.2].
- **Different equality settings.** The equality MLP with a random-init layer one
  reaches above 80% with RevNet [arxiv:2507.08802 §6]. On the trained MLP with
  identity-of-first-argument, RevNet reaches under 0.55 train IIA with 1.28M
  samples [arxiv:2507.08802 app. H]. These are different settings, and the paper
  does not reconcile them.

## Implementation sketch

Pseudocode for the common core: an interchange intervention, and one DAS step
with the network frozen. Only φ receives gradients.

```pseudocode
# Interchange intervention at one layer  [arxiv:2507.08802 §3.2]
def interchange(phi, phi_inv, base_h, source_h, mask):
    # Map both runs into the latent basis chosen by phi
    base_z = phi(base_h)
    source_z = phi(source_h)
    # Overwrite the coordinates assigned to the target variable
    # mask is True where a coordinate comes from the source run
    z = where(mask, source_z, base_z)
    # Return to the neuron basis; the forward pass continues from here
    return phi_inv(z)

# One DAS training step; the network stays frozen  [arxiv:2507.08802 §3.3]
def das_step(net, layer, phi, batch):
    base_h = net.hidden(layer, batch.base)
    source_h = net.hidden(layer, batch.source)
    patched = interchange(phi, phi.inverse, base_h, source_h, batch.mask)
    logits = net.forward_from(layer, patched)
    # Target: the algorithm's output under the same intervention
    return cross_entropy(logits, batch.algorithm_output)
```

The inverse must be exact. For the orthogonal variant it is the transpose; for
RevNet it is built into the architecture [arxiv:2507.08802 §5].

Implementation notes. The card's
[Paper vs code](/papers/2507.08802) section lists where the released scripts
differ from the paper: learning rates, epoch counts, early-stopping rules, and
which alignment map is kept. Check it before copying any hyperparameter. The
card also flags a possible label inversion in the IOI data preprocessing that
has not been verified.

## Papers

- [Non-Linear Representation Dilemma](/papers/2507.08802): shows that an
  unrestricted invertible alignment map makes causal abstraction vacuous, and
  that the map class must be restricted or controlled.