Causal abstraction

Overview

Causal abstraction asks whether a neural network computes what a simpler, step-by-step algorithm computes. The algorithm is a graph of named intermediate variables that ends in the task output. The claimed match is a map from the network's hidden states to those variables. The match passes the interventional test if overwriting a variable in the algorithm has the same effect as overwriting the matched part of the network 2507.08802 §2.1 2507.08802 §3.

The invariant all variants share is that interventions must commute with the map: intervene on the network, read off the mapped variables, and the result must agree with intervening on the algorithm. The variants differ in how flexible the map may be. A more flexible map can fit more, so a good score proves less 2507.08802 §1 2507.08802 §4.

The paper that anchors this page argues that the flexible end of that scale is vacuous. Under a few mild conditions, an invertible map can align any network with any algorithm 2507.08802 §1 2507.08802 §4. Distributed alignment search (DAS), introduced in earlier work by Geiger and colleagues (arXiv:2301.04709, arXiv:2303.02536), is the training procedure the variants below use. Those two papers have no card here, so they are named only.

Variants

Constructive abstraction (neuron partition)

Each algorithm variable reads its own disjoint set of hidden neurons, and some neurons may stay unused. No map is learned. The search only assigns neurons to variables. The paper's identity-map experiments build this partition greedily, adding one neuron at a time to each variable's set 2507.08802 §3 2507.08802 §5. On an equality task, identity, rotation and RevNet maps all score about 50% for the identity-of-first-argument algorithm 2507.08802 §6. Card: Non-Linear Representation Dilemma.

Distributed abstraction with an orthogonal map

A learned bijection φ first rotates the hidden states into latent coordinates. Each variable then reads its own latent coordinates. An intervention maps to latent space, overwrites the chosen coordinates from a source run, and maps back 2507.08802 §3.2 2507.08802 app. B. The paper's linear variant restricts φ to an orthogonal matrix 2507.08802 §5. The paper notes that most prior work implicitly assumes the linear representation hypothesis, which limits the map to linear or orthogonal transformations 2507.08802 §1 2507.08802 §5. DAS fits φ with the network frozen. The loss is cross-entropy between the intervened network's output and the algorithm's output under the same intervention 2507.08802 §3.3.

Distributed abstraction with a non-linear invertible map (RevNet)

φ is a reversible residual network, so its inverse is exact and the map can be non-linear 2507.08802 §5. The paper reports high fits for this variant even on untrained models. For the equality MLP, the most complex RevNet at layer one of a randomly initialised network scores above 80% 2507.08802 §6. For Pythia language models the paper reports 100% for randomly initialised models 2507.08802 §1 2507.08802 §6. The paper argues that this result shows the test does not tell a real implementation from a coincidence. Use this variant only with the controls in the sections below. Card: Non-Linear Representation Dilemma.

Strong versus input-restricted abstraction

The definition can require the map to realise every intervention, or only the interventions that other inputs can produce. The paper says the input-restricted version is the one used in practice, and its theorem is about that version 2507.08802 §3.2.

When to use which

Practical constraints first. The method needs hidden states at a chosen layer and forward passes that resume from that layer, so it does not apply to black-box models 2507.08802 §2.2 2507.08802 §3.3. Training uses about 1.28 million interventions for the equality MLP 2507.08802 app. I.1 and 1 million for the Pythia runs 2507.08802 app. I.2. The paper reports about three weeks on ten CPU machines for the MLP and about 30 GPU hours on one A100 for the Pythia runs, including tuning 2507.08802 app. J.

Assumptions behind the existence result

The paper's theorem says that, under five conditions, the algorithm is an input-restricted distributed abstraction of the network 2507.08802 §4 2507.08802 app. F. The conditions are: the input set is countable, which holds for text and approximately for images at fixed resolution; every prefix of the network is injective; the suffix of the network followed by the output map is surjective onto the outputs; a single-neuron assignment respects the algorithm's order, with at least one unused neuron per layer; and the network gets every input right 2507.08802 §4 2507.08802 app. F.

The proof is existential. It does not show that the map can be learned from finite data or that it generalises 2507.08802 §7. Injectivity holds almost surely for transformers at initialisation with real weights 2507.08802 app. G. Finite precision and neural collapse may break it in practice, and the softmax bottleneck may break output surjectivity 2507.08802 §4 2507.08802 §8.

Evidence and controls

All results are author-reported 2507.08802 §5. Interchange intervention accuracy (IIA) is the fraction of held-out interventions where the patched network's argmax equals the algorithm's output 2507.08802 §5.

• Equality MLP. Three hidden ReLU layers of width 16 2507.08802 §5 2507.08802 app. E. Results appear in figures and tables 2507.08802 fig. 2 2507.08802 §6.
• Pythia indirect object identification. Across five Pythia-410m seeds, RevNet aligns perfectly for three, aligns poorly for one, and one seed does not learn the task 2507.08802 app. I.2. With name-disjoint train and eval on random-init models, no alignment was found. On fully trained name-disjoint models, RevNet is perfect and rotation is reasonably high 2507.08802 app. I.2.
• Distributive-law task. RevNet is near perfect across the settings reported 2507.08802 app. I.3.
• Rotation above the network. Rotation sometimes reaches higher IIA than the network's own accuracy. The authors suggest this may mean the map learns the task itself 2507.08802 §7.

Contradictions

• Headline versus seeds. The introduction reports 100% IIA for randomly initialised Pythia with RevNet 2507.08802 §1. The appendix reports one poor seed and one non-learning seed out of five 2507.08802 app. I.2. The authors attribute the failures to suboptimal alignment training and did not tune further 2507.08802 app. I.2.
• Different equality settings. The equality MLP with a random-init layer one reaches above 80% with RevNet 2507.08802 §6. On the trained MLP with identity-of-first-argument, RevNet reaches under 0.55 train IIA with 1.28M samples 2507.08802 app. H. These are different settings, and the paper does not reconcile them.

Implementation sketch

Pseudocode for the common core: an interchange intervention, and one DAS step with the network frozen. Only φ receives gradients.

# Interchange intervention at one layer  [arxiv:2507.08802 §3.2]
def interchange(phi, phi_inv, base_h, source_h, mask):
    # Map both runs into the latent basis chosen by phi
    base_z = phi(base_h)
    source_z = phi(source_h)
    # Overwrite the coordinates assigned to the target variable
    # mask is True where a coordinate comes from the source run
    z = where(mask, source_z, base_z)
    # Return to the neuron basis; the forward pass continues from here
    return phi_inv(z)

# One DAS training step; the network stays frozen  [arxiv:2507.08802 §3.3]
def das_step(net, layer, phi, batch):
    base_h = net.hidden(layer, batch.base)
    source_h = net.hidden(layer, batch.source)
    patched = interchange(phi, phi.inverse, base_h, source_h, batch.mask)
    logits = net.forward_from(layer, patched)
    # Target: the algorithm's output under the same intervention
    return cross_entropy(logits, batch.algorithm_output)

The inverse must be exact. For the orthogonal variant it is the transpose; for RevNet it is built into the architecture 2507.08802 §5.

Implementation notes. The card's Paper vs code section lists where the released scripts differ from the paper: learning rates, epoch counts, early-stopping rules, and which alignment map is kept. Check it before copying any hyperparameter. The card also flags a possible label inversion in the IOI data preprocessing that has not been verified.

Papers

• Non-Linear Representation Dilemma: shows that an unrestricted invertible alignment map makes causal abstraction vacuous, and that the map class must be restricted or controlled.