---
kind: concept
slug: conditional-randomization-test
title: Conditional randomisation test
summary: >-
  One source so far. Tests whether a model responds to actions, using the known action-randomisation law as the null, and returns an e-value valid whatever the model. Pooling site scores collapses when sites share one action sequence; a fixed-grid tempering mixture avoids this. Trade-off: the law must be known, and simpler replacement checks can fire with no effect.
tags: [hypothesis-testing, e-values, model-evaluation]
papers: ["2611.01666"]
related: [world-model, causal-inference, conformal-prediction]
status: draft
updated_at: 2026-11-21
---

## Overview

To ask whether a model's predictions depend on an action input, the usual check
removes or swaps the actions and watches the loss. The conditional randomisation
test instead redraws the future actions from the randomisation law that
generated them, then compares the observed block with the redraws
[arxiv:2611.01666 §2.4] [§3.1]. Under no effect the observed block is one of
many equally likely draws, so the false-alarm rate holds whatever the model is.
The model only sets the power [arxiv:2611.01666 §3]. The output is an e-value,
and a rejection needs `E >= 1/alpha`, or `E >= m/alpha` for `m` prespecified
hypotheses [arxiv:2611.01666 §3.2].

## Variants

### Replacement diagnostics (not a valid test)

Zeroing the actions, swapping in another unit's actions, or retraining without
them. When actions are autocorrelated, these checks can flag an effect that
does not exist, and the chance of doing so tends to one under the global null,
because the future block stands in for omitted history [arxiv:2611.01666 §4.4] [§5.1]. The authors show this on
ERK and CcaS designs [arxiv:2611.01666 §5.2] and in a pixel world whose logging
policy the model cannot see [arxiv:2611.01666 §7.1]. Card:
[Stand-In Effects and Pooling Amplification](/papers/2611.01666).

### Pooled score

Score a candidate block by summing per-site Gaussian log-likelihoods, with each
site's density floored before the log [arxiv:2611.01666 §3.1] [eq. 1]. Redraw
`B = 127` blocks from the design law and form the Monte Carlo e-value, capped at
`B + 1` [arxiv:2611.01666 eq. 3] [§9.1]. This is the test with tempering factor
`lambda = 1` [arxiv:2611.01666 §3.4]. When many sites share one action
sequence, the sum grows with the site count and drives the e-value towards zero
when the model's mistakes line up with the action contrast [arxiv:2611.01666 §3.3] [§6.1]. In
simulation, the median log e-value fell as pixels grew from 64 to 4096
[arxiv:2611.01666 §6.3].

### Fixed-grid tempering mixture

Replace the pooled score `S` by `T = lambda * S` with `0 < lambda <= 1`
[arxiv:2611.01666 eq. 6] [§3.4], then average whole-unit e-values over the grid
`lambda_g = 2^-g`, `g = 0..G-1`, with `G = 15` (13 in the pixel world)
[arxiv:2611.01666 eq. 9] [§3.5] [§6.4]. The grid is fixed before any test, so
no test outcome chooses it. On DIAMOND the mixture rejected 96 of 96 active
units, while the pooled score rejected none [arxiv:2611.01666 §9.2]. The mixture
can lose up to `log G` against the best grid member, and averages are taken per
decision unit, not per origin [arxiv:2611.01666 §3.5] [§4.5].

### Validation-weighted mixture

Put weight `w` on the grid factor with the largest median validation log e-value
and spread `1 - w` evenly over the other factors [arxiv:2611.01666 eq. 10]
[§3.5]. The primary choice is `w = 3/4`, fixed before test data existed
[arxiv:2611.01666 §3.5] [§10.1]. On IRIS it rejected in 24 of 26 games, and on
TD-MPC2 in all 6 tasks, where the pooled score rejected in 1 of 6
[arxiv:2611.01666 §10.2]. The factor is picked on validation data, so these
results depend on how that split was made.

### Estimated factor (Kish)

Set `lambda = 1 / (1 + (n - 1) * rho)`, where `rho` is the median over training
fields of each field's median pairwise residual correlation, clipped to
`[0, 1]` [arxiv:2611.01666 eq. 7] [§3.4]. On one unopened ERK field at 10
minutes, the registered test gave log e-value 23.1 and rejected at `E >= 20`
[arxiv:2611.01666 §8.4]. The estimate is fragile: on DIAMOND, the Kish and
contrast estimates were roughly 11 to 82 times the best grid factor in the
median unit [arxiv:2611.01666 §12]. On ERK, contrast is shared across cells, so
agreement with Kish is partly built in [arxiv:2611.01666 §8.5]. A contrast-based
factor [arxiv:2611.01666 eq. 8] is exploratory, and the authors do not recommend
it [arxiv:2611.01666 §6.3] [§9.3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| The action-randomisation law is known and the full recorded state is available | Conditional randomisation test, redraws conditioned on the full history | Validity holds whatever the model; the resampler must condition on the complete state [arxiv:2611.01666 §7.2] |
| The law is reconstructed, approximate or undocumented | Report as provisional; do not treat the result as a clean test | The paper's total-variation bound does not license its real-data tests [arxiv:2611.01666 §4.3]; validity depends on the law [arxiv:2611.01666 §13] |
| The law is unknown or cannot be recovered | No valid test from this paper; do not fall back on replacement checks | Replacement checks can fire with no effect [arxiv:2611.01666 §4.4] [§13] |
| Many sites (cells, pixels) share one action sequence | Fixed-grid mixture, not the pooled score | The pooled score collapses with shared sites [arxiv:2611.01666 §3.3] [§6.1] |
| Validation data exist and you want one reported statistic | Validation-weighted mixture, `w = 3/4` | Rejected in 24 of 26 IRIS games and all 6 TD-MPC2 tasks [arxiv:2611.01666 §10.2] |
| Only training residuals are available and you want a single factor | Kish estimate, checked against the grid | One confirmatory rejection on one field [arxiv:2611.01666 §8.4]; can sit far from the best grid factor [arxiv:2611.01666 §12] |

Every result is author-reported; the paper's own simulations and real-data tests
are the evidence [arxiv:2611.01666 §9] [§10] [§11].

## Implementation sketch

Pseudocode from the paper, shortened. The score and e-value follow equations 1
and 3; the mixture follows equation 9.

```pseudocode
# Pooled log score of one candidate block  [arxiv:2611.01666 §3.1] [eq. 1]
# Input:  block (h actions); targets (n); histories (n); model; floor
# Output: scalar log score

def pooled_score(block, targets, histories, model, floor):
    # Predictive mean and variance per site, fixed on training data  [§3.1]
    means, variances = model.predict(histories, block)
    # Gaussian density of each observed target  [§3.1]
    densities = gaussian_pdf(targets, means, variances)
    # Floor each site's density before taking the log  [§3.1] [eq. 1]
    floored = maximum(densities, floor)
    # Sum over sites: independence log-likelihood  [§3.1]
    return sum(log(floored))
```

```pseudocode
# Monte Carlo e-value at one origin  [arxiv:2611.01666 eq. 3]
# Input:  observed block; targets; histories; model; design law; B
# Output: e-value in [1, B + 1]

def mc_e_value(observed, targets, histories, model, design, floor, B):
    # Score the observed future actions  [§3.1]
    obs = pooled_score(observed, targets, histories, model, floor)
    # Draw B blocks from the known design law at this origin  [§3.1]
    draws = [design.sample(histories) for _ in range(B)]
    # Score each redraw against the same targets and histories  [§3.1]
    draw_scores = [
        pooled_score(a, targets, histories, model, floor)
        for a in draws
    ]
    # Same value as eq. 3, computed in log space  [eq. 3]
    log_total = logsumexp([obs] + draw_scores)
    return exp(log(B + 1) + obs - log_total)
```

```pseudocode
# Fixed-grid mixture over whole units  [arxiv:2611.01666 eq. 9] [§3.5]
# Input:  unit; grid size G
# Output: mean of the G whole-unit e-values

def mixture_e_value(unit, G):
    # Factors fixed before any test: lambda_g = 2^-g  [§3.5]
    factors = [2.0 ** -g for g in range(G)]
    # Whole-unit e-value at each tempered score T = lambda * S  [eq. 6]
    unit_values = [
        tempered_unit_e_value(unit, lam) for lam in factors
    ]
    # Average the grid members per decision unit  [eq. 9]
    return sum(unit_values) / G
```

Variants differ in the factor. The pooled test uses `lambda = 1`. The
mixture averages over a grid. The validation-weighted version replaces the
plain average with weight `w` on the validation-chosen member [eq. 10].

## Contradictions

None known across sources; this page has one card. Within the paper, the
authors say the fixed-grid mixture never failed in their tests [arxiv:2611.01666
§9.2] [§10.2], yet its bound allows a loss of up to `log G` against the best
grid member [arxiv:2611.01666 §3.5] [§4.5]. Both are the authors' statements;
they are not resolved here.

## Papers

- [Stand-In Effects and Pooling Amplification](/papers/2611.01666): introduces the conditional randomisation test for world models, the pooling collapse, and the fixed-grid and validation-weighted mixtures.
