Concept · 1 paper · updated 2026-11-21
Conditional randomisation test
One source so far. Tests whether a model responds to actions, using the known action-randomisation law as the null, and returns an e-value valid whatever the model. Pooling site scores collapses when sites share one action sequence; a fixed-grid tempering mixture avoids this. Trade-off: the law must be known, and simpler replacement checks can fire with no effect.
hypothesis-testinge-valuesmodel-evaluation
Markdown5 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
To ask whether a model's predictions depend on an action input, the usual check
removes or swaps the actions and watches the loss. The conditional randomisation
test instead redraws the future actions from the randomisation law that
generated them, then compares the observed block with the redraws
2611.01666 §2.4 §3.1. Under no effect the observed block is one of
many equally likely draws, so the false-alarm rate holds whatever the model is.
The model only sets the power 2611.01666 §3. The output is an e-value,
and a rejection needs E >= 1/alpha, or E >= m/alpha for m prespecified
hypotheses 2611.01666 §3.2.
Variants
Replacement diagnostics (not a valid test)
Zeroing the actions, swapping in another unit's actions, or retraining without them. When actions are autocorrelated, these checks can flag an effect that does not exist, and the chance of doing so tends to one under the global null, because the future block stands in for omitted history 2611.01666 §4.4 §5.1. The authors show this on ERK and CcaS designs 2611.01666 §5.2 and in a pixel world whose logging policy the model cannot see 2611.01666 §7.1. Card: Stand-In Effects and Pooling Amplification.
Pooled score
Score a candidate block by summing per-site Gaussian log-likelihoods, with each
site's density floored before the log 2611.01666 §3.1 eq. 1. Redraw
B = 127 blocks from the design law and form the Monte Carlo e-value, capped at
B + 1 2611.01666 eq. 3 §9.1. This is the test with tempering factor
lambda = 1 2611.01666 §3.4. When many sites share one action
sequence, the sum grows with the site count and drives the e-value towards zero
when the model's mistakes line up with the action contrast 2611.01666 §3.3 §6.1. In
simulation, the median log e-value fell as pixels grew from 64 to 4096
2611.01666 §6.3.
Fixed-grid tempering mixture
Replace the pooled score S by T = lambda * S with 0 < lambda <= 1
2611.01666 eq. 6 §3.4, then average whole-unit e-values over the grid
lambda_g = 2^-g, g = 0..G-1, with G = 15 (13 in the pixel world)
2611.01666 eq. 9 §3.5 §6.4. The grid is fixed before any test, so
no test outcome chooses it. On DIAMOND the mixture rejected 96 of 96 active
units, while the pooled score rejected none 2611.01666 §9.2. The mixture
can lose up to log G against the best grid member, and averages are taken per
decision unit, not per origin 2611.01666 §3.5 §4.5.
Validation-weighted mixture
Put weight w on the grid factor with the largest median validation log e-value
and spread 1 - w evenly over the other factors 2611.01666 eq. 10
§3.5. The primary choice is w = 3/4, fixed before test data existed
2611.01666 §3.5 §10.1. On IRIS it rejected in 24 of 26 games, and on
TD-MPC2 in all 6 tasks, where the pooled score rejected in 1 of 6
2611.01666 §10.2. The factor is picked on validation data, so these
results depend on how that split was made.
Estimated factor (Kish)
Set lambda = 1 / (1 + (n - 1) * rho), where rho is the median over training
fields of each field's median pairwise residual correlation, clipped to
[0, 1] 2611.01666 eq. 7 §3.4. On one unopened ERK field at 10
minutes, the registered test gave log e-value 23.1 and rejected at E >= 20
2611.01666 §8.4. The estimate is fragile: on DIAMOND, the Kish and
contrast estimates were roughly 11 to 82 times the best grid factor in the
median unit 2611.01666 §12. On ERK, contrast is shared across cells, so
agreement with Kish is partly built in 2611.01666 §8.5. A contrast-based
factor 2611.01666 eq. 8 is exploratory, and the authors do not recommend
it 2611.01666 §6.3 §9.3.
When to use which
| Situation | Use | Why |
|---|---|---|
| The action-randomisation law is known and the full recorded state is available | Conditional randomisation test, redraws conditioned on the full history | Validity holds whatever the model; the resampler must condition on the complete state 2611.01666 §7.2 |
| The law is reconstructed, approximate or undocumented | Report as provisional; do not treat the result as a clean test | The paper's total-variation bound does not license its real-data tests 2611.01666 §4.3; validity depends on the law 2611.01666 §13 |
| The law is unknown or cannot be recovered | No valid test from this paper; do not fall back on replacement checks | Replacement checks can fire with no effect 2611.01666 §4.4 §13 |
| Many sites (cells, pixels) share one action sequence | Fixed-grid mixture, not the pooled score | The pooled score collapses with shared sites 2611.01666 §3.3 §6.1 |
| Validation data exist and you want one reported statistic | Validation-weighted mixture, w = 3/4 | Rejected in 24 of 26 IRIS games and all 6 TD-MPC2 tasks 2611.01666 §10.2 |
| Only training residuals are available and you want a single factor | Kish estimate, checked against the grid | One confirmatory rejection on one field 2611.01666 §8.4; can sit far from the best grid factor 2611.01666 §12 |
Every result is author-reported; the paper's own simulations and real-data tests are the evidence 2611.01666 §9 §10 §11.
Implementation sketch
Pseudocode from the paper, shortened. The score and e-value follow equations 1 and 3; the mixture follows equation 9.
# Pooled log score of one candidate block [arxiv:2611.01666 §3.1] [eq. 1]
# Input: block (h actions); targets (n); histories (n); model; floor
# Output: scalar log score
def pooled_score(block, targets, histories, model, floor):
# Predictive mean and variance per site, fixed on training data [§3.1]
means, variances = model.predict(histories, block)
# Gaussian density of each observed target [§3.1]
densities = gaussian_pdf(targets, means, variances)
# Floor each site's density before taking the log [§3.1] [eq. 1]
floored = maximum(densities, floor)
# Sum over sites: independence log-likelihood [§3.1]
return sum(log(floored))# Monte Carlo e-value at one origin [arxiv:2611.01666 eq. 3]
# Input: observed block; targets; histories; model; design law; B
# Output: e-value in [1, B + 1]
def mc_e_value(observed, targets, histories, model, design, floor, B):
# Score the observed future actions [§3.1]
obs = pooled_score(observed, targets, histories, model, floor)
# Draw B blocks from the known design law at this origin [§3.1]
draws = [design.sample(histories) for _ in range(B)]
# Score each redraw against the same targets and histories [§3.1]
draw_scores = [
pooled_score(a, targets, histories, model, floor)
for a in draws
]
# Same value as eq. 3, computed in log space [eq. 3]
log_total = logsumexp([obs] + draw_scores)
return exp(log(B + 1) + obs - log_total)# Fixed-grid mixture over whole units [arxiv:2611.01666 eq. 9] [§3.5]
# Input: unit; grid size G
# Output: mean of the G whole-unit e-values
def mixture_e_value(unit, G):
# Factors fixed before any test: lambda_g = 2^-g [§3.5]
factors = [2.0 ** -g for g in range(G)]
# Whole-unit e-value at each tempered score T = lambda * S [eq. 6]
unit_values = [
tempered_unit_e_value(unit, lam) for lam in factors
]
# Average the grid members per decision unit [eq. 9]
return sum(unit_values) / GVariants differ in the factor. The pooled test uses lambda = 1. The
mixture averages over a grid. The validation-weighted version replaces the
plain average with weight w on the validation-chosen member eq. 10.
Contradictions
None known across sources; this page has one card. Within the paper, the
authors say the fixed-grid mixture never failed in their tests [arxiv:2611.01666
§9.2] §10.2, yet its bound allows a loss of up to log G against the best
grid member 2611.01666 §3.5 §4.5. Both are the authors' statements;
they are not resolved here.
Papers
- Stand-In Effects and Pooling Amplification: introduces the conditional randomisation test for world models, the pooling collapse, and the fixed-grid and validation-weighted mixtures.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.