wikillm

Concept · 1 paper · updated 2026-11-21

Conditional randomisation test

One source so far. Tests whether a model responds to actions, using the known action-randomisation law as the null, and returns an e-value valid whatever the model. Pooling site scores collapses when sites share one action sequence; a fixed-grid tempering mixture avoids this. Trade-off: the law must be known, and simpler replacement checks can fire with no effect.

hypothesis-testinge-valuesmodel-evaluation

Markdown5 min read

AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

To ask whether a model's predictions depend on an action input, the usual check removes or swaps the actions and watches the loss. The conditional randomisation test instead redraws the future actions from the randomisation law that generated them, then compares the observed block with the redraws 2611.01666 §2.4 §3.1. Under no effect the observed block is one of many equally likely draws, so the false-alarm rate holds whatever the model is. The model only sets the power 2611.01666 §3. The output is an e-value, and a rejection needs E >= 1/alpha, or E >= m/alpha for m prespecified hypotheses 2611.01666 §3.2.

Variants

Replacement diagnostics (not a valid test)

Zeroing the actions, swapping in another unit's actions, or retraining without them. When actions are autocorrelated, these checks can flag an effect that does not exist, and the chance of doing so tends to one under the global null, because the future block stands in for omitted history 2611.01666 §4.4 §5.1. The authors show this on ERK and CcaS designs 2611.01666 §5.2 and in a pixel world whose logging policy the model cannot see 2611.01666 §7.1. Card: Stand-In Effects and Pooling Amplification.

Pooled score

Score a candidate block by summing per-site Gaussian log-likelihoods, with each site's density floored before the log 2611.01666 §3.1 eq. 1. Redraw B = 127 blocks from the design law and form the Monte Carlo e-value, capped at B + 1 2611.01666 eq. 3 §9.1. This is the test with tempering factor lambda = 1 2611.01666 §3.4. When many sites share one action sequence, the sum grows with the site count and drives the e-value towards zero when the model's mistakes line up with the action contrast 2611.01666 §3.3 §6.1. In simulation, the median log e-value fell as pixels grew from 64 to 4096 2611.01666 §6.3.

Fixed-grid tempering mixture

Replace the pooled score S by T = lambda * S with 0 < lambda <= 1 2611.01666 eq. 6 §3.4, then average whole-unit e-values over the grid lambda_g = 2^-g, g = 0..G-1, with G = 15 (13 in the pixel world) 2611.01666 eq. 9 §3.5 §6.4. The grid is fixed before any test, so no test outcome chooses it. On DIAMOND the mixture rejected 96 of 96 active units, while the pooled score rejected none 2611.01666 §9.2. The mixture can lose up to log G against the best grid member, and averages are taken per decision unit, not per origin 2611.01666 §3.5 §4.5.

Validation-weighted mixture

Put weight w on the grid factor with the largest median validation log e-value and spread 1 - w evenly over the other factors 2611.01666 eq. 10 §3.5. The primary choice is w = 3/4, fixed before test data existed 2611.01666 §3.5 §10.1. On IRIS it rejected in 24 of 26 games, and on TD-MPC2 in all 6 tasks, where the pooled score rejected in 1 of 6 2611.01666 §10.2. The factor is picked on validation data, so these results depend on how that split was made.

Estimated factor (Kish)

Set lambda = 1 / (1 + (n - 1) * rho), where rho is the median over training fields of each field's median pairwise residual correlation, clipped to [0, 1] 2611.01666 eq. 7 §3.4. On one unopened ERK field at 10 minutes, the registered test gave log e-value 23.1 and rejected at E >= 20 2611.01666 §8.4. The estimate is fragile: on DIAMOND, the Kish and contrast estimates were roughly 11 to 82 times the best grid factor in the median unit 2611.01666 §12. On ERK, contrast is shared across cells, so agreement with Kish is partly built in 2611.01666 §8.5. A contrast-based factor 2611.01666 eq. 8 is exploratory, and the authors do not recommend it 2611.01666 §6.3 §9.3.

When to use which

SituationUseWhy
The action-randomisation law is known and the full recorded state is availableConditional randomisation test, redraws conditioned on the full historyValidity holds whatever the model; the resampler must condition on the complete state 2611.01666 §7.2
The law is reconstructed, approximate or undocumentedReport as provisional; do not treat the result as a clean testThe paper's total-variation bound does not license its real-data tests 2611.01666 §4.3; validity depends on the law 2611.01666 §13
The law is unknown or cannot be recoveredNo valid test from this paper; do not fall back on replacement checksReplacement checks can fire with no effect 2611.01666 §4.4 §13
Many sites (cells, pixels) share one action sequenceFixed-grid mixture, not the pooled scoreThe pooled score collapses with shared sites 2611.01666 §3.3 §6.1
Validation data exist and you want one reported statisticValidation-weighted mixture, w = 3/4Rejected in 24 of 26 IRIS games and all 6 TD-MPC2 tasks 2611.01666 §10.2
Only training residuals are available and you want a single factorKish estimate, checked against the gridOne confirmatory rejection on one field 2611.01666 §8.4; can sit far from the best grid factor 2611.01666 §12

Every result is author-reported; the paper's own simulations and real-data tests are the evidence 2611.01666 §9 §10 §11.

Implementation sketch

Pseudocode from the paper, shortened. The score and e-value follow equations 1 and 3; the mixture follows equation 9.

Pseudocode
# Pooled log score of one candidate block  [arxiv:2611.01666 §3.1] [eq. 1]
# Input:  block (h actions); targets (n); histories (n); model; floor
# Output: scalar log score

def pooled_score(block, targets, histories, model, floor):
    # Predictive mean and variance per site, fixed on training data  [§3.1]
    means, variances = model.predict(histories, block)
    # Gaussian density of each observed target  [§3.1]
    densities = gaussian_pdf(targets, means, variances)
    # Floor each site's density before taking the log  [§3.1] [eq. 1]
    floored = maximum(densities, floor)
    # Sum over sites: independence log-likelihood  [§3.1]
    return sum(log(floored))
Pseudocode
# Monte Carlo e-value at one origin  [arxiv:2611.01666 eq. 3]
# Input:  observed block; targets; histories; model; design law; B
# Output: e-value in [1, B + 1]

def mc_e_value(observed, targets, histories, model, design, floor, B):
    # Score the observed future actions  [§3.1]
    obs = pooled_score(observed, targets, histories, model, floor)
    # Draw B blocks from the known design law at this origin  [§3.1]
    draws = [design.sample(histories) for _ in range(B)]
    # Score each redraw against the same targets and histories  [§3.1]
    draw_scores = [
        pooled_score(a, targets, histories, model, floor)
        for a in draws
    ]
    # Same value as eq. 3, computed in log space  [eq. 3]
    log_total = logsumexp([obs] + draw_scores)
    return exp(log(B + 1) + obs - log_total)
Pseudocode
# Fixed-grid mixture over whole units  [arxiv:2611.01666 eq. 9] [§3.5]
# Input:  unit; grid size G
# Output: mean of the G whole-unit e-values

def mixture_e_value(unit, G):
    # Factors fixed before any test: lambda_g = 2^-g  [§3.5]
    factors = [2.0 ** -g for g in range(G)]
    # Whole-unit e-value at each tempered score T = lambda * S  [eq. 6]
    unit_values = [
        tempered_unit_e_value(unit, lam) for lam in factors
    ]
    # Average the grid members per decision unit  [eq. 9]
    return sum(unit_values) / G

Variants differ in the factor. The pooled test uses lambda = 1. The mixture averages over a grid. The validation-weighted version replaces the plain average with weight w on the validation-chosen member eq. 10.

Contradictions

None known across sources; this page has one card. Within the paper, the authors say the fixed-grid mixture never failed in their tests [arxiv:2611.01666 §9.2] §10.2, yet its bound allows a loss of up to log G against the best grid member 2611.01666 §3.5 §4.5. Both are the authors' statements; they are not resolved here.

Papers

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.