Paper · 2026-10-21 · cs.LG
Loopholing All the Way: Language Modeling with pure Self-Conditioning
LAW is a diffusion language model built only on self-conditioning: each denoiser pass passes the next one the Expected Embedding under its predicted token distribution, and tokens are decoded once at the end. Training unrolls one or two passes with a stop-gradient and random temperature scaling. Author-reported: 42.2% on GSM8K at 1024 NFE (LDDM-U 34.5%) and 96.5% exact match on hard Sudoku.
Discrete diffusionContinuous text diffusionSelf-conditioningdiffusionlanguage-modelingself-conditioninguniform-diffusionsampling
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Many text diffusion models start from random tokens and clean them up over several rounds. Each round usually keeps only the words it picked and throws away the rest of what the model thought. Self-conditioning is a trick that hands the previous round's guess back to the model as an extra input. So far it has been treated as a helper on top of the usual word-by-word updates.
This paper asks whether self-conditioning can carry the whole process. Its model, LAW, never picks words between rounds. It passes along a blended "soft" guess: for each position, the average of the word vectors weighted by how likely each word is. Only after the last round does it turn that guess into actual words. Training prepares the model for this: it runs one or two rounds, blocks gradients through the first round, and sometimes sharpens or flattens the probabilities at random. The authors report that this beats earlier diffusion models on Sudoku and on small math and language tasks at matched training compute, with caveats.
TL;DR
LAW is a continuous-state diffusion language model. Its sampler carries only the Expected Embedding of the denoiser's prediction from step to step and decodes tokens once, after the final step §3.1. The training recipe is uniform discrete diffusion with one or two unrolled passes, a stop-gradient between passes and random temperature regularization §3.2. Author-reported results beat the uniform Loopholing baseline (LDDM-U) on GSM8K, Sudoku and Maze, and LAW-style fine-tuning of a Duo checkpoint raises GSM8K from 17.2% to 48.1% at 1024 NFE §4.3.
Problem
In discrete diffusion, each position is drawn separately and the model's guess is thrown away after every step. Factorized parallel sampling hurts quality in few-step generation, and the discarded guess means its uncertainty is lost (information collapse) §1. Loopholing (LDDM) and other self-conditioning methods keep a continuous state across steps, but they still sample discrete tokens between steps and treat the state as an auxiliary input §1 §2.2. Continuous diffusion language models avoid factorized sampling but, according to the authors, at most match discrete models on reasoning tasks §1. The paper asks whether a model can be based purely on self-conditioning §1.
Method
Background (uniform discrete diffusion)
The forward process corrupts each clean token to a uniform random token with probability 1 - α_t, independently per position §2.1 eq. 1. The denoiser is a DiT: token embeddings, a stack of Transformer blocks, an LM head and a softmax §2.1 eq. 2. Sampling is ancestral over a uniform time grid t_n = n/N, where N is the number of function evaluations (NFE) §2.1 eq. 4. Each step uses the bridge plug-in posterior evaluated at the predicted distribution §2.1 app. A.2. The predictions are discarded after each step, which the paper calls information collapse §2.2.
LDDM adds a self-conditioning state c, the layer-normalised final hidden representation, which is added to the token embeddings at the next step §2.2 eq. 5. LDDM still samples tokens at every step.
Generative process (LAW)
- Initialization. Draw uniform prior tokens and embed them: c_{t_N} = z_{t_N} E_θ §3.1 eq. 6.
- Continuous state updates. For n = N, ..., 2: pass the state through the backbone at time t_n, then map the hidden state to the next state with the Expected Embedding map §3.1 eq. 7 eq. 8. No tokens are sampled between steps.
- Decoding. The final pass at t_1 gives a distribution per position. Draw each token from it with the raw decoder §3.1 eq. 9 app. C.1.
The state map returns a probability-weighted average of token embeddings, with the weights computed from the predicted distribution at temperature T eq. 8. The sampling temperature T is fixed, usually 1 §3.1. The state replaces the token embeddings as the only denoiser input; LDDM instead adds its state to them §3.1 app. A.4.
For conditional tasks, the prompt positions are reset to their clean embeddings at every step §3.1 app. C.2 app. C.4.
Key equations (author notation, sg = stop-gradient):
\Phi^{\mathrm{E.E.}}_\theta(\mathbf{h};T) = \operatorname{softmax}\!\left(\frac{g_\theta(\mathbf{h})}{T}\right) E_\theta eq. 8
\mathbf{c}^{(1)} = \mathbf{z}_t E_\theta, \qquad \mathbf{c}^{(m+1)} = \mathrm{sg}\!\left[\Phi^{\mathrm{E.E.}}_\theta\big(f_\theta(\mathbf{c}^{(m)}, t^{(m)});T\big)\right], \qquad t^{(m)} = t-(m-1)\delta eq. 10
T \sim \tfrac{1}{2}\,\mathcal{U}\!\left[\rho^{-1},1\right] + \tfrac{1}{2}\,\mathcal{U}\!\left[1,\rho\right] eq. 12
Per position, the training loss is the NELBO or the cross-entropy of the clean token under the untempered prediction x̂ §3.2 eq. 11 app. A.1:
\mathcal{L}^{(\ell)}_{\mathrm{CE}} = -\log\langle \hat{\mathbf{x}}^{(\ell)}, \mathbf{x}^{(\ell)}\rangle eq. 11
Training the continuous trajectory
- Multi-pass training. A corrupted sequence z_t eq. 1 is trained with a single pass with probability 1 - p, or with M passes with probability p. Earlier passes receive the state, not tokens; only the last pass receives gradients. Defaults are p = 0.9 and M = 2 §3.2 eq. 10 app. A.3.
- Time schedule. The first pass is at time t, later passes at t - (m-1)δ, with δ = 1/L, which is the spacing used when N = L NFE §3.2. Multi-pass updates sample t from U[(M-1)δ, 1], so the last pass stays at t ≥ 0 §3.2 eq. 10.
- Temperature regularization. One temperature T per sequence, shared across passes and across positions, is drawn with the two-sided law of eq. 12. Training uses the untempered prediction in the loss, so T only changes the states §3.2. The two sides are sampled equally often so that the LM head cannot absorb a fixed temperature into its weights §3.2.
- Architecture. The same pre-normalised DiT as UDMs. An optional learned mode embedding, zero-initialised, tells the denoiser whether its input is token embeddings or a state. It is used only on OpenWebText §3.2 app. A.6. A Duo checkpoint loads without change, so fine-tuning from Duo is possible §3.2 app. C.1.
# Sampling with LAW [alg. 1] [eq. 6] [eq. 8] [eq. 9]
# L positions, d embedding size, V vocabulary, N = NFE
def sample_law(backbone, embedding, lm_head, num_nfe, temperature):
# Draw uniform prior tokens and embed them [eq. 6]
prior = sample_uniform_tokens(num_positions, vocab_size) # (L,)
state = embedding[prior] # (L, d)
# Passes at t_n = n / N for n = N, ..., 2; no tokens are sampled [§3.1]
for n in range(num_nfe, 1, -1):
hidden = backbone(state, n / num_nfe) # (L, d)
# Expected Embedding under the prediction [eq. 7] [eq. 8]
probs = softmax(lm_head(hidden) / temperature, dim=-1) # (L, V)
state = probs @ embedding # (L, d)
# Final pass at t_1, then one raw categorical draw per position [eq. 9]
hidden = backbone(state, 1 / num_nfe) # (L, d)
probs = softmax(lm_head(hidden) / temperature, dim=-1) # (L, V)
return sample_categorical(probs) # (L,)# Training update of LAW [alg. 2] [eq. 1] [eq. 10] [eq. 11] [eq. 12]
# x: clean tokens (L,); p: multi-pass probability; M: passes; rho: range
def training_step(x, p, num_passes, delta, rho):
# Use M passes with probability p, otherwise a single pass [§3.2]
if random() < p:
passes = num_passes
else:
passes = 1
# Sample the time; passes start at t and step back by delta [§3.2]
t = sample_uniform(low=(passes - 1) * delta, high=1)
# Uniform corruption of the clean tokens [eq. 1]
z_t = corrupt(x, t) # (L,)
# One temperature per sequence, shared by all passes [eq. 12]
temperature = draw_temperature(rho)
# The first pass sees the token embeddings of z_t [eq. 10]
state = embedding[z_t] # (L, d)
for m in range(1, passes):
hidden = backbone(state, t - (m - 1) * delta) # (L, d)
probs = softmax(lm_head(hidden) / temperature, dim=-1) # (L, V)
# Stop-gradient between passes [eq. 10] [app. A.5]
state = stop_gradient(probs @ embedding) # (L, d)
# Only the last pass is trained, on the untempered prediction [§3.2]
hidden = backbone(state, t - (passes - 1) * delta)
logits = lm_head(hidden) # (L, V)
# Cross-entropy of the clean token, or the per-token NELBO [eq. 11]
loss = cross_entropy(logits, x)
take_gradient_step(loss)
def draw_temperature(rho):
# Half the mass in [1/rho, 1] (sharpen), half in [1, rho] (flatten) [eq. 12]
if random() < 0.5:
return sample_uniform(low=1 / rho, high=1)
return sample_uniform(low=1, high=rho)Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Optimizer | AdamW, lr 3e-4, (β1, β2) = (0.9, 0.999), weight decay 0 | app. C.1 | 2.5k warmup steps, then constant lr |
| Clipping, EMA, dropout | grad clip 1.0; EMA decay 0.9999 (EMA weights used for eval); dropout 0.1 | app. C.1 | all DiT backbones |
| Passes M, multi-pass probability p | M = 2, p = 0.9 default | §3.2 app. A.5 | main Maze run uses p = 0.75 fig. 3; ablations cover p ∈ {0.1, 0.25, 0.5, 0.75, 0.9} app. C.2 |
| Time shift δ | 1/L | §3.2 | L = 180 Sudoku, 512 TinyGSM, 800 Maze, 1024 OWT |
| Temperature range ρ | 2 (Sudoku, Maze main); 3 (GSM8K, OWT) | tab. 1 fig. 3 §4.3 §4.4 | ρ = 3 or 4 on OWT; ρ = 4 similar to 3 on GSM8K app. D.2 |
| Loss | CE (Sudoku main, OWT); NELBO (GSM8K, Maze main) | §4.1 §4.2 §4.3 §4.4 | Sudoku CE better than NELBO in every configuration app. D.1 |
| Sudoku | 8 blocks, width 512, 8 heads, rotary embeddings; batch 256; 15,385 steps (FLOP-matched); 48k train / 2k val puzzles | app. C.2 tab. 3 | single GPU |
| Maze | 8 blocks, width 256, 8 heads, rotary embeddings; sequence 800; batch 256; 80k steps (main run) | §4.2 fig. 3 app. C.4 | 1k train / 1k eval mazes, 20×20 grids |
| TinyGSM / GSM8K | 12 blocks, width 768, 12 heads; sequence 512; batch 512; 192k steps (FLOP-matched); SmolLM-135M tokenizer | app. C.3 tab. 4 | 11.8M TinyGSM problems |
| OpenWebText | 12 blocks, width 768, 12 heads; context 1024; batch 512; 308k steps (FLOP-matched); GPT-2 tokenizer; mode embedding on | app. C.5 tab. 6 app. A.6 | 16 GPUs in the paper |
| Fine-tuning from Duo | Duo 250k steps, then 50k LAW steps; batch 64, 128, 256 or 512; optimizer and warmup restart; EMA continues from Duo | app. C.3 §4.3 | M = 2, p = 0.9 or M = 4, p = 0.75 |
| Sampling | N = NFE; T = 1 unless stated; raw decoder | §3.1 app. C.1 | NFE swept over 1 to 1024 |
| Gen. PPL protocol | K = 512 samples of L = 1024 tokens; GPT-2 Large; tokens after the first end-of-text ignored | app. C.5 | reported as mean over samples |
Results
Author-reported. Setups differ between tasks; see the notes under each table.
Sudoku, exact match at 180 NFE (hard split is the headline) tab. 1. Baselines are taken from prior work, except LDDMs, which were trained with matched FLOPs tab. 1 app. C.2.
| Model | Easy | Medium | Hard |
|---|---|---|---|
| Autoregressive, sample | 13.9 | 5.1 | 0.6 |
| Autoregressive, greedy | 14.6 | 5.1 | 1.0 |
| MDLM | 92.0 | 77.1 | 30.2 |
| Duo | 96.3 | 84.7 | 58.4 |
| LDDM-M | 98.3 | 95.9 | 87.5 |
| LDDM-U | 99.8 | 97.9 | 91.0 |
| CANDI | 79.3 | 45.9 | 16.7 |
| FLM | 94.2 | 82.7 | 44.5 |
| S-FLM | 94.8 | 85.2 | 45.0 |
| LAW, NELBO, no reg., p = 0.9 | 98.7 | 94.6 | 85.3 |
| LAW, CE, no reg., p = 0.9 | 99.8 | 99.0 | 92.2 |
| LAW, CE, no reg., p = 0.75 | 99.9 | 98.9 | 94.5 |
| LAW, CE, reg. ρ = 2, p = 0.9 | 99.8 | 99.4 | 96.5 |
Maze, valid-solution accuracy (the paper's own metric; exact match is reported only in an appendix) fig. 3 app. C.4 app. D.5:
- LAW (NELBO, M = 2, p = 0.75, ρ = 2) solves 90.9% of evaluation mazes with 4 NFE, more than any baseline at any NFE except LDDM-M §4.2.
- Up to 64 NFE, LAW is the most accurate model in the comparison. At 512 NFE and above, the curves converge near 92%, with LDDM-M slightly higher §4.2. LDDM-U levels off under 89%, and the continuous models stay under 82% §4.2.
- Greedy ARM with KV cache: 9.4% fig. 3.
- Note: the Maze set is a 20×20 regeneration of HRM's Maze-Hard with a 71-cell path threshold, not the original 30×30 benchmark app. C.4.
GSM8K (TinyGSM training, test accuracy) fig. 1 §4.3 tab. 10 §6:
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| From scratch, T = 1, 1024 NFE | accuracy | LDDM-U 34.5% | 42.2% | §4.3 fig. 1 |
| From scratch, T = 0.1, 8 NFE | accuracy | LDDM-U 12.4% | 26.4% | §4.3 tab. 10 |
| From scratch, T = 0.1, 4 to 512 NFE | accuracy | LDDMs match it only at 1024 NFE | most accurate model in this range | §4.3 |
| Duo fine-tuned 50k steps, T = 1, 1024 NFE | accuracy | Duo 17.2% | 48.1% | §4.3 |
| Duo fine-tuned 50k steps, T = 0.1, 1024 NFE | accuracy | Duo 36.0% | 54.0% | §6 |
| Duo fine-tuned 15k steps, T = 1, 32 NFE | accuracy | ARM (512 NFE, T = 1) 53.9% | 56.2% | §4.3 tab. 10 |
| Fine-tuned 15k steps, T = 1, 16 NFE | accuracy | AlphaDLM (k = 8, greedy) 49.2% | 50.1% | §4.3 tab. 10 |
The paper's few-step comparison tab. 10 also lists MDLM, Duo, IDLM, FMLM+, DBTM, AlphaDLM and SDM at 1 to 64 NFE. The authors say AlphaDLM is the most accurate method up to 8 NFE and beats LAW trained from scratch. LAW fine-tuned for 15k steps has higher accuracy than AlphaDLM starting at 16 sampling steps, at T = 1 §4.3.
Ablations on GSM8K (all at matched training FLOPs) §4.3 app. D.2:
- State map: with the hidden-representation state, LAW solves fewer than 1.5% (no regularization). With the Expected Embedding it solves 35.8% §4.3.
- Temperature regularization: at T = 0.1 with 1024 NFE it doubles accuracy, from 24.8% to 48.8% §4.3.
- Stop-gradient between passes: removing it drops accuracy from 42.2% to 20.2% at matched FLOPs and at equal steps §4.3 app. D.2.
OpenWebText (author-reported; Gen. PPL against unigram entropy) fig. 5 §4.4:
- At 64 NFE, LAW's Gen. PPL at equal entropy is lower than every diffusion or flow comparator, and it comes close to the ARM fig. 5.
- At 1024 NFE, its entropy over the swept temperatures reaches only 5.1–5.2, against 5.8 at 64 NFE. Its samples are less diverse §4.4.
- Example samples: Gen. PPL 18.31 at unigram entropy 5.60 (64 NFE); Gen. PPL 22.09 at entropy 5.69 (1024 NFE) app. D.9.
Paper vs code
Repository github.com/ilievanadezhda/law at commit
3055c55d42ef9ad2964fc500a95f6d416648bb16. The repository owner matches a
co-author (Nadezhda Ilieva; the README links her homepage), and the paper links the
repository. Searches were not needed. There is no LICENSE file, so the code
cannot be quoted or copied; it is cited by path and line only. Run-script paths are
shortened after their directory, e.g. scripts/train/owt/law/…rng-3-mode-emb.sh,
where … stands for the rest of the run name in the repository README.
Checked and consistent with the paper: δ = 1/L in every reported script …mode-emb.sh …rng-2.sh …rng-2.sh; the temperature draw is per sequence and shared across passes algo.py; the loss uses the untempered prediction and only the last pass algo.py …mode-emb.sh; the sampling temperature applies to both the state map and the decoder samplers.py. Step counts match the paper's tables: Sudoku 15,385 …rng-2.sh, Maze 80,000 …rng-2.sh, OWT 308,000 …mode-emb.sh.
Differences:
- Stop-gradient covers only the probabilities. The paper's Eq. eq. 10 puts a stop-gradient on the whole next state. The code detaches the probabilities but multiplies them by the live embedding matrix, so the matrix still gets gradient through the carry from the last pass dit_loop.py. Implementing sg literally as written in eq. 10 will differ from the release. This is from reading the code; I did not run it.
- The config defaults are not the paper's method.
configs/algo/law.yamlaverages the loss over all passes and samples one pass only, with temperature regularization off law.yaml. Every reported script overrides these to last-only loss, p = 0.1 or 0.25 vs 0.9 or 0.75, and ρ > 1. A run that uses the config defaults is single-pass and does not match the paper. - Gaussian and rotational carry noise. The code implements extra carry
regularization (Gaussian noise and a rotation toward random directions) with
gaussian_stdandrotation_fractiondit_loop.py law.yaml. The paper does not describe it, and no reported script sets it (both default to 0). - Sampling time grid. The paper's grid is t_n = n/N §2.1. The code uses
linspace(1, eps, N+1)witheps = 1e-5by default samplers.py samplers.py. The last pass lands at about 1/N + 1e-5; the effect is probably negligible. - Maze step count. The main Maze run uses 80k steps fig. 3. The FLOP table's default LAW row for Maze is 76,923 steps at p = 0.9 app. B.2 tab. 5. Use 80k for the reported run.
Pitfalls
- Expected Embedding, not the hidden state. The hidden-state carry collapses to below 1.5% on GSM8K; the Expected Embedding is the only state that works here §4.3. Carrying the layer-normalised hidden state, as LDDM does, is not a drop-in variant.
- Temperature regularization is needed. Without it, LAW reaches only low entropies on OpenWebText, and its Gen. PPL is worse than LDDM-U's §4.4 app. D.7. On GSM8K, removing it halves accuracy at low temperature §4.3.
- Stop-gradient is needed. Backpropagating through all M passes drops GSM8K accuracy from 42.2% to 20.2% at matched FLOPs §4.3 app. A.5 app. D.2. Backpropagation costs more per step, so the step budget also shrinks.
- FLOP matching. A LAW update with M = 2, p = 0.9 costs about 3.9 F against 3 F for a baseline update, so LAW trains for about 77% of the baseline step count app. B.1. Compare at the same FLOPs, not the same steps app. B.
- Fine-tuning from Duo with M = 2 tops out early. The best GSM8K checkpoint falls at roughly 15k–20k fine-tuning steps, and later checkpoints score lower; with M = 4 the later checkpoints do not drop §4.3 app. D.3. Report the fine-tuning step count next to every fine-tuned number.
- The 15k-step numbers in §6 do not state the NFE. The conclusion gives 58.7% (T = 1) and 53.1% (T = 0.1) after 15k steps §6. The 1024-NFE setting is implied by the sentence before, but not restated.
- Decoder choice. The paper defines an argmax decoder eq. 9 but states that LAW always uses the raw decoder §3.1 app. C.1. Greedy decoding would change the diversity numbers, so do not assume it.
- Maze metric. Exact match against one BFS path underestimates accuracy because several shortest paths exist. The paper reports valid-solution accuracy and gives exact match only in an appendix app. C.4 app. D.5.
- Maze is not HRM's Maze-Hard. The authors regenerated 20×20 grids with a 71-cell threshold; the original uses 30×30 with 110 cells app. C.4. Do not compare the numbers to HRM's.
- Prompts and temperature. Prompt positions are reset to clean embeddings every step during conditional generation §3.1 app. C.4. Leaving them as states drifts from the paper's procedure.
- Diversity limit on OpenWebText. At 1024 NFE the entropy range is narrower than for the baselines §6. Gen. PPL alone is not a fair comparison §4.4.
- Scale. All experiments are at small scale, and the authors say larger scale is untested §6.
Lineage
- Builds on: Loopholing Discrete Diffusion (LDDM; Jo et al., 2025; arXiv:2510.19304), which provides the multi-pass training scheme and the self-conditioning baseline LAW replaces §2.2 app. A.4. Uniform diffusion follows Duo §2.1 app. C.1.
- Related self-conditioning and recurrent-state methods: SUNDAE, Relay, PUMBA, FReDA and Soft-Masked Diffusion, which the paper contrasts with its own pure state propagation §5.
- Compared against: ARM, MDLM, Duo, LDDM-M, LDDM-U, CANDI, FLM, S-FLM, AlphaDLM, SDM, IDLM, FMLM+, DBTM §4 tab. 1 tab. 10.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “Loopholing All the Way: Language Modeling with pure Self-Conditioning” by Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Caglar Gulcehre, Pascal Frossard (arXiv:2610.30068), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ilievanadezhda/law at commit 3055c55d42ef (none (no LICENSE file in repository)). Any quoted code is shown under that license.