Discrete diffusion

Overview

Standard diffusion adds Gaussian noise to continuous values. Discrete diffusion corrupts categorical variables instead: a token becomes a mask, or a node or edge becomes another class, according to a Markov chain over the classes 2406.07524 §3.1 2209.14734 §3. Generation starts from a fully corrupted state and runs the chain backwards.

The invariant all variants share: the network predicts a distribution over clean classes at every position, and the reverse step combines that prediction with the forward kernel 2406.07524 §3.2.1 2209.14734 §2 2209.14734 eq. 1 2303.08137 eq. 4. In the masked and graph variants the forward kernel has a closed form, so a noisy state can be drawn from the clean one in one step 2406.07524 §3.2.1 2209.14734 §3.1.

The variants differ in what the noise does to a class, which fixes the sampler. Masking makes known positions fixed and unmasked positions predictable 2406.07524 §3.2.3. Resampling from class frequencies lets every position change at every step 2209.14734 §4.1. Layout diffusion mixes both: it masks a class or moves it to a uniformly chosen other class, so the reverse model can revise a value it has already filled 2303.08137 §3.1 2303.08137 eq. 3.

The wiki has seven compiled cards on this topic. Continuous (embedding-space) text diffusion is a different family, covered by continuous text diffusion. One continuous flow, S-FLM, is benchmarked against the discrete models here, so it appears below as a comparison point.

Variants

Masked diffusion for text

Each token is independently replaced by a mask token with a probability set by a schedule. Once masked, a token stays masked 2406.07524 §3.2.1 2406.07524 eq. 4. The denoiser is an encoder-only transformer. Its output is forced so that the mask is never produced and unmasked positions are copied through unchanged 2406.07524 §3.2.3 2406.07524 eq. 7.

Training minimises a weighted average of masked-token cross-entropy losses. The weights come from the continuous-time limit of the ELBO 2406.07524 §3.4 2406.07524 eq. 10 2406.07524 §3.5. Sampling starts from a row of masks and unmasks over a fixed number of steps 2406.07524 §4.1. To write text longer than the training window, the model generates block by block with the earlier output as a clean prefix 2406.07524 §4.2. Card: Simple and Effective Masked Diffusion Language Models.

Author-reported results: a perplexity bound of 23.00 on LM1B after 327B tokens, against 20.86 for a retrained autoregressive model 2406.07524 tab. 1. On OpenWebText the bound is 23.21, against 17.54 for autoregressive 2406.07524 tab. 2. Diffusion perplexities are upper bounds, so the gap is not an exact comparison 2406.07524 tab. 1. The block sampler reaches a generative perplexity of 27.18 in 89.3 seconds per sequence, against 35.43 in 2473.9 seconds for SSD-LM 2406.07524 tab. 5.

Categorical diffusion for graphs

Each node and each edge carries a one-hot class, and the absence of an edge is one of the edge classes. Noise is a per-node and per-edge Markov chain, and the chain moves each class toward the empirical class frequencies of the training set rather than toward a uniform distribution 2209.14734 §3 2209.14734 §4.1 2209.14734 eq. 6. The denoiser is a graph transformer trained with cross-entropy on nodes and edges. It is permutation equivariant, so training does not need a node ordering 2209.14734 §3.2 2209.14734 §3.3. Card: DiGress: Discrete Denoising diffusion for graph generation.

Author-reported: 74% and 75% valid-unique-novel graphs on the SBM and planar benchmarks 2209.14734 tab. 1. Validity is 99.0% on QM9 2209.14734 tab. 2 and 85.7% on MOSES, with FCD 1.19 2209.14734 tab. 3. The marginal noise does not win on every metric: with explicit hydrogens, validity rises from 89.8% to 92.3% but molecule stability falls from 70.5% to 66.8% 2209.14734 tab. 6. The paper reports conditional generation with a regressor guidance rule, which lowers the dipole-moment error from 1.71 to 0.81 on QM9 2209.14734 §5 2209.14734 §7.3. The guidance scale is not stated 2209.14734 alg. 3, and the released code does not include a guidance step in its sampler, so these numbers cannot be checked from the code alone.

Per-step watermark tilting for diffusion LMs

This variant changes sampling, not training. A red-green watermark needs the preceding tokens to pick its green list, and a diffusion model often fills a later position before its context is known. At each denoising step the method adds to the logits a tilt proportional to the gradient of an expected green-count energy, taken over the distribution of the still-undecoded context hashes 2509.24368 §3.1 2509.24368 §3.2 [arxiv:2509.24368 alg. 2]. The default hash is a sum over context offsets, with top-k 50 2509.24368 §4.1. Detection is the unchanged red-green binomial test, after removing repeated (context, token) pairs 2509.24368 §3.2. Card: Watermarking Diffusion Language Models.

Author-reported on LLaDA-8B-Instruct, at a 1% false-positive rate: the naive adaptation detects 63% of texts, and the tilt detects 99% at context {-1} and strength 4. Average benchmark accuracy falls from 59.4 to 56.0 2509.24368 tab. 1. Detection is weak on short texts: at about 50 tokens the tilt matches the naive baseline at about 350 tokens 2509.24368 §4.1. The strength scale in the paper and the released configs do not agree, so calibrate it on your own texts 2509.24368 §4.1 2509.24368 app. A.1.

Parallel block drafter for speculative decoding

A drafter for speculative decoding fills a block of masked slots in one forward pass, instead of drafting one token per pass. The draft is a small block diffusion transformer. The target model's hidden features enter every draft layer as extra keys and values, not as input embeddings. Queries come only from the draft tokens 2602.06036 §4.1 2602.06036 app. A.3. The target verifies the drafted block, so the output matches the target 2602.06036 §4.1 2602.06036 §3.1. Training samples anchor positions as block starts and weights earlier block positions more 2602.06036 §4.2 2602.06036 eq. 4. Card: DFlash: Block Diffusion for Flash Speculative Decoding.

Author-reported on Qwen3-4B, greedy decoding, averaged over seven tasks: 4.91× speedup with an acceptance length of 6.54 tokens, against 2.08× for EAGLE-3 with a 60-node draft tree 2602.06036 tab. 1. The 6.1× headline is a single-task peak on MATH-500 2602.06036 tab. 1. Speedup falls as concurrency rises: from 4.8× at one request to 2.9× at 32 on Qwen3-4B MATH-500 2602.06036 tab. 3. Feeding target features as keys and values beats feeding them as input, in block diffusion drafting 2602.06036 tab. 9.

Weighted policy optimisation for masked diffusion LLMs

This variant changes training, not sampling. The usual policy-gradient objective needs the likelihood of each sampled answer under the current and old model, and a masked diffusion LLM has no exact value for that. The wd1 method uses one approximation per completion, and it drops the importance ratio and the reference-model term 2507.08838 §1 [arxiv:2507.08838 §2.3] 2507.08838 §3.2. For a group of sampled answers, each answer's group-relative advantage sets two softmax weights. A positive term raises the likelihood of high-advantage answers, and a negative term lowers the likelihood of low-advantage ones 2507.08838 §3.2 [arxiv:2507.08838 eq. 8] 2507.08838 eq. 9. The likelihood estimate masks the prompt at random and masks all completion tokens, then sums the log-probabilities of the completion from one forward pass 2507.08838 §2.3 2507.08838 app. B.4. Card: wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models.

Author-reported on LLaDA-8B-Instruct with LoRA, at 256 generated tokens: Sudoku 76.4% against 17.6% for the authors' reproduction of d1, and Countdown 51.2% against 25.8% 2507.08838 tab. 1. At 512 tokens, GSM8K is 82.3% against 82.0%, and MATH500 is 39.0% against 38.0% 2507.08838 tab. 1. At 256 tokens, MATH500 ties at 34.4% 2507.08838 tab. 1. The d1 numbers the d1 authors report differ from the reproduction, and on Sudoku they are lower 2507.08838 tab. 7. The paper reports one RL step at 81.16 s against 103.5 s for d1 on four A100 GPUs, about 22% less time per step 2507.08838 tab. 2. The main results use a single seed, and no spread is reported 2507.08838 tab. 6. The extended wd1++ variant, which also uses intermediate denoising steps, reaches 84.5% on GSM8K and 44.2% on MATH500 with full fine-tuning 2507.08838 tab. 3 [arxiv:2507.08838 §3.3]. The paper's wd1++ code was not found in the released repository, so those numbers could not be checked against code; the card has the details.

Per-attribute mask-and-replace diffusion for layouts

Each element of a layout is a category and four quantized box values, written as a flat sequence of tokens. Elements are shuffled at random and padded to a fixed length of 25 elements 2303.08137 §3.2 2303.08137 §4.1. The forward step keeps a token with probability α_t, moves it to a uniformly chosen other value with probability β_t, and masks it with probability γ_t. Each attribute has its own matrix, so a category never turns into a coordinate bin 2303.08137 §3.1 2303.08137 §3.2 2303.08137 eq. 3. The denoiser is a bidirectional Transformer encoder. Its position information is split into an element index and an attribute index 2303.08137 §3.2. Training uses the variational bound plus a weighted denoising term with weight 0.1 2303.08137 §3.1 2303.08137 §4.4 2303.08137 eq. 5. Card: LayoutDM: Discrete Diffusion Model for Controllable Layout Generation.

Conditioning needs no retraining. Known fields are masked back in after every reverse step, which is a hard constraint. Soft priors, such as refinement targets or pairwise relations, shift the reverse log-probabilities 2303.08137 §3.3 2303.08137 eq. 6. For relations, the paper applies the adjustment three times per diffusion step 2303.08137 §4.5.

Author-reported on Rico, averaged over three trials: unconditional FID 6.65, which is second only to the fixed-order LayoutTrans-fixed at 6.47 2303.08137 tab. 2. Replacing the learned geometry bins with uniform or percentile bins raises FID to 7.58 and 9.79 2303.08137 tab. 4. For refinement, Rico FID is 2.77 against 3.23 for the task-specific RUITE, but on PubLayNet the FID is 6.75 against 6.39 2303.08137 tab. 3. On relation constraints the paper reports LayoutDM behind CLG-LO on both datasets, with an average runtime of 0.5 s against 4.0 s 2303.08137 §4.5 2303.08137 fig. 5. On overlap and alignment for variable-length tasks, BART and LayoutTrans are moderately better 2303.08137 app. B. FID values across papers are not comparable, because the datasets have no official splits and FID networks differ 2303.08137 §4.4.

Continuous flow over unit-norm embeddings, as a comparison point

This variant is not discrete. S-FLM moves each token's embedding on the unit sphere and never builds one-hot vectors. It is included because the paper benchmarks it directly against discrete masked diffusion on the same tasks 2605.11125 §4.1 2605.11125 §4.2. Its denoiser predicts a distribution over the vocabulary, as the discrete variants do, and it is trained with unweighted cross-entropy on the clean token 2605.11125 §3 2605.11125 eq. 14. Sampling rotates each position toward every embedding, weighted by that distribution. The top-1 option keeps only the most likely token 2605.11125 eq. 15 2605.11125 §3.1. The noise range is truncated, and its density is set adaptively from the training loss 2605.11125 §3.2. Card: Language Modeling with Hyperspherical Flows.

Author-reported on Sudoku exact match, the truncated and adaptive S-FLM reaches 94.8%, 85.2% and 45.0% on easy, medium and hard puzzles. MDLM reaches 92.0%, 77.1% and 30.2% 2605.11125 tab. 1. The plain S-FLM row, without the truncation, is below MDLM at all three levels: 81.5%, 50.6% and 14.0% 2605.11125 tab. 1. On GSM8K at T = 1 with 1024 sampling steps, top-1 velocity reaches 18.0%, level with the 18.0% that MDLM reaches in the same table. The plain linear-schedule model reaches 1.2% 2605.11125 tab. 2. At T = 0.1 the paper reports MDLM at about 33% and S-FLM with top-1 at about 18% 2605.11125 §4.2 2605.11125 fig. 3.

The numbers are single-run. The authors trained one model per configuration 2605.11125 §7. The released sampling scripts default to 32 steps, not the 1024 in the table, and the truncation side is read two ways across the paper's text, its ablation table and its scripts. Check both before you try to reproduce the GSM8K numbers. The S-FLM card lists the details 2605.11125 tab. 2 2605.11125 tab. 7 2605.11125 §3.2.

When to use which

Implementation sketch

The shared core is a reverse loop over a corrupted state. Each variant changes the prior, the model's output and the reverse update.

# Common reverse loop for discrete diffusion
# State x: one discrete class per position (tokens, nodes, edges)
# Output: a sample of clean classes

def reverse_sample(model, corrupted, steps):
    # Start from the fully corrupted state: all masks, or class marginals
    # [arxiv:2406.07524 §4.1] [arxiv:2209.14734 alg. 2]
    x = corrupted
    for t in steps:                              # from T down toward 0
        # Distribution over clean classes at every position
        # [arxiv:2406.07524 §3.2.3] [arxiv:2209.14734 §3.2]
        p_clean = model(x, t)                    # (positions, classes)
        # Combine the prediction with the forward kernel
        # [arxiv:2209.14734 eq. 1] [arxiv:2209.14734 eq. 5]
        x = sample_reverse_step(x, p_clean, t)   # (positions,)
    return x

Masked variants keep unmasked positions fixed, so their reverse step only fills masks 2406.07524 §3.2.3. The categorical graph variant resamples every position from the posterior 2209.14734 eq. 4. The layout variant re-imposes known fields after every step and adds weak priors to the log-probabilities 2303.08137 §3.3 2303.08137 eq. 6. The watermark variant adds a tilt to p_clean before sampling 2509.24368 alg. 2. The block drafter runs the same loop once per round, with the target's features as extra context 2602.06036 §4.1. The weighted policy optimisation variant leaves the sampler alone and changes the training loss: it raises or lowers the log-likelihood of sampled completions by their softmax weights 2507.08838 eq. 8.

Contradictions

Across papers, no two cards conflict on a shared metric. The seven cards measure different things, on different tasks. Within one paper, the graph variant's marginal-noise ablation is mixed: validity rises and molecule stability falls 2209.14734 tab. 6. The S-FLM text says S-FLM and FLM outperform AR and MDLM at every Sudoku difficulty. That holds only for the truncated S-FLM rows. The plain S-FLM row is below MDLM at all three levels 2605.11125 §4.1 2605.11125 tab. 1. Treat the text's claim as applying to the truncated variants. The wd1 paper's text and table disagree on its Sudoku gain: the text gives 43% 2507.08838 §5.1, the introduction gives +58.8% 2507.08838 §1, and the table's 76.4 against 17.6 is a gain of 58.8 points, about 4.3 times 2507.08838 tab. 1. In the layout paper, the refinement table's "LayoutDM w/o logit adjustment" row has the same values as LayoutDM's category-to-size-and-position row in the conditional table 2303.08137 tab. 3 2303.08137 tab. 1. The two are different tasks, and the paper does not explain the match. Treat the ablation numbers from that table with care.

Papers

• DiGress: Discrete Denoising diffusion for graph generation: categorical diffusion over graph nodes and edges, with a graph transformer denoiser.
• Simple and Effective Masked Diffusion Language Models: masked diffusion for text, with a weighted masked-token objective and block sampling.
• LayoutDM: Discrete Diffusion Model for Controllable Layout Generation: per-attribute mask-and-replace diffusion over layout tokens, with conditions applied at inference.
• Watermarking Diffusion Language Models: red-green watermark applied as a per-step logit tilt for diffusion LMs.
• DFlash: Block Diffusion for Flash Speculative Decoding: a parallel block drafter that fills masked slots in one pass, with target features injected as keys and values.
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models: ratio-free reinforcement learning for masked diffusion LLMs, with a positive and a negative softmax-weighted likelihood term.
• Language Modeling with Hyperspherical Flows: continuous flow over unit-norm token embeddings, benchmarked against masked diffusion on Sudoku and GSM8K.