---
kind: concept
slug: discrete-diffusion
title: Discrete diffusion
summary: >-
  Diffusion over discrete variables such as tokens, graph classes, layout attributes or draft tokens. Corruption masks or replaces classes; a network predicts clean classes and sampling runs back to clean data. Covers masked text, graph, layout, watermarking, block drafting and RL variants, plus a continuous comparison flow. Seven cards.
tags: [diffusion, non-autoregressive, discrete, sampling]
papers: ["2209.14734", "2406.07524", "2303.08137", "2509.24368", "2507.08838", "2602.06036", "2605.11125"]
related: [diffusion-models, continuous-text-diffusion, masked-language-modeling, graph-neural-network, equivariance-constraint, speculative-decoding, ai-generated-text-detection, llm-serving, policy-gradient, reinforcement-learning-verifiable-rewards, constrained-decoding, flow-matching, noise-schedule]
status: draft
updated_at: 2026-10-31
---

## Overview

Standard diffusion adds Gaussian noise to continuous values. Discrete diffusion
corrupts categorical variables instead: a token becomes a mask, or a node or
edge becomes another class, according to a Markov chain over the classes
[arxiv:2406.07524 §3.1] [arxiv:2209.14734 §3]. Generation starts from a fully
corrupted state and runs the chain backwards.

The invariant all variants share: the network predicts a distribution over
clean classes at every position, and the reverse step combines that prediction
with the forward kernel [arxiv:2406.07524 §3.2.1] [arxiv:2209.14734 §2]
[arxiv:2209.14734 eq. 1] [arxiv:2303.08137 eq. 4]. In the masked and graph
variants the forward kernel has a closed form, so a noisy state can be drawn
from the clean one in one step [arxiv:2406.07524 §3.2.1] [arxiv:2209.14734 §3.1].

The variants differ in what the noise does to a class, which fixes the
sampler. Masking makes known positions fixed and unmasked positions
predictable [arxiv:2406.07524 §3.2.3]. Resampling from class frequencies lets
every position change at every step [arxiv:2209.14734 §4.1]. Layout diffusion
mixes both: it masks a class or moves it to a uniformly chosen other class, so
the reverse model can revise a value it has already filled
[arxiv:2303.08137 §3.1] [arxiv:2303.08137 eq. 3].

The wiki has seven compiled cards on this topic. Continuous (embedding-space)
text diffusion is a different family, covered by
[continuous text diffusion](/concepts/continuous-text-diffusion). One
continuous flow, S-FLM, is benchmarked against the discrete models here, so
it appears below as a comparison point.

## Variants

### Masked diffusion for text

Each token is independently replaced by a mask token with a probability set
by a schedule. Once masked, a token stays masked
[arxiv:2406.07524 §3.2.1] [arxiv:2406.07524 eq. 4]. The denoiser is an
encoder-only transformer. Its output is forced so that the mask is never
produced and unmasked positions are copied through unchanged
[arxiv:2406.07524 §3.2.3] [arxiv:2406.07524 eq. 7].

Training minimises a weighted average of masked-token cross-entropy losses. The
weights come from the continuous-time limit of the ELBO
[arxiv:2406.07524 §3.4] [arxiv:2406.07524 eq. 10] [arxiv:2406.07524 §3.5].
Sampling starts from a row of masks and unmasks over a fixed number of steps
[arxiv:2406.07524 §4.1]. To write text longer than the training window, the
model generates block by block with the earlier output as a clean prefix
[arxiv:2406.07524 §4.2]. Card: [Simple and Effective Masked Diffusion Language
Models](/papers/2406.07524).

Author-reported results: a perplexity bound of 23.00 on LM1B after 327B tokens,
against 20.86 for a retrained autoregressive model [arxiv:2406.07524 tab. 1].
On OpenWebText the bound is 23.21, against 17.54 for autoregressive
[arxiv:2406.07524 tab. 2]. Diffusion perplexities are upper bounds, so the
gap is not an exact comparison [arxiv:2406.07524 tab. 1]. The block sampler
reaches a generative perplexity of 27.18 in 89.3 seconds per sequence, against
35.43 in 2473.9 seconds for SSD-LM [arxiv:2406.07524 tab. 5].

### Categorical diffusion for graphs

Each node and each edge carries a one-hot class, and the absence of an edge is
one of the edge classes. Noise is a per-node and per-edge Markov chain, and
the chain moves each class toward the empirical class frequencies of the
training set rather than toward a uniform distribution
[arxiv:2209.14734 §3] [arxiv:2209.14734 §4.1] [arxiv:2209.14734 eq. 6]. The
denoiser is a graph transformer trained with cross-entropy on nodes and edges.
It is permutation equivariant, so training does not need a node ordering
[arxiv:2209.14734 §3.2] [arxiv:2209.14734 §3.3]. Card: [DiGress: Discrete
Denoising diffusion for graph generation](/papers/2209.14734).

Author-reported: 74% and 75% valid-unique-novel graphs on the SBM and planar
benchmarks [arxiv:2209.14734 tab. 1]. Validity is 99.0% on QM9
[arxiv:2209.14734 tab. 2] and 85.7% on MOSES, with FCD 1.19
[arxiv:2209.14734 tab. 3]. The marginal noise does not win on every metric:
with explicit hydrogens, validity rises from 89.8% to 92.3% but molecule
stability falls from 70.5% to 66.8% [arxiv:2209.14734 tab. 6]. The paper
reports conditional generation with a regressor guidance rule, which lowers
the dipole-moment error from 1.71 to 0.81 on QM9 [arxiv:2209.14734 §5]
[arxiv:2209.14734 §7.3]. The guidance scale is not stated
[arxiv:2209.14734 alg. 3], and the released code does not include a guidance
step in its sampler, so these numbers cannot be checked from the code alone.

### Per-step watermark tilting for diffusion LMs

This variant changes sampling, not training. A red-green watermark needs the
preceding tokens to pick its green list, and a diffusion model often fills a
later position before its context is known. At each denoising step the method
adds to the logits a tilt proportional to the gradient of an expected
green-count energy, taken over the distribution of the still-undecoded context
hashes [arxiv:2509.24368 §3.1] [arxiv:2509.24368 §3.2] [arxiv:2509.24368
alg. 2]. The default hash is a sum over context offsets, with top-k 50
[arxiv:2509.24368 §4.1]. Detection is the unchanged red-green binomial test,
after removing repeated (context, token) pairs [arxiv:2509.24368 §3.2]. Card:
[Watermarking Diffusion Language Models](/papers/2509.24368).

Author-reported on LLaDA-8B-Instruct, at a 1% false-positive rate: the naive
adaptation detects 63% of texts, and the tilt detects 99% at context {-1} and
strength 4. Average benchmark accuracy falls from 59.4 to 56.0
[arxiv:2509.24368 tab. 1]. Detection is weak on short texts: at about 50
tokens the tilt matches the naive baseline at about 350 tokens
[arxiv:2509.24368 §4.1]. The strength scale in the paper and the released
configs do not agree, so calibrate it on your own texts
[arxiv:2509.24368 §4.1] [arxiv:2509.24368 app. A.1].

### Parallel block drafter for speculative decoding

A drafter for speculative decoding fills a block of masked slots in one forward
pass, instead of drafting one token per pass. The draft is a small block
diffusion transformer. The target model's hidden features enter every draft
layer as extra keys and values, not as input embeddings. Queries come only
from the draft tokens [arxiv:2602.06036 §4.1] [arxiv:2602.06036 app. A.3].
The target verifies the drafted block, so the output matches the target
[arxiv:2602.06036 §4.1] [arxiv:2602.06036 §3.1]. Training samples anchor
positions as block starts and weights earlier block positions more
[arxiv:2602.06036 §4.2] [arxiv:2602.06036 eq. 4]. Card: [DFlash: Block
Diffusion for Flash Speculative Decoding](/papers/2602.06036).

Author-reported on Qwen3-4B, greedy decoding, averaged over seven tasks: 4.91×
speedup with an acceptance length of 6.54 tokens, against 2.08× for EAGLE-3
with a 60-node draft tree [arxiv:2602.06036 tab. 1]. The 6.1× headline is a
single-task peak on MATH-500 [arxiv:2602.06036 tab. 1]. Speedup falls as
concurrency rises: from 4.8× at one request to 2.9× at 32 on Qwen3-4B
MATH-500 [arxiv:2602.06036 tab. 3]. Feeding target features as keys and values
beats feeding them as input, in block diffusion drafting
[arxiv:2602.06036 tab. 9].

### Weighted policy optimisation for masked diffusion LLMs

This variant changes training, not sampling. The usual policy-gradient
objective needs the likelihood of each sampled answer under the current and
old model, and a masked diffusion LLM has no exact value for that. The wd1
method uses one approximation per completion, and it drops the importance
ratio and the reference-model term [arxiv:2507.08838 §1] [arxiv:2507.08838
§2.3] [arxiv:2507.08838 §3.2]. For a group of sampled answers, each answer's
group-relative advantage sets two softmax weights. A positive term raises the
likelihood of high-advantage answers, and a negative term lowers the
likelihood of low-advantage ones [arxiv:2507.08838 §3.2] [arxiv:2507.08838
eq. 8] [arxiv:2507.08838 eq. 9]. The likelihood estimate masks the prompt at
random and masks all completion tokens, then sums the log-probabilities of
the completion from one forward pass [arxiv:2507.08838 §2.3]
[arxiv:2507.08838 app. B.4]. Card: [wd1: Weighted Policy Optimization for
Reasoning in Diffusion Language Models](/papers/2507.08838).

Author-reported on LLaDA-8B-Instruct with LoRA, at 256 generated tokens:
Sudoku 76.4% against 17.6% for the authors' reproduction of d1, and Countdown
51.2% against 25.8% [arxiv:2507.08838 tab. 1]. At 512 tokens, GSM8K is 82.3%
against 82.0%, and MATH500 is 39.0% against 38.0% [arxiv:2507.08838 tab. 1].
At 256 tokens, MATH500 ties at 34.4% [arxiv:2507.08838 tab. 1]. The d1 numbers
the d1 authors report differ from the reproduction, and on Sudoku they are
lower [arxiv:2507.08838 tab. 7]. The paper reports one RL step at 81.16 s
against 103.5 s for d1 on four A100 GPUs, about 22% less time per step
[arxiv:2507.08838 tab. 2]. The main results use a single seed, and no
spread is reported [arxiv:2507.08838 tab. 6]. The extended wd1++ variant,
which also uses intermediate denoising steps, reaches 84.5% on GSM8K and 44.2%
on MATH500 with full fine-tuning [arxiv:2507.08838 tab. 3] [arxiv:2507.08838
§3.3]. The paper's wd1++ code was not found in the released repository, so
those numbers could not be checked against code; the card has the details.

### Per-attribute mask-and-replace diffusion for layouts

Each element of a layout is a category and four quantized box values, written
as a flat sequence of tokens. Elements are shuffled at random and padded to a
fixed length of 25 elements [arxiv:2303.08137 §3.2] [arxiv:2303.08137 §4.1].
The forward step keeps a token with probability α_t, moves it to a uniformly
chosen other value with probability β_t, and masks it with probability γ_t.
Each attribute has its own matrix, so a category never turns into a coordinate
bin [arxiv:2303.08137 §3.1] [arxiv:2303.08137 §3.2] [arxiv:2303.08137 eq. 3].
The denoiser is a bidirectional Transformer encoder. Its position information
is split into an element index and an attribute index
[arxiv:2303.08137 §3.2]. Training uses the variational bound plus a weighted
denoising term with weight 0.1 [arxiv:2303.08137 §3.1] [arxiv:2303.08137 §4.4]
[arxiv:2303.08137 eq. 5]. Card: [LayoutDM: Discrete Diffusion Model for
Controllable Layout Generation](/papers/2303.08137).

Conditioning needs no retraining. Known fields are masked back in after every
reverse step, which is a hard constraint. Soft priors, such as refinement
targets or pairwise relations, shift the reverse log-probabilities
[arxiv:2303.08137 §3.3] [arxiv:2303.08137 eq. 6]. For relations, the paper
applies the adjustment three times per diffusion step [arxiv:2303.08137 §4.5].

Author-reported on Rico, averaged over three trials: unconditional FID 6.65,
which is second only to the fixed-order LayoutTrans-fixed at 6.47
[arxiv:2303.08137 tab. 2].
Replacing the learned geometry bins with uniform or percentile bins raises FID
to 7.58 and 9.79 [arxiv:2303.08137 tab. 4]. For refinement, Rico FID is 2.77
against 3.23 for the task-specific RUITE, but on PubLayNet the FID is 6.75
against 6.39 [arxiv:2303.08137 tab. 3]. On relation constraints the paper
reports LayoutDM behind CLG-LO on both datasets, with an average runtime of
0.5 s against 4.0 s [arxiv:2303.08137 §4.5] [arxiv:2303.08137 fig. 5]. On
overlap and alignment for variable-length tasks, BART and LayoutTrans are
moderately better [arxiv:2303.08137 app. B]. FID values across papers are not
comparable, because the datasets have no official splits and FID networks
differ [arxiv:2303.08137 §4.4].

### Continuous flow over unit-norm embeddings, as a comparison point

This variant is not discrete. S-FLM moves each token's embedding on the unit
sphere and never builds one-hot vectors. It is included because the paper
benchmarks it directly against discrete masked diffusion on the same tasks
[arxiv:2605.11125 §4.1] [arxiv:2605.11125 §4.2]. Its denoiser predicts a
distribution over the vocabulary, as the discrete variants do, and it is
trained with unweighted cross-entropy on the clean token
[arxiv:2605.11125 §3] [arxiv:2605.11125 eq. 14]. Sampling rotates each
position toward every embedding, weighted by that distribution. The top-1
option keeps only the most likely token [arxiv:2605.11125 eq. 15]
[arxiv:2605.11125 §3.1]. The noise range is truncated, and its density is set
adaptively from the training loss [arxiv:2605.11125 §3.2]. Card:
[Language Modeling with Hyperspherical Flows](/papers/2605.11125).

Author-reported on Sudoku exact match, the truncated and adaptive S-FLM reaches
94.8%, 85.2% and 45.0% on easy, medium and hard puzzles. MDLM reaches 92.0%,
77.1% and 30.2% [arxiv:2605.11125 tab. 1]. The plain S-FLM row, without the
truncation, is below MDLM at all three levels: 81.5%, 50.6% and 14.0%
[arxiv:2605.11125 tab. 1]. On GSM8K at T = 1 with 1024 sampling steps, top-1
velocity reaches 18.0%, level with the 18.0% that MDLM reaches in the same
table. The plain linear-schedule model reaches 1.2% [arxiv:2605.11125 tab. 2].
At T = 0.1 the paper reports MDLM at about 33% and S-FLM with top-1 at about
18% [arxiv:2605.11125 §4.2] [arxiv:2605.11125 fig. 3].

The numbers are single-run. The authors trained one model per configuration
[arxiv:2605.11125 §7]. The released sampling scripts default to 32 steps,
not the 1024 in the table, and the truncation side is read two ways across the
paper's text, its ablation table and its scripts. Check both before you try to
reproduce the GSM8K numbers. The [S-FLM card](/papers/2605.11125) lists the
details [arxiv:2605.11125 tab. 2] [arxiv:2605.11125 tab. 7]
[arxiv:2605.11125 §3.2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Text generator that fills many positions in parallel and can see both sides of a gap | Masked diffusion (MDLM) | Trained with a bound and sampled in steps. Steps multiply cost, and likelihoods are upper bounds [arxiv:2406.07524 §4.1] [arxiv:2406.07524 tab. 1] |
| Long text from a diffusion model | MDLM semi-autoregressive blocks | Reported only at one stride, so quality at other strides is unknown [arxiv:2406.07524 §4.2] [arxiv:2406.07524 tab. 5] |
| Graphs or molecules with labelled nodes and edges, one-shot generation | Categorical graph diffusion (DiGress) | Permutation-equivariant denoiser. Each layer costs Theta(n^2) and spectral features O(n^3) [arxiv:2209.14734 §3.2] [arxiv:2209.14734 app. B.2]. Size extrapolation is weak on MOSES [arxiv:2209.14734 tab. 7] |
| Generating sets of typed elements with coordinates, where some fields are fixed at inference and no retraining is allowed | Per-attribute mask-and-replace diffusion (LayoutDM) | Fixed fields are masked in and soft rules shift logits, with no retraining [arxiv:2303.08137 §3.3]. Results cover Rico and PubLayNet only [arxiv:2303.08137 §4.1] |
| Watermark diffusion LM output when left context is not yet fixed | Per-step tilting | Needs per-step logit access and a strength tuned to your texts. Reliable detection needs a few hundred tokens [arxiv:2509.24368 §3.2] [arxiv:2509.24368 §4.1] |
| Speed up an existing autoregressive LLM without changing its output | Parallel block drafter (DFlash) | The target verifies every token [arxiv:2602.06036 §4.1]. Needs training a draft per target model [arxiv:2602.06036 §5] [arxiv:2602.06036 app. A.1] |
| Reinforcement learning on verifiable rewards for a masked diffusion LLM, with a limited compute budget | Weighted policy optimisation (wd1) | No importance ratio or reference model, and one likelihood pass per completion [arxiv:2507.08838 §3.2] [arxiv:2507.08838 §2.3]. Reported gains come from a single seed on LLaDA-8B [arxiv:2507.08838 tab. 1] [arxiv:2507.08838 tab. 6] |
| Continuous, embedding-space generator, where a sum over the vocabulary at every step is affordable and low-temperature accuracy matters less | S-FLM with top-1 velocity | Top-1 matches MDLM on GSM8K at T = 1, but trails it at T = 0.1 [arxiv:2605.11125 tab. 2] [arxiv:2605.11125 §4.2]. The exact velocity sums over the whole vocabulary at each step [arxiv:2605.11125 eq. 15] |

## Implementation sketch

The shared core is a reverse loop over a corrupted state. Each variant changes
the prior, the model's output and the reverse update.

```pseudocode
# Common reverse loop for discrete diffusion
# State x: one discrete class per position (tokens, nodes, edges)
# Output: a sample of clean classes

def reverse_sample(model, corrupted, steps):
    # Start from the fully corrupted state: all masks, or class marginals
    # [arxiv:2406.07524 §4.1] [arxiv:2209.14734 alg. 2]
    x = corrupted
    for t in steps:                              # from T down toward 0
        # Distribution over clean classes at every position
        # [arxiv:2406.07524 §3.2.3] [arxiv:2209.14734 §3.2]
        p_clean = model(x, t)                    # (positions, classes)
        # Combine the prediction with the forward kernel
        # [arxiv:2209.14734 eq. 1] [arxiv:2209.14734 eq. 5]
        x = sample_reverse_step(x, p_clean, t)   # (positions,)
    return x
```

Masked variants keep unmasked positions fixed, so their reverse step only
fills masks [arxiv:2406.07524 §3.2.3]. The categorical graph variant
resamples every position from the posterior [arxiv:2209.14734 eq. 4]. The
layout variant re-imposes known fields after every step and adds weak priors
to the log-probabilities [arxiv:2303.08137 §3.3] [arxiv:2303.08137 eq. 6].
The watermark variant adds a tilt to `p_clean` before sampling
[arxiv:2509.24368 alg. 2]. The block drafter runs the same loop once per
round, with the target's features as extra context
[arxiv:2602.06036 §4.1]. The weighted policy optimisation variant leaves the
sampler alone and changes the training loss: it raises or lowers the
log-likelihood of sampled completions by their softmax weights
[arxiv:2507.08838 eq. 8].

## Contradictions

Across papers, no two cards conflict on a shared metric. The seven cards
measure different things, on different tasks. Within one paper, the graph
variant's marginal-noise ablation is mixed: validity rises and molecule
stability falls [arxiv:2209.14734 tab. 6]. The S-FLM text says S-FLM and FLM
outperform AR and MDLM at every Sudoku difficulty. That holds only for the
truncated S-FLM rows. The plain S-FLM row is below MDLM at all three levels
[arxiv:2605.11125 §4.1] [arxiv:2605.11125 tab. 1]. Treat the text's claim as
applying to the truncated variants. The wd1 paper's text and table disagree on its
Sudoku gain: the text gives 43% [arxiv:2507.08838 §5.1], the introduction
gives +58.8% [arxiv:2507.08838 §1], and the table's 76.4 against 17.6 is a
gain of 58.8 points, about 4.3 times [arxiv:2507.08838 tab. 1]. In the layout
paper, the refinement table's "LayoutDM w/o logit adjustment" row has the same
values as LayoutDM's category-to-size-and-position row in the conditional
table [arxiv:2303.08137 tab. 3] [arxiv:2303.08137 tab. 1]. The two are
different tasks, and the paper does not explain the match. Treat the
ablation numbers from that table with care.

## Papers

- [DiGress: Discrete Denoising diffusion for graph generation](/papers/2209.14734): categorical diffusion over graph nodes and edges, with a graph transformer denoiser.
- [Simple and Effective Masked Diffusion Language Models](/papers/2406.07524): masked diffusion for text, with a weighted masked-token objective and block sampling.
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](/papers/2303.08137): per-attribute mask-and-replace diffusion over layout tokens, with conditions applied at inference.
- [Watermarking Diffusion Language Models](/papers/2509.24368): red-green watermark applied as a per-step logit tilt for diffusion LMs.
- [DFlash: Block Diffusion for Flash Speculative Decoding](/papers/2602.06036): a parallel block drafter that fills masked slots in one pass, with target features injected as keys and values.
- [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](/papers/2507.08838): ratio-free reinforcement learning for masked diffusion LLMs, with a positive and a negative softmax-weighted likelihood term.
- [Language Modeling with Hyperspherical Flows](/papers/2605.11125): continuous flow over unit-norm token embeddings, benchmarked against masked diffusion on Sudoku and GSM8K.