wikillm

Concept · 1 paper · updated 2026-11-21

Probability-flow ODE

A deterministic path replaces a stochastic sampler with a fixed trajectory. Discrete states need auxiliary randomness, since a deterministic map cannot split probability mass. One discrete-diffusion sampler draws an Exp(1) residual per position and token, moves them by an ODE, and changes a token at zero. Trade-off: the fixed path enables distillation; uniform rates need bisection.

samplingodediscrete-diffusiondistillation

Markdown5 min read

AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A probability-flow ODE turns a stochastic sampler into a deterministic path, so that every starting noise value maps to one output. The card's authors point to the continuous version as the reference point and note that a deterministic analogue on a finite state space cannot split probability mass between tokens 2610.40618 §1. Their fix is to draw continuous auxiliary randomness once, at the start, and to let a fixed ODE move it. The invariant all variants aim for is that the marginal distribution at every time is preserved 2610.40618 §3.1. Masked diffusion is the hardest case, because its terminal distribution is a point mass 2610.40618 §1.

Variants

Exponential residual flow (ERF) for discrete diffusion

Each position and token pair gets one residual, drawn from Exp(1) at the start. Between jumps the sequence is constant. The residual of a non-current token shrinks, and the residual of the current token grows, at rates set by the reverse and forward chains. The first residual to reach zero fixes the jump time, the position and the new token 2610.40618 §3 2610.40618 §3.1. Card: An ODE-based Sampler for Discrete Diffusion Models.

The paper's Gillespie comparison matters here. Gillespie-style samplers draw fresh exponentials after each jump. The ERF sampler keeps unused residuals and updates them deterministically instead 2610.40618 §3 2610.40618 §3.1.

Frozen leave-one-out sampler

The practical sampler uses a fixed time grid. On each interval the denoiser is evaluated once at the interval start and held constant. Jump times are then solved in closed form where possible 2610.40618 §3.2 2610.40618 app. A. The paper states that the frozen version has finitely many jumps, while unfrozen forward exit rates can produce infinitely many 2610.40618 §3.2.

  • Masked diffusion: closed-form jump time from the residual, the frozen denoiser and the schedule. The candidate is valid only under a positivity condition on the denoiser output and a bound on the residual; otherwise it is treated as never firing 2610.40618 app. A.
  • Uniform diffusion, canonical rates: no closed-form inverse. The experiments solve it by bisection with 40 iterations per solve 2610.40618 §6.1 2610.40618 app. F. The paper reports about 7 jumps per position at 8 NFEs 2610.40618 tab. 2.
  • Uniform diffusion, net-flux rates (U-NetFlux): a different rate definition with a closed-form jump time 2610.40618 §3.2 2610.40618 app. A. The paper reports about 1 jump per position at 8 NFEs 2610.40618 tab. 2.

Positions can be updated in parallel for masked, uniform and U-NetFlux rates, because the frozen rate at one position depends only on that position's state 2610.40618 §3.2. Other frozen-rate choices, the paper says, need sequential updates 2610.40618 app. A.

Consistency distillation on ERF paths

Because the ERF map is deterministic given the augmented state, a teacher's trajectory gives a fixed target for a student. The student predicts a clean token distribution from the state at time t, and the target is a frozen copy of the student evaluated at time t minus a gap delta 2610.40618 §4 2610.40618 eq. 16. The student is conditioned on a transform of the residual, −log E_t 2610.40618 §6.2. Multi-step sampling alternates a clean-state prediction with a re-noising step 2610.40618 app. D.

When to use which

SituationUseWhy
Masked diffusion, sampling only, no training budgetFrozen leave-one-out sampler with the masked closed formClosed-form jump times; the paper reports quality close to ancestral sampling at matched model calls 2610.40618 §6.1
Uniform diffusion with canonical ratesOnly if a bisection solve per jump is affordable40 bisection steps per solve, about 7 jumps per position at 8 NFEs 2610.40618 §6.1 2610.40618 tab. 2
Uniform diffusion, latency matters, and the net-flux form is acceptableU-NetFluxClosed form and about 1 jump per position at 8 NFEs; reported speed-up 2.83 (TinyGSM) and 2.00 (OpenWebText) at 8 NFEs 2610.40618 §6.1 2610.40618 tab. 2
Many sampling steps (512 NFEs)Expect little gainReported speed-up falls to 1.10 to 1.15 at 512 NFEs 2610.40618 tab. 6
Few-step generation from a trained teacherConsistency distillation on ERF pathsThe deterministic path gives the student a fixed target; author-reported gains largest for uniform diffusion 2610.40618 §4 2610.40618 §6
No trained teacher, or no distillation budgetPlain ERF samplingThe paper's distillation runs used 50 GPU-hours (TinyGSM) and 100 (OpenWebText), excluding queue and evaluation time; the paper gives no smaller budget 2610.40618 §6.2 2610.40618 app. F

Timing numbers are for one AMD MI210 with batches of four and do not transfer to other hardware 2610.40618 app. F. The paper's DiffusionGemma run compares an ERF sampler at 48 forward evaluations against a native procedure that averages 11.1, so timings there are not compute-matched 2610.40618 app. G.

Implementation sketch

Our own simplified outline of one ERF interval with a frozen denoiser, based on the description in 2610.40618 §3.2 and 2610.40618 app. A. It is not the paper's algorithm listing. Check the closed forms in appendix A before use.

Pseudocode
# Freeze the leave-one-out denoiser at the start of [s, t]  [§3.2]
def erf_interval(x, e, t, s, denoiser):
    d_hat = denoiser(x, t)               # per position and token
    for pos in positions:                # parallel for masked/uniform/net-flux
        now = t
        while now > s:
            # Earliest residual-zero crossing at this position  [app. A]
            t_hit, tok = next_crossing(pos, x, e, now, s, d_hat)
            # Passive residuals move by their integrated rate  [§3.1 eq. 9]
            advance_residuals(pos, x, e, now, t_hit, d_hat)
            if t_hit < s:                # no jump left in this interval
                break
            x[pos] = tok                 # the crossing token takes over
            now = t_hit
    return x, e

For distillation, the same ERF step is run by a teacher from (x_t, e_t) to time s, and the student at time t is matched to a frozen target at time s 2610.40618 §4 2610.40618 alg. 4.

Contradictions

No disagreements across cards: only one card lists this slug. Within the source paper, two points are unresolved:

  • Divergence argument order. The consistency loss is written as KL(target ‖ student) 2610.40618 eq. 16. The round-based training loop passes the student first, which reads as the reverse order 2610.40618 alg. 4 2610.40618 app. D. The text says forward KL for OpenWebText and reverse KL for TinyGSM 2610.40618 §6.2. The paper does not say which order produced those results.
  • Speed-up against ancestral sampling. The paper reports speed-ups of 2.83 and 2.00 at 8 NFEs on TinyGSM and OpenWebText 2610.40618 §6.1. On DiffusionGemma, the ERF samplers take 61.1 and 60.4 seconds per question against 59.4 for ancestral sampling 2610.40618 tab. 1. The DiffusionGemma runs use a different model and a native baseline that stops early, so the two results do not directly conflict, but they do not show the same gain.

Papers

  • An ODE-based Sampler for Discrete Diffusion Models: the exponential residual flow sampler, its frozen leave-one-out discretisation, closed-form jump times for masked and net-flux uniform diffusion, and consistency distillation on the deterministic ERF paths.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.