---
kind: concept
slug: probability-flow-ode
title: Probability-flow ODE
summary: >-
  A deterministic path replaces a stochastic sampler with a fixed trajectory. Discrete states need auxiliary randomness, since a deterministic map cannot split probability mass. One discrete-diffusion sampler draws an Exp(1) residual per position and token, moves them by an ODE, and changes a token at zero. Trade-off: the fixed path enables distillation; uniform rates need bisection.
tags: [sampling, ode, discrete-diffusion, distillation]
papers: ["2610.40618"]
related: [discrete-diffusion, diffusion-models, knowledge-distillation, flow-matching, neural-ode]
status: draft
updated_at: 2026-11-21
---

## Overview

A probability-flow ODE turns a stochastic sampler into a deterministic path, so
that every starting noise value maps to one output. The card's authors point
to the continuous version as the reference point and note that a deterministic
analogue on a finite state space cannot split probability mass between tokens
[arxiv:2610.40618 §1]. Their fix is to draw continuous auxiliary randomness once,
at the start, and to let a fixed ODE move it. The invariant all variants aim
for is that the marginal distribution at every time is preserved
[arxiv:2610.40618 §3.1]. Masked diffusion is the hardest case, because its
terminal distribution is a point mass [arxiv:2610.40618 §1].

## Variants

### Exponential residual flow (ERF) for discrete diffusion

Each position and token pair gets one residual, drawn from Exp(1) at the start.
Between jumps the sequence is constant. The residual of a non-current token
shrinks, and the residual of the current token grows, at rates set by the
reverse and forward chains. The first residual to reach zero fixes the jump
time, the position and the new token [arxiv:2610.40618 §3] [arxiv:2610.40618 §3.1].
Card: [An ODE-based Sampler for Discrete Diffusion Models](/papers/2610.40618).

The paper's Gillespie comparison matters here. Gillespie-style samplers draw
fresh exponentials after each jump. The ERF sampler keeps unused residuals and
updates them deterministically instead [arxiv:2610.40618 §3] [arxiv:2610.40618 §3.1].

### Frozen leave-one-out sampler

The practical sampler uses a fixed time grid. On each interval the denoiser is
evaluated once at the interval start and held constant. Jump times are then
solved in closed form where possible [arxiv:2610.40618 §3.2] [arxiv:2610.40618 app. A].
The paper states that the frozen version has finitely many jumps, while
unfrozen forward exit rates can produce infinitely many [arxiv:2610.40618 §3.2].

- **Masked diffusion:** closed-form jump time from the residual, the frozen
  denoiser and the schedule. The candidate is valid only under a positivity
  condition on the denoiser output and a bound on the residual; otherwise it
  is treated as never firing [arxiv:2610.40618 app. A].
- **Uniform diffusion, canonical rates:** no closed-form inverse. The
  experiments solve it by bisection with 40 iterations per solve
  [arxiv:2610.40618 §6.1] [arxiv:2610.40618 app. F]. The paper reports about
  7 jumps per position at 8 NFEs [arxiv:2610.40618 tab. 2].
- **Uniform diffusion, net-flux rates (U-NetFlux):** a different rate
  definition with a closed-form jump time [arxiv:2610.40618 §3.2] [arxiv:2610.40618 app. A].
  The paper reports about 1 jump per position at 8 NFEs [arxiv:2610.40618 tab. 2].

Positions can be updated in parallel for masked, uniform and U-NetFlux rates,
because the frozen rate at one position depends only on that position's state
[arxiv:2610.40618 §3.2]. Other frozen-rate choices, the paper says, need
sequential updates [arxiv:2610.40618 app. A].

### Consistency distillation on ERF paths

Because the ERF map is deterministic given the augmented state, a teacher's
trajectory gives a fixed target for a student. The student predicts a clean
token distribution from the state at time t, and the target is a frozen copy
of the student evaluated at time t minus a gap delta
[arxiv:2610.40618 §4] [arxiv:2610.40618 eq. 16]. The student is conditioned on
a transform of the residual, −log E_t [arxiv:2610.40618 §6.2]. Multi-step
sampling alternates a clean-state prediction with a re-noising step
[arxiv:2610.40618 app. D].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Masked diffusion, sampling only, no training budget | Frozen leave-one-out sampler with the masked closed form | Closed-form jump times; the paper reports quality close to ancestral sampling at matched model calls [arxiv:2610.40618 §6.1] |
| Uniform diffusion with canonical rates | Only if a bisection solve per jump is affordable | 40 bisection steps per solve, about 7 jumps per position at 8 NFEs [arxiv:2610.40618 §6.1] [arxiv:2610.40618 tab. 2] |
| Uniform diffusion, latency matters, and the net-flux form is acceptable | U-NetFlux | Closed form and about 1 jump per position at 8 NFEs; reported speed-up 2.83 (TinyGSM) and 2.00 (OpenWebText) at 8 NFEs [arxiv:2610.40618 §6.1] [arxiv:2610.40618 tab. 2] |
| Many sampling steps (512 NFEs) | Expect little gain | Reported speed-up falls to 1.10 to 1.15 at 512 NFEs [arxiv:2610.40618 tab. 6] |
| Few-step generation from a trained teacher | Consistency distillation on ERF paths | The deterministic path gives the student a fixed target; author-reported gains largest for uniform diffusion [arxiv:2610.40618 §4] [arxiv:2610.40618 §6] |
| No trained teacher, or no distillation budget | Plain ERF sampling | The paper's distillation runs used 50 GPU-hours (TinyGSM) and 100 (OpenWebText), excluding queue and evaluation time; the paper gives no smaller budget [arxiv:2610.40618 §6.2] [arxiv:2610.40618 app. F] |

Timing numbers are for one AMD MI210 with batches of four and do not transfer
to other hardware [arxiv:2610.40618 app. F]. The paper's DiffusionGemma run
compares an ERF sampler at 48 forward evaluations against a native procedure
that averages 11.1, so timings there are not compute-matched
[arxiv:2610.40618 app. G].

## Implementation sketch

Our own simplified outline of one ERF interval with a frozen denoiser, based on
the description in [arxiv:2610.40618 §3.2] and [arxiv:2610.40618 app. A]. It is
not the paper's algorithm listing. Check the closed forms in appendix A before
use.

```pseudocode
# Freeze the leave-one-out denoiser at the start of [s, t]  [§3.2]
def erf_interval(x, e, t, s, denoiser):
    d_hat = denoiser(x, t)               # per position and token
    for pos in positions:                # parallel for masked/uniform/net-flux
        now = t
        while now > s:
            # Earliest residual-zero crossing at this position  [app. A]
            t_hit, tok = next_crossing(pos, x, e, now, s, d_hat)
            # Passive residuals move by their integrated rate  [§3.1 eq. 9]
            advance_residuals(pos, x, e, now, t_hit, d_hat)
            if t_hit < s:                # no jump left in this interval
                break
            x[pos] = tok                 # the crossing token takes over
            now = t_hit
    return x, e
```

For distillation, the same ERF step is run by a teacher from (x_t, e_t) to
time s, and the student at time t is matched to a frozen target at time s
[arxiv:2610.40618 §4] [arxiv:2610.40618 alg. 4].

## Contradictions

No disagreements across cards: only one card lists this slug. Within the
source paper, two points are unresolved:

- **Divergence argument order.** The consistency loss is written as
  KL(target ‖ student) [arxiv:2610.40618 eq. 16]. The round-based training loop
  passes the student first, which reads as the reverse order
  [arxiv:2610.40618 alg. 4] [arxiv:2610.40618 app. D]. The text says forward KL
  for OpenWebText and reverse KL for TinyGSM [arxiv:2610.40618 §6.2]. The paper
  does not say which order produced those results.
- **Speed-up against ancestral sampling.** The paper reports speed-ups of 2.83
  and 2.00 at 8 NFEs on TinyGSM and OpenWebText [arxiv:2610.40618 §6.1]. On
  DiffusionGemma, the ERF samplers take 61.1 and 60.4 seconds per question
  against 59.4 for ancestral sampling [arxiv:2610.40618 tab. 1]. The
  DiffusionGemma runs use a different model and a native baseline that stops
  early, so the two results do not directly conflict, but they do not show the
  same gain.

## Papers

- [An ODE-based Sampler for Discrete Diffusion Models](/papers/2610.40618): the exponential residual flow sampler, its frozen leave-one-out
  discretisation, closed-form jump times for masked and net-flux uniform
  diffusion, and consistency distillation on the deterministic ERF paths.
