Concept · 2 papers · updated 2026-11-21
Distribution matching distillation
Distribution matching distillation trains a one-step or few-step diffusion student so its samples match the teacher's output distribution, using the difference of two score estimates. The base method adds a regression term to teacher outputs; the sDMD variant adds a gated optimal-transport term. Trade-offs: teacher data cost, extra training cost, and mode coverage.
diffusion-distillationfew-step-generationtext-to-image
Markdown5 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
Diffusion samplers need many network passes per image. Distribution matching distillation (DMD) trains a few-step student instead. The shared invariant is that the student's samples should look like the teacher's samples in distribution. The method does not need to reproduce a specific noise-to-image mapping 2311.18828 §1.
The generator update is the difference between two score estimates, both evaluated on diffused student samples at a random noise level. One score comes from a frozen teacher. The other comes from a fake-score denoiser that is trained alongside the student on the student's own outputs [arxiv:2311.18828 §3.2 eq. 7]. The paper presents this as an approximation of the KL gradient 2311.18828 §3.2.
Variants
DMD: distribution matching with a regression term
The student is a one-step generator that takes no timestep input and starts as a copy of the teacher 2311.18828 §3.1. The objective adds a regression term: LPIPS between the student's output and the teacher's deterministic-ODE output for the same starting noise. The weight is 0.25 by default 2311.18828 §3.3 eq. 9. Under classifier-free guidance the guidance scale is fixed during training 2311.18828 §3.4. Card: One-step Diffusion with Distribution Matching Distillation.
Reported results, author-reported: class-conditional ImageNet 64x64 reaches FID 2.62 in one forward pass, against 2.32 for the EDM teacher with 512 passes 2311.18828 tab. 1. Distilled Stable Diffusion v1.5 reaches FID 11.49 on zero-shot COCO-30k at 0.09 s per image, against 8.78 at 2.59 s for the teacher 2311.18828 tab. 3. Removing the regression term raises CIFAR-10 FID from 2.66 to 5.58 and ImageNet-64 FID from 2.62 to 5.61 2311.18828 tab. 2.
sDMD: adding a gated Sinkhorn transport term
sDMD keeps the DMD teacher-score signal and adds an entropic optimal-transport velocity, computed with Sinkhorn iterations. The velocity compares student samples with teacher samples and with the student's own batch [arxiv:2610.39889 §3 eq. 6, §5.1 eq. 9]. The paper argues that a teacher score alone is unreliable in low-density regions, which motivates the extra term [arxiv:2610.39889 §1 §4.2].
The transport term is applied only to samples that a discriminator marks as weakly supported by the teacher. The gate is lambda = 1 - D, and the discriminator is the DMD2 adversarial discriminator, not a new one 2610.39889 §5.2 eq. 13 eq. 14. Gate thresholds are 0.50 or 0.55 by application 2610.39889 tab. 6. Transport runs in MAE features for ImageNet, DINOv2 features for FLUX.1-dev, and clean latents for Qwen-Image and Wan2.1 2610.39889 tab. 6 2610.39889 §5.3. Card: sDMD: Sinkhorn Distribution Matching Distillation.
Reported results, author-reported. On ImageNet-256 one-step FID is 2.83, and 2.62 with REPA 2610.39889 tab. 7. On FLUX.1 at four steps, FID is 13.78 against 16.67 for DMD2 2610.39889 tab. 8. Diversity is higher than DMD2 on the reported metrics 2610.39889 §6.4 tab. 5, though on Wan2.1-14B the VideoMAE V2 gap is small (13.85 against 13.74) 2610.39889 tab. 5. Training cost is 118 GPU-hours with diagonal masking and 145 with two-batch self-transport, against 114 for DMD at the same ImageNet setting 2610.39889 tab. 10.
When to use which
| Situation | Use | Why |
|---|---|---|
| You can run the teacher to build a paired (noise, output) set for regression | DMD | The regression term is part of the objective, and removing it raises FID substantially 2311.18828 tab. 2. The LAION 6+ run used 12 million pairs 2311.18828 app. B.4. |
| Teacher data is the bottleneck and you can store teacher samples per prompt | Either; check the buffer cost | sDMD draws teacher samples per prompt into a buffer whose size is not stated 2610.39889 §5.3. |
| Mode coverage or diversity matters more than training cost | sDMD | Higher diversity on the reported metrics 2610.39889 tab. 5, at about 27 percent more GPU-hours than DMD in the two-batch setting 2610.39889 tab. 10. |
| You need video results | sDMD | The sDMD card reports Wan2.1 at 1.3B and 14B 2610.39889 tab. 2. The DMD card reports image and text-to-image results only. |
| You want the fewest extra components to implement | DMD | sDMD adds an OT solver, a gate and per-application choices of transport space and iteration count 2610.39889 tab. 6. |
The two papers use different ImageNet settings: 64x64 with an EDM teacher in DMD 2311.18828 tab. 1, and 256x256 with SiT-XL/2 in sDMD [arxiv:2610.39889 tab. 6]. Their FID values are not directly comparable.
No official sDMD code was found. The sDMD card's Paper vs code section lists the searches made. An implementation would work from the paper alone.
Implementation sketch
The common core is the DMD update. sDMD changes only the velocity in step 3 and the gate in step 4. Shapes are in comments; the code is pseudocode, not a library call.
# One DMD generator step [arxiv:2311.18828 alg. 1] [eq. 7]
# teacher: frozen; fake: trainable denoiser, initialised from teacher
def dmd_generator_loss(images, teacher, fake):
# 1. Random diffusion time per sample [arxiv:2311.18828 §3.2]
t = sample_uniform(T_MIN, T_MAX, shape=(B,))
alpha, sigma = schedule(t)
# 2. Diffuse the student's outputs [arxiv:2311.18828 eq. 3]
x_t = alpha * images + sigma * normal_like(images)
# 3. Mean predictions of both denoisers, no gradient [arxiv:2311.18828 §3.2]
with no_grad():
mu_real = teacher(x_t, t)
mu_fake = fake(x_t, t)
# 4. Score difference, scaled per sample [arxiv:2311.18828 eq. 4-5, eq. 8]
s_real = -(x_t - alpha * mu_real) / sigma**2
s_fake = -(x_t - alpha * mu_fake) / sigma**2
w = (sigma**2 / alpha) * (C * S) / l1(mu_real - images) # (B,)
grad = w * alpha * (s_fake - s_real)
# 5. Surrogate loss whose gradient w.r.t. images is grad [arxiv:2311.18828 app. B]
dm = 0.5 * mse(images, stop_gradient(images - grad))
# 6. Regression to teacher ODE outputs on paired noise [arxiv:2311.18828 eq. 9]
reg = lpips(student(pair_noise), pair_targets)
return dm + 0.25 * reg# sDMD: add a gated transport term to the velocity [arxiv:2610.39889 alg. 1]
# lambda = 1 - D from the DMD2 discriminator [arxiv:2610.39889 §5.2]
def sdmd_velocity(x_t, t, students, teacher_samples):
# 1. IKL velocity, as in DMD [arxiv:2610.39889 eq. 8]
v_ikl = alpha(t) * (teacher_score(x_t, t) - fake_score(x_t, t))
# 2. Per-sample gate [arxiv:2610.39889 eq. 13-14]
gate = 1 - discriminator(x_t, t) # (B,)
active = gate > threshold # (B,)
# 3. Two Sinkhorn barycentric projections [arxiv:2610.39889 alg. 2]
t_qp = sinkhorn_projection(students, teacher_samples, beta, L)
t_qq = sinkhorn_projection(students, students, beta, L)
# 4. Add the weighted term only where the gate is active [arxiv:2610.39889 eq. 14]
return where(active, v_ikl + w * (t_qp - t_qq), v_ikl)Two details are not settled by the paper. The Sinkhorn kernel is formed directly as an exponential, with beta as small as 0.05 [arxiv:2610.39889 alg. 2 tab. 6]. A log-domain form is a choice an implementer has to make. The iteration count is fixed, with no convergence test 2610.39889 alg. 2.
Contradictions
- Gating in sDMD. Algorithm 1 has one gate condition with no per-sample index, while Eq. 14 applies the indicator per sample 2610.39889 alg. 1 2610.39889 eq. 14. The sketch above uses the per-sample form.
- Self-transport in sDMD. Algorithm 1 draws the self term from the same student batch with no mask. Table 6 uses a diagonal mask for FLUX, Qwen-Image and Wan2.1, and separate batches for ImageNet 2610.39889 alg. 1 2610.39889 tab. 6.
- Transport space in sDMD. Section 5.3 recommends clean latents for large models, and Table 6 uses DINOv2 features for FLUX 2610.39889 §5.3 2610.39889 tab. 6.
- Small gaps. Some sDMD ablation gaps are small, for example 2.83 against 2.88 on ImageNet and 13.78 against 13.89 on FLUX 2610.39889 tab. 4. The paper reports no variance over seeds.
Papers
- One-step Diffusion with Distribution Matching Distillation: introduces DMD, a one-step generator trained with a score-difference gradient and an LPIPS regression term.
- sDMD: Sinkhorn Distribution Matching Distillation: adds a gated entropic optimal-transport term to DMD for better coverage and diversity, with 2- and 4-step students on image and video models.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.