---
kind: concept
slug: distribution-matching-distillation
title: Distribution matching distillation
summary: >-
  Distribution matching distillation trains a one-step or few-step diffusion student so its samples match the teacher's output distribution, using the difference of two score estimates. The base method adds a regression term to teacher outputs; the sDMD variant adds a gated optimal-transport term. Trade-offs: teacher data cost, extra training cost, and mode coverage.
tags: [diffusion-distillation, few-step-generation, text-to-image]
papers: ["2311.18828", "2610.39889"]
related: [diffusion-models, knowledge-distillation, score-distillation-sampling, optimal-transport, classifier-free-guidance]
status: draft
updated_at: 2026-11-21
---

## Overview

Diffusion samplers need many network passes per image. Distribution matching
distillation (DMD) trains a few-step student instead. The shared invariant is
that the student's samples should look like the teacher's samples in
distribution. The method does not need to reproduce a specific noise-to-image
mapping [arxiv:2311.18828 §1].

The generator update is the difference between two score estimates, both
evaluated on diffused student samples at a random noise level. One score comes
from a frozen teacher. The other comes from a fake-score denoiser that is
trained alongside the student on the student's own outputs [arxiv:2311.18828
§3.2 eq. 7]. The paper presents this as an approximation of the KL gradient
[arxiv:2311.18828 §3.2].

## Variants

### DMD: distribution matching with a regression term

The student is a one-step generator that takes no timestep input and starts as a
copy of the teacher [arxiv:2311.18828 §3.1]. The objective adds a regression
term: LPIPS between the student's output and the teacher's deterministic-ODE
output for the same starting noise. The weight is 0.25 by default
[arxiv:2311.18828 §3.3 eq. 9]. Under classifier-free guidance the guidance scale
is fixed during training [arxiv:2311.18828 §3.4]. Card:
[One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828).

Reported results, author-reported: class-conditional ImageNet 64x64 reaches FID
2.62 in one forward pass, against 2.32 for the EDM teacher with 512 passes
[arxiv:2311.18828 tab. 1]. Distilled Stable Diffusion v1.5 reaches FID 11.49 on
zero-shot COCO-30k at 0.09 s per image, against 8.78 at 2.59 s for the teacher
[arxiv:2311.18828 tab. 3]. Removing the regression term raises CIFAR-10 FID
from 2.66 to 5.58 and ImageNet-64 FID from 2.62 to 5.61 [arxiv:2311.18828 tab. 2].

### sDMD: adding a gated Sinkhorn transport term

sDMD keeps the DMD teacher-score signal and adds an entropic optimal-transport
velocity, computed with Sinkhorn iterations. The velocity compares student
samples with teacher samples and with the student's own batch [arxiv:2610.39889
§3 eq. 6, §5.1 eq. 9]. The paper argues that a teacher score alone is
unreliable in low-density regions, which motivates the extra term [arxiv:2610.39889
§1 §4.2].

The transport term is applied only to samples that a discriminator marks as
weakly supported by the teacher. The gate is lambda = 1 - D, and the
discriminator is the DMD2 adversarial discriminator, not a new one
[arxiv:2610.39889 §5.2 eq. 13 eq. 14]. Gate thresholds are 0.50 or 0.55 by
application [arxiv:2610.39889 tab. 6]. Transport runs in MAE features for
ImageNet, DINOv2 features for FLUX.1-dev, and clean latents for Qwen-Image and
Wan2.1 [arxiv:2610.39889 tab. 6] [arxiv:2610.39889 §5.3]. Card:
[sDMD: Sinkhorn Distribution Matching Distillation](/papers/2610.39889).

Reported results, author-reported. On ImageNet-256 one-step FID is 2.83, and
2.62 with REPA [arxiv:2610.39889 tab. 7]. On FLUX.1 at four steps, FID is 13.78
against 16.67 for DMD2 [arxiv:2610.39889 tab. 8]. Diversity is higher than DMD2
on the reported metrics [arxiv:2610.39889 §6.4 tab. 5], though on Wan2.1-14B
the VideoMAE V2 gap is small (13.85 against 13.74) [arxiv:2610.39889 tab. 5].
Training cost is 118 GPU-hours with diagonal masking and 145 with two-batch
self-transport, against 114 for DMD at the same ImageNet setting
[arxiv:2610.39889 tab. 10].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You can run the teacher to build a paired (noise, output) set for regression | DMD | The regression term is part of the objective, and removing it raises FID substantially [arxiv:2311.18828 tab. 2]. The LAION 6+ run used 12 million pairs [arxiv:2311.18828 app. B.4]. |
| Teacher data is the bottleneck and you can store teacher samples per prompt | Either; check the buffer cost | sDMD draws teacher samples per prompt into a buffer whose size is not stated [arxiv:2610.39889 §5.3]. |
| Mode coverage or diversity matters more than training cost | sDMD | Higher diversity on the reported metrics [arxiv:2610.39889 tab. 5], at about 27 percent more GPU-hours than DMD in the two-batch setting [arxiv:2610.39889 tab. 10]. |
| You need video results | sDMD | The sDMD card reports Wan2.1 at 1.3B and 14B [arxiv:2610.39889 tab. 2]. The DMD card reports image and text-to-image results only. |
| You want the fewest extra components to implement | DMD | sDMD adds an OT solver, a gate and per-application choices of transport space and iteration count [arxiv:2610.39889 tab. 6]. |

The two papers use different ImageNet settings: 64x64 with an EDM teacher in DMD
[arxiv:2311.18828 tab. 1], and 256x256 with SiT-XL/2 in sDMD [arxiv:2610.39889
tab. 6]. Their FID values are not directly comparable.

No official sDMD code was found. The sDMD card's Paper vs code section lists the
searches made. An implementation would work from the paper alone.

## Implementation sketch

The common core is the DMD update. sDMD changes only the velocity in step 3 and
the gate in step 4. Shapes are in comments; the code is pseudocode, not a
library call.

```pseudocode
# One DMD generator step  [arxiv:2311.18828 alg. 1] [eq. 7]
# teacher: frozen; fake: trainable denoiser, initialised from teacher
def dmd_generator_loss(images, teacher, fake):
    # 1. Random diffusion time per sample  [arxiv:2311.18828 §3.2]
    t = sample_uniform(T_MIN, T_MAX, shape=(B,))
    alpha, sigma = schedule(t)
    # 2. Diffuse the student's outputs  [arxiv:2311.18828 eq. 3]
    x_t = alpha * images + sigma * normal_like(images)
    # 3. Mean predictions of both denoisers, no gradient  [arxiv:2311.18828 §3.2]
    with no_grad():
        mu_real = teacher(x_t, t)
        mu_fake = fake(x_t, t)
    # 4. Score difference, scaled per sample  [arxiv:2311.18828 eq. 4-5, eq. 8]
    s_real = -(x_t - alpha * mu_real) / sigma**2
    s_fake = -(x_t - alpha * mu_fake) / sigma**2
    w = (sigma**2 / alpha) * (C * S) / l1(mu_real - images)  # (B,)
    grad = w * alpha * (s_fake - s_real)
    # 5. Surrogate loss whose gradient w.r.t. images is grad  [arxiv:2311.18828 app. B]
    dm = 0.5 * mse(images, stop_gradient(images - grad))
    # 6. Regression to teacher ODE outputs on paired noise  [arxiv:2311.18828 eq. 9]
    reg = lpips(student(pair_noise), pair_targets)
    return dm + 0.25 * reg
```

```pseudocode
# sDMD: add a gated transport term to the velocity  [arxiv:2610.39889 alg. 1]
# lambda = 1 - D from the DMD2 discriminator  [arxiv:2610.39889 §5.2]
def sdmd_velocity(x_t, t, students, teacher_samples):
    # 1. IKL velocity, as in DMD  [arxiv:2610.39889 eq. 8]
    v_ikl = alpha(t) * (teacher_score(x_t, t) - fake_score(x_t, t))
    # 2. Per-sample gate  [arxiv:2610.39889 eq. 13-14]
    gate = 1 - discriminator(x_t, t)                     # (B,)
    active = gate > threshold                            # (B,)
    # 3. Two Sinkhorn barycentric projections  [arxiv:2610.39889 alg. 2]
    t_qp = sinkhorn_projection(students, teacher_samples, beta, L)
    t_qq = sinkhorn_projection(students, students, beta, L)
    # 4. Add the weighted term only where the gate is active  [arxiv:2610.39889 eq. 14]
    return where(active, v_ikl + w * (t_qp - t_qq), v_ikl)
```

Two details are not settled by the paper. The Sinkhorn kernel is formed
directly as an exponential, with beta as small as 0.05 [arxiv:2610.39889 alg.
2 tab. 6]. A log-domain form is a choice an implementer has to make. The
iteration count is fixed, with no convergence test [arxiv:2610.39889 alg. 2].

## Contradictions

- **Gating in sDMD.** Algorithm 1 has one gate condition with no per-sample
  index, while Eq. 14 applies the indicator per sample [arxiv:2610.39889 alg. 1]
  [arxiv:2610.39889 eq. 14]. The sketch above uses the per-sample form.
- **Self-transport in sDMD.** Algorithm 1 draws the self term from the same
  student batch with no mask. Table 6 uses a diagonal mask for FLUX, Qwen-Image
  and Wan2.1, and separate batches for ImageNet [arxiv:2610.39889 alg. 1]
  [arxiv:2610.39889 tab. 6].
- **Transport space in sDMD.** Section 5.3 recommends clean latents for large
  models, and Table 6 uses DINOv2 features for FLUX [arxiv:2610.39889 §5.3]
  [arxiv:2610.39889 tab. 6].
- **Small gaps.** Some sDMD ablation gaps are small, for example 2.83 against
  2.88 on ImageNet and 13.78 against 13.89 on FLUX [arxiv:2610.39889 tab. 4].
  The paper reports no variance over seeds.

## Papers

- [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828): introduces DMD, a one-step generator trained with a score-difference gradient and an LPIPS regression term.
- [sDMD: Sinkhorn Distribution Matching Distillation](/papers/2610.39889): adds a gated entropic optimal-transport term to DMD for better coverage and diversity, with 2- and 4-step students on image and video models.
