Adversarial Diffusion Distillation

In plain words

Diffusion models make very good images, but they build each image by running a large network many times, so they are slow. GANs make an image in one pass but have been hard to scale to text-to-image quality. This paper combines the two ideas. It starts from a big pretrained diffusion model and trains a new, small student that produces an image in one to four passes.

The student learns from two sources at once. A judge network learns to tell student images from real photos, and pushes the student toward realistic results. The original diffusion model also acts as a teacher and tells the student what it would have drawn, which keeps the prompt-following and the overall composition of the big model. The authors report that the four-step result beats its own teacher on quality and prompt alignment, with less variety between samples.

TL;DR

• A frozen diffusion model is the teacher; a copy of it, initialised from its weights, becomes the student.
• Training uses an adversarial hinge loss on frozen ViT features plus a score distillation loss, combined with weight λ = 2.5.
• The student is a single U-Net evaluation at inference, with no classifier-free guidance; it can run 1, 2 or 4 steps.
• Author-reported headline: ADD-XL (SDXL backbone) with one step beats LCM-XL with four steps in human preference, and with four steps it beats SDXL at 50 steps in most comparisons.

Problem

Few-step diffusion samplers and distilled models often look blurry or show artifacts at 1 to 4 steps, and the artifacts grow as the step count drops §2. Single-step GANs are fast but lag diffusion models in quality, and scaling GANs to large text-to-image data has saturated §1 §2. Score distillation alone can make the teacher's outputs match, but on its own the paper finds it ineffective at few steps tab. 1. The method targets the one- to four-step regime with a foundation text-to-image model as teacher §1.

Method

The training setup has three networks §3.1:

1. Student: a denoiser initialised from the pretrained diffusion U-Net (weights θ). It receives a noised real image and outputs a sample.
2. Discriminator: a frozen pretrained feature network F with several trainable, lightweight heads D_{φ,k} on features from different layers §3.2. Heads are conditioned by projection on a text embedding and an image embedding §3.2.
3. Teacher: a frozen diffusion model (weights ψ) §3.1.

Step 1: noising and sampling. The student is trained on noised real images x_s = α_s x_0 + σ_s ε. The coefficients are those of the student diffusion model. The timestep s is drawn uniformly from N = 4 student timesteps {τ_1, …, τ_N}, with τ_N = 1000 and zero-terminal SNR enforced §3.1.

Step 2: adversarial term. The student output is scored by the heads on frozen features. The generator term is the negative sum of head scores §3.2 eq. 2. The discriminator is trained with the hinge loss, plus an R1 penalty computed on the inputs of each head rather than on pixels §3.2 eq. 3. The paper says R1 matters most above 128 px output resolution §3.2.

Step 3: distillation term. The student sample is re-noised with the teacher's forward process, x̂_{θ,t} = α_t x̂_θ + σ_t ε′, and the teacher denoises it. The teacher's clean estimate is x̂_ψ = (x_t − σ_t ε̂_ψ) / α_t §3.3. The teacher is not applied to the raw student sample, because the paper considers that out of distribution for the teacher §3.3. The distance is squared L2, the target is detached with stop-gradient, and the per-timestep weight is c(t) §3.3 eq. 4.

Step 4: weighting. Two choices of c(t) are compared: an exponential weight c(t) = α_t, which gives more diverse samples, and SDS-style weighting §3.3 §4.1 tab. 1. A third option, noise-free score distillation (NFSD), is also evaluated, and the final model uses NFSD §3.3 §4.1. The appendix shows that squared L2 with c(t) = α_t / (2σ_t) · w(t) gives the same gradient as the SDS objective of DreamFusion app. A.

Total objective. The student minimises the adversarial term plus λ times the distillation term §3.1 eq. 1. The discriminator minimises its own hinge loss §3.2 eq. 3.

# Student (generator) loss for one batch  [§3.1] [eq. 1]
# Inputs: real images x0 (B, 3, H, W) in pixel space
# Output: scalar loss for the student parameters theta

def student_loss(x0, text_emb, image_emb):
    # Draw one of the N = 4 student timesteps  [§3.1]
    s = sample_student_timestep()
    # Noise the real image with the student's forward process  [§3.1]
    eps = sample_gaussian_noise(x0)
    x_s = alpha_s * x0 + sigma_s * eps
    # One student evaluation gives the sample  [§3.1]
    x_hat = student(x_s, s)
    # Adversarial term: fool the heads on frozen features  [§3.2] [eq. 2]
    feats = frozen_vit(x_hat)
    adv = -sum_over_heads(heads(feats, text_emb, image_emb))
    # Re-noise the sample with the teacher's forward process  [§3.3]
    t = sample_teacher_timestep()   # distribution not stated [§3.3]
    eps_prime = sample_gaussian_noise(x_hat)
    x_hat_t = alpha_t * x_hat + sigma_t * eps_prime
    # Teacher's clean estimate is a fixed target  [§3.3] [eq. 4]
    x_teacher = stop_gradient(teacher_x0_pred(x_hat_t, t))
    # Weighted squared-L2 distillation term  [§3.3] [eq. 4]
    distill = c(t) * sum_of_squares(x_hat - x_teacher)
    # Combine with lambda = 2.5  [§4] [eq. 1]
    return adv + lambda_distill * distill

# Discriminator (head) loss in hinge form  [§3.2] [eq. 3]
# Real and generated images pass through the same frozen features
def discriminator_loss(x0, x_hat, text_emb, image_emb):
    # Scores for real images and for student samples  [§3.2]
    real_scores = heads(frozen_vit(x0), text_emb, image_emb)
    fake_scores = heads(frozen_vit(x_hat), text_emb, image_emb)
    # Hinge terms, summed over the heads  [§3.2] [eq. 3]
    loss = sum(maximum(0, 1 - real_scores)) + sum(maximum(0, 1 + fake_scores))
    # R1 penalty on each head's input, gamma = 1e-5  [§3.2] [§4]
    loss = loss + gamma * r1_penalty(real_scores, real_head_inputs)
    return loss

Key equations, as typeset in the paper:

\mathcal{L} = \mathcal{L}_{\mathrm{adv}}^{G} + \lambda\, \mathcal{L}_{\mathrm{distill}} eq. 1

\mathcal{L}_{\mathrm{adv}}^{G} = -\,\mathbb{E}_{s,\epsilon,x_0}\Big[\sum_k \mathcal{D}_{\phi,k}\big(F_k(\hat{x}_\theta(x_s,s))\big)\Big] eq. 2

\mathcal{L}_{\mathrm{adv}}^{D} = \mathbb{E}_{x_0}\Big[\sum_k \max\big(0,\,1-\mathcal{D}_{\phi,k}(F_k(x_0))\big) + \gamma\, R1(\phi)\Big] + \mathbb{E}_{\hat{x}_\theta}\Big[\sum_k \max\big(0,\,1+\mathcal{D}_{\phi,k}(F_k(\hat{x}_\theta))\big)\Big] eq. 3

\mathcal{L}_{\mathrm{distill}} = \mathbb{E}_{t,\epsilon'}\Big[c(t)\; d\big(\hat{x}_\theta,\; \hat{x}_\psi(\mathrm{sg}(\hat{x}_{\theta,t});\, t)\big)\Big], \quad d(x,y) = \lVert x-y\rVert_2^2 eq. 4

Hyperparameters

Results

Author-reported. The paper evaluates on COCO2017 with FID-5k and CLIP score for its ablations and for the SD1.5-based comparison, and uses human preference for the state-of-the-art comparison. All numbers below are from the paper.

Ablations (single student step, COCO zero-shot FID-5k ↓ and CLIP score ↑; defaults marked) tab. 1:

Comparison with other few-step methods, all SD1.5-based, COCO FID-5k ↓ and CLIP ↑ tab. 2: the paper gives per-sample time but does not state the hardware for these times.

Human preference, SDXL-scale comparison §4.2: the ELO values below are read from the data behind the performance-versus-speed figure in §4.2, which the paper says combines the ELO scores with speed on an A100 at 512 × 512 in mixed precision. Their order matches the text.

In text, the paper reports: ADD-XL with one step beats LCM-XL with four steps; ADD-XL with four steps beats SDXL at 50 steps in the majority of comparisons; ADD-XL beats its SDXL-Base teacher on quality and prompt alignment at a cost in diversity §4.2 §4.3. The discussion adds that four steps beat SDXL, IF and OpenMUSE §5.

Paper vs code

The official repository is https://github.com/Stability-AI/generative-models at commit e8cd657656fa5d61688191730d0e03242bf4ed44, licensed MIT LICENSE-CODE. The paper links it, and its owner is the authors' organisation.

• No ADD training code. The repository contains inference and demo code for SDXL-Turbo, which its README pairs with this paper's report. A search of the code for adversarial, discriminator, hinge and distillation terms finds only an autoencoder loss, not the ADD student, discriminator or distillation loss discriminator_loss.py. Every training detail therefore comes from the paper alone.
• Sampler for 1 to 4 steps. The demo builds a sampler with 1000 steps, eta 1.0 and a legacy DDPM discretisation turbo.py. It selects the sigma indices [0, 100, 200, 300, 1000] and keeps the first n values plus the last, which gives n steps turbo.py. It starts from noise scaled by sqrt(1 + σ²) turbo.py. The paper does not describe a multi-step sampler, so these indices are only inference settings from the demo. Whether they match the paper's training timesteps is not stated.
• No guidance at inference. The demo sets the unconditional input to the conditional one turbo.py, which matches the paper's statement that no classifier-free guidance is used §1.
• Resolution. The demo fixes 512 × 512 turbo.py, as in the paper's experiments §4.

Nothing in the repo confirms the paper's ADD-M or ADD-XL training schedule.

Pitfalls

• Training recipe is incomplete. The paper gives no learning rate, optimiser, batch size, iteration count or precision for ADD-M or ADD-XL. The only schedule stated is the ablation run (4000 iterations, batch 128) tab. 1. Any reimplementation must choose these and report them.
• Timesteps are only partly specified. Only N = 4 and τ_N = 1000 are given §3.1. The other three student timesteps and the distribution of the teacher timestep t are not stated. The demo's index set is an inference hint, not a training value.
• Heads and feature layers are vague. The heads are "lightweight", and the layers that feed them are not listed §3.2. Feature network choice moves the metric a lot: the ablation shows DINOv2 ViT-S best and ViT-L worse tab. 1.
• Adversarial weight is implicit. Eq. 1 has no coefficient on the adversarial term, so it is 1 by reading; the paper does not say so.
• Weighting choice changes behaviour. The exponential weighting gives more diverse outputs, while SDS and NFSD give better quality and text alignment §4.1. The final model uses NFSD but the ablation default is exponential. The NFSD formula is not given here, so the cited NFSD work is needed to implement it.
• Do not mix SDS and the exponential weight. The SDS equivalence in the appendix needs c(t) = α_t / (2σ_t) · w(t) and squared L2 app. A. The exponential default c(t) = α_t is a different weighting.
• Pixel-space distillation has a hidden cost. For latent diffusion teachers the paper computes the distillation loss in pixel space, which it says gives more stable gradients §3.1. That means decoding the student output inside the training loop. The paper does not measure the extra cost.
• Eq. 3 typesetting. The R1 term sits inside the real-data expectation as printed. It is computed on each head's input, not on pixels, and it is only said to matter above 128 px §3.2.
• Update order is not stated. The paper does not say whether the discriminator sees detached student samples during the generator step, or how the two updates alternate.
• More teacher steps do not help clearly. The paper finds no clear gain from more than one teacher step §4.1 tab. 1.
• Diversity drops. The student takes on the teacher's traits, and the paper notes lower sample diversity for ADD-XL than for its teacher §4.3.
• Evaluation caveats. The main comparison uses human preference, which the paper prefers to FID and CLIP because those capture aesthetics poorly app. B. Baselines are the best public models, with SDXL-Base used without its refiner §4. The StyleGAN-T++ baseline is the authors' own retrained model §4.
• Licence conflict for weights. The paper is CC BY 4.0, but the released SDXL-Turbo weights in the repository are under the Stability AI Non-Commercial Research Community License LICENSE-SDXL-Turbo. The paper's licence does not extend to them. Do not use these weights in commercial work on the strength of this card.
• Name mismatch. The repository and README use the name SDXL-Turbo. The paper calls its models ADD-M and ADD-XL and does not name SDXL-Turbo. Treat any claim that they are the same model as unconfirmed.

Lineage

• Builds on: score distillation sampling (DreamFusion, Poole et al.), whose SDS loss the distillation term reduces to under a particular weight app. A; the discriminator design of StyleGAN-T, with ViT features and hinge loss §3.2; pretrained diffusion weights as student initialisation §3.1.
• Compared against: latent consistency models (LCM and LCM-LoRA), progressive distillation, guidance distillation, InstaFlow, UFOGen, DPM-Solver, GANs (StyleGAN-T++), IF, OpenMUSE and SDXL §2 tab. 2 §4.2.
• Related approaches named in the background: Denoising Diffusion GANs and Adversarial Score Matching, which add an adversarial term to diffusion training §2.