Understanding Diffusion Models: A Unified Perspective

In plain words

Diffusion models make images by starting from pure random noise and removing it step by step. Training them works the other way round: take a real image, add a little noise, then add a bit more, until the picture is only noise. The model learns to undo one small step of that corruption at a time.

This paper is a long, careful tutorial on why that recipe works. It starts from the standard tool for latent-variable models, the evidence lower bound, and shows that the noising process can be written as a stack of simple Gaussian layers. The training objective then collapses to a simple question: given a noisy picture and how noisy it is, can the model predict what was added or what the clean picture looked like? The paper also shows that the same model can be read as learning which direction makes the data more likely.

The last part covers steering the output with a condition, such as a text prompt. Two common tricks are explained, classifier guidance and classifier-free guidance, both of which trade sample variety for closer agreement with the condition. The paper is a derivation, not a benchmark, so it gives no new results to compare against.

TL;DR

• Diffusion models are presented as a special case of a Markovian hierarchical VAE with a fixed, Gaussian, variance-preserving encoder §3.
• Under those assumptions the ELBO splits into a reconstruction term, a prior term that is zero in practice, and a sum of denoising terms §3 (eq:51).
• The denoising objective can be written as predicting the clean image, the source noise, or the score; the paper shows these are equivalent up to time-dependent weights §3.
• Conditioning and guidance are explained in closed form; the paper reports no experiments.

Problem

Generative models need a tractable training objective. Maximising the exact likelihood of a latent-variable model requires integrating out the latents or knowing the true posterior, and neither is practical for complex models. The ELBO is used as a proxy; it becomes exact when the approximate posterior matches the true one §2.

Standard VAEs use a learned encoder. The paper argues that a hierarchy of latents with a learned encoder at each level is hard to optimise, and that a hierarchy whose encoder is fixed and Gaussian is easier to handle. The question it answers is how far that fixed-encoder simplification goes §3.

The paper states the limits of its own approach: sampling needs many sequential steps, latents have the same dimension as the data and are not interpretable, and the encoder cannot be learned §6.

Method

Section numbers in this card follow the source's starred headings, counted in order, since the source does not number them: §1 Introduction: Generative Models, §2 Background (ELBO, VAE, hierarchical VAE), §3 Variational Diffusion Models (including its subsections on learning the noise and on the three interpretations), §4 Score-based Generative Models, §5 Guidance, §6 Closing. Equation citations give the TeX label in parentheses, for example (eq:27), because printed equation numbers depend on compile order. The labels are in main.tex.

Notation follows the paper. \alpha_t is the per-step coefficient, and \bar\alpha_t = \prod_{i=1}^{t} \alpha_i its cumulative product. Timestep t=0 is clean data, t=T is pure noise §3.

1. Forward (noising) process. Each step is a Gaussian centred on the scaled previous state, with variance 1 - \alpha_t §3 (eq:27):

q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\left(\mathbf{x}_t;\ \sqrt{\alpha_t}\,\mathbf{x}_{t-1},\ (1-\alpha_t)\,\mathbf{I}\right)

The paper calls this variance-preserving: the coefficients are chosen so the variance stays at a similar scale §3.

2. Closed-form marginal. Repeated application of the reparameterisation trick gives the noisy sample at any t directly from \mathbf{x}_0 §3 (eq:68, eq:61):

\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\bm{\epsilon}_0, \qquad \bm{\epsilon}_0 \sim \mathcal{N}(\bm{0}, \mathbf{I})

3. Ground-truth denoising step. Bayes' rule gives a Gaussian posterior q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) with the following mean and a scalar variance §3 (eq:78, eq:79):

\bm{\mu}_q(\mathbf{x}_t, \mathbf{x}_0) = \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,\mathbf{x}_t + \frac{\sqrt{\bar\alpha_{t-1}}\,(1-\alpha_t)}{1-\bar\alpha_t}\,\mathbf{x}_0

\sigma_q^2(t) = \frac{(1-\alpha_t)(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}

4. ELBO. The paper gives two derivations. The one with the lower-variance estimator is the form used later §3 (eq:51):

\log p(\mathbf{x}_0) \geq \underbrace{\mathbb{E}_{q(\mathbf{x}_1 \mid \mathbf{x}_0)}\left[\log p_{\theta}(\mathbf{x}_0 \mid \mathbf{x}_1)\right]}_{\text{reconstruction}} - \underbrace{D_{KL}\left(q(\mathbf{x}_T \mid \mathbf{x}_0)\ \|\ p(\mathbf{x}_T)\right)}_{\text{prior matching}} - \sum_{t=2}^{T} \underbrace{\mathbb{E}_{q(\mathbf{x}_t \mid \mathbf{x}_0)}\left[D_{KL}\left( q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0)\ \|\ p_{\theta}(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\right)\right]}_{\text{denoising matching}}

The prior term has no trainable parameters and is treated as zero for large enough T §3.

5. Reverse model. The learned transition is Gaussian with the same variance as the posterior, \sigma_q^2(t) \mathbf{I}, and a mean that is parameterised through a network §3. The paper writes the mean as \bm{\mu}_\theta = \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\mathbf{x}_t + \frac{\sqrt{\bar\alpha_{t-1}}(1-\alpha_t)}{1-\bar\alpha_t}\hat{\mathbf{x}}_\theta(\mathbf{x}_t, t), where \hat{\mathbf{x}}_\theta predicts \mathbf{x}_0 §3.

6. Three equivalent parameterisations. Each one gives the same per-timestep objective up to a weight:

• Predict the clean image \mathbf{x}_0. The weighted squared error is §3 (eq:93) \frac{\bar\alpha_{t-1}(1-\alpha_t)^2}{2\sigma_q^2(t)\,(1-\bar\alpha_t)^2}\,\|\hat{\mathbf{x}}_\theta(\mathbf{x}_t, t) - \mathbf{x}_0\|^2.
• Predict the source noise \bm{\epsilon}_0. Using the reparameterisation (eq:62), the mean becomes \bm{\mu}_\theta = \frac{1}{\sqrt{\alpha_t}}\mathbf{x}_t - \frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}\sqrt{\alpha_t}}\hat{\bm{\epsilon}}_\theta(\mathbf{x}_t, t), and the weighted error is \frac{(1-\alpha_t)^2}{2\sigma_q^2(t)(1-\bar\alpha_t)\alpha_t}\,\|\bm{\epsilon}_0 - \hat{\bm{\epsilon}}_\theta(\mathbf{x}_t, t)\|^2 §3.
• Predict the score \nabla\log p(\mathbf{x}_t). Tweedie's formula links the score to \mathbf{x}_0 (eq:109), and the weighted error is \frac{(1-\alpha_t)^2}{2\sigma_q^2(t)\,\alpha_t}\,\|\mathbf{s}_\theta(\mathbf{x}_t, t) - \nabla\log p(\mathbf{x}_t)\|^2 §3 (eq:123).

The score and noise are related by \nabla\log p(\mathbf{x}_t) = -\bm{\epsilon}_0 / \sqrt{1-\bar\alpha_t} §3. The paper notes that predicting the noise has been found to work better in practice, citing other work §3.

7. Training objective used in practice. The per-timestep terms are averaged over timesteps sampled uniformly from \{2,\dots,T\} §3 (eq:94). The paper's closing statement is that the model is trained by sampling timesteps and minimising the norm of the prediction error against its target §3. See Pitfalls for what is dropped.

8. Learning the noise schedule. Instead of fixing \alpha_t, the paper parameterises the signal-to-noise ratio \mathrm{SNR}(t) = \bar\alpha_t / (1-\bar\alpha_t) §3 (eq:108) by a monotonically increasing network \omega_\eta(t) through \mathrm{SNR}(t) = \exp(-\omega_\eta(t)) §3 (eq:105). This gives \bar\alpha_t = \mathrm{sigmoid}(-\omega_\eta(t)) and 1-\bar\alpha_t = \mathrm{sigmoid}(\omega_\eta(t)) §3. The per-timestep objective is then written in terms of the SNR difference §3 (eq:104).

9. Score-based view. Energy-based densities p(\mathbf{x}) \propto e^{-f(\mathbf{x})} are learned through their score \nabla\log p(\mathbf{x}), which removes the normalising constant §4. Samples come from Langevin dynamics, which moves \mathbf{x} along the score plus Gaussian noise §4. Training with several noise levels fixes three problems with plain score matching (the manifold issue, low density regions, and mixing between modes) §4. Annealed Langevin runs over decreasing noise levels; the paper calls this the same procedure as the reverse chain of the VDM §4. The paper says the infinite-step limit becomes a stochastic differential equation whose reverse is used for sampling, citing other work for the details §4.

10. Conditional models and guidance. Conditioning is added to every transition, p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t, y) §5. The two guidance forms are:

• Classifier guidance adds the classifier gradient, scaled by \gamma, to the unconditional score §5 (eq:150): \nabla\log p(\mathbf{x}_t \mid y) = \nabla\log p(\mathbf{x}_t) + \gamma\,\nabla\log p(y \mid \mathbf{x}_t).
• Classifier-free guidance uses one conditional network with the condition dropped sometimes. Its score is §5: \gamma\,\nabla\log p(\mathbf{x}_t \mid y) + (1-\gamma)\,\nabla\log p(\mathbf{x}_t).

# Classifier-free guidance combination  [§5]
# Input:  conditional and unconditional scores (B, C, H, W); weight gamma
# Output: guided score (B, C, H, W)
def guided_score(score_cond, score_uncond, gamma):
    # gamma = 1 is the plain conditional model; gamma > 1 pushes away
    # from the unconditional score  [§5]
    return gamma * score_cond + (1 - gamma) * score_uncond

Training step and sampler (derived from the method above)

The paper does not give a single algorithm box. The two procedures below are assembled from the equations cited above; the variance and the step at t=1 are the parts the paper leaves open (see Pitfalls).

# Noise-prediction training step  [§3]
# Input:  clean batch x0 (B, C, H, W); fixed alpha_bar (T,)
# Output: scalar loss
def training_step(x0, alpha_bar):
    # One timestep per example, t in {2..T}  [§3] (eq:94)
    t = sample_uniform_integers(2, T, size=B)
    # Source noise eps_0 ~ N(0, I)  [§3] (eq:68)
    eps = sample_standard_normal(x0.shape)
    # Closed-form corruption of the clean batch  [§3] (eq:68)
    x_t = (sqrt(alpha_bar[t]) * x0
           + sqrt(1 - alpha_bar[t]) * eps)
    # Predict the noise from noisy input and timestep  [§3]
    eps_hat = network(x_t, t)
    # Unweighted squared error, as the closing step suggests  [§3]
    return mean_squared_error(eps_hat, eps)

# Ancestral sampler over the learned reverse chain  [§3]
# Input:  network; alpha (T,), alpha_bar (T,), sigma_q_sq (T,)
# Output: sample x0 (B, C, H, W)
def sample(network, shape):
    # Start from the Gaussian prior p(x_T)  [§3] (eq:36)
    x = sample_standard_normal(shape)
    # Walk the chain from t = T down to t = 1  [§3]
    for t in range(T, 0, -1):
        # Predict the noise for the current state  [§3]
        eps_hat = network(x, t)
        # Mean of p_theta(x_{t-1} | x_t)  [§3]
        mean = (x - (1 - alpha[t]) / sqrt(1 - alpha_bar[t]) * eps_hat)
        mean = mean / sqrt(alpha[t])
        # Variance sigma_q^2(t) noise, but none at t = 1  [§3] (eq:79)
        noise = sample_standard_normal(shape) if t > 1 else zeros(shape)
        x = mean + sqrt(sigma_q_sq[t]) * noise
    return x

Hyperparameters

The paper is a derivation and reports no training runs, so most values are not stated. Only the free parameters that the method introduces are listed.

Results

Not applicable. The paper is a tutorial and derivation. It reports no experiments, datasets, metrics or baselines. Its statements that diffusion models power text-to-image systems such as Imagen and DALL-E 2 are background claims about other work §6 §5, not results of this paper.

Paper vs code

No official code.

The paper has no code link in its abstract or TeX source. A GitHub repository search for "diffusion models unified perspective calvin" and "calvinluo diffusion" returned no repositories. The paper is an expository derivation, so no implementation accompanies it.

Pitfalls

• Unnumbered sections. Sections use starred headings, so the source has no section numbers. This card numbers them in source order (see Method). Printed equation numbers were not checked; use the TeX labels.
• Derived weights are dropped. The three per-timestep objectives carry weights (see Method step 6). The closing statement only says the model is trained by minimising the norm of the prediction error §3. Whether to keep the derived weights or use the plain unweighted loss is left to the implementer. The paper does not say which one it means in practice.
• Tweedie step is informal. The formula is applied to \mathbf{x}_t with its noise-free mean \sqrt{\bar\alpha_t}\mathbf{x}_0 and read as an equality §3 (eq:109). Tweedie's formula gives a posterior mean in general; the paper takes the pointwise identity as given.
• Sign of the score. The relation \nabla\log p(\mathbf{x}_t) = -\bm{\epsilon}_0 / \sqrt{1-\bar\alpha_t} has a minus sign. Mixing up the sign of the score target or of the noise target reverses the update direction.
• Reverse variance is fixed. The reverse step's variance is set to \sigma_q^2(t) from the posterior and is not learned §3 (eq:79). Only the mean is learned.
• The final sampling step is undefined. The paper never states how the chain ends at t=1. The product \bar\alpha_0 is not defined in the text; an empty product would give 1, which makes \sigma_q^2(1) = 0. The sampler above drops the noise at t=1, which is inferred, not stated.
• Training timesteps. The denoising sum runs over t \ge 2 §3 (eq:51), and the t=1 term is the reconstruction term. The training sampler §3 (eq:94) uses \{2,\dots,T\}. How the reconstruction term enters the training loss is not given.
• Monotonicity is assumed. The learned SNR must decrease in t. The text requires a monotonically increasing \omega_\eta, but gives no architecture that enforces it §3 (eq:105).
• Langevin schedule is missing. Annealed Langevin "reduces the step size over time", but no schedule, number of steps per level, or noise-level values are given §4.
• Guidance convention. The paper's \gamma multiplies the conditional score directly, so \gamma = 1 is the plain conditional model §5 (eq:150). The paper gives no mapping from this \gamma to other guidance weight conventions.
• The "adversarial gradient" name. The paper calls \nabla\log p(y \mid \mathbf{x}_t) an adversarial gradient §5 (eq:148). It is the gradient of a classifier that must work on noisy inputs. It has no link to GAN training.
• Unconditional branch. Classifier-free training needs the condition replaced by a fixed value "such as zeros" §5. The dropout probability and the exact null value are not stated.
• Costs the paper says it has. Sampling needs T sequential denoising steps. The training sum over timesteps dominates the cost §3 §6.

Lineage

• Builds on: the VAE and the hierarchical VAE, described in the Background section §2 and cited in the source by bib key only (not linked here). The Markovian hierarchical VAE is the starting structure §2.
• Builds on: score matching, Langevin dynamics and annealed Langevin sampling, presented in the Score-based section with their own citations §4.
• Compared against: classifier guidance and classifier-free guidance, both explained in the Guidance section §5.