State of the Art on Diffusion Models for Visual Computing

In plain words

Diffusion models make pictures by starting from static, like a TV with no signal, and removing the noise step by step until an image appears. A network is trained to predict the noise in a slightly noisy picture, and running that prediction in reverse turns random noise into a new picture. A text prompt can steer the process toward the image it describes.

This report is a guide to the field as it stood in 2023. It first explains the mathematics behind these models in plain terms, then shows how the popular Stable Diffusion model works: it does the noise removal on a compressed version of the image, not on every pixel. Next it covers how to steer the output with text or other inputs, and how to edit a real photo or teach the model a new subject. The second half applies the same ideas beyond flat images, to video, 3D objects, 3D scenes and scenes that change over time, and then discusses the data, the ways to measure quality, the open problems and the social concerns.

It does not train or test any model itself. Where it gives numbers, they come from the works it cites.

TL;DR

• A state-of-the-art report, not a method: it explains diffusion models for visual computing and surveys the 2021–2023 literature §1 §2.
• Core framework: forward noising and reverse denoising as SDEs and a probability-flow ODE, sampled numerically §3.1 alg. 1.
• Stable Diffusion runs diffusion in an autoencoder's latent space with cross-attention conditioning §3.2 §3.3 fig. 2.
• It runs no experiments and has no results table of its own. Use it as a map and a reading list; take training recipes and hyperparameters from the original papers it cites, since it gives almost none.

Problem

Diffusion models are slow at inference because they run a large network many times, and training them on high-resolution or high-dimensional data (video, 3D, 4D) is expensive. Extending image models to these domains is not obvious, and large 3D and video training sets are scarce. The report also notes that the field is growing faster than anyone can track §1.

Method

The report has no single method. Its content is a framework plus a survey, organised as follows.

Framework (§3.1). Data are noised by a stochastic differential equation (SDE) that adds Gaussian noise over time. The paper picks the variance-exploding form, in which the noise level σ(t) grows with t. A denoiser that minimises a squared error gives the score (the gradient of the log-density) through D(x; σ). Sampling runs the reverse-time SDE, approximated numerically with the Euler–Maruyama scheme alg. 1. A deterministic probability-flow ODE gives the same marginals and allows encoding an image into noise and back §3.1.

Stable Diffusion, latent diffusion (§3.2). An encoder compresses the image into a smaller latent, a U-Net denoises latents, and a decoder maps the result back to pixels. The U-Net contains self-attention and cross-attention blocks. The report names VQ-GAN as the most common choice of compression model, and cites a retrieval-augmented variant §3.2 fig. 2.

Conditioning and guidance (§3.3). Conditions are injected by concatenation or by cross-attention, where queries come from the U-Net and keys and values from an encoded condition eq. 8. Classifier guidance adds the gradient of a classifier's log-likelihood eq. 10. Classifier-free guidance trains one network with the condition sometimes dropped and combines the two predictions eq. 11. ControlNet adds trainable copies of encoder blocks linked to the frozen model through zero-initialised convolutions §3.3 fig. 3.

Editing, inversion and customisation (§3.4). Editing methods include SDEdit (partial noising, then denoising with a new condition), prompt-based attention control, instruction-based fine-tuning and drag-style edits. Inversion finds a latent that reproduces a given real image, for example by DDIM inversion or textual inversion. Customisation adapts a model to one subject, for example DreamBooth, LoRA-based fine-tuning and encoder-based methods §3.4.

Beyond images (§4 to §7). The report covers video generation and editing (§5), 3D object and scene generation with diffusion directly or through 2D priors (§6), and 4D spatio-temporal diffusion, including articulated objects and human motion (§7). Each part compares methods narratively; the report gives no uniform comparison table for them.

Data, metrics and open problems (§8 to §11). Dataset families, quality metrics, open challenges and social concerns are surveyed in §8–§11.

Key equations, in the report's notation. Equation numbers are counted in the LaTeX source order; see Pitfalls.

Forward SDE with drift f and diffusion g:

dx = f(x, t)\,dt + g(t)\,dw eq. 1

Variance-exploding forward process, with the closed form it implies:

dx = \sqrt{2\,\sigma(t)\,\sigma'(t)}\;dw eq. 2

p(x_t \mid x_0) = \mathcal{N}\big(x_t;\, x_0,\, [\sigma^2(t) - \sigma^2(0)]\,I\big) eq. 3

Reverse-time SDE, which needs the score:

dx = -2\,\sigma\sigma' \,\nabla_x \log p(x; \sigma(t))\,dt + \sqrt{2\,\sigma\sigma'}\;dw eq. 4

Probability-flow ODE, deterministic, same marginals:

dx = \big[f(x,t) - \tfrac{1}{2} g(t)^2 \nabla_x \log p(x;\sigma(t))\big]\,dt eq. 5

Cross-attention, with queries from the U-Net features φ(z_t) and keys and values from the encoded condition τ(c):

Q = W_Q\,\varphi(z_t), \quad K = W_K\,\tau(c), \quad V = W_V\,\tau(c) eq. 8

Classifier guidance, with guidance weight w and noise scale σ_t:

\tilde\epsilon_\theta(x_t, c) = \epsilon_\theta(x_t, c) - w\,\sigma_t\,\nabla_{x_t} \log p_\phi(c \mid x_t) eq. 10

Classifier-free guidance, combining the conditional and the null-condition predictions:

\hat\epsilon_\theta(x_t) = (1+w)\,\epsilon_\theta(x_t, c) - w\,\epsilon_\theta(x_t, \varnothing) eq. 11

Euler–Maruyama approximation of an SDE dx = a dt + b dw alg. 1:

# Euler–Maruyama sampler for dx = a(x, t) dt + b(x, t) dw  [alg. 1]
# Input:  drift a, diffusion b, x_0, time span [0, T], N steps
# Output: the trajectory x_0 ... x_N (see Pitfalls on the indexing)

def simulate(drift, diffusion, x_0, T, N):
    # Equal steps, delta t = T / N  [alg. 1]
    dt = T / N
    trajectory = [x_0]
    for n in range(N):
        # Wiener increment over one step, dw_n = w(t_{n+1}) - w(t_n)  [alg. 1]
        dw = sample_wiener_increment(dt)
        # Euler step along the drift and the noise  [alg. 1]
        x = x + drift(x, n * dt) * dt + diffusion(x, n * dt) * dw
        trajectory.append(x)
    return trajectory

Classifier-free guidance at one denoising step, following eq. 11:

# Classifier-free guidance for one denoising step  [eq. 11]
# Inputs: noisy latent z_t (B, C, H, W); condition c; weight w
# Output: guided noise estimate (B, C, H, W)

def guided_noise(model, z_t, t, condition, w):
    # Prediction with the real condition  [§3.3]
    eps_cond = model(z_t, t, condition)
    # Prediction with the null condition, trained by random dropping  [§3.3]
    eps_uncond = model(z_t, t, null_token)
    # Extrapolate away from the unconditional prediction  [eq. 11]
    return (1 + w) * eps_cond - w * eps_uncond

Hyperparameters

The report gives almost no training or sampling settings. The values below are the few it states.

Results

The report has no experiments. Its quantitative content is dataset statistics and one training-cost figure, both quoted from other sources. Dataset numbers are as the report gives them.

Paper vs code

No official code. The source links no code repository or project page in its active text; the only links are in commented-out lines and point to datasets and third-party pages. A GitHub repository search on the title words with the Eurographics organisation returned nothing. A survey has no implementation of its own.

Pitfalls

• No experiments. Nothing here is a benchmark result. Numbers in the Results table are inputs quoted from other sources, with the caveats the report attaches to them.
• Algorithm 1 indexing. The output line lists the trajectory as x̂_0 … x̂_{N−1}, but the loop computes x̂_{n+1} for n = 0 … N−1, so it produces N+1 states including x̂_N. Keep x̂_N as the final sample; the output line in the source does not say so.
• Guidance weight conventions differ between sections. Classifier-free guidance uses (1 + w) on the conditional term eq. 11. Classifier guidance uses the sign −w σ_t ∇ log p eq. 10. The report does not reconcile these with the weight scales used by public Stable Diffusion code, so do not mix conventions. The text's w = −1 example for empty output is the unconditional limit of eq. 11. The commented-out draft of this passage used w = 0 for the same behaviour, so the source's versions disagree; this card follows the active text.
• Noise-predictor relation. The report states ε = x − D(x; σ) and a score of (D(x; σ) − x)/σ². These are written for the variance-exploding form with no scale factor. My inference: a DDPM-style checkpoint, which parametrises noise with a scale factor, needs its own conversion. The report does not say this.
• Only one SDE family is written out. The derivations are for the variance-exploding form eq. 2. Other families are not written out, so implementers should use the cited original for them.
• Attention notation is loose. The report defines Attention(Q, K) as a softmax of QKᵀ/√d and then writes the block output as A·V, but it does not name the head count or the per-head size, and it does not write out the output projection.
• Equation and figure numbers are counted from the LaTeX source, in the order they appear, so numbered equations run eq. 1 to eq. 11. The teaser appears unnumbered in the source, so the figures cited here (fig. 2, fig. 3) are counted from the first figure in §3. Check both against the compiled PDF before relying on them.
• Dates. The report covers papers from 2021 to 2023 §2, so its coverage stops there. It does not claim completeness.
• Secondary sources. The Stable Diffusion training cost is attributed to a secondary source and not verified by the report §11. The report also marks an external estimate of another model's cost as unverified.
• Tables not summarised here. The report includes tables on 3D direct generation, compositional generation, novel-view synthesis and relighting (in the source directory tables/). They are not summarised on this card. Read the source for them.
• Figures. Several figures are reproduced from other papers, as their captions state. This card uses no figures.

Lineage

• Builds on: Stable Diffusion / latent diffusion (Rombach et al., cited in §3.2; arXiv id not given in the source); score-based SDE framework (Song et al., arXiv:2011.13456 §3.1); DDIM sampling and inversion (Song et al., arXiv:2010.02502 §3.4); classifier guidance (Dhariwal and Nichol, arXiv:2105.05233 §3.3); classifier-free guidance (Ho and Salimans, arXiv:2207.12598 §3.3); ControlNet (Zhang et al., arXiv:2302.05543 §3.3).
• Compared against: GANs and other generative models, which the report excludes from scope §2.
• Superseded by: not stated in the source.