Common Diffusion Noise Schedules and Sample Steps are Flawed

In plain words

Image generators such as Stable Diffusion learn by mixing pictures with noise and then learning to undo the mixing. A schedule sets how much picture is kept at each step. The paper finds that the final step of common schedules still keeps a faint trace of the picture, most visibly its overall brightness. The model learns to rely on that trace. When an image is generated, though, the model starts from pure random noise, which has no such trace. This mismatch pulls the output toward medium brightness, so the model struggles to make very dark or very bright images, even for prompts like "solid black background".

The authors propose four changes that make training and generation match. Rescale the schedule so the last step is pure noise. Train the model to predict a combined target instead of the noise itself. Always start sampling from that last step. Finally, rescale the guidance that steers the image toward the prompt, so strong guidance does not overexpose the picture. They report that these changes let the model produce a wider range of brightness.

TL;DR

• Linear, cosine and Stable Diffusion schedules do not reach zero signal-to-noise ratio (SNR) at the last step, and some samplers skip that step §1 tab. 1 tab. 2.
• The leftover signal makes training inputs differ from the pure noise given at inference, which limits Stable Diffusion to medium-brightness images §1 §3.1 fig. 1.
• Four fixes: rescale the schedule to zero terminal SNR, train with v prediction, sample from the last timestep, and rescale classifier-free guidance §3.1–§3.4.
• Fine-tuning SD 2.1-base with the fixes gives FID 21.66 and IS 36.16 on 10k COCO captions, against 23.76 and 32.84 for the official model tab. 3. Author-reported.
• No official code was found [see Paper vs code].

Problem

• Stable Diffusion, up to version 2.1 at the time of writing, produces images whose mean brightness stays near the middle of the −1 to 1 scale, even for prompts that ask for much brighter or darker results. It also fails on simple prompts such as "Solid black color" or "A white background" §1 fig. 1.
• At the final timestep T = 1000, the terminal SNR of common schedules is not zero: linear 4.035993e-05, cosine 2.428735e-09, Stable Diffusion 0.004682 tab. 1.
• The stated cause: during training the last step still leaks low-frequency content, such as each channel's mean, which the model learns to read. At inference the model receives pure Gaussian noise, whose mean is zero, so it keeps the training-time mean and stays at medium brightness §1 §3.1.
• Some sampler implementations do not start from the last timestep. The paper names the official DDIM and PNDM implementations §1 §3.3 tab. 2.
• Offset noise, an earlier workaround, is described by the authors as a trick that does not address the cause §5.4.
• The argument is made for the variance-preserving formulation. The paper argues variance-exploding models cannot reach zero terminal SNR §3.1. It says the flaw is general to diffusion models, not only Stable Diffusion §1 §3.1.

Method

The four fixes aim to make the training input at the last step match the pure noise given at inference. The text and the paper's listings are both given below in our own words.

Notation and forward process §2

With \alpha_t = 1 - \beta_t and \bar\alpha_t = \prod_{s=1}^{t} \alpha_s, the forward process has the closed form below.

q(x_{1:T}|x_0) = \prod_{t=1}^{T} q(x_t|x_{t-1}) eq. 1

q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\, x_{t-1}, \beta_t \mathbf{I}) eq. 2

q(x_t|x_0) = \mathcal{N}(x_t; \sqrt{\bar\alpha_t}\, x_0, (1-\bar\alpha_t)\mathbf{I}) eq. 3

x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) eq. 4

\mathrm{SNR}(t) = \frac{\bar\alpha_t}{1-\bar\alpha_t} eq. 5

p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1}|x_t) eq. 6

p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \tilde\mu_t, \tilde\beta_t \mathbf{I}) eq. 7

\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}} \Big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\, \epsilon\Big) eq. 8

\tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\, \beta_t eq. 9

The paper's Background section says the reverse step is Gaussian when \beta_t is small, and that models are commonly reparameterized to predict \epsilon §2.

Step 1: enforce zero terminal SNR §3.1

1. Keep \sqrt{\bar\alpha_1} unchanged, set \sqrt{\bar\alpha_T} to zero, and rescale the values for t in 2..T-1 linearly. The rescaling is done in \sqrt{\bar\alpha} space, which the authors find preserves the curve better than scaling SNR §3.1.
2. The paper's example shows the original Stable Diffusion schedule at the last step, where signal survives: \sqrt{\bar\alpha_T} is 0.068265 tab. 1 eq. 10 §3.1.

x_T = 0.068265\, x_0 + 0.997667\, \epsilon eq. 10 §3.1

3. Procedure from the paper's listing alg. 1:

# Rescale a noise schedule to zero terminal SNR  [alg. 1] [§3.1]
# Input:  betas (T,), where betas[0] is the first step t = 1
# Output: betas (T,) with alpha_bar at the last step equal to 0

def enforce_zero_terminal_snr(betas):
    # Work in sqrt(alpha_bar) space  [§3.1] [eq. 4]
    alphas = 1 - betas
    root = cumprod(alphas).sqrt()
    # Keep the first value; the last one must become 0  [alg. 1]
    head = root[0]
    tail = root[-1]
    # Shift so the tail is 0, then scale so the head is unchanged  [alg. 1]
    root = (root - tail) * head / (head - tail)
    # Back to alpha_bar, then to per-step alphas  [alg. 1]
    cum = root * root
    step_alphas = cum[1:] / cum[:-1]
    step_alphas = concat([cum[0], step_alphas])
    return 1 - step_alphas

4. For cosine schedules, the paper says removing the clipping of \beta_t at 0.999 is enough §3.1. Future schedules should set \beta_T = 1 §3.1.

Step 2: train with v prediction §3.2

1. Use the v target below in place of \epsilon §3.2 eq. 11.
2. The loss weight is \lambda_t §3.2 eq. 12.
3. At t = T, with \bar\alpha_T = 0, the target becomes v_T = x_0. The model then receives pure noise and must predict the conditional data mean, so the paper says it is repurposed at that step §3.2.

v_t = \sqrt{\bar\alpha_t}\, \epsilon - \sqrt{1-\bar\alpha_t}\, x_0 eq. 11 §3.2

\mathcal{L} = \lambda_t \, \lVert v_t - \tilde v_t \rVert_2^2 eq. 12 §3.2

Step 3: sample from the last timestep §3.3

1. Include t = T in the sample steps. The paper's table of selection rules is reproduced below, with T = 1000 and S = 10 for illustration tab. 2.
2. The paper picks trailing steps, which start at T and go down by T/S, and uses DDIM as the sampler §3.3.

Trailing procedure, from the paper's rule tab. 2 §3.3:

# Trailing sample steps  [tab. 2] [§3.3]
# Input:  T (total timesteps, 1000), S (number of sample steps)
# Output: S timesteps in ascending order, the largest equal to T

def trailing_steps(T, S):
    # Count down from T in steps of T / S, so T is always included  [tab. 2]
    steps = [round(T - k * T / S) for k in range(S)]
    # Ascending order for the sampling loop  [tab. 2]
    return sorted(steps)

Step 4: rescale classifier-free guidance §3.4

1. Standard guidance combines a positive-prompt output and a negative-prompt output with weight w §3.4 eq. 13.
2. Compute the standard deviation of the positive output and of the guided output §3.4 eq. 14.
3. Rescale the guided output so its standard deviation matches the positive output §3.4 eq. 15.
4. Blend the rescaled and the plain guided output with factor \phi §3.4 eq. 16.

x_{cfg} = x_{neg} + w\,(x_{pos} - x_{neg}) eq. 13 §3.4

\sigma_{pos} = \mathrm{std}(x_{pos}), \quad \sigma_{cfg} = \mathrm{std}(x_{cfg}) eq. 14 §3.4

x_{rescaled} = x_{cfg} \cdot \frac{\sigma_{pos}}{\sigma_{cfg}} eq. 15 §3.4

x_{final} = \phi\, x_{rescaled} + (1-\phi)\, x_{cfg} eq. 16 §3.4

The paper's listing alg. 2 computes the same thing, which is x_{cfg} times \phi\,\sigma_{pos}/\sigma_{cfg} + (1-\phi):

# Classifier-free guidance with a standard-deviation rescale  [alg. 2] [§3.4]
# Input:  pos, neg with shape (B, C, H, W) assumed; weight w; rescale phi
# Output: guided output, same shape as pos

def apply_cfg(pos, neg, weight=7.5, rescale=0.7):
    # Standard guidance  [eq. 13] [alg. 2]
    guided = neg + weight * (pos - neg)
    # Per-sample standard deviations over dimensions 1..3  [eq. 14] [alg. 2]
    pos_spread = std(pos, dims=(1, 2, 3))
    guided_spread = std(guided, dims=(1, 2, 3))
    # Blend the ratio with 1 using phi  [eq. 15] [eq. 16]
    ratio = pos_spread / guided_spread
    blend = rescale * ratio + (1 - rescale)
    return guided * blend

Step 5: sampler implementation §6

1. Do not convert v to \epsilon and then use the \epsilon update. At t = T that conversion loses all signal, and \alpha_T = 0 causes a division by zero §6 eq. 17 eq. 18.
2. Convert v to x_0 instead, then use the x_0 form of the DDPM posterior mean §6 eq. 19 eq. 20.

\epsilon = \sqrt{\bar\alpha_t}\, v + \sqrt{1-\bar\alpha_t}\, x_t eq. 17 §6

\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}} \Big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\, \epsilon\Big) eq. 18 §6 (same form as eq. 8)

x_0 = \sqrt{\bar\alpha_t}\, x_t - \sqrt{1-\bar\alpha_t}\, v eq. 19 §6

\tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\, x_0 + \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\, x_t eq. 20 §6

3. For DDIM, convert v to \epsilon and x_0 with eqs. 17 and 19, then step with eq. 21. The paper states z \sim \mathcal{N}(0, \mathbf{I}) and \eta \in [0, 1] §6 eq. 21 eq. 22.

x_{t-1} = \sqrt{\bar\alpha_{t-1}}\, x_0 + \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\, \epsilon + \sigma_t z eq. 21 §6

\sigma_t(\eta) = \eta \sqrt{\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}} \sqrt{1-\frac{\bar\alpha_t}{\bar\alpha_{t-1}}} eq. 22 §6

The paper says the same logic applies to other samplers §6.

The paper's listings in own words:

# Convert v prediction to x0 without going through eps  [§6] [eq. 19]
# Valid at t = T, where alpha_bar_t = 0  [§6]

def v_to_x0(x_t, v, alpha_bar_t):
    return sqrt(alpha_bar_t) * x_t - sqrt(1 - alpha_bar_t) * v

Hyperparameters

Results

Author-reported. Setup: Stable Diffusion 2.1-base; 10k random COCO 2014 validation captions; DDIM with 50 steps; guidance weight 7.5; no negative prompt. "Fixes" are the zero terminal SNR schedule, v prediction, trailing steps and φ = 0.7 tab. 3 §4.2. "Our data" means the authors' LAION subset §4.

Qualitative results only for brightness and prompt following. fig. 3 shows eight prompt pairs, each with the reference Stable Diffusion model on the left and the fixed model on the right, using the same seed. The paper says the flawed model always gives medium brightness and cannot produce prompts such as "Solid black background" or "White background", while the fixed model follows them §4.1 fig. 3. The paper reports no numeric brightness metric.

The sample-step comparison fig. 4 and the rescale-factor comparison fig. 6 are visual. The paper says trailing is noticeably better than linspace at S = 5, and the difference is subtle at S = 25 §5.1 fig. 4. It says φ between 0.5 and 0.75 gives the most appealing results, and φ = 0 overexposes §5.3 fig. 6.

Paper vs code

No official code.

• The arXiv abstract, the paper body and the metadata record contain no code link. The only link is the footnote to a Hugging Face model card, ByteDance/sd2.1-base-zsnr-laionaes5 p. 1. That card is a model page and names no code repository. Model weights are not used here.
• Searches (GitHub API, repositories and users, via WebFetch): "zero terminal snr" (2 results, both third-party personal projects with no license, not used), "zsnr bytedance", "common diffusion noise schedules", "rescale classifier free guidance v prediction terminal", "zero snr org:bytedance", "sd2.1 zsnr", and the first author's name with bytedance (no users found). None found an author-owned repository.

The paper's own listings were checked against its text:

• Algorithm 1 alg. 1 matches the text in §3.1: the first value is kept, the last is set to 0, and the rescale is applied in \sqrt{\bar\alpha} space.
• Algorithm 2 alg. 2 matches eqs. 13–16 §3.4.
• Equation 18 is the same as equation 8 eq. 8 eq. 18.
• The listings are the paper's own code. They use PyTorch, but no training or sampling loop is given. The card cannot verify them against a running implementation.

Pitfalls

• Rescale index convention. The text uses t = 1..T and keeps \sqrt{\bar\alpha_1} §3.1. The listing uses index 0 as the first step alg. 1. Table 2 uses 1..T, while most implementations use 0..T−1 tab. 2. A shift error moves the kept value by one step.
• Zero division in samplers. Equations 8 and 18 divide by \sqrt{\alpha_t}. At t = T with \beta_T = 1, \alpha_T = 0. Use the x_0 form, eqs. 19–20 §6.
• Eps conversion at t = T loses signal. The paper says so directly §6. Do not use the eps conversion as a shortcut.
• Sample steps must include T. The leading rule used by DDIM and PNDM omits T for T = 1000, S = 10. Its last step is 901 tab. 2. Trailing includes it §3.3.
• Std axes are not stated in the text. Equation 14 says only "std" §3.4. The listing takes the std per sample over dimensions 1–3 alg. 2. A tensor with another layout would need the same choice. Whether the std is the biased or unbiased estimator is not stated.
• Parameterization of the guided outputs is not stated. Eq. 13 uses x_{pos} and x_{neg} as "model outputs" §3.4. The paper does not say whether these are \epsilon, v or x_0. Its fine-tune uses v prediction §3.2.
• w and \phi are chosen empirically. The text says the values "work great" §3.4. The ablation is visual, on a few prompts fig. 6 §5.3.
• The FID comparison mixes two changes. The "official" row uses SD's own training data, while "ours" uses the authors' filtered LAION subset tab. 3 §4. The gain from the fixes on the same data is FID 22.96 → 21.66 and IS 34.11 → 36.16 (differences from tab. 3).
• Single run. The paper reports no seeds, error bars, or repeated runs §4.2. The FID is on 10k samples.
• Missing training details. Optimizer, LR schedule, precision, resolution, CFG dropout, dataset filter rules and hardware are not stated §4.
• Cosine schedule. The paper says the \beta_t clip at 0.999 can simply be removed to reach zero terminal SNR §3.1. The paper does not show the resulting schedule's values.
• Notation typo. Eqs. 3 and 4 write \bar a_t where the definition is \bar\alpha_t §2 eq. 3 eq. 4.
• Offset noise is only compared qualitatively. The paper argues against it §5.4 without a metric.
• One-prompt analysis. The claim that the model's output at t = T is nearly the same for every noise input is shown with one prompt §5.2 fig. 5.

Lineage

• Builds on: diffusion models (Sohl-Dickstein et al. 2015; Ho et al. 2020) §2. Linear and cosine schedules (Ho et al. 2020; Nichol and Dhariwal 2021) §1 tab. 1. DDIM and PNDM sampling conventions §1 tab. 2. Trailing steps from DPM-Solver §3.3. v prediction (Salimans and Ho 2022) §3.2. Classifier-free guidance (Ho and Salimans 2022) §3.4. Dynamic thresholding from Imagen (Saharia et al. 2022) §3.4. Variance-exploding formulation (Song et al. 2021) is discussed as a non-option §3.1.
• Compared against: SD v2.1-base official tab. 3; offset noise §5.4; the original leading, linspace and trailing step rules tab. 2.
• Superseded by: not known from content already in this wiki.