Noise schedule

Overview

A diffusion model corrupts data with Gaussian noise over a time variable t and learns to reverse the corruption. The noise schedule is the function that says how much signal and how much noise are present at each t. In the variance-preserving form the paper uses, the noisy state is alpha_t x + sigma_t eps with alpha_t^2 + sigma_t^2 = 1, and the schedule is the signal-to-noise ratio SNR(t) = alpha_t^2 / sigma_t^2, which decreases in t 2107.00630 §3.1 2107.00630 eq. 1 2107.00630 eq. 2. Writing the schedule as the log-SNR, gamma(t) = -log SNR(t), gives a function that increases in t and fixes both noise factors 2107.00630 §3.2 2107.00630 eq. 3 2107.00630 eq. 4.

The shared invariant: in continuous time, the training loss depends on the schedule only through its two endpoint SNRs, SNR(0) and SNR(1). Two schedules with the same endpoints give the same p(x) once the denoiser's input is rescaled 2107.00630 §5.1 2107.00630 eq. 18 2107.00630 app. G. The interior of the schedule therefore changes the estimator's variance, not the bound it targets 2107.00630 §5.3. This invariance concerns the bound. It does not say that sample quality is insensitive to the interior: LangFlow reports a large change in generative perplexity from its interior schedule 2604.11748 §4.1.

The tutorial by Luo uses the same two-sided form. It writes the SNR as exp(-omega(t)) with a monotonically increasing network omega, so that alpha_bar_t = sigmoid(-omega(t)) and 1 - alpha_bar_t = sigmoid(omega(t)) 2208.11970 §3. It keeps the per-step coefficient fixed or learnable and gives no values for either 2208.11970 §3.

Changing the loss weight over the same SNR axis is a separate lever, not a change of schedule. It is covered under its own variant below.

Variants

Learned monotone schedule

The paper's method. gamma(t) is a small network with positive weights, so it is monotone by construction. It is trained jointly with the denoiser on the variational bound 2107.00630 §3.2 2107.00630 app. H. The continuous-time loss then weights the squared noise-prediction error by the schedule's derivative, 1/2 gamma'(t) 2107.00630 eq. 17. Card: Variational Diffusion Models.

The paper also trains the interior of the schedule to reduce the variance of the loss estimator, with the endpoints still trained on the bound 2107.00630 §5.3 2107.00630 app. I.2. The released code does not contain that step, and trains the schedule on the plain bound gradient 2107.00630 §5.3. Treat the variance result as reproducible from the paper only if you implement it yourself.

The tutorial describes the same monotone-network parameterisation, trained with the noise-prediction error 2208.11970 §3. It states that the network must be monotone but gives no architecture that enforces this, whereas the variational paper builds monotonicity in with positive weights 2107.00630 §3.2. It reports no experiments, so it gives no bits per dimension or variance figures to compare against. Card: Understanding Diffusion Models: A Unified Perspective.

Learned endpoints, linear in log-SNR

The released configs for CIFAR-10 without augmentation and for both ImageNet sizes use a simpler learned form. gamma(t) = b + |w| t, with two learned scalars, starting from gamma_min = -13.3 and gamma_max = 5 cifar10.py model_vdm.py. The paper describes the monotone network as its method, so this form is a code-level choice rather than a result the paper reports 2107.00630 §3.2. The augmented CIFAR-10 config uses the monotone network cifar10_aug.py.

Learned Gumbel-shaped schedule over log-SNR

Used for text diffusion on continuous token embeddings. The motivation is that a uniform schedule over the usual time variable spends more than half of its steps at noise levels where the loss is nearly zero 2604.11748 §4.1. The schedule instead models the per-level cross-entropy as a Gumbel-shaped curve, H_inf exp(-exp(-(gamma - P_mu) / P_beta)), with three learned parameters. Those parameters are fitted to the stop-gradient cross-entropy, so the denoiser is not affected by the fit. Training levels are drawn from the inverse CDF of the same Gumbel distribution, clipped to the 1e-5 quantiles 2604.11748 §4.1 2604.11748 alg. 1. The paper reports that Gumbel scheduling lowers LangFlow's generative perplexity on LM1B from about 1000 to 154.2 2604.11748 §4.1. The training code is not released, so the fitting loss has to be implemented from the algorithm box. The repository's default parameter values are initial values, not the trained ones config.py. Card: LangFlow.

Truncated and adaptive schedule for flow language models

Used for a continuous flow language model whose tokens are unit-norm embeddings on a hypersphere. Two changes sit on the schedule. The first trains only on a truncated range [0, a] of the schedule, where a comes from an approximate bound, eq. 17, that gives 0.121 for a vocabulary of about 50k and d = 768 2605.11125 §3.2 2605.11125 tab. 4. The authors call that bound a heuristic 2605.11125 §7. The second is adaptive: the density over the range is refitted every 50 steps from recent (time, loss) pairs, using the positive part of the slope of a spline fit to the loss, then smoothed with an exponential moving average 2605.11125 §3.2 2605.11125 alg. 4 2605.11125 app. B.2.

On GSM8K at T = 1 with 1024 sampling steps, each row of the paper's ablation adds one component to the row above it. Linear schedule 1.2%, truncated 7.7%, then no embedding re-projection 8.1%, then adaptive 11.1% 2605.11125 tab. 2. The authors train one model per configuration, so these gaps are single-seed 2605.11125 §7. Card: Language Modeling with Hyperspherical Flows.

The paper's text says the truncated schedule trains only at the higher noise levels 2605.11125 §3.2. The released adaptive scripts keep that side. The released fixed-truncation script instead sets a lower bound of 0.869 and leaves the upper bound at 1, which keeps the other end of the schedule sfm_sphere_dit_truncated_fixed_no_renorm.sh. The code's time variable also runs the other way, with t = 1 as pure noise noise_schedules.py. Check the truncation side on a short run before you rely on either.

Loss weighting over SNR (P2)

This variant keeps the baseline schedule and changes the weight of the simple noise-prediction loss. The per-step term is multiplied by 1 / (k + SNR(t))^gamma, which down-weights the lowest-noise steps, where the model learns imperceptible detail 2204.00227 §3.2 2204.00227 eq. 8. The authors use k = 1 and gamma of 0.5 or 1, and report noise artefacts for gamma above 2 2204.00227 §3.4. With the linear schedule of the released code at T = 1000, the first step's weight is about 1e-4 for gamma = 1 gaussian_diffusion.py. In the author-reported 256 by 256 image setup, FID-50k on FFHQ at 1000 steps falls from 7.86 to 6.92 2204.00227 tab. 1.

The paper says that its weighting and the noise schedule are related but not interchangeable 2204.00227 §3.3. It also reports that a cosine schedule is much weaker than a linear one even with the weighting 2204.00227 §3.3 2204.00227 fig. 4. The released code applies the factor to the simple loss only, not to the learned-variance term of the hybrid objective gaussian_diffusion.py. Card: Perception Prioritized Training of Diffusion Models.

Fixed hand-designed schedules

The paper compares fixed schedules against the learned one on the same endpoints: log-SNR-linear, beta-linear and alpha-cosine. Each gives the same VLB estimate of 2.66 2107.00630 fig. 4 caption, but the variance of the per-example estimate on the CIFAR-10 test set differs widely. The learned schedule gives 0.53, log-SNR-linear 6.35, beta-linear 31.6 and alpha-cosine 31.1 2107.00630 fig. 4. In an ablation, a fixed schedule from Ho et al. keeps the maximum log-SNR near 8 and stays above 4 BPD on the test set, while learning the endpoints lets the maximum reach 13.3 2107.00630 §6.2. The tutorial gives no fixed schedule values and no comparison between fixed schedules 2208.11970 §3.

The depth paper uses a fixed DDPM schedule with 1000 steps 2312.02145 §4.1. Its code also rescales the schedule to zero terminal SNR and uses trailing timestep spacing marigold_depth_trainer.py. The paper does not compare this with other fixed schedules. The endpoint rescaling in the code is the same change as the zero-terminal variant below.

The NoMaD diffusion policy uses a square-cosine schedule, cited in the paper's bibliography as pmlr-v139-nichol21a 2310.07896 §IV-C. Its code uses the diffusers squaredcos_cap_v2 schedule with epsilon prediction and clipped samples, over 10 denoising steps train.py 2310.07896 §IV-C. The paper's sampling update leaves its coefficients unspecified 2310.07896 §IV-B, and its reported results cover encoders and goal masking, not schedules 2310.07896 §V-C. Card: NoMaD.

Zero terminal SNR rescaling

This variant keeps the shape of a fixed schedule but moves its last step to pure noise. At T = 1000, the terminal SNR is about 4.0e-05 for the linear schedule, 2.4e-09 for the cosine schedule and 0.0047 for the Stable Diffusion schedule, so the last training input still carries part of the image 2305.08891 tab. 1. The authors tie a brightness failure in Stable Diffusion to this leftover signal. At sampling time the model receives pure noise, which does not match what it saw at the last training step 2305.08891 §1 2305.08891 §3.1. They present the flaw as general to variance-preserving diffusion, and argue that variance-exploding models cannot reach zero terminal SNR 2305.08891 §3.1.

The rescale works on the square root of the cumulative alpha. It keeps the first value and sets the last to zero 2305.08891 §3.1 2305.08891 alg. 1. For cosine schedules, removing the clip on beta at 0.999 is enough 2305.08891 §3.1. The paper pairs the rescale with three other changes: v prediction 2305.08891 §3.2, sampling from the last step with trailing spacing 2305.08891 §3.3 2305.08891 tab. 2, and a rescaled classifier-free guidance output 2305.08891 §3.4. Its fine-tune of Stable Diffusion 2.1-base uses all four, for 50k iterations at batch size 2048 2305.08891 §4.

The only numeric result compares the full bundle. On 10k COCO captions with DDIM at 50 steps, the fixed model reaches FID 21.66. The same data without the fixes gives 22.96, and the official model, trained on different data, gives 23.76 2305.08891 tab. 3. The individual fixes are shown only visually 2305.08891 fig. 4 2305.08891 fig. 6. The brightness claim rests on prompt pairs with no numeric metric 2305.08891 §4.1, and the paper reports a single run 2305.08891 §4.2. Card: Common Diffusion Noise Schedules and Sample Steps are Flawed.

When to use which

Learning the endpoints mattered in the paper's ablation, so do not drop that step when simplifying the schedule 2107.00630 §6.2.

Implementation sketch

Common core of the variants: the schedule, the forward noising, and the schedule-weighted loss. The learned monotone network and the fixed schedules differ only in how gamma is produced. The Gumbel variant changes how training levels are drawn and fitted. The truncated and adaptive variant changes the range and density of the drawn times. The P2 variant leaves the schedule alone and multiplies the simple loss by its SNR factor 2204.00227 eq. 8. The multi-resolution variant changes the noise eps. The zero-terminal variant changes the endpoint of the schedule before this core runs; the last block of the sketch shows that step.

# Noise factors from the log-SNR gamma(t) = -log SNR(t)  [§3.2] [eq. 3-4]
def noise_factors(t, gamma):
    var = sigmoid(gamma(t))          # sigma_t^2
    alpha_sq = sigmoid(-gamma(t))    # alpha_t^2, so alpha^2 + sigma^2 = 1
    return alpha_sq, var

# Noisy state at time t  [§3.1] [eq. 1]
def noise_state(x, t, gamma, eps):
    alpha_sq, var = noise_factors(t, gamma)
    return sqrt(alpha_sq) * x + sqrt(var) * eps

# Continuous-time loss for one example  [eq. 17]
def schedule_loss(x, t, gamma, eps, eps_hat):
    z_t = noise_state(x, t, gamma, eps)
    # Weight is 1/2 times the schedule derivative at t, not a fixed weight
    return 0.5 * gamma_prime(t) * sum((eps - eps_hat) ** 2)

# Gumbel proposal for training levels, clipped to the 1e-5 quantiles  [§4.1] [alg. 1]
def sample_gamma(u, P_mu, P_beta):
    u = clip(u, 1e-5, 1 - 1e-5)      # u ~ U(0, 1)
    return P_mu - P_beta * log(-log(u))

# Rescale cumulative alpha so the last step has zero signal  [§3.1] [alg. 1]
# Input:  alpha_bar (T,), first entry for t = 1
# Output: alpha_bar (T,) with the last entry 0 and the first unchanged
def zero_terminal_alpha_bar(alpha_bar):
    # Work in sqrt(alpha_bar) space  [§3.1]
    root = sqrt(alpha_bar)
    head, tail = root[0], root[-1]
    # Shift so the tail is 0, then scale so the head keeps its value  [alg. 1]
    root = (root - tail) * head / (head - tail)
    # Recover per-step alphas as ratios of consecutive values  [alg. 1]
    return root ** 2

Contradictions

The two variational and tutorial papers do not disagree on the SNR convention. Both use gamma(t) = -log SNR(t), with alpha_bar_t = sigmoid(-gamma(t)) 2107.00630 §3.2 2208.11970 §3.

They differ on the loss weight. The variational paper's continuous-time loss weights the noise-prediction error by 1/2 gamma'(t) 2107.00630 eq. 17. The tutorial derives per-timestep weights, but its practical loss is the unweighted norm of the prediction error, and it does not say which weight it means in practice 2208.11970 §3. Whether the two losses match for a given schedule is not stated in either paper.

The variational paper says the interior of the schedule only changes the estimator's variance 2107.00630 §5.3. LangFlow reports that its interior schedule changes generative perplexity 2604.11748 §4.1. The two papers measure different things, the bound and sample quality, so they do not contradict each other. Compare them with that scope in mind.

The P2 paper reports that a cosine schedule is much worse than a linear one even with its weighting 2204.00227 §3.3 2204.00227 fig. 4. The variational paper's invariance result concerns only the bound, which depends on the endpoints 2107.00630 §5.1. These are different measurements: sample quality under a changed loss weight, and a variational bound under the unweighted objective. Neither result settles the other.

LangFlow's own tables give two different LM1B generative perplexities for the same model with self-conditioning: 81.5 in 2604.11748 tab. 1 and 92.2 in 2604.11748 tab. 2. The appendix NFE comparison on LM1B supports 92.2 2604.11748 app.. Use 92.2 unless the other value is confirmed.

The released code and the variational paper also differ on the schedule form, as described under the variants above. The S-FLM paper's text and its released fixed-truncation script also differ on which end of the schedule is kept, as described under its variant. Both differences are within one paper, not between papers.

The zero-terminal change moves an endpoint. The variational paper says the loss depends on the schedule only through its endpoint SNRs 2107.00630 §5.1, so under that result setting SNR(1) to zero changes the bound's endpoint. The common-diffusion paper reports FID and IS, not bound values 2305.08891 tab. 3. Neither paper measures the other's quantity, so the FID gain is not evidence about the bound, and the variational invariance is not evidence about sample brightness.

Papers

• Variational Diffusion Models: a learned monotone noise schedule trained on the continuous-time variational bound, with endpoints learned and the interior trained for variance reduction.
• Perception Prioritized Training of Diffusion Models: a fixed SNR-based weight on the simple loss that down-weights the lowest-noise steps, with the schedule unchanged.
• Understanding Diffusion Models: A Unified Perspective: a tutorial derivation of the SNR parameterisation and the noise-prediction loss, with no experiments.
• Common Diffusion Noise Schedules and Sample Steps are Flawed: rescales fixed schedules to zero terminal SNR, with v prediction, trailing sample steps and a rescaled guidance output, tested by fine-tuning Stable Diffusion 2.1-base.
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation: annealed multi-resolution training noise and a fixed 1000-step DDPM schedule for a depth-from-image latent diffusion model.
• NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration: a robot diffusion policy with ten denoising steps and a square-cosine schedule that the paper names but does not study.
• LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling: a learned Gumbel-shaped schedule over log-SNR for continuous text diffusion, fitted to the observed cross-entropy.
• Language Modeling with Hyperspherical Flows: a truncated and adaptive noise schedule for a continuous flow language model whose tokens live on a hypersphere.