Classifier guidance

Overview

A diffusion model turns noise into a sample in many small denoising steps. Classifier guidance adds a second input to that loop: a differentiable score for the target, whose gradient nudges each step toward the target. The generator itself is left as it is. The shared invariant is that the score must be differentiable in the quantity it shifts, so the gradient exists at every step. For Gaussian samplers that quantity is the noisy sample; for discrete tokens it is the token probabilities 2112.05744 §3.1 [arxiv:2206.09012 §2.2] 2303.08137 §3.3.

The four method cards in this wiki show four ways to use that invariant. A fifth, a tutorial, gives the score-level formula they all build on, with no experiments 2208.11970 §5 [eq:150]:

• Score-level formula. The guided score is the unconditional score plus a weight times the gradient of the classifier's log-likelihood of the label given the noisy sample 2208.11970 §5 [eq:150]. Card: Understanding Diffusion Models: A Unified Perspective.
• Sampler shift. Each reverse step's mean is moved by a scaled gradient of the score. The score here is a CLIP-based similarity to a text or image target 2112.05744 §3.1 eq. 5. Card: Semantic Diffusion Guidance.
• Mismatch gradient on a partner modality. Two modalities are denoised jointly. One is conditioned on the other by moving its step along the gradient of how far the partner's prediction is from the real condition 2212.09478 §3.4 eq. 12. Card: MM-Diffusion.
• Point estimate under a constraint. The output is a single estimate that is optimised by gradient steps on the denoising error plus the log of a differentiable constraint 2206.09012 §2.2 alg. 1. Card: Diffusion models as plug-and-play priors.
• Logit adjustment on discrete tokens. The reverse distribution over categorical tokens is shifted in log space by a prior, which is the negative gradient of summed differentiable losses on the token probabilities. The known fields are fixed by masking 2303.08137 §3.3 eq. 6 eq. 7. Card: LayoutDM: Discrete Diffusion Model for Controllable Layout Generation.

The first three change how samples are drawn, and the mismatch gradient does so for a pair of modalities. The fourth replaces sampling with optimisation, so it returns one estimate per run, and each run is stochastic [arxiv:2206.09012 §2.2]. The fifth keeps sampling, but the guidance acts on discrete probabilities, not on a continuous sample, and it is only one of two mechanisms the paper uses for conditions 2303.08137 §3.3.

Variants

Sampler shift with a continuous score

Each reverse step is the usual Gaussian step, with its mean shifted by the user's scale times the variance times the gradient of the score. The paper says this follows the classifier-guidance rule of Dhariwal and Nichol, with the log-probability of a class replaced by a continuous score [arxiv:2112.05744 §3.1] alg. 1. The score can be a language term, a reference-image content term or a style term, and a weighted sum of them is allowed [arxiv:2112.05744 §3.2] §3.3 §3.4 eq. 10. The CLIP encoder used for the score is adapted to noisy inputs. Its batch-norm layers are made timestep-conditioned, and it is finetuned with a contrastive loss between clean and noised versions of the same images, so no text labels are needed 2112.05744 §3.5. Card: Semantic Diffusion Guidance.

Mismatch gradient on a partner modality

The paper generates a short video and its audio with two coupled denoisers, and each reverse step reads both noisy inputs 2212.09478 §3.2 §3.3. To condition one modality on the other without training for that task, the partner is replaced at each step by a forward-noised copy of the real condition. The paper also offers a gradient variant. The target is moved against the gradient of the squared mismatch between the partner's prediction at that step and the real condition, with the update scaled by the square root of one minus the cumulative coefficient 2212.09478 §3.4 eq. 10 eq. 11 eq. 12. The paper calls the scale a gradient weight and gives no value for it. The abstract calls both forms zero-shot conditional generation, but the paper reports them in figures only 2212.09478 §4.5 fig. 5. Card: MM-Diffusion.

Point estimate under a differentiable constraint

The estimate is updated by gradient steps on the denoising error minus the log of a constraint c(x, y). Fresh noise is drawn at every step, and the noise level follows a schedule that falls from high to low, so early steps set the layout and late steps add detail 2206.09012 §2.2 alg. 1. The constraint is whatever the task needs. The paper uses a log-likelihood from an attribute classifier for faces 2206.09012 §3.2 eq. 14, a mutual information term plus a weak-label KL for land cover 2206.09012 §4 eq. 15, and a differentiable tour matrix with no length term for the travelling salesman problem 2206.09012 §5. Card: Diffusion models as plug-and-play priors.

Logit adjustment on discrete tokens

The inputs are a sequence of categorical tokens, such as the category and quantised box coordinates of each layout element. The denoiser is not retrained. At each reverse step, its log-probabilities are shifted by a weight times a prior 2303.08137 §3.3 eq. 6. The prior is the negative gradient, with respect to those probabilities, of a sum of differentiable losses eq. 7. A hand-written prior is also allowed. The paper's refinement task uses one: a token gets a bonus when its bin lies within a margin of a noisy observed coordinate eq. 8. Relations between elements are handled the same way, with a loss that penalises violated relations app. A.2. Because the bins are discrete, the paper first converts each probability vector to an expected coordinate, then applies the relation adjustment three times per diffusion step §4.5. Hard conditions are a separate mechanism: known fields are replaced by their known values after every step §3.3. Card: LayoutDM.

When to use which

Scale and step counts from the cited papers are settings for their own tasks, not general defaults. The sampler-shift paper reports a default image scale of 100 and language scale of 120 2112.05744 §4.1. Its experiments are on FFHQ and LSUN cat, horse and bedroom data 2112.05744 §4.1. The point-estimate paper's experiments are on digits, faces, land cover and tours 2206.09012 §3 §4 §5. The mismatch-gradient paper gives no value for its gradient weight; its released conditional scripts use 3.0 [arxiv:2212.09478 §3.4] audio2video_sample_sr.sh. The discrete paper searches its refinement weight over {1.0, 2.0, 3.0, 4.0, 5.0} and its margin over {0.1, 0.2} 2303.08137 app. A.3.

Implementation sketch

Language-agnostic pseudocode of the four cores, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a # comment line above each step (with its citation), short trailing comments only for shapes or units.

# Sampler shift with a continuous score  [arxiv:2112.05744 alg. 1]
# Input:  target y, scale s; unconditional model (mu, sigma^2) for x_t
# Output: sample x_0

def sample_shifted(model, score, y, s):
    # Start from standard Gaussian noise  [alg. 1]
    x = random_normal(shape)                      # (3, H, W)
    for t in reversed(range(T)):
        # Unconditional mean and variance from the pretrained model  [§3.1]
        mu, sigma_sq = model(x, t)
        # Gradient of the score with respect to the noisy sample  [alg. 1]
        grad = gradient_of(score, x, y, t)
        # Shift the mean by the scaled, variance-weighted gradient  [alg. 1]
        x = sample_gaussian(mu + s * sigma_sq * grad, sigma_sq)
    return x

# Mismatch gradient for one modality given the other  [arxiv:2212.09478 §3.4] [eq. 12]
# Input:  audio_next (noisy audio at step t+1), video_real (clean video), weight lam
# Output: audio_t (guided audio at step t)

def guided_audio_step(audio_next, video_real, t, lam):
    # Replace the partner by its forward-noised version at step t+1  [§3.4]
    video_hat = forward_noise(video_real, step=t + 1)
    # One joint denoising step gives both modalities at step t  [eq. 11]
    audio_t, video_t = joint_unet(audio_next, video_hat, step=t + 1)
    # Squared mismatch between the predicted and the real condition  [eq. 12]
    mismatch = sum((video_t - video_hat) ** 2)
    grad = gradient_of(mismatch, wrt=audio_t)
    # Move the target against the gradient; the paper's prefactor is
    # sqrt(1 - alpha_bar_t), and the code's differs (see Pitfalls)  [eq. 12]
    return audio_t - lam * sqrt(1 - alpha_bar[t]) * grad

# Point estimate under a constraint  [arxiv:2206.09012 alg. 1]
# Input:  denoiser eps_theta, data y, constraint c, noise schedule, rate lr
# Output: point estimate x

def infer_point_estimate(denoiser, y, constraint, schedule, lr):
    # Start from pure Gaussian noise  [alg. 1]
    x = random_normal(shape)
    for t_i in schedule:          # schedule falls from high to low noise  [§2.2]
        # Draw fresh noise and noise the current estimate  [alg. 1]
        eps = random_normal(shape)
        x_t = sqrt(alpha_bar[t_i]) * x + sqrt(1 - alpha_bar[t_i]) * eps
        # Denoising error minus the log-constraint  [alg. 1] [eq. 12]
        denoise_error = sum_of_squares(eps - denoiser(x_t, t_i))
        loss = denoise_error - log(constraint(x, y))
        # Plain gradient step; the experiments use Adam  [app. B.1] [app. B.2]
        x = x - lr * gradient_wrt_x(loss)
    return x

# Logit adjustment on discrete tokens  [arxiv:2303.08137 eq. 4] [eq. 6]
# Input:  state z_t (B, N), known mask, known values, prior losses L_i
# Output: state z_{t-1}

def adjusted_step(denoiser, z_t, t, known, known_mask, losses, lam):
    # Reverse distribution over tokens from the denoiser  [eq. 4]
    log_p = log_posterior(denoiser(z_t, t), z_t, t)
    # Prior: minus the gradient of the summed losses  [eq. 7]
    pi = -gradient_wrt_probs(sum_of(losses, exp(log_p)), exp(log_p))
    # Shift in log space by the weighted prior  [eq. 6]
    log_p = log_p + lam * pi
    # Known fields are replaced by their known values  [§3.3]
    log_p = where(known_mask, known_log_probs(known), log_p)
    return sample(log_p)

Notes on where variants differ. The sampler-shift and mismatch-gradient forms change each reverse step, so one run gives one sample. The point-estimate form changes the optimisation, so it needs a schedule and a stochastic loop. The sampler-shift form costs one score forward and backward per step, on top of the generator pass 2112.05744 §4. The mismatch-gradient form backpropagates through the joint denoiser at each step, which is a full backward pass through the generator 2212.09478 §3.4 eq. 12. The point-estimate form costs one denoiser forward and backward per step, and extra classifier passes if the constraint needs them, such as eight augmented views per step for faces 2206.09012 app. B.2 infer_with_attributes.ipynb. The logit-adjustment form adds the loss evaluations on the token probabilities at each step. The paper does not state their cost. In the code, the prior is added only to fields that are not known task.py.

Contradictions

No direct numerical conflict between the sampler-shift, mismatch-gradient and point-estimate papers. They report on different tasks, so their numbers cannot be compared. The discrete paper is a fourth task family and is not compared with the others. The tutorial reports no numbers, so it adds nothing to any comparison 2208.11970 §6. Within each paper, the reported settings do not all agree:

• Sampler-shift scale, within one paper. The text gives defaults of 100 for image guidance and 120 for language guidance 2112.05744 §4.1. The text row of the FFHQ table has FID 28.38, which equals the scale-320 row of the language ablation, not the scale-120 default 2112.05744 tab. 1 tab. 3.
• Sampler-shift metrics disagree on the winner. On FFHQ text guidance, the sampler-shift method has the lower FID (28.38 against 57.45), but the baseline has the higher retrieval scores. The paper says the baseline's retrieval comes at the cost of low diversity 2112.05744 tab. 1 §4.2.
• Mismatch gradient, claim without numbers. The text says the gradient variant is better than replacement 2212.09478 §4.5. The paper's tables do not report that comparison, and the figures are the only evidence fig. 5 fig. C.4. The gap is therefore open.
• Point-estimate land cover, within one paper. The text calls the transfer results comparable to Rolf et al. 2206.09012 tab. 1. On the Durham and Phoenix IoU columns, the implicit-posterior baseline is higher: 48.4 against 46.4 on Durham and 46.0 against 32.4 on Phoenix [arxiv:2206.09012 tab. 1].
• Logit-adjustment ablation, within one paper. The refinement table's row for LayoutDM without logit adjustment has the same four values as LayoutDM's category-to-size-and-position row in the conditional table: Rico FID 3.55, Rico Max 0.277, PubLayNet FID 7.95 and PubLayNet Max 0.310 2303.08137 tab. 1 tab. 3. The two rows come from different tasks, so either the same run is reported under both, or one row is a copy error. The card did not check the source, so this is open.

Pitfalls

• Scale is not one number. Larger sampler-shift scale gives a closer match to the target and less diversity. In the image ablation, LPIPS falls from 0.617 to 0.565 when scale doubles from 100 to 200 on LSUN Cat, while top-1 retrieval rises from 0.178 to 0.278 2112.05744 tab. 2. In the language ablation FID rises as scale goes from 120 to 320, from 19.60 to 28.38, while top-1 rises from 0.248 to 0.553 2112.05744 tab. 3. Choose a target metric before tuning the scale.
• Score magnitude differs by term. The language score is a cosine in [-1, 1]. The effective step also depends on the model variance at each timestep, and the released sampler applies no normalisation 2112.05744 eq. 6 gaussian_diffusion.py. Early and late steps therefore move the sample by different amounts.
• Style weight does not transfer. The style term's effective strength depends on feature magnitudes, so a weight tuned on one dataset may not carry over 2112.05744 §3.3.
• The schedule is the heavy lifting, and the paper does not give it. The point-estimate paper describes the noise schedule only as a shape, falling from high to low 2206.09012 §2.2. Each experiment uses a different formula in its code, with different constants inference.ipynb inference.py infer_with_attributes.ipynb.
• Loss scale changes the weights. The paper writes the denoising term as a sum of squared errors. The released code uses a mean. Constraint weights tuned against the mean do not mean the same thing under the paper's sum 2206.09012 eq. 12 inference.py.
• Relation prior is a loss average with a large rate. The paper's prior is the negative gradient of a sum of losses 2303.08137 eq. 7. The code averages the losses and takes one gradient step with a learning rate of 3e6, so the effective weight is set by that rate, not by a λπ value. The paper does not state λπ for relations logit_adjustment.py logit_adjustment.py.
• Guidance timing differs from the paper. The paper applies the relation adjustment three times per diffusion step 2303.08137 §4.5. The code applies it only from step 10 onwards logit_adjustment.py. Results from the paper's schedule may not match runs of the release.
• Plain gradient descent is not what runs. The algorithm uses one rate, but the experiments use Adam or Adamax. Weights such as 0.01 were tuned in that setting 2206.09012 app. B.1 app. B.2.
• Single runs. The point-estimate paper ran each main experiment once, with no error bars, and inference is stochastic 2206.09012 §2.2. Compare seeds before reading small gaps as real.
• Attribute sign. The released face example labels its target with a comment that contradicts the attribute name. Check the sign before reusing it infer_with_attributes.ipynb.
• Structure guidance is missing from the release. The sampler-shift paper gives a spatial-alignment content term, but the released code does not implement it, so its structure-preserving result cannot be reproduced from the release 2112.05744 eq. 8 guidance.py.
• Mismatch-gradient formula differs from the code. The paper scales the update by the square root of one minus the cumulative coefficient, and compares the prediction at step t with the condition noised to step t. The code scales by the square root of the cumulative coefficient, compares the prediction at step t−1, and skips the update at t = 0 2212.09478 §3.4 eq. 12 multimodal_gaussian_diffusion.py. Implement one of them on purpose; the paper does not justify the difference.
• Mismatch-gradient equations disagree with each other. Equation 10 has a prefactor of one over the square root of one minus the cumulative coefficient, and equation 12 has the square root of it. The predicted-condition symbol in equation 12 is never defined 2212.09478 §3.4 eq. 10 eq. 12.
• Mismatch-gradient replacement reuses noise. The code re-noises the given condition at every step with one fixed noise tensor for the whole sample. The paper does not say whether noise is reused multimodal_gaussian_diffusion.py.
• Weight convention is the paper's own. The tutorial's weight multiplies the classifier gradient directly, and a weight of 1 gives the plain conditional model 2208.11970 §5 [eq:150]. It gives no mapping to the scales used by the other cards, so a value copied from one card does not carry over to another.
• The classifier must accept noisy inputs. The tutorial calls the classifier gradient "adversarial", but it is the gradient of a classifier trained on noisy samples, not a GAN term 2208.11970 §5 [eq:148].
• Rounding can fail. For tours, the optimised matrix does not always give a single cycle, and the paper reports disjoint cycles as a common failure 2206.09012 §5.
• Refinement noise is redrawn. The LayoutDM refinement task draws new Gaussian noise on every call, so a run is not reproducible without fixing the seed 2303.08137 §4.3 task.py.

Papers

• Semantic Diffusion Guidance: steers an unconditional DDPM by language, reference-image content or style, using a time-conditioned CLIP score finetuned without labels.
• Diffusion models as plug-and-play priors: uses a diffusion model as a prior and optimises a point estimate under a differentiable constraint, with an annealed noise level.
• Understanding Diffusion Models: A Unified Perspective: a tutorial derivation of diffusion models that writes classifier guidance as a score-level formula. It has no experiments.
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation: jointly denoises a video and its audio, and conditions one on the other by replacement or by a gradient on a mismatch, without training for that task.
• LayoutDM: Discrete Diffusion Model for Controllable Layout Generation: generates layouts as discrete tokens, and applies known fields by masking and weak priors by logit adjustment, without retraining.