---
kind: concept
slug: classifier-guidance
title: Classifier guidance
summary: >-
  Classifier guidance steers a pretrained diffusion model toward a target by adding a differentiable score's gradient during sampling, without retraining. Covers four forms (sampler shift, cross-modal mismatch, point estimate under a constraint, discrete logit adjustment), their costs, and scale pitfalls.
tags: [diffusion, guidance, sampling, controllable-generation, multimodal]
papers: ["2112.05744", "2206.09012", "2208.11970", "2212.09478", "2303.08137"]
related: [diffusion-models, discrete-diffusion, classifier-free-guidance, neural-style-transfer, semantic-segmentation, video-generation, generative-prior-inverse-problems]
status: draft
updated_at: 2026-11-01
---

## Overview

A diffusion model turns noise into a sample in many small denoising steps.
Classifier guidance adds a second input to that loop: a differentiable score
for the target, whose gradient nudges each step toward the target. The
generator itself is left as it is. The shared invariant is that the score must
be differentiable in the quantity it shifts, so the gradient exists at every
step. For Gaussian samplers that quantity is the noisy sample; for discrete
tokens it is the token probabilities [arxiv:2112.05744 §3.1] [arxiv:2206.09012
§2.2] [arxiv:2303.08137 §3.3].

The four method cards in this wiki show four ways to use that invariant. A
fifth, a tutorial, gives the score-level formula they all build on, with no
experiments [arxiv:2208.11970 §5] [eq:150]:

- **Score-level formula.** The guided score is the unconditional score plus a
  weight times the gradient of the classifier's log-likelihood of the label
  given the noisy sample [arxiv:2208.11970 §5] [eq:150]. Card:
  [Understanding Diffusion Models: A Unified Perspective](/papers/2208.11970).
- **Sampler shift.** Each reverse step's mean is moved by a scaled gradient of
  the score. The score here is a CLIP-based similarity to a text or image
  target [arxiv:2112.05744 §3.1] [eq. 5]. Card: [Semantic Diffusion Guidance](/papers/2112.05744).
- **Mismatch gradient on a partner modality.** Two modalities are denoised
  jointly. One is conditioned on the other by moving its step along the gradient
  of how far the partner's prediction is from the real condition
  [arxiv:2212.09478 §3.4] [eq. 12]. Card: [MM-Diffusion](/papers/2212.09478).
- **Point estimate under a constraint.** The output is a single estimate that is
  optimised by gradient steps on the denoising error plus the log of a
  differentiable constraint [arxiv:2206.09012 §2.2] [alg. 1]. Card:
  [Diffusion models as plug-and-play priors](/papers/2206.09012).
- **Logit adjustment on discrete tokens.** The reverse distribution over
  categorical tokens is shifted in log space by a prior, which is the negative
  gradient of summed differentiable losses on the token probabilities. The
  known fields are fixed by masking [arxiv:2303.08137 §3.3] [eq. 6] [eq. 7].
  Card: [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](/papers/2303.08137).

The first three change how samples are drawn, and the mismatch gradient does so
for a pair of modalities. The fourth replaces sampling with optimisation, so it
returns one estimate per run, and each run is stochastic [arxiv:2206.09012
§2.2]. The fifth keeps sampling, but the guidance acts on discrete
probabilities, not on a continuous sample, and it is only one of two mechanisms
the paper uses for conditions [arxiv:2303.08137 §3.3].

## Variants

### Sampler shift with a continuous score

Each reverse step is the usual Gaussian step, with its mean shifted by the
user's scale times the variance times the gradient of the score. The paper says
this follows the classifier-guidance rule of Dhariwal and Nichol, with the
log-probability of a class replaced by a continuous score [arxiv:2112.05744
§3.1] [alg. 1]. The score can be a language term, a reference-image content
term or a style term, and a weighted sum of them is allowed [arxiv:2112.05744
§3.2] [§3.3] [§3.4] [eq. 10]. The CLIP encoder used for the score is adapted to
noisy inputs. Its batch-norm layers are made timestep-conditioned, and it is
finetuned with a contrastive loss between clean and noised versions of the
same images, so no text labels are needed [arxiv:2112.05744 §3.5]. Card:
[Semantic Diffusion Guidance](/papers/2112.05744).

### Mismatch gradient on a partner modality

The paper generates a short video and its audio with two coupled denoisers, and
each reverse step reads both noisy inputs [arxiv:2212.09478 §3.2] [§3.3]. To
condition one modality on the other without training for that task, the
partner is replaced at each step by a forward-noised copy of the real
condition. The paper also offers a gradient variant. The target is moved
against the gradient of the squared mismatch between the partner's prediction
at that step and the real condition, with the update scaled by the square root
of one minus the cumulative coefficient [arxiv:2212.09478 §3.4] [eq. 10]
[eq. 11] [eq. 12]. The paper calls the scale a gradient weight and gives no
value for it. The abstract calls both forms zero-shot conditional generation, but the paper
reports them in figures only [arxiv:2212.09478 §4.5] [fig. 5]. Card: [MM-Diffusion](/papers/2212.09478).

### Point estimate under a differentiable constraint

The estimate is updated by gradient steps on the denoising error minus the log
of a constraint c(x, y). Fresh noise is drawn at every step, and the noise
level follows a schedule that falls from high to low, so early steps set the
layout and late steps add detail [arxiv:2206.09012 §2.2] [alg. 1]. The
constraint is whatever the task needs. The paper uses a log-likelihood from an
attribute classifier for faces [arxiv:2206.09012 §3.2] [eq. 14], a mutual
information term plus a weak-label KL for land cover [arxiv:2206.09012 §4]
[eq. 15], and a differentiable tour matrix with no length term for the
travelling salesman problem [arxiv:2206.09012 §5]. Card:
[Diffusion models as plug-and-play priors](/papers/2206.09012).

### Logit adjustment on discrete tokens

The inputs are a sequence of categorical tokens, such as the category and
quantised box coordinates of each layout element. The denoiser is not
retrained. At each reverse step, its log-probabilities are shifted by a weight
times a prior [arxiv:2303.08137 §3.3] [eq. 6]. The prior is the negative
gradient, with respect to those probabilities, of a sum of differentiable
losses [eq. 7]. A hand-written prior is also allowed. The paper's refinement
task uses one: a token gets a bonus when its bin lies within a margin of a
noisy observed coordinate [eq. 8]. Relations between elements are handled the
same way, with a loss that penalises violated relations [app. A.2]. Because the
bins are discrete, the paper first converts each probability vector to an
expected coordinate, then applies the relation adjustment three times per
diffusion step [§4.5]. Hard conditions are a separate mechanism: known fields
are replaced by their known values after every step [§3.3]. Card:
[LayoutDM](/papers/2303.08137).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Want a sample that follows a text or image target, one sample per run, and a score you can train without labels | Sampler shift with a continuous score | The sampler stays the same, and the score can be adapted from unlabelled images [arxiv:2112.05744 §3.5] |
| Two coupled outputs, such as video and audio, where one should follow the other, and no conditional model is trained for the pair | Mismatch gradient, or replacement without the gradient | The paper conditions one modality on the other with no extra training [arxiv:2212.09478 §3.4]. Its gain from the gradient is shown in figures only [§4.5] [fig. 5] |
| Need a structured output (a route, a label map) that a sample from the generator cannot express directly | Point estimate under a constraint | The constraint is written on the output, and the generator acts as a prior [arxiv:2206.09012 §2.1] [§5] |
| Constraint is a classifier log-likelihood on the output | Point estimate under a constraint | The paper uses this form for face attributes [arxiv:2206.09012 §3.2] [eq. 14] |
| Output is a set of discrete tokens (categories, quantised coordinates) and the conditions are weak priors such as refinement or relations | Logit adjustment on discrete tokens | The prior is added in log space at each step, and the paper applies it to refinement and relation tasks [arxiv:2303.08137 §3.3] [§4.3] |
| Output is discrete tokens and the conditions are known fields | Masking, with logit adjustment for the rest | Known fields are fixed at every step, so no guidance gradient is needed for them [arxiv:2303.08137 §3.3] |
| Need several samples of the same target | Sampler shift, or repeated point-estimate runs | The point-estimate runs are stochastic, and the paper ran each experiment once [arxiv:2206.09012 §2.2] |
| Latency or compute budget is tight | Sampler shift with fewer steps, after tuning, or replacement instead of the gradient | The gradient forms need a backward pass per step. The point-estimate paper reports 200 to 1000 steps for its experiments [arxiv:2206.09012 app. B.1] [app. B.2]. The discrete paper uses 100 steps [arxiv:2303.08137 §4.4] |
| Score is a learned net that must read noisy inputs | Adapt it to noisy inputs before use | The sampler-shift paper finetunes CLIP on noised images [arxiv:2112.05744 §3.5]. The point-estimate paper does not adapt its classifier; it applies it to the current estimate with augmentation [arxiv:2206.09012 app. B.2] |

Scale and step counts from the cited papers are settings for their own tasks,
not general defaults. The sampler-shift paper reports a default image scale of
100 and language scale of 120 [arxiv:2112.05744 §4.1]. Its experiments are on
FFHQ and LSUN cat, horse and bedroom data [arxiv:2112.05744 §4.1]. The
point-estimate paper's experiments are on digits, faces, land cover and tours
[arxiv:2206.09012 §3] [§4] [§5]. The mismatch-gradient paper gives no value for
its gradient weight; its released conditional scripts use 3.0 [arxiv:2212.09478
§3.4] [code: ssh_scripts/audio2video_sample_sr.sh#L13]. The discrete paper
searches its refinement weight over {1.0, 2.0, 3.0, 4.0, 5.0} and its margin
over {0.1, 0.2} [arxiv:2303.08137 app. A.3].

## Implementation sketch

Language-agnostic pseudocode of the four cores, with notes on where variants
differ. Same style as paper cards: Python-flavoured, lines of at most 80
characters, a `# comment` line above each step (with its citation), short
trailing comments only for shapes or units.

```pseudocode
# Sampler shift with a continuous score  [arxiv:2112.05744 alg. 1]
# Input:  target y, scale s; unconditional model (mu, sigma^2) for x_t
# Output: sample x_0

def sample_shifted(model, score, y, s):
    # Start from standard Gaussian noise  [alg. 1]
    x = random_normal(shape)                      # (3, H, W)
    for t in reversed(range(T)):
        # Unconditional mean and variance from the pretrained model  [§3.1]
        mu, sigma_sq = model(x, t)
        # Gradient of the score with respect to the noisy sample  [alg. 1]
        grad = gradient_of(score, x, y, t)
        # Shift the mean by the scaled, variance-weighted gradient  [alg. 1]
        x = sample_gaussian(mu + s * sigma_sq * grad, sigma_sq)
    return x
```

```pseudocode
# Mismatch gradient for one modality given the other  [arxiv:2212.09478 §3.4] [eq. 12]
# Input:  audio_next (noisy audio at step t+1), video_real (clean video), weight lam
# Output: audio_t (guided audio at step t)

def guided_audio_step(audio_next, video_real, t, lam):
    # Replace the partner by its forward-noised version at step t+1  [§3.4]
    video_hat = forward_noise(video_real, step=t + 1)
    # One joint denoising step gives both modalities at step t  [eq. 11]
    audio_t, video_t = joint_unet(audio_next, video_hat, step=t + 1)
    # Squared mismatch between the predicted and the real condition  [eq. 12]
    mismatch = sum((video_t - video_hat) ** 2)
    grad = gradient_of(mismatch, wrt=audio_t)
    # Move the target against the gradient; the paper's prefactor is
    # sqrt(1 - alpha_bar_t), and the code's differs (see Pitfalls)  [eq. 12]
    return audio_t - lam * sqrt(1 - alpha_bar[t]) * grad
```

```pseudocode
# Point estimate under a constraint  [arxiv:2206.09012 alg. 1]
# Input:  denoiser eps_theta, data y, constraint c, noise schedule, rate lr
# Output: point estimate x

def infer_point_estimate(denoiser, y, constraint, schedule, lr):
    # Start from pure Gaussian noise  [alg. 1]
    x = random_normal(shape)
    for t_i in schedule:          # schedule falls from high to low noise  [§2.2]
        # Draw fresh noise and noise the current estimate  [alg. 1]
        eps = random_normal(shape)
        x_t = sqrt(alpha_bar[t_i]) * x + sqrt(1 - alpha_bar[t_i]) * eps
        # Denoising error minus the log-constraint  [alg. 1] [eq. 12]
        denoise_error = sum_of_squares(eps - denoiser(x_t, t_i))
        loss = denoise_error - log(constraint(x, y))
        # Plain gradient step; the experiments use Adam  [app. B.1] [app. B.2]
        x = x - lr * gradient_wrt_x(loss)
    return x
```

```pseudocode
# Logit adjustment on discrete tokens  [arxiv:2303.08137 eq. 4] [eq. 6]
# Input:  state z_t (B, N), known mask, known values, prior losses L_i
# Output: state z_{t-1}

def adjusted_step(denoiser, z_t, t, known, known_mask, losses, lam):
    # Reverse distribution over tokens from the denoiser  [eq. 4]
    log_p = log_posterior(denoiser(z_t, t), z_t, t)
    # Prior: minus the gradient of the summed losses  [eq. 7]
    pi = -gradient_wrt_probs(sum_of(losses, exp(log_p)), exp(log_p))
    # Shift in log space by the weighted prior  [eq. 6]
    log_p = log_p + lam * pi
    # Known fields are replaced by their known values  [§3.3]
    log_p = where(known_mask, known_log_probs(known), log_p)
    return sample(log_p)
```

Notes on where variants differ. The sampler-shift and mismatch-gradient forms
change each reverse step, so one run gives one sample. The point-estimate form
changes the optimisation, so it needs a schedule and a stochastic loop. The
sampler-shift form costs one score forward and backward per step, on top of the
generator pass [arxiv:2112.05744 §4]. The mismatch-gradient form backpropagates
through the joint denoiser at each step, which is a full backward pass through
the generator [arxiv:2212.09478 §3.4] [eq. 12]. The point-estimate form costs
one denoiser forward and backward per step, and extra classifier passes if the
constraint needs them, such as eight augmented views per step for faces
[arxiv:2206.09012 app. B.2] [code: ffhq/infer_with_attributes.ipynb]. The
logit-adjustment form adds the loss evaluations on the token probabilities at
each step. The paper does not state their cost. In the code, the prior is added
only to fields that are not known [code: src/trainer/trainer/helpers/task.py#L216].

## Contradictions

No direct numerical conflict between the sampler-shift, mismatch-gradient and
point-estimate papers. They report on different tasks, so their numbers cannot
be compared. The discrete paper is a fourth task family and is not compared
with the others. The tutorial reports no numbers, so it adds nothing to any
comparison [arxiv:2208.11970 §6]. Within each paper, the reported settings do
not all agree:

- **Sampler-shift scale, within one paper.** The text gives defaults of 100 for
  image guidance and 120 for language guidance [arxiv:2112.05744 §4.1]. The
  text row of the FFHQ table has FID 28.38, which equals the scale-320 row of
  the language ablation, not the scale-120 default [arxiv:2112.05744 tab. 1]
  [tab. 3].
- **Sampler-shift metrics disagree on the winner.** On FFHQ text guidance,
  the sampler-shift method has the lower FID (28.38 against 57.45), but the
  baseline has the higher retrieval scores. The paper says the baseline's
  retrieval comes at the cost of low diversity [arxiv:2112.05744 tab. 1] [§4.2].
- **Mismatch gradient, claim without numbers.** The text says the gradient
  variant is better than replacement [arxiv:2212.09478 §4.5]. The paper's
  tables do not report that comparison, and the figures are the only evidence
  [fig. 5] [fig. C.4]. The gap is therefore open.
- **Point-estimate land cover, within one paper.** The text calls the transfer
  results comparable to Rolf et al. [arxiv:2206.09012 tab. 1]. On the Durham
  and Phoenix IoU columns, the implicit-posterior baseline is higher: 48.4
  against 46.4 on Durham and 46.0 against 32.4 on Phoenix [arxiv:2206.09012
  tab. 1].
- **Logit-adjustment ablation, within one paper.** The refinement table's row
  for LayoutDM without logit adjustment has the same four values as LayoutDM's
  category-to-size-and-position row in the conditional table: Rico FID 3.55,
  Rico Max 0.277, PubLayNet FID 7.95 and PubLayNet Max 0.310
  [arxiv:2303.08137 tab. 1] [tab. 3]. The two rows come from different tasks,
  so either the same run is reported under both, or one row is a copy error.
  The card did not check the source, so this is open.

## Pitfalls

- **Scale is not one number.** Larger sampler-shift scale gives a closer match
  to the target and less diversity. In the image ablation, LPIPS falls from
  0.617 to 0.565 when scale doubles from 100 to 200 on LSUN Cat, while top-1
  retrieval rises from 0.178 to 0.278 [arxiv:2112.05744 tab. 2]. In the language
  ablation FID rises as scale goes from 120 to 320, from 19.60 to 28.38, while
  top-1 rises from 0.248 to 0.553 [arxiv:2112.05744 tab. 3]. Choose a target
  metric before tuning the scale.
- **Score magnitude differs by term.** The language score is a cosine in
  [-1, 1]. The effective step also depends on the model variance at each
  timestep, and the released sampler applies no normalisation
  [arxiv:2112.05744 eq. 6] [code: sdg/gaussian_diffusion.py#L372-L384]. Early
  and late steps therefore move the sample by different amounts.
- **Style weight does not transfer.** The style term's effective strength
  depends on feature magnitudes, so a weight tuned on one dataset may not carry
  over [arxiv:2112.05744 §3.3].
- **The schedule is the heavy lifting, and the paper does not give it.** The
  point-estimate paper describes the noise schedule only as a shape, falling
  from high to low [arxiv:2206.09012 §2.2]. Each experiment uses a different
  formula in its code, with different constants [code: mnist/inference.ipynb]
  [code: tsp/inference.py#L163-L165] [code: ffhq/infer_with_attributes.ipynb].
- **Loss scale changes the weights.** The paper writes the denoising term as a
  sum of squared errors. The released code uses a mean. Constraint weights tuned
  against the mean do not mean the same thing under the paper's sum
  [arxiv:2206.09012 eq. 12] [code: tsp/inference.py#L175].
- **Relation prior is a loss average with a large rate.** The paper's prior is
  the negative gradient of a sum of losses [arxiv:2303.08137 eq. 7]. The code
  averages the losses and takes one gradient step with a learning rate of 3e6,
  so the effective weight is set by that rate, not by a λπ value. The paper does
  not state λπ for relations [code: src/trainer/trainer/models/categorical_diffusion/logit_adjustment.py#L103-L105]
  [code: src/trainer/trainer/models/categorical_diffusion/logit_adjustment.py#L121-L122].
- **Guidance timing differs from the paper.** The paper applies the relation
  adjustment three times per diffusion step [arxiv:2303.08137 §4.5]. The code
  applies it only from step 10 onwards [code: src/trainer/trainer/models/categorical_diffusion/logit_adjustment.py#L107].
  Results from the paper's schedule may not match runs of the release.
- **Plain gradient descent is not what runs.** The algorithm uses one rate, but
  the experiments use Adam or Adamax. Weights such as 0.01 were tuned in that
  setting [arxiv:2206.09012 app. B.1] [app. B.2].
- **Single runs.** The point-estimate paper ran each main experiment once, with
  no error bars, and inference is stochastic [arxiv:2206.09012 §2.2]. Compare
  seeds before reading small gaps as real.
- **Attribute sign.** The released face example labels its target with a
  comment that contradicts the attribute name. Check the sign before reusing it
  [code: ffhq/infer_with_attributes.ipynb].
- **Structure guidance is missing from the release.** The sampler-shift paper
  gives a spatial-alignment content term, but the released code does not
  implement it, so its structure-preserving result cannot be reproduced from
  the release [arxiv:2112.05744 eq. 8] [code: sdg/guidance.py#L24-L33].
- **Mismatch-gradient formula differs from the code.** The paper scales the
  update by the square root of one minus the cumulative coefficient, and
  compares the prediction at step t with the condition noised to step t. The
  code scales by the square root of the cumulative coefficient, compares the
  prediction at step t−1, and skips the update at t = 0
  [arxiv:2212.09478 §3.4] [eq. 12] [code: mm_diffusion/multimodal_gaussian_diffusion.py#L795-L817].
  Implement one of them on purpose; the paper does not justify the difference.
- **Mismatch-gradient equations disagree with each other.** Equation 10 has a
  prefactor of one over the square root of one minus the cumulative coefficient,
  and equation 12 has the square root of it. The predicted-condition symbol in
  equation 12 is never defined [arxiv:2212.09478 §3.4] [eq. 10] [eq. 12].
- **Mismatch-gradient replacement reuses noise.** The code re-noises the given
  condition at every step with one fixed noise tensor for the whole sample. The
  paper does not say whether noise is reused [code: mm_diffusion/multimodal_gaussian_diffusion.py#L642-L700].
- **Weight convention is the paper's own.** The tutorial's weight multiplies
  the classifier gradient directly, and a weight of 1 gives the plain
  conditional model [arxiv:2208.11970 §5] [eq:150]. It gives no mapping to
  the scales used by the other cards, so a value copied from one card does not
  carry over to another.
- **The classifier must accept noisy inputs.** The tutorial calls the
  classifier gradient "adversarial", but it is the gradient of a classifier
  trained on noisy samples, not a GAN term [arxiv:2208.11970 §5] [eq:148].
- **Rounding can fail.** For tours, the optimised matrix does not always give a
  single cycle, and the paper reports disjoint cycles as a common failure
  [arxiv:2206.09012 §5].
- **Refinement noise is redrawn.** The LayoutDM refinement task draws new
  Gaussian noise on every call, so a run is not reproducible without fixing the
  seed [arxiv:2303.08137 §4.3] [code: src/trainer/trainer/helpers/task.py#L127].

## Papers

- [Semantic Diffusion Guidance](/papers/2112.05744): steers an unconditional DDPM by language, reference-image content or style, using a time-conditioned CLIP score finetuned without labels.
- [Diffusion models as plug-and-play priors](/papers/2206.09012): uses a diffusion model as a prior and optimises a point estimate under a differentiable constraint, with an annealed noise level.
- [Understanding Diffusion Models: A Unified Perspective](/papers/2208.11970): a tutorial derivation of diffusion models that writes classifier guidance as a score-level formula. It has no experiments.
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](/papers/2212.09478): jointly denoises a video and its audio, and conditions one on the other by replacement or by a gradient on a mismatch, without training for that task.
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](/papers/2303.08137): generates layouts as discrete tokens, and applies known fields by masking and weak priors by logit adjustment, without retraining.