Classifier-free guidance

Overview

A sampler repeatedly predicts a correction at each step. Classifier-free guidance (CFG) makes two predictions at each step, one with the condition (usually a text prompt) and one with the condition removed, and combines them with a weight. The output is pushed toward the condition, and the weight sets how hard 2211.12572 §4 eq. 5.

The shared invariant: at inference, every step needs both branches, and the unconditional branch only exists because the model was trained with the condition sometimes removed. GLIGEN states this directly: caption and grounding tokens are dropped during training, which is what enables CFG [arxiv:2301.07093 app. A]. eDiff-I and MLD do the same with their own conditions 2211.01324 §4.2 2212.04048 §3.3. Composer and SDFusion do it for several conditions at once 2302.09778 §2.3 2212.04493 §3.3. DreamPose gives its image and pose inputs separate weights [arxiv:2304.06025 §4.4], and VASA-1 gives each of its conditions a weight in one sum 2404.10667 §3.2. Transformer and autoregressive generators reuse the same recipe: MDM drops its text or action condition for 10% of samples 2209.14916 §3, DiT trains a learned null label 2212.09748 §3.1, and SDXL drops its conditions 10% of the time [arxiv:2307.01952 app. Diffusion Models]. unCLIP applies it only in its pixel decoder, and its prior is not guided 2204.06125 §2.1 §5.5.

The same idea is not limited to diffusion. MoMask applies it to the logits of a masked token transformer 2312.00063 §3.4, and Infinity applies it to the per-bit logits of a bitwise autoregressive model 2412.04431 §4.7. Muse applies the same weighted sum to the logits of its masked transformer 2301.00704 §2.7 eq. 1.

The condition need not come from a user. A physics-informed diffusion model uses the gradient of a PDE residual at the noisy sample as its condition, and drops that condition in training so that an unconditional branch exists 2211.14680 alg. 1 eq. 6.

Video and 3D pipelines use the same weighting. MagicAnimate guides a pose-driven video model with a scale of 7.5 animation.yaml, and Make-It-3D sets the guidance weight to 10 for novel views in score distillation 2303.14184 §4.1. Drive-WM drops each of its conditions 20% of the time in training and samples with a scale of 5.0 2311.17918 app. B.1.

Layout-conditioned diffusion uses the same recipe for boxes and class labels. LayoutDiffusion, trained from scratch, replaces its whole layout with a null layout in 20% of training samples and guides with one scale 2303.17189 §3.4. Some work changes the guided output itself: the common-schedules paper rescales its standard deviation so that strong guidance does not overexpose the image 2305.08891 §3.4.

Variants

Standard CFG with a conditional weight

The guided prediction is a weighted sum of the conditional and unconditional predictions, with the weight on the conditional term. In this notation a weight of 1 gives the conditional prediction alone, so larger values push harder toward the prompt 2211.12572 §4 eq. 5. Card: PnP. Composer uses the same form, with ω on the conditional term 2302.09778 §2.1 eq. 2. Card: Composer. Infinity applies it to the bit logits with a scale of 4 2412.04431 §4.7. Card: Infinity. A derivation-only tutorial writes the same form, with γ on the conditional score and γ = 1 giving the conditional model. It presents the form as one of two guidance routes, next to classifier guidance 2208.11970 §5. Card: Understanding Diffusion Models.

The same form runs in a VAE latent for text-to-motion, with scale s = 7.5 on the conditional term 2212.04048 §3.3 eq. 4. Card: MLD.

Scale on the difference in transformer denoisers

This form starts from the unconditional prediction and adds the scaled difference to the conditional one, so s = 1 gives the conditional prediction alone. MDM uses it for text and action conditions, with s = 2.5 for text-to-motion 2209.14916 §3 eq. 7 §4.1. Card: MDM. DiT uses it for class labels in a latent transformer, with the unconditional branch set by a learned null embedding 2212.09748 §3.1. Its tables report DiT-XL/2 at s = 1.25 and s = 1.50 2212.09748 tab. 2 tab. 3. Card: DiT. In both papers the same network produces the unconditional prediction, so no second model is needed.

SDXL is a latent text-to-image model with two text encoders. It writes the guided prediction with w on the unconditional term, and w = 0 gives the conditional prediction alone 2307.01952 app. Diffusion Models. Its released guider uses the scale-on-difference form with s = 1 + w, so the demo default of 5.0 is w = 4 guiders.py. Card: SDXL.

Negative prompting

The unconditional term is replaced by a blend of the empty prompt and a negative prompt. The blend weight alpha starts at a set value and decreases over the sampling steps. The authors describe the effect as strong for textureless inputs such as silhouettes and minor for natural images 2211.12572 §4 eq. 6 app. C. Card: PnP.

Object removal in PowerPaint uses the same slot. The object prompt takes the unconditional place and the context prompt the conditional one, with weight w on the context prompt 2312.03594 §3.2 eq. 5. The paper gives no value for w §3.2. Card: PowerPaint.

StableSR applies the same form to a frozen Stable Diffusion 2.1-base super-resolution model. Its unconditional term comes from the negative prompt and its conditional term from the null prompt, and a scale of 1 gives the default 2305.07015 §5.1 eq. 8. Its sweep uses two negative prompts at scales 0.0, 2.5, 5.0 and 7.5 2305.07015 tab. 6. With one prompt, moving the scale from 0.0 to 7.5 raises RealSR MUSIQ from 64.86 to 67.26 and CLIP-IQA from 0.6176 to 0.6323, and lowers PSNR from 24.80 to 23.53 and SSIM from 0.7097 to 0.6673; LPIPS rises from 0.3105 to 0.3399 tab. 6. The text notes that 7.5 can over-sharpen 2305.07015 §5.1 fig. 16. Card: StableSR.

Whole-layout null token for box and class conditions

LayoutDiffusion takes one scale for its layout. Its null branch is a whole-layout token with box (0, 0, 1, 1) and class 0, with padding slots elsewhere 2303.17189 §3.1 §3.4. Its main text writes the guided prediction as (1 − s) times the unconditional prediction plus s times the conditional one, the weight-1 form under Pitfalls. Its appendix and shipped code write cond + s (cond − uncond) instead, the weight-0 form, so the code default of s = 1.0 is s = 2.0 in the main-text form 2303.17189 app. B.2 classifier-free_sample_for_layout.py. Card: LayoutDiffusion. The model is trained from scratch on box-labelled data and has no text prompt 2303.17189 §5.

Rescaling the guided output

The common-schedules paper writes its guidance in the negative-prompt form, x_neg + w (x_pos − x_neg), with w = 7.5 2305.08891 §3.4 eq. 13. It then matches the standard deviation of the guided output to that of the positive output, and blends the rescaled and the plain guided output with factor φ §3.4 eq. 14 to eq. 16. The paper uses φ = 0.7 and reports 0.5 to 0.75 as giving the most appealing results. φ = 0 overexposes §3.4 §5.3 fig. 6. Its eq. 13 does not say whether the outputs are noise, velocity or clean-image predictions, and that choice sets the spread being matched §3.4. Card: Common Diffusion Noise Schedules. The FID of 21.66 that the paper reports is for all four of its fixes together, so this rescale is not measured alone 2305.08891 tab. 3.

Per-condition weights on one model

When a model takes several inputs, each input can get its own weight. SDFusion starts from the unconditional prediction and adds, for each modality, its weight times the difference between the prediction with only that modality present and the unconditional one 2212.04493 §3.3 eq. 4. A larger weight makes that input more influential. Card: SDFusion. The released code runs this form only in one branch, and its default multi-modal path scales the input features instead; see Contradictions.

Leave-one-out weights over several conditions

VASA-1 guides all of its conditions at once. The guided prediction is the full-condition prediction scaled by one plus the sum of the weights, minus, for each condition, its weight times the prediction with that condition emptied 2404.10667 §3.2 eq. 2. With one condition it reduces to the (1 + w) form under Pitfalls. Card: VASA-1. Its defaults are 0.5 for audio and 1.0 for gaze 2404.10667 §4. Raising the audio weight from 0.5 to 2.0 raises the sync confidence from 7.957 to 8.295, while the authors report that higher audio weights also amplify mouth motion and jitter 2404.10667 tab. 4 §4.3.

Separate weights for image and pose

DreamPose applies two weights in two steps. It starts from the prediction with no conditions, adds the image term with weight s_I, then adds the pose term with weight s_p, measured against the image-only prediction 2304.06025 §4.4 eq. 4. Three predictions per step are needed §4.4. The authors report that a large s_I keeps the appearance and a large s_p keeps the pose, and that separating the two avoids overfitting to the one pose seen in subject fine-tuning §4.4 fig. 7. Card: DreamPose.

Guidance between two condition sets

Composer weights two condition sets rather than a condition and nothing. Its second set is emphasised with ω and the first is suppressed with 1 − ω, and conditions shared by both keep weight 1 2302.09778 §2.1 eq. 3. This form lets an edit move from one set of conditions to another, together with DDIM inversion of the input image 2302.09778 §2.1 §3.2. Card: Composer.

Text and image terms in a subgoal generator

TaKSIE generates the next subgoal image from the current image, the task instruction and a progress embedding. The generator is a diffusion model that takes the current image as a condition. The paper reports ControlNet and InstructPix2Pix as options 2410.11013 §3.4.1 app. D. It uses two guidance terms, one for the instruction with a negative prompt made by swapping the instruction's antonym, and one for the current image. Both scales are 2.5 2410.11013 §3.4.1 §4.2.3. Card: TaKSIE. The paper does not write out how the two terms combine. The released pipeline uses an InstructPix2Pix-style form with three predictions pipeline_controlnet.py. The ablation without the negative prompt lowers the CALVIN average length from 3.18 to 3.07 2410.11013 tab. 2.

Guidance with a zero null on a latent hand-pose generator

FastGrasp's paper does not mention guidance, and its inference section describes DDIM sampling only 2411.14786 §3.4. The released sampler still combines a conditional and an unconditional prediction in the (1 + w) form, with the unconditional input set to zeros ddim.py ddim.py. The weight is 2 in the HO-3D config and 0.5 in the GRAB and OakInk evaluation configs ho3d.json eval_adapt.json eval_adapt.json. Card: FastGrasp. The released training code does not drop the condition, so this zero branch is one the model never trains on (see Pitfalls).

Blending two models

DiffusionBlending takes the output of a first model and adds a scalar s times the difference to a second model's output, with both models conditioned on different signals 2303.01418 §3.3 eq. 3. The authors call it a generalisation of CFG, and they use s = 0.5 for two signals §4.3. Card: Human Motion Diffusion. Each control model is fine-tuned from the same base MDM 2303.01418 §4.3.

Two-model guidance for early steps

SINE mixes a model fine-tuned on one image into the conditional term. Its guided prediction is a weighted sum of the unconditional prediction and a mix of the pretrained and fine-tuned conditional predictions, with the weight w on the mix, while the step index is above a threshold K. Below K it runs the pretrained model alone 2212.04489 §3.2 eq. 2 eq. 3. The paper sets K = 400 and puts v = 0.7 on the pretrained term of the mix [arxiv:2212.04489 §4]. It gives no guidance scale; its notebook uses 7.5 SINE.ipynb. Card: SINE.

Logit-space guidance for token models

A masked or bitwise autoregressive generator outputs logits over a vocabulary rather than a noise prediction. Guidance is applied to those logits before sampling. MoMask writes the paper's form with (1 + s) on the conditional logits and s on the unconditional ones 2312.00063 §3.4 eq. 5. Its code uses the other form, with the same weight on the conditional term transformer.py. Infinity guides each bit's two logits the same way, with cfg = 4 2412.04431 §4.7. Cards: MoMask, Infinity.

Muse writes its logit guidance with (1 + t) on the conditional logits and t on the unconditional ones, and it lets a negative prompt replace the unconditional branch 2301.00704 §2.7 eq. 1. Its editing run uses scale 4 with top-k sampling at k = 3 2301.00704 §3.3.2. Card: Muse.

Guidance that ramps over the scales

VAR generates one token map per scale, from coarse to fine, and guides the logits of each map. Its released sampler uses a scale that grows with the scale index, from 0 at the first map to the full value at the last var.py. The paper reports one fixed ratio of 2.0 for its ablation and does not describe a schedule 2404.02905 tab. 3 §3.2. Card: VAR.

Muse ramps its guidance scale linearly over the sampling steps, so early tokens are less guided than late ones 2301.00704 §2.7. The start and end values are not stated §2.7. Card: Muse.

Safety guidance away from a text concept

Safe latent diffusion adds a third prediction, conditioned on a text description of unwanted content, and moves the prompt estimate away from it. The move is scaled per element by a term that is zero where the prompt estimate already exceeds the concept estimate by at least a threshold λ [arxiv:2211.05105 §3] eq. 3 eq. 4 eq. 5. The term applies only after a warm-up of δ steps, and a momentum term carries it across steps 2211.05105 §3 alg. 1. Card: Safe Latent Diffusion. The paper gives four presets, from weak to max, for warm-up, concept scale, threshold and momentum 2211.05105 tab. 1. Author-reported: on the I2P prompt benchmark the inappropriate-image probability for Stable Diffusion 1.4 falls from 0.39 to 0.09 at the strongest preset, and COCO FID-30k rises from 14.43 to 18.76 2211.05105 tab. 2 tab. 3. Negative prompting is the baseline the paper compares against §4 tab. 2.

Guidance with a planner that generates video

UniPi plans with a text-conditioned video diffusion model. Its guided denoiser uses the (1 + w) form with w on the unconditional term, so w = 0 gives the conditional prediction alone 2302.00111 §2.3. The paper does not state w, the conditioning dropout rate, or the sampler §2.3. Card: UniPi.

Guidance inside distillation

A one-step student can be trained against a guided teacher. In DMD, the regression pairs come from the guided model and the real score uses the guided model's mean prediction, while the fake score is unchanged. The guidance scale is fixed during training: 3 for the LAION 6.25+ set and 8 for LAION 6+ 2311.18828 §3.4 §4.3. The paper does not write the exact guided mean §3.4. Card: DMD. The guidance scale is therefore part of the student, not a sampler setting that can change later.

Guidance applied to overlapping views

MultiDiffusion runs one pretrained text-to-image model on overlapping windows of a larger canvas, and it averages the outputs per pixel at each step 2302.08113 §3. The paper's update step has no guidance term, but the released code applies CFG with scale 7.5 and a negative prompt to every window call, batched as two passes 2302.08113 §3 panorama.py. Card: MultiDiffusion.

Guidance over a video with a reference branch

MagicAnimate guides a pose-driven video UNet with the scale-on-difference form, the unconditional prediction plus 7.5 times the difference animation.yaml pipeline_animation.py. Predictions from overlapping frame windows are averaged before the guidance step 2311.16498 §3.3. Card: MagicAnimate.

Its unconditional branch differs from the conditional one in more than the prompt. In reference attention, the unconditional half attends only to its own tokens and receives none of the reference features mutual_self_attention.py. The paper does not describe this, so the guidance also changes how the reference is used.

Guidance inside score-based 3D optimisation

Score Jacobian Chaining optimises a voxel radiance field so that its renders receive the score of a 2D denoiser, chained through a differentiable renderer 2212.00774 §4. The paper states the guidance scale and not the guidance form. The scale is 100 for text-to-3D with Stable Diffusion [arxiv:2212.00774 app. A3] and 3.0 and 10.0 in its 2D tests §5.1. The released code runs classifier-free guidance with an empty negative prompt 2212.00774 §5 adapt_sd.py. Card: SJC. The 3D value is ten times the largest 2D value tested, so the two settings are not interchangeable.

Make-It-3D uses the same kind of guidance inside score distillation, applied to a NeRF. Novel views get the distillation loss with guidance weight ω = 10, and the timestep is drawn between 200 and 600 2303.14184 §4.1. The card's reading of the mix puts (1 + ω) on the conditional prediction and ω on the unconditional one, so it is the weight-0 form described in Pitfalls 2303.14184 §4.1. The reference view gets a masked pixel loss instead. At timesteps up to 400, a CLIP-based loss replaces the distillation term 2303.14184 §3.2 app. C.1. Card: Make-It-3D.

Attention methods that sit next to guidance

DenseDiffusion changes attention scores early in sampling and does not train the model 2308.12964 §3.2. Its equations never mention guidance. The code runs the modulation on the conditional half of a batch that also holds the unconditional prompt, so the guidance step and the attention edit are coupled gradio_app.py. The paper states no guidance scale, and the code does not set one gradio_app.py. Card: DenseDiffusion.

Guidance on the decoder of a two-stage model

unCLIP maps a caption to a CLIP image embedding with a prior, then renders the embedding with a diffusion decoder. Only the decoder is guided; the paper does not apply guidance to the prior 2204.06125 §5.5. The guidance scale of 1.25 is the value the paper reports for FID 2204.06125 tab. 2, and the scales used for human evaluation are not stated. This card does not record the weight convention, so check it against the paper before copying a value. The authors report that unCLIP keeps its content as the scale rises, while GLIDE's semantics converge 2204.06125 fig. 9, and that at the high scales used for photorealistic output it is more diverse than GLIDE at similar photorealism and caption similarity 2204.06125 fig. 10. Card: unCLIP.

Physics-residual gradient as the condition

The condition here is not a user input. The network takes c = dr_t/dx_t, the gradient of a PDE residual at the noisy sample, as an extra input 2211.14680 eq. 6 alg. 1. Training drops c with probability p_u, so the same network also gives an unconditional prediction alg. 1. Sampling mixes the two outputs with (w + 1) on the conditional one and −w on the unconditional one 2211.14680 alg. 2 eq. 8. Card: Physics-informed Diffusion Model.

The paper's Table 3 compares this learned form with a direct gradient step (eq. 9) and with no physics term. On the sparse task, the learned form has the lowest equation residual at each printed iteration. At the third iteration it reports 0.27, against 0.38 for the direct step and 0.39 for the baseline. On the 8x task, at its last printed step count, it reports 0.20 against 0.99 and 1.47, while L2 stays within 0.04 across the three rows 2211.14680 tab. 3. The table gives one number per cell, with no seeds or spread, so the size of the gaps is not quantified.

Training-time condition dropout

This is not a sampling rule. The model is trained with each condition dropped at a fixed rate, so that the unconditional prediction exists at sampling time. A model that takes new conditions, here bounding boxes with phrases, is trained this way for each of them. The GLIGEN paper uses 10% for both caption and grounding tokens 2301.07093 app. A. Card: GLIGEN.

unCLIP's decoder drops its CLIP image embedding and its caption, with the caption dropped 50% of the time 2204.06125 §2.1. The embedding rate is 10% in that section and 5% in §5.1; see Contradictions. Its prior drops the caption 10% of the time 2204.06125 §2.2. Card: unCLIP.

A model with several embeddings drops each one on its own. eDiff-I uses independent rates of 0.25 for T5, 0.2 for CLIP text and 0.9 for CLIP image embeddings, and dropping all three gives the unconditional branch 2211.01324 §4.2 tab. 7. Card: eDiff-I. MLD uses one rate of 0.1 for its single condition 2212.04048 §3.3.

Composer drops each condition independently with probability 0.5. It also drops all conditions with probability 0.1 and keeps all with probability 0.1, and it uses 0.7 for its intensity representation, which otherwise carries most of the image information 2302.09778 §2.3. Card: Composer. Other models set their own rates: ComMDM drops the text with 10% 2303.01418 §3.2, MoMask with 10% for its masked transformer 2312.00063 §3.4, and Infinity's released default is 0.1 arg_util.py. LayoutDiffusion drops its whole layout in 20% of training samples in every run 2303.17189 app. C.1 tab. 4. Muse drops the text for 10% of training samples, and does not say what replaces the dropped text [arxiv:2301.00704 §2.7]. SDFusion's paper gives no rate, while its multi-modal code keeps each input with probability 0.5 2212.04493 §3.3.

DreamPose drops the pose alone, the image alone, and both together, with 5% for each, so 15% of training samples lose at least one condition 2304.06025 supp. §1. VASA-1 drops each condition with probability 0.1 and drops its two previous-window conditions with 0.5, which covers the first window that has no history 2404.10667 §3.2. Card: VASA-1.

The transformer and token models add their own rates. DiT's paper says only that labels are dropped at random, and its code uses 0.1 [arxiv:2212.09748 §3.1] models.py. VAR's paper does not mention label dropout, and its code replaces 10% of the labels with a null class var.py. Card: VAR.

Drive-WM trains its multiview driving video model with each condition dropped 20% of the time, which provides the unconditional prediction used at sampling 2311.17918 app. B.1. The appendix does not say what replaces a dropped condition app. B.1. Card: Drive-WM.

The physics-residual model drops its residual-gradient condition with probability p_u. The paper does not give p_u, and the released training code defaults to 0.1 2211.14680 alg. 1 losses.py. Card: Physics-informed Diffusion Model.

When to use which

Weight values from the cited papers are settings for their own tasks, not general recommendations. PnP reports 15.0 for real-image translation and 7.5 for generated guidance images 2211.12572 app. C. GLIGEN does not state a guidance scale in the paper 2301.07093 app. A. Composer states 3.0 for its prior and base 2302.09778 §3.5. Infinity uses 4 on all scales 2412.04431 tab. 9. MoMask uses 4 for HumanML3D and 2 for KIT-ML, with 5 for its residual transformer 2312.00063 §4. SDFusion's released values differ by model: 1.0 for single-view, 3.0 for text, and 1.0 for multi-modal 2212.04493 §3.3 sdfusion_txt2shape_model.py. VASA-1 uses 0.5 for audio and 1.0 for gaze, and states no scale for head distance or emotion 2404.10667 §3.2 §4. DreamPose states no values for its reported results 2304.06025 §4.4. JavisDiT uses 7.0 2503.23377 app. C.1. SINE's paper gives no scale; its notebook uses 7.5 and its README 10 SINE.ipynb README.md. PowerPaint's paper gives no w, and its app defaults to 7.5 app.py. The survey gives no scale of its own; it says that for Stable Diffusion w = −1 gives empty output and w above 10 gives saturation 2310.07204 §3.3. SJC uses 100 for text-to-3D and 3.0 and 10.0 for its 2D tests 2212.00774 app. A3 §5.1, and StableSR's negative-prompt sweep runs up to 7.5 2305.07015 tab. 6. Make-It-3D uses 10 on novel views 2303.14184 §4.1, MagicAnimate's released config uses 7.5 animation.yaml, and Drive-WM uses 5.0 2311.17918 app. B.1. The guidance form for Drive-WM is not stated. LayoutDiffusion's code default is s = 1.0 in the (1 + s) form, and its appendix figures use 1.0, 0.6 and 0.5 2303.17189 app. A.4 app. A.5. The common-schedules paper uses w = 7.5 2305.08891 §3.4.

MDM's text-to-motion scale is 2.5 2209.14916 §4.1, against 7.5 for MLD on a related task, so the two numbers come from different models and should not be copied across. DiT's 1.25 and 1.50 correspond to w = 0.25 and 0.5 in the weight-on-unconditional form 2212.09748 tab. 2. VAR's ablation uses 2.0 and its FID command 1.5 2404.02905 tab. 3 README.md. DMD uses 3 and 8 for its two text-to-image sets 2311.18828 §4.3. Muse uses 4 for editing 2301.00704 §3.3.2, and its base sampling scale is not stated §2.7. SLD does not state a guidance scale for its I2P and quality runs; its ablation figures use 15 2211.05105 app. I, and the released code defaults to 7.5 sld_pipeline.py. TaKSIE uses 2.5 for both its text and image terms 2410.11013 §4.2.3. FastGrasp states no weight; its released configs use 2 for HO-3D and 0.5 for GRAB and OakInk ho3d.json eval_adapt.json.

MLD's sweep over s, at dropout 0.1, gives the lowest FID at s = 2.5 (0.350). R-Precision rises up to s = 7.5, which the paper chooses (FID 0.473) 2212.04048 tab. 11. A FID-only target and a text-match target therefore pick different scales. eDiff-I does not state the guidance weight behind its Table 1 FID numbers. It shows FID–CLIP curves for weights 0 to 10 instead 2211.01324 §5.1 fig. 7.

For classifier training data, Imagen's base guidance was swept from 1.0 to 5.0 2304.08466 §4.2. Added synthetic data helped ResNet-50 up to 4.8M total images at 256 px and hurt above 6.0M total 2304.08466 tab. 4.

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a # comment line above each step (with its citation), short trailing comments only for shapes or units.

# Standard CFG, conditional-weight form  [arxiv:2211.12572 §4] [eq. 5]
# Input:  x (noisy latent), t (timestep), cond (prompt embedding),
#         uncond (empty-prompt embedding), w (weight on the conditional term)
# Output: eps (noise estimate passed to the sampler)

def cfg_eps(x, t, cond, uncond, w):
    # Two U-Net passes per step: with the prompt and with the empty prompt
    eps_cond = unet(x, t, cond)
    eps_uncond = unet(x, t, uncond)
    # Weighted sum; w = 1 gives the conditional prediction alone  [eq. 5]
    return w * eps_cond + (1 - w) * eps_uncond

# Negative prompting: the unconditional term becomes a blend  [eq. 6]
# Input:  alpha (blend weight, decays over sampling), neg (negative prompt)
# Output: eps (noise estimate passed to the sampler)

def negative_cfg_eps(x, t, cond, neg, alpha, w):
    eps_cond = unet(x, t, cond)
    eps_uncond = unet(x, t, "")
    eps_neg = unet(x, t, neg)
    # Blend the empty and negative predictions  [eq. 6]
    eps_mix = alpha * eps_uncond + (1 - alpha) * eps_neg
    # Same outer weighting as standard CFG  [eq. 5]
    return w * eps_cond + (1 - w) * eps_mix

# Per-condition weights  [arxiv:2212.04493 §3.3] [eq. 4]
# Input:  conds (one per modality, None if absent), weights s_i
# Output: eps (noise estimate passed to the sampler)

def per_condition_eps(x, t, conds, weights):
    # Unconditional base: every modality zero-filled  [eq. 4]
    eps_uncond = unet(x, t, empty_context())
    eps = eps_uncond
    for i, (c_i, s_i) in enumerate(zip(conds, weights)):
        # Prediction with only modality i present  [eq. 4]
        only_i = unet(x, t, context_with_only(i, c_i))
        # Add the weighted difference from the base  [eq. 4]
        eps = eps + s_i * (only_i - eps_uncond)
    return eps

# Condition dropout during training  [arxiv:2301.07093 app. A]
# Input:  caption tokens, grounding tokens, drop_prob (paper: 0.1)

def dropout_conditions(caption, grounding, drop_prob):
    # Replace each condition with its null embedding at random, so the
    # model learns the unconditional prediction used by CFG at sampling
    if random() < drop_prob:
        caption = null_caption
    if random() < drop_prob:
        grounding = null_grounding
    return caption, grounding

The per-condition sketch follows eq. 4 and not the released code, whose multi-modal default differs; see Contradictions.

# Guidance with a standard-deviation rescale  [arxiv:2305.08891 §3.4]
# Equations 13 to 16 of that section
# Input:  pos, neg (model outputs, same shape), w (weight), phi (rescale factor)
# Output: guided output, same shape as pos

def cfg_std_rescale(pos, neg, w, phi):
    # Negative-prompt form of guidance  [eq. 13]
    guided = neg + w * (pos - neg)
    # Spread of the positive output and of the guided output  [eq. 14]
    # Per-sample std over the non-batch axes; the paper does not state this
    ratio = std(pos) / std(guided)
    # Rescaled output, blended with the plain guided output  [eq. 15] [eq. 16]
    rescaled = guided * ratio
    return phi * rescaled + (1 - phi) * guided

Notes on where variants differ: negative prompting changes only the unconditional term and needs one extra pass per step. The rescale above takes the guided output as its input, so the value of w depends on the form it is written in; the paper writes it only in the negative-prompt form. Composer's two-set form and DiffusionBlending change which predictions are combined, not only the weights. DreamPose applies its two weights in sequence rather than as one sum, and VASA-1 subtracts one emptied prediction per condition [arxiv:2304.06025 eq. 4] 2404.10667 eq. 2. The physics-residual sampler 2211.14680 alg. 2 fits the same function, with c set to the residual gradient for the conditional call and None for the unconditional one. Token models apply the same weighted sum to logits, before sampling 2312.00063 §3.4 eq. 5. Feature injection in PnP applies to every branch, see Pitfalls. The training-time version changes the training loop, not the sampler. The latent-space version (MLD) applies the same function to the denoiser's latent, with the empty string as the unconditional text and a zero vector for action labels 2212.04048 eq. 4 mld_denoiser.py.

Contradictions

No direct conflict between the papers' reported numbers. Several places where a paper and its code, or a paper and its config, disagree:

• Dropout rate, GLIGEN. The paper states 10% for caption and grounding tokens 2301.07093 app. A. The released box config sets a caption probability of 0.5 GoldG+SBU+CC3M+O365_box_text_image.yaml. The GLIGEN card lists both.
• Guidance weight, PnP. The paper gives 15.0 for real images 2211.12572 app. C; the shipped real-image config uses 10.0 pnp-real.yaml. The PnP card lists both.
• Per-condition weights, SDFusion. The paper's eq. 4 weights each modality 2212.04493 §3.3. The default multi-modal inference path instead scales the condition features and runs ordinary CFG; eq. 4 is only in a second branch sdfusion_mm_model.py sdfusion_mm_model.py. In that branch, one global scale multiplies every weight, so the effective weight is scale × w_i.
• Dropout rate, SDFusion. The paper gives no rate for its task-specific encoders 2212.04493 §3.3. The multi-modal code keeps each input with probability 0.5 sdfusion_mm_model.py.
• Scale convention, MoMask. The paper's eq. 5 and the released code write the same guidance with different weights; the paper's published values match the code's convention 2312.00063 §3.4 eq. 5 transformer.py. The residual transformer's condition drop is 10% in the paper, and the README's command uses 0.2 2312.00063 §3.4 README.md.
• Blend, DiffusionBlending. The paper's blend has a free scale s 2303.01418 eq. 3. The released code averages the models' outputs with equal weights model_blending.py. The two agree at s = 0.5 only.
• Guidance in the second take, DoubleTake. Figure 3 says the refinement step is conditioned on the neighbouring intervals 2303.01418 fig. 3. The code turns text guidance off for that step sampling_utils.py sampling_utils.py.
• Sampling temperature, Infinity. The paper's best decoding uses tau = 1.00 2412.04431 tab. 9. The inference script uses 0.5 infer.sh. This sits next to the guidance step, so set both explicitly.
• Guidance weights, DreamPose. The paper states no s_I or s_p for its reported results 2304.06025 §4.4. The code defaults to 0.5 for both test.py, the README example uses 8 and 3 README.md, and the pipeline's sweep uses 0, 3, 5, 7 and 9 dual_encoder_pipeline.py.
• Guidance form, SJC. The paper states a guidance scale and no form 2212.00774 §5. The released code adds a classifier-free guidance step with an empty negative prompt, and divides the noisy latent by √(1+σ²) before the denoiser adapt_sd.py. The SJC card lists both.
• Guidance not stated, MultiDiffusion. The paper's update has no guidance term 2302.08113 §3; the code uses 7.5 with a negative prompt panorama.py.
• Guidance window, SINE. The paper applies the fine-tuned term for t > K 2212.04489 §3.2. The CLI default applies it for 1 ≤ t ≤ 400 stable_txt2img_guidance.py guidance_ddim.py. The notebook and README match the paper SINE.ipynb README.md. The SINE card lists both.
• Mix weight, SINE. The paper and notebook use v = 0.7 §4. The CLI default is 0.5 stable_txt2img_guidance.py, and the README uses 0.4 README.md.
• Guidance scope, DenseDiffusion. The paper's equations never mention guidance 2308.12964 §3.2. The code modulates only the conditional half of the batch gradio_app.py. The DenseDiffusion card lists both.
• Safety scale, SLD. The paper's eq. 5 sets the scale to max(1, |φ|) 2211.05105 eq. 5 alg. 1, while its text says the scale is clipped 2211.05105 §3. The released code clips to min(1, |φ|) sld_pipeline.py. The SLD card lists both.
• Momentum sign, SLD. Algorithm 1 as printed has no plus sign between the two momentum terms 2211.05105 alg. 1. Eq. 8 has one 2211.05105 eq. 8, and the code adds the two sld_pipeline.py.
• CLIP-embedding dropout, unCLIP. §2.1 and its pseudocode zero the CLIP embedding 10% of the time 2204.06125 §2.1. §5.1 and the fig. 8 caption give 5% 2204.06125 §5.1 fig. 8. The paper does not explain the difference. The unCLIP card lists both.
• Guidance in FastGrasp. The paper does not mention classifier-free guidance 2411.14786 §3.4. The released sampler applies it ddim.py, and the released training code drops no condition ddim.py. The FastGrasp card lists both.
• Update rule, physics-residual sampler. Algorithm 2 has no gradient term in its update 2211.14680 alg. 2. The released guided DDIM step subtracts the residual gradient at weight 1 denoising_step.py, and its conditional configs set the mix weight to 0. The 2211.14680 card lists both.
• Guidance combination, TaKSIE. The paper names two guidance terms, both at scale 2.5, and does not write out how they combine 2410.11013 §3.4.1 §4.2.3. The released pipeline uses uncond + w_text·(text − image) + w_image·(image − uncond) pipeline_controlnet.py. The TaKSIE card lists both.
• Guidance in the refine stage, Make-It-3D. The paper's refine stage uses a coarse-style objective 2303.14184 §3.3. The released refine code applies score distillation with guidance 5 utils.py, while its main default is 10 main.py. The Make-It-3D card lists both.
• Classifier-free form, LayoutDiffusion. The main text writes the guided prediction as (1 − s) uncond + s cond 2303.17189 §3.4. Appendix B.2 and the shipped code write cond + s (cond − uncond) 2303.17189 app. B.2 classifier-free_sample_for_layout.py. The code default s = 1.0 is therefore s = 2.0 in the main-text form. The LayoutDiffusion card lists both.
• Padding mask, LayoutDiffusion. The paper does not describe a key-padding mask 2303.17189 §3.2. The code has one, but the shipped configs turn it off, so padding slots take part in the attention LayoutDiffusion_large.yaml. The LayoutDiffusion card lists both.

Pitfalls

• Weight conventions. Papers write the same guidance in at least three forms, and a scale copied from one into another changes the strength:
  • Weight 1 is conditional-only: ω·cond + (1 − ω)·uncond. Used by PnP, Composer, Infinity, MoMask's code, the tutorial, and SDFusion's eq. 4 for one modality 2211.12572 eq. 5 2302.09778 eq. 2 2412.04431 §4.7 2208.11970 §5 2212.04493 eq. 4. LayoutDiffusion's main text uses the same form, with s on the conditional term 2303.17189 §3.4.
  • Weight 0 is conditional-only: (1 + w)·cond − w·uncond. Used by UniPi, by MoMask's paper, by Muse's logits, by SDXL's paper and by the survey's eq. 11 2302.00111 §2.3 2312.00063 eq. 5 2301.00704 eq. 1 2307.01952 app. Diffusion Models 2310.07204 eq. 11, and per condition in VASA-1's eq. 2 2404.10667 eq. 2, and FastGrasp's released sampler ddim.py, LayoutDiffusion's appendix and code 2303.17189 app. B.2, and the physics-residual sampler 2211.14680 alg. 2. A value of 4 here equals 5 in the first form. VAR's code uses it with a w that grows per scale [code: models/var.py#L172].
  • Scale on the difference: uncond + s·(cond − uncond). Same as the first form, written differently. Used by DiT, MDM, and SDXL's released guider 2212.04493 eq. 4 2212.09748 §3.1 2209.14916 eq. 7 guiders.py, and the common-schedules paper's eq. 13, with the negative prompt as the unconditional term 2305.08891 eq. 13. Make-It-3D's distillation mix is written in this form 2303.14184 §4.1. Check which form a paper or library uses before copying a value.
• Weight 0 in the physics sampler is not no guidance. The shipped conditional configs set w = 0, so the mix is off. The released sampler still subtracts the residual gradient in its update kmflow_re1000_rs256_conditional.yml denoising_step.py. The paper gives no w alg. 2. Test the sampler with the mix on and off before reporting either.
• Null condition differs by model. The unconditional branch is an empty prompt for PnP and MLD's text, a zero-filled set of inputs in SDFusion, an all-zero image in SDFusion's single-view model sdfusion_img2shape_model.py, and a zero vector for MLD's actions 2212.04048 eq. 4. FastGrasp's released sampler uses an all-zero input ddim.py. DiT's null is a learned embedding at an index past the last class 2212.09748 §3.1 models.py. MDM's unconditional token is the time embedding alone mdm.py, and VAR's null is an extra class index 2404.02905 §3.2. Use the form your model was trained with. DreamPose's dropped image and pose are all-zero tensors that still pass through CLIP and the VAE, so the null context is whatever those networks output for black input train.py train.py. JavisDiT uses a learned null embedding for its prior, which the paper does not describe base_model.py 2503.23377 app. C.2.1.
• Two-model guidance costs an extra pass. SINE's eq. 3 needs the pretrained conditional, the fine-tuned conditional and the pretrained unconditional prediction. The code also runs the fine-tuned unconditional branch, whose output does not enter the update guidance_ddim.py.
• Attention edits and guidance share a batch. In DenseDiffusion's code, only the conditional half of the batch is modulated. An implementation that modulates the unconditional half too differs from the code gradio_app.py gradio_app.py.
• Safety guidance costs an extra pass. SLD runs the unconditional, prompt and concept predictions every step sld_pipeline.py sld_pipeline.py. The paper reports no timing. Its pipeline also runs Stable Diffusion's own safety checker when one is loaded, which changes the output path sld_pipeline.py.
• A user negative prompt changes the SLD baseline. The code uses a negative prompt in place of the empty unconditional branch [code: src/sld/sld_pipeline.py#L349-L367], so the paper's protocol no longer holds.
• A token model's ramp endpoints are missing. Muse's linear ramp has no published start or end value 2301.00704 §2.7. Choose them and report them, since the ramp changes which tokens are guided.
• Guidance can silently do nothing. In SDFusion's blended-diffusion completion path, the guidance call omits the unconditional conditioning, so the scale has no effect there sdfusion_model.py ddim.py 2212.04493 §3.3.
• Negative prompting costs more. The code runs the U-Net on two batches of two per step, recomputing the unconditional branch, which is about twice the usual CFG cost [Plug-and-Play card, Pitfalls] ddim.py.
• Per-condition guidance costs one pass per input. SDFusion's eq. 4 needs the unconditional pass plus one pass per input 2212.04493 eq. 4. Its multi-modal guided branch runs three denoiser passes per step by stacking the input three times ddim.py.
• Leave-one-out guidance costs one pass per condition. VASA-1's eq. 2 needs the full prediction and one emptied prediction per condition 2404.10667 eq. 2. Its text does not say whether each emptied term removes one condition or all of them; the per-condition reading is the one the notation gives §3.2. It also gives no scale for head distance or emotion §3.2 §4, so those values must be chosen before reuse.
• Guidance can lift a metric above real data. VASA-1's SyncNet lip-sync scores exceed those of real videos when audio guidance is on, and the authors attribute this to CFG 2404.10667 §4.1 §4.3. Weigh the score with that in mind.
• Priors are resampled per call. JavisDiT's estimator draws a fresh prior on every call, so the same prompt yields different conditions unless the random state is fixed. The paper does not say whether the prior is fixed per prompt base_model.py 2503.23377 §3.2.
• Negative prompt can be a silent no-op. In the PnP code, the negative prompt defaults to the source prompt, which is forced to empty for real images, so negative prompting does nothing unless it is set explicitly run_pnp.py 2211.12572 §4.
• The DreamPose guidance argument can be a silent no-op. Its guidance_scale only decides whether the three-pass branch runs. The update uses only the two per-condition weights dual_encoder_pipeline.py dual_encoder_pipeline.py. Set s_I and s_p explicitly.
• Invert with the empty prompt. PnP inverts a real image with an empty prompt at CFG scale 1, so any prompt and scale can be used at sampling 2211.12572 §5.1 app. C.
• Feature injection hits every branch. When features are injected, the conditional, unconditional and negative passes all receive them ddim.py 2211.12572 §4.
• The unconditional branch loses the reference. In MagicAnimate's reference attention, the unconditional half gets no reference features, and the guided step doubles the batch for each window mutual_self_attention.py pipeline_animation.py. The paper's text does not say which features the unconditional branch gets. Choose the behaviour on purpose and report it.
• Dropout needs a null embedding. Training with dropout needs a fixed null embedding for each condition. GLIGEN's code also uses learned null embeddings for padded grounding slots, which the paper does not describe text_image_grounding_net.py.
• Guidance can use a null the model never trained on. FastGrasp's released sampler guides against a zero input, but its training code keeps every condition: the dropout probability is stored and the line that would mask the condition is commented out ddim.py ddim.py. The paper reports no guidance experiment, so compare outputs at the released weight and at w = 0 before relying on the term.
• The null condition is not defined in MLD's paper. The paper's empty condition is undefined 2212.04048 eq. 4. The code encodes the empty string with CLIP for text and uses a zero vector for actions mld.py mld_denoiser.py.
• Dropped embeddings are zeroed, separately from the null token. eDiff-I zeroes a dropped embedding. Its cross-attention also has a learnable null token, which is a different thing 2211.01324 §4.2.
• Dropout rate moves FID and R-Precision in opposite directions. In MLD's sweep at s = 7.5, p = 0.25 gives the lowest FID (0.229) but a lower R-Precision (0.757) than the chosen p = 0.1 (0.772, FID 0.473) 2212.04048 tab. 11.
• Headline FID without a stated weight. eDiff-I's Table 1 does not state the guidance weight behind its FID numbers 2211.01324 §5.1. Treat the numbers as the authors' sampler setting, not a value you can copy.
• Unstated dropout and weight in some papers. UniPi gives neither its guidance weight nor its dropout rate 2302.00111 §2.3. The Imagen synthetic-data paper states the sweep but not the prompt template 2304.08466 §4. Drive-WM states its guidance scale and dropout rate, but not its guidance form or the replacement for a dropped condition 2311.17918 app. B.1. Any reimplementation has to choose these and say so.
• Selection on the evaluation set. The Imagen paper picks guidance and sampling settings on FID, IS and classifier accuracy measured on the ImageNet validation set, and reports classifier accuracy on that same set 2304.08466 §4.2 §5.2. Treat the headline numbers as tuned values.
• Guidance on a subset of channels. DiT's code applies the guidance only to the first three latent channels by default models.py app. A. The paper's appendix reports that three channels at scale 1.5 give FID-50K 2.27, and four channels at 1.375 give 2.20 app. A. The cause is left open app. A. Copy the channel choice along with the scale.
• The guided mean is not written out in the distillation. DMD's real score uses the guided model's mean prediction, but the exact formula is not given 2311.18828 §3.4. Reimplementations must choose it and state the choice.
• A ramped scale is easy to miss. VAR's code ramps the scale per scale index var.py. Copying a fixed cfg of 2.0 into its sampler therefore changes the output. The paper does not describe the ramp 2404.02905 §3.2.
• Guiding the prior can hurt. unCLIP reports that guidance on its prior hurt results, so the prior runs unguided 2204.06125 §5.5. Do not add guidance to a prior by default.
• Short runs may show no grounding effect. In GLIGEN the gated layer starts closed and the paper reports grounding appears only after tens of thousands of iterations 2301.07093 app. F.
• The rescale is not measured alone. The common-schedules paper's FID compares the official model with a model that has all four fixes, and its data differ from the official model's 2305.08891 tab. 3 §4.2. Its rescale-factor sweep is visual fig. 6. A reimplementation that adds only the CFG rescale has no reported number to match.

Papers

• Muse: masked-token text-to-image model with logit-space guidance on the (1 + t) form, ramped over the steps; the ramp endpoints are not stated.
• Safe Latent Diffusion: safety guidance that adds a text-defined concept as a third prediction to CFG; the paper's scale clip differs from its code.
• unCLIP: two-stage text-to-image model with dropout-trained guidance in the decoder only; the CLIP-embedding dropout is 10% in one section and 5% in another.
• PnP: training-free image-to-image editing that uses CFG with a conditional weight and negative prompting, with the weight and blend settings listed above.
• GLIGEN: trains a frozen diffusion model with condition dropout so that CFG applies to added grounding inputs.
• eDiff-I: a text-to-image cascade with independent dropout on three embeddings; its guidance sweep appears only as curves.
• MLD: text-to-motion latent diffusion with CFG in the latent and a published dropout and scale sweep.
• SDFusion: 3D shape diffusion with per-input guidance weights, and a released code that differs from the paper's eq. 4.
• UniPi: text-conditioned video planner that uses CFG with the (1 + w) form; the guidance and dropout values are not stated.
• MultiDiffusion: training-free panoramas and region control by averaging guided windows; the paper has no guidance term, the code does.
• Composer: composable conditions with dropout rates per condition and a two-set guidance form.
• Human Motion Diffusion as a Generative Prior: DiffusionBlending, a blend of two fine-tuned models that the authors describe as a generalisation of CFG.
• Synthetic Data from Diffusion Models Improves ImageNet Classification: per-stage guidance and sampling sweeps for Imagen, judged by downstream accuracy.
• MoMask: logit-space CFG for a masked token transformer, with the paper and code forms differing.
• Infinity: bitwise autoregressive model with CFG on bit logits at scale 4 and a temperature that differs between paper and script.
• DreamPose: image-to-video fashion animation with two separate weights, one for the photo and one for the pose; the paper gives no values for its results.
• VASA-1: talking-face generation with a leave-one-out weight per condition; the audio weight trades lip sync against mouth motion.
• Score Jacobian Chaining: text-to-3D by back-propagating a 2D diffusion score through a voxel renderer; the guidance scale is 100 for 3D, and the paper reports no quantitative metric.
• StableSR: super-resolution from a frozen Stable Diffusion, with negative-prompt guidance whose scale trades no-reference scores against PSNR, SSIM and LPIPS.
• JavisDiT: joint audio-video diffusion transformer that samples with one guidance scale of 7.0 and a learned null prior.
• SINE: one-image editing with a fine-tuned copy mixed into the early guidance steps; the CLI and the notebook use different windows.
• DenseDiffusion: training-free layout control by attention modulation, which shares a batch with CFG in the code; the paper gives no guidance scale.
• State of the Art on Diffusion Models for Visual Computing: a survey whose classifier-free guidance uses the (1 + w) form, with its weight warnings for Stable Diffusion.
• Understanding Diffusion Models: a derivation-only tutorial that writes CFG as a convex weighting of the conditional and unconditional scores; it reports no experiments.
• A Task is Worth One Word: PowerPaint, one inpainting model for several tasks, with object removal by a negative prompt in classifier-free guidance.
• TaKSIE: visual subgoal generator with separate text and image guidance terms, both at 2.5; the paper does not write out how they combine.
• FastGrasp: hand-pose latent diffusion whose released sampler uses guidance with a zero null, though the paper omits guidance and the training code drops no condition.
• Make-It-3D: single-image 3D with score distillation at weight 10 on novel views; the released refine stage uses guidance 5.
• MagicAnimate: pose-driven human video with scale 7.5 on the difference, and an unconditional branch that drops the reference features in the released code.
• Drive-WM: multiview driving video model with guidance 5.0 and 20% condition dropout; the guidance form is not stated.
• LayoutDiffusion: layout-to-image diffusion trained from scratch, with a whole-layout null token dropped 20% of the time; the main text and the code write the same scale in different forms.
• Common Diffusion Noise Schedules and Sample Steps are Flawed: four fixes for the training and sampling mismatch, one of which rescales the guided output by its standard deviation with φ = 0.7; the rescale is not measured alone.
• Human Motion Diffusion Model: MDM, with the scale-on-difference form and 10% null-condition training for text and action.
• Scalable Diffusion Models with Transformers: DiT, class-conditional latent transformer with a learned null label; the paper reports FID at scales 1.25 and 1.50.
• SDXL: latent text-to-image model with two text encoders, whose guided form puts w on the unconditional term; the released guider uses the scale-on-difference form.
• Visual Autoregressive Modeling: next-scale autoregressive generator with a ramped logit guidance in its released sampler; the paper's ablation uses one fixed value.
• One-step Diffusion with Distribution Matching Distillation: one-step student trained against a guided teacher at a fixed scale.
• A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction: diffusion prior for flow-field super-resolution whose condition is the PDE residual gradient, mixed in the (1 + w) form; the paper gives no w and no dropout rate.