--- kind: concept slug: classifier-free-guidance title: Classifier-free guidance summary: >- Classifier-free guidance (CFG) steers a generative sampler toward a condition by combining a conditional and an unconditional prediction at every step. It needs training with the condition sometimes dropped. Covers weight conventions, negative prompting, rescaling, per-condition weights, blending two models, logit guidance for token models, training-time dropout, and physics-residual conditions. tags: [generative-models, sampling, guidance, text-to-image, text-to-motion, video-generation, 3d-generation] papers: ["2204.06125", "2208.11970", "2209.14916", "2211.12572", "2301.07093", "2211.01324", "2212.00774", "2212.04048", "2212.04489", "2212.04493", "2212.09748", "2302.00111", "2302.08113", "2302.09778", "2303.01418", "2304.06025", "2304.08466", "2305.07015", "2307.01952", "2308.12964", "2310.07204", "2311.18828", "2312.00063", "2312.03594", "2404.02905", "2404.10667", "2412.04431", "2503.23377", "2211.05105", "2301.00704", "2410.11013", "2411.14786", "2303.14184", "2311.16498", "2311.17918", "2303.17189", "2305.08891", "2211.14680"] related: [diffusion-models, text-to-image-generation, latent-diffusion, layout-to-image-generation, ddim-inversion, masked-language-modeling, text-guided-image-editing, diffusion-inpainting, attention-mechanism, knowledge-distillation, vision-transformer, autoregressive-image-generation, image-super-resolution, differentiable-rendering, safety-filter, score-distillation-sampling, image-to-3d-generation, neural-radiance-field, video-generation, world-model, noise-schedule, physics-informed-neural-network] status: draft updated_at: 2026-11-01 --- ## Overview A sampler repeatedly predicts a correction at each step. Classifier-free guidance (CFG) makes two predictions at each step, one with the condition (usually a text prompt) and one with the condition removed, and combines them with a weight. The output is pushed toward the condition, and the weight sets how hard [arxiv:2211.12572 §4] [eq. 5]. The shared invariant: at inference, every step needs both branches, and the unconditional branch only exists because the model was trained with the condition sometimes removed. GLIGEN states this directly: caption and grounding tokens are dropped during training, which is what enables CFG [arxiv:2301.07093 app. A]. eDiff-I and MLD do the same with their own conditions [arxiv:2211.01324 §4.2] [arxiv:2212.04048 §3.3]. Composer and SDFusion do it for several conditions at once [arxiv:2302.09778 §2.3] [arxiv:2212.04493 §3.3]. DreamPose gives its image and pose inputs separate weights [arxiv:2304.06025 §4.4], and VASA-1 gives each of its conditions a weight in one sum [arxiv:2404.10667 §3.2]. Transformer and autoregressive generators reuse the same recipe: MDM drops its text or action condition for 10% of samples [arxiv:2209.14916 §3], DiT trains a learned null label [arxiv:2212.09748 §3.1], and SDXL drops its conditions 10% of the time [arxiv:2307.01952 app. Diffusion Models]. unCLIP applies it only in its pixel decoder, and its prior is not guided [arxiv:2204.06125 §2.1] [§5.5]. The same idea is not limited to diffusion. MoMask applies it to the logits of a masked token transformer [arxiv:2312.00063 §3.4], and Infinity applies it to the per-bit logits of a bitwise autoregressive model [arxiv:2412.04431 §4.7]. Muse applies the same weighted sum to the logits of its masked transformer [arxiv:2301.00704 §2.7] [eq. 1]. The condition need not come from a user. A physics-informed diffusion model uses the gradient of a PDE residual at the noisy sample as its condition, and drops that condition in training so that an unconditional branch exists [arxiv:2211.14680 alg. 1] [eq. 6]. Video and 3D pipelines use the same weighting. MagicAnimate guides a pose-driven video model with a scale of 7.5 [code: configs/prompts/animation.yaml#L15], and Make-It-3D sets the guidance weight to 10 for novel views in score distillation [arxiv:2303.14184 §4.1]. Drive-WM drops each of its conditions 20% of the time in training and samples with a scale of 5.0 [arxiv:2311.17918 app. B.1]. Layout-conditioned diffusion uses the same recipe for boxes and class labels. LayoutDiffusion, trained from scratch, replaces its whole layout with a null layout in 20% of training samples and guides with one scale [arxiv:2303.17189 §3.4]. Some work changes the guided output itself: the common-schedules paper rescales its standard deviation so that strong guidance does not overexpose the image [arxiv:2305.08891 §3.4]. ## Variants ### Standard CFG with a conditional weight The guided prediction is a weighted sum of the conditional and unconditional predictions, with the weight on the conditional term. In this notation a weight of 1 gives the conditional prediction alone, so larger values push harder toward the prompt [arxiv:2211.12572 §4] [eq. 5]. Card: [PnP](/papers/2211.12572). Composer uses the same form, with ω on the conditional term [arxiv:2302.09778 §2.1] [eq. 2]. Card: [Composer](/papers/2302.09778). Infinity applies it to the bit logits with a scale of 4 [arxiv:2412.04431 §4.7]. Card: [Infinity](/papers/2412.04431). A derivation-only tutorial writes the same form, with γ on the conditional score and γ = 1 giving the conditional model. It presents the form as one of two guidance routes, next to classifier guidance [arxiv:2208.11970 §5]. Card: [Understanding Diffusion Models](/papers/2208.11970). The same form runs in a VAE latent for text-to-motion, with scale s = 7.5 on the conditional term [arxiv:2212.04048 §3.3] [eq. 4]. Card: [MLD](/papers/2212.04048). ### Scale on the difference in transformer denoisers This form starts from the unconditional prediction and adds the scaled difference to the conditional one, so s = 1 gives the conditional prediction alone. MDM uses it for text and action conditions, with s = 2.5 for text-to-motion [arxiv:2209.14916 §3] [eq. 7] [§4.1]. Card: [MDM](/papers/2209.14916). DiT uses it for class labels in a latent transformer, with the unconditional branch set by a learned null embedding [arxiv:2212.09748 §3.1]. Its tables report DiT-XL/2 at s = 1.25 and s = 1.50 [arxiv:2212.09748 tab. 2] [tab. 3]. Card: [DiT](/papers/2212.09748). In both papers the same network produces the unconditional prediction, so no second model is needed. SDXL is a latent text-to-image model with two text encoders. It writes the guided prediction with w on the unconditional term, and w = 0 gives the conditional prediction alone [arxiv:2307.01952 app. Diffusion Models]. Its released guider uses the scale-on-difference form with s = 1 + w, so the demo default of 5.0 is w = 4 [code: sgm/modules/diffusionmodules/guiders.py#L24-L30]. Card: [SDXL](/papers/2307.01952). ### Negative prompting The unconditional term is replaced by a blend of the empty prompt and a negative prompt. The blend weight alpha starts at a set value and decreases over the sampling steps. The authors describe the effect as strong for textureless inputs such as silhouettes and minor for natural images [arxiv:2211.12572 §4] [eq. 6] [app. C]. Card: [PnP](/papers/2211.12572). Object removal in PowerPaint uses the same slot. The object prompt takes the unconditional place and the context prompt the conditional one, with weight w on the context prompt [arxiv:2312.03594 §3.2] [eq. 5]. The paper gives no value for w [§3.2]. Card: [PowerPaint](/papers/2312.03594). StableSR applies the same form to a frozen Stable Diffusion 2.1-base super-resolution model. Its unconditional term comes from the negative prompt and its conditional term from the null prompt, and a scale of 1 gives the default [arxiv:2305.07015 §5.1] [eq. 8]. Its sweep uses two negative prompts at scales 0.0, 2.5, 5.0 and 7.5 [arxiv:2305.07015 tab. 6]. With one prompt, moving the scale from 0.0 to 7.5 raises RealSR MUSIQ from 64.86 to 67.26 and CLIP-IQA from 0.6176 to 0.6323, and lowers PSNR from 24.80 to 23.53 and SSIM from 0.7097 to 0.6673; LPIPS rises from 0.3105 to 0.3399 [tab. 6]. The text notes that 7.5 can over-sharpen [arxiv:2305.07015 §5.1] [fig. 16]. Card: [StableSR](/papers/2305.07015). ### Whole-layout null token for box and class conditions LayoutDiffusion takes one scale for its layout. Its null branch is a whole-layout token with box (0, 0, 1, 1) and class 0, with padding slots elsewhere [arxiv:2303.17189 §3.1] [§3.4]. Its main text writes the guided prediction as (1 − s) times the unconditional prediction plus s times the conditional one, the weight-1 form under Pitfalls. Its appendix and shipped code write cond + s (cond − uncond) instead, the weight-0 form, so the code default of s = 1.0 is s = 2.0 in the main-text form [arxiv:2303.17189 app. B.2] [code: scripts/classifier-free_sample_for_layout.py#L125]. Card: [LayoutDiffusion](/papers/2303.17189). The model is trained from scratch on box-labelled data and has no text prompt [arxiv:2303.17189 §5]. ### Rescaling the guided output The common-schedules paper writes its guidance in the negative-prompt form, x_neg + w (x_pos − x_neg), with w = 7.5 [arxiv:2305.08891 §3.4] [eq. 13]. It then matches the standard deviation of the guided output to that of the positive output, and blends the rescaled and the plain guided output with factor φ [§3.4] [eq. 14] to [eq. 16]. The paper uses φ = 0.7 and reports 0.5 to 0.75 as giving the most appealing results. φ = 0 overexposes [§3.4] [§5.3] [fig. 6]. Its eq. 13 does not say whether the outputs are noise, velocity or clean-image predictions, and that choice sets the spread being matched [§3.4]. Card: [Common Diffusion Noise Schedules](/papers/2305.08891). The FID of 21.66 that the paper reports is for all four of its fixes together, so this rescale is not measured alone [arxiv:2305.08891 tab. 3]. ### Per-condition weights on one model When a model takes several inputs, each input can get its own weight. SDFusion starts from the unconditional prediction and adds, for each modality, its weight times the difference between the prediction with only that modality present and the unconditional one [arxiv:2212.04493 §3.3] [eq. 4]. A larger weight makes that input more influential. Card: [SDFusion](/papers/2212.04493). The released code runs this form only in one branch, and its default multi-modal path scales the input features instead; see Contradictions. ### Leave-one-out weights over several conditions VASA-1 guides all of its conditions at once. The guided prediction is the full-condition prediction scaled by one plus the sum of the weights, minus, for each condition, its weight times the prediction with that condition emptied [arxiv:2404.10667 §3.2] [eq. 2]. With one condition it reduces to the (1 + w) form under Pitfalls. Card: [VASA-1](/papers/2404.10667). Its defaults are 0.5 for audio and 1.0 for gaze [arxiv:2404.10667 §4]. Raising the audio weight from 0.5 to 2.0 raises the sync confidence from 7.957 to 8.295, while the authors report that higher audio weights also amplify mouth motion and jitter [arxiv:2404.10667 tab. 4] [§4.3]. ### Separate weights for image and pose DreamPose applies two weights in two steps. It starts from the prediction with no conditions, adds the image term with weight s_I, then adds the pose term with weight s_p, measured against the image-only prediction [arxiv:2304.06025 §4.4] [eq. 4]. Three predictions per step are needed [§4.4]. The authors report that a large s_I keeps the appearance and a large s_p keeps the pose, and that separating the two avoids overfitting to the one pose seen in subject fine-tuning [§4.4] [fig. 7]. Card: [DreamPose](/papers/2304.06025). ### Guidance between two condition sets Composer weights two condition sets rather than a condition and nothing. Its second set is emphasised with ω and the first is suppressed with 1 − ω, and conditions shared by both keep weight 1 [arxiv:2302.09778 §2.1] [eq. 3]. This form lets an edit move from one set of conditions to another, together with DDIM inversion of the input image [arxiv:2302.09778 §2.1] [§3.2]. Card: [Composer](/papers/2302.09778). ### Text and image terms in a subgoal generator TaKSIE generates the next subgoal image from the current image, the task instruction and a progress embedding. The generator is a diffusion model that takes the current image as a condition. The paper reports ControlNet and InstructPix2Pix as options [arxiv:2410.11013 §3.4.1] [app. D]. It uses two guidance terms, one for the instruction with a negative prompt made by swapping the instruction's antonym, and one for the current image. Both scales are 2.5 [arxiv:2410.11013 §3.4.1] [§4.2.3]. Card: [TaKSIE](/papers/2410.11013). The paper does not write out how the two terms combine. The released pipeline uses an InstructPix2Pix-style form with three predictions [code: taksie/models/pipeline_controlnet.py#L1013-L1022]. The ablation without the negative prompt lowers the CALVIN average length from 3.18 to 3.07 [arxiv:2410.11013 tab. 2]. ### Guidance with a zero null on a latent hand-pose generator FastGrasp's paper does not mention guidance, and its inference section describes DDIM sampling only [arxiv:2411.14786 §3.4]. The released sampler still combines a conditional and an unconditional prediction in the (1 + w) form, with the unconditional input set to zeros [code: network/diffusion/ddim.py#L135] [code: network/diffusion/ddim.py#L170-L171]. The weight is 2 in the HO-3D config and 0.5 in the GRAB and OakInk evaluation configs [code: config/grab/ho3d.json#L3] [code: config/grab/eval_adapt.json#L3] [code: config/oakink/eval_adapt.json#L8]. Card: [FastGrasp](/papers/2411.14786). The released training code does not drop the condition, so this zero branch is one the model never trains on (see Pitfalls). ### Blending two models DiffusionBlending takes the output of a first model and adds a scalar s times the difference to a second model's output, with both models conditioned on different signals [arxiv:2303.01418 §3.3] [eq. 3]. The authors call it a generalisation of CFG, and they use s = 0.5 for two signals [§4.3]. Card: [Human Motion Diffusion](/papers/2303.01418). Each control model is fine-tuned from the same base MDM [arxiv:2303.01418 §4.3]. ### Two-model guidance for early steps SINE mixes a model fine-tuned on one image into the conditional term. Its guided prediction is a weighted sum of the unconditional prediction and a mix of the pretrained and fine-tuned conditional predictions, with the weight w on the mix, while the step index is above a threshold K. Below K it runs the pretrained model alone [arxiv:2212.04489 §3.2] [eq. 2] [eq. 3]. The paper sets K = 400 and puts v = 0.7 on the pretrained term of the mix [arxiv:2212.04489 §4]. It gives no guidance scale; its notebook uses 7.5 [code: SINE.ipynb#L251]. Card: [SINE](/papers/2212.04489). ### Logit-space guidance for token models A masked or bitwise autoregressive generator outputs logits over a vocabulary rather than a noise prediction. Guidance is applied to those logits before sampling. MoMask writes the paper's form with (1 + s) on the conditional logits and s on the unconditional ones [arxiv:2312.00063 §3.4] [eq. 5]. Its code uses the other form, with the same weight on the conditional term [code: models/mask_transformer/transformer.py#L321-L323]. Infinity guides each bit's two logits the same way, with cfg = 4 [arxiv:2412.04431 §4.7]. Cards: [MoMask](/papers/2312.00063), [Infinity](/papers/2412.04431). Muse writes its logit guidance with (1 + t) on the conditional logits and t on the unconditional ones, and it lets a negative prompt replace the unconditional branch [arxiv:2301.00704 §2.7] [eq. 1]. Its editing run uses scale 4 with top-k sampling at k = 3 [arxiv:2301.00704 §3.3.2]. Card: [Muse](/papers/2301.00704). ### Guidance that ramps over the scales VAR generates one token map per scale, from coarse to fine, and guides the logits of each map. Its released sampler uses a scale that grows with the scale index, from 0 at the first map to the full value at the last [code: models/var.py#L161-L173]. The paper reports one fixed ratio of 2.0 for its ablation and does not describe a schedule [arxiv:2404.02905 tab. 3] [§3.2]. Card: [VAR](/papers/2404.02905). Muse ramps its guidance scale linearly over the sampling steps, so early tokens are less guided than late ones [arxiv:2301.00704 §2.7]. The start and end values are not stated [§2.7]. Card: [Muse](/papers/2301.00704). ### Safety guidance away from a text concept Safe latent diffusion adds a third prediction, conditioned on a text description of unwanted content, and moves the prompt estimate away from it. The move is scaled per element by a term that is zero where the prompt estimate already exceeds the concept estimate by at least a threshold λ [arxiv:2211.05105 §3] [eq. 3] [eq. 4] [eq. 5]. The term applies only after a warm-up of δ steps, and a momentum term carries it across steps [arxiv:2211.05105 §3] [alg. 1]. Card: [Safe Latent Diffusion](/papers/2211.05105). The paper gives four presets, from weak to max, for warm-up, concept scale, threshold and momentum [arxiv:2211.05105 tab. 1]. Author-reported: on the I2P prompt benchmark the inappropriate-image probability for Stable Diffusion 1.4 falls from 0.39 to 0.09 at the strongest preset, and COCO FID-30k rises from 14.43 to 18.76 [arxiv:2211.05105 tab. 2] [tab. 3]. Negative prompting is the baseline the paper compares against [§4] [tab. 2]. ### Guidance with a planner that generates video UniPi plans with a text-conditioned video diffusion model. Its guided denoiser uses the (1 + w) form with w on the unconditional term, so w = 0 gives the conditional prediction alone [arxiv:2302.00111 §2.3]. The paper does not state w, the conditioning dropout rate, or the sampler [§2.3]. Card: [UniPi](/papers/2302.00111). ### Guidance inside distillation A one-step student can be trained against a guided teacher. In DMD, the regression pairs come from the guided model and the real score uses the guided model's mean prediction, while the fake score is unchanged. The guidance scale is fixed during training: 3 for the LAION 6.25+ set and 8 for LAION 6+ [arxiv:2311.18828 §3.4] [§4.3]. The paper does not write the exact guided mean [§3.4]. Card: [DMD](/papers/2311.18828). The guidance scale is therefore part of the student, not a sampler setting that can change later. ### Guidance applied to overlapping views MultiDiffusion runs one pretrained text-to-image model on overlapping windows of a larger canvas, and it averages the outputs per pixel at each step [arxiv:2302.08113 §3]. The paper's update step has no guidance term, but the released code applies CFG with scale 7.5 and a negative prompt to every window call, batched as two passes [arxiv:2302.08113 §3] [code: panorama.py#L123-L130]. Card: [MultiDiffusion](/papers/2302.08113). ### Guidance over a video with a reference branch MagicAnimate guides a pose-driven video UNet with the scale-on-difference form, the unconditional prediction plus 7.5 times the difference [code: configs/prompts/animation.yaml#L15] [code: magicanimate/pipelines/pipeline_animation.py#L586]. Predictions from overlapping frame windows are averaged before the guidance step [arxiv:2311.16498 §3.3]. Card: [MagicAnimate](/papers/2311.16498). Its unconditional branch differs from the conditional one in more than the prompt. In reference attention, the unconditional half attends only to its own tokens and receives none of the reference features [code: magicanimate/models/mutual_self_attention.py#L243-L256]. The paper does not describe this, so the guidance also changes how the reference is used. ### Guidance inside score-based 3D optimisation Score Jacobian Chaining optimises a voxel radiance field so that its renders receive the score of a 2D denoiser, chained through a differentiable renderer [arxiv:2212.00774 §4]. The paper states the guidance scale and not the guidance form. The scale is 100 for text-to-3D with Stable Diffusion [arxiv:2212.00774 app. A3] and 3.0 and 10.0 in its 2D tests [§5.1]. The released code runs classifier-free guidance with an empty negative prompt [arxiv:2212.00774 §5] [code: adapt_sd.py#L137-L141]. Card: [SJC](/papers/2212.00774). The 3D value is ten times the largest 2D value tested, so the two settings are not interchangeable. Make-It-3D uses the same kind of guidance inside score distillation, applied to a NeRF. Novel views get the distillation loss with guidance weight ω = 10, and the timestep is drawn between 200 and 600 [arxiv:2303.14184 §4.1]. The card's reading of the mix puts (1 + ω) on the conditional prediction and ω on the unconditional one, so it is the weight-0 form described in Pitfalls [arxiv:2303.14184 §4.1]. The reference view gets a masked pixel loss instead. At timesteps up to 400, a CLIP-based loss replaces the distillation term [arxiv:2303.14184 §3.2] [app. C.1]. Card: [Make-It-3D](/papers/2303.14184). ### Attention methods that sit next to guidance DenseDiffusion changes attention scores early in sampling and does not train the model [arxiv:2308.12964 §3.2]. Its equations never mention guidance. The code runs the modulation on the conditional half of a batch that also holds the unconditional prompt, so the guidance step and the attention edit are coupled [code: gradio_app.py#L147-L163]. The paper states no guidance scale, and the code does not set one [code: gradio_app.py#L270]. Card: [DenseDiffusion](/papers/2308.12964). ### Guidance on the decoder of a two-stage model unCLIP maps a caption to a CLIP image embedding with a prior, then renders the embedding with a diffusion decoder. Only the decoder is guided; the paper does not apply guidance to the prior [arxiv:2204.06125 §5.5]. The guidance scale of 1.25 is the value the paper reports for FID [arxiv:2204.06125 tab. 2], and the scales used for human evaluation are not stated. This card does not record the weight convention, so check it against the paper before copying a value. The authors report that unCLIP keeps its content as the scale rises, while GLIDE's semantics converge [arxiv:2204.06125 fig. 9], and that at the high scales used for photorealistic output it is more diverse than GLIDE at similar photorealism and caption similarity [arxiv:2204.06125 fig. 10]. Card: [unCLIP](/papers/2204.06125). ### Physics-residual gradient as the condition The condition here is not a user input. The network takes c = dr_t/dx_t, the gradient of a PDE residual at the noisy sample, as an extra input [arxiv:2211.14680 eq. 6] [alg. 1]. Training drops c with probability p_u, so the same network also gives an unconditional prediction [alg. 1]. Sampling mixes the two outputs with (w + 1) on the conditional one and −w on the unconditional one [arxiv:2211.14680 alg. 2] [eq. 8]. Card: [Physics-informed Diffusion Model](/papers/2211.14680). The paper's Table 3 compares this learned form with a direct gradient step (eq. 9) and with no physics term. On the sparse task, the learned form has the lowest equation residual at each printed iteration. At the third iteration it reports 0.27, against 0.38 for the direct step and 0.39 for the baseline. On the 8x task, at its last printed step count, it reports 0.20 against 0.99 and 1.47, while L2 stays within 0.04 across the three rows [arxiv:2211.14680 tab. 3]. The table gives one number per cell, with no seeds or spread, so the size of the gaps is not quantified. ### Training-time condition dropout This is not a sampling rule. The model is trained with each condition dropped at a fixed rate, so that the unconditional prediction exists at sampling time. A model that takes new conditions, here bounding boxes with phrases, is trained this way for each of them. The GLIGEN paper uses 10% for both caption and grounding tokens [arxiv:2301.07093 app. A]. Card: [GLIGEN](/papers/2301.07093). unCLIP's decoder drops its CLIP image embedding and its caption, with the caption dropped 50% of the time [arxiv:2204.06125 §2.1]. The embedding rate is 10% in that section and 5% in §5.1; see Contradictions. Its prior drops the caption 10% of the time [arxiv:2204.06125 §2.2]. Card: [unCLIP](/papers/2204.06125). A model with several embeddings drops each one on its own. eDiff-I uses independent rates of 0.25 for T5, 0.2 for CLIP text and 0.9 for CLIP image embeddings, and dropping all three gives the unconditional branch [arxiv:2211.01324 §4.2] [tab. 7]. Card: [eDiff-I](/papers/2211.01324). MLD uses one rate of 0.1 for its single condition [arxiv:2212.04048 §3.3]. Composer drops each condition independently with probability 0.5. It also drops all conditions with probability 0.1 and keeps all with probability 0.1, and it uses 0.7 for its intensity representation, which otherwise carries most of the image information [arxiv:2302.09778 §2.3]. Card: [Composer](/papers/2302.09778). Other models set their own rates: ComMDM drops the text with 10% [arxiv:2303.01418 §3.2], MoMask with 10% for its masked transformer [arxiv:2312.00063 §3.4], and Infinity's released default is 0.1 [code: infinity/utils/arg_util.py#L80]. LayoutDiffusion drops its whole layout in 20% of training samples in every run [arxiv:2303.17189 app. C.1] [tab. 4]. Muse drops the text for 10% of training samples, and does not say what replaces the dropped text [arxiv:2301.00704 §2.7]. SDFusion's paper gives no rate, while its multi-modal code keeps each input with probability 0.5 [arxiv:2212.04493 §3.3]. DreamPose drops the pose alone, the image alone, and both together, with 5% for each, so 15% of training samples lose at least one condition [arxiv:2304.06025 supp. §1]. VASA-1 drops each condition with probability 0.1 and drops its two previous-window conditions with 0.5, which covers the first window that has no history [arxiv:2404.10667 §3.2]. Card: [VASA-1](/papers/2404.10667). The transformer and token models add their own rates. DiT's paper says only that labels are dropped at random, and its code uses 0.1 [arxiv:2212.09748 §3.1] [code: models.py#L158]. VAR's paper does not mention label dropout, and its code replaces 10% of the labels with a null class [code: models/var.py#L201]. Card: [VAR](/papers/2404.02905). Drive-WM trains its multiview driving video model with each condition dropped 20% of the time, which provides the unconditional prediction used at sampling [arxiv:2311.17918 app. B.1]. The appendix does not say what replaces a dropped condition [app. B.1]. Card: [Drive-WM](/papers/2311.17918). The physics-residual model drops its residual-gradient condition with probability p_u. The paper does not give p_u, and the released training code defaults to 0.1 [arxiv:2211.14680 alg. 1] [code: train_ddpm/functions/losses.py#L77]. Card: [Physics-informed Diffusion Model](/papers/2211.14680). ## When to use which | Situation | Use | Why | |---|---|---| | Off-the-shelf model, you control the sampler only | Standard CFG with a conditional weight | Needs no training. The unconditional branch is a sampler-side choice. [arxiv:2211.12572 §4] | | Input carries little meaning on its own (silhouettes, flat shapes) | Negative prompting on top of CFG | The authors report it helps most for primitive inputs [arxiv:2211.12572 §4] [app. A.1] | | Natural images with a text prompt | Standard CFG; negative prompting is optional | The authors report minor effect on natural images [arxiv:2211.12572 §4] | | Latency or compute budget is tight | Standard CFG; avoid negative prompting | Negative prompting adds passes per step, see Pitfalls | | Diffusion in a latent space over a non-image signal (text to motion) | Standard CFG in the latent, with a scale and dropout sweep on your data | The paper's sweep shows scale trades FID against R-Precision, see Weight values [arxiv:2212.04048 tab. 11] | | You train the model and want new conditions usable with CFG | Dropout of each condition during training | Without dropped conditions there is no unconditional prediction to guide against [arxiv:2301.07093 app. A] | | Several text or image embeddings feed one model | Independent dropout per embedding, one rate each | eDiff-I drops each embedding alone at its own rate [arxiv:2211.01324 §4.2] [tab. 7] | | Several inputs that should each have a strength you can set at sampling | Per-condition weights (SDFusion eq. 4) | Each input gets its own weight; the cost grows with the number of inputs [arxiv:2212.04493 §3.3] [eq. 4] | | Changing one aspect of an existing image under new conditions | Two condition sets with ω (Composer eq. 3), after inversion | Interpolates between the old and new conditions [arxiv:2302.09778 §2.1] [§3.2] | | Edit one real image from one prompt, with a copy of the model fine-tuned on that image | Two-model guidance above a step threshold (SINE eq. 3) | Only the early steps use the fine-tuned term; the paper's K = 400 and v = 0.7 differ from the CLI default, see Contradictions [arxiv:2212.04489 §4] [§4.3] | | Removing an object from a masked area with a text inpainting model | Object prompt as the negative term (PowerPaint eq. 5) | The paper gives no weight; the app defaults to 7.5 and recommends 10 or higher for removal [arxiv:2312.03594 §3.2] [code: app.py#L620] | | Two fine-tuned models for different signals, no joint training | Blending with one scalar (DiffusionBlending) | Each signal keeps its own model; the paper's control results use s = 0.5 [arxiv:2303.01418 §4.3] | | Discrete token generator (masked or bitwise) | Guidance on the logits, with the scale from the paper's own setting | MoMask and Infinity both guide logits; the scales differ by task and model [arxiv:2312.00063 §4] [arxiv:2412.04431 §4.7] | | A differentiable physical constraint is known, and you can train the generator on the target data | Physics-residual gradient as a learned condition, with dropout (2211.14680 alg. 1 and 2) | In the authors' ablation the learned form gives the lowest equation residual; the paper's w and p_u are not stated [arxiv:2211.14680 tab. 3] [alg. 2] | | Large canvas or panorama from a model trained on squares | Run the model per window with guidance, then average (MultiDiffusion) | The paper reports the method without training; cost grows with the number of windows [arxiv:2302.08113 §4.1] [app. B.1] | | Several models in a cascade | Tune the scale per stage | Settings differ by stage: Imagen uses 1.25 for the base and 1.0 for its 256 stage [arxiv:2304.08466 §4.2]; Composer uses 3.0 for its prior and base [arxiv:2302.09778 §3.5] | | Generating training data for a classifier | Choose the guidance by downstream accuracy as well as FID | The Imagen paper selects settings on FID, IS and classifier accuracy together [arxiv:2304.08466 §4.2] [§5.2] | | Grounding (boxes) added to a frozen model | Keep the 10% dropout the paper uses, then tune at sampling | The paper's own grounding setting uses 10%; the code uses a different rate, see Contradictions [arxiv:2301.07093 app. A] | | Several conditions, each needing its own strength (for example audio and gaze) | Leave-one-out weights per condition (VASA-1 eq. 2) | Each condition has its own weight; the paper's defaults are 0.5 for audio and 1.0 for gaze [arxiv:2404.10667 §4] [tab. 4] | | One photo animated by a pose sequence, with per-subject fine-tuning | Separate image and pose weights (DreamPose eq. 4) | Appearance and pose can be tuned on their own; the cost is three predictions per step [arxiv:2304.06025 §4.4] | | Joint audio and video from one text prompt | The single scale the paper reports, 7.0, with its sampler | The paper's ablations in §5.3 vary the modules and the prior, not the guidance scale [arxiv:2503.23377 app. C.1] [§5.3] | | Two-stage text-to-image model: an embedding prior and a pixel decoder | Guide the decoder only; leave the prior unguided | The paper reports that guiding the prior hurt results [arxiv:2204.06125 §5.5]. Its decoder keeps more diversity than guided GLIDE at high scales [arxiv:2204.06125 fig. 10] | | Non-image signal from a transformer that predicts the clean signal, with a text or action condition | Scale on the difference with 10% null-condition training (MDM eq. 7) | The paper uses s = 2.5 for text-to-motion; the released action models set dropout to 0 and cannot use guidance [arxiv:2209.14916 §4.1] [code: utils/parser_util.py#L22-L23] | | Class-conditional latent diffusion with a transformer backbone | Scale on the difference with a learned null label (DiT §3.1) | The paper reports FID at s = 1.25 and 1.50; the code guides only three latent channels by default [arxiv:2212.09748 tab. 2] [app. A] | | Text-to-image with two text encoders and size or crop conditions | Weight-on-unconditional form at the paper's scale (SDXL) | The paper's comparisons use cfg 8.0; the demo uses 5.0, which is w = 4 in that form [arxiv:2307.01952 §2.2] [code: sgm/modules/diffusionmodules/guiders.py#L24-L30] | | One-step student of a guided teacher | Guided regression pairs and guided real score at a fixed scale (DMD) | The scale is fixed in training, 3 for LAION 6.25+ and 8 for LAION 6+; the student carries it [arxiv:2311.18828 §3.4] [§4.3] | | Autoregressive generator over scales with a class label | Ramped logit guidance (VAR) | The paper's ablation uses 2.0 without a schedule; the released sampler ramps it per scale [arxiv:2404.02905 tab. 3] [code: models/var.py#L172] | | Text-to-3D from a 2D diffusion model, with no 3D training data and one optimisation per object | Score-based optimisation with CFG in the denoiser (SJC) | Needs no 3D data. The paper reports about 25 minutes per object on one A6000 and shows results only qualitatively [arxiv:2212.00774 app. A3] [§5.2] | | Super-resolution with a frozen text-to-image model, where a negative prompt should steer quality | Negative prompt in the eq. 8 form (StableSR) | A higher scale raises the no-reference scores (CLIP-IQA, MUSIQ) and lowers PSNR and SSIM, while LPIPS rises, so the scale is a trade-off [arxiv:2305.07015 tab. 6] | | Suppress unwanted content during sampling, on a model with CFG and no retraining, where the unwanted content can be described in words | Safety guidance with a text concept (SLD eq. 3) | No classifier or retraining. The paper's gain on I2P comes with a higher FID; both are author-reported [arxiv:2211.05105 tab. 2] [tab. 3] | | Masked-token text-to-image model that decodes in parallel over a few steps | Logit guidance that ramps over the steps (Muse eq. 1) | The paper ramps the scale linearly but does not give its endpoints, so the schedule has to be set and reported [arxiv:2301.00704 §2.7] | | Instruction-conditioned image generator that must stay close to the current image | Two guidance terms, one for the text and one for the image, each with its own scale (TaKSIE) | The paper's ablation without the negative prompt lowers CALVIN average length from 3.18 to 3.07 [arxiv:2410.11013 tab. 2] [§4.2.3] | | Object-conditioned latent diffusion that you train yourself, with guidance at sampling | Train with condition dropout first, then apply the (1 + w) form | Without dropped conditions the unconditional branch is never trained. FastGrasp's released code samples with guidance and drops no condition [code: network/diffusion/ddim.py#L73-L74] | | Pose-driven animation of one reference image with a video diffusion model | Scale-on-difference guidance at 7.5 over overlapping windows (MagicAnimate) | The released config uses 7.5. The reference-free unconditional branch is code-only, so copy it only if you want the released behaviour [code: configs/prompts/animation.yaml#L15] [code: magicanimate/models/mutual_self_attention.py#L243-L256] | | Single image to 3D with a 2D diffusion prior and no 3D training data | Score distillation with guidance 10 on novel views, and a CLIP term at small timesteps (Make-It-3D) | The paper's weight is 10 [arxiv:2303.14184 §4.1]; the released refine stage uses 5 [code: nerf/utils.py#L880] | | Multiview video world model with several conditions, trained from scratch | Training-time dropout at 20% per condition, then sampling with guidance 5.0 (Drive-WM) | Both values are in the appendix; the guidance form and the replacement for a dropped condition are not stated [arxiv:2311.17918 app. B.1] | | Box and class layout control, and you can train from scratch on labelled boxes | Whole-layout null token with 20% dropout and the (1 + s) form (LayoutDiffusion) | The paper reports FID 15.61 on COCO-Stuff 256 and 15.63 on VG 256, author-reported. It trains from scratch and names text-image pretraining as future work [arxiv:2303.17189 tab. 1] [§5] | | Strong guidance overexposes a fixed model, and you can change the sampler | Standard-deviation rescale of the guided output with φ = 0.7 (common-schedules paper) | The paper's φ sweep is visual, and φ = 0 overexposes [arxiv:2305.08891 §5.3] [fig. 6]. The rescale is not measured alone [tab. 3] | | A model cannot make very dark or very bright images, and you can retrain it | The four fixes: zero terminal SNR, v prediction, trailing sample steps, CFG rescale | Reported FID 21.66 against 23.76 for the official model, but the training data also differ [arxiv:2305.08891 tab. 3] [§4.2] | Weight values from the cited papers are settings for their own tasks, not general recommendations. PnP reports 15.0 for real-image translation and 7.5 for generated guidance images [arxiv:2211.12572 app. C]. GLIGEN does not state a guidance scale in the paper [arxiv:2301.07093 app. A]. Composer states 3.0 for its prior and base [arxiv:2302.09778 §3.5]. Infinity uses 4 on all scales [arxiv:2412.04431 tab. 9]. MoMask uses 4 for HumanML3D and 2 for KIT-ML, with 5 for its residual transformer [arxiv:2312.00063 §4]. SDFusion's released values differ by model: 1.0 for single-view, 3.0 for text, and 1.0 for multi-modal [arxiv:2212.04493 §3.3] [code: models/sdfusion_txt2shape_model.py#L76]. VASA-1 uses 0.5 for audio and 1.0 for gaze, and states no scale for head distance or emotion [arxiv:2404.10667 §3.2] [§4]. DreamPose states no values for its reported results [arxiv:2304.06025 §4.4]. JavisDiT uses 7.0 [arxiv:2503.23377 app. C.1]. SINE's paper gives no scale; its notebook uses 7.5 and its README 10 [code: SINE.ipynb#L251] [code: README.md#L89]. PowerPaint's paper gives no w, and its app defaults to 7.5 [code: app.py#L677-L681]. The survey gives no scale of its own; it says that for Stable Diffusion w = −1 gives empty output and w above 10 gives saturation [arxiv:2310.07204 §3.3]. SJC uses 100 for text-to-3D and 3.0 and 10.0 for its 2D tests [arxiv:2212.00774 app. A3] [§5.1], and StableSR's negative-prompt sweep runs up to 7.5 [arxiv:2305.07015 tab. 6]. Make-It-3D uses 10 on novel views [arxiv:2303.14184 §4.1], MagicAnimate's released config uses 7.5 [code: configs/prompts/animation.yaml#L15], and Drive-WM uses 5.0 [arxiv:2311.17918 app. B.1]. The guidance form for Drive-WM is not stated. LayoutDiffusion's code default is s = 1.0 in the (1 + s) form, and its appendix figures use 1.0, 0.6 and 0.5 [arxiv:2303.17189 app. A.4] [app. A.5]. The common-schedules paper uses w = 7.5 [arxiv:2305.08891 §3.4]. MDM's text-to-motion scale is 2.5 [arxiv:2209.14916 §4.1], against 7.5 for MLD on a related task, so the two numbers come from different models and should not be copied across. DiT's 1.25 and 1.50 correspond to w = 0.25 and 0.5 in the weight-on-unconditional form [arxiv:2212.09748 tab. 2]. VAR's ablation uses 2.0 and its FID command 1.5 [arxiv:2404.02905 tab. 3] [code: README.md#L155]. DMD uses 3 and 8 for its two text-to-image sets [arxiv:2311.18828 §4.3]. Muse uses 4 for editing [arxiv:2301.00704 §3.3.2], and its base sampling scale is not stated [§2.7]. SLD does not state a guidance scale for its I2P and quality runs; its ablation figures use 15 [arxiv:2211.05105 app. I], and the released code defaults to 7.5 [code: src/sld/sld_pipeline.py#L212-L213]. TaKSIE uses 2.5 for both its text and image terms [arxiv:2410.11013 §4.2.3]. FastGrasp states no weight; its released configs use 2 for HO-3D and 0.5 for GRAB and OakInk [code: config/grab/ho3d.json#L3] [code: config/grab/eval_adapt.json#L3]. MLD's sweep over s, at dropout 0.1, gives the lowest FID at s = 2.5 (0.350). R-Precision rises up to s = 7.5, which the paper chooses (FID 0.473) [arxiv:2212.04048 tab. 11]. A FID-only target and a text-match target therefore pick different scales. eDiff-I does not state the guidance weight behind its Table 1 FID numbers. It shows FID–CLIP curves for weights 0 to 10 instead [arxiv:2211.01324 §5.1] [fig. 7]. For classifier training data, Imagen's base guidance was swept from 1.0 to 5.0 [arxiv:2304.08466 §4.2]. Added synthetic data helped ResNet-50 up to 4.8M total images at 256 px and hurt above 6.0M total [arxiv:2304.08466 tab. 4]. ## Implementation sketch Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a `# comment` line above each step (with its citation), short trailing comments only for shapes or units. ```pseudocode # Standard CFG, conditional-weight form [arxiv:2211.12572 §4] [eq. 5] # Input: x (noisy latent), t (timestep), cond (prompt embedding), # uncond (empty-prompt embedding), w (weight on the conditional term) # Output: eps (noise estimate passed to the sampler) def cfg_eps(x, t, cond, uncond, w): # Two U-Net passes per step: with the prompt and with the empty prompt eps_cond = unet(x, t, cond) eps_uncond = unet(x, t, uncond) # Weighted sum; w = 1 gives the conditional prediction alone [eq. 5] return w * eps_cond + (1 - w) * eps_uncond ``` ```pseudocode # Negative prompting: the unconditional term becomes a blend [eq. 6] # Input: alpha (blend weight, decays over sampling), neg (negative prompt) # Output: eps (noise estimate passed to the sampler) def negative_cfg_eps(x, t, cond, neg, alpha, w): eps_cond = unet(x, t, cond) eps_uncond = unet(x, t, "") eps_neg = unet(x, t, neg) # Blend the empty and negative predictions [eq. 6] eps_mix = alpha * eps_uncond + (1 - alpha) * eps_neg # Same outer weighting as standard CFG [eq. 5] return w * eps_cond + (1 - w) * eps_mix ``` ```pseudocode # Per-condition weights [arxiv:2212.04493 §3.3] [eq. 4] # Input: conds (one per modality, None if absent), weights s_i # Output: eps (noise estimate passed to the sampler) def per_condition_eps(x, t, conds, weights): # Unconditional base: every modality zero-filled [eq. 4] eps_uncond = unet(x, t, empty_context()) eps = eps_uncond for i, (c_i, s_i) in enumerate(zip(conds, weights)): # Prediction with only modality i present [eq. 4] only_i = unet(x, t, context_with_only(i, c_i)) # Add the weighted difference from the base [eq. 4] eps = eps + s_i * (only_i - eps_uncond) return eps ``` ```pseudocode # Condition dropout during training [arxiv:2301.07093 app. A] # Input: caption tokens, grounding tokens, drop_prob (paper: 0.1) def dropout_conditions(caption, grounding, drop_prob): # Replace each condition with its null embedding at random, so the # model learns the unconditional prediction used by CFG at sampling if random() < drop_prob: caption = null_caption if random() < drop_prob: grounding = null_grounding return caption, grounding ``` The per-condition sketch follows eq. 4 and not the released code, whose multi-modal default differs; see Contradictions. ```pseudocode # Guidance with a standard-deviation rescale [arxiv:2305.08891 §3.4] # Equations 13 to 16 of that section # Input: pos, neg (model outputs, same shape), w (weight), phi (rescale factor) # Output: guided output, same shape as pos def cfg_std_rescale(pos, neg, w, phi): # Negative-prompt form of guidance [eq. 13] guided = neg + w * (pos - neg) # Spread of the positive output and of the guided output [eq. 14] # Per-sample std over the non-batch axes; the paper does not state this ratio = std(pos) / std(guided) # Rescaled output, blended with the plain guided output [eq. 15] [eq. 16] rescaled = guided * ratio return phi * rescaled + (1 - phi) * guided ``` Notes on where variants differ: negative prompting changes only the unconditional term and needs one extra pass per step. The rescale above takes the guided output as its input, so the value of w depends on the form it is written in; the paper writes it only in the negative-prompt form. Composer's two-set form and DiffusionBlending change which predictions are combined, not only the weights. DreamPose applies its two weights in sequence rather than as one sum, and VASA-1 subtracts one emptied prediction per condition [arxiv:2304.06025 eq. 4] [arxiv:2404.10667 eq. 2]. The physics-residual sampler [arxiv:2211.14680 alg. 2] fits the same function, with c set to the residual gradient for the conditional call and None for the unconditional one. Token models apply the same weighted sum to logits, before sampling [arxiv:2312.00063 §3.4] [eq. 5]. Feature injection in PnP applies to every branch, see Pitfalls. The training-time version changes the training loop, not the sampler. The latent-space version (MLD) applies the same function to the denoiser's latent, with the empty string as the unconditional text and a zero vector for action labels [arxiv:2212.04048 eq. 4] [code: mld/models/architectures/mld_denoiser.py#L262-L274]. ## Contradictions No direct conflict between the papers' reported numbers. Several places where a paper and its code, or a paper and its config, disagree: - **Dropout rate, GLIGEN.** The paper states 10% for caption and grounding tokens [arxiv:2301.07093 app. A]. The released box config sets a caption probability of 0.5 [code: configs/GoldG+SBU+CC3M+O365_box_text_image.yaml#L65]. The GLIGEN card lists both. - **Guidance weight, PnP.** The paper gives 15.0 for real images [arxiv:2211.12572 app. C]; the shipped real-image config uses 10.0 [code: configs/pnp/pnp-real.yaml#L6]. The PnP card lists both. - **Per-condition weights, SDFusion.** The paper's eq. 4 weights each modality [arxiv:2212.04493 §3.3]. The default multi-modal inference path instead scales the condition features and runs ordinary CFG; eq. 4 is only in a second branch [code: models/sdfusion_mm_model.py#L464-L465] [code: models/sdfusion_mm_model.py#L527-L548]. In that branch, one global scale multiplies every weight, so the effective weight is scale × w_i. - **Dropout rate, SDFusion.** The paper gives no rate for its task-specific encoders [arxiv:2212.04493 §3.3]. The multi-modal code keeps each input with probability 0.5 [code: models/sdfusion_mm_model.py#L390-L394]. - **Scale convention, MoMask.** The paper's eq. 5 and the released code write the same guidance with different weights; the paper's published values match the code's convention [arxiv:2312.00063 §3.4] [eq. 5] [code: models/mask_transformer/transformer.py#L321-L323]. The residual transformer's condition drop is 10% in the paper, and the README's command uses 0.2 [arxiv:2312.00063 §3.4] [code: README.md#L185]. - **Blend, DiffusionBlending.** The paper's blend has a free scale s [arxiv:2303.01418 eq. 3]. The released code averages the models' outputs with equal weights [code: model/model_blending.py#L15-L21]. The two agree at s = 0.5 only. - **Guidance in the second take, DoubleTake.** Figure 3 says the refinement step is conditioned on the neighbouring intervals [arxiv:2303.01418 fig. 3]. The code turns text guidance off for that step [code: utils/sampling_utils.py#L63-L64] [code: utils/sampling_utils.py#L105]. - **Sampling temperature, Infinity.** The paper's best decoding uses tau = 1.00 [arxiv:2412.04431 tab. 9]. The inference script uses 0.5 [code: scripts/infer.sh#L13]. This sits next to the guidance step, so set both explicitly. - **Guidance weights, DreamPose.** The paper states no s_I or s_p for its reported results [arxiv:2304.06025 §4.4]. The code defaults to 0.5 for both [code: test.py#L26-L27], the README example uses 8 and 3 [code: README.md#L17], and the pipeline's sweep uses 0, 3, 5, 7 and 9 [code: pipelines/dual_encoder_pipeline.py#L506-L507]. - **Guidance form, SJC.** The paper states a guidance scale and no form [arxiv:2212.00774 §5]. The released code adds a classifier-free guidance step with an empty negative prompt, and divides the noisy latent by √(1+σ²) before the denoiser [code: adapt_sd.py#L133, L137-L141, L166]. The SJC card lists both. - **Guidance not stated, MultiDiffusion.** The paper's update has no guidance term [arxiv:2302.08113 §3]; the code uses 7.5 with a negative prompt [code: panorama.py#L123-L130]. - **Guidance window, SINE.** The paper applies the fine-tuned term for t > K [arxiv:2212.04489 §3.2]. The CLI default applies it for 1 ≤ t ≤ 400 [code: scripts/stable_txt2img_guidance.py#L190-L191] [code: ldm/models/diffusion/guidance_ddim.py#L455]. The notebook and README match the paper [code: SINE.ipynb#L249-L263] [code: README.md#L94]. The SINE card lists both. - **Mix weight, SINE.** The paper and notebook use v = 0.7 [§4]. The CLI default is 0.5 [code: scripts/stable_txt2img_guidance.py#L192], and the README uses 0.4 [code: README.md#L93]. - **Guidance scope, DenseDiffusion.** The paper's equations never mention guidance [arxiv:2308.12964 §3.2]. The code modulates only the conditional half of the batch [code: gradio_app.py#L147-L163]. The DenseDiffusion card lists both. - **Safety scale, SLD.** The paper's eq. 5 sets the scale to max(1, |φ|) [arxiv:2211.05105 eq. 5] [alg. 1], while its text says the scale is clipped [arxiv:2211.05105 §3]. The released code clips to min(1, |φ|) [code: src/sld/sld_pipeline.py#L478-L484]. The SLD card lists both. - **Momentum sign, SLD.** Algorithm 1 as printed has no plus sign between the two momentum terms [arxiv:2211.05105 alg. 1]. Eq. 8 has one [arxiv:2211.05105 eq. 8], and the code adds the two [code: src/sld/sld_pipeline.py#L494]. - **CLIP-embedding dropout, unCLIP.** §2.1 and its pseudocode zero the CLIP embedding 10% of the time [arxiv:2204.06125 §2.1]. §5.1 and the fig. 8 caption give 5% [arxiv:2204.06125 §5.1] [fig. 8]. The paper does not explain the difference. The unCLIP card lists both. - **Guidance in FastGrasp.** The paper does not mention classifier-free guidance [arxiv:2411.14786 §3.4]. The released sampler applies it [code: network/diffusion/ddim.py#L135], and the released training code drops no condition [code: network/diffusion/ddim.py#L73-L74]. The FastGrasp card lists both. - **Update rule, physics-residual sampler.** Algorithm 2 has no gradient term in its update [arxiv:2211.14680 alg. 2]. The released guided DDIM step subtracts the residual gradient at weight 1 [code: functions/denoising_step.py#L186-L198], and its conditional configs set the mix weight to 0. The 2211.14680 card lists both. - **Guidance combination, TaKSIE.** The paper names two guidance terms, both at scale 2.5, and does not write out how they combine [arxiv:2410.11013 §3.4.1] [§4.2.3]. The released pipeline uses uncond + w_text·(text − image) + w_image·(image − uncond) [code: taksie/models/pipeline_controlnet.py#L1013-L1022]. The TaKSIE card lists both. - **Guidance in the refine stage, Make-It-3D.** The paper's refine stage uses a coarse-style objective [arxiv:2303.14184 §3.3]. The released refine code applies score distillation with guidance 5 [code: nerf/utils.py#L880], while its main default is 10 [code: main.py#L30]. The Make-It-3D card lists both. - **Classifier-free form, LayoutDiffusion.** The main text writes the guided prediction as (1 − s) uncond + s cond [arxiv:2303.17189 §3.4]. Appendix B.2 and the shipped code write cond + s (cond − uncond) [arxiv:2303.17189 app. B.2] [code: scripts/classifier-free_sample_for_layout.py#L125]. The code default s = 1.0 is therefore s = 2.0 in the main-text form. The LayoutDiffusion card lists both. - **Padding mask, LayoutDiffusion.** The paper does not describe a key-padding mask [arxiv:2303.17189 §3.2]. The code has one, but the shipped configs turn it off, so padding slots take part in the attention [code: configs/COCO-stuff_256x256/LayoutDiffusion_large.yaml#L38]. The LayoutDiffusion card lists both. ## Pitfalls - **Weight conventions.** Papers write the same guidance in at least three forms, and a scale copied from one into another changes the strength: - **Weight 1 is conditional-only:** ω·cond + (1 − ω)·uncond. Used by PnP, Composer, Infinity, MoMask's code, the tutorial, and SDFusion's eq. 4 for one modality [arxiv:2211.12572 eq. 5] [arxiv:2302.09778 eq. 2] [arxiv:2412.04431 §4.7] [arxiv:2208.11970 §5] [arxiv:2212.04493 eq. 4]. LayoutDiffusion's main text uses the same form, with s on the conditional term [arxiv:2303.17189 §3.4]. - **Weight 0 is conditional-only:** (1 + w)·cond − w·uncond. Used by UniPi, by MoMask's paper, by Muse's logits, by SDXL's paper and by the survey's eq. 11 [arxiv:2302.00111 §2.3] [arxiv:2312.00063 eq. 5] [arxiv:2301.00704 eq. 1] [arxiv:2307.01952 app. Diffusion Models] [arxiv:2310.07204 eq. 11], and per condition in VASA-1's eq. 2 [arxiv:2404.10667 eq. 2], and FastGrasp's released sampler [code: network/diffusion/ddim.py#L135], LayoutDiffusion's appendix and code [arxiv:2303.17189 app. B.2], and the physics-residual sampler [arxiv:2211.14680 alg. 2]. A value of 4 here equals 5 in the first form. VAR's code uses it with a w that grows per scale [code: models/var.py#L172]. - **Scale on the difference:** uncond + s·(cond − uncond). Same as the first form, written differently. Used by DiT, MDM, and SDXL's released guider [arxiv:2212.04493 eq. 4] [arxiv:2212.09748 §3.1] [arxiv:2209.14916 eq. 7] [code: sgm/modules/diffusionmodules/guiders.py#L24-L30], and the common-schedules paper's eq. 13, with the negative prompt as the unconditional term [arxiv:2305.08891 eq. 13]. Make-It-3D's distillation mix is written in this form [arxiv:2303.14184 §4.1]. Check which form a paper or library uses before copying a value. - **Weight 0 in the physics sampler is not no guidance.** The shipped conditional configs set w = 0, so the mix is off. The released sampler still subtracts the residual gradient in its update [code: configs/kmflow_re1000_rs256_conditional.yml#L40] [code: functions/denoising_step.py#L186-L198]. The paper gives no w [alg. 2]. Test the sampler with the mix on and off before reporting either. - **Null condition differs by model.** The unconditional branch is an empty prompt for PnP and MLD's text, a zero-filled set of inputs in SDFusion, an all-zero image in SDFusion's single-view model [code: models/sdfusion_img2shape_model.py#L241], and a zero vector for MLD's actions [arxiv:2212.04048 eq. 4]. FastGrasp's released sampler uses an all-zero input [code: network/diffusion/ddim.py#L170-L171]. DiT's null is a learned embedding at an index past the last class [arxiv:2212.09748 §3.1] [code: models.py#L74]. MDM's unconditional token is the time embedding alone [code: model/mdm.py#L227-L229], and VAR's null is an extra class index [arxiv:2404.02905 §3.2]. Use the form your model was trained with. DreamPose's dropped image and pose are all-zero tensors that still pass through CLIP and the VAE, so the null context is whatever those networks output for black input [code: train.py#L212-L216] [code: train.py#L361-L370]. JavisDiT uses a learned null embedding for its prior, which the paper does not describe [code: javisdit/models/prior_encoder/base_model.py#L248-L253] [arxiv:2503.23377 app. C.2.1]. - **Two-model guidance costs an extra pass.** SINE's eq. 3 needs the pretrained conditional, the fine-tuned conditional and the pretrained unconditional prediction. The code also runs the fine-tuned unconditional branch, whose output does not enter the update [code: ldm/models/diffusion/guidance_ddim.py#L489-L492]. - **Attention edits and guidance share a batch.** In DenseDiffusion's code, only the conditional half of the batch is modulated. An implementation that modulates the unconditional half too differs from the code [code: gradio_app.py#L260] [code: gradio_app.py#L147-L163]. - **Safety guidance costs an extra pass.** SLD runs the unconditional, prompt and concept predictions every step [code: src/sld/sld_pipeline.py#L456] [code: src/sld/sld_pipeline.py#L461]. The paper reports no timing. Its pipeline also runs Stable Diffusion's own safety checker when one is loaded, which changes the output path [code: src/sld/sld_pipeline.py#L517-L523]. - **A user negative prompt changes the SLD baseline.** The code uses a negative prompt in place of the empty unconditional branch [code: src/sld/sld_pipeline.py#L349-L367], so the paper's protocol no longer holds. - **A token model's ramp endpoints are missing.** Muse's linear ramp has no published start or end value [arxiv:2301.00704 §2.7]. Choose them and report them, since the ramp changes which tokens are guided. - **Guidance can silently do nothing.** In SDFusion's blended-diffusion completion path, the guidance call omits the unconditional conditioning, so the scale has no effect there [code: models/sdfusion_model.py#L438] [code: models/networks/diffusion_networks/samplers/ddim.py#L187-L188] [arxiv:2212.04493 §3.3]. - **Negative prompting costs more.** The code runs the U-Net on two batches of two per step, recomputing the unconditional branch, which is about twice the usual CFG cost [Plug-and-Play card, Pitfalls] [code: ldm/models/diffusion/ddim.py#L216-L234]. - **Per-condition guidance costs one pass per input.** SDFusion's eq. 4 needs the unconditional pass plus one pass per input [arxiv:2212.04493 eq. 4]. Its multi-modal guided branch runs three denoiser passes per step by stacking the input three times [code: models/networks/diffusion_networks/samplers/ddim.py#L199-L201]. - **Leave-one-out guidance costs one pass per condition.** VASA-1's eq. 2 needs the full prediction and one emptied prediction per condition [arxiv:2404.10667 eq. 2]. Its text does not say whether each emptied term removes one condition or all of them; the per-condition reading is the one the notation gives [§3.2]. It also gives no scale for head distance or emotion [§3.2] [§4], so those values must be chosen before reuse. - **Guidance can lift a metric above real data.** VASA-1's SyncNet lip-sync scores exceed those of real videos when audio guidance is on, and the authors attribute this to CFG [arxiv:2404.10667 §4.1] [§4.3]. Weigh the score with that in mind. - **Priors are resampled per call.** JavisDiT's estimator draws a fresh prior on every call, so the same prompt yields different conditions unless the random state is fixed. The paper does not say whether the prior is fixed per prompt [code: javisdit/models/prior_encoder/base_model.py#L226-L229] [arxiv:2503.23377 §3.2]. - **Negative prompt can be a silent no-op.** In the PnP code, the negative prompt defaults to the source prompt, which is forced to empty for real images, so negative prompting does nothing unless it is set explicitly [code: run_pnp.py#L42] [arxiv:2211.12572 §4]. - **The DreamPose guidance argument can be a silent no-op.** Its `guidance_scale` only decides whether the three-pass branch runs. The update uses only the two per-condition weights [code: pipelines/dual_encoder_pipeline.py#L479] [code: pipelines/dual_encoder_pipeline.py#L552-L553]. Set s_I and s_p explicitly. - **Invert with the empty prompt.** PnP inverts a real image with an empty prompt at CFG scale 1, so any prompt and scale can be used at sampling [arxiv:2211.12572 §5.1] [app. C]. - **Feature injection hits every branch.** When features are injected, the conditional, unconditional and negative passes all receive them [code: ldm/models/diffusion/ddim.py#L223-L234] [arxiv:2211.12572 §4]. - **The unconditional branch loses the reference.** In MagicAnimate's reference attention, the unconditional half gets no reference features, and the guided step doubles the batch for each window [code: magicanimate/models/mutual_self_attention.py#L243-L256] [code: magicanimate/pipelines/pipeline_animation.py#L659-L776]. The paper's text does not say which features the unconditional branch gets. Choose the behaviour on purpose and report it. - **Dropout needs a null embedding.** Training with dropout needs a fixed null embedding for each condition. GLIGEN's code also uses learned null embeddings for padded grounding slots, which the paper does not describe [code: ldm/modules/diffusionmodules/text_image_grounding_net.py#L36-L38]. - **Guidance can use a null the model never trained on.** FastGrasp's released sampler guides against a zero input, but its training code keeps every condition: the dropout probability is stored and the line that would mask the condition is commented out [code: network/diffusion/ddim.py#L58] [code: network/diffusion/ddim.py#L73-L74]. The paper reports no guidance experiment, so compare outputs at the released weight and at w = 0 before relying on the term. - **The null condition is not defined in MLD's paper.** The paper's empty condition is undefined [arxiv:2212.04048 eq. 4]. The code encodes the empty string with CLIP for text and uses a zero vector for actions [code: mld/models/modeltype/mld.py#L224-L231] [code: mld/models/architectures/mld_denoiser.py#L262-L274]. - **Dropped embeddings are zeroed, separately from the null token.** eDiff-I zeroes a dropped embedding. Its cross-attention also has a learnable null token, which is a different thing [arxiv:2211.01324 §4.2]. - **Dropout rate moves FID and R-Precision in opposite directions.** In MLD's sweep at s = 7.5, p = 0.25 gives the lowest FID (0.229) but a lower R-Precision (0.757) than the chosen p = 0.1 (0.772, FID 0.473) [arxiv:2212.04048 tab. 11]. - **Headline FID without a stated weight.** eDiff-I's Table 1 does not state the guidance weight behind its FID numbers [arxiv:2211.01324 §5.1]. Treat the numbers as the authors' sampler setting, not a value you can copy. - **Unstated dropout and weight in some papers.** UniPi gives neither its guidance weight nor its dropout rate [arxiv:2302.00111 §2.3]. The Imagen synthetic-data paper states the sweep but not the prompt template [arxiv:2304.08466 §4]. Drive-WM states its guidance scale and dropout rate, but not its guidance form or the replacement for a dropped condition [arxiv:2311.17918 app. B.1]. Any reimplementation has to choose these and say so. - **Selection on the evaluation set.** The Imagen paper picks guidance and sampling settings on FID, IS and classifier accuracy measured on the ImageNet validation set, and reports classifier accuracy on that same set [arxiv:2304.08466 §4.2] [§5.2]. Treat the headline numbers as tuned values. - **Guidance on a subset of channels.** DiT's code applies the guidance only to the first three latent channels by default [code: models.py#L262] [app. A]. The paper's appendix reports that three channels at scale 1.5 give FID-50K 2.27, and four channels at 1.375 give 2.20 [app. A]. The cause is left open [app. A]. Copy the channel choice along with the scale. - **The guided mean is not written out in the distillation.** DMD's real score uses the guided model's mean prediction, but the exact formula is not given [arxiv:2311.18828 §3.4]. Reimplementations must choose it and state the choice. - **A ramped scale is easy to miss.** VAR's code ramps the scale per scale index [code: models/var.py#L172]. Copying a fixed cfg of 2.0 into its sampler therefore changes the output. The paper does not describe the ramp [arxiv:2404.02905 §3.2]. - **Guiding the prior can hurt.** unCLIP reports that guidance on its prior hurt results, so the prior runs unguided [arxiv:2204.06125 §5.5]. Do not add guidance to a prior by default. - **Short runs may show no grounding effect.** In GLIGEN the gated layer starts closed and the paper reports grounding appears only after tens of thousands of iterations [arxiv:2301.07093 app. F]. - **The rescale is not measured alone.** The common-schedules paper's FID compares the official model with a model that has all four fixes, and its data differ from the official model's [arxiv:2305.08891 tab. 3] [§4.2]. Its rescale-factor sweep is visual [fig. 6]. A reimplementation that adds only the CFG rescale has no reported number to match. ## Papers - [Muse](/papers/2301.00704): masked-token text-to-image model with logit-space guidance on the (1 + t) form, ramped over the steps; the ramp endpoints are not stated. - [Safe Latent Diffusion](/papers/2211.05105): safety guidance that adds a text-defined concept as a third prediction to CFG; the paper's scale clip differs from its code. - [unCLIP](/papers/2204.06125): two-stage text-to-image model with dropout-trained guidance in the decoder only; the CLIP-embedding dropout is 10% in one section and 5% in another. - [PnP](/papers/2211.12572): training-free image-to-image editing that uses CFG with a conditional weight and negative prompting, with the weight and blend settings listed above. - [GLIGEN](/papers/2301.07093): trains a frozen diffusion model with condition dropout so that CFG applies to added grounding inputs. - [eDiff-I](/papers/2211.01324): a text-to-image cascade with independent dropout on three embeddings; its guidance sweep appears only as curves. - [MLD](/papers/2212.04048): text-to-motion latent diffusion with CFG in the latent and a published dropout and scale sweep. - [SDFusion](/papers/2212.04493): 3D shape diffusion with per-input guidance weights, and a released code that differs from the paper's eq. 4. - [UniPi](/papers/2302.00111): text-conditioned video planner that uses CFG with the (1 + w) form; the guidance and dropout values are not stated. - [MultiDiffusion](/papers/2302.08113): training-free panoramas and region control by averaging guided windows; the paper has no guidance term, the code does. - [Composer](/papers/2302.09778): composable conditions with dropout rates per condition and a two-set guidance form. - [Human Motion Diffusion as a Generative Prior](/papers/2303.01418): DiffusionBlending, a blend of two fine-tuned models that the authors describe as a generalisation of CFG. - [Synthetic Data from Diffusion Models Improves ImageNet Classification](/papers/2304.08466): per-stage guidance and sampling sweeps for Imagen, judged by downstream accuracy. - [MoMask](/papers/2312.00063): logit-space CFG for a masked token transformer, with the paper and code forms differing. - [Infinity](/papers/2412.04431): bitwise autoregressive model with CFG on bit logits at scale 4 and a temperature that differs between paper and script. - [DreamPose](/papers/2304.06025): image-to-video fashion animation with two separate weights, one for the photo and one for the pose; the paper gives no values for its results. - [VASA-1](/papers/2404.10667): talking-face generation with a leave-one-out weight per condition; the audio weight trades lip sync against mouth motion. - [Score Jacobian Chaining](/papers/2212.00774): text-to-3D by back-propagating a 2D diffusion score through a voxel renderer; the guidance scale is 100 for 3D, and the paper reports no quantitative metric. - [StableSR](/papers/2305.07015): super-resolution from a frozen Stable Diffusion, with negative-prompt guidance whose scale trades no-reference scores against PSNR, SSIM and LPIPS. - [JavisDiT](/papers/2503.23377): joint audio-video diffusion transformer that samples with one guidance scale of 7.0 and a learned null prior. - [SINE](/papers/2212.04489): one-image editing with a fine-tuned copy mixed into the early guidance steps; the CLI and the notebook use different windows. - [DenseDiffusion](/papers/2308.12964): training-free layout control by attention modulation, which shares a batch with CFG in the code; the paper gives no guidance scale. - [State of the Art on Diffusion Models for Visual Computing](/papers/2310.07204): a survey whose classifier-free guidance uses the (1 + w) form, with its weight warnings for Stable Diffusion. - [Understanding Diffusion Models](/papers/2208.11970): a derivation-only tutorial that writes CFG as a convex weighting of the conditional and unconditional scores; it reports no experiments. - [A Task is Worth One Word](/papers/2312.03594): PowerPaint, one inpainting model for several tasks, with object removal by a negative prompt in classifier-free guidance. - [TaKSIE](/papers/2410.11013): visual subgoal generator with separate text and image guidance terms, both at 2.5; the paper does not write out how they combine. - [FastGrasp](/papers/2411.14786): hand-pose latent diffusion whose released sampler uses guidance with a zero null, though the paper omits guidance and the training code drops no condition. - [Make-It-3D](/papers/2303.14184): single-image 3D with score distillation at weight 10 on novel views; the released refine stage uses guidance 5. - [MagicAnimate](/papers/2311.16498): pose-driven human video with scale 7.5 on the difference, and an unconditional branch that drops the reference features in the released code. - [Drive-WM](/papers/2311.17918): multiview driving video model with guidance 5.0 and 20% condition dropout; the guidance form is not stated. - [LayoutDiffusion](/papers/2303.17189): layout-to-image diffusion trained from scratch, with a whole-layout null token dropped 20% of the time; the main text and the code write the same scale in different forms. - [Common Diffusion Noise Schedules and Sample Steps are Flawed](/papers/2305.08891): four fixes for the training and sampling mismatch, one of which rescales the guided output by its standard deviation with φ = 0.7; the rescale is not measured alone. - [Human Motion Diffusion Model](/papers/2209.14916): MDM, with the scale-on-difference form and 10% null-condition training for text and action. - [Scalable Diffusion Models with Transformers](/papers/2212.09748): DiT, class-conditional latent transformer with a learned null label; the paper reports FID at scales 1.25 and 1.50. - [SDXL](/papers/2307.01952): latent text-to-image model with two text encoders, whose guided form puts w on the unconditional term; the released guider uses the scale-on-difference form. - [Visual Autoregressive Modeling](/papers/2404.02905): next-scale autoregressive generator with a ramped logit guidance in its released sampler; the paper's ablation uses one fixed value. - [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828): one-step student trained against a guided teacher at a fixed scale. - [A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction](/papers/2211.14680): diffusion prior for flow-field super-resolution whose condition is the PDE residual gradient, mixed in the (1 + w) form; the paper gives no w and no dropout rate.