More Control for Free! Image Synthesis with Semantic Diffusion Guidance

In plain words

Image generators that start from noise can make striking pictures, but it is hard to say what they should show. This paper adds a steering wheel to an existing diffusion model. At each step of turning noise into an image, the picture is nudged in the direction that makes it match a target: a sentence such as "a smiling woman with curly hair", a reference photo, or both. The steering signal comes from a vision-language model that scores how well a picture matches the target.

The model that does the drawing is never retrained. Only the scoring model is adjusted, and it is adjusted without any text labels, so the method works on image collections that have no captions. A reference photo can steer the content, meaning the pose and layout are allowed to change, or the style, meaning colours and textures are copied. A single strength setting trades off how closely the result follows the target against how varied the results are.

The results are reported by the authors on face, room, cat and horse datasets. The released code does not match every formula in the paper.

TL;DR

• Guidance is added at sampling time to a pretrained unconditional DDPM, with no retraining of the diffusion model §3.1.
• Language, content and style guidance, and their weighted sum, all use the same scheme: shift each reverse step by a scaled gradient of a CLIP-based score §3.1 eq. 5 eq. 10.
• CLIP is adapted to noisy images with time-conditioned batch normalisation and a contrastive finetune between clean and noised images, with no text labels §3.5.
• Author-reported FFHQ results: image guidance FID 14.37 versus ILVR 17.15; text guidance FID 28.38 versus StyleGAN+CLIP 57.45, but StyleGAN+CLIP has the higher retrieval scores tab. 1.

Problem

Text-to-image models usually need paired image-caption data for training, and image-conditioned methods mostly transfer style or only work on narrow domains with fixed structure such as faces §1 §2. The paper asks whether a pretrained unconditional diffusion model can be steered by language, by a reference image, or by both, at fine-grained and continuous strength, and without retraining the generator §1. Its evaluation covers FFHQ and LSUN cat, horse and bedroom data §4.1.

Method

The method has four parts. The underlying generator is an unconditional DDPM trained elsewhere §3.1. A guidance function F(x_t, y, t) scores how well a noisy image x_t matches a target y at step t §3.1. The sampler adds a scaled gradient of F to each reverse-step mean alg. 1. The guidance network is a CLIP image encoder that has been adapted to noisy inputs §3.5.

Sampler §3.1 alg. 1. The reverse step is the usual Gaussian step with its mean shifted by s · Σ · ∇F, where s is a user-set scale and Σ = σ²I is the model's variance alg. 1. The shift follows the classifier-guidance rule of Dhariwal and Nichol, with the log-probability of the target replaced by a continuous score §3.1 eq. 5 eq. 6. Scale s multiplies the score, so the paper treats it as the guidance strength §3.1.

Language guidance §3.2 eq. 6. The score is the cosine similarity between the time-conditioned CLIP image embedding of x_t and the CLIP text embedding of the prompt. The text encoder is the original, unmodified CLIP text encoder sample.py.

Image content guidance §3.3 eq. 7. The reference image x'_0 is noised to the same level t as x_t, and the score is the similarity of their time-conditioned CLIP image embeddings. Pooled embeddings have no spatial dimensions, so pose and layout can change between samples §3.3.

Structure-preserving content guidance §3.3 eq. 8. Per-layer spatial feature maps of the two images are compared by mean squared difference, summed over layers j. The paper says this keeps pose and layout of the reference §3.3. The released code does not implement this variant (see Paper vs code).

Image style guidance §3.3 eq. 9. The score is the negative sum over layers of the squared difference between Gram matrices of the feature maps of x_t and the noised reference §3.3. The Gram matrix of a feature map is the matrix of its channel-by-channel inner products §3.3 eq. 9.

Multimodal guidance §3.4 eq. 10. The guidance is a weighted sum of a language score and an image score, with their scales as weights §3.4. The paper gives no default for the two weights beyond the per-modality defaults in §4.1.

Time-conditioned CLIP §3.5. The CLIP image encoder is changed in one place: each batch-normalisation layer becomes an adaptive one whose scale and bias are predicted from the timestep t. Weights are copied from pretrained CLIP, except the new adaptive layers §3.5.

Self-supervised finetune §3.5. For a batch of N clean images and their noised versions at sampled timesteps, the clean-image encoder stays frozen and the noised-image encoder is trained with CLIP's contrastive loss. Matching clean and noised images are positives; other pairs in the batch are negatives §3.5. No text is needed §3.5.

# Semantic diffusion guidance sampler  [alg. 1]
# Input:  target y, scale s; unconditional model (mu, sigma^2) for x_t
# Output: image x_0, shape (3, H, W)

def sample_guided(model, guidance_score, y, s):
    # Start from standard Gaussian noise  [alg. 1]
    x = sample_gaussian_noise()                  # (3, H, W)
    for t in reversed(range(T)):
        # Unconditional mean and variance from the pretrained model  [§3.1]
        mu, sigma_sq = model(x, t)
        # Gradient of the score with respect to the noisy image  [alg. 1]
        grad = gradient_of(guidance_score, x, y, t)
        # Shift the mean by the scaled, variance-weighted gradient  [alg. 1]
        x = sample_gaussian(mu + s * sigma_sq * grad, sigma_sq)
    return x

# Guidance scores for the sampler  [§3.2] [§3.3] [§3.4]
# Each returns one scalar per image; higher means a better match.

def language_score(x_t, text_embedding, t):
    # Time-conditioned CLIP image embedding of the noisy image  [§3.2] [§3.5]
    image_embedding = clip_encode(x_t, t)        # (D,)
    # Cosine similarity with the prompt embedding  [eq. 6]
    return cosine(image_embedding, text_embedding)

def content_score(x_t, reference_0, t):
    # Noise the reference to the same level as x_t  [§3.3] [eq. 2]
    reference_t = add_noise(reference_0, t)      # (3, H, W)
    # Similarity of the two time-conditioned embeddings  [eq. 7]
    return cosine(clip_encode(x_t, t), clip_encode(reference_t, t))

def style_score(x_t, reference_0, t):
    # Feature maps of each encoder layer j, for both images  [eq. 9]
    maps_x = clip_feature_maps(x_t, t)           # list over layers j
    maps_r = clip_feature_maps(add_noise(reference_0, t), t)
    # Negative sum of squared Gram-matrix differences  [eq. 9]
    total = 0.0
    for j in range(len(maps_x)):
        total = total + gram_diff_sq(maps_x[j], maps_r[j])
    return -total

def multimodal_score(x_t, prompt_embedding, reference_0, t, s1, s2):
    # Weighted sum of the language and image scores  [eq. 10]
    return s1 * language_score(x_t, prompt_embedding, t) \
        + s2 * content_score(x_t, reference_0, t)

# Self-supervised finetune of time-conditioned CLIP  [§3.5]
# Input:  batch of clean images (N, 3, H, W); no captions

def finetune_loss(clean_images, timesteps):
    # Noise each image to its own sampled timestep  [§3.5]
    noisy_images = add_noise(clean_images, timesteps)
    # Frozen original encoder on clean images  [§3.5]
    clean_features = clip_encode_frozen(clean_images)     # (N, D)
    # Trainable time-conditioned encoder on noisy images  [§3.5]
    noisy_features = clip_encode_tuned(noisy_images, timesteps)
    # Contrastive loss; matching clean/noisy pairs are positives  [§3.5]
    return contrastive_loss(noisy_features, clean_features)

Hyperparameters

Results

Author-reported. FFHQ, Table 1 tab. 1. Higher is better except FID. Retrieval uses the original CLIP RN50×16 with 99 negatives §4.2.

Image guidance ablation on LSUN, scale s in brackets tab. 2:

Language guidance ablation on FFHQ tab. 3:

Qualitative results are in fig. 3 to fig. 7: image content guidance, language guidance, joint guidance, comparison with ILVR and StyleGAN+CLIP, and style and out-of-domain guidance. The paper reports these as examples, not as metrics.

Paper vs code

Official code: github.com/xh-liu/SDG_code, linked from the authors' project page §1 footnote at commit 62fb2725035f2cc327bcfc3710384d9cec0dac3c. The repository has no license file, so it can be read but not quoted verbatim (see code_license). Searches: the paper, its arXiv metadata and the project page were checked for links; the project page links only this repository.

• Scale s. The paper gives defaults of 100 for image and 120 for language guidance §4.1. The main FFHQ text row in Table 1 has FID 28.38, LPIPS 0.610 and retrieval 0.553 to 0.947, which equal the s = 320 row of tab. 3, not the s = 120 default. The released sampling flags use text weight 160 with image weight 100 for bedroom README.md. The code does not multiply the score by any separate s: the two weights multiply the scores directly sample.py with defaults of 1.0 parser.py.
• Update rule. The code adds the model variance times the gradient of the weighted score to the model mean gaussian_diffusion.py. This matches Algorithm 1 with s folded into the weights.
• Time-conditioned CLIP. The released encoder keeps each batch-norm layer and applies a timestep-predicted scale and bias after it, in every block and in the stem clip_guidance.py clip_guidance.py. This matches the adaptive-batch-norm description in form §3.5. The release does not state which batch-norm statistics are used at sampling time.
• Content guidance. The code's content score uses the pooled final embedding, normalised, and cosine similarity guidance.py. This matches eq. 7 in words.
• Structure guidance eq. 8 is not in the code. sdg/guidance.py has only semantic and style modes guidance.py, and no spatial-alignment loss exists in the repository. The structure-preserving result in fig. 7 cannot be reproduced from the release.
• Style guidance eq. 9 differs. The code normalises each feature map by the square root of channel × height × width before the Gram matrix, takes a mean squared error over Gram entries rather than a sum of squared Frobenius norms, multiplies by 1e10, and averages over five layers, using the first five feature maps and not the attention-pool output guidance.py.
• Finetune loss. The released loss uses a fixed logit scale of 100 and symmetric cross-entropy over the batch clip_finetune_noise_nolabel.py. The paper names the CLIP contrastive objective but gives no temperature §3.5.
• Finetune batch size and weight decay. The paper gives 256 and 1e-3 §4.1. The released script's defaults are 32 and 0 parser.py parser.py. The paper's values may have been passed as flags; the release does not record them.
• Finetune script does not run as released. It imports from a package named sgd clip_finetune_noise_nolabel.py, while the package is sdg. It reads args.structure clip_finetune_noise_nolabel.py, but the parser has no such flag. It also unpacks each batch as (image, reference), but the release's data loader yields (image, kwargs) image_datasets.py. Treat the finetune script as illustrative. The sampler sample.py is the path the authors document.
• Gram batching. gram_matrix reshapes (B, C, H, W) to (B·C, H·W), so channels from different images in one batch are mixed guidance.py. The comment assumes batch size 1. The reference batch is replicated image_datasets.py, so the reference side is the same image in every slot; the generated side mixes samples.
• Evaluation. The paper states the CLIP model and dataset sizes §4.1. The scripts/ folder has only the sampler and the finetune script, so the FID, LPIPS and retrieval numbers cannot be rerun from the release.

Pitfalls

• Scale is not one number. Use the scale that matches the result you want. The paper's defaults (100, 120) differ from the Table 1 text row (equal to 320), and from the released bedroom command (160). Expect to tune it per dataset. Larger scale gives more semantic match and less diversity tab. 2 tab. 3.
• Score magnitude. The language score is a cosine in [-1, 1]. The effective update is scaled by the model variance, which depends on the timestep. Early and late steps therefore move the image by different amounts, and no normalisation is applied gaussian_diffusion.py.
• Gradient cost. Each reverse step runs a CLIP RN50×16 forward and backward pass through the noisy image, and the diffusion model runs once per step. The paper gives no runtime or memory numbers §4. Sampling cost scales with the number of steps.
• Style scale. The style score is multiplied by 1e10 in code guidance.py. Its effective strength depends on feature magnitudes, so the weight does not transfer between layers or datasets without retuning.
• Equation and figure numbers. The numbers here were counted from the LaTeX source in order, not read from the compiled PDF: eq. 1 to eq. 10 for the displayed equations in §3, fig. 1 to fig. 7, tab. 1 to tab. 3, and alg. 1. Check against the compiled PDF before relying on them. The paper says a visual example of scale is in an appendix §4.3, but the source has no appendix, so that content is not stated here.
• Text side is never finetuned. The finetuned image encoder is compared with the original CLIP text embedding sample.py. The finetune changes only the image side, so the text and image embeddings are matched only through the original pretraining.
• Results are author-reported. The Table 1 comparison is on FFHQ only. StyleGAN+CLIP has the highest retrieval scores, which the paper says come at the cost of low diversity §4.2.

Lineage

• Builds on: DDPM sampling §2; the classifier-guidance rule of Dhariwal and Nichol, generalised here from class labels to continuous embeddings §3.1; CLIP for the scoring model §3.2.
• Compared against: ILVR, an image-guided diffusion method that keeps the same structure §4.2; StyleGAN+CLIP, a latent-optimisation method §4.2.
• Related concurrent work named in the paper: GLIDE for classifier-free text guidance, and Blended Diffusion for mask-based editing §2.
• Superseded by: not stated in the source.