Controllable Generation with Text-to-Image Diffusion Models: A Survey

In plain words

Text-to-image diffusion models turn a written prompt into a picture. A prompt is often not enough, though. Some needs are hard to put into words: a face you want to reproduce, a particular art style, a pose, a layout, or a region of the picture. This survey collects the research that adds such extra inputs, called conditions, to a pretrained text-to-image model so that it follows them.

The survey groups the work in three ways. The first asks how one new kind of condition gets into the model: by training the model on it, by attaching a small add-on network, or by steering the existing model without any training. It also covers a second route, where a separate predictor checks the picture during generation and nudges it toward the condition. The second group handles several conditions at once, such as a character's identity together with a pose. The third group looks for frameworks that accept many kinds of condition with one design.

The survey also describes what these models are used for, such as editing, filling in missing parts of an image, composing objects, and building 3D or video models. It is a map of the field rather than a new method, so a reader looking for one algorithm to implement should follow the citations it gives.

TL;DR

A survey of controllable generation with text-to-image diffusion models, with a taxonomy keyed on the condition type: one new condition, several conditions, or a universal framework §1 §3 §4 §5. The two core mechanisms are conditional score prediction (the model is changed or given an extra input) and condition-guided score estimation (a predictor's gradient steers sampling) §3. It reports no experiments of its own; its tables list model specifications and evaluation dimensions tab. 1 tab. 2.

Problem

Text alone cannot carry every user requirement. Conditions such as an unseen person or a specific art style are hard to convey through a prompt §1. The survey says that earlier surveys of generative AI covered text-to-image control only briefly or focused on other modalities §1. It argues that the effect of adding new conditions to text-to-image models needs an in-depth review §1.

The text does not describe a search protocol, inclusion criteria or a date cut-off for the literature it covers. The reader cannot reconstruct which papers were considered and which were left out. §1 §2.6

Method

The survey has no method of its own. It structures the literature as follows.

Preliminaries §2. Denoising diffusion probabilistic models (DDPM) define a forward process that adds Gaussian noise over T steps and a learned reverse process whose mean and variance a network predicts §2.1 eq. 1 eq. 2 eq. 3. The survey points to DDIM, score-based modelling, flow matching and EDM as extensions and focuses on flow matching, which it says underlies Stable Diffusion 3 and Flux §2.1.

Flow matching trains a network to predict a velocity field that carries noise to data along a conditional path, and sampling integrates the learned ODE §2.2 eq. 4 eq. 5 eq. 6. Rectified flow uses a straight line between a prior sample and a data sample, and its training target is the constant difference between the two endpoints §2.2 eq. 7 eq. 8.

Text-to-image models differ in where the text enters the denoiser §2.3 tab. 1. The survey distinguishes U-Net models with cross-attention, where image latents are queries and text embeddings are keys and values, from transformer models: DiT, which uses self-attention over patch tokens, and MMDiT, which uses joint attention over text and image tokens §2.3.

Controllable generation tasks §2.4 fig. 2. The survey lists spatial control (layout, pose, depth, segmentation), image personalization, view- conditioned generation, advanced text conditioning (longer prompts, prompt rewriting, multilingual encoders), in-context generation, brain- and sound- guided generation, and text rendering.

Evaluation §2.5 tab. 2. Image quality uses FID, Inception Score, aesthetic predictors and learned human-preference models. Conditional alignment uses CLIP-based text similarity, visual similarity with CLIP or DINOv2 features, spatial consistency (mIoU, MSE) and content accuracy.

Taxonomy §2.6. Three directions: novel conditions, multiple conditions, and universal generation.

Core formulation §3. The survey writes the score of a text-conditioned model with classifier-free guidance as a weighted combination of an unconditional and a conditional prediction, with weight w §3. Novel conditions are added in one of two ways:

• Conditional score prediction §3.1: the denoiser itself takes the novel condition, so the sampler needs no extra gradient. The survey names three families: tuning-based (§3.1.1), adapter-based (§3.1.2) and training-free (§3.1.3).
• Condition-guided score estimation §3.2: a predictor estimates how well the current latent matches the condition, and the gradient of that estimate is added to the noise prediction with a scale γ.

# Classifier-free guidance on the text condition  [§3]
# Inputs: x_t (latent), c_text, timestep t, weight w
# Output: text-guided noise prediction, same shape as x_t

def cfg_noise(x_t, c_text, t, w):
    # Prediction with the null prompt phi in place of the text  [§3]
    eps_uncond = eps_model(x_t, phi, t)
    # Prediction with the text condition  [§3]
    eps_cond = eps_model(x_t, c_text, t)
    # Combine the two predictions with weight w  [§3]
    return (1 - w) * eps_uncond + w * eps_cond

# Conditional score prediction with a novel condition  [§3.1]
# Inputs: x_t, c_text, c_novel, timestep t, weight w
# Output: noise prediction, same shape as x_t

def predict_noise(x_t, c_text, c_novel, t, w):
    # Novel-condition model with the text dropped; the paper's
    # unconditional term omits c_novel, and the card follows it  [§3.1]
    eps_uncond = novel_eps_model(x_t, phi, t)
    # Novel-condition model with both the text and the novel condition  [§3.1]
    eps_cond = novel_eps_model(x_t, c_text, c_novel, t)
    # Combine the two predictions with weight w  [§3.1]
    return (1 - w) * eps_uncond + w * eps_cond

# Condition-guided score estimation  [§3.2]
# Inputs: x_t, c_text, c_novel, timestep t, weight w, scale gamma
# Output: noise prediction shifted by a condition gradient

def guided_noise(x_t, c_text, c_novel, t, w, gamma):
    # Text-conditioned prediction with classifier-free guidance  [§3]
    eps_text = cfg_noise(x_t, c_text, t, w)
    # Log-likelihood of the novel condition given the latent  [§3.2]
    log_likelihood = condition_critic(x_t, c_novel)
    # Gradient of that log-likelihood with respect to the latent  [§3.2]
    grad = gradient_wrt(log_likelihood, x_t)
    # Add the gradient scaled by gamma  [§3.2]
    return eps_text + gamma * grad

Rectified-flow training step §2.2 eq. 7 eq. 8.

# One rectified-flow training step  [§2.2]
# Inputs: x0 (B, C, H, W) data; model predicts a velocity
# Output: scalar loss

def rectified_flow_loss(x0, model):
    # Draw one prior sample per data point; prior not named  [§2.2]
    x1 = sample_prior(x0.shape)
    # Draw one time per example, uniform on (0, 1)  [§2.2]
    t = uniform(0.0, 1.0, size=(B,))
    # Straight-line interpolation  [§2.2] [eq. 7]
    x_t = (1 - t) * x1 + t * x0
    # Regress the velocity onto the constant difference  [§2.2] [eq. 8]
    target = x0 - x1
    return mean_squared_error(model(x_t, t), target)

Multiple conditions §4. The survey groups methods into joint training §4.1, continual learning §4.2, weight fusion §4.3, attention-based integration §4.4 and guidance composition §4.5. Guidance composition combines per-condition predictions with weights and spatial masks §4.5.

Universal generation §5. Universal conditional score prediction builds one framework that encodes any condition and feeds it to the noise predictor §5.1. Universal condition-guided estimation uses off-the-shelf predictors (for example CLIP, segmentation, face recognition or object detection) as guidance functions §5.2.

Applications §6 fig. 8. Image manipulation §6.1, completion and inpainting §6.2, composition §6.3, text- and image-to-3D generation using Score Distillation Sampling §6.4, and camera-controlled video generation as a route to world models §6.5.

Hyperparameters

The survey sets no hyperparameters of its own. The table lists every numeric value stated in the text. Items marked "not stated" are missing from the paper and cannot be taken from it.

Results

The survey reports no benchmark numbers of its own, and its text describes no experiments. Its numeric content is the model table and the evaluation dimensions. Author-reported figures for individual models are not in the text and come from the cited papers.

Model specifications tab. 1. The table covers pixel-space and latent-space models. Parameter counts marked † count the U-Net and text encoder together; unmarked counts refer to the U-Net only, according to the table caption.

Source: tab. 1. "Open" is the table's own column; "–" is the table's dash.

Evaluation dimensions tab. 2. Image quality: FID, Inception Score, aesthetic score, HPS and PickScore. Conditional alignment: CLIP text similarity, CLIPScore and DINO for visual similarity, mIoU and MSE for spatial consistency, and content accuracy, which the table cites to two benchmark papers without naming a metric.

Figures. fig. 1 plots the yearly number of papers on controllable text-to-image generation; the caption says the count rises after strong generators were released. The figure carries no values in the text.

Paper vs code

No official code. The abstract links a curated repository, PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models, as an index of the surveyed literature §1. It is a paper list, not code. I did not clone it or run a GitHub search, because the survey proposes no method and no code is expected for it. The paper's source contains no project page link.

Pitfalls

• Two conventions for time. The flow-matching section integrates from data at t = 0 §2.2. The rectified-flow formula puts data at t = 1, since x_t = (1 − t) x_1 + t x_0 with x_0 as data §2.2 eq. 7. Choose one convention and convert the other before implementing.
• Two forms of the reverse-mean formula. The equation at the start of §3 uses 1/√ᾱ_t and a minus sign on the score term §3. The ε-form written right after it uses 1/√(1 − ᾱ_t) §3. The two do not agree as printed. Use the ε-form, which matches the score relation the survey states §3.
• Unconditional branch. In the novel-condition formula the unconditional term is written without c_novel §3.1. The text does not say whether that omission is deliberate, so an implementation has to choose.
• Guidance weight convention. The survey writes classifier-free guidance as (1 − w) times the unconditional prediction plus w times the conditional one §3. It gives no range for w, so a value copied from another paper may mean something different here.
• Notation slips in §2.2. The flow-matching velocity is written u_t in one place and v_t in the text, and one equation has a broken symbol. Read the equations, not the prose, for the target.
• Unstated values. Sampler step count, prior distribution, noise schedule, guidance scales and the weight γ are not stated §2.1 §2.2 §3.2.
• Citation keys. Some keys in the source are duplicated or look wrong (for example a stray key in the feature-injection paragraph of §3.1.3 and a repeated key in the same list). Check a reference against its own paper before relying on it.
• Version. This card follows v2 of the arXiv record, the version fetch recorded. The record was updated on 2026-01-07. Later or earlier versions may differ.
• Coverage. The text states no search protocol or inclusion rule, so the list of methods is the authors' selection, not a complete census §1.

Lineage

• Builds on: the DDPM, DDIM, score-based, flow-matching and EDM lines, named in §2.1 without links, since they have no cards here.
• Compared against: earlier surveys named in §1 (a diffusion-theory survey, an efficient-diffusion survey, a multimodal-synthesis survey, two visual-diffusion surveys and a text-to-3D survey). Named only, not linked.
• Superseded by: not stated.