--- kind: concept slug: layout-to-image-generation title: Layout-to-image generation summary: >- Makes a diffusion model place objects where you ask, given as boxes with phrases, boxes with classes, painted or region masks, or a layout an LLM writes. GLIGEN and LayoutDiffusion train box-conditioned layers; the mask-based controls, box guidance on attention and LayoutGPT's planning need no training. Covers inputs, training cost and the constraints that choose between them. tags: [diffusion, text-to-image, grounding, layout, controllable-generation, training-free, llm-prompting] papers: ["2301.07093", "2211.01324", "2302.08113", "2305.15393", "2308.12964", "2303.17189", "2304.03373"] related: [diffusion-models, latent-diffusion, classifier-free-guidance, attention-mechanism, self-attention, diffusion-inpainting, in-context-learning, prompt-engineering, text-to-image-generation] status: draft updated_at: 2026-11-01 --- ## Overview A caption says what to draw but not where. Layout-to-image generation adds a second input, a set of spatial conditions, so each object gets a region. The classic approach trains a generator from scratch on a fixed set of categories [arxiv:2301.07093 §1] [§2]. LayoutDiffusion is one such model: it is trained from scratch on box-labelled images with a fixed class list [arxiv:2303.17189 §3] [tab. 4]. Most of the other approaches here keep a large pretrained text-to-image model and add spatial control on top of it; eDiff-I is the exception among those, since it trains its own cascade. GLIGEN trains new gated layers that read boxes and phrases, and its base weights stay frozen [arxiv:2301.07093 §4.2]. eDiff-I trains its own cascade, but its paint-with-words control adds no training and biases cross-attention with user-painted masks [arxiv:2211.01324 §4] [§4.3]. MultiDiffusion runs a pretrained model on overlapping crops or masked regions and fuses the outputs, with no training [arxiv:2302.08113 §3] [§4.2]. DenseDiffusion keeps Stable Diffusion frozen and modulates its attention with one region map per phrase [arxiv:2308.12964 §3.2]. LayoutGPT moves the layout step out of the generator: a language model writes the boxes from the caption, and a separate layout-to-image model draws them [arxiv:2305.15393 §3.2] [§4.1]. Layout-Guidance is a training-free alternative that edits the latent of Stable Diffusion during sampling, so that each guided word's cross-attention falls inside its box [arxiv:2304.03373 §3.2]. Beyond training, the approaches differ in what they take as input and what they measure. The GLIGEN invariant is that the new layers start in a state that reproduces the original model, and the conditions are encoded in the same space as the caption, so the model can use phrases it did not see with boxes during training [arxiv:2301.07093 §4.1] [§4.2]. ## Variants ### Boxes with phrases on a frozen latent diffusion model Each entity is a phrase and a box. The phrase goes through the caption's text encoder, the box coordinates are Fourier-embedded, and a small MLP fuses them into one grounding token [arxiv:2301.07093 §4.1] [app. A]. New gated self-attention layers attend over visual and grounding tokens together, and their output is added back through a gate that starts at zero, so the first forward pass equals the original model [arxiv:2301.07093 §4.2] [eq. 8]. Card: [GLIGEN](/papers/2301.07093). ### Gated self-attention versus gated cross-attention The paper also tried gated cross-attention, where grounding tokens enter through cross-attention instead. Its ablation on COCO2014 with detection data gives AP 16.6 for cross-attention and 21.7 for self-attention, at similar FID [arxiv:2301.07093 app. B]. The paper keeps self-attention for its main model. Card: [GLIGEN](/papers/2301.07093). ### Other grounding inputs through the same path The same grounding path takes other conditions with a different encoder: - **Keypoints.** One learned embedding per person ties that person's keypoints together [arxiv:2301.07093 app. A]. Reported on COCO2017 with a caption [app. D]. - **Reference images.** An image encoder replaces the text encoder for image entities [arxiv:2301.07093 §4.1] [app. A]. - **Spatial maps.** A small convolutional backbone turns an edge, depth or semantic map into a fixed number of tokens. The map is also concatenated to the U-Net input [arxiv:2301.07093 §4.1] [app. A]. Card: [GLIGEN](/papers/2301.07093). Only the box-with-phrase setting is evaluated in the paper's main tables; the others are supplementary [arxiv:2301.07093 app. D] [app. E]. ### Scheduled grounding at sampling time This is a sampler rule, not a training change. Grounding is on for the first `tau * T` denoising steps and off afterwards, so the layout is fixed early and the original model finishes the details [arxiv:2301.07093 §4.2] [eq. 10]. The paper shows this only qualitatively, using `tau = 0.2` [arxiv:2301.07093 §5.4] [fig. 7]. Card: [GLIGEN](/papers/2301.07093). eDiff-I reports that its text prompt drives the layout at high noise and is largely ignored at low noise [arxiv:2211.01324 §4] [fig. 3] [fig. 4]. That supports the early timing of grounding, but the observation is about the prompt, not about grounding, and eDiff-I does not test a grounding switch. DenseDiffusion also modulates only the early steps, t from 1 to 0.7, because the authors saw no clear gain beyond that [arxiv:2308.12964 §3.2]. ### Boxes and classes, trained from scratch LayoutDiffusion has no text prompt and no pretrained image model. Each object is a box and a class id. A six-layer transformer first lets the objects attend to each other, and an object-aware cross-attention block then lets each image patch attend to the objects, using each patch's and each object's box position [arxiv:2303.17189 §3.2] [§3.3] [tab. 4]. It is trained with the layout dropped 20% of the time, so classifier-free guidance is available [arxiv:2303.17189 §3.4] [app. C.1]. Author-reported, on COCO-Stuff at 256 px, FID 15.61 against 29.10 for PLGAN and 31.18 for LostGAN-v2; on Visual Genome at 256 px, 15.63 [arxiv:2303.17189 tab. 1]. These baselines are GANs only. The model is larger than LDM-4 (569M against 306M parameters), and the paper reports FID 31.68 against 40.91 for LDM-4 on 2,048 images [arxiv:2303.17189 tab. 3]. Card: [LayoutDiffusion](/papers/2303.17189). ### Painted masks with an attention bias The user selects a prompt phrase and paints a binary mask for it. The masks are resized to each cross-attention layer's resolution. A bias built from the masks is added to the cross-attention logits before the softmax, and its strength grows with the noise level and with the user's weight `w'` [arxiv:2211.01324 §4.3] [fig. 6]. No weights are trained. The input is a mask for a prompt phrase, not a box, so it does not name entities separately from the caption. The paper shows paint-with-words results as samples only [arxiv:2211.01324 §5.4] [fig. 17]. Card: [eDiff-I](/papers/2211.01324). ### Region maps with attention modulation on Stable Diffusion DenseDiffusion splits the caption into non-overlapping segments, each with a binary region map [arxiv:2308.12964 §3]. For the first 30% of the denoising steps it changes the scores of both cross-attention and self-attention. Pairs inside one segment are raised toward the row maximum, and pairs that do not share a segment are lowered toward the row minimum [arxiv:2308.12964 eq. 2] [eq. 6]. The change is scaled by a factor that grows for small segments [§3.2]. The weights are 1 for cross-attention and 0.3 for self-attention, with a power of 5 on the step schedule [§3.2] [eq. 3]. The paper also encodes each phrase on its own and substitutes that encoding into the caption's text features [§3.2]. On 250 MS-COCO captions the authors report SOA-I 77.61 against 73.08 for Stable Diffusion, and layout IoU 34.99 against 23.76 for SD-Pww [arxiv:2308.12964 tab. 2] [tab. 3]. Card: [DenseDiffusion](/papers/2308.12964). ### Box energy on cross-attention, latent updated at sampling Layout-Guidance keeps Stable Diffusion v1.5 unchanged. For each guided word and box, it defines an energy: one minus the share of that word's cross-attention mass that falls inside the box [arxiv:2304.03373 §3.2] [eq. 2]. During the first 10 denoising steps, it takes gradient steps on the latent to lower the summed energy of several layers, with strength η = 30 by default [arxiv:2304.03373 §4.1] [eq. 3]. The authors also describe a forward variant that blends attention maps directly; the released code omits it, see Contradictions [arxiv:2304.03373 §3.2]. On 1,000 random VISOR prompts the backward variant reaches 96.9% conditional accuracy, against 59.8% for Stable Diffusion, at about 8 s per image against 4 s [arxiv:2304.03373 tab. 1]. On 1,000 COCO 2014 images, mAP with a detector rises from 19.2 to 35.7, while FID moves from 72.3 to 73.3 [arxiv:2304.03373 tab. 3]. Applied on top of GLIGEN, the gain is smaller: mAP 62.8 to 65.1 [arxiv:2304.03373 tab. 3]. Card: [Layout-Guidance](/papers/2304.03373). ### Mask-weighted fusion of denoising views MultiDiffusion has no grounding input and adds no layers. At every denoising step it runs the pretrained Stable Diffusion v2 once per view, where a view is either a fixed crop of the canvas or the whole canvas weighted by a region mask. The next latent is a per-pixel weighted average of the view outputs [arxiv:2302.08113 §3] [eq. 5] [alg. 1]. Wide panoramas use fixed crops with equal weights. Region control gives each object a binary mask and a prompt, and for tight masks a bootstrapping rule in the early steps replaces the view outside the mask with a constant colour, so the object fills the mask [arxiv:2302.08113 §4.2] [eq. 9]. The paper sets this rule for the first 20% of the process [§4.2]. Author-reported results on a COCO subset: region IoU 0.26 against 0.17 for Blended Latent Diffusion and 0.16 for Stable Inpainting, and 0.18 without bootstrapping [arxiv:2302.08113 tab. 2]. Panorama FID is 10.3, against 18.4 for Blended Latent Diffusion and 45.5 for Stable Inpainting [arxiv:2302.08113 tab. 1]. Card: [MultiDiffusion](/papers/2302.08113). ### LLM-written layouts LayoutGPT does not train or change the generator. The prompt holds a task instruction, a few retrieved solved examples written as CSS-style rules with normalized coordinates, and the new caption [arxiv:2305.15393 §3.2] [§3.3]. The reply is parsed back into boxes, and a layout-to-image model draws them. The main 2D results use GLIGEN as the renderer, and an ablation uses a training-free method [arxiv:2305.15393 §4.1] [tab. 4]. On its NSR-1K numerical benchmark, GPT-3.5 reaches 86.33% layout accuracy against 92.56% for human layouts [arxiv:2305.15393 tab. 2]. For 3D bedrooms it reports fewer out-of-bounds placements than the supervised ATISS baseline, 43.26% against 49.88% [arxiv:2305.15393 tab. 5]. Card: [LayoutGPT](/papers/2305.15393). ## When to use which | Situation | Use | Why | |---|---|---| | You must keep a strong pretrained generator unchanged, and can train | Boxes with phrases on a frozen model, gated self-attention | Only the new layers and the grounding MLP are trained; original weights stay frozen [arxiv:2301.07093 §4.2] | | You cannot run a training job but can change cross-attention at sampling | Painted masks with an attention bias | No training; the strength `w'` is user-specified [arxiv:2211.01324 §4.3]. The paper reports samples only, no control metric [arxiv:2211.01324 §5.4] | | Dense captions, one region per phrase, on a frozen Stable Diffusion model, with no training | Region maps with attention modulation | No training. The paper reports SOA-I 77.61 against 73.08 for Stable Diffusion and layout IoU 34.99 against 23.76 for SD-Pww, on its own 250 captions [arxiv:2308.12964 tab. 2] [tab. 3]. Each segment is a part of the caption, so its phrase must appear in the prompt [§3] | | You want wide canvases or rough region masks, with no training, on Stable Diffusion v2 | Mask-weighted fusion of denoising views | No training; the paper reports panorama FID and region IoU against Blended Latent Diffusion and Stable Inpainting [arxiv:2302.08113 tab. 1] [tab. 2]. The panorama must be at least one crop in height and width [§4.1] | | Masks must be tight around each object | Mask-weighted fusion with bootstrapping | Bootstrapping raises region IoU from 0.18 to 0.26 on the paper's COCO subset [arxiv:2302.08113 tab. 2] | | Stable Diffusion v1.5, no training, named objects must land in given boxes, and about twice the sampling time is acceptable | Box energy on cross-attention, latent updated at sampling | No weights change. The paper reports 96.9% conditional VISOR accuracy against 59.8% for Stable Diffusion, and COCO mAP 19.2 to 35.7, with FID about unchanged (72.3 to 73.3) [arxiv:2304.03373 tab. 1] [tab. 3]. Its evaluations are on its own protocols, and the release has no licence file [code: README.md#L1-L4] | | Fixed category list, box and class labels for every training image, no text prompt needed, and a full training run is affordable | Boxes and classes, trained from scratch | Reports COCO-Stuff 256 FID 15.61 against GAN baselines [arxiv:2303.17189 tab. 1]. Conditional-stage cost is 216.55 V100-days, and the authors name combining it with text pretraining as future work [arxiv:2303.17189 tab. 3] [§5] | | You have only a text prompt, no layout data, and need counts and spatial relations | LLM-written layouts, plus a layout-to-image renderer | No training; the paper reports GPT-3.5 layout accuracy of 86.33% and GPT-4 at 78.43% on numerical prompts [arxiv:2305.15393 tab. 2]. Needs an LLM that takes long prompts, and CLIP features for retrieval [§3.3] | | 3D room layouts from a room type and size, with no 3D layout training | LLM-written layouts in 3D | The paper reports out-of-bounds and FID against the supervised ATISS baseline [arxiv:2305.15393 tab. 5]. Retrieval of examples is essential here [app. C.1] | | Objects come from an open vocabulary, not a fixed list | Phrase-based grounding | Phrases use the caption's text encoder, so unseen names can be grounded [arxiv:2301.07093 §4.1] [tab. 3] | | You need reported placement metrics on a benchmark | Boxes with phrases on a frozen model | Reports YOLO AP on COCO2014 and a zero-shot LVIS GLIP-score [arxiv:2301.07093 tab. 1] [tab. 3] | | Only COCO-style boxes are available | Train on the COCO-only setting and expect weaker open-vocabulary grounding | COCO-only models reach zero-shot LVIS GLIP-score 4.4 to 6.4, versus 10.6 to 11.1 with larger box data [arxiv:2301.07093 tab. 3] | | Compute is limited | Plan for the COCO-scale run first | The COCO run is batch 64 on 16 V100 GPUs for 100k iterations [arxiv:2301.07093 app. A]. Larger runs use 500k iterations at batch 32 [app. A] | | Need to interpolate between the original model and the grounded one at sampling time | Scheduled grounding with `tau` | The paper reports the switch only qualitatively [arxiv:2301.07093 §5.4] | | Need fixed categories and a GAN-style generator | Not covered here | Layout GANs are named as baselines in the card's tables, but no card exists for them in this wiki [arxiv:2301.07093 tab. 2] | Cost differs by approach. The GLIGEN paper gives no inference cost for the extra tokens, so a latency comparison with the unmodified model is not supported by the cited results [arxiv:2301.07093 app. A]. The eDiff-I card reports no cost for the attention bias either [arxiv:2211.01324 §4.3]. MultiDiffusion makes one reference call per view at every step, so its cost grows with the canvas area, and the paper notes the calls can be batched [arxiv:2302.08113 app. B.1]. LayoutGPT makes LLM calls for the layout step, and its 2D run samples five layouts per prompt [arxiv:2305.15393 app. A]. DenseDiffusion adds no weights, and the paper reports no timing or memory figures. Reading its code, the modulated steps build the full score matrix and take row extremes, while later steps use the library attention path [code: gradio_app.py#L139-L141] [code: gradio_app.py#L168-L169]. Backward box guidance adds a backward pass through the U-Net for each update in the first steps; the paper's timing is about 8 s against 4 s per image, with the hardware not stated [arxiv:2304.03373 tab. 1]. LayoutDiffusion's inference cost is 0.308 images per second on the paper's hardware, which it does not name, and its training cost is in V100-days [arxiv:2303.17189 tab. 3]. ## Implementation sketch Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a `# comment` line above each step (with its citation), short trailing comments only for shapes or units. ```pseudocode # Grounding token for one entity [arxiv:2301.07093 §4.1] [eq. 5] [app. A] # Input: phrase_feat (768,) text feature of the phrase, box (4,) # Output: grounding_token (768,) def grounding_token(phrase_feat, box): # Fourier-embed each of the four coordinates [app. A] box_feat = fourier_embedding(box) # (64,) # Fuse phrase and box with an MLP [eq. 5] return mlp(concat(phrase_feat, box_feat)) # (768,) ``` ```pseudocode # Gated self-attention, inserted into a frozen block [§4.2] [eq. 8] # Input: visual (B, M, d), grounding (B, N, d), gamma (learned, init 0) # Output: visual (B, M, d) def gated_self_attention(visual, grounding, gamma, beta): M = visual.shape[1] # Joint self-attention over visual and grounding tokens [eq. 8] joint = concat(visual, grounding, axis=1) # (B, M + N, d) attended = self_attention(joint)[:, :M] # Gate is zero at initialization, so the block starts as the original return visual + beta * tanh(gamma) * attended ``` ```pseudocode # Condition dropout during training [arxiv:2301.07093 app. A] # Input: caption tokens, grounding tokens, drop_prob (paper: 0.1) def dropout_conditions(caption, grounding, drop_prob): # Replace each condition with a null embedding at random, so the # unconditional branch used by classifier-free guidance exists if random() < drop_prob: caption = null_caption if random() < drop_prob: grounding = null_grounding return caption, grounding ``` ```pseudocode # Scheduled grounding at sampling [arxiv:2301.07093 §4.2] [eq. 10] # tau in [0, 1]: fraction of denoising steps with grounding on def grounding_weight(step, total_steps, tau): # beta = 1 for the first tau * T steps, then 0 return 1.0 if step <= tau * total_steps else 0.0 ``` ```pseudocode # Box energy and latent update, first steps only [arxiv:2304.03373 eq. 2-3] # Input: latent z_t; per guided word i, box B_i; cross-attention maps A per layer # Output: latent z_t after the guidance updates def box_guidance(z_t, t, guided, layers, eta, sigma_t): for _ in range(max_repeats): # the paper uses 5 per step [§4.1] energy = 0 for layer in layers: A = cross_attention_maps(z_t, t, layer) for word, box in guided: # Share of the word's attention mass inside its box [eq. 2] inside = A[in_box(box), word].sum() energy = energy + (1 - inside / A[:, word].sum()) ** 2 # Gradient step on the latent, scaled by sigma_t^2 [eq. 3] z_t = z_t - sigma_t**2 * eta * grad(energy, z_t) return z_t ``` ```pseudocode # Painted-mask bias in one cross-attention layer [arxiv:2211.01324 §4.3] # Input: queries (N_i, d_k); keys, values (N_t, d_k) # token_mask (N_i, N_t): column j holds the mask of the phrase that # token j belongs to, resized to this layer's resolution; 0 if unpainted # Output: attended values (N_i, d_k) def painted_attention(queries, keys, values, token_mask, w_user, sigma): # Attention logits between image and text tokens [§4.3] logits = queries @ keys.T # (N_i, N_t) # Strength grows with the noise level and scales with the largest # logit; the scope of that max is not stated in the paper [§4.3] strength = w_user * log(1 + sigma) * max(logits) weights = softmax((logits + strength * token_mask) / sqrt(d_k)) return weights @ values ``` ```pseudocode # One MultiDiffusion step [arxiv:2302.08113 alg. 1] [eq. 5] # Input: pretrained model phi; views with prompts and per-pixel weights # Output: canvas after one step (H', W', C) def fused_step(phi, canvas, views, prompts, weights, t): num = zeros_like(canvas) den = zeros_like(canvas) for view, prompt, w in zip(views, prompts, weights): # Crop the canvas to the view and run one denoising step [alg. 1] crop = canvas[view.slice] out = phi(bootstrapped(crop, view, t), prompt, t) # Accumulate weighted outputs back at canvas positions [eq. 5] num[view.slice] += w * out den[view.slice] += w # Pixels no view covers are undefined in the paper; keep them at zero return safe_divide(num, den) # guard against den == 0 ``` ```pseudocode # Prompt for LLM-written layouts [arxiv:2305.15393 §3.2] [§3.3] # Input: query caption; demos (caption, objects); k; canvas size in px # Output: prompt text for the LLM def build_layout_prompt(query, demos, k, canvas): # Keep the k demos closest to the query, most similar last [§3.3] shots = list(reversed(top_k_similar(query, demos, k))) text = task_instruction(canvas) # [§3.2] for demo in shots: # One CSS-style rule per object, normalized values [§3.2] text += demo_block(demo, canvas) # Leave the query's layout open for the model to complete return text + f"Prompt: {query}\nLayout:\n" ``` ```pseudocode # Region-modulated attention in one layer at one step [§3.2] [eq. 2] [eq. 6] # queries (Nq, d), keys and values (Nk, d); pair_map (Nq, Nk): 1 if the # query and key share a segment; area (Nq, 1): share of the image covered # by the query's segment; w: 1 for cross-attention, 0.3 for self-attention def region_attention(queries, keys, values, pair_map, area, t, w): # Modulate only in the first 30% of steps, t from 1 to 0.7 [§3.2] if t < 0.7: return softmax(queries @ keys.T / sqrt(d)) @ values scores = queries @ keys.T / sqrt(d) # Per-query range of the scores, so the change stays inside it [eq. 6] top = scores.max(axis=-1, keepdims=True) bottom = scores.min(axis=-1, keepdims=True) # Strength lambda_t = w * t^p, with p = 5 [eq. 3] [§3.2] strength = w * t ** 5 # Same-segment pairs move toward the row maximum [eq. 2] raise_up = strength * pair_map * (top - scores) * (1 - area) # Other pairs move toward the row minimum [eq. 2] lower_down = strength * (1 - pair_map) * (scores - bottom) * (1 - area) return softmax(scores + raise_up - lower_down, axis=-1) @ values ``` Notes on where variants differ: keypoints and spatial maps replace the phrase encoder and token count, but keep the gated layers. Gated cross-attention swaps the attention type in the gated layer. Scheduled grounding changes only the sampler. The painted-mask bias adds no layers and no training; it changes only the cross-attention logits of the frozen model. DenseDiffusion changes self-attention as well, with a positive and a negative term that are each scaled by the score range of their query. MultiDiffusion changes neither the weights nor the attention: it changes how the sampler combines views. LayoutGPT changes only the prompt and the parser, and its boxes go to a renderer that is not part of the method. LayoutDiffusion has no pretrained weights to reuse: its layout encoder and object-aware cross-attention are trained with the image model from scratch. Box guidance changes neither weights nor the attention scores; it changes only the latent, and only in the first steps. ## Contradictions Within GLIGEN, the paper and released code disagree on several details. The paper's numbers come from its own setups, and the baselines in its tables are copied from other papers, so differences in tables are not contradictions by themselves [arxiv:2301.07093 tab. 1] [tab. 3]. - **Caption and grounding dropout.** The paper states 10% for both [arxiv:2301.07093 app. A]. The released box config sets the caption probability to 0.5 [code: configs/GoldG+SBU+CC3M+O365_box_text_image.yaml#L65]. - **Grounding MLP depth.** The paper states three hidden layers of width 512 [arxiv:2301.07093 app. A]. The code has two hidden layers of width 512 and a final projection [code: ldm/modules/diffusionmodules/text_image_grounding_net.py#L19-L25]. - **Feed-forward gate and normalization.** Equation 8 gates only the attention output and has no normalization [arxiv:2301.07093 eq. 8]. The code adds a second gated residual around the feed-forward block and applies LayerNorm before the gated attention [code: ldm/modules/attention.py#L241-L242]. - **Scheduled sampling.** Equation 10 is a hard switch at `tau * T` [arxiv:2301.07093 eq. 10]. The code has a hard switch in its example settings but a three-stage schedule in general: on, then a linear decay, then off [code: gligen_inference.py#L31-L66] [code: gligen_inference.py#L475]. - **Optimizer.** The paper says Adam [arxiv:2301.07093 app. A]. The code uses AdamW with weight decay defaulting to 0 [code: trainer.py#L245]. - **Separate text and image MLPs.** The code runs separate MLPs for text and image entities, so each box gives two tokens [code: ldm/modules/diffusionmodules/text_image_grounding_net.py#L19-L33]. The paper does not say whether they share weights. The GLIGEN card lists these in full. Within LayoutDiffusion, the paper and released code disagree on the variance schedule, the classifier-free formula, the query positional embedding, and the padding mask [arxiv:2303.17189 §4.2] [§3.4] [§3.3] [code: configs/COCO-stuff_256x256/LayoutDiffusion_large.yaml#L38]. The paper's main text gives a fixed linear variance, while the code learns it [code: configs/COCO-stuff_256x256/LayoutDiffusion_large.yaml#L45-L49]. The guidance scale is written one way in the main text and another in the appendix and code, and the shipped default 1.0 corresponds to 2.0 in the main-text form [arxiv:2303.17189 app. B.2] [code: scripts/classifier-free_sample_for_layout.py#L125]. Within Layout-Guidance, the released code differs from the paper in the stopping rule, the loss normalisation and the noise schedule. The code repeats guidance while the normalised loss exceeds 0.2 [code: inference.py#L61], a threshold the paper does not give, and averages the energy in a way that changes the effective η [code: utils.py#L57-L59]. The paper says linear betas, and the code uses scaled linear [code: conf/base_config.yaml#L19-L21]. The forward variant is not in the release [code: README.md#L89-L94] [arxiv:2304.03373 §3.2]. Across papers, LayoutGPT's ablation renders the same GPT-3.5 spatial layouts with two renderers. With GLIGEN the image accuracy is 52.86 and with Layout-Guidance it is 31.02, while the layout accuracy is identical at 82.54 [arxiv:2305.15393 tab. 4]. This is not a conflict; it shows that the renderer sets the image-level result. DenseDiffusion's card has one paper-versus-code gap. The paper replaces parts of the text features with separately encoded segment phrases [arxiv:2308.12964 §3.2]. The code replaces only the first occurrence of each phrase's token span [code: gradio_app.py#L234-L243]. Across the seven papers, the protocols differ, so their numbers are not directly comparable. GLIGEN reports YOLO AP and GLIP-based scores [arxiv:2301.07093 tab. 1] [tab. 3]. MultiDiffusion reports FID and IoU against Blended Latent Diffusion and Stable Inpainting [arxiv:2302.08113 tab. 1] [tab. 2]. LayoutGPT reports layout accuracy and GLIP-based image accuracy with GLIGEN as renderer [arxiv:2305.15393 tab. 2]. DenseDiffusion and MultiDiffusion both report a region IoU, but on different prompt sets: 250 captions in DenseDiffusion [arxiv:2308.12964 §4.1] and a 1K COCO subset in MultiDiffusion [arxiv:2302.08113 §5.2], so their IoU values are not comparable. Layout-Guidance reports COCO mAP with a detector on 1,000 images, and LayoutDiffusion reports FID on COCO-Stuff with a 3,097-layout protocol [arxiv:2304.03373 tab. 3] [arxiv:2303.17189 tab. 1]. The eDiff-I card has no layout metric to set against them [arxiv:2211.01324 §5.4]. The cited results do not conflict with one another, but no two of the seven sets of numbers can be read as the same test. ## Pitfalls - **Box coordinate convention.** The paper gives boxes as corner coordinates but does not say whether they are normalized or in pixels. The example settings use values in [0, 1] [code: gligen_inference.py#L474]. - **Short runs may show no grounding.** The gate starts closed, and the paper reports the grounded layers start attending to boxes only around 60k to 70k iterations [arxiv:2301.07093 app. F]. - **Do not drop the Fourier box embedding.** Replacing it with a plain MLP keeps FID close (5.82 against 5.80) but cuts box AP from 21.7 to 3.2 [arxiv:2301.07093 app. B]. - **Image tokens need a rescaling constant.** Image embeddings are projected and rescaled to norm 28.7, the average text-feature norm the authors observed [arxiv:2301.07093 app. A]. Raw image embeddings have a different scale. - **Projected text features converge slowly.** The paper reports slow convergence when the projected CLIP embedding replaces the EOS feature [arxiv:2301.07093 app. A]. - **Null captions for detection data.** A null caption is the paper's choice. A pseudo-caption built from entity names raises FID from 5.61 to 7.40 [arxiv:2301.07093 app. B]. - **Maximum entities.** The code and paper cap entities at 30 per image [arxiv:2301.07093 app. A] [code: configs/GoldG+SBU+CC3M+O365_box_text_image.yaml#L64]. The paper does not say what happens above the cap. - **One detector does three jobs.** GLIP produces pseudo-boxes for training data, and the LVIS GLIP-score uses GLIP as the evaluation detector [arxiv:2301.07093 §5.1] [§5.2] [tab. 3]. Read the metric with that in mind. - **Not like for like.** LAMA is trained on LVIS, while GLIGEN is evaluated zero-shot on LVIS [arxiv:2301.07093 §5.2]. - **Inpainting adds input channels.** The paper adds five trainable input channels for inpainting [arxiv:2301.07093 app. C]. The code README says they start at zero when converting a checkpoint [code: README.md#L97]. - **Released weights carry a licence condition.** The code README says the checkpoints and demo follow the Latent Diffusion and Stable Diffusion licence terms [code: README.md#L115]. This applies to any use of released weights, not to the method itself. The paper's own license is CC BY 4.0, and the code is MIT. - **No strength default for paint-with-words.** The user weight `w'` has no default, and the paper gives no rule for choosing it [arxiv:2211.01324 §4.3]. - **Scope of the max is not stated.** It is not stated whether the maximum logit in the strength term is taken per head, per sample or over the whole matrix [arxiv:2211.01324 §4.3]. - **Samples, not metrics.** Paint-with-words results are shown as samples, so the paper gives no number for how closely the painted region is followed [arxiv:2211.01324 §5.4] [fig. 17]. - **No official code.** eDiff-I has no released code or weights, and the sketch above follows the method description, not an implementation [arxiv:2211.01324 §5] [§5.1]. - **Bootstrapping step count.** The paper gives the stop point as 20% of the process, or step 800, but does not state the total step count. The code bootstraps the first 20 steps of a 50-step default, which is 40% [arxiv:2302.08113 §4.2] [code: region_based.py#L108]. Pick one convention and state it. - **Panorama size floor.** The panorama must have height and width at least as large as the reference's crop [arxiv:2302.08113 §4.1]. - **Mask resizing is unspecified.** The paper does not say how masks are resized to latent size. The code uses nearest-neighbour resizing and a 0.5 threshold, so thin structures can vanish [code: region_based.py#L165-L173]. - **Claims versus table.** The text says MultiDiffusion outperforms the baselines in all metrics [arxiv:2302.08113 §5.1]. In the table, CLIP-score ties Blended Latent Diffusion at 0.27 [tab. 1]. The standard deviations are large next to the region gains [tab. 2]. - **Random exemplars break 3D LayoutGPT.** Random exemplars raise bedroom out-of-bounds from 43.26% to 85.58%. In 2D the effect is small [arxiv:2305.15393 app. C.1] [app. B]. - **Canvas size conflicts in LayoutGPT.** The main text gives a 256 px maximum, and the appendix instruction text says 64 px [arxiv:2305.15393 §3.2] [app. A]. Use one value and state it. - **Rendering is the bottleneck for LayoutGPT.** Layout accuracy is close to human layouts, but image accuracy is lower, which the paper attributes to rendering and detection [arxiv:2305.15393 §4.2]. - **LLM snapshots change the numbers.** The results depend on the API model snapshots and on sampling at temperature 0.7. The tables show no variance, so re-run before quoting [arxiv:2305.15393 app. A]. - **Reported claims without numbers.** LayoutGPT's 100% colour binding at layout level has no table [arxiv:2305.15393 §4.2], and its keypoint planning result has no metrics [app. D]. - **Phrases must match the prompt.** The paper says each segment is a part of the full caption [arxiv:2308.12964 §3]. The code finds each phrase by string match, and a missing phrase gets no positive pairs without an error [code: gradio_app.py#L234-L245]. Check every phrase before running. - **The step gate is hard-coded.** The code modulates while the attention-call counter is below 50 × 0.3, which assumes 50 steps and 32 attention calls per UNet pass [code: gradio_app.py#L132]. If you change the step count, update the gate too. - **Only half of the guidance batch is changed.** The code modulates the conditional half of the classifier-free batch [code: gradio_app.py#L147-L163]. The paper's equations do not mention guidance [arxiv:2308.12964 eq. 2]. - **Unlabelled pixels.** The paper does not say how pixels outside every segment are treated. The code gives them the negative update [code: gradio_app.py#L156-L163]. - **Small segments lose quality.** The paper reports quality loss when one segment is much smaller than the others [§3.2]. The area factor is near zero when one segment covers almost the whole image [eq. 2]. - **SOA-I depends on a detector.** SOA-I and IoU both use YOLOv7 [§4.1]. The authors say SOA-I correlates only loosely with human judgement [§4.2]. The 250-caption benchmark was built by the authors [§4.1]. - **No text, no pretrained prior (LayoutDiffusion).** The paper trains from scratch on box-labelled data and names combining it with text-to-image pretraining as future work [arxiv:2303.17189 §5]. Expect no text prompts, and a fixed class list (185 classes for COCO-Stuff, 180 for Visual Genome) [arxiv:2303.17189 tab. 4]. - **Padding takes part in attention (LayoutDiffusion).** The shipped configs do not mask padded layout slots, so changing the padded length changes the model's inputs, not only its speed [code: layout_diffusion/layout_encoder.py#L252]. - **Gradient steps and fidelity (box guidance).** The loss scale η trades layout for image quality: FID is 88.4 at η = 100 and 129.7 at η = 500 [arxiv:2304.03373 app. C]. The paper gives η = 30 to 50 as the working range. - **Guidance needs the prompt tokens (box guidance).** The code finds each phrase's token by splitting the prompt on spaces, so a word with an attached comma raises an error, and a repeated noun maps to its first occurrence [code: utils.py#L62-L73]. Boxes that cover no latent cell give no gradient [code: utils.py#L21-L26]. - **Forward guidance is not released (box guidance).** Only the backward variant ships. Do not assume the forward variant's numbers from [arxiv:2304.03373 tab. 1] apply to code you have. ## Papers - [GLIGEN](/papers/2301.07093): boxes with phrases, plus keypoints, reference images and spatial maps, added to a frozen latent diffusion model through gated self-attention; scheduled grounding at sampling time. - [eDiff-I](/papers/2211.01324): a text-to-image cascade with expert denoisers; its training-free paint-with-words control biases cross-attention with painted phrase masks. - [MultiDiffusion](/papers/2302.08113): one pretrained text-to-image model run on overlapping crops or masked regions at every step, with the outputs fused by weighted averaging; no training. - [LayoutGPT](/papers/2305.15393): a training-free prompt method in which a language model writes the layout as CSS-style rules, which a layout-to-image model then renders. - [DenseDiffusion](/papers/2308.12964): a training-free method for frozen Stable Diffusion that modulates cross- and self-attention with one region map per caption segment, in the first 30% of steps. - [LayoutDiffusion](/papers/2303.17189): an image-space diffusion model trained from scratch on boxes and classes, with a layout transformer and object-aware cross-attention; no text prompt. - [Layout-Guidance](/papers/2304.03373): a training-free method for Stable Diffusion that takes gradient steps on the latent in the first denoising steps, so that each guided word's cross-attention falls inside its box.