Layout-to-image generation

Overview

A caption says what to draw but not where. Layout-to-image generation adds a second input, a set of spatial conditions, so each object gets a region. The classic approach trains a generator from scratch on a fixed set of categories 2301.07093 §1 §2. LayoutDiffusion is one such model: it is trained from scratch on box-labelled images with a fixed class list [arxiv:2303.17189 §3] tab. 4. Most of the other approaches here keep a large pretrained text-to-image model and add spatial control on top of it; eDiff-I is the exception among those, since it trains its own cascade. GLIGEN trains new gated layers that read boxes and phrases, and its base weights stay frozen 2301.07093 §4.2. eDiff-I trains its own cascade, but its paint-with-words control adds no training and biases cross-attention with user-painted masks 2211.01324 §4 §4.3. MultiDiffusion runs a pretrained model on overlapping crops or masked regions and fuses the outputs, with no training 2302.08113 §3 §4.2. DenseDiffusion keeps Stable Diffusion frozen and modulates its attention with one region map per phrase 2308.12964 §3.2. LayoutGPT moves the layout step out of the generator: a language model writes the boxes from the caption, and a separate layout-to-image model draws them 2305.15393 §3.2 §4.1. Layout-Guidance is a training-free alternative that edits the latent of Stable Diffusion during sampling, so that each guided word's cross-attention falls inside its box 2304.03373 §3.2.

Beyond training, the approaches differ in what they take as input and what they measure. The GLIGEN invariant is that the new layers start in a state that reproduces the original model, and the conditions are encoded in the same space as the caption, so the model can use phrases it did not see with boxes during training 2301.07093 §4.1 §4.2.

Variants

Boxes with phrases on a frozen latent diffusion model

Each entity is a phrase and a box. The phrase goes through the caption's text encoder, the box coordinates are Fourier-embedded, and a small MLP fuses them into one grounding token 2301.07093 §4.1 app. A. New gated self-attention layers attend over visual and grounding tokens together, and their output is added back through a gate that starts at zero, so the first forward pass equals the original model 2301.07093 §4.2 eq. 8. Card: GLIGEN.

Gated self-attention versus gated cross-attention

The paper also tried gated cross-attention, where grounding tokens enter through cross-attention instead. Its ablation on COCO2014 with detection data gives AP 16.6 for cross-attention and 21.7 for self-attention, at similar FID 2301.07093 app. B. The paper keeps self-attention for its main model. Card: GLIGEN.

Other grounding inputs through the same path

The same grounding path takes other conditions with a different encoder:

• Keypoints. One learned embedding per person ties that person's keypoints together 2301.07093 app. A. Reported on COCO2017 with a caption app. D.
• Reference images. An image encoder replaces the text encoder for image entities 2301.07093 §4.1 app. A.
• Spatial maps. A small convolutional backbone turns an edge, depth or semantic map into a fixed number of tokens. The map is also concatenated to the U-Net input 2301.07093 §4.1 app. A.

Card: GLIGEN. Only the box-with-phrase setting is evaluated in the paper's main tables; the others are supplementary 2301.07093 app. D app. E.

Scheduled grounding at sampling time

This is a sampler rule, not a training change. Grounding is on for the first tau * T denoising steps and off afterwards, so the layout is fixed early and the original model finishes the details 2301.07093 §4.2 eq. 10. The paper shows this only qualitatively, using tau = 0.2 2301.07093 §5.4 fig. 7. Card: GLIGEN.

eDiff-I reports that its text prompt drives the layout at high noise and is largely ignored at low noise 2211.01324 §4 fig. 3 fig. 4. That supports the early timing of grounding, but the observation is about the prompt, not about grounding, and eDiff-I does not test a grounding switch. DenseDiffusion also modulates only the early steps, t from 1 to 0.7, because the authors saw no clear gain beyond that 2308.12964 §3.2.

Boxes and classes, trained from scratch

LayoutDiffusion has no text prompt and no pretrained image model. Each object is a box and a class id. A six-layer transformer first lets the objects attend to each other, and an object-aware cross-attention block then lets each image patch attend to the objects, using each patch's and each object's box position 2303.17189 §3.2 §3.3 tab. 4. It is trained with the layout dropped 20% of the time, so classifier-free guidance is available 2303.17189 §3.4 app. C.1. Author-reported, on COCO-Stuff at 256 px, FID 15.61 against 29.10 for PLGAN and 31.18 for LostGAN-v2; on Visual Genome at 256 px, 15.63 2303.17189 tab. 1. These baselines are GANs only. The model is larger than LDM-4 (569M against 306M parameters), and the paper reports FID 31.68 against 40.91 for LDM-4 on 2,048 images 2303.17189 tab. 3. Card: LayoutDiffusion.

Painted masks with an attention bias

The user selects a prompt phrase and paints a binary mask for it. The masks are resized to each cross-attention layer's resolution. A bias built from the masks is added to the cross-attention logits before the softmax, and its strength grows with the noise level and with the user's weight w' 2211.01324 §4.3 fig. 6. No weights are trained. The input is a mask for a prompt phrase, not a box, so it does not name entities separately from the caption. The paper shows paint-with-words results as samples only 2211.01324 §5.4 fig. 17. Card: eDiff-I.

Region maps with attention modulation on Stable Diffusion

DenseDiffusion splits the caption into non-overlapping segments, each with a binary region map 2308.12964 §3. For the first 30% of the denoising steps it changes the scores of both cross-attention and self-attention. Pairs inside one segment are raised toward the row maximum, and pairs that do not share a segment are lowered toward the row minimum 2308.12964 eq. 2 eq. 6. The change is scaled by a factor that grows for small segments §3.2. The weights are 1 for cross-attention and 0.3 for self-attention, with a power of 5 on the step schedule §3.2 eq. 3. The paper also encodes each phrase on its own and substitutes that encoding into the caption's text features §3.2. On 250 MS-COCO captions the authors report SOA-I 77.61 against 73.08 for Stable Diffusion, and layout IoU 34.99 against 23.76 for SD-Pww 2308.12964 tab. 2 tab. 3. Card: DenseDiffusion.

Box energy on cross-attention, latent updated at sampling

Layout-Guidance keeps Stable Diffusion v1.5 unchanged. For each guided word and box, it defines an energy: one minus the share of that word's cross-attention mass that falls inside the box 2304.03373 §3.2 eq. 2. During the first 10 denoising steps, it takes gradient steps on the latent to lower the summed energy of several layers, with strength η = 30 by default 2304.03373 §4.1 eq. 3. The authors also describe a forward variant that blends attention maps directly; the released code omits it, see Contradictions 2304.03373 §3.2. On 1,000 random VISOR prompts the backward variant reaches 96.9% conditional accuracy, against 59.8% for Stable Diffusion, at about 8 s per image against 4 s 2304.03373 tab. 1. On 1,000 COCO 2014 images, mAP with a detector rises from 19.2 to 35.7, while FID moves from 72.3 to 73.3 2304.03373 tab. 3. Applied on top of GLIGEN, the gain is smaller: mAP 62.8 to 65.1 2304.03373 tab. 3. Card: Layout-Guidance.

Mask-weighted fusion of denoising views

MultiDiffusion has no grounding input and adds no layers. At every denoising step it runs the pretrained Stable Diffusion v2 once per view, where a view is either a fixed crop of the canvas or the whole canvas weighted by a region mask. The next latent is a per-pixel weighted average of the view outputs 2302.08113 §3 eq. 5 alg. 1. Wide panoramas use fixed crops with equal weights. Region control gives each object a binary mask and a prompt, and for tight masks a bootstrapping rule in the early steps replaces the view outside the mask with a constant colour, so the object fills the mask 2302.08113 §4.2 eq. 9. The paper sets this rule for the first 20% of the process §4.2. Author-reported results on a COCO subset: region IoU 0.26 against 0.17 for Blended Latent Diffusion and 0.16 for Stable Inpainting, and 0.18 without bootstrapping 2302.08113 tab. 2. Panorama FID is 10.3, against 18.4 for Blended Latent Diffusion and 45.5 for Stable Inpainting 2302.08113 tab. 1. Card: MultiDiffusion.

LLM-written layouts

LayoutGPT does not train or change the generator. The prompt holds a task instruction, a few retrieved solved examples written as CSS-style rules with normalized coordinates, and the new caption 2305.15393 §3.2 §3.3. The reply is parsed back into boxes, and a layout-to-image model draws them. The main 2D results use GLIGEN as the renderer, and an ablation uses a training-free method 2305.15393 §4.1 tab. 4. On its NSR-1K numerical benchmark, GPT-3.5 reaches 86.33% layout accuracy against 92.56% for human layouts 2305.15393 tab. 2. For 3D bedrooms it reports fewer out-of-bounds placements than the supervised ATISS baseline, 43.26% against 49.88% 2305.15393 tab. 5. Card: LayoutGPT.

When to use which

Cost differs by approach. The GLIGEN paper gives no inference cost for the extra tokens, so a latency comparison with the unmodified model is not supported by the cited results 2301.07093 app. A. The eDiff-I card reports no cost for the attention bias either 2211.01324 §4.3. MultiDiffusion makes one reference call per view at every step, so its cost grows with the canvas area, and the paper notes the calls can be batched 2302.08113 app. B.1. LayoutGPT makes LLM calls for the layout step, and its 2D run samples five layouts per prompt 2305.15393 app. A. DenseDiffusion adds no weights, and the paper reports no timing or memory figures. Reading its code, the modulated steps build the full score matrix and take row extremes, while later steps use the library attention path gradio_app.py gradio_app.py. Backward box guidance adds a backward pass through the U-Net for each update in the first steps; the paper's timing is about 8 s against 4 s per image, with the hardware not stated 2304.03373 tab. 1. LayoutDiffusion's inference cost is 0.308 images per second on the paper's hardware, which it does not name, and its training cost is in V100-days 2303.17189 tab. 3.

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a # comment line above each step (with its citation), short trailing comments only for shapes or units.

# Grounding token for one entity  [arxiv:2301.07093 §4.1] [eq. 5] [app. A]
# Input:  phrase_feat (768,) text feature of the phrase, box (4,)
# Output: grounding_token (768,)

def grounding_token(phrase_feat, box):
    # Fourier-embed each of the four coordinates  [app. A]
    box_feat = fourier_embedding(box)              # (64,)
    # Fuse phrase and box with an MLP  [eq. 5]
    return mlp(concat(phrase_feat, box_feat))      # (768,)

# Gated self-attention, inserted into a frozen block  [§4.2] [eq. 8]
# Input:  visual (B, M, d), grounding (B, N, d), gamma (learned, init 0)
# Output: visual (B, M, d)

def gated_self_attention(visual, grounding, gamma, beta):
    M = visual.shape[1]
    # Joint self-attention over visual and grounding tokens  [eq. 8]
    joint = concat(visual, grounding, axis=1)      # (B, M + N, d)
    attended = self_attention(joint)[:, :M]
    # Gate is zero at initialization, so the block starts as the original
    return visual + beta * tanh(gamma) * attended

# Condition dropout during training  [arxiv:2301.07093 app. A]
# Input:  caption tokens, grounding tokens, drop_prob (paper: 0.1)

def dropout_conditions(caption, grounding, drop_prob):
    # Replace each condition with a null embedding at random, so the
    # unconditional branch used by classifier-free guidance exists
    if random() < drop_prob:
        caption = null_caption
    if random() < drop_prob:
        grounding = null_grounding
    return caption, grounding

# Scheduled grounding at sampling  [arxiv:2301.07093 §4.2] [eq. 10]
# tau in [0, 1]: fraction of denoising steps with grounding on

def grounding_weight(step, total_steps, tau):
    # beta = 1 for the first tau * T steps, then 0
    return 1.0 if step <= tau * total_steps else 0.0

# Box energy and latent update, first steps only  [arxiv:2304.03373 eq. 2-3]
# Input: latent z_t; per guided word i, box B_i; cross-attention maps A per layer
# Output: latent z_t after the guidance updates

def box_guidance(z_t, t, guided, layers, eta, sigma_t):
    for _ in range(max_repeats):              # the paper uses 5 per step [§4.1]
        energy = 0
        for layer in layers:
            A = cross_attention_maps(z_t, t, layer)
            for word, box in guided:
                # Share of the word's attention mass inside its box  [eq. 2]
                inside = A[in_box(box), word].sum()
                energy = energy + (1 - inside / A[:, word].sum()) ** 2
        # Gradient step on the latent, scaled by sigma_t^2  [eq. 3]
        z_t = z_t - sigma_t**2 * eta * grad(energy, z_t)
    return z_t

# Painted-mask bias in one cross-attention layer  [arxiv:2211.01324 §4.3]
# Input:  queries (N_i, d_k); keys, values (N_t, d_k)
#         token_mask (N_i, N_t): column j holds the mask of the phrase that
#         token j belongs to, resized to this layer's resolution; 0 if unpainted
# Output: attended values (N_i, d_k)

def painted_attention(queries, keys, values, token_mask, w_user, sigma):
    # Attention logits between image and text tokens  [§4.3]
    logits = queries @ keys.T                      # (N_i, N_t)
    # Strength grows with the noise level and scales with the largest
    # logit; the scope of that max is not stated in the paper  [§4.3]
    strength = w_user * log(1 + sigma) * max(logits)
    weights = softmax((logits + strength * token_mask) / sqrt(d_k))
    return weights @ values

# One MultiDiffusion step  [arxiv:2302.08113 alg. 1] [eq. 5]
# Input:  pretrained model phi; views with prompts and per-pixel weights
# Output: canvas after one step (H', W', C)

def fused_step(phi, canvas, views, prompts, weights, t):
    num = zeros_like(canvas)
    den = zeros_like(canvas)
    for view, prompt, w in zip(views, prompts, weights):
        # Crop the canvas to the view and run one denoising step  [alg. 1]
        crop = canvas[view.slice]
        out = phi(bootstrapped(crop, view, t), prompt, t)
        # Accumulate weighted outputs back at canvas positions  [eq. 5]
        num[view.slice] += w * out
        den[view.slice] += w
    # Pixels no view covers are undefined in the paper; keep them at zero
    return safe_divide(num, den)                   # guard against den == 0

# Prompt for LLM-written layouts  [arxiv:2305.15393 §3.2] [§3.3]
# Input:  query caption; demos (caption, objects); k; canvas size in px
# Output: prompt text for the LLM

def build_layout_prompt(query, demos, k, canvas):
    # Keep the k demos closest to the query, most similar last  [§3.3]
    shots = list(reversed(top_k_similar(query, demos, k)))
    text = task_instruction(canvas)                # [§3.2]
    for demo in shots:
        # One CSS-style rule per object, normalized values  [§3.2]
        text += demo_block(demo, canvas)
    # Leave the query's layout open for the model to complete
    return text + f"Prompt: {query}\nLayout:\n"

# Region-modulated attention in one layer at one step  [§3.2] [eq. 2] [eq. 6]
# queries (Nq, d), keys and values (Nk, d); pair_map (Nq, Nk): 1 if the
# query and key share a segment; area (Nq, 1): share of the image covered
# by the query's segment; w: 1 for cross-attention, 0.3 for self-attention

def region_attention(queries, keys, values, pair_map, area, t, w):
    # Modulate only in the first 30% of steps, t from 1 to 0.7  [§3.2]
    if t < 0.7:
        return softmax(queries @ keys.T / sqrt(d)) @ values
    scores = queries @ keys.T / sqrt(d)
    # Per-query range of the scores, so the change stays inside it  [eq. 6]
    top = scores.max(axis=-1, keepdims=True)
    bottom = scores.min(axis=-1, keepdims=True)
    # Strength lambda_t = w * t^p, with p = 5  [eq. 3] [§3.2]
    strength = w * t ** 5
    # Same-segment pairs move toward the row maximum  [eq. 2]
    raise_up = strength * pair_map * (top - scores) * (1 - area)
    # Other pairs move toward the row minimum  [eq. 2]
    lower_down = strength * (1 - pair_map) * (scores - bottom) * (1 - area)
    return softmax(scores + raise_up - lower_down, axis=-1) @ values

Notes on where variants differ: keypoints and spatial maps replace the phrase encoder and token count, but keep the gated layers. Gated cross-attention swaps the attention type in the gated layer. Scheduled grounding changes only the sampler. The painted-mask bias adds no layers and no training; it changes only the cross-attention logits of the frozen model. DenseDiffusion changes self-attention as well, with a positive and a negative term that are each scaled by the score range of their query. MultiDiffusion changes neither the weights nor the attention: it changes how the sampler combines views. LayoutGPT changes only the prompt and the parser, and its boxes go to a renderer that is not part of the method. LayoutDiffusion has no pretrained weights to reuse: its layout encoder and object-aware cross-attention are trained with the image model from scratch. Box guidance changes neither weights nor the attention scores; it changes only the latent, and only in the first steps.

Contradictions

Within GLIGEN, the paper and released code disagree on several details. The paper's numbers come from its own setups, and the baselines in its tables are copied from other papers, so differences in tables are not contradictions by themselves 2301.07093 tab. 1 tab. 3.

• Caption and grounding dropout. The paper states 10% for both 2301.07093 app. A. The released box config sets the caption probability to 0.5 GoldG+SBU+CC3M+O365_box_text_image.yaml.
• Grounding MLP depth. The paper states three hidden layers of width 512 2301.07093 app. A. The code has two hidden layers of width 512 and a final projection text_image_grounding_net.py.
• Feed-forward gate and normalization. Equation 8 gates only the attention output and has no normalization 2301.07093 eq. 8. The code adds a second gated residual around the feed-forward block and applies LayerNorm before the gated attention attention.py.
• Scheduled sampling. Equation 10 is a hard switch at tau * T 2301.07093 eq. 10. The code has a hard switch in its example settings but a three-stage schedule in general: on, then a linear decay, then off gligen_inference.py gligen_inference.py.
• Optimizer. The paper says Adam 2301.07093 app. A. The code uses AdamW with weight decay defaulting to 0 trainer.py.
• Separate text and image MLPs. The code runs separate MLPs for text and image entities, so each box gives two tokens text_image_grounding_net.py. The paper does not say whether they share weights.

The GLIGEN card lists these in full.

Within LayoutDiffusion, the paper and released code disagree on the variance schedule, the classifier-free formula, the query positional embedding, and the padding mask 2303.17189 §4.2 §3.4 §3.3 LayoutDiffusion_large.yaml. The paper's main text gives a fixed linear variance, while the code learns it LayoutDiffusion_large.yaml. The guidance scale is written one way in the main text and another in the appendix and code, and the shipped default 1.0 corresponds to 2.0 in the main-text form 2303.17189 app. B.2 classifier-free_sample_for_layout.py.

Within Layout-Guidance, the released code differs from the paper in the stopping rule, the loss normalisation and the noise schedule. The code repeats guidance while the normalised loss exceeds 0.2 inference.py, a threshold the paper does not give, and averages the energy in a way that changes the effective η utils.py. The paper says linear betas, and the code uses scaled linear base_config.yaml. The forward variant is not in the release README.md 2304.03373 §3.2.

Across papers, LayoutGPT's ablation renders the same GPT-3.5 spatial layouts with two renderers. With GLIGEN the image accuracy is 52.86 and with Layout-Guidance it is 31.02, while the layout accuracy is identical at 82.54 2305.15393 tab. 4. This is not a conflict; it shows that the renderer sets the image-level result.

DenseDiffusion's card has one paper-versus-code gap. The paper replaces parts of the text features with separately encoded segment phrases [arxiv:2308.12964 §3.2]. The code replaces only the first occurrence of each phrase's token span gradio_app.py.

Across the seven papers, the protocols differ, so their numbers are not directly comparable. GLIGEN reports YOLO AP and GLIP-based scores 2301.07093 tab. 1 tab. 3. MultiDiffusion reports FID and IoU against Blended Latent Diffusion and Stable Inpainting 2302.08113 tab. 1 tab. 2. LayoutGPT reports layout accuracy and GLIP-based image accuracy with GLIGEN as renderer 2305.15393 tab. 2. DenseDiffusion and MultiDiffusion both report a region IoU, but on different prompt sets: 250 captions in DenseDiffusion 2308.12964 §4.1 and a 1K COCO subset in MultiDiffusion 2302.08113 §5.2, so their IoU values are not comparable. Layout-Guidance reports COCO mAP with a detector on 1,000 images, and LayoutDiffusion reports FID on COCO-Stuff with a 3,097-layout protocol 2304.03373 tab. 3 2303.17189 tab. 1. The eDiff-I card has no layout metric to set against them 2211.01324 §5.4. The cited results do not conflict with one another, but no two of the seven sets of numbers can be read as the same test.

Pitfalls

• Box coordinate convention. The paper gives boxes as corner coordinates but does not say whether they are normalized or in pixels. The example settings use values in [0, 1] gligen_inference.py.

• Short runs may show no grounding. The gate starts closed, and the paper reports the grounded layers start attending to boxes only around 60k to 70k iterations 2301.07093 app. F.

• Do not drop the Fourier box embedding. Replacing it with a plain MLP keeps FID close (5.82 against 5.80) but cuts box AP from 21.7 to 3.2 2301.07093 app. B.

• Image tokens need a rescaling constant. Image embeddings are projected and rescaled to norm 28.7, the average text-feature norm the authors observed 2301.07093 app. A. Raw image embeddings have a different scale.

• Projected text features converge slowly. The paper reports slow convergence when the projected CLIP embedding replaces the EOS feature 2301.07093 app. A.

• Null captions for detection data. A null caption is the paper's choice. A pseudo-caption built from entity names raises FID from 5.61 to 7.40 2301.07093 app. B.

• Maximum entities. The code and paper cap entities at 30 per image 2301.07093 app. A GoldG+SBU+CC3M+O365_box_text_image.yaml. The paper does not say what happens above the cap.

• One detector does three jobs. GLIP produces pseudo-boxes for training data, and the LVIS GLIP-score uses GLIP as the evaluation detector 2301.07093 §5.1 §5.2 tab. 3. Read the metric with that in mind.

• Not like for like. LAMA is trained on LVIS, while GLIGEN is evaluated zero-shot on LVIS 2301.07093 §5.2.

• Inpainting adds input channels. The paper adds five trainable input channels for inpainting 2301.07093 app. C. The code README says they start at zero when converting a checkpoint README.md.

• Released weights carry a licence condition. The code README says the checkpoints and demo follow the Latent Diffusion and Stable Diffusion licence terms README.md. This applies to any use of released weights, not to the method itself. The paper's own license is CC BY 4.0, and the code is MIT.

• No strength default for paint-with-words. The user weight w' has no default, and the paper gives no rule for choosing it 2211.01324 §4.3.

• Scope of the max is not stated. It is not stated whether the maximum logit in the strength term is taken per head, per sample or over the whole matrix 2211.01324 §4.3.

• Samples, not metrics. Paint-with-words results are shown as samples, so the paper gives no number for how closely the painted region is followed 2211.01324 §5.4 fig. 17.

• No official code. eDiff-I has no released code or weights, and the sketch above follows the method description, not an implementation 2211.01324 §5 §5.1.

• Bootstrapping step count. The paper gives the stop point as 20% of the process, or step 800, but does not state the total step count. The code bootstraps the first 20 steps of a 50-step default, which is 40% 2302.08113 §4.2 region_based.py. Pick one convention and state it.

• Panorama size floor. The panorama must have height and width at least as large as the reference's crop 2302.08113 §4.1.

• Mask resizing is unspecified. The paper does not say how masks are resized to latent size. The code uses nearest-neighbour resizing and a 0.5 threshold, so thin structures can vanish region_based.py.

• Claims versus table. The text says MultiDiffusion outperforms the baselines in all metrics 2302.08113 §5.1. In the table, CLIP-score ties Blended Latent Diffusion at 0.27 tab. 1. The standard deviations are large next to the region gains tab. 2.

• Random exemplars break 3D LayoutGPT. Random exemplars raise bedroom out-of-bounds from 43.26% to 85.58%. In 2D the effect is small 2305.15393 app. C.1 app. B.

• Canvas size conflicts in LayoutGPT. The main text gives a 256 px maximum, and the appendix instruction text says 64 px 2305.15393 §3.2 app. A. Use one value and state it.

• Rendering is the bottleneck for LayoutGPT. Layout accuracy is close to human layouts, but image accuracy is lower, which the paper attributes to rendering and detection 2305.15393 §4.2.

• LLM snapshots change the numbers. The results depend on the API model snapshots and on sampling at temperature 0.7. The tables show no variance, so re-run before quoting 2305.15393 app. A.

• Reported claims without numbers. LayoutGPT's 100% colour binding at layout level has no table 2305.15393 §4.2, and its keypoint planning result has no metrics app. D.

• Phrases must match the prompt. The paper says each segment is a part of the full caption 2308.12964 §3. The code finds each phrase by string match, and a missing phrase gets no positive pairs without an error gradio_app.py. Check every phrase before running.

• The step gate is hard-coded. The code modulates while the attention-call counter is below 50 × 0.3, which assumes 50 steps and 32 attention calls per UNet pass gradio_app.py. If you change the step count, update the gate too.

• Only half of the guidance batch is changed. The code modulates the conditional half of the classifier-free batch gradio_app.py. The paper's equations do not mention guidance 2308.12964 eq. 2.

• Unlabelled pixels. The paper does not say how pixels outside every segment are treated. The code gives them the negative update gradio_app.py.

• Small segments lose quality. The paper reports quality loss when one segment is much smaller than the others §3.2. The area factor is near zero when one segment covers almost the whole image eq. 2.

• SOA-I depends on a detector. SOA-I and IoU both use YOLOv7 §4.1. The authors say SOA-I correlates only loosely with human judgement §4.2. The 250-caption benchmark was built by the authors §4.1.

• No text, no pretrained prior (LayoutDiffusion). The paper trains from scratch on box-labelled data and names combining it with text-to-image pretraining as future work 2303.17189 §5. Expect no text prompts, and a fixed class list (185 classes for COCO-Stuff, 180 for Visual Genome) 2303.17189 tab. 4.

• Padding takes part in attention (LayoutDiffusion). The shipped configs do not mask padded layout slots, so changing the padded length changes the model's inputs, not only its speed layout_encoder.py.

• Gradient steps and fidelity (box guidance). The loss scale η trades layout for image quality: FID is 88.4 at η = 100 and 129.7 at η = 500 2304.03373 app. C. The paper gives η = 30 to 50 as the working range.

• Guidance needs the prompt tokens (box guidance). The code finds each phrase's token by splitting the prompt on spaces, so a word with an attached comma raises an error, and a repeated noun maps to its first occurrence utils.py. Boxes that cover no latent cell give no gradient utils.py.

• Forward guidance is not released (box guidance). Only the backward variant ships. Do not assume the forward variant's numbers from 2304.03373 tab. 1 apply to code you have.

Papers

• GLIGEN: boxes with phrases, plus keypoints, reference images and spatial maps, added to a frozen latent diffusion model through gated self-attention; scheduled grounding at sampling time.
• eDiff-I: a text-to-image cascade with expert denoisers; its training-free paint-with-words control biases cross-attention with painted phrase masks.
• MultiDiffusion: one pretrained text-to-image model run on overlapping crops or masked regions at every step, with the outputs fused by weighted averaging; no training.
• LayoutGPT: a training-free prompt method in which a language model writes the layout as CSS-style rules, which a layout-to-image model then renders.
• DenseDiffusion: a training-free method for frozen Stable Diffusion that modulates cross- and self-attention with one region map per caption segment, in the first 30% of steps.
• LayoutDiffusion: an image-space diffusion model trained from scratch on boxes and classes, with a layout transformer and object-aware cross-attention; no text prompt.
• Layout-Guidance: a training-free method for Stable Diffusion that takes gradient steps on the latent in the first denoising steps, so that each guided word's cross-attention falls inside its box.