Training-Free Layout Control with Cross-Attention Guidance

In plain words

Text-to-image models often ignore instructions about where things should go in the picture. Asking for "a dog to the left of a cat" may give you a dog, a cat, or a strange mix of both, placed wherever the model likes.

This paper's method keeps the image model unchanged and nudges the picture while it is being made. For each word in the prompt, the model has an internal map of where it is "painting" that word. The method checks whether that map sits inside the box the user asked for, and if not, it adjusts the noisy image representation so the map moves into the box. This adjustment is only applied during the first steps of generation, because that is when the overall layout is settled.

The authors also compare this with a simpler alternative that directly edits the maps instead of the image representation, and find the adjustment approach works better. They further show the same idea can reposition a subject in a real photo after the model has been personalised on it.

TL;DR

• Training-free layout control for Stable Diffusion v1.5: no weights change §3.2 §4.1.
• Backward guidance: gradient steps on the latent that minimise an energy on cross-attention maps inside the target boxes, applied to the first 10 denoising steps, up to 5 updates each §3.2 eq. 2 eq. 3 §4.1.
• Beats the forward (attention-biasing) variant on VISOR and on COCO and Flickr30K layout metrics, at about twice the per-image time in the authors' setup tab. 1 tab. 3.
• Code releases only the backward guidance on text-to-image; forward guidance, noise selection and all benchmark evaluation are absent README.md.

Problem

Text-conditioned diffusion models do not reliably follow spatial instructions such as "left of", "on top of" or unusual compositions. For a prompt like "a dog to the left of a cat", the relation is not always depicted, and the model can produce a dog-cat hybrid §1. Fixing this by training layout-conditioned models requires new data and training; training-free methods existed (for example attention biasing), but the paper argues they are limited because other words, the special start and end tokens, and the initial noise all influence layout §3.3 §2.

Method

Background: cross-attention

Each cross-attention layer computes, for a spatial location u and prompt token index i, a weight that is a softmax over the N tokens §3.1:

A^{(\gamma)}_{ui} = \frac{\exp\langle Q_u^{(\gamma)}, K_i^{(\gamma)}\rangle} {\sum_{j=1}^{N} \exp\langle Q_u^{(\gamma)}, K_j^{(\gamma)}\rangle}

§3.1 (unnumbered in the paper). The weights for each location sum to one, so tokens "compete" for each location §3.2. Layout control is done by changing these weights in selected layers \gamma \in \Gamma §3.2.

Forward guidance (not in the released code)

Replace the map of a guided token i by a blend with a normalised window g placed on the box, scaled by the total mass of that token eq. 1 §3.2:

A^{(\gamma)}_{ui} \leftarrow (1-\lambda)\, A^{(\gamma)}_{ui} + \lambda\, g^{(\gamma)}_u \sum_{v} A^{(\gamma)}_{vi}

eq. 1. The window is a Gaussian blob whose \sigma depends on the box size §3.2, footnote. The paper normalises with a softmax over tokens after the blend §3.2. The default is \lambda = 0.8, applied to every layer for the first 40 steps §4.1. Applying it to the start and end tokens as well is reported to improve it, but it can damage image fidelity §3.3.

Backward guidance (the main method)

Instead of editing maps, define an energy per guided token i, box B and layer \gamma eq. 2 §3.2:

E(A^{(\gamma)}, B, i) = \left( 1 - \frac{\sum_{u \in B} A^{(\gamma)}_{ui}}{\sum_{u} A^{(\gamma)}_{ui}} \right)^2

eq. 2. At each application of the denoiser, for each layer in \Gamma, the gradient of this energy updates the latent eq. 3 §3.2:

\mathbf{z}_t \leftarrow \mathbf{z}_t - \sigma_t^2\, \eta\, \nabla_{\mathbf{z}_t} \sum_{\gamma \in \Gamma} E(A^{(\gamma)}, B, i)

eq. 3, with \sigma_t = \sqrt{(1-\alpha_t)/\alpha_t} and \eta > 0 the guidance strength §3.2. Gradient updates alternate with ordinary denoising steps §3.2. Because the latent is changed, all tokens' maps move, not only the guided one §3.2 fig. 3.

Layers used: the mid-block layer and the layers of the first up-sampling block of the U-Net §4.1. Strength: \eta = 30 by default; values 30 to 50 "work well" §4.1. Schedule: guidance on the first 10 steps, repeated 5 times per step §4.1.

# Backward layout guidance during sampling  [§3.2] [§4.1]
# Input:  latent (B, 4, h, w), prompt embedding, guided (token, box) pairs
# Output: latent after guidance and denoising

def guided_sampling(latent, prompt_emb, guided_pairs, scheduler):
    for step, t in enumerate(scheduler.timesteps):
        # Guidance only on the first 10 steps  [§4.1]
        if step < 10:
            # Up to 5 gradient updates per step  [§4.1]
            for _ in range(5):
                # Cross-attention maps of the layers in Gamma  [§3.2]
                maps = cross_attention_maps(latent, t, prompt_emb)
                loss = 0
                for token, box in guided_pairs:
                    # Energy of eq. 2 for one token and one layer  [eq. 2]
                    inside = maps[:, box_mask(box), token].sum()
                    total = maps[:, :, token].sum()
                    loss = loss + (1 - inside / total) ** 2
                # Latent update of eq. 3 with eta = 30 by default  [eq. 3]
                grad = gradient(loss, latent)
                latent = latent - eta * sigma[step] ** 2 * grad
        # Ordinary denoising step with classifier-free guidance 7.5  [app. A]
        noise = denoise_with_cfg(latent, t, prompt_emb, scale=7.5)
        latent = scheduler.step(noise, t, latent)
    return latent

The paper states the energy per token and layer eq. 2 and sums it over layers in the update eq. 3; how several guided tokens are combined into one loss is not stated in the text. The code's choice is in Paper vs code.

Analysis that motivates backward guidance

• Start and end tokens carry layout information: the end-token maps tend to mark the salient foreground, and the start-token maps the background §3.3 fig. 4 app. E. Backward guidance needs no special handling of them, since it updates the latent §3.3 app. E.
• Initial noise carries an intrinsic layout: the same seed gives similar layouts across different prompts §3.3 app. D. Choosing the noise whose attention already matches the boxes helps: sample several noises, run guidance for a few steps, keep the one with the lowest energy §3.3 app. D. The number of candidates is not stated.
• Forward guidance struggles because non-guided tokens with spatial overlap (for example "climbing" near "two climbers") pull the map away from the box §3.3 fig. 3.

Real-image layout editing

Textual Inversion learns a token for the subject from the photo (500 steps), then DreamBooth fine-tunes the text encoder and U-Net (150 steps). Backward guidance is then applied with a prompt that contains the learned token and user boxes §3.4 app. A app. F fig. 8.

Hyperparameters

Results

All numbers are author-reported. Hardware is not stated in the text §4.

Forward vs backward, VISOR protocol, 1,000 random prompts tab. 1:

Comparison with text-to-image models on the VISOR protocol tab. 2. The table caption does not give a prompt count; the appendix describes the full protocol with 25,280 prompts app. B. The Stable Diffusion row here differs from Table 1 because the sample differs tab. 1:

Layout-to-image on COCO 2014 (1,000 images) and Flickr30K Entities (1,000 images) tab. 3. Layout is measured by a detector (Detic) on generated images. The caption states mAP uses an IoU threshold of 0.3. FID is lower-is-better; AP and mAP are higher-is-better:

The text reports gains of 11.6 mAP and 9.6 AP_P over BoxDiff on these benchmarks §4.3, which match the table entries.

Ablations (from the appendix; Flickr30K, 1,000 captions app. C):

Guidance steps app. C:

Layers receiving the loss app. C:

Loss scale \eta app. C:

Noise selection (NS) app. D:

Qualitative results: fig. 5 (forward vs backward), fig. 6 (VISOR prompts), fig. 7 (box control of size and position), fig. 10 (loss scale). The appendix has further VISOR and editing examples app. F.

Paper vs code

Repository: silent-chen/layout-guidance, the first author's account, linked from the project page named in the paper's author block [arxiv.tex] and from the project page itself. Commit 50fcf75. There is no LICENSE file, so cite by path and line only. README.md

• Early stop inside the guidance loop. The paper repeats guidance 5 times per step on the first 10 steps §4.1. The code repeats while the normalised loss is above 0.2 and stops after 5 iterations or at step 10 inference.py base_config.yaml. The threshold is not in the paper.
• Loss normalisation. The paper's energy is a squared term per token and layer, summed over layers eq. 2 eq. 3. The code averages over the batch and heads dimension utils.py utils.py, averages over the positions of a phrase utils.py utils.py, then divides the total by (number of objects × number of layers) utils.py. The effective \eta therefore depends on the normalisation, which the paper does not state.
• Layers. The paper says "block" throughout; the appendix says each cross-attention block holds 2, 1 or 3 repeated layers app. A. The code takes all three layers of the first up block and the single mid-block layer unet_2d_condition.py unet_2d_blocks.py utils.py. Whether the ablation row "Up-1" app. C also used all three layers is not stated.
• Noise schedule. The paper says "linear" beta with start 0.00085 and end 0.012 app. A. The code uses scaled_linear with the same endpoints base_config.yaml text_inversion.py.
• Gradient branch. The gradient is taken through the conditional text embedding only, without classifier-free guidance, while the update is applied to the same latent inference.py. The paper does not state this.
• Latent update. The code multiplies the loss by \eta and then multiplies the gradient by the scheduler's \sigma_t^2 inference.py. This matches eq. 3 in form. Whether the scheduler's sigma equals the paper's \sqrt{(1-\alpha_t)/\alpha_t} was not checked against the library.
• Mask. Box coordinates are truncated to integers, and the upper bounds are exclusive utils.py. The paper writes u \in B with no discretisation rule.
• Not in the release. Forward guidance eq. 1 and noise selection §3.3 are absent from the repository. The README lists forward guidance and layer choice as open to-do items README.md. Benchmark evaluation (VISOR, COCO, Flickr30K) is not in the repository. The repository contains only example inference, Textual Inversion and DreamBooth scripts inference.py text_inversion.py dreambooth.py.
• Editing hyperparameters. The paper gives 500 and 150 steps app. A; the shipped configuration matches real_image_editing_config.yaml real_image_editing_config.yaml.

Pitfalls

• Stopping rule and cost. The early stop in the code means runtime and quality depend on the threshold, which the paper does not mention. Backward guidance needs a backward pass through the U-Net for each update. The authors' runtime is about 8 s versus 4 s for the baseline on the VISOR sample tab. 1. Hardware is not stated.
• Loss scale. Large \eta improves layout but damages fidelity: FID is 88.4 at \eta=100 and 129.7 at \eta=500 app. C fig. 10.
• Ablation text is inconsistent with its table. The appendix says loss factors 20 to 50 have "the lowest" AP_P and mAP app. C. The table shows the highest values there (for example mAP 16.7 at 30, AP_P 35.8 at 50) app. C. Read the numbers, not the sentence.
• Guidance steps. The text says mAP gains stop after 10 steps and FID then worsens; the authors attribute the FID decline to drifting away from the text embedding app. C. This is a hypothesis.
• Fragile prompt-to-token mapping. The code finds phrase positions by splitting the prompt on spaces, stripping a trailing period, taking the first match of each phrase word, and adding 1 for the start token utils.py utils.py. A word that does not appear as an exact space-separated token (for example with a comma attached) raises an error. A repeated noun always maps to its first occurrence.
• Tiny boxes. Boxes whose truncated integer extent covers no latent cell give a zero mask and no gradient utils.py.
• Fixed resolution. The code fixes a 512 px latent of 4 × 64 × 64 inference.py. The paper does not state the resolution it used.
• Seed-dependent results. Results depend on the random seed, which is fixed in the code base_config.yaml. The paper's seed is not stated. Noise selection is the paper's remedy §3.3 app. D.
• Backbone coverage. Main results use Stable Diffusion v1.5 §4.1 app. A. The only other backbone reported is GLIGEN with the method added on top §4.3 tab. 3. No other diffusion models are reported.

Lineage

• Builds on: Stable Diffusion §3.1, cross-attention control by prompt-to-prompt [arXiv:2208.01626] §3.2, and personalisation with Textual Inversion [arXiv:2208.01618] and DreamBooth [arXiv:2208.12242] for the editing application §3.4.
• Compared against (named only, nothing from their content used): GLIGEN [/papers/2301.07093]; eDiff-I [/papers/2211.01324]; MultiDiffusion [/papers/2302.08113]; BoxDiff [arXiv:2307.10816]; HFG §4.3; VISOR benchmark [arXiv:2212.10015] §4.2.
• Superseded by: not stated.