Paper · 2023-06-08 · cs.CV
Grounded Text-to-Image Synthesis with Attention Refocusing
Training-free fix for layout-grounded text-to-image diffusion. Two losses on cross- and self-attention maps (CAR, SAR) are minimised by gradient steps on the latent during early denoising steps; GPT-4 supplies the boxes. Author-reported: on HRS, GLIGEN's spatial accuracy rises from 30.74 to 40.22.
Layout-to-image generationText-to-image generationCross-attentionDiffusion modelstext-to-imagediffusionlayout-controltraining-freeattention
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Text-to-image models often get the details of a prompt wrong. Ask for three parrots and you may get four, or objects may go missing or take on each other's attributes. The paper argues that the model's internal attention maps are the cause. Each word and each image region has a spot in those maps, and when two things look alike, their spots blur together.
The method has two parts. First, a large language model turns the prompt into a rough sketch: a labelled box for each object. Second, while the picture is being drawn, a few small corrections nudge the image so that each word's attention lands inside its box and stays out of the others. Nothing in the image model is retrained; the corrections happen at generation time.
The point is control. Given a layout, the picture follows it more closely, and the paper reports that this helps with counting, positions, sizes and colours across several existing models.
TL;DR
- No model is retrained. A prompt-to-boxes step (GPT-4 with examples) feeds a grounded or plain diffusion model, which is steered during sampling §3.3 §3.2.
- Two losses act on attention maps: CAR pulls each object word's cross-attention into its box, and SAR makes pixels inside a box attend less to pixels outside it §3.2.1 §3.2.2.
- The latent is updated by gradient descent in the first denoising steps, with a few repeats per step §3.2.3 alg. 1.
- Author-reported: adding the losses raises the HRS spatial accuracy of five base models and improves counting F1 for all five (GLIGEN spatial 30.74 to 40.22) tab. 1.
- The official code differs from the written losses in several ways (see "Paper vs code"). Implement from the code if you need the reported numbers.
Problem
Diffusion models tie prompt words to image regions through cross-attention, and they relate image pixels to each other through self-attention. When two regions have similar features, their attention can leak into each other. The paper identifies this as the cause of missing objects and blended attributes §1 §3.2.1.
Earlier training-free fixes changed cross-attention only §1 §2. Layout-based methods also need the boxes, which are costly to write by hand §3.2. The paper assumes that boxes with a caption per box are available, and that each caption word appears in the prompt §3.2.
Method
Overview. The pipeline has two phases, both without training §3. Phase one (text-to-layout) asks GPT-4 for labelled boxes on a 512 × 512 canvas, using a fixed instruction and in-context examples §3.3 tab. 7 fig. 3. Phase two (grounded generation) runs a pretrained diffusion model and, at each sampling step, adds the attention-refocusing update §3.2 fig. 3.
Attention. For queries Q and keys K of dimension d, a map A^t = softmax(QK^T / √d) gives, for each image location, a distribution over tokens (cross-attention) or over locations (self-attention) eq. 1 §3.1. The paper works on the 16 × 16 maps app. Implementation details.
Layout tokens. A grounded token i may own one or more boxes B_i, and Mask(B_i) is 1 inside those boxes and 0 outside §3.2.
CAR (cross-attention refocusing).
- Drop the
map, apply softmax over the remaining tokens, then apply Gaussian smoothing §3.2.1 app. Implementation details. - Foreground term: push each token's peak inside its boxes, L_FG = (1/q) Σ_i (1 − max(A_i · Mask(B_i))) eq. 2.
- Background term: push the peak outside the boxes down, L_BG = (1/q) Σ_i max(A_i · (1 − Mask(B_i))) eq. 3.
- L_CAR = L_FG + L_BG §3.2.1.
SAR (self-attention refocusing).
- For a pixel p inside a box B_i, take its self-attention map S_p and keep only the outside part, S_p^BG = S_p · (1 − Mask(B_i)) eq. 4 §3.2.2.
- Per pixel, average the outside part over the outside pixels: L_p = Σ S_p^BG / Σ(1 − Mask(B_i)) eq. 5.
- L_SAR = (1/q) Σ_i Σ_{p ∈ B_i} L_p eq. 6 §3.2.2.
Update rule. Each step modifies the noisy latent by x̂_t ← x_t − α ∇_{x_t} (L_CAR + L_SAR), then resumes the normal denoising step from x̂_t eq. 7 §3.2.3. A single update is often not enough, so τ updates are made per early step §3.2.3.
Algorithm 1 (one denoising step with refocusing) alg. 1.
- Run the diffusion model to get cross-attention A^t and self-attention S^t.
- Drop
and apply softmax to A^t. - For each token i: Gaussian-smooth A_i^t, compute L_FG, L_BG and L_CAR; for each pixel in its boxes compute L_p, and sum into L_SAR.
- Set L = L_CAR + L_SAR and x̂_t = x_t − α_t ∇ L.
- If t is one of the refinement steps and L > 1 − T, set x_t = x̂_t and go back to step 1 (threshold T; value not stated).
- Return the denoised latent x_{t−1} computed from x̂_t.
# Refocus one denoising step [alg. 1] [eq. 7] [§3.2.3]
def refocus(latent, boxes, token_index, alpha, tau):
for k in range(tau):
# Forward pass that also returns the attention maps [§3.2]
cross_maps, self_maps = attention_forward(latent)
# CAR and SAR losses on the layout tokens [§3.2.1] [§3.2.2]
car = car_loss(cross_maps, boxes, token_index)
sar = sar_loss(self_maps, boxes, token_index)
# Total loss: plain sum of the two terms [§3.2.3]
loss = car + sar
# Gradient step on the latent with step size alpha [eq. 7]
latent = latent - alpha * grad(loss, latent)
# Continue with the usual denoising update [alg. 1]
return latent# Cross-attention refocusing loss [§3.2.1] [eq. 2] [eq. 3]
def car_loss(cross_maps, boxes, token_index):
# Skip the <sot> map, softmax the rest, smooth [§3.2.1]
attn = gaussian(softmax(cross_maps[..., 1:])) # (16, 16, n - 1)
loss = 0
for i in token_index:
# Foreground: peak inside the boxes of token i [eq. 2]
loss += 1 - max(attn[..., i] * mask(boxes[i]))
# Background: peak outside those boxes [eq. 3]
loss += max(attn[..., i] * (1 - mask(boxes[i])))
return loss / len(token_index) # 1/q [eq. 2] [eq. 3]# Self-attention refocusing loss [§3.2.2] [eq. 4] [eq. 5] [eq. 6]
def sar_loss(self_maps, boxes, token_index):
loss = 0
for i in token_index:
# Every pixel p inside the boxes of token i [§3.2.2]
for p in pixels_inside(boxes[i]):
# Keep the attention from p to pixels outside [eq. 4]
outside = self_maps[p] * (1 - mask(boxes[i]))
# Average over the outside pixels for this p [eq. 5]
loss += sum(outside) / sum(1 - mask(boxes[i]))
return loss / len(token_index) # 1/q, summed over p [eq. 6]Notes on the pseudocode: token_index refers to positions after the
Layout generation. The prompt sent to GPT-4 is a fixed instruction (ChatGPT-4 role, output order left, top, right, bottom, 512 × 512 canvas), followed by two worked examples and the user prompt tab. 7 §3.3. The paper compares GPT-4 with GPT-3, Llama 1 and Llama 2 on 200 HRS prompts tab. 6.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Attention resolution for both losses | 16 × 16 | app. Implementation details | Code uses 16 × 16 cross-attention and 256-key self-attention loss.py loss.py |
| Denoising steps | 50 | app. Implementation details | Code PLMS sampler uses 50 steps inference.py |
| Sampler | not stated | app. Implementation details | Code default is PLMS; DDIM with 250 steps behind --no_plms inference.py |
| Step size α, first five steps | 4 | app. Implementation details | Code uses a loss scale of 4 for steps 0 to 4 plms.py |
| Step size α, later steps | 3 | app. Implementation details | Paper: "then decreases to 3". Code: 3 for steps 5 to 19 and 1 from step 20 plms.py |
| Repeats τ at t = 0, 1 | 2 | app. Implementation details | |
| Repeats τ at t = 2, 3, 4 | 6 | app. Implementation details | Code repeats 6 times at steps 2 to 4 plms.py |
| Repeats τ after the early steps | not stated | app. Implementation details | Code: 3 at steps 5 to 9, 2 from step 10 on plms.py |
| Late-step updates | only when tokens misaligned, after step 10 | app. Implementation details | Code: after step 10, only at steps 10, 15, …, 35 plms.py |
| Refinement threshold T | not stated | alg. 1 | Code, steps before 10: stop when the loss is at most 0.1 or the minimum in-box peak reaches 0.2 plms.py plms.py |
| Gaussian smoothing | kernel 3 × 3, σ = 0.5 | app. Implementation details | Code: same values, reflect-padded by 1 loss.py |
| Weight of CAR and SAR | equal sum | §3.2.3 | Code weights them differently; see "Paper vs code" |
| Baseline settings | defaults of each original paper | app. Implementation details | |
| Stable Diffusion version | v1.4 for the box experiments §4.1; v1.4 and v2.1 for TIFA tab. 2 | §4.1 | Version for the mask experiments: not stated |
| Layout LLM | GPT-4 | §3.3 | Code calls model "gpt-4" with no temperature set chatGPT.py. Snapshot and temperature: not stated |
| Canvas for layouts | 512 × 512 | tab. 7 | |
| Guidance scale | not stated | not in paper | Code default 7.5 inference.py |
| Negative prompt | not stated | not in paper | Code default string inference.py |
| GLIGEN grounding scale schedule | not stated | not in paper | Code: [0.3, 0.0, 0.7], so on for 15 of 50 steps, then off inference.py inference.py |
| HRS prompts (counting / spatial / size / color) | 3,000 / 1,002 / 501 / 501 | §4.1 | |
| DrawBench prompts | 39 | §4.1 | Labelled by the authors |
| TIFA prompts | 4,000 | §4.1 | |
| Mask benchmark | about 250 masks | §4.1 | From DenseDiffusion |
Results
Author-reported. Setup: HRS (counting, spatial, size, colour), DrawBench, TIFA and a COCO 2014 FID check; baselines at their default settings; hardware not stated §4.1 app. Implementation details. A "no" row is the base model without the losses. Accuracy is in %.
HRS, base models with and without CAR and SAR tab. 1. Counting is F1, the rest is accuracy.
| Method | CAR & SAR | Counting F1 | Spatial | Size | Colour |
|---|---|---|---|---|---|
| Stable Diffusion | no | 58.31 | 8.48 | 9.18 | 12.61 |
| Stable Diffusion | yes | 60.62 | 24.45 | 16.97 | 23.54 |
| Attend-and-excite | no | 60.47 | 9.98 | 10.58 | 19.56 |
| Attend-and-excite | yes | 62.71 | 20.76 | 14.17 | 20.83 |
| Layout-guidance | no | 56.22 | 16.47 | 12.38 | 14.39 |
| Layout-guidance | yes | 63.01 | 25.84 | 15.56 | 21.50 |
| MultiDiffusion | no | 55.18 | 14.27 | 10.58 | 17.15 |
| MultiDiffusion | yes | 57.37 | 22.65 | 10.78 | 24.59 |
| GLIGEN | no | 66.58 | 30.74 | 26.75 | 18.78 |
| GLIGEN | yes | 67.54 | 40.22 | 27.74 | 26.32 |
The text states an average counting gain of about 2 points and a spatial gain of about 10 points, with size and colour gains up to 10.9 points §4.2. In Table 1 the counting F1 gain ranges from +0.96 (GLIGEN) to +6.79 (Layout-guidance) tab. 1.
Ablation on GLIGEN, HRS tab. 5.
| CAR | SAR | Precision | Recall | Counting F1 | Spatial | Size | Colour |
|---|---|---|---|---|---|---|---|
| no | no | 78.81 | 59.44 | 66.58 | 30.74 | 26.75 | 18.78 |
| no | yes | 79.76 | 59.34 | 67.03 | 36.43 | 30.34 | 18.39 |
| yes | no | 82.11 | 59.35 | 67.59 | 36.92 | 28.94 | 23.88 |
| yes | yes | 81.25 | 59.39 | 67.54 | 40.22 | 27.74 | 26.32 |
DrawBench tab. 9. Counting precision, recall and F1, and spatial accuracy.
| Method | CAR & SAR | Precision | Recall | F1 | Spatial acc. |
|---|---|---|---|---|---|
| Stable Diffusion | no | 73.32 | 70.00 | 71.55 | 12.50 |
| Stable Diffusion | yes | 78.53 | 73.63 | 75.81 | 43.50 |
| Attend-and-excite | no | 77.64 | 74.85 | 76.20 | 20.50 |
| Attend-and-excite | yes | 74.06 | 77.58 | 75.66 | 38.00 |
| Layout-guidance | no | 79.15 | 70.61 | 74.48 | 36.50 |
| Layout-guidance | yes | 78.45 | 75.45 | 76.82 | 52.50 |
| MultiDiffusion | no | 75.37 | 65.61 | 69.90 | 38.00 |
| MultiDiffusion | yes | 84.30 | 68.03 | 75.20 | 54.50 |
| GLIGEN | no | 81.66 | 80.89 | 81.18 | 48.00 |
| GLIGEN | yes | 90.28 | 86.21 | 88.16 | 64.00 |
HRS counting, precision and recall, with Boxdiff tab. 10. The Boxdiff row is from the paper's table.
| Method | CAR & SAR | Precision | Recall | F1 |
|---|---|---|---|---|
| Stable Diffusion | no | 71.86 | 52.19 | 58.31 |
| Stable Diffusion | yes | 81.56 | 51.19 | 60.62 |
| Attend-and-excite | no | 73.10 | 54.79 | 60.47 |
| Attend-and-excite | yes | 75.94 | 56.31 | 62.71 |
| Layout-guidance | no | 80.60 | 45.83 | 56.22 |
| Layout-guidance | yes | 78.15 | 55.65 | 63.01 |
| MultiDiffusion | no | 78.96 | 45.18 | 55.18 |
| MultiDiffusion | yes | 83.26 | 45.71 | 57.37 |
| GLIGEN | no | 78.81 | 59.44 | 66.58 |
| GLIGEN | yes | 81.25 | 59.39 | 67.54 |
| Boxdiff | - | 83.78 | 57.81 | 67.02 |
Boxdiff comparison on GLIGEN, HRS tab. 11.
| Method | Counting F1 | Spatial | Size | Colour |
|---|---|---|---|---|
| GLIGEN + ours | 67.54 | 40.22 | 27.74 | 26.32 |
| GLIGEN + Boxdiff | 67.02 | 33.93 | 28.54 | 22.50 |
TIFA score tab. 2. Average over the benchmark, with counting and spatial columns.
| Base model | CAR & SAR | Counting | Spatial | Average |
|---|---|---|---|---|
| Stable Diffusion 1.4 | no | 68.15 | 72.01 | 78.38 |
| Stable Diffusion 1.4 | yes | 69.37 | 73.33 | 78.87 |
| Stable Diffusion 2.1 | no | 73.63 | 76.11 | 81.84 |
| Stable Diffusion 2.1 | yes | 74.44 | 76.29 | 81.89 |
Mask-guided generation tab. 3. Compared with DenseDiffusion and SD-Pww results taken from the DenseDiffusion paper.
| Method | IoU ↑ | CLIP score ↑ | SOA-I ↑ |
|---|---|---|---|
| SD-Pww | 23.76 ± 0.50 | 0.2800 ± 0.0005 | 73.92 ± 1.84 |
| DenseDiffusion | 34.99 ± 1.13 | 0.2814 ± 0.0005 | 77.61 ± 1.75 |
| Stable Diffusion + ours | 38.97 ± 0.56 | 0.3177 ± 0.0011 | 78.80 ± 1.27 |
FID on COCO 2014 tab. 4. The text calls the FID values compatible with the originals.
| SD no | SD yes | GLIGEN no | GLIGEN yes | |
|---|---|---|---|---|
| FID ↓ | 20.82 | 21.03 | 20.63 | 20.37 |
LLM layout quality, 200 HRS prompts tab. 6. Column definitions are in §4.5.
| Model | Format ↑ | Valid ↑ | Correct ↑ |
|---|---|---|---|
| Llama 1 | 67.5 | 46.0 | 38.5 |
| Llama 2 (13b-chat) | 98.5 | 84.0 | 63.5 |
| GPT-3 | 98.5 | 97.5 | 83.5 |
| GPT-4 | 98.5 | 98.5 | 88.5 |
Inference time, seconds per 10 images tab. 8. Hardware not stated.
| Stable Diffusion | +AE | +MD | +LG | +Ours | GLIGEN | GLIGEN +Ours |
|---|---|---|---|---|---|---|
| 54.33 | 101.67 | 74.16 | 111.13 | 102.97 | 205.90 | 279.08 |
The appendix says the losses are compatible with Attend-and-excite and faster than Layout-guidance tab. 8.
Paper vs code
Official code: Attention-Refocusing/attention-refocusing, linked from the project page and the README. It is built on GLIGEN README.md. Its LICENSE file is MIT and names "Yuheng Li" as copyright holder LICENSE. The repository is the one at code_commit above.
- CAR loss form. The paper uses one foreground term and one background term per token, with equal weight eq. 2 eq. 3. The code applies a per-box foreground penalty with thresholds: 6 × (1 − peak) if the peak is below 0.1, (1 − peak) if below 0.2, and otherwise (1 − peak) only while step index < 15 loss.py. The outside term is 4× the peak above 0.15, 1× above 0.1, and 1× below 0.1 only while step index < 15 loss.py. The background term is the maximum outside all boxes, scaled by the box count and divided by 2 loss.py.
- Softmax. The paper removes
and applies softmax §3.2.1. The code multiplies the logits by 100, then applies softmax over tokens 1 to n−2, which also drops the last token loss.py. - Gaussian smoothing. Kernel 3 × 3 and σ = 0.5 match the appendix app. Implementation details loss.py.
- SAR normalisation. The paper divides each pixel's outside sum by the number of outside pixels, then sums over the box's pixels eq. 5 eq. 6. The code averages the box's attention rows and sums over outside keys, with no division by the outside size loss.py. For one box, the paper's value equals the code's times |B| / |bg|, where |B| is the box size and |bg| the outside size (derived from the two formulas).
- A third term. The code adds a loss on the attention of the gated grounding self-attention (token index 286, that is 256 image keys plus 30 grounding keys), active only in early steps, weighted 3 × scale loss.py plms.py. The paper describes only two losses §3.2.
- Weights and step size. The paper uses L = L_CAR + L_SAR and a step α in eq. 7 §3.2.3. The code multiplies each loss by the step's scale and adds a further weight on the third term. The total in early steps is loss2 + loss1 + 3 × scale × loss3, with loss1 and loss2 each scaled plms.py. The update is x ← x − ∇loss, with no separate α plms.py.
- Repeat and step schedule. Paper: τ = 2 at t = 0, 1 and 6 at t = 2 to 4 app. Implementation details. Code: 3 repeats at steps 5 to 9, 2 from step 10 on, and a loss scale of 1 from step 20 onward plms.py. The paper says nothing about steps 5 to 9 or after 20.
- Late steps. The paper updates after step 10 only for misaligned tokens app. Implementation details. The code updates only at steps 10, 15, …, 35, and only while the minimum in-box peak is below 0.2 plms.py.
- Threshold. The paper's T is not given alg. 1. For steps before 10, the code stops once the total loss is at most 0.1 plms.py.
- GLIGEN grounding schedule. Not described in the paper. The code switches the grounding scale to 1 for the first 15 of 50 steps and to 0 afterwards, and restores the first convolution from Stable Diffusion at scale 0 inference.py inference.py plms.py.
- CAR-only path. The ablation's CAR-only setting runs a separate code path, with loss scales 3, 2 and 1, up to 5 repeats (1 after step 20), and no updates from step 30 on plms.py. The paper does not describe this path.
- Few-shot examples. The code's example file gives different example boxes from the paper's table, and includes a correction turn about relative object sizes, which the paper's table omits (the turn is commented out in the source) example3.json tab. 7.
- Guidance and negative prompt. The paper does not state a guidance scale or negative prompt. The code defaults are 7.5 and a fixed negative prompt inference.py.
- Steps and attention size. 50 denoising steps and 16 × 16 attention match app. Implementation details inference.py loss.py.
No discrepancies were checked for the HRS, DrawBench or TIFA evaluation scripts; they are under data_evaluate_LLM/ in the repository and were not read.
Pitfalls
- Implement the code, not only the equations, if you want the reported numbers. The code's CAR thresholds, SAR normalisation, extra grounding term and step schedule all differ from the written losses (see above).
- Empty slices count as satisfied. If a box has no 16 × 16 cells after integer truncation (for example, a box narrower than one cell), the code sets the in-box peak to 1, so the foreground penalty is zero loss.py. Small boxes can silently drop out of the loss.
- Token matching is by string. Box labels are matched to prompt words by whitespace splitting, with simple plural rules and the first occurrence only. Only periods and commas are stripped. A label that matches nothing is dropped without warning chatGPT.py. The paper does not state its matching rule §3.2.
- The grounding branch is switched off mid-sampling. With the code's schedule, the GLIGEN grounding scale is 0 after step 15 of 50, and the losses still run on the attention maps at steps 10 to 35 inference.py plms.py. Results for GLIGEN depend on this.
- Hard-coded CUDA. Gaussian smoothing is moved to CUDA unconditionally loss.py. The path needs a GPU as written.
- Cost depends on the data. Each refinement is a forward pass that keeps attention maps plus a backward pass through the UNet. The number of refinements varies per image, because of the early stop plms.py tab. 8. The paper gives one timing table, without hardware tab. 8. Stable Diffusion takes 54.33 s per 10 images without the method and 102.97 s with it; GLIGEN takes 205.90 s and 279.08 s tab. 8.
- Layout errors pass through. GPT-4 can miscount objects or draw small boxes, and GLIGEN can fail on layouts it has not seen § Limitations. The losses cannot fix a wrong layout.
- Pinned model. The code calls the
gpt-4alias without a temperature setting chatGPT.py. Boxes generated again may differ from the authors'. The repository ships the authors' GPT-4 boxes for HRS and DrawBench underdata_evaluate_LLM/, which is the safer way to reproduce the numbers README.md. - Reproduce the ablation with the right path. CAR-only and CAR-plus-SAR run different functions in the code plms.py plms.py. The SAR-only path (
update_only_self) was not read in detail. - Author-reported only. All numbers above come from the paper and its tables. Only the mask table (Table 3) gives ± values, and the sections read do not say what they measure tab. 3.
Lineage
- Builds on: GLIGEN, the grounded baseline used for the grounded-generation results §1 tab. 1.
- Builds on: Stable Diffusion, the base model used for the training-free plug-in results §4.2 tab. 1.
- Compared against: MultiDiffusion, Layout-guidance, Attend-and-excite, Boxdiff, DenseDiffusion and SD-Pww §4.2 tab. 1 tab. 11 tab. 3.
- Related concurrent work: LLM-grounded diffusion, which also uses GPT-4 for layouts §2.
- Superseded by: not stated in the source.
Source and license
Adapted from “Grounded Text-to-Image Synthesis with Attention Refocusing” by Quynh Phung, Songwei Ge, Jia-Bin Huang (arXiv:2306.05427), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Attention-Refocusing/attention-refocusing at commit 2c265cbcac2e (MIT). Any quoted code is shown under that license.