wikillm

Paper · 2023-06-08 · cs.CV

Grounded Text-to-Image Synthesis with Attention Refocusing

Training-free fix for layout-grounded text-to-image diffusion. Two losses on cross- and self-attention maps (CAR, SAR) are minimised by gradient steps on the latent during early denoising steps; GPT-4 supplies the boxes. Author-reported: on HRS, GLIGEN's spatial accuracy rises from 30.74 to 40.22.

Layout-to-image generationText-to-image generationCross-attentionDiffusion modelstext-to-imagediffusionlayout-controltraining-freeattention

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Text-to-image models often get the details of a prompt wrong. Ask for three parrots and you may get four, or objects may go missing or take on each other's attributes. The paper argues that the model's internal attention maps are the cause. Each word and each image region has a spot in those maps, and when two things look alike, their spots blur together.

The method has two parts. First, a large language model turns the prompt into a rough sketch: a labelled box for each object. Second, while the picture is being drawn, a few small corrections nudge the image so that each word's attention lands inside its box and stays out of the others. Nothing in the image model is retrained; the corrections happen at generation time.

The point is control. Given a layout, the picture follows it more closely, and the paper reports that this helps with counting, positions, sizes and colours across several existing models.

TL;DR

  • No model is retrained. A prompt-to-boxes step (GPT-4 with examples) feeds a grounded or plain diffusion model, which is steered during sampling §3.3 §3.2.
  • Two losses act on attention maps: CAR pulls each object word's cross-attention into its box, and SAR makes pixels inside a box attend less to pixels outside it §3.2.1 §3.2.2.
  • The latent is updated by gradient descent in the first denoising steps, with a few repeats per step §3.2.3 alg. 1.
  • Author-reported: adding the losses raises the HRS spatial accuracy of five base models and improves counting F1 for all five (GLIGEN spatial 30.74 to 40.22) tab. 1.
  • The official code differs from the written losses in several ways (see "Paper vs code"). Implement from the code if you need the reported numbers.

Problem

Diffusion models tie prompt words to image regions through cross-attention, and they relate image pixels to each other through self-attention. When two regions have similar features, their attention can leak into each other. The paper identifies this as the cause of missing objects and blended attributes §1 §3.2.1.

Earlier training-free fixes changed cross-attention only §1 §2. Layout-based methods also need the boxes, which are costly to write by hand §3.2. The paper assumes that boxes with a caption per box are available, and that each caption word appears in the prompt §3.2.

Method

Overview. The pipeline has two phases, both without training §3. Phase one (text-to-layout) asks GPT-4 for labelled boxes on a 512 × 512 canvas, using a fixed instruction and in-context examples §3.3 tab. 7 fig. 3. Phase two (grounded generation) runs a pretrained diffusion model and, at each sampling step, adds the attention-refocusing update §3.2 fig. 3.

Attention. For queries Q and keys K of dimension d, a map A^t = softmax(QK^T / √d) gives, for each image location, a distribution over tokens (cross-attention) or over locations (self-attention) eq. 1 §3.1. The paper works on the 16 × 16 maps app. Implementation details.

Layout tokens. A grounded token i may own one or more boxes B_i, and Mask(B_i) is 1 inside those boxes and 0 outside §3.2.

CAR (cross-attention refocusing).

  1. Drop the map, apply softmax over the remaining tokens, then apply Gaussian smoothing §3.2.1 app. Implementation details.
  2. Foreground term: push each token's peak inside its boxes, L_FG = (1/q) Σ_i (1 − max(A_i · Mask(B_i))) eq. 2.
  3. Background term: push the peak outside the boxes down, L_BG = (1/q) Σ_i max(A_i · (1 − Mask(B_i))) eq. 3.
  4. L_CAR = L_FG + L_BG §3.2.1.

SAR (self-attention refocusing).

  1. For a pixel p inside a box B_i, take its self-attention map S_p and keep only the outside part, S_p^BG = S_p · (1 − Mask(B_i)) eq. 4 §3.2.2.
  2. Per pixel, average the outside part over the outside pixels: L_p = Σ S_p^BG / Σ(1 − Mask(B_i)) eq. 5.
  3. L_SAR = (1/q) Σ_i Σ_{p ∈ B_i} L_p eq. 6 §3.2.2.

Update rule. Each step modifies the noisy latent by x̂_t ← x_t − α ∇_{x_t} (L_CAR + L_SAR), then resumes the normal denoising step from x̂_t eq. 7 §3.2.3. A single update is often not enough, so τ updates are made per early step §3.2.3.

Algorithm 1 (one denoising step with refocusing) alg. 1.

  1. Run the diffusion model to get cross-attention A^t and self-attention S^t.
  2. Drop and apply softmax to A^t.
  3. For each token i: Gaussian-smooth A_i^t, compute L_FG, L_BG and L_CAR; for each pixel in its boxes compute L_p, and sum into L_SAR.
  4. Set L = L_CAR + L_SAR and x̂_t = x_t − α_t ∇ L.
  5. If t is one of the refinement steps and L > 1 − T, set x_t = x̂_t and go back to step 1 (threshold T; value not stated).
  6. Return the denoised latent x_{t−1} computed from x̂_t.
Pseudocode
# Refocus one denoising step  [alg. 1] [eq. 7] [§3.2.3]
def refocus(latent, boxes, token_index, alpha, tau):
    for k in range(tau):
        # Forward pass that also returns the attention maps  [§3.2]
        cross_maps, self_maps = attention_forward(latent)
        # CAR and SAR losses on the layout tokens  [§3.2.1] [§3.2.2]
        car = car_loss(cross_maps, boxes, token_index)
        sar = sar_loss(self_maps, boxes, token_index)
        # Total loss: plain sum of the two terms  [§3.2.3]
        loss = car + sar
        # Gradient step on the latent with step size alpha  [eq. 7]
        latent = latent - alpha * grad(loss, latent)
    # Continue with the usual denoising update  [alg. 1]
    return latent
Pseudocode
# Cross-attention refocusing loss  [§3.2.1] [eq. 2] [eq. 3]
def car_loss(cross_maps, boxes, token_index):
    # Skip the <sot> map, softmax the rest, smooth  [§3.2.1]
    attn = gaussian(softmax(cross_maps[..., 1:]))  # (16, 16, n - 1)
    loss = 0
    for i in token_index:
        # Foreground: peak inside the boxes of token i  [eq. 2]
        loss += 1 - max(attn[..., i] * mask(boxes[i]))
        # Background: peak outside those boxes  [eq. 3]
        loss += max(attn[..., i] * (1 - mask(boxes[i])))
    return loss / len(token_index)  # 1/q  [eq. 2] [eq. 3]
Pseudocode
# Self-attention refocusing loss  [§3.2.2] [eq. 4] [eq. 5] [eq. 6]
def sar_loss(self_maps, boxes, token_index):
    loss = 0
    for i in token_index:
        # Every pixel p inside the boxes of token i  [§3.2.2]
        for p in pixels_inside(boxes[i]):
            # Keep the attention from p to pixels outside  [eq. 4]
            outside = self_maps[p] * (1 - mask(boxes[i]))
            # Average over the outside pixels for this p  [eq. 5]
            loss += sum(outside) / sum(1 - mask(boxes[i]))
    return loss / len(token_index)  # 1/q, summed over p  [eq. 6]

Notes on the pseudocode: token_index refers to positions after the map is removed. The paper does not give an explicit rule for matching box captions to prompt tokens §3.2.

Layout generation. The prompt sent to GPT-4 is a fixed instruction (ChatGPT-4 role, output order left, top, right, bottom, 512 × 512 canvas), followed by two worked examples and the user prompt tab. 7 §3.3. The paper compares GPT-4 with GPT-3, Llama 1 and Llama 2 on 200 HRS prompts tab. 6.

Hyperparameters

NameValueWhereNotes
Attention resolution for both losses16 × 16app. Implementation detailsCode uses 16 × 16 cross-attention and 256-key self-attention loss.py loss.py
Denoising steps50app. Implementation detailsCode PLMS sampler uses 50 steps inference.py
Samplernot statedapp. Implementation detailsCode default is PLMS; DDIM with 250 steps behind --no_plms inference.py
Step size α, first five steps4app. Implementation detailsCode uses a loss scale of 4 for steps 0 to 4 plms.py
Step size α, later steps3app. Implementation detailsPaper: "then decreases to 3". Code: 3 for steps 5 to 19 and 1 from step 20 plms.py
Repeats τ at t = 0, 12app. Implementation details
Repeats τ at t = 2, 3, 46app. Implementation detailsCode repeats 6 times at steps 2 to 4 plms.py
Repeats τ after the early stepsnot statedapp. Implementation detailsCode: 3 at steps 5 to 9, 2 from step 10 on plms.py
Late-step updatesonly when tokens misaligned, after step 10app. Implementation detailsCode: after step 10, only at steps 10, 15, …, 35 plms.py
Refinement threshold Tnot statedalg. 1Code, steps before 10: stop when the loss is at most 0.1 or the minimum in-box peak reaches 0.2 plms.py plms.py
Gaussian smoothingkernel 3 × 3, σ = 0.5app. Implementation detailsCode: same values, reflect-padded by 1 loss.py
Weight of CAR and SARequal sum§3.2.3Code weights them differently; see "Paper vs code"
Baseline settingsdefaults of each original paperapp. Implementation details
Stable Diffusion versionv1.4 for the box experiments §4.1; v1.4 and v2.1 for TIFA tab. 2§4.1Version for the mask experiments: not stated
Layout LLMGPT-4§3.3Code calls model "gpt-4" with no temperature set chatGPT.py. Snapshot and temperature: not stated
Canvas for layouts512 × 512tab. 7
Guidance scalenot statednot in paperCode default 7.5 inference.py
Negative promptnot statednot in paperCode default string inference.py
GLIGEN grounding scale schedulenot statednot in paperCode: [0.3, 0.0, 0.7], so on for 15 of 50 steps, then off inference.py inference.py
HRS prompts (counting / spatial / size / color)3,000 / 1,002 / 501 / 501§4.1
DrawBench prompts39§4.1Labelled by the authors
TIFA prompts4,000§4.1
Mask benchmarkabout 250 masks§4.1From DenseDiffusion

Results

Author-reported. Setup: HRS (counting, spatial, size, colour), DrawBench, TIFA and a COCO 2014 FID check; baselines at their default settings; hardware not stated §4.1 app. Implementation details. A "no" row is the base model without the losses. Accuracy is in %.

HRS, base models with and without CAR and SAR tab. 1. Counting is F1, the rest is accuracy.

MethodCAR & SARCounting F1SpatialSizeColour
Stable Diffusionno58.318.489.1812.61
Stable Diffusionyes60.6224.4516.9723.54
Attend-and-exciteno60.479.9810.5819.56
Attend-and-exciteyes62.7120.7614.1720.83
Layout-guidanceno56.2216.4712.3814.39
Layout-guidanceyes63.0125.8415.5621.50
MultiDiffusionno55.1814.2710.5817.15
MultiDiffusionyes57.3722.6510.7824.59
GLIGENno66.5830.7426.7518.78
GLIGENyes67.5440.2227.7426.32

The text states an average counting gain of about 2 points and a spatial gain of about 10 points, with size and colour gains up to 10.9 points §4.2. In Table 1 the counting F1 gain ranges from +0.96 (GLIGEN) to +6.79 (Layout-guidance) tab. 1.

Ablation on GLIGEN, HRS tab. 5.

CARSARPrecisionRecallCounting F1SpatialSizeColour
nono78.8159.4466.5830.7426.7518.78
noyes79.7659.3467.0336.4330.3418.39
yesno82.1159.3567.5936.9228.9423.88
yesyes81.2559.3967.5440.2227.7426.32

DrawBench tab. 9. Counting precision, recall and F1, and spatial accuracy.

MethodCAR & SARPrecisionRecallF1Spatial acc.
Stable Diffusionno73.3270.0071.5512.50
Stable Diffusionyes78.5373.6375.8143.50
Attend-and-exciteno77.6474.8576.2020.50
Attend-and-exciteyes74.0677.5875.6638.00
Layout-guidanceno79.1570.6174.4836.50
Layout-guidanceyes78.4575.4576.8252.50
MultiDiffusionno75.3765.6169.9038.00
MultiDiffusionyes84.3068.0375.2054.50
GLIGENno81.6680.8981.1848.00
GLIGENyes90.2886.2188.1664.00

HRS counting, precision and recall, with Boxdiff tab. 10. The Boxdiff row is from the paper's table.

MethodCAR & SARPrecisionRecallF1
Stable Diffusionno71.8652.1958.31
Stable Diffusionyes81.5651.1960.62
Attend-and-exciteno73.1054.7960.47
Attend-and-exciteyes75.9456.3162.71
Layout-guidanceno80.6045.8356.22
Layout-guidanceyes78.1555.6563.01
MultiDiffusionno78.9645.1855.18
MultiDiffusionyes83.2645.7157.37
GLIGENno78.8159.4466.58
GLIGENyes81.2559.3967.54
Boxdiff-83.7857.8167.02

Boxdiff comparison on GLIGEN, HRS tab. 11.

MethodCounting F1SpatialSizeColour
GLIGEN + ours67.5440.2227.7426.32
GLIGEN + Boxdiff67.0233.9328.5422.50

TIFA score tab. 2. Average over the benchmark, with counting and spatial columns.

Base modelCAR & SARCountingSpatialAverage
Stable Diffusion 1.4no68.1572.0178.38
Stable Diffusion 1.4yes69.3773.3378.87
Stable Diffusion 2.1no73.6376.1181.84
Stable Diffusion 2.1yes74.4476.2981.89

Mask-guided generation tab. 3. Compared with DenseDiffusion and SD-Pww results taken from the DenseDiffusion paper.

MethodIoU ↑CLIP score ↑SOA-I ↑
SD-Pww23.76 ± 0.500.2800 ± 0.000573.92 ± 1.84
DenseDiffusion34.99 ± 1.130.2814 ± 0.000577.61 ± 1.75
Stable Diffusion + ours38.97 ± 0.560.3177 ± 0.001178.80 ± 1.27

FID on COCO 2014 tab. 4. The text calls the FID values compatible with the originals.

SD noSD yesGLIGEN noGLIGEN yes
FID ↓20.8221.0320.6320.37

LLM layout quality, 200 HRS prompts tab. 6. Column definitions are in §4.5.

ModelFormat ↑Valid ↑Correct ↑
Llama 167.546.038.5
Llama 2 (13b-chat)98.584.063.5
GPT-398.597.583.5
GPT-498.598.588.5

Inference time, seconds per 10 images tab. 8. Hardware not stated.

Stable Diffusion+AE+MD+LG+OursGLIGENGLIGEN +Ours
54.33101.6774.16111.13102.97205.90279.08

The appendix says the losses are compatible with Attend-and-excite and faster than Layout-guidance tab. 8.

Paper vs code

Official code: Attention-Refocusing/attention-refocusing, linked from the project page and the README. It is built on GLIGEN README.md. Its LICENSE file is MIT and names "Yuheng Li" as copyright holder LICENSE. The repository is the one at code_commit above.

  • CAR loss form. The paper uses one foreground term and one background term per token, with equal weight eq. 2 eq. 3. The code applies a per-box foreground penalty with thresholds: 6 × (1 − peak) if the peak is below 0.1, (1 − peak) if below 0.2, and otherwise (1 − peak) only while step index < 15 loss.py. The outside term is 4× the peak above 0.15, 1× above 0.1, and 1× below 0.1 only while step index < 15 loss.py. The background term is the maximum outside all boxes, scaled by the box count and divided by 2 loss.py.
  • Softmax. The paper removes and applies softmax §3.2.1. The code multiplies the logits by 100, then applies softmax over tokens 1 to n−2, which also drops the last token loss.py.
  • Gaussian smoothing. Kernel 3 × 3 and σ = 0.5 match the appendix app. Implementation details loss.py.
  • SAR normalisation. The paper divides each pixel's outside sum by the number of outside pixels, then sums over the box's pixels eq. 5 eq. 6. The code averages the box's attention rows and sums over outside keys, with no division by the outside size loss.py. For one box, the paper's value equals the code's times |B| / |bg|, where |B| is the box size and |bg| the outside size (derived from the two formulas).
  • A third term. The code adds a loss on the attention of the gated grounding self-attention (token index 286, that is 256 image keys plus 30 grounding keys), active only in early steps, weighted 3 × scale loss.py plms.py. The paper describes only two losses §3.2.
  • Weights and step size. The paper uses L = L_CAR + L_SAR and a step α in eq. 7 §3.2.3. The code multiplies each loss by the step's scale and adds a further weight on the third term. The total in early steps is loss2 + loss1 + 3 × scale × loss3, with loss1 and loss2 each scaled plms.py. The update is x ← x − ∇loss, with no separate α plms.py.
  • Repeat and step schedule. Paper: τ = 2 at t = 0, 1 and 6 at t = 2 to 4 app. Implementation details. Code: 3 repeats at steps 5 to 9, 2 from step 10 on, and a loss scale of 1 from step 20 onward plms.py. The paper says nothing about steps 5 to 9 or after 20.
  • Late steps. The paper updates after step 10 only for misaligned tokens app. Implementation details. The code updates only at steps 10, 15, …, 35, and only while the minimum in-box peak is below 0.2 plms.py.
  • Threshold. The paper's T is not given alg. 1. For steps before 10, the code stops once the total loss is at most 0.1 plms.py.
  • GLIGEN grounding schedule. Not described in the paper. The code switches the grounding scale to 1 for the first 15 of 50 steps and to 0 afterwards, and restores the first convolution from Stable Diffusion at scale 0 inference.py inference.py plms.py.
  • CAR-only path. The ablation's CAR-only setting runs a separate code path, with loss scales 3, 2 and 1, up to 5 repeats (1 after step 20), and no updates from step 30 on plms.py. The paper does not describe this path.
  • Few-shot examples. The code's example file gives different example boxes from the paper's table, and includes a correction turn about relative object sizes, which the paper's table omits (the turn is commented out in the source) example3.json tab. 7.
  • Guidance and negative prompt. The paper does not state a guidance scale or negative prompt. The code defaults are 7.5 and a fixed negative prompt inference.py.
  • Steps and attention size. 50 denoising steps and 16 × 16 attention match app. Implementation details inference.py loss.py.

No discrepancies were checked for the HRS, DrawBench or TIFA evaluation scripts; they are under data_evaluate_LLM/ in the repository and were not read.

Pitfalls

  • Implement the code, not only the equations, if you want the reported numbers. The code's CAR thresholds, SAR normalisation, extra grounding term and step schedule all differ from the written losses (see above).
  • Empty slices count as satisfied. If a box has no 16 × 16 cells after integer truncation (for example, a box narrower than one cell), the code sets the in-box peak to 1, so the foreground penalty is zero loss.py. Small boxes can silently drop out of the loss.
  • Token matching is by string. Box labels are matched to prompt words by whitespace splitting, with simple plural rules and the first occurrence only. Only periods and commas are stripped. A label that matches nothing is dropped without warning chatGPT.py. The paper does not state its matching rule §3.2.
  • The grounding branch is switched off mid-sampling. With the code's schedule, the GLIGEN grounding scale is 0 after step 15 of 50, and the losses still run on the attention maps at steps 10 to 35 inference.py plms.py. Results for GLIGEN depend on this.
  • Hard-coded CUDA. Gaussian smoothing is moved to CUDA unconditionally loss.py. The path needs a GPU as written.
  • Cost depends on the data. Each refinement is a forward pass that keeps attention maps plus a backward pass through the UNet. The number of refinements varies per image, because of the early stop plms.py tab. 8. The paper gives one timing table, without hardware tab. 8. Stable Diffusion takes 54.33 s per 10 images without the method and 102.97 s with it; GLIGEN takes 205.90 s and 279.08 s tab. 8.
  • Layout errors pass through. GPT-4 can miscount objects or draw small boxes, and GLIGEN can fail on layouts it has not seen § Limitations. The losses cannot fix a wrong layout.
  • Pinned model. The code calls the gpt-4 alias without a temperature setting chatGPT.py. Boxes generated again may differ from the authors'. The repository ships the authors' GPT-4 boxes for HRS and DrawBench under data_evaluate_LLM/, which is the safer way to reproduce the numbers README.md.
  • Reproduce the ablation with the right path. CAR-only and CAR-plus-SAR run different functions in the code plms.py plms.py. The SAR-only path (update_only_self) was not read in detail.
  • Author-reported only. All numbers above come from the paper and its tables. Only the mask table (Table 3) gives ± values, and the sections read do not say what they measure tab. 3.

Lineage

  • Builds on: GLIGEN, the grounded baseline used for the grounded-generation results §1 tab. 1.
  • Builds on: Stable Diffusion, the base model used for the training-free plug-in results §4.2 tab. 1.
  • Compared against: MultiDiffusion, Layout-guidance, Attend-and-excite, Boxdiff, DenseDiffusion and SD-Pww §4.2 tab. 1 tab. 11 tab. 3.
  • Related concurrent work: LLM-grounded diffusion, which also uses GPT-4 for layouts §2.
  • Superseded by: not stated in the source.

Source and license

Adapted from “Grounded Text-to-Image Synthesis with Attention Refocusing” by Quynh Phung, Songwei Ge, Jia-Bin Huang (arXiv:2306.05427), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Attention-Refocusing/attention-refocusing at commit 2c265cbcac2e (MIT). Any quoted code is shown under that license.