Dense Text-to-Image Generation with Attention Modulation

In plain words

Text-to-image models turn a prompt into a picture in many small steps, starting from noise. When the prompt describes several objects in detail, such as "a black and white dog next to a red bicycle by a blue door", the model often drops one of the objects or blends two of them together. Adding a layout, where each object gets its own region, is possible but usually means training a new model.

DenseDiffusion keeps the model as it is. The idea is that, early in the generation, the model's internal attention decides roughly where each object will appear. The method nudges that attention: words from one region are encouraged to look at pixels inside that region, and pixels in different regions are discouraged from looking at each other. The nudge is strongest at the start and fades out, so the image keeps the model's usual look. The method needs no extra training data.

On a benchmark built from MS-COCO captions, the authors report better text fidelity than plain Stable Diffusion and several training-free alternatives, and layout fidelity well above the one baseline that accepts a layout. The cost is that each region has to be given as a mask and each region's words have to appear in the prompt.

TL;DR

DenseDiffusion is a training-free method for Stable Diffusion. Each prompt segment is paired with a binary region map §3. For the first 30% of the denoising steps (t from 1 to 0.7), it adds a bonus to attention scores for same-region pairs and a penalty to other pairs, in cross-attention and self-attention layers §3.2 eq. 2. The size of the adjustment is scaled by the score range and by each region's area eq. 2 eq. 6. Author-reported, on 250 MS-COCO captions, against Stable Diffusion: SOA-I 77.61 vs 73.08 and CLIP-score 0.2814 vs 0.2732 tab. 2; layout IoU 34.99 vs 23.76 against SD-Pww tab. 3. Relevant if you need layout or dense-prompt control on a frozen diffusion model and can supply region masks.

Problem

Text-to-image diffusion models handle dense captions, where each phrase describes one image region, poorly: they often omit or blend the features of different objects §1 fig. 1. Layout control is also hard to get from text alone §1.

The paper groups prior fixes into two families. Layout-conditioned models are trained or fine-tuned for each new condition, domain or base model, which is costly §1. Compositional or inference-time methods avoid training but either cost time that grows with the number of phrases, or do not take a layout at all §4.1 app. A.1.

DenseDiffusion targets both problems with no training and no extra dataset §1 §3.

Method

Problem setup §3. The condition is a set of N segments {(c_n, m_n)}. Each c_n is a non-overlapping part of the full caption, and m_n is a binary map of its region. The modulation is applied to the attention maps of every attention layer at inference time §3.

Diffusion and attention background §3.1. Sampling is 50 DDIM steps on Stable Diffusion §3.1. An attention map is A = softmax(QK^T / sqrt(d)) eq. 1. Self-attention uses image features as keys and values; cross-attention uses the CLIP text features of the caption §3.1.

Attention analysis §3.1 fig. 2 tab. 1. The authors generated 250 MS-COCO validation captions and detected objects with YOLOv7. Attention maps come to resemble the layout as sampling proceeds fig. 2. Scores between a query and a key that belong to the same object are higher than for unrelated pairs, in both layer types tab. 1. This motivates the method §3.1.

A matched key is a text token that equals the class label of a detected box (cross-attention), or an image token inside the box (self-attention) tab. 1.

Modulated attention §3.2 eq. 2. Scores are modified before the softmax:

• A' = softmax((QK^T + M) / sqrt(d)) eq. 2.
• M = lambda_t · R ⊙ M_pos ⊙ (1 − S) − lambda_t · (1 − R) ⊙ M_neg ⊙ (1 − S) eq. 2.
• R is a query-key condition map: 1 marks a positive pair (same segment) and 0 a negative pair eq. 2.
• S is, for each query token, the fraction of the image covered by its segment §3.2.

Strength schedule §3.2 eq. 3. lambda_t = w · t^p, with t normalised to [0, 1] eq. 3. The authors use one exponent p for both layer types and separate weights w^c (cross) and w^s (self) §3.2. Modulation is applied only to the first denoising steps, t from 1 to 0.7, because the authors saw no clear gain beyond that §3.2.

Cross-attention pair map §3.2 eq. 4. Each binary region map is flattened and resized to the layer's resolution. For text token j, let k[j] be its segment index, or 0 if it belongs to no segment. Column j of R^cross is zero when k[j] = 0, and otherwise equals the flattened map of segment k[j] eq. 4. Pixels in that region therefore form a positive pair with that segment's words, and all other pixel-word pairs are negative eq. 4.

Self-attention pair map §3.2 eq. 5. R^self_ij = 1 when some segment n contains both image tokens i and j, and 0 otherwise eq. 5. Positive pairs are pixels in the same segment, negative pairs are all other pixel pairs eq. 5.

Value-range adaptive term §3.2 eq. 6. For each query, M_pos = max(QK^T) − QK^T and M_neg = QK^T − min(QK^T), with the per-query max and min repeated along the key axis eq. 6. The adjustment therefore stays inside the original range of scores for that query §3.2 eq. 6.

Mask-area adaptive term §3.2 eq. 2. The factor (1 − S) increases the modulation for small segments and reduces it for large ones §3.2. The authors report quality loss when one segment is much smaller than the others, which this term addresses §3.2.

Phrase-embedding replacement §3.2. Each segment prompt c_n is also encoded on its own. Its encoding replaces the matching span of the full-caption text features. The authors say this helps when two objects are closely related, such as a microwave and an oven §3.2.

Pseudocode for one attention layer at one step (written from eq. 2 to eq. 6; the paper gives no algorithm box):

# Modulated attention for one layer at one denoising step  [eq. 2] [eq. 6]
# queries (B, Nq, d), keys (B, Nk, d), values (B, Nk, d) -> output (B, Nq, d)
# pair_map (Nq, Nk): 1 = positive pair, 0 = negative pair  [eq. 4] [eq. 5]
# area (Nq, 1): fraction of the image covered by the query's segment  [§3.2]
def modulated_attention(queries, keys, values, pair_map, area, t):
    # Scaled scores, same as QK^T / sqrt(d)  [eq. 1]
    scores = queries @ keys.transpose(-1, -2) / sqrt(d)
    # Per-query maximum and minimum over the keys  [eq. 6]
    top = scores.max(dim=-1, keepdim=True).values
    bottom = scores.min(dim=-1, keepdim=True).values
    # Strength lambda_t = w * t^p, with t in [0, 1]  [eq. 3]
    strength = w * t ** p
    # Raise positive pairs toward the maximum  [eq. 2] [eq. 6]
    raise_up = strength * pair_map * (top - scores) * (1 - area)
    # Lower negative pairs toward the minimum  [eq. 2] [eq. 6]
    lower_down = strength * (1 - pair_map) * (scores - bottom) * (1 - area)
    scores = scores + raise_up - lower_down
    weights = softmax(scores, dim=-1)
    return weights @ values

The code adds the adjustment after dividing by sqrt(d). Because both terms in the paper's formula are divided by sqrt(d) in the same way, this is the same update gradio_app.py gradio_app.py.

Hyperparameters

Results

Author-reported. Setup: 250 MS-COCO validation captions, each with at least two objects. Each class label is replaced by a phrase from the caption, for example "dog" becomes "a black and white dog". Four images per caption, so 1,000 images per method §4.1. Metrics: SOA-I (a YOLOv7 detector checks object presence), CLIP-score, IoU of a YOLOv7 segmentation against the layout, local CLIP-score on object crops, and a user study on Amazon Mechanical Turk §4.1.

Text fidelity tab. 2. Human preference is the share of raters who preferred DenseDiffusion over that baseline; 50% means a tie §4.1.

Layout fidelity tab. 3. SD-Pww is the only baseline with layout control, so it is the only one compared here tab. 3.

Ablation tab. 4 fig. 7. Each row removes one component from the full method.

The authors say removing (d) gives a flatter background that is easier for the detector to segment, which lifts IoU while lowering text scores §4.2.

Other results. The appendix compares qualitatively with Attend-and-Excite, Universal Guided Diffusion and MultiDiffusion, on the authors' prompts and on MS-COCO prompts. The authors say MultiDiffusion often looks like a collage of objects, and the other two often fail to follow the conditions app. A.2 fig. 9 fig. 14. Against the layout-trained MAS and SpaText, only qualitative comparisons on SpaText's own examples are given §4.2 fig. 5.

Paper vs code

Official code: naver-ai/DenseDiffusion at commit 54a2ea645a6211f855212efff1720875cef2c090 (Apache-2.0). The paper's abstract links it, and its README lists the authors at NAVER AI Lab, the affiliation on the paper. The modulation is in gradio_app.py (the mod_forward function that replaces each attention call), and the masks and token replacement are in process_generation gradio_app.py gradio_app.py. The helper utils.py builds the masks from a sketch utils.py.

Discrepancies and undocumented choices:

1. Classifier-free guidance. The paper's equations apply to every query row and never mention guidance. The code concatenates unconditional and conditional text embeddings, with unconditional first, and modulates only the second half of the batch gradio_app.py gradio_app.py. An implementation that modulates the unconditional branch too differs from the code.
2. Step gating. The paper says t from 1 to 0.7 §3.2. The code counts attention calls and applies modulation while COUNT/32 < 50 × 0.3, so the first 15 steps, assuming 32 attention calls per UNet pass gradio_app.py. The timestep used is the scheduler's timestep divided by 1000 gradio_app.py.
3. Area term. The paper uses (1 − S) with no coefficient eq. 2. The code uses (1 − sizereg · S), and the slider defaults to 1.0, which matches the paper gradio_app.py gradio_app.py.
4. Scaling of M. The paper places M before the division by sqrt(d) eq. 2. The code adds the update to the scaled scores gradio_app.py gradio_app.py. These are the same update, as noted above.
5. Unlabelled pixels and tokens. The paper does not say how unlabelled regions are treated. In the code, the GUI sketch reads white as a background segment with its own prompt utils.py gradio_app.py. The example loader reads black as background utils.py. Any token or pixel outside every segment gets zero in the pair map, so it receives the negative update gradio_app.py.
6. Token replacement. The paper says parts of the text features are replaced with separately encoded segment features §3.2. The code replaces only the first occurrence of each segment's token span in the full prompt gradio_app.py.
7. Model version, guidance, size. Not stated in the paper. The code loads Stable Diffusion v1.5 and does not set a guidance scale or image size gradio_app.py gradio_app.py.
8. Environment. The code exits unless diffusers is exactly 0.20.2 gradio_app.py.

Pitfalls

• Missing phrase fails silently. If a segment's phrase is not found verbatim in the full prompt, the matching loop finds nothing and moves on. The segment then gets no positive cross-attention pairs gradio_app.py. The error message in the code is raised only when an exception occurs, not when the phrase is missing. Check that every segment phrase appears in the full caption before running.
• The step count is hard-coded. The gate depends on 50 steps and on 32 attention calls per UNet pass gradio_app.py. Changing the number of steps, or the UNet, needs the gate updated too.
• Guidance interacts with the method. Only the conditional half of the batch is modulated gradio_app.py. If you change the guidance setup, verify which half is which.
• Cost in modulated steps. In the first 15 steps the code builds the full score matrix and takes row maxima and minima gradio_app.py gradio_app.py. The later steps use the library's attention path gradio_app.py. The paper reports no timing or memory figures, so this is from reading the code only.
• Segment size. Area term (1 − S) is near zero when one segment covers almost the whole image, so the adjustment fades in that case eq. 2. The paper says quality drops when one segment is much smaller than the others §3.2.
• Fine detail and missing concepts. The authors report failure cases: the base model cannot draw a concept, and the coarse attention maps cannot follow fine shapes such as leaves §5 fig. 8.
• Metrics rely on a detector. SOA-I and IoU use YOLOv7 §4.1. The authors note that SOA-I correlates only loosely with human judgement, because of a domain gap between LAION-trained Stable Diffusion and MS-COCO-trained YOLOv7 §4.2.
• Benchmark is author-built. The 250-caption set and its phrase substitutions were made by the authors §4.1. The code ships dataset/testset.pkl and valset.pkl testset.pkl valset.pkl.

Lineage

• Builds on: Stable Diffusion [ldm] and DDIM sampling, cited in §3.1. Neither has a card in this wiki.
• Compared against: Composable Diffusion [compdiff], Structure Diffusion [strucdiff], SD-Pww (eDiffi-Pww) [ediffi], MultiDiffusion [multidiffusion], Attend-and-Excite [attendnexcite], Universal Guided Diffusion [universaldiffusion] §4.1 app. A.1; MAS [makeascene] and SpaText [spatext] qualitatively §4.2.
• Superseded by: not stated in the paper.