wikillm

Paper · 2023-03-20 · cs.CV, cs.GR, cs.LG

Localizing Object-level Shape Variations with Text-to-Image Diffusion Models

Training-free method (Prompt Mix & Match) for shape variations of one object in a Stable Diffusion image. A proxy word replaces the object word only in the Values of a middle denoising interval; self-attention injection and self-segmentation keep the rest. Author-reported: a better diversity, faithfulness and preservation trade-off than the baselines tested.

Text-guided image editingDiffusion modelsCross-attentionSelf-attentiondiffusionimage-editingprompt-mixingself-attention-injectionstable-diffusiontraining-free

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A text-to-image model usually gives you a new picture every time you change the random seed, and the whole picture changes with it. This paper asks a narrower question: how do you get several versions of one object in a picture, say a basket with a different handle or a different shape, while the rest of the scene stays put?

The model draws an image in stages. Early on it settles the overall layout, in the middle it decides the shapes of things, and late on it adds fine texture. The authors use that order. They describe the object with one word while the layout is being set, and swap in a different, nearby word for the middle stage, so the shape changes while the layout is kept. A second set of tools keeps the rest of the picture in place: the model's own attention maps show which pixels belong to which object, and the background is copied back from the original image.

The method needs no training and no per-image optimisation for images the model generated itself. It is aimed at object-level exploration: you get a small gallery of shape options for one object without writing new instructions for each one.

TL;DR

Training-free prompt mixing on Stable Diffusion: the object word is replaced by a CLIP-nearby "proxy" word only for the middle denoising interval, while Keys stay on the original prompt. Two localisation tools, self-attention row/column injection and self-segmentation with background blending, keep the rest of the image. Author-reported: a better trade-off between diversity, object faithfulness and preservation than the baselines tested; no numbers in the text.

Problem

Text-to-image sampling with new seeds changes layout, shape and appearance together, so users cannot explore one object §1. Inpainting and SDEdit mostly change texture, and changing a shape while keeping the rest of the image is hard for them §1 fig. 2 fig. 8. Text-guided editing methods such as Prompt-to-Prompt and Plug-and-Play inject the whole self-attention map to keep structure, which blocks the shape change the user wants §3 §5.1. The method assumes Stable Diffusion, an object word present in the prompt, and that proxy words in CLIP's text space give plausible shape alternatives §4.2 §6.1.

Method

Overview. The denoising process is treated as coarse to fine: layout first, then shape, then detail §1 §4.1. The paper checks this by changing the prompt word in each interval of a 50-step run and seeing which attribute follows which interval. Layout follows the early interval, shape follows the middle one, and texture follows the last one §4.1 fig. 4.

Prompt-mixing (Mix-and-Match). Let P(w) be the original prompt and w the object word. Let w' be a proxy word. Three intervals are used, [T, T3), [T3, T2) and [T2, 0]. The first and last intervals use P(w), and the middle interval uses P(w') §4.2. Only the cross-attention Values are taken from the altered prompt in the middle interval. Keys always come from the original prompt §4.1 app. A.

Pseudocode
# Mix-and-Match inside the cross-attention layers  [§4.2] [§4.1]
# Steps count from the first denoising step; T = 50  [app. B]
# Keys always use the original prompt P(w)  [§4.1] [app. A]
def cross_attention_inputs(step, original_prompt, proxy_prompt):
    keys = encode_keys(original_prompt)
    # Values switch to the proxy prompt only in the middle interval
    # [T3, T2), i.e. steps 7 to 16 in the released defaults  [app. B]
    if start_step <= step < end_step:
        values = encode_values(proxy_prompt)
    else:
        values = encode_values(original_prompt)
    return keys, values

Proxy words. For the object word w, take the k nearest tokens in CLIP's text space, using prompts of the form "A photo of a " §4.2. Rank those k tokens by CLIP distance between the full prompt with t and the prompt with w, and keep the top m as proxies §4.2. Tokens need not be real words; the paper reports that loosely related tokens can still give useful variations §4.2 app. A.

Pseudocode
# Proxy words: CLIP-space neighbours of the object word  [§4.2]
# Step 1: nearest k tokens to "A photo of a <w>"  [§4.2]
# Step 2: re-rank by distance of the full prompt with each token [§4.2]
def proxy_words(object_word, prompt, vocabulary, k, m):
    candidates = nearest_tokens(
        vocabulary, "A photo of a " + object_word, k)
    ranked = sorted(
        candidates,
        key=lambda t: clip_distance(prompt_with(t), prompt_with(object_word)),
    )
    return ranked[:m]

Attention-based shape localisation. To keep a non-target object (for example a dog) while the target changes, the self-attention map of the generated image is overwritten, row by row and column by column, for the pixels of the preserved object §5.1 eq. 1 eq. 2. Those pixels come from thresholding the cross-attention map of the object's token §5.1.

Pseudocode
# Self-attention injection in one layer and one step  [§5.1] [eq. 1] [eq. 2]
# source_map, generated_map: (N2, N2) self-attention maps  [§5.1]
# object_pixels: indices in O_t, from a cross-attention threshold  [§5.1]
def inject_self_attention(source_map, generated_map, object_pixels):
    # Mask is 1 in the rows and columns of the preserved object  [eq. 1]
    mask = zeros_like(generated_map)
    mask[object_pixels, :] = 1
    mask[:, object_pixels] = 1
    # Copy the source map where the mask is 1  [eq. 2]
    return mask * source_map + (1 - mask) * generated_map

Self-segmentation. The image is segmented with the model's own self-attention, not an outside segmenter §5.2. At the segmentation step, the 32²×32² self-attention maps are averaged over the whole denoising run. Each pixel is then a 1024-dim vector, and K-means with 5 clusters gives the segments §5.2 app. B.

Pseudocode
# Self-segmentation from averaged self-attention  [§5.2] [app. B]
# avg_self_attn: (1024, 1024) averaged over all steps  [§5.2]
def segment(avg_self_attn, num_clusters):
    # Each of the 32 x 32 pixels is a 1024-dim feature  [§5.2]
    features = reshape(avg_self_attn, (32, 32, 1024))
    # K = 5 clusters in the released configuration  [app. B]
    labels = kmeans(features, num_clusters)
    return labels

Labelling segments. Each segment is matched to a noun of the prompt, using the normalised aggregated cross-attention maps of the nouns §5.2. For a segment mask M_i and a noun map A_n, the score is the sum of their product divided by the segment's area. The best noun is the label, unless the best score is at most σ = 0.3, in which case the segment is background §5.2 app. B. The paper states that σ is fixed across experiments §5.2.

Pseudocode
# Label a segment with its best noun, or background  [§5.2] [app. B]
# score = sum(M_i * A_n) / sum(M_i), sigma = 0.3 [app. B]
def label_segment(segment_mask, noun_maps, sigma):
    scores = {}
    for noun, noun_map in noun_maps.items():
        scores[noun] = sum(segment_mask * noun_map) / sum(segment_mask)
    best = argmax(scores)
    if scores[best] > sigma:
        return best
    return "background"

Controllable background preservation. At a fixed step in the denoising run, the generated latents are merged with the original latents. Pixels come from the original image only where the segment is background or a user-listed object in both the original and the new image. The rest come from the generated image §5.2 §6.2. This keeps the edit local, so a changed handle does not drag the rest of the basket with it §5.2 fig. 3.

Pseudocode
# Blend original and generated latents at the blend step  [§5.2] [§6.2]
# keep_original: 1 where background or user object in both images  [§5.2]
def blend_latents(generated, original, keep_original):
    # Take the original where it is kept, otherwise the generated latent
    return keep_original * original + (1 - keep_original) * generated

Real images. The image is first inverted with null-text inversion §6.2, then the same pipeline runs. The card does not analyse the inversion settings; see Paper vs code.

Hyperparameters

Paper values are given where stated. Code defaults are in main.py and are not stated in the paper unless noted.

NameValueWhereNotes
Diffusion modelStable Diffusion (version not stated; released code uses CompVis v1-4)§3 diffusion_model_wrapper.pyPaper names only "publicly available Stable Diffusion" §3
Denoising steps T50app. B main.py
Image size512×512, latent 64×64diffusion_model_wrapper.pynot stated in paper
Guidance scale7.5main.py diffusion_model_wrapper.pynot stated in paper
Prompt-mixing intervalT3 = 44 ± 1, T2 = 34 ± 1app. BCode uses step indices 7 and 17 main.py. Paper: the optimum may move by a step between prompts and seeds app. B
Self-attention injectionMask only on layers with at most 32² tokensprompt_mixing.pynot stated in paper
Self-attention injection stepsnot statedmain.pycode default: first 40 steps
Object-pixel threshold"fixed threshold", value not stated§5.1code default 0.05 on min-max-normalised cross-attention main.py prompt_mixing.py
Remove object from self masknot stated in main textapp. Acode default True main.py
Segmentation clusters K5app. Bcode default 5 main.py
K-means initialisationnot stated—code: 10 inits, seed 1 attention_based_segmentation.py
Label threshold σ0.3app. Bcode default 0.3 main.py
Background blend steppaper §5.2: "t = T1"; §6.2: "t = 35"; app. B: T1 = 15 ± 1§5.2 §6.2 app. Bcode default 35 counted from the first step main.py
Proxy candidates knot stated§4.2code: 50 nearest from vocabulary prompt_utils.py
Proxy words m20 variations per image in evaluation§6.1 app. Bcode default 20 main.py
Evaluation set150 images (3 templates × 5 words × 10 seeds), 20 variations per method per imageapp. B
Random seednot stated—code default 10 main.py
Mask dilation3×3, one iteration; mask upsampled to 64×64diffusion_model_wrapper.pynot stated in paper
Null-text inversionnot stated in paper§6.2not analysed in this card

Results

Author-reported. Setup: 150 generated images, 20 variations per method per image, against inpainting, SDEdit, Prompt-to-Prompt, Instruct-Pix2Pix, Plug-and-Play, Zero-shot Image2Image and Imagic §6.1 app. B.

Metrics: diversity is 1 − IoU of CLIPSeg masks averaged over pairs; faithfulness is CLIP cosine similarity between the average embedding of the object class and each output; preservation is LPIPS against the original §6.1.

SettingMetricBaselineThis paperSource
Object variations, 150 imagesDiversity, faithfulness, preservationInpainting, SDEdit, edit methodsNumbers appear only in the plot; the text reports a "good trade-off"fig. 10 §6.1
Methods with higher diversityPreservation, faithfulnessEdit methodsThe text says higher-diversity methods drift from the original image or the object class§6.1
AblationQualitativeMix-and-Match aloneAdding attention localisation keeps the dog; adding background preservation keeps the background§6.1 fig. 11
P2P with our localisationQualitativeP2P with 40% and 20% full self-attention injectionHat changes to crown while the dog is preserved§6.2 fig. 12
P2P with background preservationQualitativeP2P with local blendingLocalises the edit better: without it, parts of a cat (tail, feet) are added to the bird§6.2 fig. 13
Imagic comparisonTime, qualityOptimisation-based ImagicPaper says the method is more time-efficient; no timing values in text§6.1

The paper gives no numeric scores in the text. Only the plots and the qualitative claims above are available.

Paper vs code

Code: the repository github.com/orpatashnik/local-prompt-mixing at commit 4bc225db751ae755604677cb2cdbb7bd8e732fe8 is linked from the paper's project page and is owned by the first author. The repository has no LICENSE file, so no code is quoted; all code references are pseudocode.

Consistent with the paper:

Differences:

  • Blend timing: §5.2 says "t = T1" while §6.2 says step 35 §5.2 §6.2. The code uses step index 35 counted from the first step, which is T1 = 15 with T = 50 main.py diffusion_model_wrapper.py. Both are the same point, but only if T1 is counted from the end of the run.
  • Object removed from the self-attention mask by default main.py prompt_mixing.py. The main text's eq. 1 keeps rows and columns of preserved objects only. The appendix app. A says removal of the target object is a user choice.
  • Cross-attention smoothing on the target token (a 3×3 median blend at 0.2/0.8) is coded but never runs with default settings: the replace range is set to empty (start = end = 50) prompt_mixing.py prompt_mixing.py. The paper does not describe it.
  • Self-attention injection stops at 40 steps by default and only on layers with at most 32² tokens main.py prompt_mixing.py. The paper gives neither number §5.1.
  • Noun map normalisation subtracts |min| and divides by max attention_based_segmentation.py. The paper says only "normalized". This is not plain min-max, so the scores may differ.
  • Proxy vocabulary: the paper ranks "all tokens" of CLIP's tokenizer §4.2. The code keeps alphabetic tokens of length two or more from vocab.json, takes the top 50 by pooled-text cosine similarity and then the top 20 prompt_utils.py.
  • Proxy ranking uses the pooled text embedding and cosine similarity prompt_utils.py. The paper says "CLIP distance" without naming the embedding.
  • Real images are inverted with null-text inversion and the DDIM scheduler main.py diffusion_model_wrapper.py. The paper says only "invert each image" §6.2. The inversion settings are not analysed here.

Pitfalls

  • Step counting. The paper writes intervals as T3, T2 and T1 counted down from T = 50. The code uses step indices counted up from the first step. Paper T3 = 44 ± 1 and T2 = 34 ± 1 correspond roughly to indices 7 and 17 app. B main.py. Mixing the two conventions moves the interval by about 10 steps.
  • Blend-step ambiguity. §5.2 and §6.2 disagree on how the blend time is written (T1 versus 35). Use index 35 with T = 50 to match the code.
  • Unstated thresholds. The object-pixel threshold (0.05 in code) is "fixed" in the paper with no value §5.1. Changing it changes which rows and columns are injected, so the result is sensitive to it.
  • Injection cost and scope. The injection mask is applied only to attention layers with at most 32² tokens, and only for the first 40 steps by default prompt_mixing.py main.py. Inject too long and the object stops changing; too short and the rest of the image drifts.
  • Keys versus Values. Only the Values use the altered prompt. The appendix reports that taking the altered prompt for Keys as well degrades the layout app. A. Do not swap the Keys.
  • Averaging for segmentation. Self-attention is averaged over all denoising steps before clustering §5.2 attention_based_segmentation.py. Cross-attention for labelling is also averaged over all steps and layers at matching resolution attention_utils.py prompt_to_prompt_controllers.py. The paper does not state this averaging.
  • Clustering randomness. K-means is seeded with 1 and run 10 times in the code attention_based_segmentation.py. Other seeds can give different segments, so changing the seed alters the labelled background.
  • Shared latent. Every variation in the released pipeline starts from the same initial latent main.py main.py. This matches the analysis in §4.1, which uses one noise for all rows.
  • Cost. The paper says the method needs no optimisation §1 §4.2. For real images it first inverts the photo with null-text inversion §6.2; the paper gives no cost for that step. The released code, in its default low-resource mode, makes the unconditional and conditional UNet predictions as separate calls instead of one batched call main.py diffusion_model_wrapper.py.
  • Proxy quality. The paper notes that some proxy words are unrelated tokens such as "something", and that these can still give shape changes app. A. Object-class faithfulness is measured only by CLIP similarity §6.1.
  • Continuous words. The conclusion says the word space is discrete and that continuous exploration is future work §7. Do not expect interpolation between proxies.

Lineage

  • Builds on: Latent Diffusion Models and the publicly available Stable Diffusion model §3. Not in this wiki.
  • Builds on: Prompt-to-Prompt, which the paper uses for cross-attention Key and Value roles and self-attention injection §3 §6.1. Not in this wiki.
  • Compared against: Plug-and-Play (self-attention injection of the whole map), inpainting, SDEdit, Instruct-Pix2Pix, Zero-shot Image2Image and Imagic §6.1. Only Plug-and-Play has a card.
  • Uses for real images: null-text inversion §6.2. Not in this wiki.

Source and license

Adapted from “Localizing Object-level Shape Variations with Text-to-Image Diffusion Models” by Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, Daniel Cohen-Or (arXiv:2303.11306), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/orpatashnik/local-prompt-mixing at commit 4bc225db751a (none (no LICENSE file in repository)). Any quoted code is shown under that license.