Text-guided image editing

Overview

Given an input image and a target prompt, the goal is an edited image that follows the prompt while keeping some part of the input. The eight covered variants differ in what they keep. Plug-and-Play keeps the spatial layout of a real image 2211.12572 §4. VQGAN-CLIP starts from the input image, but its loss only scores the match to the text, plus an optional L2 penalty on the latent 2204.08583 §2 §2.4; the paper reports no metric for how much of the input survives an edit 2204.08583 §4.1. FashionGAN changes only the clothing of a person in a photo, keeping pose and body shape, and is trained for that one task 1710.07346 §1 §3.1. BLIP-Diffusion takes its subject from a reference photo rather than from the layout of the input image 2305.14720 §3.2. SINE trains a copy of Stable Diffusion on the input photo itself, so the source content is carried by the weights rather than by the layout alone 2212.04489 §3.1 §3.2. StyleCLIP works on a pretrained StyleGAN2 rather than a diffusion model. A real photo is first inverted into the generator's latent space, and the edit moves that latent under CLIP's guidance 2103.17249 §4 §5. Text2LIVE keeps the whole input under a learned edit layer, so where the layer's opacity is zero the input shows through unchanged 2204.02491 eq. 1 §3.1. CLIPstyler needs no style image: it restyles a content photo to match a text description, training a small network per photo whose output is scored by CLIP 2112.00374 §3.1 §4.1.

Text alone gives little control over layout: the same prompt yields different poses and layouts from different initial noise 2211.12572 §1. The Plug-and-Play variant addresses this by running the model on the source image first and carrying part of its internal state into the edited run.

The invariant of the Plug-and-Play variant: the edited run starts from the same initial noise as the source's reconstruction. With a different seed, injected features do not match the encoder skip features and the method fails 2211.12572 app. B.

A 2023 survey groups editing into partial-noising methods (SDEdit-style), attention control, instruction-based fine-tuning, drag-style edits and inversion-based editing 2310.07204 §3.4. It gives no numbers for these, so this page does not rank them and covers only the cards listed below.

Variants

Plug-and-Play diffusion features (training-free)

The source image is encoded and DDIM-inverted with an empty prompt to a noise latent. A guidance pass then denoises that latent back while recording a decoder ResBlock feature map and the decoder self-attention queries and keys. The edit pass starts from the same latent, samples with the target prompt, and overwrites the matching feature map and attention queries and keys with the recorded ones during the early sampling steps 2211.12572 §4 alg. 1. The base model is a frozen Stable Diffusion v1.4 2211.12572 app. C.

The authors pick the feature map from the fourth decoder layer because, in their PCA, it encodes semantic parts (head, legs, torso) consistently across images with very different appearance 2211.12572 §4. Injecting only that feature map keeps too little of the structure; adding it at deeper layers keeps structure but leaks source colors into the result. Adding self-attention at higher-resolution layers on top of the feature map balances the two 2211.12572 §4.

Reported defaults: 50 translation steps, feature injection in the first 40 and self-attention injection in the first 25, with classifier-free guidance scale 15 for real images and 7.5 for generated guidance images 2211.12572 app. C. Negative prompting, which replaces the unconditional term with a blend that includes a description of the source, mainly helps textureless inputs such as silhouettes 2211.12572 §4 2211.12572 app. A.1.

Reported results are the authors' own, on benchmarks they built. Against SDEdit at three noise levels, P2P, DiffuseIT and VQGAN-CLIP, the authors report a structure distance comparable to SDEdit at a very low noise level and a text score comparable to SDEdit at a very high one 2211.12572 §5.1 fig. 9. On the Wild-TI2I real-image set, the full method has a structure distance of 0.058, against 0.090 without the feature injection and 0.097 without the self-attention injection 2211.12572 tab. 1. The paper's P2P comparison covers only its two ImageNet-R benchmarks, because P2P needs word-aligned prompts 2211.12572 §5.1.

Card: Plug-and-Play Diffusion Features.

VQGAN-CLIP latent optimisation (training-free, slow per image)

Editing uses the same loop as generation. A pretrained VQGAN decoder turns a latent grid of codebook vectors into an image, and a pretrained CLIP model scores augmented crops of that image against the prompt. Each step averages the score over many crops, and Adam updates the latent grid 2204.08583 §2 §2.3. For editing, the start latent is an encoded input image instead of random codes 2204.08583 §2 §2.1. Nothing is trained 2204.08583 §1.

The paper states a learning rate of 0.15 and 400 iterations 2204.08583 §2. Its runtime per image is 188 s on a V100 and 935 s on a K80 [arxiv:2204.08583 tab. 2]. A Colab run is reported under 3 minutes with under 11 GB of video memory 2204.08583 §6.1. The editing comparison with Open-Edit is visual only, with no metric 2204.08583 §4.1.

The human study covers 12 prompts, rated 1 to 5. This method averages 4.6, and the best GLIDE baseline with CLIP guidance averages 3.3. The baselines were picked best-of-five and this method best-of-one 2204.08583 tab. 1 §3.2. These are generation prompts, not edits.

Card: VQGAN-CLIP.

Layered per-input editing with CLIP losses (trained per input, image and video)

Text2LIVE trains a new generator for each input. The generator is a small U-Net that takes an augmented crop of the input and outputs an edit layer, a colour image plus an opacity map. The output is the opacity-weighted layer laid over the input 2204.02491 eq. 1 §3.1 app. A.1. Training uses CLIP only. It has a composition loss against the target prompt, a loss that asks CLIP to match the layer alone over a green background, and a structure loss that compares CLIP token self-similarity of input and output. Sparsity and bootstrap terms act on the opacity map 2204.02491 eq. 2 §3.1. The method needs no task-specific pretraining and no mask 2204.02491 §3.1.

For video, the frames are mapped to a pretrained, frozen neural layered atlas. The same image procedure runs on atlas crops, and its losses are applied to the reconstructed frames, so one edit carries across the video [arxiv:2204.02491 §3.2] app. A.4. The paper trains the foreground and background atlases as two separate generators 2204.02491 app. A.4.

Cost is per input. The image setting uses 1000 iterations, and the authors report about nine minutes at 512 x 512 on an RTX 6000 [arxiv:2204.02491 app. A.3]. The video setting uses 3000 iterations on 70 frames and takes about an hour per layer on an RTX 8000 2204.02491 app. A.3. Nothing carries over between inputs 2204.02491 app. A.3.

Reported results are the authors' own, from a two-alternative human preference study on Mechanical Turk. For images, Text2LIVE received 0.82 to 0.86 of votes against three mask-free baselines: CLIPStyler, VQ-GAN+CLIP and Diffusion+CLIP 2204.02491 tab. 1 §4.3. For video it received 0.73 and 0.74 against two video baselines, a single-image method run on the atlas and a per-frame generator 2204.02491 tab. 1. Blended-Diffusion and GLIDE need masks and were compared only qualitatively [arxiv:2204.02491 §4.2]. The paper reports that CLIP can pull new objects into an edit, for example candles on a "birthday cake", and calls the method unsuited to creating new objects 2204.02491 §4.5 fig. 9.

The shipped code differs from the paper in several settings, including the image sparsity weights, the structure-loss form and the bootstrap schedule. Check the card's Paper vs code section before you copy a value.

Card: Text2LIVE: Text-Driven Layered Image and Video Editing.

Two-stage GAN for outfit change (trained, one domain)

Given a photo of a person and a text description of an outfit, the model returns a 128x128 image of the same person, with the same pose and body shape, in the new outfit 1710.07346 §1 §3.5. Stage 1 generates a 7-label segmentation map from a coarse 8x8 map of the wearer in which all clothing labels are merged into one class 1710.07346 §3.2. Stage 2 gives each label its own output channel and combines them pixel by pixel using the label map 1710.07346 §3.3 eq. 4. Hair and face pixels are then copied back from the original photo 1710.07346 §3.5.

Both conditional GANs are trained separately, with Adam 1710.07346 §3.4. Training needs one photo, one description and its segmentation for each person. No pairs of one person in two outfits are used 1710.07346 §3.1. The dataset has 78,979 images, split 70,000 for training and 8,979 for testing 1710.07346 §3.5 §4. On attribute detection, the generated images score mAP 82.6, against 71.8 to 74.7 for the baselines and 86.8 for real photos 1710.07346 tab. 1. These are author-reported. The body does not state the learning rate, batch size or epoch count 1710.07346 §3.4.

Card: Be Your Own Prada: Fashion Synthesis with Structural Coherence.

Subject-driven editing from a reference photo (pre-trained, zero-shot or short fine-tune)

A frozen BLIP-2 image encoder and a trainable multimodal encoder turn a subject photo and its category word into 16 subject tokens. These are added to the prompt, and Stable Diffusion v1-5 denoises as usual 2305.14720 §3.2 §3.1 §4.1. For editing an existing image, the cross-attention maps of a chosen token give a mask. Unedited regions keep the original generation's latents, and the masked regions take the subject-driven result 2305.14720 §3.3 fig. 4. The paper gives no numeric values for the editing thresholds 2305.14720 §3.3.

No per-subject training is needed for the zero-shot setting. A short fine-tune of the U-Net on a few photos runs 40 to 120 steps, with a mean of 76 2305.14720 §3.3 app. A.5 tab. 3. Pre-training used 500K steps on 16 A100 40GB GPUs over six days, and only the paper reports it; the released repository has fine-tuning configs, not that stage 2305.14720 §4.1 finetune-db-dog.yaml.

Reported results are the authors' own. On DreamBench, the zero-shot model scores DINO 0.594 and the fine-tuned model 0.670, against 0.668 for DreamBooth 2305.14720 tab. 1. That table measures subject fidelity in generated images, not edit quality, and the quantitative comparison covers no editing benchmark 2305.14720 §4.2.

Card: BLIP-Diffusion.

Single-image fine-tune with guidance mixing (per-image training, qualitative results)

SINE fine-tunes a pretrained Stable Diffusion on one image. Each training sample is a random patch of that image, resized to the model's native size, and the denoiser also gets a positional code for where the patch sits 2212.04489 §3.3. At sampling time the full-image code is used, and the noise prediction for early steps (timestep above K = 400) mixes the fine-tuned and pretrained models with weight v = 0.7. Later steps use the pretrained model alone 2212.04489 §3.2 §4 §4.3.

The reported training setup is a base learning rate of 1e-6, batch size 1, 800 steps without patches and 10,000 with patches, on one RTX 8000 GPU 2212.04489 §4. The paper states no wall-clock time. Patch training is what the authors credit for edits at a larger size without duplicated objects, though very different aspect ratios still produce artifacts they show 2212.04489 §4.3 app. D. Expect to choose K and v per image; the appendix says the best values differ by image 2212.04489 app. C.

Reported results are qualitative. The paper has no numeric headline, its K and v sweeps are plots without values in the text 2212.04489 fig. 8, and its comparisons keep the best of four samples per prompt 2212.04489 §4.1. The baselines are DreamBooth and Textual Inversion 2212.04489 §4.2.

Card: SINE: SINgle Image Editing with Text-to-Image Diffusion Models.

Implementation warning: the released CLI applies the fine-tuned term at the other end of the schedule from the paper. Its default window is 1 to 400, while the paper, README and notebook use the window above K = 400 stable_txt2img_guidance.py. Copy the notebook's values.

Text-driven StyleGAN editing (pretrained GAN, three routes)

StyleCLIP drives a pretrained StyleGAN2 generator with the CLIP cosine distance to the prompt 2103.17249 §4 eq. 1. The paper offers three routes that trade cost per image against dependence on the input:

• Latent optimisation (per image and prompt). Gradient descent on the latent minimises the CLIP term, a penalty on the distance from the source latent, and an optional ArcFace identity term 2103.17249 §4 eq. 1 eq. 2. The paper states 200 to 300 iterations §4. Inference takes 98 s on a single GTX 1080Ti 2103.17249 tab. 1. The paper calls this method sensitive to its parameters and leaves it out of its quantitative comparison 2103.17249 §5 §7.
• Latent mapper (trained per prompt). Three small networks, one each for coarse, medium and fine layer groups of the W+ latent, predict a residual for any input latent in one forward pass 2103.17249 §5 eq. 3. Training takes 10 to 12 h and inference 75 ms, both on a GTX 1080Ti tab. 1.
• Global direction (input-independent, in style space S). A text direction is computed from prompt embeddings, and per-channel CLIP relevance keeps only the style channels that move with it 2103.17249 §6 eq. 6 eq. 7. The strength is a user dial. Pre-processing takes 4 h and inference 72 ms, and the same direction applies to any input tab. 1.

The paper's comparisons are visual. It reports no quantitative image-quality or identity metric 2103.17249 §7 app. B.

Card: StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.

CLIP-trained text-only style transfer (trained per image, or once per text)

CLIPstyler restyles a content photo from a text prompt alone. For each content image and text, it trains a small U-Net from scratch; the network's output is the stylised image 2112.00374 §3.1 §4.1. The main term is a whole-image directional CLIP loss. It compares how the output moves in CLIP's embedding space from the content photo with the move from a source word to the target text 2112.00374 eq. 2. A patch term applies the same test to 64 random 128-pixel crops of the output, each with a random perspective change. Patches whose loss is at or below a threshold of 0.7 count as zero 2112.00374 §3.2 eq. 3 eq. 4 app. A. A VGG-19 content term on two layers and a total-variation term complete the loss 2112.00374 §4.1.

Cost is per image and per text. The paper trains 200 iterations with Adam, at a learning rate of 5 × 10⁻⁴ halved at iteration 100, and reports about 40 seconds per text on one RTX 2080 Ti at 512 × 512 2112.00374 §4.1. The paper states that real-time per-input use needs its fast variant [arxiv:2112.00374 app. G]. The fast variant trains only the decoder of a pretrained encoder-decoder, on random crops of DIV2K images. It reports under 0.5 s of inference per content image 2112.00374 §5.1 app. A. The authors say its quality is lower than the per-image method 2112.00374 §5.1.

Reported results are the authors' own. In a 25-rater study, the method scores 3.78 for style, the highest of seven methods, against 3.32 for AdaIN with CLIP 2112.00374 tab. 1. Its content score is 4.18, below StyleCLIP's 4.35 tab. 1. Its CLIP score is 0.2515, against 0.2487 for AdaIN with CLIP 2112.00374 tab. 2; the paper does not name the CLIP model used for that score app. B.3. In the ablation, the full loss is preferred at 3.92 and removing the patch term drops it to 1.69 2112.00374 tab. 3.

Implementation warning: the paper's introduction and threshold paragraph say that high-scoring patches are rejected, but its equation 4 zeroes patches at or below the threshold 2112.00374 §1 §3.2 eq. 4. Follow the equation.

Card: CLIPstyler: Image Style Transfer with a Single Text Condition.

Other editing methods (no card yet)

The Plug-and-Play paper compares against SDEdit, Prompt-to-Prompt, DiffuseIT, VQGAN-CLIP, Text2LIVE, DiffusionCLIP and FlexIT 2211.12572 §5.1. Of these, only VQGAN-CLIP and Text2LIVE have cards here. The VQGAN-CLIP paper names Open-Edit, minDALL-E and GLIDE as its baselines 2204.08583 §3.2 §4.1. The Text2LIVE paper names Blended-Diffusion, GLIDE, CLIPStyler, Diffusion+CLIP, VQ-GAN+CLIP, StyleCLIP and StyleGAN-NADA among its comparisons [arxiv:2204.02491 §4.2]. None of the methods without a card has one here, so this page says nothing about how they work or how they score. Prompt-to-Prompt is linked on arXiv for reference: arXiv:2208.01626. SDEdit is linked on arXiv for reference: arXiv:2108.01073.

When to use which

The rows below rest on the cards above. They do not rank the variants against each other, because the cards report different metrics on different tasks (see Contradictions).

Implementation sketch

Own-words sketch of the Plug-and-Play variant. It omits classifier-free guidance and the negative prompt for brevity 2211.12572 §4.

# Structure-preserving edit by reusing source features  [arxiv:2211.12572 alg. 1]
# Input:  source image I, target prompt P, steps=50, feat_steps=40, attn_steps=25
# Output: edited image

def edit(I, P, steps=50, feat_steps=40, attn_steps=25):
    # Invert the source with an empty prompt to a shared start noise  [app. C]
    x_T = ddim_invert(encode(I), prompt="", n=1000)
    # Guidance pass: record one decoder feature map and decoder q, k per step  [§4]
    recorded = []
    x = x_T
    for t in timesteps(steps):
        eps, feat, qk = unet_record(x, t, prompt="")
        recorded.append((feat, qk))
        x = ddim_step(x, eps, t)
    # Edit pass: same start noise, target prompt, overwrite the early steps  [§4]
    x = x_T
    for i, t in enumerate(timesteps(steps)):
        feat, qk = recorded[i]
        eps = unet_inject(
            x, t, P,
            feat=feat if i < feat_steps else None,
            qk=qk if i < attn_steps else None,
        )
        x = ddim_step(x, eps, t)
    return decode(x)

Where variants differ: the other variants do not share this loop. VQGAN-CLIP optimises a latent grid against CLIP's score 2204.08583 §2, Text2LIVE trains a per-input generator whose layer is composited over the input 2204.02491 §3.1, FashionGAN trains two conditional GANs 1710.07346 §3.1, BLIP-Diffusion conditions the U-Net on subject tokens instead of on a source feature map 2305.14720 §3.2, and SINE fine-tunes the U-Net on the source and mixes its noise prediction into the pretrained one for early steps 2212.04489 §3.2. StyleCLIP has no diffusion trajectory at all: it optimises or maps a StyleGAN latent 2103.17249 §4 §5. CLIPstyler trains a small U-Net per content image against CLIP's directional loss, with no latent to invert [arxiv:2112.00374 §3.1]. Their cards hold their own descriptions.

Contradictions

No card states a result that another card contradicts. The results are not comparable across cards, and the cards do not share a metric:

• Plug-and-Play reports structure distance and CLIP text score on benchmarks built by its authors 2211.12572 §5.1 tab. 1.
• VQGAN-CLIP reports a 1-to-5 human rating on 12 generation prompts 2204.08583 tab. 1.
• FashionGAN reports attribute-detection mAP on its own test set 1710.07346 tab. 1.
• Text2LIVE reports the share of human votes in two-alternative comparisons, on 82 image-text pairs and 19 video-text pairs 2204.02491 tab. 1 §4.3. Its image baselines include VQ-GAN+CLIP, but its vote share is not on the 1-to-5 scale that the VQGAN-CLIP card uses, so the two sets of numbers do not compare.
• BLIP-Diffusion reports DINO, CLIP-I and CLIP-T on DreamBench, a subject-fidelity benchmark for generation 2305.14720 tab. 1.
• SINE reports no numbers in its text. Its LPIPS and CLIP values are only in plots 2212.04489 fig. 8.
• StyleCLIP reports timings and visual comparisons, with no quantitative image-quality metric 2103.17249 tab. 1 §7.
• CLIPstyler reports StyleCLIP scores of 1.47 for style and 4.35 for content, and a CLIP score of 0.1982, in its own study on face images for that baseline 2112.00374 tab. 1 tab. 2 app. B.2. These come from a setup that CLIPstyler chose, not from StyleCLIP's paper, so do not compare them with StyleCLIP's own visual results.
• CLIPstyler's headline numbers are 1-to-5 user ratings from 25 raters and a CLIP cosine score, both from its own study 2112.00374 tab. 1 tab. 2. They are not on the scale of the VQGAN-CLIP ratings or the Text2LIVE votes.
• StyleCLIP's text says latent optimisation takes "several minutes" per edit 2103.17249 §4 §5, while its table gives 98 s on a GTX 1080Ti 2103.17249 tab. 1. The paper does not reconcile the two.

The Plug-and-Play authors state that VQGAN-CLIP and DiffuseIT optimise the CLIP and DINO self-similarity metrics directly 2211.12572 §5.1. That is their reading of the method; the VQGAN-CLIP card does not measure structure.

Guidance-weight convention differs between the survey and the editing cards. The survey writes classifier-free guidance as (1+w) times the conditional prediction minus w times the unconditional one 2310.07204 eq. 11. The Plug-and-Play and SINE cards write w times the conditional plus (1-w) times the unconditional 2211.12572 eq. 5 2212.04489 eq. 2. The two forms are the same with the weight shifted by one, so a survey w equals the editing cards' w minus 1. Check which form a code base uses before you copy a value.

Papers

• Plug-and-Play Diffusion Features: training-free editing of a real image by copying one decoder feature map and decoder self-attention queries and keys from a source pass into a target-prompt pass on a frozen Stable Diffusion.
• VQGAN-CLIP: training-free text-driven generation and editing by optimising a pretrained VQGAN latent grid against CLIP's score.
• Be Your Own Prada: Fashion Synthesis with Structural Coherence: a two-stage conditional GAN that changes the outfit in a person's photo while keeping pose and body shape.
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing: a pre-trained subject encoder that conditions Stable Diffusion on a reference photo, with a mask-based editing mode.
• SINE: SINgle Image Editing with Text-to-Image Diffusion Models: fine-tunes Stable Diffusion on one image with patch crops and a positional code, then mixes its prediction into the pretrained model for early steps.
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery: text-driven edits of a pretrained StyleGAN2 through per-image latent optimisation, a per-prompt latent mapper, or a global direction in style space.
• CLIPstyler: Image Style Transfer with a Single Text Condition: a per-image U-Net trained with patch-wise CLIP directional losses to restyle a photo from text alone, with a fast decoder-only variant.
• Text2LIVE: Text-Driven Layered Image and Video Editing: a per-input generator outputs a colour-and-opacity layer over the image, trained with CLIP losses, with a neural-atlas route for video.
• State of the Art on Diffusion Models for Visual Computing: a survey with no experiments of its own; its editing section is a narrative taxonomy, used here only as background.