LEDITS++: Limitless Image Editing using Text-to-Image Models

In plain words

Changing a real photo with a text-to-image model has usually meant one of two trade-offs. You either retrain or fine-tune the model on that one photo, which is slow, or you push the photo back into the model's random starting noise and regenerate, which tends to drift away from the original.

LEDITS++ takes a third route. It runs the photo back into noise with a faster solver, so that if you ask for no edit, the photo comes back unchanged. Each edit you request, such as "add sunglasses" or "smile", then adds a small push toward that concept. The push is limited to the image regions where the model associates that concept, which the method estimates on the fly from the model's attention maps and from how the noise changes. Several edits can be requested together, each with its own push and region, so they interfere less. No training is needed for each new image.

TL;DR

• Edits a real image with a diffusion model without tuning or optimisation per image §1 §3.2.
• Inverts the image with a second-order SDE DPM-Solver++ so an unedited input is reconstructed exactly §3.2, Component 1.
• Each edit prompt gets its own guidance term, so several edits can be applied at once §3.2, Component 2.
• Each term is masked by the intersection of a cross-attention mask and a noise-difference mask §3.2, Component 3.
• Author-reported: 1.78 s versus 349.98 s for Imagic on a proxy oil-painting task on one A100 tab. 1 app. C.1.2; 87% edit success on TEdBench++ with SD-XL tab. 2. Code is MIT-licensed and published by the ml-research organisation.

Problem

Three barriers are named §1:

• Methods that tune the model on each input, such as Imagic, are expensive §1 §2.
• Inversion with DDIM needs many steps, and small errors accumulate, so the reconstruction drifts §2.
• Most semantically grounded editors handle one concept at a time, so multiple simultaneous edits interfere §1 §3.2.

Assumptions: the editing model must use classifier-free guidance (CFG) and a latent autoencoder in the experiments §3.1 app. C. The paper claims the approach is architecture-agnostic for any model using CFG §1. All reported experiments use Stable Diffusion 1.5 or SD-XL app. C §6.2. The cross-attention mask needs access to the U-Net's cross-attention maps; the paper does not discuss other architectures pipeline_stable_diffusion_ledits.py.

Method

Background notation §3.1: a DDPM-style forward process adds Gaussian noise to x_0 over T steps, with cumulative signal coefficient ᾱ_t. The U-Net predicts the noise ε̂_θ(x_t). CFG combines a conditional and an unconditional prediction with a guidance scale s_g §3.1.

Overview: the method has three components §3.2: (1) perfect inversion of the input with a higher-order SDE solver, (2) per-concept guidance for textual editing, and (3) semantic grounding of each change through implicit masks.

Component 1: inversion §3.2

1. Build a reconstruction path x_t = sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε̃_t, where each ε̃_t is drawn independently, unlike the correlated noise of the forward process §3.1 §3.2.
2. Compute noise maps z_t for t = T..1 by inverting one step of the second-order SDE solver. Its mean μ̂_t uses the latents at t and t+1, and the U-Net is evaluated once per step, reusing the previous estimate §3.2.
3. Optionally stop inversion at an intermediate index and start generation there. The paper reports that starting between 0.9T and 0.8T gave edits of the same fidelity as starting at T §3.2.

Key solver quantities as written in the paper §3.2:

\sigma_t = \sqrt{1-\bar{\alpha}_{t-1}}\,\sqrt{1-e^{-2h_{t-1}}}

\hat{\mu}_t(x_t,x_{t+1}) = \tfrac{\sqrt{1-\bar{\alpha}_{t-1}}}{\sqrt{1-\bar{\alpha}_t}}\,e^{-h_{t-1}}\,x_t + \sqrt{\bar{\alpha}_{t-1}}\,(1-e^{-2h_{t-1}})\,\hat{\epsilon}_\theta(x_t) + \tfrac{1}{2}\sqrt{\bar{\alpha}_{t-1}}\,(1-e^{-2h_{t-1}})\,\tfrac{-h_t}{h_{t-1}}\,\big(\hat{\epsilon}_\theta(x_{t+1})-\hat{\epsilon}_\theta(x_t)\big)

The paper defines h_t as a difference of ratios of logarithms of ᾱ terms §3.2. See Pitfalls for why the code's definition should be used instead.

# Reconstruction path: one independent noise draw per step  [§3.2]
def reconstruction_path(x0, timesteps, alpha_bar):
    xts = []
    for t in timesteps:
        eps = sample_normal(x0.shape)                      # (B, 4, h, w)
        xts.append(sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps)
    return xts

# Noise map z_t from the second-order SDE step  [§3.2, Component 1]
def noise_map(x_prev, x_t, x_next, t, eps_t, eps_next):
    # eps_t and eps_next are U-Net outputs at x_t and x_next  [§3.2]
    mu = mu_hat(x_t, x_next, eps_t, eps_next, t)           # [§3.2]
    sigma = sigma_t(t)                                     # [§3.2]
    return (x_prev - mu) / sigma                           # [§3.2]

Component 2: per-concept guidance §3.2

For each edit prompt e_i the method forms a direction ψ_i from the difference between the edit-conditioned and the unconditional estimate. The sign is flipped for negative guidance. The guidance term is the masked, scaled direction. The terms of all prompts are summed and added to the unconditional estimate. Setting the guidance to zero reproduces the input. With one prompt and a uniform mask, the term reduces to standard CFG §3.1 §3.2.

Multi-edit behaviour depends on each concept having its own mask, threshold and scale, so that the terms remain largely isolated §3.2, Component 3.

Component 3: implicit semantic masks §3.2

• M1 (cross-attention). Take the cross-attention maps of the smallest resolution (16x16 for SD), average over heads and layers, and sum over the tokens of the edit prompt. Upsample to the latent size, and set M1 = 1 where the map is at least the λ-th percentile §3.2. The same U-Net pass that produces the edit estimate supplies these maps §3.2.
• M2 (noise difference). Set M2 = 1 where |ψ| is at least the λ-th percentile §3.2.
• Effective mask: φ = s_e · M1 · M2, with the scale s_e §3.2.

The paper says λ should be small for global edits such as style transfer and roughly proportional to the object's prominence for localised edits §3.2.

# One denoising step with K edit prompts  [§3.2, Components 2 and 3]
def guided_noise(eps_uncond, eps_edits, attn_maps, cfg):
    guidance = zeros_like(eps_uncond)
    for i in range(K):
        # Direction towards (or away from) concept i  [§3.2]
        psi = eps_edits[i] - eps_uncond                    # (B, 4, h, w)
        if cfg.reverse[i]:
            psi = -psi                                     # [§3.2]
        m1 = attention_mask(attn_maps[i], cfg.lam[i])      # [§3.2]
        m2 = percentile_mask(abs(psi), cfg.lam[i])         # [§3.2]
        guidance = guidance + cfg.scale[i] * m1 * m2 * psi # [§3.2]
    return eps_uncond + guidance                           # [§3.2]

# Cross-attention mask M1 for one edit prompt  [§3.2, Component 3]
def attention_mask(attn_maps, lam):
    # Average heads and layers, sum the edit-prompt tokens  [§3.2]
    attn = sum_over_tokens(mean_over_heads_and_layers(attn_maps))
    attn = upsample(attn, latent_size)                     # (h, w) [§3.2]
    return (attn >= percentile(attn, lam)).float()         # [§3.2]

# Noise-difference mask M2  [§3.2, Component 3]
def percentile_mask(psi_abs, lam):
    # psi_abs keeps the latent's shape; threshold per element  [§3.2]
    return (psi_abs >= percentile(psi_abs, lam)).float()   # [§3.2]

Sampling with the edit

The paper states the guided denoising step and the inversion. It does not spell out, in the sampling section, that the stored noise maps z_t are reused as the stochastic noise during generation. The code does exactly this, so an implementation should do the same pipeline_stable_diffusion_ledits.py.

# Generation after inversion  [§3.2] [code: pipeline_stable_diffusion_ledits.py#L967]
def edit_image(x_start, zs, timesteps, edit_prompts, cfg):
    x_t = x_start                                          # from inversion
    for k, t in enumerate(timesteps):
        # U-Net on the unconditional and each edit prompt, batch 1 + K
        eps_uncond, eps_edits, attn = unet_all(x_t, t, edit_prompts)
        eps = guided_noise(eps_uncond, eps_edits, attn, cfg)  # [§3.2]
        x_t = solver_step(eps, t, x_t, zs[k])              # reuse z_t [§3.2]
    return decode(x_t)                                     # VAE decoder

Hyperparameters

Paper values, then code defaults. "not stated" means the paper gives no value.

Results

Author-reported. Setup: CelebA multi-attribute edits and a revised benchmark, TEdBench++, both with SD 1.5 or SD-XL §6 app. C app. E.

Qualitative and figure-only results with no table values in the paper:

• CelebA, three simultaneous attributes: this paper and edit-friendly DDPM are the only methods that reliably reach an average CLIP score above 0.25 §6.1. The paper's plot of CLIP against LPIPS places this method closest to the ideal corner §6.1. Pix2Pix-Zero and Imagic often fail to perform all three edits or change the input strongly §6.1.
• COCO panoptic masks: the intersection of cross-attention and noise masks beats each part alone and is close to CLIPSeg; no numbers are given in text §5 (mask figure, described in text).
• Ablation without grounding keeps CLIP alignment but raises LPIPS app. G.

Paper vs code

Code: ml-research/ledits_pp at commit c2ccffb9d65d39da6f696676d85d09b62fdc8bd5, MIT. The repository is owned by the ml-research organisation, its README links this paper's arXiv id and names the authors, and the first commit is by the first author README.md. The code is a set of diffusers pipelines in src/leditspp/ [code: src/leditspp/init.py].

• Inversion step h. Paper: h_t is a difference of ratios of logarithms ln(sqrt(ᾱ))/ln(sqrt(1-ᾱ)) §3.2. Code: λ_t = log α_t − log σ_t and h = λ_t − λ_s scheduling_dpmsolver_multistep_inject.py [code: pipeline_stable_diffusion_ledits.py#L1166-L1184]. Implement the code's log-SNR form, which is the standard DPM-Solver quantity.
• Threshold meaning. Paper text says λ is the percentage of pixels selected §3.2. The defining equations select pixels above the λ-th percentile, so the selected fraction is 1 − λ §3.2. Code keeps values at or above quantile(edit_threshold) pipeline_stable_diffusion_ledits.py pipeline_stable_diffusion_ledits.py. The code agrees with the equations, and the paper's grid values (0.7 to 0.9) only make sense that way.
• M1 ordering. Paper upsamples the attention map and then takes the percentile §3.2. Code takes the percentile on the 16x16 map and then interpolates the binary mask pipeline_stable_diffusion_ledits.py. Also the code applies a 3x3 Gaussian smoothing (sigma 0.5) to the map, which the paper's equations do not mention pipeline_stable_diffusion_ledits.py pipeline_stable_diffusion_ledits.py.
• M2 per element. Paper uses |ψ| element by element §3.2. Code sums |ψ| over channels, thresholds the sum, and applies that mask to all four latent channels pipeline_stable_diffusion_ledits.py.
• Mask combination. The paper's default is the intersection M1 · M2 §3.2. The library defaults turn off both mask options and use M2 alone pipeline_stable_diffusion_ledits.py [code: pipeline_stable_diffusion_ledits.py#L497]. The notebook enables use_intersect_mask=True LEdits.ipynb.
• Negative weights. Code zeroes negative concept weights before summing pipeline_stable_diffusion_ledits.py. The paper's sum has no such step §3.2.
• Warm-up and cooldown. Code has warm-up and cooldown steps, default 0 for SD and 10 for SDXL pipeline_stable_diffusion_ledits.py pipeline_stable_diffusion_xl_ledits.py. The paper does not describe either.
• Scheduler. The class default is dpmsolver++, but the inversion path only supports sde-dpmsolver++ with order 2 and raises otherwise scheduling_dpmsolver_multistep_inject.py pipeline_stable_diffusion_ledits.py. The notebook sets the SDE variant explicitly.
• First solver step. The code uses a first-order update for the first step and second-order afterwards pipeline_stable_diffusion_ledits.py. The paper does not state the first-step rule.
• Inversion guidance. The paper does not state inversion guidance. Code uses the empty prompt with no CFG unless a source prompt is given, and then uses scale 3.5 pipeline_stable_diffusion_ledits.py.
• Noise reuse. Code feeds the inverted noise maps back in as generation noise pipeline_stable_diffusion_ledits.py. The paper's sampling section does not say so explicitly.
• Other pipelines. The repository also contains a DeepFloyd IF pipeline pipeline_if_ledits.py. The paper does not mention it.

No discrepancy in the basic guidance form: the direction ψ, the sign flip for reverse editing, and the sum over edit prompts match §3.2 [code: pipeline_stable_diffusion_ledits.py#L838-L843] [code: pipeline_stable_diffusion_ledits.py#L952-L960].

Pitfalls

• The solver is the core of "perfect inversion". Use the code's log-SNR definition of h, not the paper's displayed ratio-of-logs formula. Implement only the sde-dpmsolver++ second-order branch with the inversion, or the code raises pipeline_stable_diffusion_ledits.py.
• Reconstruction is exact only when the same noise maps and timesteps are used for regeneration without edits. The paper's 0.00 RMSE is rounded tab. 1 app. C.1.1.
• Cost scales with the number of edit prompts. The U-Net batch grows to 1 plus K per step, so K edits cost about K+1 U-Net passes per step [code: pipeline_stable_diffusion_ledits.py#L742]. The paper's timing numbers are for a single edit prompt app. C.1.2.
• Memory. Code copies the per-step unconditional estimates, edit estimates and masks to CPU memory pipeline_stable_diffusion_ledits.py pipeline_stable_diffusion_ledits.py. For many steps and edits this is large. The paper gives no memory figures.
• Hand selection. The TEdBench++ user-study images were chosen by hand from a grid of 72 candidates per sample and three seeds app. E. Success rates are therefore not a fixed-setting result. The authors note that edit success and quality depend on the seed app. F.
• Model dependence. Some edits fail on a weaker base model, for example a "sitting giraffe" with SD 1.5 app. F. Choose the base model with care.
• Identity. A generic target word such as "cat" can change object identity; the authors suggest more descriptive prompts or Textual Inversion app. F.
• Threshold choice. The paper gives no rule for λ beyond "style transfer uses small λ; object edits use λ near the object's prominence" §3.2, and the mask study approximates λ from box size without a formula §5. Use the grid.
• Sign and direction. Negative guidance uses the same term with the sign flipped §3.2. The paper's removal and replacement examples are shown in figures and not described in the method text §4.
• The user-study preference result (60%) is one the authors themselves attribute partly to study design app. E.

Lineage

• Builds on: Edit-friendly DDPM inversion, which gives perfect inversion for DDPM sampling. This paper replaces DDPM with a second-order SDE DPM-Solver++ to get the same guarantee in fewer steps §3.2.
• Builds on the noise-based semantic guidance family (SEGA, Brack et al. 2023) and on cross-attention masking (Prompt-to-Prompt, Pix2Pix-Zero, MasaCtrl) §2 §5. None of these have cards in this wiki yet.
• Compared against: SDEdit, DDIM inversion, Imagic, Pix2Pix-Zero, DiffEdit, and edit-friendly DDPM §4 tab. 1; Imagic with Imagen on TEdBench §6.2.
• Superseded by: not known from content in this wiki.