An Edit Friendly DDPM Noise Space: Inversion and Manipulations

In plain words

Diffusion image generators start from random static and clean it up step by step. The random draws made along the way act like a hidden code for the picture. The trouble is that this code is awkward to edit. If you keep the same draws and change the text prompt, the layout of the picture usually changes too.

This paper builds a different hidden code for a given photo. It works out which random draws, arranged in a particular way, would reproduce that photo exactly. Those draws are easier to work with: nudging them sideways shifts the picture, and changing a region of them changes its colour. Keeping the draws fixed while changing the prompt keeps the layout and changes the content. Because the generator is still random, running the same edit again gives somewhat different results.

TL;DR

Edit-friendly DDPM inversion takes a real image and returns a sequence of noise maps that reproduce it exactly, with the maps placed so that they carry the image structure. Fixing these maps while changing the text prompt gives prompt edits that keep the layout. The authors report that this preserves structure better than DDIM inversion and gives diverse outputs, which DDIM inversion does not. The results are author-reported and use Stable Diffusion v1-4 with 100 steps. The official code is MIT-licensed.

Problem

• In DDPM the noise maps that generate an image are a latent code for it, but this native code is not edit-friendly. Fixing the generated maps and changing the prompt does not preserve the image structure §3 fig. 4 fig. 7.
• Most diffusion editing methods rely on DDIM, which maps one noise map to one image deterministically. This leaves no way to get diverse edits of the same inverted image §3.1 §2.
• A previous DDPM-based inversion that builds its auxiliary images from the DDPM forward chain gives maps with the native statistics, and so inherits the same editing problem §3.1 fig. 4 app. C.
• Exact DDIM inversion methods are said to roughly double the computation of the diffusion process §2.

Method

Background. The forward DDPM process adds Gaussian noise to a clean image x_0 in T steps §3, eq. 1. Its closed form, with the cumulative product abar_t, is §3, eq. 2. In that closed form the noise vectors at different steps are not independent of each other §3, after eq. 2. The reverse sampler at each step takes a mean from a noise prediction, then adds sigma_t z_t, where z_t is a standard normal draw §3, eq. 3. The mean is written in terms of the predicted clean image and the predicted noise §3, eq. 4. The variance schedule parameter eta lies in [0, 1]; eta = 1 is DDPM and eta = 0 is DDIM. The method is stated to work for any eta in (0, 1] §3.

Step 1: maps from the reverse step. Given any sequence of images x_0, ..., x_T, where x_0 is the real image, each noise map can be solved from the reverse step as z_t = (x_{t-1} - mu_t(x_t)) / sigma_t §3.1, eq. 5. The paper notes that z_1 is commonly set to zero in DDPM §3.1, footnote. The sequence must be built carefully, because arbitrary images lie away from the training distribution of the noise predictor §3.1.

Step 2: auxiliary chain. The proposed chain noises the real image directly, with fresh draws at every step §3.1, eq. 6: x_t = sqrt(abar_t) x_0 + sqrt(1 - abar_t) eps~_t, with the eps~_t statistically independent across t. The draws in the usual closed form are highly correlated from step to step, so this chain is a different process. Consecutive states are farther apart, which makes each extracted z_t have a larger variance than in regular sampling §3.1.

Step 3: extraction. Algorithm 1 builds the chain alg. 1, lines 1-4, then walks back from T to 1, computing each z_t with eq. 5 and resetting x_{t-1} to mu_t(x_t) + sigma_t z_t to limit numerical error build-up alg. 1, lines 6-8. The output is x_T and the maps z_T, ..., z_1 alg. 1. The method reconstructs the image up to machine precision when the drift correction is applied §3.1. Any conditional variant (classifier-free, guided, text) works by using the matching mu_t §3.1.

# Edit-friendly DDPM inversion  [alg. 1] [eq. 4] [eq. 5] [eq. 6]
# Input: real latent x0 (B, 4, 64, 64); Output: chain aux, maps zs
def edit_friendly_inversion(x0, prompt, guidance, num_steps):
    # Noise x0 at every step with fresh, mutually independent draws
    # [alg. 1 lines 1-4] [eq. 6]
    aux = [x0]
    for t in 1 to num_steps:
        eps_tilde = normal(0, I)  # same shape as x0
        aux.append(sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps_tilde)
    # Walk back from T to 1; solve for the map that takes x_t to aux[t-1]
    # [alg. 1 lines 6-8] [eq. 5]
    zs = [None] * (num_steps + 1)
    x_t = aux[num_steps]
    for t in num_steps down to 1:
        # Predicted mean, with classifier-free guidance  [eq. 4] [tab. S1]
        mu = mu_hat(x_t, t, prompt, guidance)
        zs[t] = (aux[t - 1] - mu) / sigma[t]  # [eq. 5]
        # Reset the chain to mu + sigma_t z_t, which equals aux[t-1] in
        # exact arithmetic; this only removes floating-point drift [alg. 1]
        x_t = mu + sigma[t] * zs[t]
    return aux, zs

Step 4: text-guided editing. Invert the real image once with the source prompt, keep the maps fixed, then sample with the target prompt, starting at step T - T_skip from the stored chain value §4 §5. The guidance strength for the target is 15 and T_skip is 36 in the reported runs §5.

# Text-guided editing with fixed maps  [§4] [§5] [eq. 3] [eq. 4]
# Inversion uses the source prompt; sampling uses the target prompt
def edit_image(x0, source_prompt, target_prompt, num_steps, skip):
    # Invert once with the source prompt and guidance 3.5  [tab. S1]
    aux, zs = edit_friendly_inversion(x0, source_prompt, 3.5, num_steps)
    # Resume from the stored chain value at step T - skip  [§5]
    x_t = aux[num_steps - skip]
    for t in num_steps - skip down to 1:
        # Guidance toward the target prompt, strength 15  [§5]
        eps = eps_uncond + 15 * (eps_target - eps_uncond)
        # One sampling step with the fixed map zs[t], eta = 1  [eq. 3]
        x_t = ddpm_step(x_t, eps, zs[t], eta=1)
    return decode_latent(x_t)

Step 5: properties. In a 2D toy model with 40 steps, the new extracted maps have larger spread, and consecutive maps are negatively correlated. The angle histogram peaks at 180 degrees §3.2 fig. 5. The same holds for an unconditional ImageNet model with 100 steps, computed over 10 images §3.2 fig. 6. The paper argues this stronger correlation with the input image is what makes the maps editable §3.2.

Step 6: manipulations.

• Shifting. Shift all T+1 maps by d pixels, after inserting d new columns or rows at the edge. The inserted values are copied from a fixed chunk of the same noise map (indices 50 to 50+d-1), which worked better than white Gaussian fill §3.2 app. A. The authors report a small loss of structure for shifts up to 16 pixels, against a complete loss for the native maps §3.2 fig. 7 fig. S1.
• Colour. For a binary mask B and target colours M, modify each map as z_t' = z_t + s B * (M - P(f_t(x_t))) over a range of steps [T_1, T_2], where s sets the strength §3.2, eq. 7 fig. 8. The example uses s = 0.05 with the range 20 to 70 on an unconditional ImageNet model with 100 steps fig. 8.

# Colour edit on the extracted maps  [§3.2] [eq. 7]
# Applied for timesteps t in [T_1, T_2]; B is a binary mask, M target colours
def colour_edit(zs, x_t_states, mask, target, strength, t_range):
    # Only steps in the range change the maps  [§3.2] [fig. 8]
    for t in t_range:
        # Difference between target colours and predicted clean image
        # inside the mask  [eq. 7]
        gap = target - predicted_x0(x_t_states[t], t)
        zs[t] = zs[t] + strength * mask * gap
    return zs

Combinations with other editors. The maps can replace the DDIM inversion inside prompt-to-prompt. In that case the cross-attention replacement fraction is lowered from 0.8 to 0.6 and the self-attention fraction is 0.2 app. E tab. S1. The maps can also replace the inversion in Zero-Shot image-to-image translation, with the cross-attention guidance weight lowered from 0.1 to 0.03 app. F.2 tab. S3.

Hyperparameters

Values used for the paper's own method and the core settings. The comparison hyperparameters for every baseline are in the tables below.

Table S1: comparison hyperparameters on the ImageNet-R-TI2I set tab. S1. Strength is the classifier-free scale. The inversion-stage strength is 3.5 for every method except PnP and CycleDiffusion, which use 1 tab. S1 caption.

Table S3: hyperparameters on the zero-shot I2IT set tab. S3. The inversion strength is 1 for Zero-Shot and 3.5 for Zero-Shot + Our tab. S3 caption. With our inversion, T_skip is 0 and the source prompt is empty app. F.2.

Results

Author-reported. Setup: modified ImageNet-R-TI2I set (main text: 48 images and 212 image-text pairs; see Paper vs code), Stable Diffusion v1-4, 100 steps, timed on an RTX A6000 §5 app. F. The authors ran every baseline with fixed parameters for all images, not per-image tuned ones as in their original papers §5. Metrics: CLIP similarity (higher is better), LPIPS to the input (lower is better), diversity (higher is better) and time in seconds §5 app. F.

Table S2: ImageNet-R-TI2I tab. S2

• Headline: our inversion gives the highest CLIP score in this table (0.32) with LPIPS 0.29 and a 36 s edit tab. S2 §5.
• CycleDiffusion with eta = 1 has a slightly lower LPIPS (0.26) but a lower CLIP score (0.30) tab. S2. The main text says CycleDiffusion struggles to apply a strong edit while keeping structure §5.
• The CLIP-LPIPS trade-off curves fig. 10 use three parameter settings for each method. Our inversion: (strength, T_skip) = (15, 36), (12, 36), (9, 36). P2P + Our: (7.5, 8), (7.5, 12), (9, 20). CycleDiffusion: (3, 30), (4, 25), (4, 15). DDIM inversion mid-way: T_skip 20, 40, 60 at guidance 9 §5 fig. 10 fig. S8.
• Diversity: 8 outputs per image and source prompt, averaged over the 28 pairs app. F. Our inversion scores 0.18 tab. S2.

Table S4: modified zero-shot I2IT set (60 pairs) tab. S4

• With the maps, Zero-Shot keeps the same CLIP accuracy and lowers LPIPS from 0.35 to 0.27 tab. S4 fig. 11. The authors say fine textures such as fur and flowers are retained §5 fig. 11.

Other numbers reported in the text: integrating our maps into P2P moves LPIPS from 0.61 to 0.25 with CLIP at 0.30 and 0.31 tab. S2.

Paper vs code

Code: official repository inbarhub/DDPM_inversion, MIT licence, at commit 58fc881772d34f0c24be9e34725d213731aa009d. The paper's project page links this repository, and the owner name matches the first author's project page [abstract] [project page]. The repo covers the text-editing pipeline only. It has the inversion, the reverse sampler, and P2P integration. It has no code for shifting, colour edits, the statistics figures, or the Zero-Shot I2IT baseline.

• Defaults differ from the paper's per-method settings. main_run.py sets the editing guidance to 15 main_run.py and T_skip to 36 main_run.py for every mode, including ddim. The paper's table gives DDIM inversion strength 9 and T_skip 0 tab. S1. Pass --cfg_tar=9 --skip=0 to match. The README gives skip 12 and guidance 9 for p2pinv [README].
• Inversion guidance. Inversion uses classifier-free guidance with the source prompt, default 3.5 main_run.py [code: ddm_inversion/inversion_utils.py#L146]. Editing applies guidance 15 to the target prompt in sampling inversion_utils.py.
• z_1 is zeroed in code. After extraction, the code sets the first map (the final step, the paper's z_1) to zero [code: ddm_inversion/inversion_utils.py#L175-L176]. The paper says this is common practice §3.1, footnote.
• Sigma form. The paper's text gives sigma_t = eta beta_t (1 - abar_{t-1}) / (1 - abar_t) §3, after eq. 3. The code uses the standard DDIM form: variance = beta_t (1 - abar_{t-1}) / (1 - abar_t) [code: ddm_inversion/inversion_utils.py#L92-L99], with std = eta sqrt(variance) inversion_utils.py. The paper's printed form lacks the square root. The direction term in the code subtracts eta * variance, not eta^2 * variance inversion_utils.py. At eta = 1 both forms agree, which is all the paper reports.
• Latent encoding. The code uses the VAE posterior mode, scaled by 0.18215 main_run.py. The paper does not state how the latent is obtained.
• Scheduler. The sampler is a DDIM scheduler built from the SD v1-4 checkpoint config, with 100 steps main_run.py. Its beta schedule and offsets are not stated in the paper.
• Seed. The seed for the auxiliary draws is commented out [code: ddm_inversion/inversion_utils.py#L36]. Runs are not reproducible unless the user sets a seed. This is the source of diversity, as the paper describes.
• Empty source prompt. When the source prompt is empty, the code skips guidance and uses the unconditional prediction only [code: ddm_inversion/inversion_utils.py#L141-L148]. Zero-Shot + Our uses this setting app. F.2.
• Shifting and colour. Not in the repository main_run.py. Those results cannot be reproduced from the released code.
• Dataset. The repository ships two example images and a two-entry test.yaml test.yaml. The full evaluation sets, which the paper says will be published app. F, are not in the repository.

Pitfalls

• Numerical drift grows with guidance. Reconstruction is exact only when the prompt, T_skip and strength are the same in inversion and sampling [app. B]. Without the drift correction, PSNR falls below 17 dB at strength 30 [app. B] fig. S2. For editing at strength 15 the paper says the correction is not needed, with PSNR 67.4 dB between edits with and without it app. B. Keep the correction in place anyway.
• Sigma and direction term. Implement the code's form (see Paper vs code). The printed formula, read literally, would give a different direction term.
• The maps are not Gaussian. Their variance is larger than in regular sampling and they are negatively correlated across steps §3.2 fig. 5 fig. 6. Do not replace them with fresh Gaussian noise, because the edits depend on their structure. The shift result also depends on the copied columns and rows app. A.
• Shifts depend on the fill. Filling the new columns and rows from a white Gaussian with the same mean and variance was worse than copying a chunk of the same map app. A. The paper does not report a number for the random-fill variant.
• The colour example uses one setting. s = 0.05 and the [20, 70] range come from one figure fig. 8. Nothing else is swept, and the code does not include the operation. The caption writes the range as 70 to 20, while the text writes [T_1, T_2].
• Cost is about double per step. Both the inversion and the sampler call the UNet twice per step, unconditional and conditional [code: ddm_inversion/inversion_utils.py#L140-L142] [code: ddm_inversion/inversion_utils.py#L243-L250]. Inversion stores the whole chain and every map: T + 1 latents plus T maps inversion_utils.py inversion_utils.py.
• The eta range is not tested. The method is said to work for any eta in (0, 1] §3. Only eta = 1 appears in the paper's experiments §5.
• Diversity comes from fresh draws. Results differ between runs [code: ddm_inversion/inversion_utils.py#L36]. Report the spread across several seeds, not one sample.
• Baselines use fixed parameters. All baselines are run with one fixed parameter set §5 app. F. These numbers therefore differ from the baselines' own papers.
• Dataset size disagrees. The main text says the ImageNet-R-TI2I set has 48 images and 212 pairs §5. Appendix F.1 says it contains 44 images (30 from PnP and 14 added) app. F.1. The card does not resolve which is correct; check the released data before quoting either count.
• Timing depends on hardware. The 36 s and 46 s times are on one RTX A6000 app. F.

Lineage

• Builds on: the DDPM sampler [Ho20], the DDIM sampler and its inversion [Song21], latent diffusion / Stable Diffusion [Rombach22], and classifier-free guidance [Ho21]. None of these have cards on this wiki.
• Compared against: CycleDiffusion [Wu22], Plug-and-Play [Narek22], EDICT [Bram22], null-text inversion [Mokady22], prompt-to-prompt [Hertz22], and Zero-Shot image-to-image translation [Parmar23]. None of these have cards on this wiki.
• Superseded by: not stated in this card's sources.

This card was compiled by an AI model from the paper and the official code listed above. It is not endorsed by the authors. Changes: pseudocode, tables and commentary written for wikillm; equations and tables copied with citation; no figures reproduced. Review status: draft.