DDIM inversion

Overview

A diffusion model normally maps noise to an image. DDIM inversion runs the deterministic DDIM sampler in the other direction: it starts from a real image, encodes it, and walks toward noise to get a latent x_T that a sampler could map back to (approximately) that image 2211.12572 §4 2211.12572 app. C. The useful property is that x_T is a concrete starting point. If you sample from the same x_T with a different prompt, the output keeps some of the layout of the original while the text changes its content 2211.12572 §4.

Three reuse-licensed cards cover this concept so far. They differ in the conditions used during inversion: Plug-and-Play inverts with an empty prompt, Composer inverts with the image's own set of conditions [arxiv:2302.09778 §3.2], and StyleAligned inverts a single style reference under a caption 2312.02133 §4.3. All three rely on sampling from the inverted latent of the image that is being edited or used as a style source.

The invariant that Plug-and-Play relies on is that the translated image must start from the inverted noise of the guidance image. With a different seed, the injected features no longer match the encoder skip features and the method fails 2211.12572 app. B 2211.12572 fig. 14.

Variants

Empty-prompt inversion with 1000 steps (Plug-and-Play)

The guidance image is encoded with the autoencoder and inverted with the empty prompt and a CFG scale of 1 2211.12572 §5.1. The paper uses 1000 inversion steps and 1000 reconstruction steps for real images [arxiv:2211.12572 app. C]. The guidance trajectory is then re-run deterministically, and its internal features are saved at the 50 timesteps used for translation 2211.12572 app. C. Card: Plug-and-Play.

Inverting with the empty prompt is the design choice the authors give for why any prompt and CFG scale can be used at sampling time. They attribute the weaker results of Prompt-to-Prompt on real images to inverting with a source text 2211.12572 §5.1. Inverting with the target or source prompt is therefore not what this variant does.

Inversion of generated guidance images

For guidance images the model generated itself, the paper's guidance features come from the generating prompt P_G instead of the empty prompt, with a CFG scale of 7.5 2211.12572 §4 2211.12572 app. C. The paper does not state whether the latent is still obtained by DDIM inversion in this case or taken directly from the generation run.

Inversion under the image's own conditions (Composer)

Composer inverts an image to a latent with DDIM inversion conditioned on a set of the image's representations, c_i. It then runs DDIM sampling from that latent under a modified set, c_j 2302.09778 §3.2. The same procedure serves reconfiguration, where one aspect of the image changes and the rest is kept 2302.09778 §3.2 fig. 3. Card: Composer.

The paper states neither the number of inversion steps nor the guidance scale used during inversion 2302.09778 §3.2 2302.09778 §3.5. The reconfiguration results are shown only as figures, without metrics 2302.09778 fig. 3.

Inversion of one style reference under a caption (StyleAligned)

StyleAligned uses a real image as the style source. It inverts that image with DDIM inversion, conditioned on a text caption produced with BLIP, to recover a noise trajectory from x_T to x_0. New images are then generated while the reference follows that trajectory 2312.02133 §4.3. Card: StyleAligned.

The paper states no inversion step count and no inversion guidance scale 2312.02133 §4.3. The official notebook uses 50 steps with guidance 2, which the card lists in its paper-vs-code section, so these values come from the code and not from the paper StyleAligned card. The paper says that inversion sometimes fails or gives an erroneous trajectory, and it describes no fallback 2312.02133 §4.3.

When to use which

The Plug-and-Play paper compares its full method against SDEdit, Prompt-to-Prompt, DiffuseIT and VQGAN-CLIP, but reports its results as plotted curves without numeric tables for the main comparison 2211.12572 §5.1. Composer's comparison is on COCO text-to-image generation, not on editing 2302.09778 §3.5. StyleAligned's quantitative table compares style consistency for text prompts against personalisation and editing baselines 2312.02133 tab. 2. The table above therefore reflects what the papers say, not a benchmark of inversion alone.

Implementation sketch

The Plug-and-Play steps, expressed as an inversion loop. Each inversion step runs the U-Net on the current latent with the empty prompt and moves the latent one step toward noise, using the deterministic DDIM update in reverse. The exact update rule and its scaling are standard DDIM and are not restated in the paper's main text.

# DDIM inversion of a guidance image with an empty prompt
# [arxiv:2211.12572 §4] [arxiv:2211.12572 app. C]
# Input:  image I_G, number of inversion steps N (paper: 1000)
# Output: latent x_T used as the shared starting noise

def ddim_invert(I_G, N):
    # Encode the image into the model's latent space  [arxiv:2211.12572 §4]
    x = encode(I_G)
    # Walk from the clean latent toward noise, one step at a time  [arxiv:2211.12572 app. C]
    for t in ddim_timesteps_ascending(N):
        # Predict noise with the empty prompt, CFG scale 1  [arxiv:2211.12572 §5.1]
        eps = unet(x, t, prompt="")
        x = ddim_step_inverse(x, eps, t)
    return x  # the same x_T is the start of the translated sample  [arxiv:2211.12572 app. B]

Notes on where variants differ:

• Step count. The paper's 1000 steps are what it states; its shipped code uses 999 for the same schedule. This mismatch is noted on the Plug-and-Play card, not here. Composer states no inversion step count. StyleAligned states none either; its notebook uses 50 StyleAligned card.
• Conditions for inversion. Empty prompt in the Plug-and-Play real-image path. Composer inverts under the image's own condition set c_i instead 2302.09778 §3.2. StyleAligned conditions on a BLIP caption 2312.02133 §4.3. A non-empty prompt changes the latent that comes out of inversion.
• Sampling afterwards. Plug-and-Play uses the same x_T with deterministic DDIM sampling (eta = 0) 2211.12572 app. C. For Composer's sampler, see Contradictions.

Contradictions

The papers invert with different conditioning, and Composer's sampler is described inconsistently. No paper tests another's choice, so this page does not settle either point.

• Conditioning for inversion. Plug-and-Play inverts with an empty prompt and attributes weaker real-image results to inverting with a source text 2211.12572 §5.1. Composer inverts under the image's own conditions, one of which can be a caption 2302.09778 §2.2 2302.09778 §3.2. StyleAligned inverts under a BLIP caption 2312.02133 §4.3. These are different setups for different tasks, not conflicting numbers.
• Sampler after inversion. Composer's text describes DDIM sampling from the inverted latent 2302.09778 §3.2. Its hyperparameter table caption says DPM-Solver++ is the sampler for all diffusion models 2302.09778 tab. 1. The paper does not say which one the reconfiguration step uses.

Papers

• Plug-and-Play (Tumanyan et al., text-driven image-to-image translation): uses DDIM inversion with an empty prompt as the first step of zero-shot, structure-preserving image editing on frozen Stable Diffusion v1.4.
• Composer (Huang et al., controllable image synthesis from composable conditions): inverts an image under its own condition set and resamples under a modified set, for reconfiguration.
• StyleAligned (Hertz et al., style-aligned image generation via shared attention): inverts one style reference under a BLIP caption, then shares attention with it to generate a style-consistent set on SDXL.