MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

In plain words

A text-to-image model is trained on square pictures, so it does not know how to draw a wide panorama, and it gives little control over where things go in the frame. Retraining it for each new kind of control is slow and expensive.

The authors' idea is to leave the model alone and change how it is run. The image is split into several overlapping windows. At each step of the drawing process, the model proposes how each window should look. Where windows overlap, those proposals disagree, so the method takes a per-pixel average and uses that as the single next step for the whole picture. The result is one image that the model considers plausible in every window.

The same mechanism covers a second kind of control. Instead of fixed windows, each region of the picture gets its own mask and text prompt, and the averaging is weighted by those masks. A rough box with a label can then steer where an object appears. The authors show both uses with the same procedure and no extra training.

TL;DR

MultiDiffusion treats a pretrained Stable Diffusion model as a "reference" for local patches. At each denoising step it runs the reference on every crop or masked region, and sets the next whole-image latent to the least-squares fit of those outputs, which for these maps is a weighted per-pixel average. This gives wide panoramas and region-guided images without training. Results are author-reported, against Blended Latent Diffusion and Stable Inpainting.

Problem

A diffusion model trained at one fixed resolution and aspect ratio does not directly produce images at other shapes such as a wide panorama §1. Prior adaptations are task-specific, and controlling content through spatial inputs usually means training or finetuning a model for that input type §1 §2. The paper asks whether one fixed, pretrained text-to-image model can be used for several controlled tasks without training §1.

Naive per-crop sampling fails: independently denoised crops of one panorama are unrelated samples and do not join up §4.1 fig. 3.

Method

Setup. The reference model maps a noisy image and a condition to a less-noisy image, one denoising step at a time, from pure noise to a clean sample §3 eq. 1. The target canvas J lives in a new image space, while the reference works on crops of the usual size. Each target-to-reference mapping F_i takes a crop (or a region) of J; each condition map λ_i gives the prompt for that crop §3.

Objective. The next canvas is defined as the minimiser of a weighted least-squares fit between each crop of the canvas and the reference's denoised version of that crop, with per-pixel weights W_i §3 eq. 3 eq. 4. The paper calls this the "FTD" loss §3 eq. 4.

Closed form. When each F_i copies pixels (a crop or a mask-selected region), the minimiser is available analytically. Each canvas pixel is the W-weighted average of all reference outputs that cover it §3 eq. 5. The sampler therefore runs the reference on every view, scatters the results back to canvas coordinates, and divides by the summed weights alg. 1.

# MultiDiffusion sampling  [alg. 1] [eq. 5]
# Input:  reference model phi; views F_i with prompts y_i and weights W_i
# Output: canvas J_0 of shape (H', W', C)

def multidiffusion_sample(phi, views, prompts, weights, timesteps):
    # Start from Gaussian noise on the whole canvas  [§3] [alg. 1]
    canvas = sample_noise(canvas_shape)
    for t in timesteps:
        # Crop each view and run one reference denoising step  [alg. 1]
        crops = [crop(canvas, F) for F in views]          # F_i(J_t)
        steps = [phi(c, y) for c, y in zip(crops, prompts)]  # Phi(I_t^i)
        # Scatter weighted outputs back and average where views overlap  [eq. 5]
        numerator = zeros_like(canvas)
        denominator = zeros_like(canvas)
        for F, w, s in zip(views, weights, steps):
            numerator = numerator + scatter(F, w * s)     # F_i^{-1}(W_i * Phi)
            denominator = denominator + scatter(F, w)     # F_i^{-1}(W_i)
        # Pixels covered by no view stay zero  [code: panorama.py#L138]
        canvas = safe_divide(numerator, denominator)
    return canvas

Panorama §4.1. Views are fixed-size crops of the canvas with weights W_i = 1, so the closed form is a plain average over overlapping crops eq. 7. The panorama must have height and width at least as large as the reference's crop size §4.1. Crops are 64 x 64 in latent units (512 x 512 pixels) with a stride of 8 latent units (64 pixels) app. B.1 alg. 2.

Region-based generation §4.2. Each region i has a binary mask M_i and a prompt y_i. The views are the full canvas and the weights are the masks, W_i = M_i §4.2 eq. 8. The background is handled as a region whose prompt describes the full image §5.2. For tight masks, a bootstrapping phase is applied in the early steps only §4.2 eq. 9:

# Bootstrapped region view F_i(J_t, t)  [eq. 9] [app. B.2]
# Input:  canvas view (h, w, C); mask M_i (h, w); background S_t (h, w, C)
# Output: view passed to the reference model at step t

def bootstrapped_view(canvas_view, mask, t, t_init, background):
    # Late steps (t <= T_init) keep the full view unchanged  [eq. 9]
    if t <= t_init:
        return canvas_view
    # Early steps keep the mask region; the rest is constant colour  [eq. 9]
    return mask * canvas_view + (1 - mask) * background

Here S_t is a constant-colour image, encoded with the autoencoder and noised to the level of step t app. B.2. The paper sets T_init to 20% of the process, i.e. T_init = 800 §4.2. Its ablation shows that bootstrapping makes the generated object fit the tight mask more closely §4.2 tab. 2.

Reference model. Stable Diffusion, version 2, through the HuggingFace implementation §5. Diffusion runs in latent space of size 64 x 64 x 4 for a 512 x 512 image, and a decoder maps the canvas to pixels §5.

Cost. Each canvas step calls the reference once per view, and the paper notes these calls can be batched app. B.1. The number of views n for a panorama is given by a formula in B.1 (see Pitfalls for how the code computes it).

Hyperparameters

Results

Author-reported. Setup: Stable Diffusion v2 as the reference model §5. Panorama: 512 x 4608 px, FID between 512 x 512 crops and a reference set of independent reference samples, N = 2000 per prompt over 8 prompts; baselines are Blended Latent Diffusion (BLD) and Stable Inpainting (SI) §5.1 tab. 1. Region: IoU of a segmenter's output on generated images against COCO ground-truth masks §5.2 tab. 2. Make-A-Scene and SpaText were compared only qualitatively, using their published examples §5.2.

Panorama tab. 1. Mean ± standard deviation for FID; CLIP scores have no spread in the table.

Region-based generation on a COCO subset tab. 2. IoU, mean ± standard deviation.

Qualitative results (panorama, rough and tight region layouts, COCO comparisons) are shown in figures in §5.1, §5.2 and the appendix app. A.

Paper vs code

The repository is the one linked from the project page; its README cites this paper with the same authors README.md. It has no LICENSE file, so no reuse licence is known; code is cited here, not copied.

• Bootstrapping length. Paper: T_init is 20% of the process, i.e. 800 steps §4.2. Code: the first 20 loop steps are bootstrapped, counted out of num_inference_steps (default 50), i.e. 40% at defaults region_based.py region_based.py. The paper's 800 implies T = 4000, which is not stated.
• Background noise in bootstrapping. Paper: S_t is one constant-colour image, encoded and noised to the level of step t app. B.2. Code: keeps a pool of bootstrapping constant-colour images, draws a random one per view and per step, and noises it with the initial noise slice for that view, which is fixed across steps region_based.py region_based.py region_based.py.
• Bootstrapping applies to foreground only. Paper: eq. 9 applies to every view i §4.2 eq. 9. Code: masking is applied to the foreground prompts only; the background view (index 0) is never masked region_based.py.
• Background region. Paper: region weights are the masks M_i, with no background region defined §4.2. Code: builds the background mask as the complement of the summed foreground masks, clipped at zero, and adds it as an extra region with its own prompt region_based.py.
• Panorama view count. Paper: n = (H' - 64)/step · (W' - 64)/step app. B.1. This is ambiguous about units and omits the +1 of a sliding window. Code: views per axis = floor((L - 64)/8) + 1 on latent length L panorama.py. For the 512 x 4608 evaluation size the latent is 64 x 576, which the code turns into 1 x 65 = 65 views (derived from the code's formula).
• Guidance. Paper: Φ is a diffusion step, with no mention of guidance §3. Code: each reference call uses classifier-free guidance with scale 7.5 and a negative prompt, batched as two passes panorama.py region_based.py.
• Empty coverage. Paper: eq. 5 is undefined where the denominator is zero §3 eq. 5. Code: keeps the zero accumulator where no view covers a pixel, so such pixels come out as zero latent panorama.py region_based.py.
• Batching. Paper: views can be processed in batches app. B.1. Code: views are processed one at a time, with a TODO noting batching is possible panorama.py.

Otherwise the panorama update matches eq. 5 with W_i = 1: accumulate denoised crops, divide by count panorama.py. The region update matches eq. 5 with W_i = M_i region_based.py.

Pitfalls

• Reading eq. 5. F_i^{-1} denotes the adjoint of the view map (scatter back and sum), not a matrix inverse. For crops and masks, implement it as scatter-add into a canvas and divide by the scattered weights, as the code does panorama.py.
• Time direction in eq. 9. t counts down from T to 1, so "t ≤ T_init" means the late steps. The early steps are the masked ones. The code's loop index counts up from zero, so i < bootstrapping marks the same early steps region_based.py. Mixing the two conventions reverses the bootstrapping.
• Step counts vs fractions. The paper gives T_init as a fraction of the process (20%) and as a number (800). The code takes a step count (20). At the code's defaults these differ (40% vs 20%). Pick one convention and state it.
• Guarantee scope. The proposition says a zero loss at every step reproduces the reference's diffusion path §3. The sampler does not minimise a loss; it applies one closed-form average per step, and the paper's claim of convergence to a global optimum refers to that per-step fit §6. It is not a claim about the whole trajectory.
• Model-dependence. The method only works as well as the reference's paths for the given crops §6. The paper's limitation example, a vertical panorama, shows a bias: the reference keeps adding a waterfall to each crop, and a region constraint is used to counter it §6.
• Cost. A panorama step costs one reference call per view, with classifier-free guidance doubling the batch. At code defaults (50 steps, 65 views for the evaluation size) that is 3,250 reference calls on 64 x 64 latents, derived from the code's view formula panorama.py panorama.py. The cost grows with the panorama area.
• Mask resizing. Masks are downsampled to latent size with nearest-neighbour and thresholded at 0.5 region_based.py. Thin structures can vanish at this resolution; the paper does not say how masks are resized.
• Precision. The code runs the sampler inside CUDA autocast (reduced precision) panorama.py region_based.py. Not stated in the paper.
• Claims versus table. The text says the method "outperforms the existing baselines in all metrics" §5.1. In the table, CLIP-score ties with BLD (0.27), and CLIP-aesthetic ties with the reference Stable Diffusion row (6.36) tab. 1. The standard deviations are large relative to the region gains (0.26 ± 0.12 vs 0.18 ± 0.10 without bootstrapping) tab. 2.
• Figure numbering. The appendix refers to "Fig. 4" for the tight-mask example app. B.2, which does not match caption order in the source. Cite the figures by section rather than by number.
• Notation. The source writes the reference as \dif (Φ) and uses Φ for both the one-step output and the model §3 eq. 1. Implement Φ as one reference denoising step.

Lineage

• Builds on: Stable Diffusion, cited as [rombach2022high]; the paper uses its v2 weights and latent space §5. No arXiv id in the paper body; no card on the wiki yet.
• Compared against: Blended Latent Diffusion [avrahami2022blended] and Stable Inpainting [rombach2022high] for panoramas and regions §5.1 §5.2; Make-A-Scene [gafni2022_make_a_scene] and SpaText [avrahami2022spatext] qualitatively for regions §5.2. No arXiv ids in the paper body.
• Superseded by: not known from the wiki.