Diffusion inpainting

Overview

Diffusion inpainting keeps part of a sample tied to known content while the model generates the rest. The six cards use two families. A third design, the restoration paper's, diffuses a compact prior and is covered below as well.

The first family writes known values into the sample at sampling time. The base motion model does this at every sampling step, with no retraining: the known entries of its clean-motion estimate are overwritten with the input motion 2209.14916 §3. The motion paper does this after each denoising step, and its noise on those features is either zeroed or scaled by a mask 2303.01418 alg. 2 2303.01418 §3.1. The robot paper's sampler overwrites the committed action prefix at every step 2512.05964 alg. 1. The motion paper uses this for two jobs: steering a generated motion with a control signal such as a root path or one joint, and joining separately generated segments into one longer motion 2303.01418 §1 2303.01418 §3.1 2303.01418 §3.3. The robot paper uses it to keep the next action chunk continuous with the actions the robot has already committed to while the previous chunk is still running 2512.05964 §1 2512.05964 §4. Its inference-time baseline enforces the prefix with pseudoinverse guidance, which needs a vector-Jacobian product at every denoising step 2512.05964 §4.

The second family trains the model to take the known part as input, so the sampler needs no write-back. The image paper feeds the masked image and the mask in as extra input channels and learns a task prompt for each inpainting job 2312.03594 §3.1 2312.03594 §3.2. The robot paper's training-time variant does the same with a clean action prefix 2512.05964 §4. SmartBrush does the same for the background: its training input keeps the pixels outside the mask clean, and a mask-precision level is an extra input 2212.05034 §4.1 2212.05034 §4.2. At inference the background is set from the input image once, at the start, and not rewritten at each step 2212.05034 §4.1.

The restoration paper has no known-value write-back either. It diffuses a compact prior rather than pixels, and the diffusion model that predicts the prior is conditioned on the damaged image 2303.09472 §4.2 2303.09472 eq. 12. Its inpainting task is one of three restoration tasks, not a separate sampler 2303.09472 §5.2.

The paper's baseline for motion is the inpainting scheme inherited from its base motion model (MDM). The authors report that this baseline does not follow trajectory and end-effector constraints well 2303.01418 §1 2303.01418 §4.3.

Variants

Baseline: inherited MDM inpainting

The editing scheme of the base model, used as the comparison point for control. It needs no retraining. The base paper's in-betweening fixes the first and last 25% of frames and generates the middle 50%; its body-part editing fixes the joints that stay 2209.14916 §5.1. The base paper shows these edits only as figures and does not measure them 2209.14916 fig. 3. The control paper compares against this scheme on HumanML3D control samples 2303.01418 §4.3 2303.01418 tab. 4. Cards: Human Motion Diffusion Model for the mechanics and Human Motion Diffusion as a Generative Prior for the comparison.

Fine-tuned control with zeroed noise

The model is fine-tuned, starting from the same base model, so that the noise on the controlled features is zero during the forward process. In training, the ground-truth control features reach the input, and the model learns to rely on them to reconstruct the other features 2303.01418 §3.3 2303.01418 alg. 1. At sampling time, the given control values are written in after each prediction, and the noise on those features is zeroed again 2303.01418 alg. 2. Reported control results are in 2303.01418 tab. 4: the fine-tuned model has lower FID than baseline inpainting for trajectory (0.54 vs 0.98) and for the left wrist (0.34 vs 0.82) 2303.01418 §4.3.

DiffusionBlending: composing control models

For two models aligned to one motion space but conditioned on different signals, the sample is the first model's output plus a scalar s times the difference to the second model's output 2303.01418 §3.3 2303.01418 eq. 3. The paper describes it as a generalisation of classifier-free guidance 2303.01418 §1. Its composite-control experiments use equal weights, s = 0.5 2303.01418 §4.3. For left wrist plus trajectory, the reported FID is 0.22 against 1.18 for baseline inpainting 2303.01418 tab. 4.

Soft inpainting for joins: DoubleTake

A fixed prior generates several text-prompted intervals in one batch. At every denoising step, the overlapping frames of neighbouring intervals (the handshake) are set to one shared value, a linear blend of the two sides 2303.01418 §3.1 2303.01418 eq. 2. A second take then partially re-noises each join, sandwiched between its two neighbours, and denoises it back. The paper calls this a "soft inpainting scheme": each frame gets a mask value between 0 and 1 that sets how much refinement it receives, with a linear transition over b frames 2303.01418 §3.1. Ablations on HumanML3D report FID 0.60 for the full method, 1.00 for the first take alone and 1.09 for the second take alone 2303.01418 tab. 2 2303.01418 §4.1.

Inference-time prefix inpainting (RTC)

For a robot policy that is a flow-matching model, the prefix of the next action chunk is held to the committed actions while the rest is generated. The robot paper calls the inference-time form real-time chunking (RTC), from arXiv:2506.07339, which it does not reproduce 2512.05964 §1 2512.05964 §4. The guidance step needs backpropagation through the model at every denoising step, which the authors say adds latency 2512.05964 §1. They also say it handles large inference delays poorly 2512.05964 §5.1. Card for the comparison: Training-Time Action Conditioning for Efficient Real-Time Chunking.

Training-time prefix conditioning

The model is trained to see a clean prefix. During training, a delay is sampled per example, and the first steps of the chunk are given as ground-truth actions with flow timestep 1. Each action step has its own flow timestep, which needs per-token time conditioning. The loss covers only the postfix 2512.05964 §4 2512.05964 alg. 1. At run time the committed actions are written into the prefix, with no extra computation and no change to the robot software 2512.05964 §1 2512.05964 alg. 1.

In simulation (Kinetix, horizon 8), the authors report a higher solve rate than inference-time RTC at inference delays 2 to 4, with the gap growing as the delay grows. At delays 0 and 1 it is slightly lower, which they attribute to fewer supervised first actions 2512.05964 §5.1 2512.05964 fig. 3. On two real tasks (box building and espresso) with the pi0.6 model, the authors report success and speed about the same as inference-time RTC. Average end-to-end latency is 108 ms at about delay 5, against 135 ms at about delay 7 for inference-time RTC 2512.05964 §5.2 2512.05964 fig. 5.

Learned task prompts for image inpainting: PowerPaint

One Stable Diffusion-based model handles several inpainting jobs, and a learned task prompt selects the job. The masked image and the mask are extra input channels of the denoiser, so the model learns to fill the hole during training 2312.03594 §3.1 2312.03594 eq. 3. Three prompts are trained: one for filling a random mask from its context, one for inserting a captioned object into a bounding box, and one for an object that follows a segmentation mask 2312.03594 §3.2 2312.03594 eq. 4.

Object removal reuses the context prompt as the positive prompt and the object prompt as the negative prompt in classifier-free guidance 2312.03594 §3.2 2312.03594 eq. 5. The paper argues that a diffusion model otherwise copies content from the surroundings into the hole, which makes removal hard in crowded scenes 2312.03594 §3.2 2312.03594 fig. 3. In a user study on removal, the paper reports 73.2% preference against 15.2% for LaMa and 11.6% for SD-Inpainting 2312.03594 tab. 5.

For shape-guided insertion, the text embeddings of the context and shape prompts are blended with a fitting degree α, so the user sets how closely the object follows the mask 2312.03594 §3.2 2312.03594 eq. 7. α is an area ratio during training and a slider at inference, and the paper's dilation of training masks is in 2312.03594 eq. 6.

On object inpainting with bounding-box and layout masks, the paper reports the lowest Local-FID and FID on both OpenImages and MSCOCO 2312.03594 tab. 1 2312.03594 tab. 2. Card: A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting.

Mask-precision control for named objects: SmartBrush

The model fills a masked region with the object named by a short label or caption. A precision input sets how closely the object follows the mask. Each training instance mask is blurred to one of several precision levels, from the exact mask at the lowest level to its bounding box at the highest 2212.05034 §4.2 2212.05034 eq. 7. The network takes the blurred mask, the level, the prompt and the noisy image 2212.05034 eq. 8. Pixels outside the mask stay clean in the training input, so the model can read the background while it generates the object 2212.05034 §4.1 2212.05034 eq. 5. A second output channel predicts the exact mask, with a DICE loss weighted by λ = 0.01. At inference the coarse mask is used first, and the predicted mask replaces it so that only the predicted object region is denoised 2212.05034 §4.3 2212.05034 §4.4 2212.05034 fig. 7.

Author-reported results: on bounding-box masks, SmartBrush has lower Local FID than the best of four baselines on both datasets, 9.71 against 12.57 for Stable Inpainting on OpenImages and 13.22 against 18.13 on MSCOCO 2212.05034 tab. 1. On object layout masks the gap is similar: 7.82 against 10.98 on OpenImages and 9.80 against 15.16 on MSCOCO 2212.05034 tab. 2. CLIP scores are close to the baselines. On OpenImages bounding boxes SmartBrush scores 0.266, against 0.265 for Blended Diffusion and Stable Diffusion 2212.05034 tab. 1. The background-preservation benefit is shown in figures only 2212.05034 fig. 6. Training used 8 A100 GPUs for about 20K steps; the total batch size across GPUs is not stated 2212.05034 §5.1. The card records no official code Paper vs code. Card: SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model.

Diffusion on a restoration prior: DiffIR

The diffusion model does not work on pixels. A stage-1 encoder maps the ground-truth and the damaged image to a prior of 4C' values, with C' = 64 2303.09472 §4.1 2303.09472 §5.1. A diffusion model with 4 steps predicts that prior from the damaged image alone, and a transformer restorer turns the prior into the output 2303.09472 §4.2 2303.09472 alg. 2. Known pixels are not written back during sampling. The inpainting task uses the LaMa mask generator and its narrow and wide mask settings 2303.09472 app. D 2303.09472 tab. 1.

The paper reports FID and LPIPS on Places at 512 x 512 and CelebA-HQ at 256 x 256. On wide masks it has up to 0.2706 (Places) and 0.5583 (CelebA-HQ) lower FID than LaMa, at similar parameters 2303.09472 §5.2 2303.09472 tab. 1. In its CelebA-HQ ablation, joint training of the prior's diffusion term gives FID 5.14, against 5.98 for traditional training and 5.68 with no diffusion on the prior 2303.09472 tab. 4. Card: DiffIR: Efficient Diffusion Model for Image Restoration.

When to use which

The evidence is six papers in three domains. Human motion (HumanML3D, KIT, BABEL, 3DPW, HumanAct12, UESTC) reports author-reported FID and user studies. Robot action chunking (simulated Kinetix and two real tasks) reports solve rate and success, mostly in figures. Image inpainting (OpenImages, MSCOCO, Places2, Flickr-Scenery) reports FID, CLIP score, LPIPS and user preference. SmartBrush reports Local FID and FID on OpenImages and MSCOCO for the same kind of object insertion. Image restoration (Places, CelebA-HQ, plus super-resolution and deblurring) reports FID, LPIPS, DISTS and PSNR. The motion and robot rows do not transfer to images, and the image rows do not transfer to motion or robot control. Audio and text generation are not covered.

Implementation sketch

Fixed-value control: the core loop of the fine-tuned variant. Lines are shown as pseudocode; the paper's own algorithms are the reference 2303.01418 alg. 1 2303.01418 alg. 2.

# Training step for one control feature  [arxiv:2303.01418 alg. 1]
# Input: clean motion x0, control mask over features
# Output: loss for one gradient step

def finetune_loss(x0, controlled):
    # Sample a diffusion step and Gaussian noise  [arxiv:2303.01418 alg. 1]
    t = uniform_int(1, T)
    eps = normal(0, I)
    # No noise on the controlled features, so they reach the input  [§3.3]
    eps[controlled] = 0
    x_t = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps
    # Predict the clean motion  [§3]
    return norm(x0 - model(x_t, t))

# Sampling with known control values  [arxiv:2303.01418 alg. 2]
# Input: given values for the controlled features
# Output: motion that follows the given values

def sample_with_control(given, controlled, T):
    x0 = zeros_like(given)
    for t in range(T, 0, -1):
        # Write the known values in after each prediction  [alg. 2]
        x0[controlled] = given
        eps = normal(0, I)
        # Zero the noise on the controlled features  [alg. 2]
        eps[controlled] = 0
        x0 = denoise_step(x0, eps, t)
    return x0

Soft join: each frame's mask sets how much of the first take is refined. The mask schedule and the handshake length are in 2303.01418 §4.1.

# Soft-masked join between two neighbouring intervals  [arxiv:2303.01418 §3.1]
# Input: suffix of interval A, prefix of interval B, handshake h, mask m
# Output: refined join frames

def refine_join(suffix, prefix, h, m, T_prime):
    # Blend the overlap into one shared value  [arxiv:2303.01418 eq. 2]
    handshake = linear_blend(suffix[-h:], prefix[:h])
    sandwich = concat(suffix, handshake, prefix)
    # Re-noise partially, then denoise back to t = 0  [§3.1]
    noisy = add_noise(sandwich, T_prime)
    # Each frame's mask value sets its share of refinement  [§3.1]
    return denoise_with_mask(noisy, m, T_prime)

Training-time prefix conditioning: the loss only. Sampling is the same overwrite loop as above, with flow timestep 1 for the prefix 2512.05964 alg. 1.

# Training loss with a simulated inference delay  [arxiv:2512.05964 alg. 1]
# Input: observation, action chunk (B, H, D), max delay
# Output: scalar loss

def prefix_training_loss(model, observation, chunk, max_delay):
    # Sample a simulated delay per example  [arxiv:2512.05964 §4]
    delay = sample_delay(max_delay)                    # (B,)
    # The first `delay` steps are the committed prefix  [§4]
    prefix = arange(H)[None, :] < delay[:, None]       # (B, H)
    # Prefix steps are clean, so their flow timestep is 1  [§4]
    time = where(prefix, 1.0, sample_uniform_01(B))    # (B, H)
    noise = normal(0, I)                               # (B, H, D)
    x_t = time[..., None] * chunk + (1 - time[..., None]) * noise
    # Loss only on the postfix steps  [§4] [alg. 1]
    err = (model(observation, x_t, time) - (chunk - noise)) ** 2
    postfix = (~prefix)[..., None]
    return sum(err * postfix) / (sum(postfix) + 1e-8)

Object removal with a learned context prompt: the guidance step only. The prompts are encoded once, and the weight w is not stated in the paper 2312.03594 eq. 5.

# Guided noise prediction for object removal  [arxiv:2312.03594 eq. 5]
# Input: noisy latent; embeddings of P_ctxt (positive) and P_obj (negative)
# Output: guided noise prediction

def removal_noise(noisy_latent, t, ctxt_emb, obj_emb, w):
    # Predict noise under the context prompt  [§3.2]
    noise_pos = unet(noisy_latent, t, ctxt_emb)
    # Predict noise under the object prompt, used as the negative  [§3.2]
    noise_neg = unet(noisy_latent, t, obj_emb)
    # Weight w on the positive term, 1 - w on the negative  [eq. 5]
    return w * noise_pos + (1 - w) * noise_neg

Contradictions

Within 2303.01418:

• Control fine-tuning length. The paper says the fine-tuning stage "requires less than 20K steps" for visually pleasing results 2303.01418 §3.3, and reports 80K steps for its fine-tuned models 2303.01418 §4.3. Neither side is resolved here.

Within 2312.03594:

• Best in every setting. The text says PowerPaint is state of the art for visual quality and text alignment in object inpainting 2312.03594 §4.1. Its own table shows SmartBrush with the higher CLIP score on OpenImages bounding boxes, 27.61 against 27.56 2312.03594 tab. 1.
• Single model against task-specific models. The paper presents one unified model 2312.03594 §4.2. Its table shows task-specific models with better FID and CLIP in several bounding-box and layout cells, which the paper calls comparable 2312.03594 tab. 7.

Paper against its own released code (the cards' Paper vs code sections list these in full; the code is not a paper, so the code side is not cited to a paper here): for the motion paper, the code's default for control fine-tuning is 600,000 steps, the second take starts from a different step than the paper's T' = 700, and the soft-mask value M_soft = 0.1 does not appear in the code 2303.01418 §4.1. Check any of these before reimplementing. For the image paper, the released v2 model places the task prompts on a separate branch rather than in the U-Net's text input, so the paper's guidance equation does not act on the same tensors 2312.03594 eq. 5. See the card's Paper vs code section before reimplementing.

Within 2303.09472, paper against its released code: the inpainting config uses channel width 52, against the paper's 48 to 384 range, and the launch script gives a total batch of 32, against the paper's 30 Paper vs code (items 9 and 13). The inpainting numbers are not checked against a rerun.

Across the six papers: no direct conflict is known. The two motion papers both report FID on HumanML3D, but for different tasks: single-prompt generation in 2209.14916 tab. 1, and long multi-prompt sequences in 2303.01418 tab. 2. The robot paper reports solve rate and success 2512.05964 §5.1. The image papers use different metrics and datasets: PowerPaint with CLIP score on OpenImages and MSCOCO 2312.03594 tab. 1, SmartBrush with Local FID, FID and CLIP score on the same two datasets 2212.05034 tab. 1, and DiffIR with LPIPS on Places and CelebA-HQ 2303.09472 tab. 1. The same method can also appear on different scales: SmartBrush's OpenImages bounding-box CLIP is 0.266 in its own table 2212.05034 tab. 1, and PowerPaint's table lists SmartBrush at 27.61 2312.03594 tab. 1. Numbers across these tasks and tables are not comparable.

Pitfalls

• Mask direction. The paper says the soft mask "dictates the amount of refinement" 2303.01418 §3.1. The card's code reading says the mask multiplies the injected noise, so a value near 1 injects less fresh noise Human Motion Diffusion as a Generative Prior. These two readings do not obviously agree about which frames refine more, so confirm the direction against the code before reusing the schedule.
• Blend at every step. The handshake is blended after every denoising step, not once at the end 2303.01418 §3.1.
• Blend length and handshake size. The paper's ablation gives a one-second handshake and a blend of 10 frames for HumanML3D 2303.01418 tab. 2 2303.01418 §4.1. The card records that the code's default blend length differs.
• Interval length. The prior was trained on clips up to 10 s 2303.01418 §4.1. The paper does not say how long intervals behave.
• Loss sign. Eq. 2 of the robot paper writes the regression target as eps minus the action, but its algorithm and code use the action minus eps. The sampler integrates from noise to data, so implement the algorithm's sign 2512.05964 eq. 2 2512.05964 alg. 1.
• Prefix is hard. The training-time form cannot use the overlapping actions beyond the prefix with decreasing weights, as inference-time RTC can 2512.05964 §6.
• Delay distribution. The authors say the training delay distribution must match the expected latency, and give no rule for choosing it. A delay above the training maximum is not covered 2512.05964 §6 2512.05964 §5.2.
• Per-action timestep needs per-token conditioning. The method relies on a time signal that differs per action step. For architectures without that, the paper gives no recipe 2512.05964 §4 2512.05964 fig. 2.
• Prefix after the last step. The listing overwrites the prefix at the start of each step but applies the last update to it too, so the returned prefix is not exactly the committed one. The code discards those entries and executes the old chunk instead 2512.05964 alg. 1.
• Guidance scale for removal. The paper gives no value for the guidance weight w 2312.03594 eq. 5. The released app defaults to 7.5 and recommends 10 or higher for removal and outpainting, and low values leave objects in the hole Paper vs code. Choose w by testing and state it as an assumption.
• Prompt wording. The released app adds extra words to the prompts for removal, and its v1 and v2 paths differ. The paper's prompts are not given verbatim, so reproduce results with the prompt text the card describes Paper vs code.
• Mask generation is not described. The random masks for context filling follow prior work, and the Places2 subset criteria are not given 2312.03594 §4.1. The Places2 numbers cannot be reproduced exactly.
• Human study scale. The user study uses 2,995 votes and compares against only the two strongest baselines per task, with a rater pool the paper does not describe 2312.03594 §4.1. Its percentages are not comparable with the automatic metrics.
• No write-back in DiffIR. Its sampler outputs a prior, which the restorer turns into the image 2303.09472 alg. 2. Reusing the known-value overwrite loop from the first family would not match its design.
• DiffIR code checks are partial. The beta schedule and the clean-prior head were checked on the SR code, not the inpainting denoiser. The paper gives no learning-rate schedule for inpainting 2303.09472 §5.2 Paper vs code.
• Clipping during editing. The base motion paper does not mention clipping of the clean-motion estimate 2209.14916 §3. Its library sampler clamps it by default, but the editing script turns clipping off, so pick one and state it Paper vs code.
• SmartBrush core values are unstated. The precision range, the blur kernel and deviation for each level, and how the level enters the network are not given 2212.05034 §4.2 2212.05034 eq. 7. An implementer has to choose them, so the mask-control behaviour will not match the paper exactly.
• SmartBrush mask switch. The step at which the coarse mask is replaced by the predicted one, and any binarisation threshold, are not given 2212.05034 §4.3. Figure 7 shows the prediction sharpening after about 10 steps 2212.05034 fig. 7.
• SmartBrush sampler and guidance. The sampler, step count and guidance scale are not stated 2212.05034 §5.1; section 3 names DDIM and PNDM only as generic options 2212.05034 §3.
• SmartBrush mask loss target. The noise loss takes the blurred mask as input, while the mask loss targets the accurate mask 2212.05034 eq. 8 2212.05034 eq. 9. The paper does not say at which timesteps the mask loss applies.
• SmartBrush resolution. Blended Diffusion and GLIDE were run at 256 px and resized, against 512 px for SmartBrush, so part of the gap may be resolution 2212.05034 §5.1.
• SmartBrush long shadows. Shadows that extend beyond the object mask are not covered, because the coarsest mask is the bounding box 2212.05034 §6.

Papers

• Human Motion Diffusion Model: the base text-to-motion model, whose editing overwrites known entries at each sampling step with no retraining. It is the inherited baseline for the motion control paper.
• Human Motion Diffusion as a Generative Prior: the inherited inpainting baseline, a fine-tuned control method with zeroed noise, DiffusionBlending, and soft-masked joins for long motion.
• Training-Time Action Conditioning for Efficient Real-Time Chunking: a clean, delay-simulated action prefix trained into a flow-matching policy, as a training-time alternative to inference-time prefix inpainting.
• A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting: one Stable Diffusion-based inpainting model with learned task prompts for context filling, object insertion, shape-guided insertion and object removal.
• SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model: a named object inserted into exact or coarse masks, with a blurred-mask precision input and a predicted-mask output that keeps the background.
• DiffIR: Efficient Diffusion Model for Image Restoration: a diffusion model on a compact restoration prior rather than pixels, with inpainting, x4 super-resolution and deblurring as its three restoration tasks.