A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting

In plain words

Inpainting means filling in a part of a picture you have marked as missing. Earlier models did this in one of two ways. Some look only at the pixels around the hole, which is good for removing things and extending a background, but they cannot invent a new object. Others are told in words what to draw, which lets them add objects, but they tend to draw something even where you wanted the area left empty.

PowerPaint trains one model to do both. It learns a short hidden "task word" for each job, and the word that is attached to the request tells the model which job it is doing. One word means "fill in the background from the surroundings". Another means "draw the object this text describes". A third is used to make a new object follow the outline of the marked area more or less closely.

The trick for object removal is to use the object word as a negative prompt. The model is steered away from drawing that object, so the area is filled with what surrounds it. In a user study on object removal, people preferred its results over those of the two baselines tested.

TL;DR

• One Stable Diffusion-based inpainting model with three learned task prompts: P_obj (insert an object), P_ctxt (fill from context), P_shape (shape-fitted object) §3.2.
• Object removal reuses the context prompt as the positive prompt and the object prompt as a negative prompt in classifier-free guidance §3.2 eq. 5.
• Shape control interpolates the text embeddings of the context and shape prompts, weighted by a fitting degree §3.2 eq. 7.
• Author-reported: best FID in all four object-inpainting settings of tab. 1 and tab. 2, best FID on Places2 context filling tab. 3, best FID on outpainting tab. 4. Code is released for the inference side only.

Problem

Context-aware inpainting (filling a mask from its surroundings) is trained by masking random regions of real images and reconstructing them §3.2. It cannot synthesize objects that the context does not imply §1. Text-guided inpainting, fine-tuned on masks and captions, can add objects, but it assumes an object is wanted in the masked region. For clean background filling it then needs prompt engineering or extra workflow, and it can produce incoherent artifacts §1. The paper also reports that a diffusion model copies content from the surrounding context into the hole, which makes removing objects in crowded scenes hard §3.2 fig. 3.

Training a single model for both regimes is difficult because the two tasks use different training strategies [abstract, §1].

Method

Preliminary: the base model

PowerPaint starts from a text-to-image latent diffusion model, Stable Diffusion §3.1. The forward process adds Gaussian noise in closed form eq. 1:

x_t = \sqrt{\bar{\alpha_t}}\, x_0 + \sqrt{1 - \bar{\alpha_t}}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I) eq. 1

The classical diffusion objective trains a network to predict the added noise eq. 2:

\mathcal{L} = \mathbb{E}_{x_0, t, \epsilon_t} \lVert \epsilon_t - \epsilon_\theta(x_t, t) \rVert_2^2 eq. 2

For inpainting, the first convolution of the denoiser is extended with five extra input channels. These carry the masked image x_0 ⊙ (1 − m) and the mask m, so the network input is the noisy latent, the masked image and the mask §3.1. The text condition τ_θ(y) comes from the CLIP text encoder, and the training loss is eq. 3:

\mathcal{L} = \mathbb{E}_{x_0, m, t, y, \epsilon_t} \lVert \epsilon_t - \epsilon_\theta(x_t', \tau_\theta(y), t) \rVert_2^2 eq. 3

The paper's change is to let the text condition contain learnable task prompts, so the text can be a caption, a task prompt, or a caption with a task prompt appended §3.1.

Learning with task prompts

The method uses three learned prompts §3.2 fig. 2:

• P_ctxt for context-aware filling. The mask is random, and the text condition is the prompt alone §3.2 eq. 4.
• P_obj for text-guided object inpainting. The mask is the bounding box of an object, and P_obj is appended as a suffix to the caption of the masked region §3.2.
• P_shape for shape-guided object inpainting. It is trained on precise object segmentation masks and their descriptions §3.2.

The context prompt is learned by minimising the loss over the prompt p, which starts as a randomly initialised array of tokens §3.2 eq. 4:

p_{ctxt} = \arg\min_{p} \; \mathbb{E}_{x_0, m, t, p, \epsilon_t} \lVert \epsilon_t - \epsilon_\theta(x_t', \tau_\theta(p), t) \rVert_2^2 eq. 4

Object removal by classifier-free guidance

The paper uses the classifier-free guidance form of §3.2 eq. 5. The positive prompt is P_ctxt and the negative prompt is P_obj. The guidance scale w is the weight on the positive prediction:

\tilde{\epsilon}_\theta = w \cdot \epsilon_\theta(x_t', \tau_\theta(p_{ctxt}), t) + (1 - w) \cdot \epsilon_\theta(x_t', \tau_\theta(p_{obj}), t) eq. 5

Guidance pushes the sample away from the object prompt and towards the context prompt, which lowers the likelihood of generating an object §3.2.

Shape-guided inpainting by prompt interpolation

To control how closely an object follows the mask, the training masks are grown by convolutional dilation §3.2 eq. 6:

m' = D(m, k, it) eq. 6

Here k is the kernel size and it is the number of dilation iterations §3.2. Each training mask gets an area ratio α, where a larger α means a closer fit to the mask shape §3.2.

At inference the two text embeddings are computed separately, the caption with P_ctxt appended and the caption with P_shape appended. They are then linearly blended by α §3.2 fig. 4 eq. 7:

\tau'_\theta = (1 - \alpha) \cdot \tau_\theta(y, p_{ctxt}) + \alpha \cdot \tau_\theta(y, p_{shape}) eq. 7

Users set α to control the fitting degree after training §3.2.

Training procedure (summary)

The text-guided and context tasks use the same objective as eq. 4, with the masks and prompts above. The shape task uses the interpolation in eq. 7 for its text input. The model is fine-tuned from Stable Diffusion, with the task-prompt embeddings in the CLIP text encoder and the U-Net trained §3.3. Text-to-image generation is treated as inpainting with the whole image masked, and it is sampled with probability 20% against 80% for the main task §3.3.

# One training step for a sample  [§3.1] [§3.2] [eq. 1] [eq. 3]
# Input:  image (3, H, W); mask (1, H, W); caption; task in {object,
#         context, shape}
# Output: scalar loss
def training_step(image, mask, caption, task):
    # Masked image is the image with the hole removed  [§3.1]
    masked_image = image * (1 - mask)
    # Noise the clean latent to a random timestep  [eq. 1]
    latent = encode(image)
    noise = sample_gaussian_like(latent)
    timestep = sample_timestep()
    noisy_latent = add_noise(latent, noise, timestep)
    # Choose the text condition from the task  [§3.2]
    # Context: learned prompt only  [eq. 4]
    # Object: caption with P_obj appended  [§3.2]
    # Shape: blend of the two prompts by the area ratio  [eq. 7]
    text_condition = text_for_task(task, caption, mask)
    # Predict the added noise from the stacked inputs  [§3.1] [eq. 3]
    prediction = unet(noisy_latent, mask, masked_image, text_condition)
    return mean_squared_error(prediction, noise)

# Object removal at one denoising step  [§3.2] [eq. 5]
# Input:  noisy latent; embedding of P_ctxt (positive) and of P_obj
#         (negative); guidance scale w
# Output: guided noise prediction
def removal_noise(noisy_latent, t, context_text, object_text, w):
    # Predict noise under the positive (context) prompt  [eq. 5]
    noise_positive = unet(noisy_latent, t, context_text)
    # Predict noise under the negative (object) prompt  [eq. 5]
    noise_negative = unet(noisy_latent, t, object_text)
    # Weighted sum: w on the positive, (1 - w) on the negative  [eq. 5]
    return w * noise_positive + (1 - w) * noise_negative

# Shape-guided text embedding with fitting degree alpha  [§3.2] [eq. 7]
# Input:  caption; alpha in [0, 1]
# Output: text embedding for the U-Net
def shape_text_embedding(caption, alpha):
    # Encode the caption with each learned prompt appended  [§3.2]
    shape_embedding = encode_text(caption + " P_shape")
    context_embedding = encode_text(caption + " P_ctxt")
    # Linear blend: alpha = 1 uses only the shape prompt  [eq. 7]
    return (1 - alpha) * context_embedding + alpha * shape_embedding

Hyperparameters

Results

Author-reported. Setup: object inpainting on OpenImages and MSCOCO, about 10K images and masks each §4.1; context filling on 10K Places2 test images §4.1; outpainting on 10K Flickr-Scenery images §4.1. Metrics are FID, Local-FID, CLIP score, LPIPS and aesthetic score §4.1. Table numbers follow the order of the table environments in the LaTeX source.

Text-guided object inpainting with bounding-box masks tab. 1. Local-FID and FID are lower-is-better, CLIP score is higher-is-better:

Shape-guided object inpainting with object layout masks tab. 2:

Context-aware inpainting on Places2 tab. 3. FID and LPIPS are lower-is-better. The left pair uses 40–50% masked images, the right pair all samples:

Outpainting on Flickr-Scenery tab. 4. FID is lower-is-better, aesthetic score is higher-is-better:

User study tab. 5. Preference shares; 2,995 valid votes in total §4.1. Each study compared PowerPaint with the two strongest baselines for that task:

Learnable prompt versus unlearnable identifiers tab. 6. The identifier variant uses rare tokens trained with the same strategy §4.2:

Single unified model versus task-specific models, with SD-Inpainting fine-tuned on the same data tab. 7. Each cell pair is OpenImages / MSCOCO:

The second part of tab. 7 compares inpainting and outpainting. Columns are FID and LPIPS for inpainting, then FID and aesthetic score for outpainting:

Paper vs code

The official repository is open-mmlab/PowerPaint, linked from the paper's project page (powerpaint.github.io). It is MIT licensed. The commit above is the head of the default branch at fetch time. The repository contains inference code only, for two architectures. Its README says the training code is on the dev branch, which this commit does not contain [README.md#L158-L160], so the training settings in §3.3 are not checked against code.

• Architecture. The paper's model adds five input channels to a Stable Diffusion U-Net §3.1. The v1 pipeline matches this: it checks for a 9-channel U-Net pipeline_PowerPaint.py and VAE-encodes the masked image pipeline_PowerPaint.py. The released v2 weights are different. The README says v2 was retrained and takes inspiration from BrushNet [README.md#L33-L34]. In the v2 pipeline the task-prompt embeddings go to the BrushNet branch pipeline_PowerPaint_Brushnet_CA.py, while the U-Net receives the user prompt and user negative prompt pipeline_PowerPaint_Brushnet_CA.py. So in v2 the contrast of eq. 5 acts on the branch outputs, not on P_ctxt and P_obj as U-Net text inputs. This reading is from the code, not from the paper.
• Base checkpoint. The paper says Stable Diffusion v1.5 §3.3. The v1 app loads runwayml/stable-diffusion-inpainting app.py.
• Shape prompt in v2. The paper appends P_shape and P_ctxt to the caption before encoding §3.2. In the v2 path the prompt prefix is empty, so the blended embedding contains only the task word and the caption goes to the U-Net separately app.py pipeline_PowerPaint_Brushnet_CA.py.
• Interpolation. The v1 pipeline blends the text embeddings after the encoder, as eq. 7 describes pipeline_PowerPaint.py. It blends the negative embeddings with the same weight pipeline_PowerPaint.py. The paper's eq. 7 covers only the positive branch.
• Fitting degree. The paper defines α as an area ratio computed per training mask §3.2. At inference the app passes the slider value directly as the blend weight app.py. The dilation in eq. 6 is not in the released code.
• Guidance. The standard guidance step is uncond + w * (text - uncond) pipeline_PowerPaint.py. With the negative prompt as the unconditional term, this equals eq. 5. The paper gives no value for w.
• Prompt words not in the paper. For v1, removal prepends "empty scene blur" to the positive prompt app.py. For v2, outpainting appends " empty scene" and removal appends " empty scene blur" app.py. Both negative prompts get quality words for shape-guided v1 app.py.
• Initialisation. The paper says the context prompt starts from a random token array §3.2. The app adds the three task tokens with ten vectors each, initialised from the embedding of "a" app.py. This is the inference-time loader; training initialisation is not in the code.
• Stale cross-reference. The appendix points to "Section 4.4" for the applications, but the source numbers that subsection §4.3. This card cites §4.3.

Pitfalls

• Ambiguous evaluation text. §4.1 says bounding boxes and layout masks are used "for testing on OpenImages and MSCOCO, respectively". tab. 1 and tab. 2 use both datasets with both mask types. Read the tables, not the sentence.
• Claims that the tables only partly support. The text says the model is state-of-the-art for both visual quality and text alignment in object inpainting §4.1. On OpenImages bounding boxes, SmartBrush has the higher CLIP score (27.61 against 27.56) tab. 1. On Places2, LaMa has the lowest LPIPS in both columns tab. 3.
• Single-model claim. tab. 7 shows task-specific models beat PowerPaint on several bounding-box and layout FID and CLIP cells, and the paper calls the gap comparable §4.2.
• Mask generation is cited, not described. Random masks for context filling follow prior work §4.1. The mask generator, its size distribution and the 40–50% subset criteria are not described, so Places2 numbers are not directly reproducible.
• Guidance scale is the main knob for removal. The paper gives no value for w §3.2. The app defaults to 7.5 and recommends 10 or higher for removal and outpainting app.py app.py. Low w leaves objects in the hole. Choose w from the code, and state it as an assumption.
• Shape fitting has a known weak range. Very small α values are hard to reach, because few training examples have a very small object area §4.3, limitations.
• The base model constrains quality. The authors note that visual quality is limited by the underlying text-to-image model §4.3, limitations.
• Prompt stacking changes the result. The v1 and v2 paths add different words to prompts app.py app.py. Use the same prompt text as the app when reproducing results, and note that the paper's prompts are not given verbatim.
• Human evaluation. The user study uses 2,995 votes, a rater pool that is not described, and only the two strongest baselines per task §4.1. The percentages are not comparable with the automatic metrics.
• Licences of weights. The released checkpoints are on Hugging Face (README). Their licences and the licence of the base Stable Diffusion checkpoint were not checked here, so they are not covered by this card's licence field.

Lineage

• Builds on: Stable Diffusion, a latent text-to-image diffusion model cited as Rombach et al. (2022) §3.1. It has no card in this wiki. Classifier-free guidance is cited as Ho and Salimans (2021) §3.2. Learned rare identifiers, the baseline in tab. 6, follow DreamBooth (Ruiz et al., 2023) §4.2.
• Compared against: LaMa, LDM-Inpainting, Blended Diffusion, Stable Diffusion, SD-Inpainting, ControlNet-Inpainting and SmartBrush §4.1. Adobe Firefly is compared on object removal qualitatively §3.2 fig. 3 app. B.
• Superseded by: not known from wiki content. The released v2 is a BrushNet-based variant [README.md#L33-L34].