Image inpainting

Overview

Inpainting takes an image and a mask. Pixels where the mask is 1 are a hole to fill; the rest must be kept as they are 2203.00867 §3.4 eq. 8. The hard part is structure: a fill that paints plausible texture can still get straight lines and object outlines wrong.

The one card on this page takes a two-stage route. A transformer first predicts the missing edges and lines as a small sketch on a 256 px grid 2203.00867 §3.1. A separate painter then fills colour and texture, guided by that sketch 2203.00867 §3.3. The painter is the LaMa architecture, a feed-forward network trained with reconstruction, adversarial and perceptual losses, not a denoising diffusion model 2203.00867 §3.3 2203.00867 §3.4. Diffusion-based fills are covered on the diffusion inpainting page.

Variants

Structure-first two-stage inpainting

The sketch stage is a transformer with row and column axial attention and a learned relative position bias 2203.00867 §3.1 eq. 1. It is trained with binary cross-entropy on the edge and line maps eq. 2. Its input is a masked image with Canny edges and wireframe lines, and it runs at 256 px 2203.00867 §3.1. At inference, Mask-Predict runs five rounds, re-masking only low-confidence edge and line pixels 2203.00867 app. E.4. The card Structure-first inpainting (ZITS) is the source for every number here.

The painter is helped by three additions. A small learned upsampler (SSU) enlarges the sketch for bigger images 2203.00867 §3.2 2203.00867 app. D. A structure encoder (SFE) turns the sketch into four feature maps 2203.00867 §3.3 eq. 3. Each map enters a layer through a scalar that starts at zero, so the painter's output is unchanged at the start of fine-tuning 2203.00867 §3.3 eq. 7. A masking positional encoding gives each hole pixel its distance to the known region and a direction code 2203.00867 §3.3 eq. 4 eq. 5.

Training has two stages. The painter is first trained as in LaMa, then the structure parts are added and the model is fine-tuned in an incremental stage 2203.00867 §3.3 2203.00867 §4.2. The loss weights are L1 10, adversarial 10, feature matching 100 and high-receptive-field perceptual 30 2203.00867 §3.4 eq. 11.

Plain texture painter (baseline)

LaMa, which the card's painter is built on, fills holes from the masked image alone, with no sketch stage (Suvorov et al., 2021, cited in 2203.00867 §3.3). It is the baseline the card compares against most often 2203.00867 tab. 1.

When to use which

Numbers are author-reported on the card's own setup (256 px, mixed masks) 2203.00867 tab. 1. Gains over LaMa are small on PSNR and clearer on FID and LPIPS on Indoor.

The sketch stage only works at 256 px, and the authors report that distant urban views lose structure 2203.00867 app. G. For scenes of that kind, do not expect the sketch to carry the fill.

Implementation sketch

Two-stage pipeline. The sketch comes first; the painter runs once on the enlarged sketch. Shapes assume 256 px input.

# Structure-first inpainting, after [arxiv:2203.00867 §3] (not a numbered algorithm)
# Input:  masked_image (B, 3, 256, 256) in [0, 1]; mask (B, 1, 256, 256), 1 = hole
# Output: inpainted_image (B, 3, 256, 256)

def inpaint(masked_image, mask, edges, lines):
    # Predict edge and line maps for the hole  [arxiv:2203.00867 §3.1]
    edge_logits, line_logits = sketch_transformer(masked_image, edges, lines, mask)
    # Re-mask low-confidence structure and repeat, five rounds  [arxiv:2203.00867 app. E.4]
    edge_map, line_map = mask_predict(edge_logits, line_logits, mask, rounds=5)
    # Upsample the sketch to the painter's working size  [arxiv:2203.00867 §3.2]
    edge_big, line_big = upsample(edge_map, line_map)
    # Four structure feature maps from the sketch  [arxiv:2203.00867 §3.3] [eq. 3]
    structure_feats = encode_structure(edge_big, line_big, mask_big)
    # Distance and direction codes for hole pixels  [arxiv:2203.00867 §3.3] [eq. 4] [eq. 5]
    distance_emb, direction_emb = masking_position_encoding(mask_big)
    # Paint; each structure map enters via a zero-initialised scalar
    # [arxiv:2203.00867 §3.3] [eq. 7]
    x = first_layer(masked_image, mask_big) + distance_emb + direction_emb
    for k in range(4):
        x = painter_block_k(x + alpha[k] * structure_feats[k])
    return decode(x)

Contradictions

The card reports a conflict inside its own tables. For Places2 mixed masks, the LPIPS of HiFill, Co-Mod and MST is 0.246, 0.122 and 0.137 in Table 1, but 0.137, 0.246 and 0.122 in Table 7 2203.00867 tab. 1 2203.00867 app. E.2. Only the other values agree. Do not quote those three values until the source is checked. The card also flags the Indoor LPIPS values in Table 5 (0.826 and 0.823 for LaMa and this paper) as far above every other LPIPS in the paper 2203.00867 tab. 5, which may be a transcription or print error.

No other paper on this page, so no cross-paper contradictions are known.

Papers

• Structure-first inpainting (ZITS): a transformer sketches edges and lines, and a LaMa-style painter fills texture with masking positional encoding and zero-initialised structure injection.