---
kind: concept
slug: image-inpainting
title: Image inpainting
summary: >-
  Filling a masked hole in an image so the result blends with the known pixels. Structure-first methods predict edges and lines before painting texture, which helps large holes and big images. One card so far, author-reported results; for diffusion-based fills see the diffusion inpainting page.
tags: [inpainting, image-restoration, transformer, structure-guidance]
papers: ["2203.00867"]
related: [diffusion-inpainting, positional-encoding, residual-learning, cnn-transformer-hybrid, canny-edge-detection, masked-image-modeling]
status: draft
updated_at: 2026-11-08
---

## Overview

Inpainting takes an image and a mask. Pixels where the mask is 1 are a hole to
fill; the rest must be kept as they are [arxiv:2203.00867 §3.4] [eq. 8]. The
hard part is structure: a fill that paints plausible texture can still get
straight lines and object outlines wrong.

The one card on this page takes a two-stage route. A transformer first
predicts the missing edges and lines as a small sketch on a 256 px grid
[arxiv:2203.00867 §3.1]. A separate painter then fills colour and texture,
guided by that sketch [arxiv:2203.00867 §3.3]. The painter is the LaMa
architecture, a feed-forward network trained with reconstruction, adversarial
and perceptual losses, not a denoising diffusion model
[arxiv:2203.00867 §3.3] [arxiv:2203.00867 §3.4]. Diffusion-based fills are
covered on the [diffusion inpainting](/concepts/diffusion-inpainting) page.

## Variants

### Structure-first two-stage inpainting

The sketch stage is a transformer with row and column axial attention and
a learned relative position bias [arxiv:2203.00867 §3.1] [eq. 1]. It is
trained with binary cross-entropy on the edge and line maps [eq. 2]. Its
input is a masked image with Canny edges and wireframe lines, and it runs at
256 px [arxiv:2203.00867 §3.1]. At inference, Mask-Predict runs five
rounds, re-masking only low-confidence edge and line pixels
[arxiv:2203.00867 app. E.4]. The card [Structure-first inpainting (ZITS)](/papers/2203.00867)
is the source for every number here.

The painter is helped by three additions. A small learned upsampler (SSU)
enlarges the sketch for bigger images [arxiv:2203.00867 §3.2] [arxiv:2203.00867 app. D].
A structure encoder (SFE) turns the sketch into four feature maps
[arxiv:2203.00867 §3.3] [eq. 3]. Each map enters a layer through a scalar
that starts at zero, so the painter's output is unchanged at the start of
fine-tuning [arxiv:2203.00867 §3.3] [eq. 7]. A masking positional encoding
gives each hole pixel its distance to the known region and a direction code
[arxiv:2203.00867 §3.3] [eq. 4] [eq. 5].

Training has two stages. The painter is first trained as in LaMa, then the
structure parts are added and the model is fine-tuned in an incremental stage
[arxiv:2203.00867 §3.3] [arxiv:2203.00867 §4.2]. The loss weights are L1 10,
adversarial 10, feature matching 100 and high-receptive-field perceptual 30
[arxiv:2203.00867 §3.4] [eq. 11].

### Plain texture painter (baseline)

LaMa, which the card's painter is built on, fills holes from the masked image
alone, with no sketch stage (Suvorov et al., 2021, cited in
[arxiv:2203.00867 §3.3]). It is the baseline the card compares against most
often [arxiv:2203.00867 tab. 1].

## When to use which

Numbers are author-reported on the card's own setup (256 px, mixed masks)
[arxiv:2203.00867 tab. 1]. Gains over LaMa are small on PSNR and clearer on
FID and LPIPS on Indoor.

| Situation | Use | Why |
|---|---|---|
| Large holes or thin structures matter, and the image is 256 px | Structure-first (ZITS) | Mixed-mask Indoor PSNR 25.57 vs 25.20 for LaMa; FID 15.93 vs 16.97 [arxiv:2203.00867 tab. 1] |
| Plain texture fill is enough, and a single painter is simpler to build | Plain painter (LaMa) | The ablation gain from the structure parts is 0.37 dB PSNR on Indoor, the full model over LaMa [arxiv:2203.00867 tab. 4] |
| Images at 512 px and above | Structure-first (ZITS), with checking | At 512 px on Indoor, PSNR 25.36 vs 24.42 for LaMa [arxiv:2203.00867 tab. 5]. On the 1k Places2 check without retraining, PSNR is equal (19.65 vs 19.69) and FID is better (55.85 vs 61.67) [arxiv:2203.00867 tab. 10] |
| Latency or GPU memory is tight | Plain painter (LaMa) | The sketch stage adds five transformer passes per image [arxiv:2203.00867 app. E.4]. TSR alone runs at 7.89 FPS and 10.5 GB with axial attention, batch 8, single V100 [arxiv:2203.00867 tab. 8] |
| Small holes (10–20% of the image) | Either; the gain is larger here | Indoor PSNR 29.87 vs 29.05 for LaMa, a 0.82 dB gap. At 40–50% the gap is 0.74 dB [arxiv:2203.00867 tab. 7] |

The sketch stage only works at 256 px, and the authors report that distant
urban views lose structure [arxiv:2203.00867 app. G]. For scenes of that kind,
do not expect the sketch to carry the fill.

## Implementation sketch

Two-stage pipeline. The sketch comes first; the painter runs once on the
enlarged sketch. Shapes assume 256 px input.

```pseudocode
# Structure-first inpainting, after [arxiv:2203.00867 §3] (not a numbered algorithm)
# Input:  masked_image (B, 3, 256, 256) in [0, 1]; mask (B, 1, 256, 256), 1 = hole
# Output: inpainted_image (B, 3, 256, 256)

def inpaint(masked_image, mask, edges, lines):
    # Predict edge and line maps for the hole  [arxiv:2203.00867 §3.1]
    edge_logits, line_logits = sketch_transformer(masked_image, edges, lines, mask)
    # Re-mask low-confidence structure and repeat, five rounds  [arxiv:2203.00867 app. E.4]
    edge_map, line_map = mask_predict(edge_logits, line_logits, mask, rounds=5)
    # Upsample the sketch to the painter's working size  [arxiv:2203.00867 §3.2]
    edge_big, line_big = upsample(edge_map, line_map)
    # Four structure feature maps from the sketch  [arxiv:2203.00867 §3.3] [eq. 3]
    structure_feats = encode_structure(edge_big, line_big, mask_big)
    # Distance and direction codes for hole pixels  [arxiv:2203.00867 §3.3] [eq. 4] [eq. 5]
    distance_emb, direction_emb = masking_position_encoding(mask_big)
    # Paint; each structure map enters via a zero-initialised scalar
    # [arxiv:2203.00867 §3.3] [eq. 7]
    x = first_layer(masked_image, mask_big) + distance_emb + direction_emb
    for k in range(4):
        x = painter_block_k(x + alpha[k] * structure_feats[k])
    return decode(x)
```

## Contradictions

The card reports a conflict inside its own tables. For Places2 mixed masks,
the LPIPS of HiFill, Co-Mod and MST is 0.246, 0.122 and 0.137 in Table 1, but
0.137, 0.246 and 0.122 in Table 7 [arxiv:2203.00867 tab. 1] [arxiv:2203.00867 app. E.2].
Only the other values agree. Do not quote those three values until the
source is checked. The card also flags the Indoor LPIPS values in Table 5
(0.826 and 0.823 for LaMa and this paper) as far above every other LPIPS in
the paper [arxiv:2203.00867 tab. 5], which may be a transcription or print
error.

No other paper on this page, so no cross-paper contradictions are known.

## Papers

- [Structure-first inpainting (ZITS)](/papers/2203.00867): a transformer sketches edges and lines, and a LaMa-style painter fills texture with masking positional encoding and zero-initialised structure injection.