---
kind: concept
slug: video-inpainting
title: Video inpainting
summary: >-
  Filling masked or unwanted regions in every frame of a video so the result looks plausible and stays steady over time. Covers flow-guided propagation with learned refinement and a mask-aware Transformer. One paper card so far, so the design space is narrow.
tags: [video, inpainting, optical-flow]
papers: ["2309.03897"]
related: [optical-flow, deformable-convolution, local-global-attention, diffusion-inpainting]
status: draft
updated_at: 2026-11-06
---

## Overview

Given a clip and a binary mask per frame (1 marks the hole), video inpainting
fills the hole in every frame so the filled region matches the surrounding
content and stays consistent from one frame to the next. The input setting
assumed by the main method is a frame sequence with one mask per frame, with
optical flow between neighbouring frames [arxiv:2309.03897 §3].

Two families of approach exist. Propagation copies pixels from neighbouring
frames along estimated motion. Learned methods let a network look across frames
and predict the missing content. Propagation inherits errors from the motion
estimate, and learned methods are limited in how many frames they can look at
because of memory and compute [arxiv:2309.03897 §1] [arxiv:2309.03897 §3.3].

The one card on this wiki for this concept combines both families. Its
pipeline has three parts: flow completion, propagation in two domains (pixels
first, then features), and a Transformer that attends sparsely over time
[arxiv:2309.03897 §3].

## Variants

### Flow-guided propagation with learned refinement and a sparse Transformer

The method first repairs the optical flow inside the hole with a recurrent
network. It then copies pixels where the completed flow passes a
forward-backward consistency check, with no learned weights. Learned features
are propagated along the same flow with flow-guided deformable convolution, and
an 8-block Transformer runs over local windows, attending mainly to windows
that touch the hole [arxiv:2309.03897 §3.1] [arxiv:2309.03897 §3.2]
[arxiv:2309.03897 §3.3] [arxiv:2309.03897 §4]. Card:
[ProPainter: Improving Propagation and Transformer for Video Inpainting](/papers/2309.03897).

The authors report the best PSNR, SSIM, VFID and warping error among the
methods they compare on YouTube-VOS and DAVIS. On DAVIS, PSNR is 34.47 dB,
against 33.01 dB for E2FGVI [arxiv:2309.03897 tab. 1]. Runtime is 0.083 s per
frame, and the table gives 808 GFLOPs for 10 frames. The table does not state
the hardware used for these runtimes [arxiv:2309.03897 §4.1].

Flow completion is the cheapest step among the flow-based methods in the
paper's comparison: 0.005 s per frame, against 0.231 s for ISVI and 0.312 s
for FGT [arxiv:2309.03897 tab. 2].

### Other methods in the same comparison

The paper also compares against DFVI, CPNet, FGVC, STTN, TSAM, FuseFormer,
ISVI, FGT and E2FGVI [arxiv:2309.03897 tab. 1]. None of them has a card on
this wiki, so their designs are not described here. Their names and reported
numbers come only from the table above.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Best reported quality on standard benchmarks at about 432 x 240, and a multi-GPU training budget is not a constraint | Flow-guided propagation with sparse Transformer (above) | Best reported scores on all four metrics in Table 1 [arxiv:2309.03897 tab. 1]. |
| Per-frame latency is the main constraint | A faster method from the comparison; not described on this wiki | Table 1 lists faster methods, such as STTN at 0.051 s and TSAM at 0.068 s per frame, with lower reported PSNR [arxiv:2309.03897 tab. 1]. |
| Flow inside the hole is unreliable or unavailable | Keep flow completion in mind; copying cannot correct a wrong flow | Copying along a wrong flow misaligns content [arxiv:2309.03897 §1]. The paper's ablation has no variant without flow completion. |
| Choosing which stages to keep | Keep both propagation steps | Removing image propagation gives 33.05 dB, and removing feature propagation gives 33.17 dB, against 34.15 dB for the full model [arxiv:2309.03897 tab. 3]. The ablation dataset is not named, and the models were trained for 450k rather than 700k iterations. |
| Long videos | Plan chunking yourself | The paper fixes a local temporal window and a set of sparse references [arxiv:2309.03897 §3.3] [arxiv:2309.03897 §4.1]. It does not describe how to process a video longer than memory allows. |
| Training budget | Expect a large one | Training uses 700k iterations, batch size 8, and 8 V100 GPUs with 32 GB each [arxiv:2309.03897 §4]. |
| Commercial reuse of the official code | Implement from the paper; do not copy code | The official repository is under a non-commercial licence (see the card's licence note). |

## Implementation sketch

The common core of the method, in our words. Variants differ mainly in how
steps 3 and 4 are done.

```pseudocode
# Video inpainting over one clip  [arxiv:2309.03897 §3]
# Input:  frames (T, H, W, 3); masks (T, H, W), 1 = hole
# Output: completed frames (T, H, W, 3)

def video_inpaint(frames, masks):
    # 1. Estimate forward and backward flow between neighbours [§4]
    flow_fwd, flow_bwd = estimate_flow(frames)
    # 2. Repair flow inside the holes with a recurrent network [§3.1]
    flow_fwd, flow_bwd = complete_flow(flow_fwd, flow_bwd, masks)
    # 3. Copy pixels where the round-trip flow check passes [§3.2]
    frames, masks = propagate_pixels(frames, masks, flow_fwd, flow_bwd)
    # 4. Propagate features along the flow and refine them [§3.2]
    feats = propagate_features(frames, masks, flow_fwd, flow_bwd)
    # 5. Run the mask-aware Transformer over local windows [§3.3]
    feats = sparse_transformer(feats, masks)
    # 6. Decode features into completed frames [§3]
    return decode(feats)
```

The sparse-attention rule in step 5:

```pseudocode
# Mask-aware window attention  [arxiv:2309.03897 §3.3] [eq. 6]
# Input:  tokens (T, H, W, C); token_mask (T, H, W), 1 = hole
# Output: tokens (T, H, W, C)

def sparse_attention(tokens, token_mask):
    # 1. A query window is active only if the hole touches it in a local frame
    #    [§3.3] [eq. 6]
    active = windows_touched_by_hole(token_mask)
    for window in all_windows:
        if active[window]:
            # 2. Keys and values come from every second frame, the parity
            #    alternating between blocks [§3.3]
            keys, values = gather_keys(window, every_second_frame=True)
            out[window] = attention(tokens[window], keys, values)
        else:
            # 3. The paper does not state the output rule for inactive
            #    windows [§3.3]
            out[window] = not_stated
    return out
```

Points where a re-implementation most often diverges from the official code.
The paper is the source for these items, and the code is named only to show
the gap [arxiv:2309.03897 §3.2] [arxiv:2309.03897 §4]:

- **Consistency threshold.** The paper uses a fixed squared round-trip error
  of 5 [§3.2] [eq. 2] [eq. 3]. The official code uses a test that depends on
  flow magnitude. The two agree only at about 15 px flow.
- **Flow learning rate.** The paper states 1e-4 for both networks [§4]. The
  official flow-completion config uses 5e-5.
- **Equation 5 notation.** The text says the current feature enters the final
  fusion, but eq. 5 writes the frame feature in that position [eq. 5]. Follow
  the text.
- **Discriminator loss.** The loss printed in the appendix is not in the usual
  form, and the official config uses a hinge loss instead [app. A].
- **Unstated numbers.** The paper does not state the optimiser betas, the
  learning-rate schedule, or the number of reference frames used in training
  [§4].

## Contradictions

- **Speed-up of flow completion.** The text says the recurrent network is about
  40 times faster than earlier flow completion [arxiv:2309.03897 §4.1]. Table 2
  gives ratios of about 46 against ISVI and about 62 against FGT [arxiv:2309.03897 tab. 2].
  These are different comparisons, but the stated factor does not match the table.
- **Margin over the best DAVIS baseline.** The text gives a margin of 1.14 dB
  over the best DAVIS baseline [arxiv:2309.03897 §4.1]. Table 1 gives 34.47 - 33.01 = 1.46 dB
  against E2FGVI [arxiv:2309.03897 tab. 1]. We could not match 1.14 dB to any pair of
  entries in Table 1.
- **Full-model PSNR.** The full model scores 34.15 dB in the ablation [arxiv:2309.03897 tab. 3]
  and 34.43 dB in Table 1 [arxiv:2309.03897 tab. 1]. The ablation was trained for 450k
  iterations and the main result for 700k [arxiv:2309.03897 §4], so the two are not comparable.

## Papers

- [ProPainter: Improving Propagation and Transformer for Video Inpainting](/papers/2309.03897): flow completion, two-domain propagation and a mask-aware sparse Transformer for video inpainting.