Video inpainting

Overview

Given a clip and a binary mask per frame (1 marks the hole), video inpainting fills the hole in every frame so the filled region matches the surrounding content and stays consistent from one frame to the next. The input setting assumed by the main method is a frame sequence with one mask per frame, with optical flow between neighbouring frames 2309.03897 §3.

Two families of approach exist. Propagation copies pixels from neighbouring frames along estimated motion. Learned methods let a network look across frames and predict the missing content. Propagation inherits errors from the motion estimate, and learned methods are limited in how many frames they can look at because of memory and compute 2309.03897 §1 2309.03897 §3.3.

The one card on this wiki for this concept combines both families. Its pipeline has three parts: flow completion, propagation in two domains (pixels first, then features), and a Transformer that attends sparsely over time 2309.03897 §3.

Variants

Flow-guided propagation with learned refinement and a sparse Transformer

The method first repairs the optical flow inside the hole with a recurrent network. It then copies pixels where the completed flow passes a forward-backward consistency check, with no learned weights. Learned features are propagated along the same flow with flow-guided deformable convolution, and an 8-block Transformer runs over local windows, attending mainly to windows that touch the hole 2309.03897 §3.1 2309.03897 §3.2 2309.03897 §3.3 2309.03897 §4. Card: ProPainter: Improving Propagation and Transformer for Video Inpainting.

The authors report the best PSNR, SSIM, VFID and warping error among the methods they compare on YouTube-VOS and DAVIS. On DAVIS, PSNR is 34.47 dB, against 33.01 dB for E2FGVI 2309.03897 tab. 1. Runtime is 0.083 s per frame, and the table gives 808 GFLOPs for 10 frames. The table does not state the hardware used for these runtimes 2309.03897 §4.1.

Flow completion is the cheapest step among the flow-based methods in the paper's comparison: 0.005 s per frame, against 0.231 s for ISVI and 0.312 s for FGT 2309.03897 tab. 2.

Other methods in the same comparison

The paper also compares against DFVI, CPNet, FGVC, STTN, TSAM, FuseFormer, ISVI, FGT and E2FGVI 2309.03897 tab. 1. None of them has a card on this wiki, so their designs are not described here. Their names and reported numbers come only from the table above.

When to use which

Implementation sketch

The common core of the method, in our words. Variants differ mainly in how steps 3 and 4 are done.

# Video inpainting over one clip  [arxiv:2309.03897 §3]
# Input:  frames (T, H, W, 3); masks (T, H, W), 1 = hole
# Output: completed frames (T, H, W, 3)

def video_inpaint(frames, masks):
    # 1. Estimate forward and backward flow between neighbours [§4]
    flow_fwd, flow_bwd = estimate_flow(frames)
    # 2. Repair flow inside the holes with a recurrent network [§3.1]
    flow_fwd, flow_bwd = complete_flow(flow_fwd, flow_bwd, masks)
    # 3. Copy pixels where the round-trip flow check passes [§3.2]
    frames, masks = propagate_pixels(frames, masks, flow_fwd, flow_bwd)
    # 4. Propagate features along the flow and refine them [§3.2]
    feats = propagate_features(frames, masks, flow_fwd, flow_bwd)
    # 5. Run the mask-aware Transformer over local windows [§3.3]
    feats = sparse_transformer(feats, masks)
    # 6. Decode features into completed frames [§3]
    return decode(feats)

The sparse-attention rule in step 5:

# Mask-aware window attention  [arxiv:2309.03897 §3.3] [eq. 6]
# Input:  tokens (T, H, W, C); token_mask (T, H, W), 1 = hole
# Output: tokens (T, H, W, C)

def sparse_attention(tokens, token_mask):
    # 1. A query window is active only if the hole touches it in a local frame
    #    [§3.3] [eq. 6]
    active = windows_touched_by_hole(token_mask)
    for window in all_windows:
        if active[window]:
            # 2. Keys and values come from every second frame, the parity
            #    alternating between blocks [§3.3]
            keys, values = gather_keys(window, every_second_frame=True)
            out[window] = attention(tokens[window], keys, values)
        else:
            # 3. The paper does not state the output rule for inactive
            #    windows [§3.3]
            out[window] = not_stated
    return out

Points where a re-implementation most often diverges from the official code. The paper is the source for these items, and the code is named only to show the gap 2309.03897 §3.2 2309.03897 §4:

• Consistency threshold. The paper uses a fixed squared round-trip error of 5 §3.2 eq. 2 eq. 3. The official code uses a test that depends on flow magnitude. The two agree only at about 15 px flow.
• Flow learning rate. The paper states 1e-4 for both networks §4. The official flow-completion config uses 5e-5.
• Equation 5 notation. The text says the current feature enters the final fusion, but eq. 5 writes the frame feature in that position eq. 5. Follow the text.
• Discriminator loss. The loss printed in the appendix is not in the usual form, and the official config uses a hinge loss instead app. A.
• Unstated numbers. The paper does not state the optimiser betas, the learning-rate schedule, or the number of reference frames used in training §4.

Contradictions

• Speed-up of flow completion. The text says the recurrent network is about 40 times faster than earlier flow completion 2309.03897 §4.1. Table 2 gives ratios of about 46 against ISVI and about 62 against FGT 2309.03897 tab. 2. These are different comparisons, but the stated factor does not match the table.
• Margin over the best DAVIS baseline. The text gives a margin of 1.14 dB over the best DAVIS baseline 2309.03897 §4.1. Table 1 gives 34.47 - 33.01 = 1.46 dB against E2FGVI 2309.03897 tab. 1. We could not match 1.14 dB to any pair of entries in Table 1.
• Full-model PSNR. The full model scores 34.15 dB in the ablation 2309.03897 tab. 3 and 34.43 dB in Table 1 2309.03897 tab. 1. The ablation was trained for 450k iterations and the main result for 700k 2309.03897 §4, so the two are not comparable.

Papers

• ProPainter: Improving Propagation and Transformer for Video Inpainting: flow completion, two-domain propagation and a mask-aware sparse Transformer for video inpainting.