Image matting

Overview

Matting predicts an alpha value in [0, 1] for each pixel, where 1 means fully foreground and 0 means fully background. The composite over a new background is I = alpha F + (1 - alpha) B 2012.07810 §4. Hard segmentation gives only 0 or 1, so it cannot represent hair or other see-through edges. The invariant every variant keeps is the compositing equation above. The variants differ in what extra input they need to solve for alpha:

• Trimap-based: a user or a generator marks certain foreground, certain background and an unknown band. Deep Image Matting (DIM) and FBA Matting are the trimap-based methods the card's paper compares against. The paper notes they need a hand-drawn or generated trimap 2012.07810 §5.
• Captured background: one frame of the empty scene is taken with the same camera. Background Matting (BGM; Sengupta et al., 2020) is the baseline here, run at 512x512 in the paper's comparisons 2012.07810 §5.1. Card: Real-Time High-Resolution Background Matting.

Variants

Background matting with patch refinement

Two networks run in sequence. A base network works on a downsampled copy of the frame and the background, and predicts a coarse alpha, a foreground residual and an error map 2012.07810 §4. A refinement network then re-predicts only the k patches with the largest predicted error, at half and full resolution 2012.07810 §4.2. Most pixels are never processed at full size. The authors report 60.0 fps at HD and 33.2 fps at 4K with a ResNet-50 backbone on an RTX 2080 Ti, against 7.8 fps for BGM at 512x512. These figures are pass-through timings without data transfer 2012.07810 tab. 3. Card: Real-Time High-Resolution Background Matting.

Trimap-based methods (named, not covered by a card)

DIM and FBA take a trimap as input. The paper reports FBA as the most accurate method on AIM and PhotoMatte85 among the methods it compares, but its speed figures are low: 3.3 fps at HD, or 2.9 fps when the trimap is generated automatically 2012.07810 tab. 1 2012.07810 tab. 3. No card for these papers exists yet, so their details are not given on this page.

When to use which

The speed and accuracy figures are the authors' own. The user study compared only against BGM: 59% of 40 raters preferred the new method, 23% preferred BGM 2012.07810 tab. 2. Dataset licences are not stated in the paper for VideoMatte240K or PhotoMatte85, so check them before commercial use 2012.07810 §3.

Implementation sketch

The common core of the captured-background variant. Refinement details, such as window sizes and channel counts, differ by implementation and are in the card.

# Background matting with patch refinement  [arxiv:2012.07810 §4]
# Input:  frame, empty_background (same camera, same size)
# Output: alpha, foreground colour

def matte(frame, empty_background, c, k):
    # Coarse pass on a downsampled copy, factor c  [arxiv:2012.07810 §4]
    small = downsample(concat(frame, empty_background), c)
    alpha_c, fg_c, err_c, hidden = base_net(small)

    # Choose the k cells with the largest predicted error  [§4.2]
    cells = top_k(upsample(err_c, cell_size), k)

    # Re-predict only those cells, at half and full resolution  [§4.2]
    alpha, fg = refine_cells(frame, empty_background,
                             cells, alpha_c, fg_c, hidden)
    return alpha, fg

Contradictions

• Within the paper, on FBA. The text says the method is "slightly worse than FBA" 2012.07810 §5.1. On Distinctions, the table shows the method lower than FBA on all four alpha metrics (SAD 9.19 against 11.03), so the statement holds only for AIM and PhotoMatte85 2012.07810 tab. 1.
• Within the paper, on BGM foreground error. The BGM row of the AIM ablation shows FG MSE 42.84, which is the BGM_a value in the main table. BGM's own value there is 47.98 2012.07810 tab. 5 2012.07810 tab. 1. The paper does not explain the difference.
• Paper against released code. The released training scripts differ from the paper in crop size, foreground scale, noise, the refinement loss and the evaluation defaults. The card lists nine differences in its Paper vs code section. Use the card's section before reproducing the paper's numbers.

Papers

• Real-Time High-Resolution Background Matting: a captured-background method with a coarse base pass and patch refinement, reporting real-time HD and 4K speeds.