Real-Time High-Resolution Background Matting

In plain words

Green-screen effects cut a person out of their background. This paper does the same without a green screen, provided you first record one photo or video frame of the empty room. Comparing the live frame with that empty frame shows where the person is, and how see-through each pixel is at the edges, such as hair.

The model works in two passes. A small, cheap pass looks at the whole picture at low resolution and makes a rough cut-out. A second pass looks only at the spots where the rough cut-out is likely wrong, such as hair, and redoes those small patches at full resolution. Most of the picture is never processed at full size, which is why it can run in real time on 4K video.

The authors also released two large matting datasets, made with green-screen footage and studio photos, to train and test this kind of model.

TL;DR

• Background matting: the input frame and one captured frame of the empty background predict the alpha matte and the foreground colour.
• Two networks: a base network at 1/c resolution (c = 4 for training) and a refinement network that re-predicts the k most error-prone 4x4 patches §4.
• Author-reported: 60.0 fps at HD and 33.2 fps at 4K with ResNet-50 on an RTX 2080 Ti, against 7.8 fps for BGM at 512x512 tab. 3.
• Code is MIT, but the released training and inference defaults differ from the paper in several places (see Paper vs code).

Problem

Matting at full resolution with one deep network is too expensive in compute and memory. Trimap-based methods need a hand-drawn or generated trimap §5. Background matting needs a captured background, and earlier variants ran at 512x512 only §5.2, §5.3. The paper aims at real-time matting at HD and 4K with strand-level hair detail, under the assumption that the camera and background stay nearly still §6, limitations.

Method

Overview

The input image I and captured background B are downsampled by c to I_c and B_c. The base network G_base predicts a coarse alpha, a foreground residual, an error map E_c and 32 hidden channels H_c §4. The refinement network G_refine uses H_c, I and B. It refines only the patches where E_c is large, at half resolution and then full resolution, and outputs the final alpha and foreground residual §4, fig. 2.

The foreground is stored as a residual F^R = F - I, recovered as F = max(min(F^R + I, 1), 0) §4. The composite over a new background B' is I' = alpha F + (1 - alpha) B' §4.

Base network §4.1 app. B.1

• Encoder: ResNet-50 by default, with ResNet-101 and MobileNetV2 as options §4.1. The first convolution takes 6 channels (image and background stacked) app. B.1.
• ASPP with dilation rates 3, 6 and 9 after the encoder §4.1 app. B.1.
• Decoder: bilinear upsampling, concatenation with the encoder skip, 3x3 convolution, BatchNorm, ReLU, with channel widths 128, 64, 48, then a final 37-channel output app. B.1. The 37 channels are 1 alpha, 3 foreground residual, 1 error and 32 hidden features app. B.1.
• Clamps: alpha and error are clamped to [0, 1]; hidden features pass ReLU app. B.1.

Refinement network §4.2 app. B.1, B.2 fig. 3

1. Build the error map at 1/4 resolution, E_4, so that each entry covers a 4x4 patch of the full-resolution image. Select the top k entries §4.2.
2. At half resolution, concatenate the coarse hidden features, alpha and foreground residual (36 channels) with I and B (6 channels), giving 42 channels app. B.1.
3. For each selected location, crop an 8x8 window. Two 3x3 valid convolutions with BatchNorm and ReLU (42 to 24 to 16 channels) reduce it to 4x4 §4.2 app. B.1.
4. Upsample the 4x4 result to 8x8 with nearest-neighbour upsampling app. B.2. Concatenate it with the full-resolution 8x8 windows of I and B (22 channels). Two 3x3 valid convolutions (22 to 12 to 4) give the refined 4x4 alpha and foreground residual §4.2 app. B.1.
5. Bilinearly upsample the coarse alpha and residual to full resolution and overwrite the selected 4x4 patches §4.2.

Refining k patches touches 16k pixels at full resolution §4.2.

# Inference: coarse pass, then patch refinement  [§4] [§4.1] [§4.2]
# Input:  image (B, 3, H, W) in [0, 1]; background (B, 3, H, W)
# Output: alpha (B, 1, H, W); foreground (B, 3, H, W)
# H and W must be divisible by 4  [code: model/model.py#L163-L164]

def matte(image, background, c, k):
    # Downsample both inputs by c for the base network  [§4]
    image_small = resize(image, 1 / c)
    background_small = resize(background, 1 / c)

    # Base network: 6 input channels, outputs at 1/c  [§4.1] [app. B.1]
    coarse = base_network(concat(image_small, background_small))
    alpha_coarse = clamp(coarse.alpha, 0, 1)            # (B, 1, h, w)
    residual_coarse = coarse.residual                   # (B, 3, h, w)
    error_coarse = clamp(coarse.error, 0, 1)            # (B, 1, h, w)
    hidden = relu(coarse.hidden)                        # (B, 32, h, w)

    # Choose the k cells with the largest predicted error  [§4.2]
    # [code: model/refiner.py#L93-L95, #L163-L179]
    error_quarter = resize(error_coarse, 1 / 4)
    selected = top_k_cells(error_quarter, k)

    # Half-resolution pass on 8x8 windows around each cell  [§4.2] [app. B.1]
    # [code: model/refiner.py#L101-L110]
    half = concat(hidden, alpha_coarse, residual_coarse)
    half = resize(half, 1 / 2)
    half_input = resize(concat(image, background), 1 / 2)
    windows = crop_8x8(concat(half, half_input), selected)  # 42 channels
    features = conv_3x3_valid(windows, out=24)
    features = conv_3x3_valid(features, out=16)             # 4x4 cells

    # Nearest upsampling to 8x8, then join full-resolution windows  [app. B.2]
    # [code: model/refiner.py#L114-L119]
    features = upsample_nearest(features, 8)
    full_windows = crop_8x8(concat(image, background), selected)
    features = concat(features, full_windows)                # 22 channels
    features = conv_3x3_valid(features, out=12)
    patch_out = conv_3x3_valid(features, out=4)              # alpha, residual

    # Write refined patches over bilinearly upsampled coarse outputs  [§4.2]
    # [code: model/refiner.py#L122-L126]
    alpha = upsample_bilinear(alpha_coarse, full_size)
    residual = upsample_bilinear(residual_coarse, full_size)
    alpha = scatter_patches(alpha, patch_out.alpha, selected)
    residual = scatter_patches(residual, patch_out.residual, selected)

    # Recover the foreground from the residual  [§4]
    foreground = clamp(residual + image, 0, 1)
    return alpha, foreground

Training losses §4.3 eq. 1 eq. 2 eq. 3 eq. 4 eq. 5

Alpha loss, with * marking ground truth eq. 1:

# Alpha: L1 on values plus L1 on Sobel gradients  [eq. 1]
def alpha_loss(pred_alpha, true_alpha):
    return (l1(pred_alpha, true_alpha)
            + l1(sobel(pred_alpha), sobel(true_alpha)))

Foreground loss, counted only where the true alpha is above zero eq. 2:

# Foreground: L1 only where true alpha > 0  [eq. 2]
def foreground_loss(pred_fgr, true_fgr, true_alpha):
    mask = true_alpha > 0                  # Boolean mask
    return l1(pred_fgr * mask, true_fgr * mask)

Error map target and base loss. The target is the absolute alpha error, which the paper says changes during training §4.3, eq. 3:

# Base loss: alpha + foreground + error map  [eq. 4]
# Error target is |alpha - alpha*|, with the prediction detached in code
# [code: train_base.py#L219-L225]
def base_loss(pred_alpha, pred_fgr, pred_err, true_alpha, true_fgr):
    target_err = abs(pred_alpha.detach() - true_alpha)
    return (alpha_loss(pred_alpha, true_alpha)
            + foreground_loss(pred_fgr, true_fgr, true_alpha)
            + mse(pred_err, target_err))

Refinement loss is alpha plus foreground loss on the final outputs eq. 5:

# Refinement loss  [eq. 5]; the released script adds coarse terms, see Paper vs code
def refine_loss(pred_alpha, pred_fgr, true_alpha, true_fgr):
    return (alpha_loss(pred_alpha, true_alpha)
            + foreground_loss(pred_fgr, true_fgr, true_alpha))

Training procedure §4.3 app. D

1. Train the base network alone until convergence. No stopping criterion is stated §4.3.
2. Add the refinement network and train jointly §4.3.
3. Initialise backbone and ASPP from DeepLabV3 weights pre-trained on ImageNet and Pascal VOC §4.3.
4. Dataset order: VideoMatte240K (base, then joint), PhotoMatte13K (joint), Distinctions-646 (joint). AIM is test-only, because adding it as a fourth stage lowered quality §4.3 §6, tab. 5.
5. Augmentation is applied independently to the foreground and background layers of each sample app. D.1. Each sample is composited onto a background as I = alpha F + (1 - alpha) B app. D.1.
6. Mixed-precision training. Multi-GPU runs use data parallelism with synchronised BatchNorm app. D.

Hyperparameters

Results

Author-reported. Setup: composites of AIM, Distinctions and PhotoMatte85 test samples onto backgrounds, with metrics from §5.1. Trimap-based methods (DIM, FBA) use a manual trimap, marked with a dagger in the paper. Lower is better for SAD, MSE, Grad and Conn. The FG column is foreground MSE. MSE values are scaled by 10^3 §5.1.

Quantitative comparison tab. 1

The text says the method beats BGM on all datasets and is slightly worse than FBA, which needs manual trimaps §5.1. On AIM and PhotoMatte85, FBA is lower than Ours on every alpha metric and on foreground MSE. On Distinctions, Ours is lower than FBA on all four alpha metrics and on foreground MSE (6.10 against 12.51), so the "slightly worse than FBA" statement holds only for the first two sets.

Speed tab. 3. Measured on an RTX 2080 Ti as PyTorch pass-through, FP32, batch 1, without data transfer.

The caption does not give the value of c used for the 4K rows tab. 3.

Model size tab. 4

The text says the ResNet-50 model has 55.7% of BGM's parameters §5.3.

User study tab. 2. 40 participants rated 34 frames, Ours against BGM, in random order §5.2.

Ours was rated better 59% of the time against 23% for BGM. For 4K and larger, Ours was rated better 75% of the time against 15% for BGM §5.2. The study was run only against BGM.

Ablations (Distinctions test set unless noted)

• Training data, on the AIM test set tab. 5. Ours: SAD 12.86, MSE 12.01, Grad 8426, Conn 11116, FG MSE 5.31. Adding AIM as a fourth stage: SAD 14.19. Without PhotoMatte13K: SAD 14.05. Without VideoMatte240K: SAD 15.17. Without Distinctions: SAD 15.95 and FG MSE 14.36.
• Backbone and refinement kernel tab. 6. ResNet-50 with 3x3 refinement: SAD 9.19, MSE 7.08, Grad 6345, Conn 7216, FG MSE 6.10. ResNet-101 with 3x3: SAD 9.30, MSE 6.82, Grad 6191, Conn 7128, FG MSE 7.68. MobileNetV2 with 3x3: SAD 10.53, MSE 9.62, Grad 7904, Conn 8808, FG MSE 8.19. ResNet-50 with 1x1: SAD 9.36, MSE 8.06, Grad 7319, Conn 7395, FG MSE 6.92.
• Refined patches k tab. 7. HD FPS with ResNet-50: k = 2,500: 62.9; 5,000: 60.0; 7,500: 55.7; full image: 42.8.
• Accuracy against refined area: most of the gain comes from refining 5% to 10% of the image §6, fig. 7.

Supplementary speed app. E. RTX 2060 Super, ResNet-50: 42.8 fps HD and 23.3 fps 4K. MobileNetV2: 75.6 fps HD and 31.3 fps 4K. On the RTX 2080 Ti, MobileNetV2 at HD, batch 8, FP16 reaches 200.0 fps, and at 4K, batch 8, FP16 reaches 64.2 fps app. E.

Paper vs code

Code: official repository PeterL1n/BackgroundMattingV2, MIT licence, at commit a8e82df9f594578edf287dbb2b289ebcc50fbf00. The abstract links a project page (abstract). That page returned no readable code link, so the repository was found by GitHub search. Its owner's profile lists the name Peter Lin and links a LinkedIn page under the first author's name (Shanchuan Lin); the repository README does not name the authors. Paths below are relative to the repository root.

Items that match the paper: the refiner channel widths (42 input, 24, 16, 22, 12, 4) refiner.py match app. B.1. Nearest upsampling of patches refiner.py matches app. B.2. The 32 hidden channels with ReLU model.py match app. B.1. Base and joint learning rates and batch sizes match §4.3 [code: train_base.py#L53, #L128-L132] train_refine.py.

Discrepancies:

1. Base training crop. Paper: height and width each uniform in [1024, 2048] per minibatch §4.3. Base script: uniform in [256, 512) after resizing to 512x512 train_base.py train_base.py. The refinement script uses the paper's [1024, 2048) range [code: train_refine.py#L265-L275].
2. Foreground scale in base training. Paper: scale 0.3 to 1 for foreground and background app. D.1. Base script: foreground scale 0.4 to 1, background scale 1 to 2 train_base.py train_base.py. The background range is not stated in the paper. The refinement script uses 0.3 to 1 for foreground train_refine.py.
3. Background jitter probability. Paper: the background misalignment group (rotation, translation, colour including hue ±0.1) is applied 30% of the time app. D.1. Code: the colour jitter alone is applied to 80% of the samples, while the affine misalignment keeps 30% train_base.py train_base.py (same in train_refine.py#L200-L208).
4. Noise. Paper: Gaussian noise with σ² ≤ 0.03, probability not stated app. D.1. Code: σ = 0.03 times a uniform draw, applied with probability 0.4 to the composite and the background, each with its own draw [code: train_base.py#L165-L170].
5. Refinement loss. Paper: L_refine = L_alpha + L_F eq. 5. Code adds L1 and Sobel terms for the coarse alpha and foreground, and an error term whose target is the absolute error of the upsampled coarse alpha, not the refined alpha. That target is not detached train_refine.py.
6. Error loss form. Paper eq. 3 writes an L2 norm; the text calls it mean squared error §4.3. Code uses MSE with a detached prediction in the base script train_base.py.
7. Evaluation defaults. Paper: k = 20,000 for photos and c = 8 for 4K §5. Repository defaults are backbone scale 0.25 and 80,000 sampled pixels (k = 5,000) inference_video.py inference_images.py. Photo and 4K settings need command-line overrides; the repo does not state them.
8. Refinement mode. Code's default refine mode is "sampling"; a threshold option exists but is set to 0.7 train_refine.py. Paper describes top-k only §4.2.
9. ASPP width. Paper does not state ASPP output channels. The code calls torchvision's ASPP without an output-width argument, so it uses the library default of 256 model.py model.py.

Paper-internal check: the BGM row in tab. 5 (AIM test) has BGM's alpha metrics from tab. 1 (SAD 16.07, MSE 21.00, Grad 15371, Conn 14123), but its FG MSE of 42.84 is the BGM_a value in tab. 1 (BGM's own is 47.98) tab. 5 tab. 1. The paper does not explain this. The ResNet-50 row of tab. 6 and the Distinctions row of tab. 1 agree.

Released weights are on the repository's GitHub releases page and a Google Drive folder [README.md#L52-L53]. Their licence was not checked.

Pitfalls

• Background must be captured on the same camera and nearly still. The paper's own limitations: handheld input only works with homography alignment of the background per frame, and only for small motion. Substantial subject shadows on the background, colour matching between subject and background, and highly textured backgrounds cause failures §6, fig. 8.
• Set the evaluation flags for photos and 4K yourself. The script defaults give backbone scale 0.25 and k = 5,000 inference_video.py. The paper's photo and 4K settings differ (see Paper vs code, item 7). Speed and accuracy change with k §5, tab. 7.
• Patches selected by error. The top-k selection can select fewer than k patches when the error map is zero in many places [code: model/refiner.py#L178-L179]. The paper does not discuss this.
• Error-map supervision moves during training. The target |alpha - alpha*| changes as alpha improves §4.3. The base script detaches the prediction for the target, but the refinement script does not train_refine.py.
• Crop sizes in base training. The base script trains on 256 to 512 crops, not the 1024 to 2048 range in the text. Follow the paper if you want the described behaviour, and note the mismatch with the released script.
• Speed numbers are pass-through, not end-to-end. Table 3 times the network without data transfer tab. 3. The repository's README says its video script is not real-time, because encoding and decoding run without hardware acceleration [README.md#L10]. Plan for hardware encoding and parallel frame loading.
• Dataset licences. The paper does not state a licence for VideoMatte240K or PhotoMatte85. PhotoMatte13K is not released for privacy and licensing reasons §3. Check the dataset licence before using the data or weights commercially. Only the background test set is said to carry a CC licence §3.
• Mixed precision is used in training app. D. FP16 inference is measured separately app. E. Inference precision for the headline speed is FP32 tab. 3.
• Input size. The model requires height and width divisible by 4 [code: model/model.py#L163-L164].

Lineage

• Builds on: Background Matting (BGM), cited as sengupta2020background, which needs a captured background and is run at 512x512 in the paper's comparisons §5.1 §5.2. Architecture follows DeepLabV3 and DeepLabV3+ with ResNet and MobileNetV2 encoders §4.1.
• Compared against: Deep Image Matting (DIM) and FBA Matting (FBA), both trimap-based, and BGM (adapted, BGM_a) §5.
• Superseded by: not stated in this card.