Robust High-Resolution Video Matting with Temporal Guidance

In plain words

Cutting a person out of a video, so that the background can be swapped, is harder than it looks. Hair, fast arm movements and camera motion all make the edge jump from frame to frame. Most earlier tools treat every frame as a separate picture, so they flicker.

This paper's model keeps a small memory that is passed from one frame to the next. The memory lets it remember what the background looked like when the person was not in front of it, and it keeps the edge steady over time. The model is also trained on ordinary person-segmentation photos as well as on synthetic cut-outs, which makes it less thrown by real scenes. It needs no extra input such as a hand-drawn outline or a photo of the empty room. A separate step sharpens the result at very high resolutions, so the model can run on 4K and HD video.

TL;DR

• Author-reported. A recurrent matting network on MobileNetV3-Large features, with a Deep Guided Filter for high resolution, lowers alpha error against FBA, BGMv2 and MODNet on three composited test sets §5.1 tab. 1.
• The network has 3.749 million parameters, 14.5 MB in FP32 tab. 3. Reported speed on a GTX 1080Ti is 76.5 FPS at 4K and 104.2 FPS at HD tab. 4.
• Training alternates a matting pass with a segmentation pass on public human-segmentation data. Ablations report that removing the segmentation objective drops COCO human-mIoU from 60.88 to 38.24 §6.2 tab. 5.
• Four training stages with a fixed learning-rate schedule are given in full. The public code matches the paper's learning rates, but runs 20 epochs in stage 1 where the paper says 15 §4.3 train.py.
• Several details are not in the paper but are in the code: the foreground is a residual added to the input frame, the encoder normalises with ImageNet statistics, and augmentation probabilities are set per dataset model.py mobilenetv3.py videomatte.py.

Problem

Background replacement for people needs an alpha matte, which gives each pixel a transparency value, and a foreground colour. Most neural matting methods rely on a trimap (a hand-drawn region of unknown alpha) or on a captured background image. Some are trained for portraits only, and background-based methods cannot handle dynamic backgrounds §2. Methods that treat frames independently flicker §1.

The paper targets three cases: real-time use without auxiliary inputs, temporally coherent video, and high resolution. Its premise is that matting and human segmentation are related, so a segmentation objective can stop the network from overfitting to synthetic composites §1 §4.

Method

The model has three parts: a per-frame encoder, a recurrent decoder, and an optional Deep Guided Filter (DGF) that upsamples the result to high resolution §3 fig. 1.

1. Downsample. For 4K or HD input, the frame is scaled by a factor s before the encoder-decoder runs. The DGF later takes the full-resolution frame as guidance §3 §3.3 model.py.
2. Encoder. MobileNetV3-Large runs on each frame and produces features at 1/2, 1/4, 1/8 and 1/16 of the working resolution. The last block uses dilated convolutions without a downsampling stride. An LR-ASPP module compresses the 1/16 features §3.1 app. B mobilenetv3.py lraspp.py.
3. Bottleneck. At 1/16, a ConvGRU runs on half of the channels. The other half passes through unchanged. The two halves are then concatenated §3.2 app. B decoder.py.
4. Upsampling blocks. Three blocks at 1/8, 1/4 and 1/2 each concatenate three inputs: the upsampled output of the previous block, the encoder feature of the same scale, and the input frame average-pooled to that scale. A convolution, batch normalisation and ReLU then merge the features. A ConvGRU again runs on half of the channels §3.2 app. B decoder.py.
5. Output block. No ConvGRU. Two convolution, batch-norm and ReLU stacks produce 16 features. A 1×1 projection gives 3 foreground channels, 1 alpha channel and 1 segmentation channel §3.2 app. B decoder.py model.py.
6. Deep Guided Filter (high resolution only). The low-resolution alpha, foreground and hidden features are upsampled to full resolution using the full-resolution frame as guidance. The DGF is a small learned module with 16-filter 1×1 convolutions. Without downsampling, the encoder-decoder runs alone §3.3 app. B deep_guided_filter.py.
7. Outputs. The foreground is predicted as a residual on the input frame, then alpha and foreground are clamped to [0, 1]. The clamp runs in training and inference app. B model.py.

The ConvGRU, as written in the paper §3.2 eq. 1:

z_t = \sigma(w_{zx} * x_t + w_{zh} * h_{t-1} + b_z) r_t = \sigma(w_{rx} * x_t + w_{rh} * h_{t-1} + b_r) o_t = \tanh(w_{ox} * x_t + w_{oh} * (r_t \odot h_{t-1}) + b_o) h_t = z_t \odot h_{t-1} + (1 - z_t) \odot o_t

The code uses the same gates, but the update gate weights the candidate state, not the previous state decoder.py. See Pitfalls.

The forward pass for one clip, in pseudocode. Names are ours.

# Forward pass over a clip  [§3] [code: model/model.py#L49-L68]
# Input:  frames (B, T, 3, H, W) in [0, 1]; downsample ratio s
# Output: foreground (B, T, 3, H, W), alpha (B, T, 1, H, W)

def forward(frames, s, states):
    # Scale the frames for the encoder-decoder  [§3] [app. B]
    small = bilinear_resize(frames, scale=s)
    # Per-frame features at 1/2, 1/4, 1/8, 1/16  [§3.1]
    feats = mobilenet_encoder(small)
    # LR-ASPP on the 1/16 features  [§3.1]
    feats[3] = lraspp(feats[3])
    # Bottleneck, then three upsampling blocks with ConvGRU  [§3.2]
    hidden, states = recurrent_decoder(small, feats, states)
    # Predict a foreground residual and alpha  [app. B]
    residual, alpha = project_matting(hidden).split([3, 1])
    # High resolution only: refine the residual and alpha with the DGF  [§3.3]
    if s < 1:
        residual, alpha = dgf(frames, small, residual, alpha, hidden)
    # Foreground is the residual added to the input frame, then clamped  [app. B]
    foreground = clamp(residual + frames, 0, 1)
    alpha = clamp(alpha, 0, 1)
    return foreground, alpha, states

Here is a ConvGRU step on one half of the channels §3.2 eq. 1. The gates follow the code's convention, not the paper's decoder.py.

# One ConvGRU step on half of the channels  [§3.2] [§3.2 eq. 1]
# Input:  x (B, C, H, W); h_prev (B, C, H, W), zeros at the first frame

def convgru_step(x, h_prev):
    # Reset and update gates from the input and the old state  [§3.2 eq. 1]
    reset, update = sigmoid(conv_gates(concat(x, h_prev))).split(C)
    # Candidate state from the input and the reset-gated old state  [§3.2 eq. 1]
    candidate = tanh(conv_candidate(concat(x, reset * h_prev)))
    # Blend: the update gate weights the candidate  [code: model/decoder.py#L171]
    h = (1 - update) * h_prev + update * candidate
    return h, h

Training uses four stages with the same loss, described below §4.3 alg. 1. Each iteration runs one low-resolution matting pass and, from stage 3, one high-resolution pass with DGF. It then runs one segmentation pass, which alternates between video and image batches.

# Four-stage training loop  [§4.3] [alg. 1]
# Stage 1: VideoMatte, low resolution, no DGF, T = 15, 15 epochs (paper)
# Stage 2: T = 50, 2 more epochs, learning rates halved
# Stage 3: adds DGF and a high-resolution pass, 1 epoch
# Stage 4: D646 and AIM images, 5 epochs, decoder learning rate raised

for stage in [1, 2, 3, 4]:
    for epoch in epochs_of(stage):
        for iteration in iterations(epoch):
            # Low-resolution matting pass: B clips, T frames, h x w  [§4.3]
            matting_step(B, T, h, w, downsample=1)
            if stage in [3, 4]:
                # High-resolution matting pass with DGF  [§4.3]
                matting_step(B, T_hat, h_hat, w_hat, downsample=0.25)
            if iteration % 2 == 0:
                # Video segmentation batch  [§4.3]
                segmentation_step(B, T, h, w)
            else:
                # Image segmentation batch, each image treated as T = 1  [§4.3]
                segmentation_step(B * T, 1, h, w)

Matting loss over all T frames of a clip §4.4. The paper's weights are 1, 1, 5, 1 and 5 for the five terms §4 eq. 6:

# Matting loss  [§4.4] [§4 eq. 6]
# Inputs: predicted and true alpha, foreground (B, T, C, H, W)

def matting_loss(pred_alpha, true_alpha, pred_fg, true_fg):
    # Alpha L1 and Laplacian pyramid loss  [§4 eq. 1] [§4 eq. 2]
    alpha_l1 = mean_abs(pred_alpha - true_alpha)
    alpha_lap = laplacian_pyramid_loss(pred_alpha, true_alpha)
    # Alpha temporal coherence over consecutive frames  [§4 eq. 3]
    alpha_tc = coherence(pred_alpha, true_alpha)
    # Foreground only where the true alpha is positive  [§4 eq. 4]
    mask = true_alpha > 0
    fg_l1 = mean_abs(mask * (pred_fg - true_fg))
    # Foreground temporal coherence, same mask  [§4 eq. 5]
    fg_tc = coherence(mask * pred_fg, mask * true_fg)
    return alpha_l1 + alpha_lap + 5 * alpha_tc + fg_l1 + 5 * fg_tc

Segmentation loss is binary cross-entropy on the segmentation head, applied to the human class only §4 eq. 7 §4.4. The segmentation loss has no explicit weight in the paper train_loss.py.

Hyperparameters

Batch size is 4 in total, across four V100 32 GB GPUs §4.3 app. C. The code sets one clip per GPU by default train.py.

Data §4.1 §4.2:

Augmentation §4.1 app. D: motion augmentation (affine, scale, rotation, shear, colour, noise, blur) changes continuously with easing curves. Temporal augmentation includes reversal, speed changes, pauses and frame skipping. Horizontal flip, grayscale and sharpening are applied the same way to every frame. Image datasets receive added artificial motion. Probabilities are not stated in the paper videomatte.py imagematte.py.

Results

Author-reported. Setup: a composited benchmark built from VideoMatte (VM), D646 and AIM test samples, each placed on 5 video and 5 image backgrounds, with 100-frame clips §5.1. Compared methods are FBA (trimap from DeepLabV3), BGMv2 (MobileNetV2, first frame's background) and MODNet (official weights, because retraining gave worse results) §5.1. Speed is FP32 throughput on a GTX 1080Ti tab. 4.

Low-resolution composition, alpha and foreground tab. 1. Bold marks the best value. MAD and MSE are ×1e3, dtSSD is ×1e2:

Reported in the caption: alpha is best for this method on all sets. Foreground is behind BGMv2 but ahead of FBA and MODNet tab. 1 §5.1.

High-resolution alpha against MODNet with the non-learned Fast Guided Filter (FGF), with the Conn metric dropped for cost tab. 2 §5.1:

Model size, FP32 tab. 3 §5.3:

Speed tab. 4 §5.3. Throughput is FP32 on a GTX 1080Ti, with models converted to TorchScript. GMACs is only a rough measure, since it counts convolutions and misses resize operations:

Segmentation on COCO validation, humans only, mIoU tab. 5 §6.2:

DGF against FGF on D646, HD speed tab. 6 §6.3:

Static against dynamic backgrounds on VM tab. 7 §6.4. BGMv2 receives the ground-truth static background, and the paper says that does not hold in use:

Larger variant with ResNet-50 backbone and more decoder channels, on VM at HD tab. 8 §6.5:

Temporal ablation §6.1 fig. 3: without recurrence, quality and consistency worsen. MODNet's per-frame error fluctuates over time, while this model's error drops in the first 15 frames and then stays stable. Figures 4 and 5 show a handrail kept steady where MODNet flickers, and recurrent channels that reconstruct the background as it is revealed and forget memory at shot cuts.

Paper vs code

Official code: github.com/PeterL1n/RobustVideoMatting at commit 53d74c6826735f01f4406b5ca9075eee27bec094, GPL-3.0 (copyleft). Only the paraphrased logic and line citations are used here. No code is quoted. The repository owner is PeterL1n, whose GitHub profile links the first author's LinkedIn page. The abstract points to a project page, which we could not open, so the link to the repository is verified through the owner.

• Stage 1 length. Paper: 15 epochs §4.3. Code: --epoch-end 20, so epochs 0 to 19 train.py. Stage 2 (2 epochs) and stage 4 (5 epochs) match.
• Learning rates. All four stages match the paper: stage 1 1e-4 and 2e-4, stage 2 halved, stage 3 DGF 2e-4 and rest 1e-5, stage 4 decoder 5e-5 train.py train.py train.py train.py.
• Foreground. Paper gives the foreground as a prediction. Code predicts a residual added to the input frame, then clamps model.py.
• Gate convention. Paper §3.2 eq. 1 writes the new state as the update gate times the old state. Code writes the update gate times the candidate decoder.py.
• Temporal coherence. Paper writes an L2 norm of the frame-difference error §4 eq. 3 §4 eq. 5. Code uses mean squared error on frame differences, times 5 train_loss.py train_loss.py. The paper's L1 terms are means in code train_loss.py train_loss.py.
• Laplacian. Matches the paper's 2^(s−1)/5 weights. Code adds a 5×5 Gaussian with reflect padding and crops to even sizes; neither is stated in the paper train_loss.py.
• Optimiser steps. Algorithm 1 lists passes without saying whether each one updates weights. Code calls the optimiser after every pass, so stage 3 and 4 iterations take three steps: low-resolution matting, high-resolution matting, segmentation train.py train.py train.py.
• Crop sizes. Paper: sides sampled between 256 and 512, and between 1024 and 2048 at high resolution §4.3. Code draws integers from half the full size to one less than the full size, so the ranges are 256 to 511 and 1024 to 2047. The crop is always centred train.py.
• Augmentation. Paper gives no probabilities. Code uses: VideoMatte fgr affine 0.3, bgr affine 0.3, noise 0.1, colour jitter 0.3, grayscale 0.02, sharpness 0.1, blur 0.02, flip 0.5, pause 0.03 videomatte.py. ImageMatte uses 0.95, 0.3, 0.05, 0.3, 0.03, 0.05, 0.02, 0.5, 0.03 imagematte.py. Frame-speed choices default to 0.5 to 5 augmentation.py. Segmentation video uses speed 1 only train.py.
• Input normalisation. Code normalises the encoder input with ImageNet mean and standard deviation. The paper does not state this mobilenetv3.py.
• DGF filter. Code computes the guide statistics with a fixed 3×3 mean filter, not a learned one deep_guided_filter.py.
• Test-time downsampling. Code's automatic ratio is min(512 / max side, 1) inference.py. That gives 0.133 at 4K and 0.267 at HD, where Table 4 uses 0.125 and 0.25.
• Segmentation schedule. Code alternates by the global step counter, step % 2, across epochs train.py. Paper says odd and even iterations alg. 1.
• Architecture. The encoder, decoder channels and LR-ASPP sizes in Table 9 match the code's MattingNetwork for both variants model.py tab. 9.
• Segmentation output. Code's segmentation head returns logits, trained with BCE-with-logits. The sigmoid is applied only for TensorBoard images model.py train_loss.py train.py.

Pitfalls

• Gate convention. §3.2 eq. 1 and the code disagree on which side the update gate weights. Pick one form for training and inference, and keep it consistent with any released weights decoder.py.
• Residual foreground. The foreground is the input plus a learned residual, then clamped. The DGF refines the residual, not the foreground. Code and paper agree on the clamp only model.py.
• Three optimiser steps per iteration. Stages 3 and 4 do three updates per loop in code. A reimplementation that backprops all three losses into one step will train differently train.py.
• Coherence scale. The coherence terms are squared differences of consecutive-frame values. The paper's norm notation does not fix the scale, so the code's means set the magnitude of the weight-5 terms.
• Stage 1 length. Paper and code differ by 5 epochs in stage 1. The sources do not say which run produced the reported numbers.
• Equation numbers. Equation numbers restart in each section, as the article class does. Citations here are section-plus-number (for example §4 eq. 3), derived from the source files, not from a rendered PDF.
• Data counts. The 420 / 15 train / validation split is one combined split of D646 and AIM. Test sets are 11 and 10 images.
• Typesetting. One MSE value in Table 2 is printed with a decimal comma (5,54) in the source. It is read here as 5.54.
• Baselines. MODNet was used with official weights, because retraining gave worse results. FBA's training code was not released §5.1.
• Speed numbers. The paper reports FP32 tensor throughput on a GTX 1080Ti, with models converted to TorchScript tab. 4. The DGF and FGF variants differ by 0.001 M parameters and about 5 FPS at HD tab. 6.
• Training cost. About 18, 2, 8 and 14 hours per stage on 4 V100 32 GB, with mixed precision app. C. The high-resolution pass needs very short sequences (T̂ = 6) to fit in GPU memory §4.3.
• Background motion. Motion augmentation in the benchmark applies only to image foregrounds and backgrounds, at lower strength than in training. Video backgrounds are not augmented app. D §5.1.
• Foreground on the dataset. The paper does not compute foreground error on VideoMatte, because its foreground ground truth is noisy §5.1. Foreground numbers are therefore only from D646 and AIM.
• Limits. The paper reports weak results when people are in the background or the scene is complex §6.6 fig. 6.
• Licence. The released code is GPL-3.0. Re-implementers should write their own code from the paper and the logic described here.

Lineage

• Builds on: MobileNetV3 with LR-ASPP §2 §3.1; ConvGRU §2 §3.2; Deep Guided Filter §3.3; FBA, BGMv2 and MODNet as baselines §5.1; DVM for temporal coherence §4.4.
• Compared against: FBA, BGMv2 and MODNet on composition, and MODNet with FGF at high resolution §5.1.
• Superseded by: not known from content already on this wiki.