--- kind: concept slug: image-matting title: Image matting summary: >- Image matting estimates a per-pixel alpha matte that separates a foreground subject from its background, including soft edges such as hair. Variants differ in the extra input they need: a manual trimap, or a captured empty background. Background matting with patch refinement reports real-time HD and 4K speeds, but assumes a nearly static camera. tags: [matting, segmentation, real-time, background-replacement] papers: ["2012.07810"] related: [semantic-segmentation, dilated-convolution, data-augmentation] status: draft updated_at: 2026-11-03 --- ## Overview Matting predicts an alpha value in [0, 1] for each pixel, where 1 means fully foreground and 0 means fully background. The composite over a new background is I = alpha F + (1 - alpha) B [arxiv:2012.07810 §4]. Hard segmentation gives only 0 or 1, so it cannot represent hair or other see-through edges. The invariant every variant keeps is the compositing equation above. The variants differ in what extra input they need to solve for alpha: - **Trimap-based:** a user or a generator marks certain foreground, certain background and an unknown band. Deep Image Matting (DIM) and FBA Matting are the trimap-based methods the card's paper compares against. The paper notes they need a hand-drawn or generated trimap [arxiv:2012.07810 §5]. - **Captured background:** one frame of the empty scene is taken with the same camera. Background Matting (BGM; Sengupta et al., 2020) is the baseline here, run at 512x512 in the paper's comparisons [arxiv:2012.07810 §5.1]. Card: [Real-Time High-Resolution Background Matting](/papers/2012.07810). ## Variants ### Background matting with patch refinement Two networks run in sequence. A base network works on a downsampled copy of the frame and the background, and predicts a coarse alpha, a foreground residual and an error map [arxiv:2012.07810 §4]. A refinement network then re-predicts only the k patches with the largest predicted error, at half and full resolution [arxiv:2012.07810 §4.2]. Most pixels are never processed at full size. The authors report 60.0 fps at HD and 33.2 fps at 4K with a ResNet-50 backbone on an RTX 2080 Ti, against 7.8 fps for BGM at 512x512. These figures are pass-through timings without data transfer [arxiv:2012.07810 tab. 3]. Card: [Real-Time High-Resolution Background Matting](/papers/2012.07810). ### Trimap-based methods (named, not covered by a card) DIM and FBA take a trimap as input. The paper reports FBA as the most accurate method on AIM and PhotoMatte85 among the methods it compares, but its speed figures are low: 3.3 fps at HD, or 2.9 fps when the trimap is generated automatically [arxiv:2012.07810 tab. 1] [arxiv:2012.07810 tab. 3]. No card for these papers exists yet, so their details are not given on this page. ## When to use which | Situation | Use | Why | |---|---|---| | A clean frame of the empty background can be captured with a nearly static camera, and real-time HD or 4K matting is needed | Background matting with patch refinement (ResNet-50) | Author-reported 60.0 fps HD and 33.2 fps 4K, against 7.8 fps for BGM at 512x512 [arxiv:2012.07810 tab. 3]. Lower SAD than BGM on AIM, Distinctions and PhotoMatte85 [arxiv:2012.07810 tab. 1] | | Same setting, but compute or memory is the tightest limit | Same method with a MobileNetV2 encoder | 5.0 M parameters, 19.10 MB [arxiv:2012.07810 tab. 4]. 100.6 fps HD [arxiv:2012.07810 tab. 3]. Distinctions SAD 10.53, against 9.19 for ResNet-50 [arxiv:2012.07810 tab. 6] | | The camera moves or is handheld, and a background frame cannot be aligned | Not covered by the card's variant | The paper says handheld input needs per-frame homography alignment and works only for small motion [arxiv:2012.07810 §6] | | No background frame, but a trimap can be made by hand or by a generator, and speed is not critical | Trimap-based methods (DIM, FBA) | Lowest SAD on AIM and PhotoMatte85 in the paper's table [arxiv:2012.07810 tab. 1]. Slow at HD [arxiv:2012.07810 tab. 3] | The speed and accuracy figures are the authors' own. The user study compared only against BGM: 59% of 40 raters preferred the new method, 23% preferred BGM [arxiv:2012.07810 tab. 2]. Dataset licences are not stated in the paper for VideoMatte240K or PhotoMatte85, so check them before commercial use [arxiv:2012.07810 §3]. ## Implementation sketch The common core of the captured-background variant. Refinement details, such as window sizes and channel counts, differ by implementation and are in the card. ```pseudocode # Background matting with patch refinement [arxiv:2012.07810 §4] # Input: frame, empty_background (same camera, same size) # Output: alpha, foreground colour def matte(frame, empty_background, c, k): # Coarse pass on a downsampled copy, factor c [arxiv:2012.07810 §4] small = downsample(concat(frame, empty_background), c) alpha_c, fg_c, err_c, hidden = base_net(small) # Choose the k cells with the largest predicted error [§4.2] cells = top_k(upsample(err_c, cell_size), k) # Re-predict only those cells, at half and full resolution [§4.2] alpha, fg = refine_cells(frame, empty_background, cells, alpha_c, fg_c, hidden) return alpha, fg ``` ## Contradictions - **Within the paper, on FBA.** The text says the method is "slightly worse than FBA" [arxiv:2012.07810 §5.1]. On Distinctions, the table shows the method lower than FBA on all four alpha metrics (SAD 9.19 against 11.03), so the statement holds only for AIM and PhotoMatte85 [arxiv:2012.07810 tab. 1]. - **Within the paper, on BGM foreground error.** The BGM row of the AIM ablation shows FG MSE 42.84, which is the BGM_a value in the main table. BGM's own value there is 47.98 [arxiv:2012.07810 tab. 5] [arxiv:2012.07810 tab. 1]. The paper does not explain the difference. - **Paper against released code.** The released training scripts differ from the paper in crop size, foreground scale, noise, the refinement loss and the evaluation defaults. The card lists nine differences in its Paper vs code section. Use the card's section before reproducing the paper's numbers. ## Papers - [Real-Time High-Resolution Background Matting](/papers/2012.07810): a captured-background method with a coarse base pass and patch refinement, reporting real-time HD and 4K speeds.