--- kind: concept slug: guided-depth-upsampling title: Guided depth upsampling summary: >- Raising a low-resolution depth map to high resolution with a colour image as guide, using sensor depth with holes and artefacts. Covers bilinear interpolation, two classical guided baselines (JBU, FGI) and two learned networks (MSG, MSPF) benchmarked on ARKitScenes at x2, x4 and x8. Numbers are author-reported, on one dataset. tags: [depth-upsampling, lidar, rgb-d, guided-upsampling] papers: ["2111.08897"] related: [image-super-resolution, monocular-depth-estimation] status: draft updated_at: 2026-10-18 --- ## Overview Guided depth upsampling takes a low-resolution (LR) depth map and a high-resolution (HR) colour image, and predicts an HR depth map. The colour image tells the method where depth edges are likely to lie. The task is defined on depth from a phone or tablet LiDAR, so the input contains real sensor artefacts and holes, not a synthetically downscaled map [arxiv:2111.08897 §4.2]. The wiki has one card for this concept. Its paper, ARKitScenes, compares bilinear interpolation, two classical guided methods and two learned networks, at three scale factors, on its own dataset [arxiv:2111.08897 §4.2] [arxiv:2111.08897 tab. 4]. No other card covers the task, so every result below comes from one author team and one dataset. ## Variants ### Bilinear interpolation The non-learned baseline. It uses no colour image and no training data [arxiv:2111.08897 §4.2]. Card: [ARKitScenes](/papers/2111.08897). ### Joint Bilateral Upsampling (JBU) and Fast Guided Global Interpolation (FGI) Two classical guided methods that use the colour image as a guide, with no learned weights. The paper reimplements both for its comparison [arxiv:2111.08897 §4.2]. On its benchmark, neither beats bilinear interpolation on L1 at any factor. On RMSE, JBU is slightly ahead only at x8, and FGI never beats bilinear [arxiv:2111.08897 tab. 4]. Card: [ARKitScenes](/papers/2111.08897). ### Multi-Scale Guided Network (MSG) A learned network trained on ARKitScenes to predict HR depth from the colour image and the LR depth. It is trained with an RMSE objective over valid ground-truth pixels [arxiv:2111.08897 §4.2]. On the paper's benchmark it beats all three non-learned baselines on L1 and RMSE at every factor [arxiv:2111.08897 tab. 4]. Card: [ARKitScenes](/papers/2111.08897). ### Multi-Scale Progressive Fusion (MSPF) A learned network, also trained on ARKitScenes, and the best method in the paper at every factor on both L1 and RMSE [arxiv:2111.08897 tab. 4]. The authors adapt its training for holes: the SSIM term is dropped, because SSIM needs every pixel to be present, and the edge loss, which runs over the whole image, is replaced by a loss following MegaDepth [arxiv:2111.08897 §4.2]. Card: [ARKitScenes](/papers/2111.08897). ## When to use which The paper's results are for LiDAR depth from one tablet, on one dataset, so the choice below only follows from what the benchmark shows. | Situation | Use | Why | |---|---|---| | No training data and no time to train | Bilinear interpolation | It is the simplest option, and in the paper it is about as accurate as the classical guided methods [arxiv:2111.08897 tab. 4] | | Training data from the same sensor and scene type is available | MSPF, or MSG if a simpler network is preferred | Both learned methods beat the baselines on the paper's benchmark. MSPF is ahead of MSG at every factor [arxiv:2111.08897 tab. 4] | | Input depth has holes and sensor noise | A learned method trained on such inputs | The paper trains on real LiDAR frames with missing depth, after filtering them by hole and error thresholds [arxiv:2111.08897 §4.2] | | Large upscaling factor (x8) | Either learned method | Both learned methods remain ahead of the baselines at x8 on L1 and RMSE; MSPF's lead over MSG is widest there on L1 [arxiv:2111.08897 tab. 4] | | Different sensor or environment | Check before relying on the numbers | The paper evaluates only on ARKitScenes with the iPad Pro LiDAR and does not show transfer to other sensors [arxiv:2111.08897 §4.2] | MSG's L1 is about 15, 17 and 22 percent higher than MSPF's at x2, x4 and x8 (0.0176 against 0.0153, 0.0175 against 0.0149, 0.0185 against 0.0152). The table gives point values only, with no spread across runs, so the paper alone does not show how stable these gaps are [arxiv:2111.08897 tab. 4]. ## Implementation sketch The paper gives no pseudocode [arxiv:2111.08897 §4.2]. The sketch below shows the common shape of a learned guided upsampler, using only the choices the paper describes: a colour guide, an LR depth input, a masked loss that ignores missing ground truth, and HR depth as output. Details such as architecture and the exact loss form are in the paper's card, not here. ```pseudocode # Learned guided depth upsampling, one training step (shapes assumed) # color: HR RGB image; lr_depth: LR depth with holes; hr_depth: HR ground truth # Output: scalar loss def training_step(color, lr_depth, hr_depth, network): # Ignore pixels with no ground-truth depth (holes in the reference) valid = hr_depth > 0 # The network sees the colour guide and the LR depth together pred = network(color, lr_depth) # Compare only on valid pixels, so holes in the ground truth do not # contribute to the loss return masked_loss(pred, hr_depth, valid) ``` Variants differ in the network and in the loss. MSG is trained with RMSE over valid pixels. For MSPF the authors drop the SSIM term and replace the edge loss, so the loss must work with incomplete depth [arxiv:2111.08897 §4.2]. ## Contradictions None known within the cards for this concept. Two points to note when comparing numbers: - The classical guided methods do not outperform bilinear interpolation on this benchmark [arxiv:2111.08897 tab. 4]. This is a finding of the paper, not a conflict between papers. - The paper's table gives no units for L1 and RMSE [arxiv:2111.08897 tab. 4], so values from other work cannot be compared without checking the unit. ## Papers - [ARKitScenes](/papers/2111.08897): a mobile-LiDAR RGB-D dataset from homes. One of its two benchmarks is guided depth upsampling at x2, x4 and x8, with bilinear, JBU, FGI, MSG and MSPF compared. The dataset and companion code are under a non-commercial licence; see the card's Pitfalls before training on the data for commercial use.