Guided depth upsampling

Overview

Guided depth upsampling takes a low-resolution (LR) depth map and a high-resolution (HR) colour image, and predicts an HR depth map. The colour image tells the method where depth edges are likely to lie. The task is defined on depth from a phone or tablet LiDAR, so the input contains real sensor artefacts and holes, not a synthetically downscaled map 2111.08897 §4.2.

The wiki has one card for this concept. Its paper, ARKitScenes, compares bilinear interpolation, two classical guided methods and two learned networks, at three scale factors, on its own dataset 2111.08897 §4.2 2111.08897 tab. 4. No other card covers the task, so every result below comes from one author team and one dataset.

Variants

Bilinear interpolation

The non-learned baseline. It uses no colour image and no training data 2111.08897 §4.2. Card: ARKitScenes.

Joint Bilateral Upsampling (JBU) and Fast Guided Global Interpolation (FGI)

Two classical guided methods that use the colour image as a guide, with no learned weights. The paper reimplements both for its comparison 2111.08897 §4.2. On its benchmark, neither beats bilinear interpolation on L1 at any factor. On RMSE, JBU is slightly ahead only at x8, and FGI never beats bilinear 2111.08897 tab. 4. Card: ARKitScenes.

Multi-Scale Guided Network (MSG)

A learned network trained on ARKitScenes to predict HR depth from the colour image and the LR depth. It is trained with an RMSE objective over valid ground-truth pixels 2111.08897 §4.2. On the paper's benchmark it beats all three non-learned baselines on L1 and RMSE at every factor 2111.08897 tab. 4. Card: ARKitScenes.

Multi-Scale Progressive Fusion (MSPF)

A learned network, also trained on ARKitScenes, and the best method in the paper at every factor on both L1 and RMSE 2111.08897 tab. 4. The authors adapt its training for holes: the SSIM term is dropped, because SSIM needs every pixel to be present, and the edge loss, which runs over the whole image, is replaced by a loss following MegaDepth 2111.08897 §4.2. Card: ARKitScenes.

When to use which

The paper's results are for LiDAR depth from one tablet, on one dataset, so the choice below only follows from what the benchmark shows.

MSG's L1 is about 15, 17 and 22 percent higher than MSPF's at x2, x4 and x8 (0.0176 against 0.0153, 0.0175 against 0.0149, 0.0185 against 0.0152). The table gives point values only, with no spread across runs, so the paper alone does not show how stable these gaps are 2111.08897 tab. 4.

Implementation sketch

The paper gives no pseudocode 2111.08897 §4.2. The sketch below shows the common shape of a learned guided upsampler, using only the choices the paper describes: a colour guide, an LR depth input, a masked loss that ignores missing ground truth, and HR depth as output. Details such as architecture and the exact loss form are in the paper's card, not here.

# Learned guided depth upsampling, one training step (shapes assumed)
# color: HR RGB image; lr_depth: LR depth with holes; hr_depth: HR ground truth
# Output: scalar loss

def training_step(color, lr_depth, hr_depth, network):
    # Ignore pixels with no ground-truth depth (holes in the reference)
    valid = hr_depth > 0
    # The network sees the colour guide and the LR depth together
    pred = network(color, lr_depth)
    # Compare only on valid pixels, so holes in the ground truth do not
    # contribute to the loss
    return masked_loss(pred, hr_depth, valid)

Variants differ in the network and in the loss. MSG is trained with RMSE over valid pixels. For MSPF the authors drop the SSIM term and replace the edge loss, so the loss must work with incomplete depth 2111.08897 §4.2.

Contradictions

None known within the cards for this concept. Two points to note when comparing numbers:

• The classical guided methods do not outperform bilinear interpolation on this benchmark 2111.08897 tab. 4. This is a finding of the paper, not a conflict between papers.
• The paper's table gives no units for L1 and RMSE 2111.08897 tab. 4, so values from other work cannot be compared without checking the unit.

Papers

• ARKitScenes: a mobile-LiDAR RGB-D dataset from homes. One of its two benchmarks is guided depth upsampling at x2, x4 and x8, with bilinear, JBU, FGI, MSG and MSPF compared. The dataset and companion code are under a non-commercial licence; see the card's Pitfalls before training on the data for commercial use.