Depth completion

Overview

A LIDAR sensor returns depth at only a few pixels. In the bottom two thirds of KITTI depth maps, 5–7% of pixels carry a return 1802.00036 §5. Depth completion turns that sparse map into a dense one, which downstream tasks such as 3D detection, obstacle avoidance and SLAM need 1802.00036 §1.

This page covers the unguided case, where only the sparse depth map is the input 1802.00036 §3. Guided methods also take a colour image; the source paper notes that they depend on image quality and on calibration and synchronisation between the camera and the depth sensor 1802.00036 §2. Learned unguided methods need a GPU and large training sets 1802.00036 §2.

The goal is stated as an error to be driven to zero against an unknown target function 1802.00036 §3, eq. 1. No card on this page optimises that objective. The method is a hand-designed estimator, and its error is not analysed 1802.00036 §3.

The invariant the variants share: valid depths are grown outward into empty pixels with a grey-scale maximum filter, so they never get overwritten by zeros. The cards here differ only in the kernels and blurs they choose.

Variants

Classical morphological pipeline (IP-Basic)

A fixed sequence of about eight steps: invert the depths, dilate, close small holes, fill the rest, optionally extend columns upward and fill large holes, then median and Gaussian blur, and invert back 1802.00036 §4. Nothing is learned, so no training data is needed, and it runs on an ordinary CPU. The authors report 0.011 s per frame on an Intel Core i7-7700K, about 90 Hz 1802.00036 §5.1, tab. 1. Card: IP-Basic card.

The kernel choices came from a greedy search on the KITTI validation split, in this order: size, then shape, then blur 1802.00036 §5.2. The best initial kernel was a 5×5 diamond 1802.00036 tab. 2. The best final blur was median followed by Gaussian 1802.00036 tab. 3.

Learned and guided methods

The source paper compares against learned and guided approaches [arxiv:1802.00036 §2, tab. 1]. No card on this wiki covers them yet, so this page does not describe their designs.

When to use which

Implementation sketch

The pipeline below is the paper's algorithm in our own words [arxiv:1802.00036 §4]. Kernel sizes and the blur follow the paper's final choices. The valid threshold is not stated in the paper, so EPS is a placeholder.

# Sparse-to-dense depth completion without training  [arxiv:1802.00036 §4]
# Input:  depth (H, W) in metres; 0 marks no return
# Output: dense depth (H, W) in metres
MAX_D = 100.0   # inversion constant; KITTI depths are at most 80 m  [§4]
EPS = 0.0       # valid threshold, not stated in the paper

def complete_depth(depth, extend=False):
    # Invert so that max-filters grow valid depths, not zeros  [§4]
    d = where(depth > EPS, MAX_D - depth, 0)
    # Grow each valid pixel with a 5x5 diamond max-filter  [§4, tab. 2]
    d = dilate(d, diamond(5))
    # Join nearby patches with a 5x5 closing  [§4]
    d = close(d, full(5))
    # Fill only the still-empty pixels from a 7x7 max-filter  [§4]
    holes = d <= EPS
    d[holes] = dilate(d, full(7))[holes]
    if extend:
        # Optional: copy the top valid value of each column upward  [§4, §5.2]
        d = extend_columns_up(d)
        # Fill the remaining holes from a 31x31 max-filter  [§4]
        holes = d <= EPS
        d[holes] = dilate(d, full(31))[holes]
    # Remove outliers, then smooth  [§4, tab. 3]
    d = median(d, 5)
    # Gaussian smoothing; sigma is not stated in the paper  [§4]
    d = gaussian(d, 5)
    # Undo the inversion on valid pixels  [§4]
    return where(d > EPS, MAX_D - d, 0)

Contradictions

None across papers: this page has one card. Within that paper, the classical pipeline does not win every metric. It has the lower test RMSE and MAE, but the learned baseline has the lower iRMSE 1802.00036 tab. 1. The released code also differs from the paper's stated configuration in places (defaults, blur, and whether the extension is on). Those differences can change a reproduction. The card lists them with code citations.

Pitfalls

• Tuning is specific to KITTI. The 100 m inversion, the kernel sizes and the blurs were chosen on KITTI validation, in a fixed order [arxiv:1802.00036 §5.2]. The paper does not show they transfer to other sensors.
• Validation and test numbers differ. Tables 2 and 3 are validation ablations; table 1 is the test server 1802.00036 §5, tab. 1–3. Do not compare an ablation row with a test row.
• Depths above the inversion constant. Depths above 100 m become negative after inversion. The paper does not discuss this case 1802.00036 §4.
• Dilation must keep the larger value. The method relies on a grey-scale maximum filter so that zeros do not overwrite real depths [arxiv:1802.00036 §4]. Replacing the dilation with a mean or median filter changes the method.
• Top-of-frame extension is a strong assumption. It assumes a tall object continues above the highest return in its column 1802.00036 §4. The authors say it adds many extrapolated points and suggest turning it off in practice 1802.00036 §5.2.
• Ranking is time-bound. "First among published methods" was true at submission 1802.00036 §5.1. Later leaderboard entries are not covered here.
• Runtime setup. The 0.011 s figure is on an i7-7700K. The paper does not say whether it ran single-threaded 1802.00036 tab. 1.

Papers

• IP-Basic: an eight-step classical pipeline with no training data, run on a CPU at about 90 Hz, with the best KITTI test RMSE at submission.