In Defense of Classical Image Processing: Fast Depth Completion on the CPU

In plain words

A LIDAR sensor measures distance at only a sparse set of points, so the depth map it produces is mostly holes. Many robot and driving systems want a full, dense map. The usual fix is a neural network trained to fill the holes, but that needs training data and often a graphics card.

This paper argues that a carefully ordered chain of ordinary image-processing operations can do the job. The method flips the depth values so that known points become the "biggest" pixels, lets them grow outward with a few differently shaped brushes, closes small gaps, fills what is left, and then smooths the result. Nothing is learned, so no training data is needed, and it runs on an ordinary processor. The authors report that it ranked first on the KITTI depth completion test server at the time they submitted it, and that it ran at about 90 frames per second.

TL;DR

Depth completion for sparse LIDAR can be solved with a fixed sequence of eight OpenCV-style steps, with no training data and no GPU. The core trick is to invert depths (100 m minus the value) so that morphological dilation, which keeps the larger value, grows near points without overwriting them with empty zeros. The authors report 1288.46 mm test RMSE on KITTI, below the best learned baseline they compare against (1419.75 mm). Every constant was chosen by a greedy search on the validation split, so the tuning is specific to KITTI.

Problem

LIDAR returns depth only for a small fraction of pixels. In the bottom two thirds of the KITTI depth maps, 5–7% of pixels carry a point §5. Missing depth limits 3D object detection, obstacle avoidance and SLAM, since they need points on the object §1. Guided methods use the colour image to fill holes, but their quality depends on image quality and on calibration and synchronisation between camera and depth sensor §2. Learned unguided methods need GPUs and large training sets §2. This paper addresses the unguided case, where only the sparse depth map is used §3.

The problem is stated as an equality, a target function f that the estimator f̂ should match §3:

\min \left\lVert \hat{f}(I, D_{sparse}) - f(I, D_{sparse}) \right\rVert_F^2 = 0 eq. 1

Here I is the image and D_sparse the sparse depth map §3. The paper does not solve this optimisation; the algorithm is a hand-designed f̂ §3, §4.

Method

The method is a fixed pipeline of eight steps, applied in order to a sparse depth map in metres §4. Values are inverted first, so that the OpenCV morphology operations, which let larger values overwrite smaller ones, do not let empty zeros overwrite real depths §4, Depth Inversion. The steps then go from small to large hole fills, so the reach of each valid pixel grows slowly and local structure is kept §5.2. Two steps, extension to the top of the frame and large hole fill, are presented as part of the algorithm §4; the authors later say they are skipped for the point-cloud application §5.2.

The eight steps §4, in order:

1. Depth inversion: valid depths become 100 m minus the depth. This leaves a 20 m buffer between valid values and empty values, since KITTI depths are at most 80 m §4.
2. Custom kernel dilation: each valid pixel is dilated with a 5×5 diamond kernel. Full, circular, cross and diamond shapes were compared, and the diamond was best §4 §5.2 tab. 2.
3. Small hole closure: a morphological close with a 5×5 full kernel joins nearby dilated patches §4.
4. Small hole fill: a mask of the still-empty pixels is built, and only those pixels are set from a 7×7 full dilation §4.
5. Extension to top of frame: the highest valid value in each column is copied to the rows above it §4. Optional in the released code [code: ip_basic/depth_map_utils.py#L100-L106].
6. Large hole fill: remaining empty pixels are set from a 31×31 full dilation, leaving valid pixels unchanged §4. Optional in the released code [code: ip_basic/depth_map_utils.py#L108-L111].
7. Median and Gaussian blur: a 5×5 median removes outliers, then a 5×5 Gaussian smooths planes and rounds edges §4 tab. 3. The Gaussian is applied to valid pixels only depth_map_utils.py.
8. Depth inversion back: output = 100 m minus the inverted value §4.

The kernel sizes were chosen greedily: first size (3×3, 5×5 and 7×7, with 5×5 best), then shape, then blur §5.2 tab. 2 tab. 3. The authors report that the kernel size and shape had no significant effect on runtime §5.2.

# Sparse-to-dense depth completion without training  [§4]
# Input:  depth (H, W) in metres; 0 marks an empty pixel
# Output: depth (H, W) in metres

def complete_depth(depth, max_depth=100.0, extrapolate=False, blur="bilateral"):
    # Invert valid depths so that morphology keeps the near edges  [§4]
    # [code: ip_basic/depth_map_utils.py#L84-L86]
    valid = depth > 0.1
    depth = where(valid, max_depth - depth, depth)
    # Dilate with the 5x5 diamond kernel; the larger value wins  [§4] [tab. 2]
    # [code: ip_basic/depth_map_utils.py#L89]
    depth = dilate(depth, diamond_kernel_5x5)
    # Close small holes with a 5x5 full kernel  [§4]
    # [code: ip_basic/depth_map_utils.py#L92]
    depth = close(depth, full_kernel_5x5)
    # Fill only the empty pixels from a 7x7 full dilation  [§4]
    # [code: ip_basic/depth_map_utils.py#L95-L97]
    empty = depth < 0.1
    depth[empty] = dilate(depth, full_kernel_7x7)[empty]
    if extrapolate:
        # Copy the first valid value of each column up to the top row  [§4]
        # [code: ip_basic/depth_map_utils.py#L100-L106]
        for each column c:
            top = first_valid_row(depth, c)
            depth[0:top, c] = depth[top, c]
        # Fill the still-empty pixels from a 31x31 full dilation  [§4]
        # [code: ip_basic/depth_map_utils.py#L108-L111]
        empty = depth < 0.1
        depth[empty] = dilate(depth, full_kernel_31x31)[empty]
    # 5x5 median blur over the whole map  [§4] [tab. 3]
    # [code: ip_basic/depth_map_utils.py#L114]
    depth = median_blur(depth, 5)
    if blur == "gaussian":
        # 5x5 Gaussian on valid pixels; sigma is not stated  [§4]
        # [code: ip_basic/depth_map_utils.py#L122-L124]
        valid = depth > 0.1
        depth[valid] = gaussian_blur(depth, 5)[valid]
    elif blur == "bilateral":
        # Bilateral filter; its parameters are not stated in the paper
        # [code: ip_basic/depth_map_utils.py#L119]
        depth = bilateral(depth, diameter=5, sigma_color=1.5, sigma_space=2.0)
    # Undo the inversion on valid pixels  [§4]
    # [code: ip_basic/depth_map_utils.py#L127-L128]
    valid = depth > 0.1
    depth = where(valid, max_depth - depth, depth)
    return depth

Hyperparameters

Results

Author-reported. The first table is the KITTI depth completion test set, evaluated by the KITTI server tab. 1. The baselines are the Nadaraya-Watson kernel method (NadarayaW), sparsity invariant CNNs (SparseConvs), and nearest neighbour interpolation followed by a CNN (NN+CNN), as named by the authors §5.1. The paper states the ranking was first among published methods on RMSE and MAE at submission time §5.1.

Setup: KITTI depth completion test set, CPU for the proposed method, GPU for the two learned baselines as stated in §5.1 tab. 1.

Versus NN+CNN, the paper reports 131.29 mm lower RMSE and 113.54 mm lower MAE, about 11 cm of mean error in the final point cloud §5.1. NN+CNN has the best iRMSE (3.25 vs 3.78); the two methods tie on iMAE tab. 1.

Ablations are on the validation set (not the test set in tab. 1) §5. Kernel size and shape, tab. 2:

Blur, tab. 3 (runtime added per blur over "No Blur" in the text §5.2):

The qualitative comparisons are in fig. 4 (KITTI test samples, error maps) and fig. 5 (projected point clouds with bilateral and Gaussian blur).

Paper vs code

Official code: the repository linked in the abstract, kujason/ip_basic, owned by the first author, at commit b6e6ce9c64e1fee63c2535f9999503cfe7e68066 (MIT). The code is the algorithm in depth_map_utils.py (fill_in_fast).

• Table 1 iRMSE is 3.78 tab. 1; the repository README reports 3.75 for the same method README.md.
• The paper presents extension to the top and large hole fill as steps of the algorithm §4. The code applies them only when extrapolate=True, and fill_in_fast defaults to False [code: ip_basic/depth_map_utils.py#L67, #L100-L111]. The released paper-result configuration turns them on with the Gaussian blur depth_completion.py; the recommended configuration turns them off and uses bilateral blur [code: demos/depth_completion.py#L36-L39].
• The paper's final blur is median plus Gaussian tab. 3. The default blur_type in fill_in_fast is 'bilateral' [code: ip_basic/depth_map_utils.py#L67], so calling the function with defaults does not reproduce the paper result.
• The paper's output formula, 100 m minus the inverted value, applies to all pixels §4. The code inverts back only pixels with value > 0.1 [code: ip_basic/depth_map_utils.py#L127-L128]. Empty pixels stay 0 in the output rather than becoming 100 m, which the literal formula would give.
• The paper does not describe a second method. fill_in_multiscale uses cross kernels of size 3, 5 and 7 selected by depth bin (above 30 m, 15–30 m, 0.1–15 m), a 9×9 full hole fill, six iterations of a 5×5 full dilation for large holes, and a bilateral blur with sigmaColor 0.5 [code: ip_basic/depth_map_utils.py#L133-L258]. The demo describes it as a slower, noise-reducing variant depth_completion.py.
• Timings differ: the demo comments give 90 Hz for the Gaussian paper-result setting, 87 Hz for bilateral and 30 Hz for multi-scale [code: demos/depth_completion.py#L31-L44]. The README's timing table gives 71 FPS for bilateral with extension, and 133 FPS for Gaussian without extension [code: README.md#L101-L105]. The paper's 90 Hz is on an i7-7700K §5.1.
• No discrepancy found in the kernel shapes and sizes: the 5×5 diamond, 5×5 full closing, 7×7 full fill and 31×31 full fill match §4 depth_map_utils.py.

Pitfalls

• Greedy, KITTI-specific tuning. Each choice was made on the validation set in a fixed order: size, shape, blur §5.2. The constants (100 m, 0.1 m threshold, kernel sizes) are not derived from first principles and may not transfer to other sensors or depth ranges.
• Validation and test numbers differ. Tables 2 and 3 are validation results, and Table 1 is the test server result §5 tab. 1–3. Do not compare an ablation row with the test row directly.
• Depth beyond 100 m. The inversion constant is 100 m §4. A depth above 100 m becomes negative after inversion and is treated as empty by the 0.1 m threshold depth_map_utils.py. This is inferred from the code; the paper does not discuss it. KITTI depths are said to be at most 80 m §4.
• Dilation is a grey-scale maximum. The method depends on the morphology keeping the larger value, as OpenCV does §4. A mean or median filter in place of dilation changes the behaviour completely.
• Blurs touch empty pixels. The median and bilateral blurs run on the whole map, so they can pull values into empty pixels next to filled regions. The Gaussian runs on valid pixels only [code: ip_basic/depth_map_utils.py#L114, #L119, #L122-L124]. The paper does not say how empty pixels are handled in the blurs.
• Unstated filter parameters. Gaussian sigma and all bilateral parameters are not in the paper. The code uses OpenCV's automatic Gaussian sigma and the bilateral values in the Hyperparameters table [code: ip_basic/depth_map_utils.py#L119, #L123].
• Top-of-frame extension is a strong assumption. It assumes tall objects continue above the highest LIDAR return in each column §4. The authors say the extension and large hole fill introduce many extrapolated points and suggest turning them off for practical use §5.2. They also note the Gaussian version adds many points §5.2.
• Output layout. The image-processing steps run on float32 arrays. The demo writes 16-bit PNGs by multiplying by 256 [code: demos/depth_completion.py#L121-L122, #L180]. Values above 256 m would wrap around in uint16; this is inferred from the code.
• NumPy version. The multi-scale function uses np.bool [code: ip_basic/depth_map_utils.py#L208, #L225]. NumPy removed that alias in version 1.24; requirements.txt does not pin NumPy requirements.txt. The fast path used for the paper result does not use it.
• Dataset not included. The demo expects KITTI depth-selection folders at a hard-coded path under the home directory [code: demos/depth_completion.py#L22-L23]. The repository has no KITTI data.
• Equation 1 is not an algorithm. It states the goal as zero error against an unknown target §3, eq. 1. The method is a hand-built estimator and its error is not analysed.
• Ranking claim is time-bound. "First among all published methods" is true at submission §5.1. Later leaderboard entries are not covered by this card.

Lineage

• Compared against: Nadaraya-Watson kernel regression (NadarayaW), sparsity invariant CNN (SparseConvs) and nearest neighbour interpolation with a CNN (NN+CNN). The paper cites Uhrig et al. 2017, "Sparsity invariant CNNs", for these baselines §2 tab. 1. No card exists for that paper and no arXiv id is given in the paper's bibliography, so it is not linked here.
• Related: guided depth completion with colour images and joint bilateral filtering is surveyed in §2. The paper argues against that family for sparse LIDAR input.
• Superseded by: not known from content in this wiki.