End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection

In plain words

Self-driving cars need to find other cars and pedestrians in 3D. The usual sensor is a LiDAR, which is expensive. A cheaper option is a pair of normal cameras: a network estimates how far away each pixel is, and those distances are turned into a fake point cloud that a LiDAR-style detector can read. This is called pseudo-LiDAR. The catch is that the depth network and the detector were trained separately, each with its own goal, so their parts did not fit well. A depth network trained on every pixel spends most of its effort on the background and nearby objects, while the detector cares about the objects.

This paper connects the two parts so that the detector's mistakes flow back into the depth network during training. The conversion step from depth map to points is changed so that it passes gradients, using two different tricks for the two kinds of detector. A plain depth loss is kept alongside, so the depth network still produces sensible distances everywhere, not just where the detector looks.

TL;DR

• Makes the stereo-depth, point-cloud conversion and 3D detector one trainable pipeline, for a point-based detector (PointRCNN) and a voxel-based one (PIXOR).
• Needs a differentiable step that turns pixels into points or voxels. Two designs are given: angular subsampling and soft quantization.
• Reports gains over the separately trained pseudo-LiDAR++ baseline on the KITTI validation set and a state-of-the-art image-only test result at submission time. All numbers are author-reported.

Problem

Pseudo-LiDAR trains two parts in two steps. A depth estimator is first trained on depth for every pixel; a LiDAR-style detector is then trained on the resulting points, with the depth network frozen §3. Two problems follow. The depth loss weights all pixels equally, while about 10% of KITTI pixels belong to cars and pedestrians/cyclists and those are mostly within 20 m §3, fig. 2. With the depth network frozen, the detector cannot correct depth errors in its input §3. Making the conversion step differentiable is the obstacle: a LiDAR detector takes points or voxels, and a naive voxel count has zero gradient almost everywhere §3.1.

Method

Pipeline §3, fig. 3:

1. A stereo depth network (SDN backbone) predicts a dense depth map Z from the left and right images §4.2.
2. Each pixel is projected to a 3D point with the pinhole relation (pseudo-LiDAR projection) §2, eq. 1. Its derivative with respect to depth is used for back-propagation §3.2.
3. A change of representation (CoR) turns the points into the detector's input. Two CoR types are given, matched to the detector:
   • Subsampling for point-based detectors such as PointRCNN §3.2: remove points above the normal LiDAR height range, then optionally keep one point per angular bin. Points that are dropped get no gradient; kept points pass their gradient straight through to the depth values §3.2.
   • Soft quantization for voxel-based detectors such as PIXOR §3.1: each point adds a radial-basis weight to its own bin and to its 26 neighbouring bins, instead of a binary occupancy. The derivative then reaches every point that influences a bin, so a bin that is empty can still pull points toward its centre §3.1, fig. 4.
4. The detector runs on the cloud and gives its loss §3.3.
5. Training uses a weighted sum of the detection loss and a depth loss. The depth loss is a smooth L1 on pixels with ground-truth depth, and it acts as a regularizer §3.3, footnote.

Training schedule as described: the depth network is first trained, then the detector is trained with the depth network fixed, then both are trained jointly with balanced loss weights §4.2.

# Joint training step for the point-based (PointRCNN) variant  [§3.3] [§4.2]
# Code: PointRCNN/tools/train_utils.py#L131-L149

def train_step(batch):
    # Stereo depth, then the depth-to-cloud change of representation  [§3.2]
    depth_loss, cloud = depth_network(batch)
    # Detector losses (RPN and RCNN) computed on the cloud  [§3.3]
    det_loss = detector(cloud, batch)
    # Weighted sum; the depth loss keeps weight 1  [§4.2]
    total = 0.01 * det_loss + depth_loss
    total.backward()
    # Gradient clipping applies to the detector parameters only  [code: L145]
    clip_grad_norm(detector.parameters(), 1.0)
    detector_optimizer.step()
    depth_optimizer.step()

# Soft quantization of points into the bin grid  [§3.1] [eq. 2-4]
# Input:  points (N, 3) inside the ROI; centres (M, 3); sigma2 = 0.01
# Output: occupancy (M,), differentiable with respect to the points

def soft_occupancy(points, centres, sigma2, offsets):
    # Hard bin of each point, detached from the graph  [§3.1] [eq. 2]
    own_bin = nearest_bin(points).detach()
    occupancy = zeros(M)
    # Offset 0 is the own bin; the other 26 offsets are neighbours  [eq. 4]
    for offset in offsets:
        target_bin = own_bin + offset
        # RBF weight of each point with respect to the target centre  [eq. 3]
        weight = exp(-sum_xyz((points - centres[target_bin]) ** 2) / sigma2)
        # Mean over the points that feed each target bin  [eq. 3]
        mean_weight = scatter_mean(weight, target_bin)
        # Own bin weight 1; each neighbour weight 1/26  [eq. 4]
        scale = 1.0 if offset == 0 else 1.0 / 26
        occupancy[unique(target_bin)] += scale * mean_weight
    return occupancy

# Angular subsampling for the point-based variant  [§3.2]
# Code: PointRCNN/lib/net/depth_net.py#L278-L296, #L384-L392
# Input:  points (N, 4) in the LiDAR frame, in input order
# Output: at most one point per angular bin

def angular_subsample(points, rows=64, cols=512):
    # Polar angle from the vertical axis, offset by 2 degrees  [code: L293]
    polar = radians(2.0) - asin(z / distance)
    # Azimuth from the forward axis  [code: L289]
    azimuth = radians(45.0) - asin(y / hypot(x, y))
    row = clamp(floor(polar / radians(0.4)), 0, rows - 1)
    col = clamp(floor(azimuth / radians(90.0 / cols)), 0, cols - 1)
    bin_id = row * cols + col
    # Keep one point per bin: the last one in input order  [code: L390-L392]
    return points[last_index_per_bin(bin_id)]

# Depth loss as implemented for the released models  [§3.3] [code: L126-L149]
# Pixels with ground-truth depth between 1 and 80 m  [code: L132]

def depth_loss(outputs, gt_depth):
    mask = (gt_depth >= 1) & (gt_depth <= 80)
    # Three network outputs with weights 0.5, 0.7 and 1.0  [code: L147-L149]
    return (0.5 * smooth_l1(outputs[0][mask], gt_depth[mask])
            + 0.7 * smooth_l1(outputs[1][mask], gt_depth[mask])
            + smooth_l1(outputs[2][mask], gt_depth[mask]))

Smooth L1 used for the depth loss §3.3, eq. 5:

\ell(x) = \begin{cases} 0.5\,x^2, & \text{if } |x| < 1 \\ |x| - 0.5, & \text{otherwise} \end{cases} eq. 5

The depth loss is the average of ℓ over the set of pixels that have ground-truth depth §3.3. The total loss is λ_det L_det + λ_depth L_depth, where L_det is the classification loss plus the regression loss of the detector §3.3.

Gradient statistics that motivated the loss weights §4.2, tab. 1: the depth loss reaches 3% of depth-map pixels, the PointRCNN loss 4%, and the PIXOR loss (through soft quantization) 70%. Balancing the sums of gradients between the detection and depth losses was reported as crucial for stable joint training §4.2.

Hyperparameters

Results

Author-reported. Setup: KITTI object detection, car class, validation set (3D and bird's-eye-view AP at IoU 0.5 and 0.7) §4.1; baselines are pseudo-LiDAR++ (image-only) and other stereo methods §4.1.

*PIXOR with visual features, from the PL++ authors' modified code §4.2. Only AP_BEV is reported for PIXOR tab. 2.

KITTI test set, car, IoU 0.7, AP_BEV / AP_3D tab. 3:

Ablation on PointRCNN, which sub-networks are trainable, car, moderate AP_BEV / AP_3D tab. 4:

Ablation on PIXOR, car, moderate AP_BEV at IoU 0.5 / 0.7 tab. 5:

Other author-reported results:

• Runtime, one GPU, PyTorch: E2E-PL with PointRCNN 0.49 s/frame; with PIXOR 0.55 s/frame; the stereo network alone 0.39 s/frame §4.6.
• Depth quality improves mainly near objects; overall depth error changes little [supp. §S6.1].
• Pedestrians and cyclists (IoU 0.5, moderate): PL++ 25.2 / 21.3 and 23.9 / 22.5 versus E2E-PL 27.8 / 24.9 and 26.2 / 24.1 [supp. §S1, tab. S1].
• Argoverse, PointRCNN (S), moderate AP_BEV / AP_3D at IoU 0.7: PL++ 17.0 / 11.1 versus E2E-PL 18.8 / 11.3 [supp. §S6.2, tab. S5].

Paper vs code

Searched: the arXiv abstract links https://github.com/mileyan/pseudo-LiDAR_e2e; the owner is an author (the LICENSE names Yan Wang, and the README contacts name two other authors). Cloned at commit 739263581ca8d59f2b269c9004c493aacae117e0. The repository has two parts, one for PointRCNN and one for PIXOR, plus a top-level MIT licence LICENSE.

• Depth loss. Paper: one smooth L1 over pixels with ground truth §3.3, eq. 5. Code: a weighted sum of three network outputs (0.5, 0.7, 1.0) over pixels with depth between 1 and 80 m [code: PointRCNN/lib/net/depth_net.py#L132, #L147-L149] train_pixor_e2e.py.
• Loss weights. Match the paper: λ_depth = 1, λ_det = 0.01 for PointRCNN train_utils.py, λ_det = 0.1 for PIXOR train_pixor_e2e.py §4.2.
• Angular subsampling. Paper: keep one point per bin, bin choice not specified, and subsampling is "optional but suggested" §3.2. Code: subsampling runs on every frame depth_net.py. It keeps the point that comes last in row-major pixel order in each bin, using a scatter over the point index, and does not compare distances even though the function is named min_dist_subsample depth_net.py. Read from the code; not run.
• Initialisation and joint schedule. Paper: depth first, then detector with the depth fixed, then joint training §4.2. Code: the joint run loads a pre-trained detector and depth checkpoint and fine-tunes both README.md README.md.
• PIXOR constants. σ² = 0.01 kitti_process_detection.py, 26 neighbours with 1/26 weight [code: PIXOR/kitti_process_detection.py#L87, #L105] and the 700 × 800 × 35 grid train_pixor_e2e.py match §3.1.
• Soft quantization details. Points outside the ROI are dropped before quantization, so kernel mass that would spill past the ROI edge is lost kitti_process_detection.py. The output axis order is height, depth, width (35, 700, 800) kitti_process_detection.py.
• Flip augmentation. Implemented by flipping the voxel tensor along its width axis train_pixor_e2e.py; the paper does not describe augmentation.
• Released PIXOR config. The config sets 15 epochs, lr 5e-5, momentum 0.9, ResNet-18 and diffused quantization fusion.cfg. None of these appear in the paper.
• Scripts. PointRCNN/run/default_train.sub calls cfgs/default.yaml and eval_rcnn.py; neither is in the repository, which ships cfgs/e2e.yaml and eval_rcnn_depth.py default_train.sub.
• Third-party code. The subtrees contain code from other projects with their own licences (for example PointRCNN/tools/kitti_object_eval_python/LICENSE). The paper's depth network is "SDN" §4.2; the code uses a stacked-hourglass model stackhourglass.py.

Pitfalls

• Subsampling is not a true nearest-point rule. Implement the rule the paper states (one point per bin) and note that the code picks the last point in pixel order. The two can differ in which point survives, which changes gradients.
• Gradient sparsity drives the design. With subsampling, only the kept points carry gradient, about 4% of depth-map pixels in the authors' statistics tab. 1. The depth loss covers the rest. The ablations never remove the depth loss while letting detection gradients reach the depth network, so its separate effect is not isolated tab. 4, tab. 5.
• Scheduler timing (inferred). The PIXOR loop calls the schedulers at the start of each epoch train_pixor_e2e.py, so a milestone of 11 takes effect in the 0-based epoch 10. The PointRCNN depth scheduler steps after each epoch train_utils.py.
• Gradient clipping is on the detector only. The depth network is not clipped train_utils.py.
• Units and frames. The height cut and the point range are applied in the LiDAR frame after the camera-to-LiDAR transform depth_net.py. Check that the calibration used for the projection matches the one used for these cuts, since the cuts are hard-coded.
• Stereo pre-training data. The depth network is pre-trained on Scene Flow and then fine-tuned on KITTI §4.2. The PointRCNN README points to a depth checkpoint that goes in a folder named stack_finetune_from_sceneflow README.md.
• Costs the abstract hides. Joint training runs the stereo network, the quantizer or sampler and the detector in one graph. Runtime is dominated by the stereo network at 0.39 s/frame §4.6. Soft quantization evaluates 27 bins per point, which the authors describe as linear in the number of points §4.6.
• Figures are not reproduced here. The main figures (fig. 1 to fig. 6) and the supplementary figures (fig. S1 to fig. S4) are described only in the text §1 §3 §4.5 [supp. §S3-S5].
• Version. This card follows arXiv v2 (updated 2020-05-14). The code commit above is the one checked.

Lineage

• Builds on: Pseudo-LiDAR++ (arXiv:1906.06310), whose stereo depth network and pipeline are the baseline §1 §4.2. The original pseudo-LiDAR projection is §2, eq. 1.
• Detectors used: PointRCNN §3.2 §4.2 and PIXOR §3.1 §4.2. Their arXiv ids are not in the wiki yet.
• Compared against: PL, PL++, 3DOP, SRCNN, RT3DStereo, OC-Stereo, MLF-stereo §4.1.
• Superseded by: not known from the wiki's content.