LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving

In plain words

Self-driving cars often use a spinning laser scanner (LiDAR) to find other vehicles, pedestrians and bikes. Most detectors first turn the scan into a top-down map of 3D points and then search that map. LaserNet keeps the scan in the form the sensor produces it: a compact grid of rows (one per laser) and columns (one per horizontal angle). A convolutional network runs on that grid, and every laser return proposes a guess at the box around the object it hits.

Each guess is not a single box but a set of possible boxes with a spread, so the detector also says how sure it is about the box. Nearby guesses for the same object are averaged together, and then a filter removes duplicate boxes. It lets the filter allow more overlap where the boxes are less certain. The approach is quick to run, and on a large dataset it does well. On a small public dataset it does worse than the top-down detectors.

TL;DR

LaserNet is a single-stage, fully convolutional 3D detector that works on the LiDAR range image rather than a bird's-eye-view grid §3.1. Each point predicts a class probability and a mixture of oriented box distributions with a log standard deviation per component §3.3. Predictions are fused by mean-shift clustering §3.4 and pruned by an adaptive NMS driven by predicted variance §3.6. Author-reported results show competitive or better vehicle detection on the large ATG4D dataset and about 2x faster total runtime than the fastest KITTI baseline, but weaker results on the small KITTI benchmark §4.1, §4.4, tab. 1, tab. 3, tab. 4.

Problem

Top-down (BEV) detectors discretise the 3D point cloud into voxels or pillars. That is sparse and costly to compute, and it discards occlusion information §1. Range-view approaches (VeloFCN, MV3D) work on the dense sensor image but have not matched BEV accuracy §1, §2. Single deterministic boxes also give no measure of localisation uncertainty §2. The authors' premise is that the range view is harder to learn from and needs a large training set to overcome the scale and occlusion problems §5. The method assumes objects lie on a common ground plane, so boxes are described in the x-y plane with a fixed height §3.3.

Method

Input representation §3.1. The sensor's point sweep is mapped to an image. Each laser id becomes a row and each azimuth bin becomes a column. If several points fall in one cell, the closest point is kept. Each cell carries five channels: range, height, azimuth, intensity, and a flag that marks whether the cell holds a point. For the Velodyne 64E the paper gives about 0.4° vertical spacing (non-uniform) and about 0.2° horizontal resolution §3.1. Rows follow the true laser geometry, not a uniform elevation grid §4.2.

Network §3.2, fig. 3, fig. 4. A deep layer aggregation (DLA) backbone §3.2 with three hierarchical levels. Kernel counts per level are 64, 64 and 128 §3.2. Vertical resolution is kept constant. Downsampling and upsampling happen only horizontally §3.2. Feature extraction and aggregation modules are residual blocks fig. 4. A final 1x1 convolution produces the encoded predictions §3.2.

Per-point outputs §3.3. For each point the network outputs:

• class probabilities over C object classes plus background §3.3;
• for each of K mixture components and each class: a relative centre (d_x, d_y), a relative orientation as (cos ω, sin ω), and the box length and width (l, w) §3.3;
• a log standard deviation s = log σ per component, with one shared variance for x and y and across all four corners §3.3;
• a mixture weight α_k per component §3.3.

Box geometry §3.3, eq. 1, eq. 2. Using the point's position (x, y) and azimuth θ, the absolute centre and heading are computed first eq. 1. The four corners follow from the centre, heading and size eq. 2. Corners are stacked into one vector b of eight values §3.3.

# Decode the box of one point and one mixture component  [§3.3] [eq. 1-2]
# Inputs: point_xy (2,), azimuth (scalar), rel_centre (2,), rel_angle (2,)
#         length, width (scalars)
# Output: corners (4, 2)

def decode_box(point_xy, azimuth, rel_centre, rel_angle, length, width):
    # Absolute centre: rotate the relative offset by the azimuth  [eq. 1]
    centre = point_xy + rotate(rel_centre, azimuth)     # (2,)
    # Absolute heading: azimuth plus the angle of the cos/sin pair  [eq. 1]
    heading = azimuth + atan2(rel_angle[1], rel_angle[0])
    # Corner offsets b1..b4 in the box frame  [eq. 2]
    offsets = [(l, w), (l, -w), (-l, -w), (-l, w)]
    # Rotate each offset by the heading and add the centre  [eq. 2]
    corners = [centre + 0.5 * rotate(o, heading) for o in offsets]
    return corners                                      # (4, 2)

Mean-shift clustering §3.4, eq. 3-5. Mean shift runs per class and per mixture component, on box centres rather than corners §3.4. The top-down plane is cut into bins of size Δx by Δy. Each occupied bin starts with the average of the centres in it and records which points it holds §3.4. A bin's mean is then updated with a kernel-weighted average over its own bin and its eight neighbours eq. 3, eq. 4. When a mean moves into another bin, the two merge and the point sets union eq. 5. The paper computes this with bin tensors shifted on a GPU §3.4.

# Approximate mean shift on a bin grid  [§3.4] [eq. 3-5]
# Inputs: bin_means (H, W, 2), bin_counts (H, W) (= |S_i|)
# Output: updated bin_means (H, W, 2), bin_counts (H, W)

def mean_shift(bin_means, bin_counts, iterations=3):
    # Repeat a fixed number of times; the paper uses three  [§3.4]
    for _ in range(iterations):
        # Kernel-weighted average over the bin and its 8 neighbours  [eq. 3]
        # kernel K_ij = exp(-|m_i - m_j|^2 / (dx^2 + dy^2))  [eq. 4]
        bin_means = weighted_neighbour_mean(bin_means, bin_counts)
        # If a mean now lies in another bin j, merge i into j  [eq. 5]
        bin_means, bin_counts = merge_bins(bin_means, bin_counts)
    return bin_means, bin_counts

Cluster fusion §3.4, eq. 6. After the iterations, the per-point distributions in one cluster are fused as a product of Gaussians. Each point's weight is w = 1/σ². The fused mean is the weighted average of its boxes, and the fused variance is the inverse of the summed precisions. Every point in the cluster takes the cluster's box and variance §3.4.

# Fuse the per-point boxes of one cluster  [§3.4] [eq. 6]
# Inputs: boxes (S, 8), sigmas (S,) for the S points in the cluster
# Output: fused_box (8,), fused_sigma (scalar)

def fuse_cluster(boxes, sigmas):
    # Precision weight per point: w = 1 / sigma^2  [§3.4]
    weights = 1.0 / sigmas**2                           # (S,)
    # Mean of the boxes weighted by w  [eq. 6]
    fused_box = sum(weights[:, None] * boxes) / sum(weights)
    # Variance: inverse of the summed precisions  [eq. 6]
    fused_sigma = (sum(1.0 / sigmas**2)) ** -0.5
    return fused_box, fused_sigma

Training losses §3.5, eq. 7-10. Each point is matched to the mixture component whose mean is closest to its ground-truth box eq. 8. Only that component is updated (hindsight-style, following multiple choice learning §3.5).

• Classification: focal loss §3.5 summed over points and divided by P, the number of points in the image eq. 7.
• Box: a Laplace-style loss, the absolute error scaled by 1/σ̂ plus log σ̂, summed over the box elements eq. 9.
• Mixture weights: cross-entropy with the matched component as the positive label §3.5.
• Regression: the box and mixture terms are averaged per object, with λ = 0.25 weighting the mixture term. Each point's term is divided by the count of points on its object n_i, and summed over objects N eq. 10.
• Total: L_cls + L_reg §3.5.

# Losses for one image  [§3.5] [eq. 7-10]
# Inputs: point_cls_probs (P, C+1), points on objects with matched component
# Output: scalar loss

def image_loss(point_cls_probs, cls_targets, matched_boxes, sigmas, n_i):
    # Focal loss per point, then mean over all P points  [eq. 7]
    cls_loss = mean(focal_loss(point_cls_probs, cls_targets))
    # Laplace-style box loss for the matched component  [eq. 9]
    box_loss = sum(abs(pred_box - gt_box) / sigma + log(sigma))
    # Cross entropy on the mixture weights, positive = matched component
    mix_loss = cross_entropy(mixture_weights, matched_index)
    # Per-object terms divided by n_i and averaged over objects  [eq. 10]
    reg_loss = mean((box_loss + 0.25 * mix_loss) / n_i)
    return cls_loss + reg_loss

Inference §3.6, eq. 11. A point counts as belonging to a class when its class probability exceeds 1/C §3.6. Each such point proposes one box per mixture mean. Adaptive NMS then prunes boxes in the same class. For each overlapping pair, the largest IoU that two side-by-side objects could show, given their predicted σ values, is computed eq. 11. The score of a mixture component is α_k / (2σ̂_k) §3.6.

# Maximum tolerated IoU for two boxes of one class  [§3.6] [eq. 11]
# Inputs: sigma1, sigma2 (predicted std dev, m), width (object width, m)
# Output: threshold t in [0, 1]

def max_tolerated_iou(sigma1, sigma2, width):
    # Worst case: two same-size objects side by side  [§3.6]
    spread = sigma1 + sigma2
    if spread < width:
        # Closed form  [eq. 11]
        return spread / (2.0 * width - spread)
    # Otherwise any overlap is allowed  [eq. 11]
    return 1.0

After thresholding, the pair's IoU is compared to t. If IoU is above t, the paper gives two options. Hard NMS removes the box with the lower likelihood. Soft-like NMS keeps it and raises its variance until t equals the observed IoU §3.6. The paper's ablation favours the soft variant tab. 2.

Hyperparameters

Results

Author-reported. Setup: ATG4D validation set (500 sequences, sampled at 0.5 Hz; 5,969 validation sweeps) and KITTI validation split, both Velodyne 64E §4.1, §4.4. AP at IoU 0.7 for vehicles and 0.5 for bikes and pedestrians §4.1. Baselines: PIXOR, PIXOR++, ContFuse (LiDAR and LiDAR+RGB) tab. 1.

Table 1: BEV detection on ATG4D (validation), AP tab. 1

LaserNet is best on the 0-70 m column for vehicles, bikes and pedestrians and at every vehicle range except 50-70 m. There ContFuse LiDAR+RGB is higher (69.83 against 67.65) tab. 1.

Ablation: vehicle AP0.7 on ATG4D tab. 2, discussed in §4.2. Each setting is listed as: output type, row spacing, mean shift on or off, IoU threshold, NMS variant, score.

• Mean-only output, laser row spacing, mean shift on, threshold 0.1, hard NMS: 77.05.
• Unimodal output, uniform row spacing, mean shift on, threshold 0.1, hard NMS: 79.14.
• Unimodal output, laser row spacing, mean shift off, threshold 0.1, hard NMS: 80.22.
• Unimodal output, laser row spacing, mean shift on, threshold 0.1, hard NMS: 80.92.
• Multimodal output, laser row spacing, mean shift on, threshold 0.1, hard NMS: 81.80.
• Multimodal output, laser row spacing, mean shift on, no threshold, soft NMS: 84.43.
• Multimodal output, laser row spacing, mean shift on, adaptive threshold, hard NMS: 83.68.
• Multimodal output, laser row spacing, mean shift on, adaptive threshold, soft NMS: 85.34.

Table 3: Runtime on KITTI (ms, NVIDIA 1080Ti) tab. 3

The paper says the total includes pre- and post-processing §4.3. Source of the baseline timings is not stated.

Table 4: BEV vehicle detection on KITTI (validation), AP0.7 tab. 4

Section 4.4 says the KITTI evaluation server computes AP at IoU 0.7 and also describes a held-out validation split. The paper does not say which of these Table 4 uses. The authors say LaserNet is worse than current BEV detectors here §4.4.

Calibration §4.5, fig. 6. On the training and validation sets, the predicted box distribution is compared to a unit-slope line. The model is calibrated on ATG4D and not on KITTI §4.5.

Paper vs code

No official code. Search steps:

1. Paper LaTeX (main.tex) and abstract: no code link.
2. No project page linked from the paper.
3. GitHub repository search for "lasernet": 23 results, none from Uber ATG or the listed authors (for example atyshka/Lasernet is a third-party TensorFlow 2 reimplementation; kareemalsawah/Modified_LaserNet_Pytorch is a third-party PyTorch version). Search inside uber-research: no results. User search for Gregory Meyer with Uber: no results.

Pitfalls

• ATG4D is the main evidence and its availability is not stated. The paper describes the dataset but gives no access route, so the headline results may not be reproducible from the paper alone §4.1.
• The distribution head needs a lot of data. On KITTI, with 5,985 training sweeps, the model did not learn a calibrated distribution §4.5, fig. 6, and it lagged BEV detectors tab. 4. The authors attribute this to data size §4.5, §5.
• Mixture width depends on class. K=3 for vehicles and K=1 for others on ATG4D §4.1; K=1 with vehicles only on KITTI §4.4. A reimplementation must use these per-class settings.
• Mean-shift parameters. Bins are 0.5 m with 8 neighbours and three iterations §3.4. After a merge, the merged bin's mean is zeroed and its point set emptied §3.4. The paper does not state how such invalidated bins are skipped in the kernel sum.
• Mean-shift is per class and per mixture component §3.4. Running it jointly across classes is not what the paper describes.
• Ambiguous C in the foreground threshold. §3.3 defines C without background, and §3.6 uses 1/C. An implementer should read C as the object-class count, but the paper does not restate this in §3.6.
• Adaptive NMS uses σ in metres and an object width w. Equation 11 switches to 1 when the summed σ reaches w. The formula is continuous at that point, so the switch does not introduce a jump. The paper gives w ≈ 2 m for vehicles but does not give values for pedestrians or bikes.
• Adaptive NMS needs the predicted σ. Soft-style handling raises the σ of a box to match the observed IoU §3.6. The paper does not state a floor or cap on σ.
• Scale of training. 300k iterations at batch 128 on 32 GPUs §4.1 is far larger than the KITTI setup §4.4. Per-GPU batch, warmup and weight decay are not stated.
• Loss details. Focal-loss γ and α are not stated §3.5. The box loss sums over the box elements; the paper writes the sum over n without stating its range explicitly. Use all eight box coordinates, since b stacks four corners §3.3.
• Box dimensions. The paper predicts l and w but does not say whether they are linear, log-scaled, or relative to a class mean §3.3.
• Runtime numbers are not directly comparable. The total includes pre- and post-processing §4.3, and the source of the baseline timings is not stated tab. 3. Table 3 cites VoxelNet for the MV3D row tab. 3; the citation looks wrong, so check the MV3D source before relying on that row.
• Unclear split for Table 4. §4.4 mentions both a held-out validation split and the KITTI evaluation server, and the paper does not say which one Table 4 reports tab. 4.
• Image row layout matters. Rows follow the laser ids rather than a uniform elevation grid §4.2. Using uniform rows instead drops unimodal vehicle AP0.7 from 80.92 to 79.14, about 1.8 points tab. 2.
• Ground-plane assumption. Boxes live in the x-y plane with a fixed height §3.3. Vertical extent and pitch are not predicted.

Lineage

• Builds on: Deep layer aggregation (Yu et al.) for the backbone §3.2; Kendall and Gal for the log-σ and Laplace-style box loss §3.5; focal loss (Lin et al.) for classification §3.5; hindsight matching from multiple choice learning (Guzman-Rivera et al.) §3.5; Soft-NMS (Bodla et al.), to which the paper's soft variant is described as analogous §3.6.
• Compared against: PIXOR and PIXOR++ tab. 1, tab. 3, tab. 4; VoxelNet tab. 3, tab. 4; MV3D, AVOD, F-PointNet and ContFuse tab. 1, tab. 3, tab. 4. The range-view methods VeloFCN and MV3D are named as the range-view predecessors §1, §2.
• Superseded by: not stated in the paper.