wikillm

Paper · 2021-06-25 · cs.CV

RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection

Range Sparse Net finds foreground points in LiDAR range images with a 2D U-Net, then runs sparse convolutions only on the points it keeps and predicts boxes with a CenterNet-style head, without NMS. Author-reported on Waymo Open Dataset validation: 74.8 vehicle 3D AP (L1) at 15.5 ms latency for the efficient CarS_3f model.

Object detectionSparse convolutionAnchor-free detectionFocal losslidar3d-object-detectionrange-imagesparse-convolutionautonomous-driving

arXivPDFMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Self-driving cars must find other vehicles and pedestrians in a laser scan of the world, and the useful range keeps growing with newer sensors. Many detectors turn the scan into a 3D grid and then process every cell, including the many empty ones, which gets expensive at long range.

RSN works in two stages. The first stage treats the scan the way the sensor captured it, as a flat image in which each pixel is one laser beam, and a small image network marks the pixels that most likely belong to an object. Only those points are kept. The second stage does the heavier 3D work on this small set of points, so its cost follows the number of kept points rather than the size of the scene. A final head predicts box centres directly, so no separate step is needed to remove duplicate boxes.

The design aims for speed without losing accuracy: the first stage is tuned to miss few object points, and the second stage removes the false ones. The model can also use several consecutive scans at little extra inference cost.

TL;DR

RSN is a LiDAR 3D object detector for driving. A small 2D network on the range image scores each pixel as foreground, and only the selected points go through sparse convolutions and a CenterNet-style head, so no non-maximum suppression is needed §3. The authors report results on the Waymo Open Dataset (WOD) that beat the LiDAR-only methods they compare with, at a fraction of their latency tab. 1 tab. 2. Use it as a reference design for efficient long-range LiDAR detection; the speed numbers depend on the authors' GPU setup §4.4.

Problem

Dense 3D grids (voxels or pillars) scale poorly to long ranges, since memory and compute grow with the grid size, and their voxel size limits resolution at distance §2. Point-based methods need nearest-neighbour search, which is costly on large point sets §2. Range-image detectors are dense and efficient, but they regress boxes from features that handle occlusion, localisation and size regression poorly, and fixing this with a second refinement stage adds significant cost §1 §2.

The problem is stated for a long-range LiDAR regime, with sensing up to 250 m and 300 m in some commercial sensors §1. The headline evaluation is on WOD, with the detection region limited to the ±79.5 m voxel box during training §4.2.

Method

RSN has five parts. Stage 1 is a 2D U-Net on the range image that produces range features and a per-pixel foreground score §3.1 §3.2. Stage 2 keeps the points whose score exceeds a threshold γ and gathers their range features §3.2. Stage 3 voxelizes those points and runs a sparse 2D or 3D convolution backbone, called SPFE §3.3. Stage 4 is a CenterNet-style head adapted to sparse voxels that predicts a centre heatmap and box parameters §3.4. A temporal variant repeats stages 1 and 2 per frame and fuses the selected points §3.5 app. B. The overview is shown in fig. 2.

  1. Range image input. Each pixel holds range, intensity and elongation. Values are clipped and rescaled to [0, 1] §3.1. The last return is used where a beam has several returns §4.2. The Waymo range image is 64 × 2650 pixels, and only the long-range LiDAR is used §4.1.

  2. Range image features (RIFE). A lightweight U-Net §3.1 fig. 3. A downsampling block D(L, C) has L residual blocks with C output channels, and the first block has stride 2. An upsampling block U(L, C) has one upsampling layer, which is a 1×1 convolution followed by bilinear interpolation, and L residual blocks at stride 1 §3.1. The text does not give the L and C values used in the final models.

  3. Foreground selection. A 1×1 convolution scores every valid pixel. Its training target is whether the pixel's 3D point lies inside any ground-truth box §3.2. The loss is the mean focal loss over the P valid pixels eq. 1. Points with score above γ are kept §3.2. The threshold is chosen for high recall, because false positives can be removed later but false negatives cannot §3.2.

  4. Dynamic voxelization. The kept points are assigned to voxels. Each point is extended with its offset to the voxel mean (p − m), the voxel covariance, and its offset to the voxel centre (p − c) §3.3. For the 2D (pillar) variant the vertical voxel size is set to infinity §3.3. An optional PointNet can further process each voxel's points §3.3.

  5. SPFE backbone. A 2D sparse convolution network (pillar style) or a 3D one is applied to the occupied voxels only §3.3 fig. 4. Pedestrian and vehicle models use different SPFE backbones §4.2 app. A.

  6. Centre heatmap target. The heatmap for a voxel centre ṽ is the maximum, over the boxes whose centres b_c contain ṽ, of \exp(-(\lVert \tilde v - b_c\rVert - \lVert \tilde V - b_c\rVert)/\sigma^2), where Ṽ is the set of voxel centres. It is 0 when no box contains the voxel. σ is a per-class constant §3.4 fig. 5. Heatmap values are regressed with a penalty-reduced focal loss eq. 2.

  7. Box regression. One fully connected layer predicts the heatmap and the box offsets (d_x, d_y, d_z) from the voxel centre, plus length, width, height and heading §3.4. Heading uses a bin loss, the other parameters use smooth L1, and an IoU loss is added eq. 3 eq. 4 eq. 5. Box losses apply only where the target heatmap exceeds δ1 eq. 5.

  8. Peak selection (no NMS). A sparse submanifold max-pooling is applied to voxels whose predicted heatmap exceeds δ2. Boxes at local maxima are the detections §3.4.

  9. Training objective. Total loss L = λ1·L_seg + λ2·L_hm + L_box eq. 6. Training is end to end, so gradients from SPFE reach RIFE §4.6 tab. 4.

  10. Temporal fusion. Each frame gets its own RIFE and selection branch, with weights shared across frames app. B. The selected points of each frame are moved into the latest frame's coordinates to remove ego-motion §3.5. Each point also gets voxel statistics from its own frame, and the time difference in seconds to the latest frame §3.5. All points are then merged and voxelized as in the single-frame model app. B. At inference only the latest frame's segmentation is computed; earlier results are reused app. B §4.3. Training uses windows of k+1 frames, and reuses the last valid frame when the window reaches the start of a sequence app. B.

Inference, as implemented by the steps above:

Pseudocode
# RSN single-frame detection  [§3] [fig. 2]
# Input:  range_image (H, W, 3); gamma, delta2 (thresholds)
# Output: boxes with scores; no NMS

def detect(range_image, gamma, delta2):
    # 1. Dense 2D features and a per-pixel foreground score  [§3.1] [§3.2]
    range_features, fg_scores = range_unet(range_image)  # (H, W, C), (H, W)

    # 2. Keep points whose score is above gamma  [§3.2]
    points, point_features = select_foreground(
        range_features, fg_scores, gamma)                # (N, 3), (N, C)

    # 3. Dynamic voxelization with per-point voxel offsets  [§3.3]
    voxels, voxel_coords = dynamic_voxelize(points, point_features)

    # 4. Sparse convolutions over occupied voxels only  [§3.3]
    voxel_features = sparse_backbone(voxels, voxel_coords)

    # 5. Centre heatmap and box parameters per voxel  [§3.4]
    heatmap, boxes = sparse_center_head(voxel_features)

    # 6. Keep local heatmap maxima above delta2 as detections  [§3.4]
    return peaks_above(heatmap, boxes, delta2)

Training objective, as written in the paper:

Pseudocode
# RSN training loss  [§3.2] [§3.4] [eq. 1-6]
# Input:  predictions and targets for one batch
# Output: scalar loss

def training_loss(preds, targets):
    # 1. Mean focal loss over valid range pixels  [eq. 1]
    seg_loss = mean_focal_loss(preds.fg_scores, targets.fg_labels)

    # 2. Penalty-reduced focal loss on the centre heatmap  [eq. 2]
    hm_loss = penalty_reduced_focal(preds.heatmap, targets.heatmap)

    # 3. Box losses on voxels with target heat above delta1  [eq. 3-5]
    box_loss = box_regression(preds.boxes, targets.boxes, targets.heatmap)

    # 4. Weighted sum with lambda1 = 400 and lambda2 = 4  [eq. 6] [§4.2]
    return 400 * seg_loss + 4 * hm_loss + box_loss

Key equations, written as in the paper:

L_{\textrm{seg}} = \frac{1}{P}\sum_{i}{L_{i}} eq. 1

L_{\textrm{hm}} = -\frac{1}{N}\sum_{\tilde{p}}\{(1 - \tilde{h})^\alpha\log(\tilde{h})I_{h > 1 - \epsilon} + (1-h)^\beta\tilde{h}^\alpha\log(1-\tilde{h})I_{h \leq 1 - \epsilon}\} eq. 2

L_{\theta_i} = L_{bin}(\theta_i, \tilde{\theta}_i) eq. 3

L_{b_i \backslash \theta_i} = \textrm{SmoothL1}(b_i \backslash \theta_i - \tilde{b}_i \backslash \tilde{\theta}_i) eq. 4

L_{\textrm{box}} = \frac{1}{N}\sum_i (L_{\theta_i} + L_{b_i \backslash \theta_i} + L_{\textrm{iou}_i})\, I_{h_i > \delta_1} eq. 5

L = \lambda_1 L_{\textrm{seg}} + \lambda_2 L_{\textrm{hm}} + L_{\textrm{box}} eq. 6

In eq. 2, α = 2, β = 4 and ε = 1e-3 §3.4 §4.2. In eq. 1, P is the number of valid range pixels §3.2. The 3D box is parameterised as b = {d_x, d_y, d_z, l, w, h, θ}, with d_z absolute when the 2D backbone is used §3.4.

Hyperparameters

NameValueWhereNotes
optimiserAdam§4.3betas, epsilon and weight decay not stated
learning ratecosine decay; initial 0.006; warmup from 0.003 over 5k steps; 120k steps in total§4.3warmup shape and whether the decay includes warmup: not stated
batch sizelargest that fits GPU memory, per backbone§4.3numeric value not stated
hardwareTesla V100; 8 GPUs (single frame), 16 GPUs (3 frames)§4.3
normalisationlayer norm in the per-voxel PointNet, not batch norm§4.3norm in the rest of the network: not stated
augmentationrandom flip along X; global rotation about Z by an angle uniform in [−π/4, π/4]§4.3applied to the selected foreground points
range image64 × 2650 pixels; range, intensity, elongation; scaled by min(v, m)/m with m = 79.5, 2.0, 2.0§4.1 §4.2last return picked
foreground cutoff γ0.15 (vehicle), 0.1 (pedestrian)§4.2 §3.2
λ1, segmentation weight400§4.2 eq. 6
λ2, heatmap weight4§4.2 eq. 6box loss weight is 1 in eq. 6
voxel regionx, y in [−79.5, 79.5] m; z in [−5, 5] m§4.2
voxel size0.2 m (vehicle), 0.1 m (pedestrian)§4.2Δz = ∞ for pillar style §3.3
σ (heatmap, per class)1.0 (vehicle), 0.5 (pedestrian)§4.2
δ1 (box-loss mask)0.2§4.2 eq. 5
δ2 (peak threshold)0.2§4.2 §3.4
focal α, β, ε (heatmap)2, 4, 1e-3§3.4 eq. 2seg focal loss parameters: not stated
heading bins12 (vehicle), 4 (pedestrian)§4.2bin loss definition not given
IoU threshold for eval0.7 (vehicle), 0.5 (pedestrian)§4.4
modelsCarS, CarL, CarXL (vehicle); PedS, PedL (pedestrian)§4.22D SPFE channels 96 for CarS, CarL, PedS, PedL; CarXL is 3D with channels 64 and no per-voxel PointNet app. A
framessingle frame (_1f) or 3 frames (_3f)§4.2 app. B
split798 train, 202 val, 150 test sequences; about 200 frames each§4.1
ensemble5 models, each on a disjoint 80% of training data; 5 augmentations each (25 sets); weighted box fusion extended to 3D with yawapp. Cfusion thresholds not stated

Results

Author-reported. Setup: Waymo Open Dataset (WOD) validation set for the single-model tables; official test set for the ensemble table. Metrics are AP and APH at difficulty L1 (easy) and L2 (hard), with 3D and BEV variants §4.4. Latency is measured on a Tesla V100 in float32 without TensorRT, averaged over 10 scenes, and excludes NMS for baselines. The PVRCNN latency comes from its authors on a Titan RTX tab. 1 §4.4. Baselines are the authors' own re-implementations where marked tab. 1.

SettingMetricBaselineThis paperSource
Vehicle, val3D AP/APH L1PVRCNN 70.3/69.7CarXL_3f 78.4/78.1tab. 1
Vehicle, val, efficient model3D AP/APH L1PVRCNN 70.3/69.7CarS_3f 74.8/74.4, 15.5 mstab. 1
Vehicle, vallatencyP.Pillars 49.0 ms; PillarMultiView 66.7 msCarS_3f 15.5 ms; CarL_3f 25.4 ms; CarXL_3f 67.5 mstab. 1
Pedestrian, val3D AP/APH L1PVRCNN WOD'20 78.9/75.1PedL_3f 79.4/76.2, 28.2 mstab. 2
Pedestrian, val, efficient model3D AP/APH L1PVRCNN WOD'20 78.9/75.1PedS_3f 78.3/75.2, 14.4 mstab. 2
Vehicle, test (overall)3D AP/APH L1PVRCNN ensemble 81.1/80.6CarXL_3f ensemble 81.4/81.0tab. 3
Vehicle, test (overall)3D AP/APH L2PVRCNN ensemble 73.7/73.2CarXL_3f ensemble 72.8/72.4tab. 3
Pedestrian, test (overall)3D AP/APH L1PVRCNN ensemble 80.3/76.3PedL_3f ensemble 82.4/78.0tab. 3
Pedestrian, test (overall)3D AP/APH L2PVRCNN ensemble 74.0/70.2PedL_3f ensemble 74.7/70.7tab. 3

Notes on the rows above:

  • The vehicle CarXL_3f single model is 78.4/78.1 on validation, a single-model result. The ensemble rows are on the test set tab. 3.
  • On test, the single CarXL_3f model is 80.7/80.3 (L1) and PedL_3f is 78.9/75.6 (L1) tab. 3. Ensembles give the gains claimed against the PVRCNN ensemble.
  • Ablations on CarS_3f (vehicle) and PedS_3f (pedestrian), validation, AP/APH L2 and latency tab. 4: baseline 64.2/63.9 and 15.4 ms (vehicle), 68.88/66.07 and 14.4 ms (pedestrian). Removing RIFE and foreground selection gives 60.9/60.3 and 27.0 ms (vehicle), 63.5/60.5 and 30.0 ms (pedestrian). Stopping gradients from SPFE into RIFE gives 60.1/59.7 (vehicle) and 64.8/61.7 (pedestrian). Adding normalised xyz inputs and removing heatmap normalisation also lower accuracy on some rows tab. 4 §4.6.
  • Foreground selection, CarS_3f: precision 77.5%, recall 99.6%; PedS_3f: precision 15.3%, recall 97.6% §4.5.
  • Internal long-range dataset (not public), CarS_3f, test set, [-250 m, 250 m] range: with RIFE and selection 61.2 3D APH at 22 ms; without, 53.4 3D APH at 44 ms tab. 5 §4.7.
  • Claims in the text: CarXL_3f "significantly outperforms all published methods" on validation §4.4; RSN ensemble beats the PVRCNN WOD challenge ensemble §4.4 tab. 3. The abstract's "more than 60 frames per second" is for a 150 m × 150 m region, and its ranking is "as of 11/2020" [abstract].

Paper vs code

No official code.

Search done on 2026-11-12. The paper has no code link, and the arXiv metadata has no comment or project link. Three GitHub repository searches (range sparse net, RSN range lidar, RSN sparse 3d detection) returned no match. A search for RSN lidar within the waymo-research organisation returned no match, and a range sparse lidar detection search found nothing either. The waymo-research/waymo-open-dataset repository is the dataset toolkit, not this method. No project page was found.

Pitfalls

  • Heatmap distance is ambiguous. The paper writes ‖Ṽ − b_c‖ without saying which point of the set Ṽ is meant. Figure 5's caption says the circle's radius is the distance from the box centre to the closest voxel centre fig. 5. Read the second term as a minimum over the set; the paper does not state this explicitly §3.4.
  • Segmentation focal loss parameters are not stated. Only the heatmap focal loss gets α = 2, β = 4 §3.4. α, γ and any normalisation for eq. 1 are not given.
  • Batch size, optimiser details and warmup shape are not stated §4.3. The learning rate is given as a cosine schedule with a 5k-step warmup starting at 0.003, but the text does not say whether the schedule is linear or how the warmup is combined with the decay.
  • Augmentation applies to selected points. The rotation and flip are applied to the selected foreground points §4.3. The text does not say how the ground- truth boxes are transformed to match, or whether the range image is augmented before selection. Implement the box transform explicitly.
  • Region and latency numbers differ. Training uses a ±79.5 m voxel region §4.2, while the abstract's frame-rate claim refers to a 150 m × 150 m region [abstract]. The text does not say which region the latency tables measure.
  • Latency is not comparable across hardware. RSN was timed on a V100 in float32 without TensorRT, PVRCNN on a Titan RTX by its authors, and the PillarMultiView latency is taken from its paper tab. 1.
  • Selection threshold tuning is manual. The authors suggest starting with a low γ, scanning λ1, then grid-searching γ §4.5. Foreground precision for pedestrians is only 15.3% §4.5, so the sparse stage carries most of the false-positive cost.
  • Temporal training cost. Three-frame models train on 16 V100s versus 8 for single-frame models §4.3. Training windows are k+1 frames and reuse the last valid frame at sequence starts app. B.
  • Shared versus separate training. Pedestrian and vehicle models are trained separately with different SPFEs but "share the same RIFE" §4.2. The text does not say whether one RIFE is trained on both classes or each copy is trained separately.
  • Heatmap normalisation is referenced but thinly defined. The ablation disables a normalisation "described in §3.4" §4.6. Section 3.4 gives only the ground-truth heatmap formula, so take that as the normalisation.
  • Ensemble fusion details are missing. The weighted box fusion step is cited to Solovyev et al. and extended to 3D with yaw, but the fusion thresholds and box-matching details are not stated app. C.
  • The ensemble claim is narrower than it reads. RSN's ensemble beats PVRCNN's ensemble on pedestrian L1 and L2 and on vehicle L1, but not on vehicle L2 (72.8 vs 73.7 overall) tab. 3. The text claims it "outperforms" the challenge submission without this caveat §4.4.
  • Direction of the γ sweep is unclear. The text argues that γ must keep recall high, since false negatives cannot be recovered §3.2. The caption of fig. 7 says latency drops sharply once γ falls below a threshold, which reads against the expectation that a higher cutoff keeps fewer points. The text does not settle this; check the sweep before choosing γ.

Lineage

  • Builds on: range-image detectors LaserNet and RCD-RCNN, which RSN extends with a segmentation stage instead of box proposals §2 §3; dynamic voxelization from MultiView §3.3; CenterNet's heatmap and head, adapted to sparse voxels §3.4; the bin loss from PointRCNN §3.4; the CenterNet-style head of AFDet §3.4.
  • Compared against: LaserNet, PointPillars, PillarMultiView, PVRCNN, RCD tab. 1 tab. 2 tab. 3.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection” by Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, Dragomir Anguelov (arXiv:2106.13365), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.