LiDAR R-CNN: An Efficient and Universal 3D Object Detector

In plain words

Self-driving cars often find objects with a LiDAR sensor, which returns a cloud of dots on the surfaces around the car. A common detector first proposes rough boxes around possible cars, pedestrians and cyclists, and a second step checks and corrects each box. LiDAR R-CNN is that second step, and it can be attached to different first-step detectors.

The authors found that a simple network reading the dots inside a box can miss how big the box is. The dots show where surfaces are, but not where the empty space ends, so boxes of different sizes can look the same to the network. The fix is to tell the network how far each dot sits from the faces of its box. With that information the network can judge the size and correct it.

The network is small and sees only the dots of one proposal at a time, which is why the method is fast. The authors report better results than their baselines on the Waymo dataset.

TL;DR

LiDAR R-CNN refines 3D boxes from any LiDAR detector with a PointNet over the raw points inside each proposal, expressed in the box frame. The paper's main claim is that plain point-based refinement cannot perceive proposal size, and it fixes this by appending each point's distances to the box faces. Author-reported on Waymo validation vehicles: 3D AP 75.6 with PointPillars proposals, against 72.1 for the PointPillars baseline. Code is released, but the repository has no LICENSE file, so this card quotes nothing from it.

Problem

A point-based refiner built on PointNet sees point coordinates but not the spacing between points. A proposal that is enlarged can therefore keep exactly the same points, and the features do not change, while its regression and classification targets do §3.2 fig. 3 (a). The paper measured this on Waymo: after enlarging PointPillars boxes by 1 m in length and width, 21.3% of the 4,387,747 test-set boxes kept the same point count, and 42.5% gained fewer than 10 points, mostly ground points §4.2. The paper also argues that voxel second stages only partly solve this, and that anchors and size normalisation break down when several classes share one model §3.2 tab. 5.

The method is a second stage: it needs proposals from a base detector, and the base detector is not re-trained §1.

Method

The network refines a 3D proposal (x, y, z, w, l, h, θ) from any detector. It reads the raw points around the proposal and outputs a class score and corrected centre, size and heading. The paper compares three ways to make the size visible (normalisation, anchors, voxels) and two candidates that it keeps: a boundary offset and virtual grid points §3.3 fig. 3 (b)–(f). The headline numbers match the boundary-offset rows of the ablations tab. 4 tab. 8. The released code implements the boundary offset only; no virtual-point code is in the repository.

1. Point selection §3.1 fig. 1. Enlarge the proposal's width and length, then take every point inside that enlarged box. The paper does not state the enlargement amount.
2. Box frame §3.1 fig. 1. Move the points so that the box centre is the origin, and rotate them so that the heading is the x-axis. The z-axis points up.
3. Boundary offset §3.3 fig. 3 (e). For each point, append the distance to the two faces on each axis, six values in total.
4. Fixed size §4.3. Use 512 points per proposal. The paper repeats points when fewer than 512 are available.
5. Backbone §3.1 fig. 2. A PointNet: a shared per-point MLP, a max-pool over points, and fully connected layers. The paper describes one shared feature feeding a classification branch and a regression branch §3.1. The code has separate centre, size and heading heads [code: src/LiDAR_RCNN/models/point_net.py#L53-L70].
6. Targets §3.1 eq. 2 eq. 3 eq. 4 eq. 5 eq. 6 eq. 7. Regression targets are defined in the proposal frame. The centre offset is divided by the proposal's w, l and h. The size target is a log ratio. The heading target is the smaller of two residuals, so that a 180° flip is not penalised as a large error.
7. Labels §3.1 §4.1. A proposal is positive when its IoU with the ground truth is at least 0.7 for vehicles and 0.5 for pedestrians and cyclists.
8. Loss eq. 1 eq. 8 eq. 9. Softmax cross-entropy for the classes, smooth L1 for the regression on positives only, with a weight of 20 on the regression term.

Key equations, as written in the paper:

\mathcal{L}_{cls} = \frac{1}{B} \sum_{i=1}^{B} -\log p_{y(i)}^{(i)} eq. 1

\tilde{\mathbf{b}}_i = (0, 0, 0, w_i, l_i, h_i, 0) eq. 2

\tilde{\mathbf{b}}_i^{gt} = (x_i^{gt} - x_i, y_i^{gt} - y_i, z_i^{gt} - z_i, w_i^{gt}, l_i^{gt}, h_i^{gt}, \theta_i^{gt} - \theta_i) eq. 3

\mathbf{t}_i^{c} = \left(\frac{x_i^{gt} - x_i}{w_i}, \frac{y_i^{gt} - y_i}{l_i}, \frac{z_i^{gt} - z_i}{h_i}\right) eq. 4

\mathbf{t}_i^{s} = \left(\log \frac{w_i^{gt}}{w_i}, \log \frac{l_i^{gt}}{l_i}, \log \frac{h_i^{gt}}{h_i}\right) eq. 5

\Delta\theta_i = (\theta_i^{gt} - \theta_i) \bmod \pi eq. 6

t_i^{o} = \begin{cases} \Delta\theta_i & \Delta\theta_i \le \frac{\pi}{2} \\ \Delta\theta_i - \pi & \Delta\theta_i > \frac{\pi}{2} \end{cases} eq. 7

\mathcal{L}_{reg} = \frac{1}{B_+} \sum_{i=1}^{B_+} \mathrm{SmoothL1}(\mathbf{o}_i - \mathbf{t}_i) eq. 8

\mathcal{L} = \mathcal{L}_{cls} + \lambda \mathcal{L}_{reg}, \qquad \lambda = 20 eq. 9

Pseudocode, written from the paper's steps and checked against the code:

# Refine one proposal  [§3.1] [§3.3] [fig. 1] [fig. 3]
# Input:  cloud (N, 3) in vehicle frame; proposal (x, y, z, w, l, h, θ)
# Output: point features (K, 9) in box frame; K = 512  [§4.3]

def point_features(cloud, proposal):
    # Enlarge width and length; keep points in that box  [§3.1]
    # Code: +3 m  [code: tools/data_processer/data_utils.py#L153-L154]
    inside = points_in_enlarged_bev_box(cloud, proposal)
    # Shift to the box centre, then rotate by heading  [§3.1]
    # [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L23-L32]
    local = to_box_frame(cloud[inside], proposal)     # (M, 3)
    # Boundary offset: distance to the two faces, per axis  [§3.3]
    # [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L33-L36]
    half = proposal.size / 2                          # (3,) for w, l, h
    offsets = concatenate(half - local, half + local) # (M, 6)
    features = concatenate(local, offsets)            # (M, 9)
    # Sample exactly K points, with replacement  [§4.3]
    # [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L27]
    return choose_with_replacement(features, num_points=512)

# Forward pass  [§3.1] [fig. 2]
# [code: src/LiDAR_RCNN/models/point_net.py#L23-L70]
def forward(features):
    # Shared per-point MLP  [§3.1]
    per_point = shared_mlp(features)                  # (K, 512)
    # Max-pool over points: one vector per proposal  [§3.1]
    pooled = max_over_points(per_point)               # (512,)
    # Two fully connected layers  [§3.1]
    # [code: src/LiDAR_RCNN/models/point_net.py#L35-L41]
    hidden = fully_connected(pooled)                  # (256,)
    # Heads: class logits, centre, size, heading  [§3.1]
    logits = class_head(hidden)
    centre = centre_head(hidden)
    size = size_head(hidden)
    heading = heading_head(hidden)
    return logits, centre, size, heading

# Regression targets in the proposal frame  [§3.1] [eq. 3-7]
def targets(proposal, gt_box):
    # Offset from proposal centre; code also rotates xy into box frame  [eq. 3]
    # [code: src/LiDAR_RCNN/datasets/waymo/loader.py#L78-L80]
    offset = rotate_to_box_frame(gt_box.centre - proposal.centre)
    # Normalise the centre offset by proposal size  [eq. 4]
    centre = offset / proposal.size
    # Log ratio of ground-truth and proposal sizes  [eq. 5]
    size = log(gt_box.size / proposal.size)
    # Smaller of two heading residuals; flips not penalised  [eq. 6-7]
    # [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L98-L105]
    heading = minimal_residual(gt_box.heading - proposal.heading)
    return centre, size, heading

# Training loss  [§3.1] [eq. 1] [eq. 8] [eq. 9]
def loss(logits, centre, size, heading, labels, targets, positive):
    # Softmax cross-entropy over the classes, averaged over the batch  [eq. 1]
    # Code uses IoU-scaled soft targets instead
    # [code: src/LiDAR_RCNN/utils/model_utils.py#L46-L53]
    cls_loss = cross_entropy(logits, labels)
    # Smooth L1 on positive proposals only  [eq. 8]
    # Code averages centre and size terms over 3 components each
    # [code: src/LiDAR_RCNN/utils/model_utils.py#L56-L69]
    reg_loss = regression_term(centre, size, heading, targets, positive)
    # Balance factor lambda = 20  [eq. 9] [§3.1]
    return cls_loss + 20 * reg_loss

Hyperparameters

Results

Author-reported. Setup: Waymo Open Dataset (WOD) validation sequences for Tables 1, 2 and 4–8 §4.1; KITTI testing set for Table 3 §4.1. Proposals come from PointPillars and SECOND in mmdetection3d, and from a range-view first stage the authors re-implemented §4.3. Baselines are the first stage itself, unless the row says otherwise. Metric: 3D AP at IoU 0.7 for vehicles §4.1. LEVEL_1 objects have at least 5 points, LEVEL_2 objects at least 1 point §4.4.

Vehicle detection, WOD validation, 3D AP with BEV AP overall. The APH columns, and the BEV per-distance columns, are omitted here; they are in tab. 1.

Multi-class WOD validation, 3D AP and APH tab. 2:

KITTI testing set, vehicle 3D AP tab. 3: PointPillars(T) 82.17 / 72.81 / 67.57 (easy / moderate / hard); PointPillars(TV) 83.12 / 74.11 / 69.14; LiDAR R-CNN on PointPillars(T) proposals, trained on train+val, 85.97 / 74.21 / 69.18. "T" means trained on the training split, "TV" on training plus validation.

Size-ambiguity ablation, WOD vehicle, 3D AP at IoU 0.7 tab. 4: PointPillars 71.6 (BEV 87.1); PointNet refinement 74.1 (87.9); voxel 72.9 (87.2); anchor 75.2 (88.2); size normalisation 75.4 (88.1); virtual point 75.4 (88.1); boundary offset 75.6 (88.3). The caption does not state the difficulty level.

Three-class model, one network, 3D AP tab. 5. Columns: vehicle, pedestrian, cyclist. PointPillars 71.6 / 70.6 / 64.4; PointNet refinement 72.1 / 69.2 / 62.2; voxel 72.1 / 69.8 / 64.5; anchor 72.5 / 70.2 / 63.5; size normalisation 72.7 / 69.9 / 64.4; virtual point 73.3 / 70.4 / 66.2; boundary offset 73.4 / 70.6 / 66.8. The caption does not state the difficulty level.

Range-view first stage with a voxel second stage vs LiDAR R-CNN, 3D AP / BEV AP tab. 6: RV baseline 53.4 / 70.5; RV + voxel 64.1 / 76.7; RV + LiDAR R-CNN 68.7 / 81.1.

Cross-model generalisation, 3D AP tab. 7: SECOND 58.5 / 63.9 / 48.6 (vehicle / pedestrian / cyclist); LiDAR R-CNN (sec) 62.6 / 68.2 / 52.8; LiDAR R-CNN (pp), trained on PointPillars outputs and run on SECOND outputs without fine-tuning, 62.1 / 67.2 / 52.9.

Sub-task ablation, AP at LEVEL_1 / LEVEL_2 tab. 8. Each row adds one or more refinement targets to PointPillars (72.1 / 63.6):

Cost, 128 proposals, batch 128 tab. 9: voxel second stage 72.9 AP, 19 ms, 3.5 M parameters; LiDAR R-CNN 75.6 AP, 4.5 ms, 0.5 M parameters. The introduction reports 200 fps for 128 proposals on a 2080 Ti §1.

Other author claims: LiDAR R-CNN (pp) outperforms PV-RCNN by 5.31% 3D AP on vehicles §4.4; the 2x variant adds 1.5 points at LEVEL_2 §4.4; the cascade reaches 76% 3D AP on vehicles §4.4; on KITTI it adds 2.8 AP on the easy set §4.4. Inference time against the number of sampled points, 64 to 1024, is plotted but its values are not given in the text fig. 4.

Paper vs code

Code at commit c693828a5ff2d65ab7293c84837df9e3f6a7786b. The repository has no LICENSE file, so the code is not used beyond citations.

• Classification target. The paper uses hard labels with a softmax cross-entropy eq. 1 §3.1. The code uses one-hot labels scaled by an IoU-based score, clip(2·IoU − 0.5, 0, 1) [code: src/LiDAR_RCNN/utils/model_utils.py#L46-L53]. The paper also gives no such score.
• PointNet widths. The paper gives 64, 64, 512 §4.3. The code builds 64, 128, 512 for X=1 point_net.py.
• Weight decay. The paper states 1e-5 §4.3. The base config sets 0.0005 lidar_rcnn.yaml.
• Repeats. The paper repeats points only when fewer than 512 exist §4.3. The code samples 512 with replacement whatever the count [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L27].
• Second point set. For single-frame data, the loader makes a second set of 512 rows from a dummy zero point, so one proposal may reach the network as 1024 rows, not 512 data_utils.py loader.py. This is inferred from reading the code; it was not run.
• Input channels. The paper describes 3 or more channels fig. 2. The base config sets PTS_DIM to 9 lidar_rcnn.yaml. The loader stacks x, y, z, a frame index and two image flags, then six offsets, which gives 12 columns data_utils.py. The empty-proposal fallback is also 12 wide [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L37-L38]. Read literally, the config and the loader disagree; not run.
• Virtual points. The paper proposes them and reports them in Table 4 and Table 5 §3.3. No virtual-point code exists in the repository; only the boundary offset is implemented data_utils.py.
• Centre target. The paper's centre equations have no rotation eq. 3 [eq. 4]. The code rotates the xy offset into the proposal's heading frame before normalising loader.py.
• Heading target. The paper uses (θgt − θ) mod π, mapped to (−π/2, π/2] eq. 6 eq. 7. The code takes the smaller in absolute value of two wrapped residuals data_utils.py. The intent matches; the behaviour at ±π/2 was not checked.
• Regression reduction. The paper writes one smooth-L1 term over the target vector, normalised by the positive count eq. 8. The code averages the three centre components and the three size components separately, adds the heading term, and divides each by the positive count plus 1e-5 [code: src/LiDAR_RCNN/utils/model_utils.py#L56-L69].
• Learning rate. The poly schedule matches §4.3 [code: src/LiDAR_RCNN/utils/utils.py#L75-L79]. The training script also builds a OneCycleLR scheduler, but its step call is commented out [code: tools/train.py#L111-L116] function.py.
• Three classes. The paper uses C+1 outputs §3.1, which is four for three classes. The 3-class config sets CLS_NUM to 5, because labels keep the raw WOD type ids 1, 2 and 4 lidar_rcnn_all_cls.yaml [code: config/lidar_rcnn_all_cls.yaml#L36].
• Inference. The paper gives no post-processing. The code scores each class with softmax(logits / 4) and runs weighted NMS (0.1, 0.5, yaw 0.3) [code: src/LiDAR_RCNN/utils/eval_utils.py#L57] eval_utils.py.
• Enlargement and jitter. The paper gives neither amount §3.1 §4.3. The code enlarges by 3 m in total and jitters with five ranges [code: tools/data_processer/config/mmdet3d_pp_train.yaml#L7] [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L9-L20]. The fourth value in each range is never read: the range index is drawn uniformly [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L12].
• Multi-frame. The code supports several frames, but the base configs have no MODEL.Frame key, while training reads it train.py [code: config/lidar_rcnn.yaml#L12-L17]. The README says to set it [code: README.md#L71]. The paper reports single-frame results only.
• Reported numbers. The README table lists 73.8 vehicle AP for 3-class 2x and 66.8 for vehicle LEVEL_2 1x README.md README.md. The paper gives 73.5 tab. 2 and 66.5 tab. 1. The README also lists 3-frame results (77.8) that the paper does not report README.md.
• Missing parts. The cascade (2xc), the range-view first stage, and the KITTI experiments are not in the repository. The paper says RCD is not open-sourced §4.1. The proposals come from mmdetection3d, outside this repository §4.3.

Pitfalls

• Which baseline. The vehicle PointPillars baseline is 71.6 in Tables 2, 4 and 5, and 72.1 (labelled PointPillars*) in Tables 1 and 8. The paper says the mmdetection3d PointPillars was fine-tuned §4.4 but does not explain the 0.5-point difference tab. 1 tab. 4. Compare against the same baseline row.
• Size is the whole argument. Plain PointNet refinement barely helps the size sub-task tab. 8. Keep the boundary-offset input; without it the refiner does not learn size. Size normalisation and anchors fail in the multi-class setting tab. 5.
• Virtual points vs boundary offset. In Table 4 the virtual point row is 75.4 and the boundary offset row is 75.6; in Table 5 the offset is again slightly ahead (66.8 vs 66.2 for cyclists). The paper calls both workable §3.3 tab. 4 tab. 5. Only the offset is in code.
• Unstated values. Enlargement amount, jitter, NMS, inference temperature and the poly exponent are not in the paper. Take them from the code and say so data_utils.py [code: src/LiDAR_RCNN/utils/eval_utils.py#L71].
• Half-precision storage. The data processor saves points, proposals and ground truth as float16 data_utils.py data_utils.py [code: tools/data_processer/data_utils.py#L164]. Float16 spacing is 0.0625 m between 64 and 128 m, so positions and box values are quantised.
• Heading flips. The paper says that unwrapped heading targets let the flipped samples dominate the gradient, which is why it uses the mod-π rule §3.1.
• Empty proposals. An empty cloud gives a zero-filled input [code: src/LiDAR_RCNN/datasets/waymo/data_utils.py#L37-L38]. The loader's comment says such proposals become negatives, but its test checks the padded point set, which always has rows. The label therefore comes from IoU alone [code: src/LiDAR_RCNN/datasets/waymo/loader.py#L63-L69]. Extraction is in the bird's-eye plane only; the height filter is commented out [code: tools/data_processer/box_aware_op/src/extract_points.cpp#L21].
• Background target. For background proposals, the code's IoU-scaled soft target still lands on the background column, which the code itself marks as a FIXME model_utils.py.
• Costs the abstract hides. Training needs proposals from a base detector on the whole training set, about 4M vehicle, 1.8M pedestrian and 72K cyclist proposals per model §4.3. The data pipeline needs Waymo's tools and compiled extensions data_utils.py. Training is seven hours on four GPUs §4.3.
• Validation only. The WOD results are on validation sequences tab. 1 [tab. 2]. The paper gives no test-server numbers for WOD.
• Cross-model transfer. PointPillars-trained refinement applied to SECOND without fine-tuning gains 3.6 vehicle AP and 3.3 pedestrian AP over SECOND. In-domain training gains 4.1 and 4.3. For cyclists the transfer gain (4.3) matches the in-domain gain (4.2) tab. 7.

Lineage

• Builds on: R-CNN-style refinement §1; PointNet as the backbone §3.1. Similar point-based R-CNN models that the paper cites: PointRCNN, Fast-PointRCNN and IPOD §3.2.
• Compared against: PointPillars, SECOND, PV-RCNN, RCD, MVF, Pillar-od and StarNet tab. 1 tab. 2; voxel second stages tab. 4 tab. 9.
• Superseded by: not stated in this wiki.