PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection

In plain words

A car's LiDAR sensor produces a cloud of 3D points, and a detector has to find the cars, people and cones in it fast enough to run on the vehicle. One cheap family of detectors flattens the cloud onto the ground, so each spot becomes a vertical column of points called a pillar. The result is a top-down image, and ordinary 2D convolutions can process it. This is fast, but these detectors have trailed the ones that keep the full 3D structure.

The authors blame two parts of the pipeline: the step that turns each pillar's points into features is too simple, and the step that combines coarse and fine maps is too weak. PillarNet tackles both. Its encoder is deeper and works on the pillar grid with sparse 2D convolutions, similar to image backbones. Its neck merges fine spatial detail with coarse, meaning-rich features. The paper also describes an overlap-based box loss that accounts for orientation, and a branch that predicts how well each box overlaps the true object so detections can be ranked better. The authors report better numbers than the baselines they compare with, at real-time speed on two driving datasets. The released code does not implement the orientation part as the paper describes it, and this card records that and other differences.

TL;DR

• PillarNet is a pillar-based LiDAR 3D detector that uses only 2D convolutions: a sparse 2D encoder for stages 1 to 4 (1x to 8x), a dense stage 5 (16x), a neck that fuses stage 4 with stage 5, and a center-based head with an IoU branch §3.2 fig. 3.
• Author-reported: nuScenes test NDS 71.4 and mAP 66.0 for PillarNet-34, against 68.5 and 62.4 for AFDetV2 tab. 1.
• The released code lacks the paper's nuScenes encoder: the configs that use the five-stage SpMiddlePillarEncoder18 reference a class that is not in this snapshot. The code also has no VGG backbone and no orientation-decoupled loss. The Waymo configs are the closest match to the stated settings. See "Paper vs code".

Problem

• Voxel detectors that use 3D sparse convolutions are accurate but expensive for onboard use §1.
• Pillar detectors are cheap because they use 2D convolutions, but they lag their voxel counterparts in accuracy §1 §3.1.
• The authors attribute the gap to a light pillar encoder and a weak fusion of coarse maps §1 §3.1.
• They also note that the pillar size sets the output map size. A finer pillar gives a larger map, better accuracy and more compute §1.
• Assumed inputs: LiDAR point clouds. nuScenes is evaluated with 10 accumulated sweeps §4. Waymo is evaluated on single frames, with two-frame results shown for reference §4 tab. 3.

Method

Pipeline fig. 3: points are assigned to pillars and pooled, then the sparse 2D encoder runs, then a dense stage 5, then the neck and the center head. The authors describe the encoder as following the VGG or ResNet layout §3.2.

Pillar features. Points are passed through a one-layer MLP and max-pooled per pillar, using augmented point features §4. The released reader adds the two offsets from each point to its pillar center to the point features, and feeds 32 channels into the MLP dynamic_pillar_encoder.py. The paper does not list the input features.

Encoder. Stages 1 to 4 are sparse 2D convolutions that downsample progressively to 1x, 2x, 4x and 8x §3.2 tab. 6. Stage 5 uses standard 2D convolutions at 16x and has a larger receptive field, which feeds the neck §1 §3.2. Three variants are named: PillarNet-vgg, -18 and -34, with complexity similar to VGGNet, ResNet-18 and ResNet-34 §3.2. The per-layer configuration is said to be in the supplementary material, which is not part of the source given here. The paper's own channel numbers are therefore "not stated".

Code, for the released PillarResNet18 backbone used by the Waymo configs: sparse widths of 32, 64, 128 and 256 at strides 1, 2, 4 and 8, each stage with two basic blocks. Stage 5 is a stride-2 3x3 convolution to 256 channels, followed by two dense 3x3 blocks, at stride 16 PillarResNet.py. This is inferred from code only.

Pillar size. A coarser pillar is obtained by removing the finest encoder stages. For example, an 8x pillar on nuScenes (0.075 m base, so 0.6 m) drops stages 1x, 2x and 4x §1. Results per size are in tab. 6.

Neck. The neck fuses stage 4 (spatial) with stage 5 (semantic) features §3.2 fig. 4. Three designs are compared:

• neckv1 (A): a naive top-down network from SECOND, which concatenates multi-scale maps §3.2.
• neckv2 (B): a group of convolutions exchanges information between the 16x semantic map and the spatial features from the encoder §3.2.
• neckv3 (C): adds convolutions on the 16x map and a second group that fuses the spatial and semantic features §3.2. The paper does not state which neck variant the headline PillarNet rows use. In the released code, the nuScenes configs use RPNV2 nusc_centerpoint_pillarnet_h.py and the Waymo configs use RPNG pillarnet_fpn_iou_centerhead_waymo.py.

Head. A center-based head, following CenterPoint, predicts a heatmap, offset, z height, size, orientation and, on nuScenes, velocity §3.2 §3.4. The Waymo configs use stride 8 for vehicles and stride 4 for pedestrians and cyclists pillarnet_fpn_iou_centerhead_waymo.py. The paper does not state head strides.

IoU-aware rectification eq. 1. A predicted IoU score is used to rectify the classification score at post-processing §3.3:

\hat{S} = S^{1-\beta} \, W_{\rm IoU}^{\beta} eq. 1

Here S is the classification score, W_IoU the predicted IoU, and beta a hyper-parameter §3.3. The IoU head is trained with an L1 loss against the target 2(W - 0.5), where W is the 3D IoU between the predicted and the ground-truth box, so the target lies in [-1, 1] §3.3.

Orientation-decoupled IoU regression fig. 5. The authors argue that orientation, center and size are coupled in IoU-based regression. A center offset can create a local optimum for orientation, and a size change can create a flat plateau §3.3 fig. 5. Their remedy is to decouple orientation theta from the other seven parameters and extend IoU, GIoU and DIoU losses in that way. These are called OD-IoU, OD-GIoU and OD-DIoU §3.3. The paper gives no closed-form equation for the decoupled loss. The formula is "not stated".

Total loss eq. 2.

\mathcal{L}_{total} = \mathcal{L}_{cls} + \mathcal{L}_{iou} + \lambda \left( \mathcal{L}_{od\text{-}iou} + \mathcal{L}_{off} + \mathcal{L}_{z} + \mathcal{L}_{size} + \mathcal{L}_{ori} \right) eq. 2

The heatmap uses focal loss. Offset, z, size and orientation use L1. The weight lambda is "empirically set as in CenterPoint", and its value is not stated in the paper §3.4.

# PillarNet training step  [§3.2] [§3.4] [eq. 2]
# Input:  point cloud (N, C); targets with class, heatmap and 3D boxes
# Output: scalar loss

def training_step(points, targets):
    # Pool the points of each pillar into one feature vector  [§4]
    pillar_features = pillar_encoder(points)       # (P, 32)
    grid = scatter_to_bev(pillar_features)         # sparse 2D grid

    # Stages 1 to 4: sparse 2D convolutions at 1x, 2x, 4x, 8x  [§3.2]
    spatial_8x = sparse_stages_1_to_4(grid)

    # Stage 5: dense 2D convolutions at 16x  [§3.2]
    semantic_16x = dense_stage_5(spatial_8x)

    # Neck fuses the 8x spatial and 16x semantic maps  [§3.2] [fig. 4]
    bev = neck(spatial_8x, semantic_16x)

    # Center head: heatmap, offset, z, size, orientation, IoU  [§3.2]
    heatmap, boxes, iou = center_head(bev)

    # Focal loss on the heatmap  [§3.4]
    loss_cls = focal_loss(heatmap, targets.heatmap)

    # L1 on regression targets  [§3.4]
    loss_reg = l1_loss(boxes, targets.boxes)

    # IoU target: overlap of the detached predicted box with ground truth
    target_iou = overlap_3d(boxes.detach(), targets.boxes)   # (K,)
    loss_iou = l1_loss(iou, 2 * target_iou - 1)              # [§3.3]

    # Overlap-based regression loss (OD variant in the paper)  [§3.3]
    loss_od_iou = overlap_loss(boxes, targets.boxes)

    return loss_cls + loss_iou + lam * (loss_od_iou + loss_reg)  # [eq. 2]

# Post-processing at inference  [§4] [eq. 1]
# Input:  heatmap, boxes, iou output for one scan
# Output: kept boxes, scores and labels

def predict(heatmap, boxes, iou):
    # Find the peak score per location  [§3.2]
    score, label = heatmap_peaks(heatmap)
    # Map the IoU output from [-1, 1] to [0, 1]  [§3.3]
    iou_score = clamp((iou + 1) / 2, 0, 1)
    # Rectified ranking score  [eq. 1]
    rank_score = score ** (1 - beta) * iou_score ** beta
    # Rotated NMS, visiting boxes in rank_score order  [§4]
    keep = rotated_nms(boxes, rank_score, iou_thresh)
    return boxes[keep], score[keep], label[keep]

Training schedule and inference §4. Adam with one-cycle learning rate, weight decay 0.01, and momentum from 0.85 to 0.95. Models are trained from scratch with batch size 16. nuScenes uses a maximum learning rate of 1e-3 for 20 epochs. Waymo uses 3e-3 for 36 epochs. For nuScenes, class-agnostic NMS uses a score threshold of 0.1 and rectification beta 0.5 for all 10 classes. The nuScenes test submission uses double-flip test-time augmentation. For Waymo, class-specific NMS uses IoU thresholds (0.8, 0.55, 0.55) and beta (0.68, 0.71, 0.65) for vehicle, pedestrian and cyclist.

Hyperparameters

Results

Author-reported. Setup: nuScenes test and val, and Waymo test and val, against CenterPoint, AFDetV2 and other LiDAR-only detectors §4 tab. 1–4. The runtime comparison uses a desktop with an i9 CPU and an RTX 3090 §4. The paper does not state the hardware for the test-set speeds (14, 13 and 12 FPS for PillarNet-vgg, -18 and -34) §4.

The nuScenes test gains are 2.3 NDS and 2.6 mAP for PillarNet-18 over AFDetV2 §4.

Ablation, nuScenes val (condensed from tab. 4). The paper's full table also lists mATE, mASE, mAOE, mAVE and mAAE, which are omitted here.

Encoder variants tab. 5.

These variants are reported with the IoU-related modules detached §4.

Pillar sizes tab. 6 (PillarNet-18 encoder; stages listed as in the table):

The paper's Waymo test and val tables tab. 2 tab. 3 cover all classes and both difficulty levels. They are not reproduced here in full. Check them in the paper.

Paper vs code

Code: repository at commit 2cfa24537cd8974a579a39f57426957bb9d8b134 (https://github.com/agent-sgs/PillarNet). The paper's abstract links it, and its README names the paper's authors and gives the author's contact address README.md. It is therefore used as the official code. Searched: the link in the paper's abstract, which was enough. No LICENSE file was found in the repository root directory.

• Orientation-decoupled IoU loss is not implemented as described. The paper says orientation is decoupled from the other parameters §3.3. The regression loss IouRegLoss calls bbox3d_overlaps_giou, which builds rectangles from the center and the width and length only, then takes min and max over their corners. It does not use the yaw angle, so the overlap ignores heading. Height enters only through a separate interval centernet_loss.py center_utils.py center_utils.py. No separate orientation-decoupled term was found. The IoU-aware branch target does use a rotated overlap centernet_loss.py.
• Paper says rectification uses beta 0.5 on nuScenes §4; the nuScenes configs set beta to 0. The released nuScenes configs set rectifier=0 nusc_centerpoint_pillarnet_h.py pillarnet_centerhead_nusc.py. Waymo uses (0.68, 0.71, 0.65), matching the paper pillarnet_fpn_iou_centerhead_waymo.py.
• Rectification in code changes NMS ordering, not the reported score. The paper's equation 1 gives a rectified score §3.3. In code, the rectified score only sets the order for NMS box_torch_ops.py. The returned score is the raw score unless use_rectify is set box_torch_ops.py. The head passes use_rectify only when the config has it, and none of the PillarNet configs set it center_head.py.
• Batch size. The paper gives 16 on 4 V100 GPUs §4. The nuScenes _h config sets samples_per_gpu=1 with 8 devices nusc_centerpoint_pillarnet_h.py nusc_centerpoint_pillarnet_h.py. The base nuScenes config and the Waymo config set 4 per GPU with 8 devices pillarnet_centerhead_nusc.py pillarnet_fpn_iou_centerhead_waymo.py. None of these matches the paper's total. The learning rate is not rescaled.
• Five-stage nuScenes encoder is missing. The paper describes five encoder stages and detachable stages for pillar sizes §1 §3.2. The nuScenes config nusc_centerpoint_pillarnet_h.py uses SpMiddlePillarEncoder18 with an Identity reader nusc_centerpoint_pillarnet_h.py. Neither class is registered in det3d in this snapshot. Four nuScenes configs and one Waymo config reference SpMiddlePillarEncoder18. Twenty-one configs reference some SpMiddle* class. The code that builds these encoders is not in the repository.
• VGG variant missing. The paper's PillarNet-vgg is not in the code. The backbones are PillarResNet18, PillarResNet18S, PillarResNet34 and PillarResNet34S only PillarResNet.py.
• Double flip is off. The paper uses double-flip test-time augmentation on the nuScenes test set §4. The PillarNet configs set DOUBLE_FLIP = False nusc_centerpoint_pillarnet_h.py. Flip TTA appears only in VoxelNet configs nusc_centerpoint_voxelnet_0075voxel_dcn_flip.py.
• NMS details not in paper. The paper gives class-agnostic NMS for nuScenes §4. The code uses rotated NMS with IoU 0.2 and at most 83 boxes nusc_centerpoint_pillarnet_h.py. Waymo pre- and post-NMS sizes appear only in code pillarnet_fpn_iou_centerhead_waymo.py.
• Loss weights. The paper uses one lambda, set "as in CenterPoint" §3.4. The code has separate weights. nuScenes uses 0.25 for the box term and 0.25 for the regression-IoU term pillarnet_centerhead_nusc.py. Waymo uses 2 and 2 pillarnet_fpn_iou_centerhead_waymo.py.
• Pillar reader. The paper says the MLP uses augmented point features with max pooling §4. The released reader is DynamicPFE with a 32-channel MLP pillarnet_fpn_iou_centerhead_waymo.py dynamic_pillar_encoder.py.
• Consistent items. The IoU target 2 * IoU - 1 with an L1 loss matches the paper §3.3 centernet_loss.py. The nuScenes detection range, pillar size and sweeps match §4 nusc_centerpoint_pillarnet_h.py nusc_centerpoint_pillarnet_h.py. Waymo's pillar size, range, learning rate and epochs match §4 pillarnet_fpn_iou_centerhead_waymo.py pillarnet_fpn_iou_centerhead_waymo.py pillarnet_fpn_iou_centerhead_waymo.py.
• Waymo subsampling. The Waymo config sets load_interval=4 for training pillarnet_fpn_iou_centerhead_waymo.py. The paper does not mention subsampling. In waymo.py the attribute is stored but not used, according to a text search waymo.py.
• Speed hardware. The paper's test-set FPS (14, 13, 12) do not state the hardware §4. The runtime comparison uses a desktop with an i9 CPU and an RTX 3090 §4.

Pitfalls

• Orientation is not in the regression overlap. The released overlap loss is axis-aligned. A reimplementation that follows the paper's OD-IoU will differ from the released code. Use the paper's description only if you accept that the paper's exact formula is missing.
• The paper's OD gain is measured against a different row. In tab. 4, the plain IoU row has mAP 59.82, above OD-DIoU at 59.72. The paper's +0.24 gain for OD-DIoU is relative to neckv3 (59.48), not to the plain IoU row §4. NDS is 67.16 for plain IoU, 67.39 for OD-DIoU. The paper does not isolate the decoupling effect cleanly.
• Rectification is off in the nuScenes configs. Set beta explicitly. For Waymo, rectification only changes NMS order unless use_rectify is set box_torch_ops.py.
• Dimension clamp. The head clamps the log-size output to [-1.2, 3.2] before taking the exponential center_head.py center_head.py. This restricts sizes to about 0.3 m to 24.5 m. The code comment says "dimensions between [0.3, 25] is valid". Predictions outside that range are silently clipped.
• Heatmap clamp. Sigmoid outputs are clamped to [1e-4, 1 - 1e-4] before the focal loss center_head.py.
• IoU output mapping. At inference the IoU output is mapped as (x + 1) / 2 and clamped to [0, 1] center_head.py. This is the inverse of the training target 2 * IoU - 1.
• Coarser pillars are slower. The paper attributes slower coarse-pillar inference to atomic max operations over many points per pillar §4. It suggests sub-sampling or streaming pillarization as fixes §4.
• Batch-size gap. Reproducing the paper's learning rate needs the paper's batch size of 16. The released configs give other effective batch sizes. Check the total before tuning.
• Supplementary material missing. The layer configuration for the variants is in the paper's supplementary material §3.2. That document is not in the source used here, so channel-level details for PillarNet-vgg and -34 are not stated.
• The nuScenes encoder cannot be built from this snapshot. The headline nuScenes configs need SpMiddlePillarEncoder18, which is not in the repository. The Waymo configs use PillarResNet18 and can be instantiated from it.
• No license for the code. There is no LICENSE file. Do not copy code snippets from this repository. Write pseudocode from the cited lines instead.
• Lambda is not given. The paper gives no value §3.4. Use the nuScenes or Waymo values from the configs, and say which one you chose.
• Velocity weight. nuScenes gives velocity a box weight of 0.2 nusc_centerpoint_pillarnet_h.py. The paper does not mention this.

Lineage

• Builds on: PointPillars for the pillar baseline and its neck §2 §3.1. SECOND for the sparse encoder and the top-down neck (neckv1) §3.1 §3.2. VGGNet and ResNet for the encoder layout §3.2. FPN for the neck concept §3.2. CenterPoint for the center-based head, the loss structure and the training schedule §3.2 §3.4 §4. AFDetV2 for the IoU rectification and Waymo NMS settings §3.3 §4. GIoU and DIoU for the overlap losses §3.3. None of these is linked to a card yet.
• Compared against: CenterPoint, AFDetV2, HotSpotNet, CVCNet, 3DVID and others on nuScenes test tab. 1. On Waymo, RSN, LiDAR R-CNN, Part-A2, PV-RCNN and others tab. 2 tab. 3.
• Superseded by: not stated. The repository README names Pillar R-CNN (arXiv:2302.13301) by the same authors as a follow-up. It is not compiled here README.md.