Sparse convolution

Overview

A sparse convolution applies a learned filter only at the occupied cells of a grid. A LiDAR scan turned into voxels or pillars is mostly empty, so the grid is kept as a list of occupied cells and their features, and the filter is evaluated only where data exists. MinkLoc3D cites the memory cost of dense voxel grids for large clouds as the reason to avoid them 2011.04530 §2.

The invariant shared by the variants is that empty cells are never stored and their values are never computed. The two cards differ in what an occupied cell holds. In MinkLoc3D each occupied voxel takes a single input feature, the constant 1 2011.04530 §3.1. In PillarNet each non-empty pillar carries a feature vector produced by a point MLP and max pooling over its points 2205.07403 §4. The paper's stated reason for sparse 2D stages is not recorded on the card, so this page does not give one.

Variants

Stride-2 sparse 3D feature pyramid with top-down fusion (MinkLoc3D)

This is the variant in MinkLoc3D. The voxel grid has a step of 0.01 and is limited to 200 voxels per axis 2011.04530 §4.1. The network is built from the blocks below.

The upsampled map is then combined with the lower lateral map. The paper says the two are concatenated 2011.04530 §3.1. The card's comparison with the released code finds that the code adds them instead (see Contradictions). The per-voxel features are pooled with generalized-mean (GeM) pooling into one 256-number descriptor 2011.04530 §3.1. The paper reports that the top-down path helped in its early experiments, but gives no table for it 2011.04530 §3.1.

Card: MinkLoc3D: Point Cloud Based Large-Scale Place Recognition.

Sparse 2D encoder on a pillar grid (PillarNet)

PillarNet works on a top-down grid of pillars, so its sparse convolutions are 2D. Stages 1 to 4 are sparse 2D convolutions that downsample to 1x, 2x, 4x and 8x. Stage 5 is a standard dense 2D convolution at 16x, and the neck fuses the 8x spatial map with the 16x semantic map 2205.07403 §3.2. The paper names three encoder sizes, PillarNet-vgg, -18 and -34, with complexity similar to VGGNet, ResNet-18 and ResNet-34. Their per-layer channels are in supplementary material that the card did not have, so the paper's channel counts are not stated 2205.07403 §3.2.

The released Waymo backbone (PillarResNet18) uses sparse widths of 32, 64, 128 and 256 at strides 1, 2, 4 and 8, then a dense stage 5 at stride 16 2205.07403 Paper vs code. That is the only encoder the card could instantiate from the released code.

Card: PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection.

Other variants

Sparse convolutions that keep the set of active sites fixed, and sparse convolutions that produce dense per-point output, are not covered by these two cards. Do not infer their behaviour from this page.

When to use which

The table uses only the constraints the cards report: model size, runtime, memory, the target task and the encoder depth.

Reported accuracy is author-reported. MinkLoc3D reports AR@1% of 97.9 on Oxford with baseline training and 98.5 with refined training 2011.04530 tab. 4 2011.04530 tab. 5. Its ablations vary the pooling and the descriptor size, not the sparse backbone, so they do not isolate the contribution of the sparse convolutions 2011.04530 tab. 7 2011.04530 tab. 8.

Implementation sketch

Language-agnostic pseudocode of the two backbones. Values are from the papers; the cards note where the released code differs. Each block is a separate pipeline.

# Sparse feature pyramid on quantized LiDAR voxels (MinkLoc3D)  [arxiv:2011.04530 §3.1]
# Input: points (N, 3) in [-1, 1], N = 4096  [arxiv:2011.04530 §4.1]
def sparse_pyramid(points):
    # Quantize to a 0.01 grid; store only occupied voxels, feature 1  [§3.1]
    coords, feats = quantize(points, step=0.01)
    # Stem: 5x5x5 sparse conv, batch norm, ReLU, 32 channels  [tab. 1]
    x = relu(bn(sparse_conv(coords, feats, k=5, c=32)))
    kept = []
    # Three stride-2 blocks: kernel-2 conv, then a residual block  [tab. 1]
    for c in [32, 64, 64]:
        x = relu(bn(sparse_conv(x, k=2, stride=2, c=c)))
        x = residual(x, k=3, c=c)   # two 3x3x3 convs with a skip
        kept.append(x)
    # 1x1 lateral convs to 256 channels on the top two maps  [tab. 1]
    top = conv1x1(kept[-1], 256)
    mid = conv1x1(kept[-2], 256)
    # Transposed sparse conv, kernel 2, stride 2, one level up  [tab. 1]
    up = sparse_transpose_conv(top, k=2, stride=2)
    # Paper: concatenate. Released code: add (see Contradictions)  [§3.1]
    fused = up + mid
    # GeM pooling over occupied voxels follows  [§3.1]
    return fused

# Sparse 2D encoder on a pillar grid, then a dense stage 5 (PillarNet)
# [arxiv:2205.07403 §3.2]. Widths are from the released Waymo backbone.
# Input: pillar features (P, 32) at (row, col) grid indices
def pillar_encoder(pillar_feats, pillar_idx):
    # Non-empty pillars only; no values stored for empty cells
    x = sparse2d(pillar_feats, pillar_idx)
    # Stages 1 to 4: sparse 2D convs at strides 1, 2, 4, 8  [§3.2]
    for width in [32, 64, 128, 256]:
        x = sparse_stage(x, width)       # [arxiv:2205.07403 Paper vs code]
    # Stage 5: dense 2D convs at 16x, larger receptive field  [§3.2]
    semantic = dense_stage5(to_dense(x))
    # Neck fuses the 8x spatial map with the 16x semantic map  [§3.2]
    return neck(x, semantic)

Contradictions

• Top-down fusion (MinkLoc3D). The paper says the upsampled and lateral maps are concatenated 2011.04530 §3.1. The released code adds them, as the card's Paper vs code section records. The two give different channel counts for the next layer, so an implementation must pick one.
• Parameter count (MinkLoc3D). Section 4.2 gives 1.5 million trainable parameters, and Table 6 gives 1.1M for the same model 2011.04530 §4.2 2011.04530 tab. 6.
• Runtime (MinkLoc3D). The text gives 22 ms per cloud and Table 6 gives 21 ms 2011.04530 §4.2 2011.04530 tab. 6.
• Point removal in augmentation (MinkLoc3D). The paper describes removing points 2011.04530 §4.1. The released code sets their coordinates to zero, so they stay in the count and fall into the origin voxel. The card records this as a paper-vs-code difference.
• Encoder variants (PillarNet). The paper's PillarNet-vgg and the nuScenes encoder are not in the released code. The paper's stated backbone set is therefore not fully reproducible from the repository [arxiv:2205.07403 Paper vs code].
• Across the two cards. The MinkLoc3D and PillarNet numbers measure different tasks (place-recognition AR@1% against detection NDS and mAP). They are not comparable, and neither paper compares against the other 2011.04530 tab. 4 2205.07403 tab. 1.

Papers

• MinkLoc3D: Point Cloud Based Large-Scale Place Recognition: a sparse 3D feature pyramid on quantized LiDAR scans, with GeM pooling and a 256-number descriptor for place recognition.
• PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection: a sparse 2D encoder on a pillar grid, a dense stage 5 and a neck that fuses spatial and semantic maps for LiDAR 3D detection.