---
kind: concept
slug: sparse-convolution
title: Sparse convolution
summary: >-
  A convolution that computes only at occupied sites of a grid, so cost follows the number of occupied cells, not the full grid. MinkLoc3D uses a sparse 3D feature pyramid for LiDAR place recognition. PillarNet uses sparse 2D convolutions on a pillar grid for 3D detection. The two cards report different tasks, so their numbers are not comparable.
tags: [point-cloud, voxel, pillar, lidar, convolution, sparsity]
papers: ["2011.04530", "2205.07403"]
related: [convolutional-neural-network, dilated-convolution, point-to-grid-interpolation, metric-learning, bev-representation, object-detection]
status: draft
updated_at: 2026-11-06
---

## Overview

A sparse convolution applies a learned filter only at the occupied cells of a
grid. A LiDAR scan turned into voxels or pillars is mostly empty, so the grid is
kept as a list of occupied cells and their features, and the filter is evaluated
only where data exists. MinkLoc3D cites the memory cost of dense voxel grids for
large clouds as the reason to avoid them [arxiv:2011.04530 §2].

The invariant shared by the variants is that empty cells are never stored and
their values are never computed. The two cards differ in what an occupied cell
holds. In MinkLoc3D each occupied voxel takes a single input feature, the
constant 1 [arxiv:2011.04530 §3.1]. In PillarNet each non-empty pillar carries a
feature vector produced by a point MLP and max pooling over its points
[arxiv:2205.07403 §4]. The paper's stated reason for sparse 2D stages is not
recorded on the card, so this page does not give one.

## Variants

### Stride-2 sparse 3D feature pyramid with top-down fusion (MinkLoc3D)

This is the variant in MinkLoc3D. The voxel grid has a step of 0.01 and is
limited to 200 voxels per axis [arxiv:2011.04530 §4.1]. The network is built
from the blocks below.

| Stage | Operation | Channels | Source |
|---|---|---|---|
| Stem | 5x5x5 sparse conv, batch norm, ReLU | 32 | [arxiv:2011.04530 tab. 1] |
| Bottom-up 1 to 3 | Stride-2 conv (kernel 2), batch norm, ReLU, then one residual block of two 3x3x3 convs | 32, 64, 64 | [arxiv:2011.04530 tab. 1] [arxiv:2011.04530 §3.1] |
| Lateral | 1x1 conv on the last bottom-up map and on the one below it | 256 | [arxiv:2011.04530 tab. 1] |
| Upsampling | Transposed conv, kernel 2, stride 2, on the top lateral map | 256 | [arxiv:2011.04530 tab. 1] |

The upsampled map is then combined with the lower lateral map. The paper says
the two are concatenated [arxiv:2011.04530 §3.1]. The card's comparison with the
released code finds that the code adds them instead (see Contradictions). The
per-voxel features are pooled with generalized-mean (GeM) pooling into one
256-number descriptor [arxiv:2011.04530 §3.1]. The paper reports that the
top-down path helped in its early experiments, but gives no table for it
[arxiv:2011.04530 §3.1].

Card: [MinkLoc3D: Point Cloud Based Large-Scale Place Recognition](/papers/2011.04530).

### Sparse 2D encoder on a pillar grid (PillarNet)

PillarNet works on a top-down grid of pillars, so its sparse convolutions are
2D. Stages 1 to 4 are sparse 2D convolutions that downsample to 1x, 2x, 4x and
8x. Stage 5 is a standard dense 2D convolution at 16x, and the neck fuses the
8x spatial map with the 16x semantic map [arxiv:2205.07403 §3.2]. The paper
names three encoder sizes, PillarNet-vgg, -18 and -34, with complexity similar
to VGGNet, ResNet-18 and ResNet-34. Their per-layer channels are in
supplementary material that the card did not have, so the paper's channel
counts are not stated [arxiv:2205.07403 §3.2].

The released Waymo backbone (PillarResNet18) uses sparse widths of 32, 64, 128
and 256 at strides 1, 2, 4 and 8, then a dense stage 5 at stride 16
[arxiv:2205.07403 Paper vs code]. That is the only encoder the card could
instantiate from the released code.

Card: [PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection](/papers/2205.07403).

### Other variants

Sparse convolutions that keep the set of active sites fixed, and sparse
convolutions that produce dense per-point output, are not covered by these two
cards. Do not infer their behaviour from this page.

## When to use which

The table uses only the constraints the cards report: model size, runtime,
memory, the target task and the encoder depth.

| Situation | Use | Why |
|---|---|---|
| One global descriptor per LiDAR scan, small model | The MinkLoc3D pyramid | Reported at 1.1M parameters and 21 ms per cloud, against 19.8M and 15 ms for PointNetVLAD [arxiv:2011.04530 tab. 6]. It is smaller but not faster than PointNetVLAD. |
| Large clouds where a dense 3D grid would not fit in memory | Sparse storage of occupied cells | The paper states that dense grids were considered too memory-hungry for large clouds [arxiv:2011.04530 §2]. |
| Real-time 3D box detection from LiDAR on a pillar grid | The PillarNet sparse 2D encoder | Author-reported on nuScenes val: PillarNet-18 at 16 FPS with 67.39 NDS, against CenterPoint-SECOND at 8 FPS with 66.76 NDS [arxiv:2205.07403 tab. 4]. |
| Choosing encoder depth for PillarNet | -vgg, -18 or -34 | PillarNet-vgg 57.67 mAP and 65.71 NDS at 16 FPS; -18 59.41 and 67.09 at 16 FPS; -34 59.98 and 67.50 at 14 FPS [arxiv:2205.07403 tab. 5]. These runs have the IoU modules detached. |
| Finer pillars for accuracy, with a runtime budget | A 0.075 m pillar, and a coarser one only if speed matters more | 0.075 m gives 59.48 mAP and 67.15 NDS at 16 FPS; 0.6 m gives 55.37 and 64.20 at 14 FPS [arxiv:2205.07403 tab. 6]. The paper attributes slower coarse-pillar inference to atomic max operations over many points per pillar [arxiv:2205.07403 §4]. |
| Reproducing the PillarNet nuScenes numbers | Check the code first | The nuScenes configs use an encoder class that is not in the released repository. The Waymo backbone can be built from the code [arxiv:2205.07403 Paper vs code]. |
| Per-point labels, segmentation or other dense outputs | Not covered by these cards | The MinkLoc3D card reports only a global descriptor and retrieval accuracy [arxiv:2011.04530 §4.2]. The PillarNet card reports 3D boxes only [arxiv:2205.07403 §4]. |

Reported accuracy is author-reported. MinkLoc3D reports AR@1% of 97.9 on Oxford
with baseline training and 98.5 with refined training [arxiv:2011.04530 tab. 4]
[arxiv:2011.04530 tab. 5]. Its ablations vary the pooling and the descriptor
size, not the sparse backbone, so they do not isolate the contribution of the
sparse convolutions [arxiv:2011.04530 tab. 7] [arxiv:2011.04530 tab. 8].

## Implementation sketch

Language-agnostic pseudocode of the two backbones. Values are from the papers;
the cards note where the released code differs. Each block is a separate
pipeline.

```pseudocode
# Sparse feature pyramid on quantized LiDAR voxels (MinkLoc3D)  [arxiv:2011.04530 §3.1]
# Input: points (N, 3) in [-1, 1], N = 4096  [arxiv:2011.04530 §4.1]
def sparse_pyramid(points):
    # Quantize to a 0.01 grid; store only occupied voxels, feature 1  [§3.1]
    coords, feats = quantize(points, step=0.01)
    # Stem: 5x5x5 sparse conv, batch norm, ReLU, 32 channels  [tab. 1]
    x = relu(bn(sparse_conv(coords, feats, k=5, c=32)))
    kept = []
    # Three stride-2 blocks: kernel-2 conv, then a residual block  [tab. 1]
    for c in [32, 64, 64]:
        x = relu(bn(sparse_conv(x, k=2, stride=2, c=c)))
        x = residual(x, k=3, c=c)   # two 3x3x3 convs with a skip
        kept.append(x)
    # 1x1 lateral convs to 256 channels on the top two maps  [tab. 1]
    top = conv1x1(kept[-1], 256)
    mid = conv1x1(kept[-2], 256)
    # Transposed sparse conv, kernel 2, stride 2, one level up  [tab. 1]
    up = sparse_transpose_conv(top, k=2, stride=2)
    # Paper: concatenate. Released code: add (see Contradictions)  [§3.1]
    fused = up + mid
    # GeM pooling over occupied voxels follows  [§3.1]
    return fused
```

```pseudocode
# Sparse 2D encoder on a pillar grid, then a dense stage 5 (PillarNet)
# [arxiv:2205.07403 §3.2]. Widths are from the released Waymo backbone.
# Input: pillar features (P, 32) at (row, col) grid indices
def pillar_encoder(pillar_feats, pillar_idx):
    # Non-empty pillars only; no values stored for empty cells
    x = sparse2d(pillar_feats, pillar_idx)
    # Stages 1 to 4: sparse 2D convs at strides 1, 2, 4, 8  [§3.2]
    for width in [32, 64, 128, 256]:
        x = sparse_stage(x, width)       # [arxiv:2205.07403 Paper vs code]
    # Stage 5: dense 2D convs at 16x, larger receptive field  [§3.2]
    semantic = dense_stage5(to_dense(x))
    # Neck fuses the 8x spatial map with the 16x semantic map  [§3.2]
    return neck(x, semantic)
```

## Contradictions

- **Top-down fusion (MinkLoc3D).** The paper says the upsampled and lateral maps
  are concatenated [arxiv:2011.04530 §3.1]. The released code adds them, as the
  card's Paper vs code section records. The two give different channel counts
  for the next layer, so an implementation must pick one.
- **Parameter count (MinkLoc3D).** Section 4.2 gives 1.5 million trainable
  parameters, and Table 6 gives 1.1M for the same model [arxiv:2011.04530 §4.2]
  [arxiv:2011.04530 tab. 6].
- **Runtime (MinkLoc3D).** The text gives 22 ms per cloud and Table 6 gives 21 ms
  [arxiv:2011.04530 §4.2] [arxiv:2011.04530 tab. 6].
- **Point removal in augmentation (MinkLoc3D).** The paper describes removing
  points [arxiv:2011.04530 §4.1]. The released code sets their coordinates to
  zero, so they stay in the count and fall into the origin voxel. The card
  records this as a paper-vs-code difference.
- **Encoder variants (PillarNet).** The paper's PillarNet-vgg and the
  nuScenes encoder are not in the released code. The paper's stated backbone
  set is therefore not fully reproducible from the repository [arxiv:2205.07403
  Paper vs code].
- **Across the two cards.** The MinkLoc3D and PillarNet numbers measure
  different tasks (place-recognition AR@1% against detection NDS and mAP). They
  are not comparable, and neither paper compares against the other
  [arxiv:2011.04530 tab. 4] [arxiv:2205.07403 tab. 1].

## Papers

- [MinkLoc3D: Point Cloud Based Large-Scale Place Recognition](/papers/2011.04530): a sparse 3D feature pyramid on quantized LiDAR scans, with GeM pooling and a 256-number descriptor for place recognition.
- [PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection](/papers/2205.07403): a sparse 2D encoder on a pillar grid, a dense stage 5 and a neck that fuses spatial and semantic maps for LiDAR 3D detection.