MinkLoc3D: Point Cloud Based Large-Scale Place Recognition

In plain words

Place recognition asks a simple question: have I been here before? A car or robot with a laser scanner (LiDAR) records a 3D point cloud of its surroundings. MinkLoc3D turns each scan into one short list of numbers, a descriptor, so that two scans of the same street give similar lists and scans of different streets give different ones. To find a place, the system searches a database of descriptors for the nearest match.

The core idea is to stop treating a scan as an unordered pile of points. The space is cut into small cubes (voxels), and only the occupied cubes are kept, much like an image where most pixels are empty. Small 3D filters can then slide over the occupied cubes and pick up local shapes. A small pyramid of these layers mixes fine and coarse detail, and a simple averaging-style step turns the result into one vector. Training pulls matching scans together and pushes non-matching ones apart, and it picks the hardest pairs inside each batch.

The approach is meant to be simpler than earlier point-based methods that add graph networks or hand-made features. It is reported to beat those earlier methods on the standard benchmarks, which makes it a useful reference for LiDAR-based relocalisation.

TL;DR

MinkLoc3D is a learned global descriptor for 3D point clouds. It quantizes each cloud into a sparse voxel grid, extracts local features with a small sparse 3D feature pyramid, and pools them with generalized-mean (GeM) pooling into 256 numbers §3.1. It is trained with batch-hard triplet loss, and its batch size grows during training §3.2. The headline result is author-reported: AR@1% of 97.9 on Oxford with baseline training and 98.5 with refined training tab. 4 tab. 5. Treat the code as the reference for the training details, because several augmentation and fusion details differ from the paper (see Paper vs code).

Problem

Learned place recognition on point clouds had been built on PointNet, which handles each point in isolation and captures local geometry poorly. PointNetVLAD used it for the local features and NetVLAD for aggregation §1 §2. Later methods added attention, graph networks or hand-made features to recover local context, at the cost of more complexity and compute §1 §2. Dense voxel grids were considered too memory-hungry for large clouds §2. Training also needed many embeddings per triplet: PointNetVLAD and LPD-Net sample 2 positives and 18 negatives per anchor and compute 21 embeddings for one triplet §3.2.

Assumptions: each query is one LiDAR scan with ground removed and about 4096 points, and positives and negatives are defined by ground-truth UTM positions §4.1.

Method

The network has two parts: a local feature extraction network and a GeM pooling layer §3.1 fig. 2.

1. Quantize. The point coordinates are shifted and scaled to [-1, 1]. They are quantized into a sparse grid with a step of 0.01; every occupied voxel gets a single feature equal to 1 §3.1 §4.1 dataset_utils.py.
2. Bottom-up pass. Conv0 uses 5x5x5 kernels with 32 channels, batch norm and ReLU tab. 1. Three further blocks, with 32, 64 and 64 channels, each start with a stride-2 convolution (kernel 2), batch norm and ReLU, followed by one residual block of two 3x3x3 convolutions tab. 1 §3.1 minkfpn.py minkfpn.py.
3. Top-down pass. The last bottom-up map goes through a 1x1 lateral convolution to 256 channels, then a transposed convolution (kernel 2, stride 2) upsamples it. A second 1x1 lateral convolution maps the map one level up to 256 channels. The paper says the two are concatenated; the code adds them (see Paper vs code) tab. 1 §3.1 minkfpn.py.
4. Pool. GeM pooling is applied per channel over all occupied voxels, with a learned exponent p, which gives the 256-d descriptor §3.1:

g^{(k)} = \left( \frac{1}{n} \sum_{j=1}^{n} \left( f_j^{(k)} \right)^{p} \right)^{1/p} §3.1

where n is the number of occupied voxels and p is a single learned scalar §3.1.

5. Batch construction. Each epoch the training set is split into batches of pairs of similar clouds, so a batch of size n holds n/2 pairs. Each cloud is used once per epoch where possible §3.2 samplers.py.
6. Mining and loss. Inside a batch, the hardest positive (farthest similar cloud) and hardest negative (closest dissimilar cloud) are chosen for each anchor. The triplet hinge uses Euclidean distance and margin m = 0.2 §3.2 loss.py:

L(a_i,p_i,n_i) = \max \{ d(a_i,p_i) - d(a_i,n_i) + m,\ 0 \} §3.2

7. Dynamic batch size. After each epoch, if the share of triplets with non-zero loss is below Θ = 0.7, the batch size is multiplied by α = 1.4, up to 256 §3.2 tab. 3 trainer.py.
8. Augmentation (training only). Per cloud: per-point jitter, removal of a random fraction of points, a random translation, and erasing of a random block. The paper gives the values in §4.1; the code's concrete implementation differs in several places (see Paper vs code) oxford.py.
9. Inference. Each scan becomes one descriptor. Queries are matched to the database by nearest neighbour in Euclidean distance §4.1 evaluate.py.

# Local feature network and GeM pooling  [§3.1] [tab. 1]
# Input:  points (N, 3), N = 4096, coords in [-1, 1]  [§4.1]
# Output: descriptor (256,)
def describe(points):
    # Quantize to a 0.01 grid; keep occupied voxels with feature 1  [§3.1]
    voxels, feats = quantize(points, step=0.01)
    # Conv0: 5x5x5 kernel, 32 channels, BatchNorm, ReLU  [tab. 1]
    x0 = relu(batch_norm(conv(voxels, feats, kernel=5, channels=32)))
    # Three bottom-up blocks: stride-2 conv (kernel 2), BN, ReLU,
    # then one residual block of two 3x3x3 convs  [tab. 1] [§3.1]
    x1 = bottom_up_block(x0, channels=32)
    x2 = bottom_up_block(x1, channels=64)
    x3 = bottom_up_block(x2, channels=64)
    # Lateral 1x1 convs map the two kept maps to 256 channels  [tab. 1]
    lateral_top = conv1x1(x3, out_channels=256)
    lateral_mid = conv1x1(x2, out_channels=256)
    # Top-down: transposed conv 2x2x2, stride 2, upsamples the top map  [tab. 1]
    upsampled = transpose_conv(lateral_top, kernel=2, stride=2)
    # Paper says concatenate; code adds the two maps (see Paper vs code)
    fused = upsampled + lateral_mid   # (occupied voxels, 256)
    # GeM over occupied voxels, one learned p shared by all channels  [§3.1]
    clamped = clamp(fused, min=1e-6)
    descriptor = mean(clamped ** p, over voxels) ** (1 / p)
    return descriptor   # (256,)

# Epoch batching and batch-hard triplet loss  [§3.2] [§4.1]
# Positive: within 10 m. Non-negative: within 50 m. Between: ignored.
def make_batches(clouds, positives, batch_size):
    # Each cloud is taken as an anchor once per epoch where possible
    unused = set(clouds)
    batches = []
    while len(unused) > 0:
        batch = []
        while len(batch) < batch_size and len(unused) > 0:
            anchor = pick_random(unused)
            unused.remove(anchor)
            # Prefer an unused partner  [code: datasets/samplers.py#L125-L132]
            free = [p for p in positives[anchor] if p in unused]
            if len(free) > 0:
                partner = pick_random(free)
                unused.remove(partner)
            else:
                partner = pick_random(positives[anchor])
            batch += [anchor, partner]
        batches.append(batch)
    return batches

# Batch-hard triplet loss on one batch of n embeddings  [§3.2]
def batch_hard_loss(emb, pos_mask, neg_mask, margin=0.2):
    # Pairwise Euclidean distances, shape (n, n)  [§3.2]
    dist = pairwise_l2(emb)
    # Mining indices come from detached distances  [code: models/loss.py#L37]
    p_idx, n_idx, keep = mine(dist.detach(), pos_mask, neg_mask)
    anchors = arange(n)[keep]
    # Hardest positive: farthest true positive; hardest negative: closest
    d_ap = dist[anchors, p_idx[keep]]
    d_an = dist[anchors, n_idx[keep]]
    # Hinge per triplet, on live (non-detached) distances  [§3.2]
    hinge = relu(d_ap - d_an + margin)
    # Average over triplets with non-zero loss only  [code: models/loss.py#L81]
    return mean(hinge[hinge > 0])

Dataset sizes tab. 2:

Training scenarios: baseline uses Oxford only; refined uses Oxford plus in-house training subsets. Both are evaluated on Oxford and the three in-house test sets §4.1. Oxford data come from a 2D SICK LiDAR; in-house data from a Velodyne HDL-64 §4.1.

Hyperparameters

Paper values are given first; code values are cited where they differ or add detail.

Results

Author-reported. Setup: Oxford and three in-house LiDAR sets, two training scenarios; one NVIDIA RTX 2080 Ti, PyTorch 1.5, MinkowskiEngine 0.4.3, PML 0.9.88 §4.1. Baselines are PointNetVLAD, PCAN, DAGC, LPD-Net and DH3D §4.2.

Baseline training (AR@1%) tab. 4

Refined training (AR@1% and AR@1) tab. 5. For PointNetVLAD and PCAN the authors used their released models; LPD-Net was trained from scratch by the authors §4.2.

Computation tab. 6

Ablations (baseline training; Oxford AR@1%, in-house AR@1%) tab. 7 tab. 8

The paper also reports that the top-down path helped in its initial experiments §3.1 (no table in the paper).

RobotCar Seasons, 6-DoF localization, % within 5 m and 10 degrees tab. 9. Columns: dawn, dusk, overcast summer, overcast winter, rain, snow, sun, night, night-rain. Image-based methods are in the first five rows and LiDAR methods in the last two (the paper's grouping).

The paper's own summary is that the point-cloud methods are weaker than image methods in day conditions and stronger at night, and that its method beats all image methods at night except NetVLAD+SP+SG §4.2.

Paper vs code

Code: official repository jac99/MinkLoc3D at commit 4700dfda6b15ed7d12bf3a83ed433dbf6f715ae3, MIT, owned by the first author. Found via the abstract link, so no search was needed. Evaluation numbers were not re-run for this card; the README reports the same numbers as Tables 4 and 5 README.md.

• Batch size. Table 3 gives an initial baseline batch size of 32; the baseline config uses 16 tab. 3 config_baseline.txt.
• Top-down fusion. The text says the upsampled map is concatenated with the lateral features §3.1. The code adds them elementwise minkfpn.py.
• Translation. Paper: a shift sampled uniformly in 0 to 0.01 §4.1. Code: a Gaussian with sigma 0.01 per axis oxford.py.
• Jitter clipping. The paper gives sigma 0.001 only §4.1. The code also clamps the jitter to ±0.002 oxford.py.
• Point removal. The paper says points are removed §4.1. The code sets the coordinates of the chosen points to (0, 0, 0), so they stay in the point count and fall into the origin voxel oxford.py.
• Block erasing. The paper gives no probability or size §4.1. The code erases with probability 0.4 and an area fraction 0.02 to 0.33 oxford.py oxford.py. The erased region is a rectangle in x and y that covers all heights oxford.py.
• Undocumented batch transforms. The code applies one random rotation (±5 degrees about z) and one random flip per batch to all clouds in it oxford.py dataset_utils.py. Neither appears in the paper.
• Refined positives. The paper gives a 10 m positive radius for both datasets §4.1. The refined training-tuple script uses 12.5 m generate_training_tuples_refine.py.
• Loss details. The code uses triplet loss with swap=True and averages over triplets with non-zero loss only loss.py. The paper describes neither.
• GeM. The code clamps features to at least 1e-6 before the power pooling.py. The paper's formula has no clamp §3.1. The top-down sum has no activation, so negative values reach GeM and are set to 1e-6 there minkfpn.py.
• Software. The paper used PyTorch 1.5, MinkowskiEngine 0.4.3 and PML 0.9.88 §4.1. The README says the code was tested with PyTorch 1.9.1 and MinkowskiEngine 0.5.4 and needs PML 1.0 or later README.md.
• Layer count and stride. Figure 2's caption gives "1/32" as an example stride fig. 2. The code's blocks are conv0 at stride 1 followed by three stride-2 blocks, a factor of 8 after conv0 minkfpn.py. The paper does not say which block the example refers to.

No discrepancies found in: epochs, learning rate, LR milestones, weight decay, margin, batch limit, expansion threshold and rate, and the 25 m evaluation radius. These match the paper tab. 3 config_baseline.txt generate_test_sets.py.

Pitfalls

• The paper contradicts itself in several places. Section 4.2 says the model has 1.5 million trainable parameters, but Table 6 gives 1.1M §4.2 tab. 6. The text gives MinkLoc3D's runtime as 22 ms, the table as 21 ms §4.2 tab. 6. The text says the refined gain over LPD-Net is 0.8 to 2.9 points on all subsets, but the Oxford gain in Table 5 is 3.6 §4.2 tab. 5. The conclusion gives a top range of 99.4 for refined AR@1%, but Table 5 shows 99.7 for U.S. §5 tab. 5. The in-house AR@1% of 93.2 in Tables 7 and 8 does not match the mean of Table 4's three in-house values (91.6); the paper does not say how it is aggregated tab. 4 tab. 7 tab. 8.
• Refined baselines mix protocols. PointNetVLAD and PCAN come from released models, and LPD-Net was retrained by the authors §4.2. Read Table 5 as a comparison of published numbers, not one shared training run.
• Batch sizes. The sampler flushes a batch once it reaches the target size, and it adds pairs. An odd target size is therefore exceeded by one element samplers.py. The expansion is int(size × 1.4), capped at 256, and never shrinks samplers.py.
• Partners reused within an epoch. When an anchor has no unused positive, its partner is drawn from all positives, so some clouds appear twice in an epoch samplers.py. Clouds with no positives are skipped samplers.py.
• Collapse guard depends on batch size. The paper says large batches can collapse. The batch grows only when the active-triplet share drops below 0.7, measured as epoch means §3.2 trainer.py. A run that never triggers expansion stays at the initial batch size.
• Mining cost. The code computes the full n-by-n distance matrix for each batch loss.py. At batch 256 the README asks for at least 11 GB of GPU RAM README.md.
• Preprocessing is not in the repo. Ground removal, downsampling to 4096 points and normalization to [-1, 1] are described §4.1, but the repo only loads the resulting files and asserts the point count oxford.py. Reproduction needs the original PointNetVLAD benchmark files.
• Negatives in the 10 to 50 m band. Clouds between 10 and 50 m are neither positives nor negatives §4.1. They are ignored in the loss, so the model is not told anything about them.
• Evaluation details. Queries with no true match in the database are skipped evaluate.py. The 1% cut-off is round(database size / 100), at least 1 evaluate.py.
• Newer code. The README points to MinkLoc3Dv2 as an improved version README.md. This card covers only the original method.
• Unstated values. GeM's initial exponent, the weight initialization, the Adam betas, the descriptor normalization and the block-erasing probability are not in the paper text. Use the code values and cite them as code-derived.

Lineage

• Builds on: PointNetVLAD (arXiv:1804.03492) for the datasets and the evaluation protocol §1 §4.1 README.md. Feature Pyramid Network for the multi-scale design, GeM pooling for aggregation, and the Minkowski Engine sparse convolutions (arXiv:1904.08755, cited in the repo) §3.1 resnet.py.
• Compared against: PointNetVLAD, PCAN, DAGC, LPD-Net and DH3D on the Oxford and in-house sets, and image-based and 6-DoF methods on RobotCar Seasons §4.2.
• Superseded by: none recorded in this wiki.