OverlapNet: Loop Closing for LiDAR-based SLAM

In plain words

A robot that maps as it moves has to notice when it returns to a place it has seen before. Then it can pull its map back into line, because the small errors built up since that first visit can be measured. This paper does that for laser scanners of the kind mounted on cars. A scan is a cloud of distance readings taken all around the vehicle, and the method turns each scan into a flat, picture-like grid.

A neural network then compares two of these grids. It answers two questions: how much of the view do the two scans share, and how far is one scan rotated relative to the other. It needs no position estimates for either scan. The shared-view number tells the mapping system whether two scans really show the same place, and the rotation gives a starting guess for lining them up precisely. The authors plug the network into an existing mapping system. It was trained on one driving dataset and still gives useful matches on a second, different dataset.

TL;DR

• A siamese CNN takes spherical range-image feature volumes of two LiDAR scans and outputs an overlap in [0, 1] and a relative yaw in 1-degree bins §3.3 §3.4.
• The network needs no poses. The SLAM integration uses pose covariance from odometry to limit which earlier scans are searched §3.5 §3.6 §4.1.
• Author-reported on KITTI sequence 00: AUC/F1 0.87/0.88 with all cues, and 0.96/0.96 for the covariance-prior variant tab. 2 tab. 3.
• Only the MIT-licensed official code is used here. The default config and pretrained model differ from the paper's best model, and several training values differ. Read "Paper vs code" before implementing.

Problem

Loop closures are needed to remove drift in graph-based SLAM §1. The SuMa system proposes only the nearest earlier frame as a candidate. That fails for loops with few large revisits and when odometry drift moves the estimated pose far from the true place §3.5. Scoring overlap directly with the ground-truth definition needs relative poses, which drift makes unreliable. Brute force over orientations costs about 1.2 s per pair, which is too slow for online use §3.1 §3.2. The paper also assumes that ICP gives a correct pose only when overlap is around 30% or more §4.1 §4.6.

Method

1. Input per scan. Project the 3D points onto a grid of 64 × 900 pixels. Each pixel keeps the nearest point. Use the vertical field of view f = f_up + f_down eq. 1 §3.2. Four cues form the input: range, normals derived from neighbouring vertices, remission (intensity) from the sensor, and semantic class probabilities from RangeNet++ with 20 classes reduced to 3 by PCA §3.3. The input tensor is 64 × 900 × D, where D depends on the cues chosen. D = 8 when all four are used (derived from the channel counts) §3.3 §4.1.
2. Shared leg. A fully convolutional stack of 11 conv layers, with weights shared by both inputs, maps each input to a feature volume of shape 1 × 360 × 128 §3.3.1 tab. 1. Because the input is cyclic in yaw, a yaw change appears as a horizontal shift of the feature volume §3.3.1.
3. Overlap head (delta head). Stack the two feature volumes so every pixel of one is paired with every pixel of the other. Take the absolute difference of each pair eq. 4 eq. 5 §3.3.2 fig. 4. Three conv layers and one dense output with a sigmoid then give the overlap tab. 1 generateNet.py.
4. Yaw head (correlation head). Pad the first feature volume circularly along its width. Slide the second volume over it as a kernel, which gives a 1-D output of 360 values. Its argmax is the relative yaw in degrees, at 1-degree resolution §3.3.3 fig. 3.
5. Training loss. Overlap term: sigmoid(s(|ŷ − y| + a) − b), with a = 0.25, b = 12, s = 24 eq. 7 §4. Yaw term: binary cross-entropy over the 360 yaw bins eq. 8 §3.4, trained only on pairs with overlap above 30% §3.4. Total: L_O + α·L_Y with α = 5 eq. 6 §4.
6. Ground truth from poses. For training pairs, compute overlap with eq. 3, using a tolerance ε = 1 m eq. 3 §4. Compute yaw from the relative rotation, quantised to 1-degree bins §3.4. Only points within 75 m of the sensor are used §4.
7. Loop-closure query inside SLAM. Propagate the pose covariance along odometry eq. 14 eq. 15 §3.6. Gate candidates by Mahalanobis distance eq. 16 inside the 3σ region §4.1. Skip the 100 most recent scans §4.1. Keep the 10 candidates with the highest predicted overlap. Pick the one nearest in Mahalanobis distance as the loop candidate §4.1. Use the predicted yaw as the initial guess for ICP §4.6.

# Overlap and yaw for one pair of scans  [§3.3] [tab. 1]
# Input:  range-image stacks x1, x2 of shape (64, 900, D)
# Output: overlap in [0, 1]; yaw index in 0..359 (1 degree per bin)

def overlapnet(x1, x2):
    # 1. Shared leg, same weights for both scans  [§3.3.1]
    feat1 = leg(x1)                              # (1, 360, 128)
    feat2 = leg(x2)                              # (1, 360, 128)

    # 2. Overlap head: all pairwise absolute differences  [§3.3.2] [eq. 4-5]
    diffs = abs(tile_rows(feat1) - tile_cols(feat2))   # (360, 360, 128)
    overlap = overlap_head(diffs)                # sigmoid output

    # 3. Yaw head: circular cross-correlation over 360 shifts  [§3.3.3]
    padded = pad_circular_width(feat1)           # width 360 -> 719 in code
    scores = correlate(padded, feat2)            # (360,)
    yaw_index = argmax(scores)                   # 1-degree bins

    return overlap, yaw_index

# Training loss for one pair  [eq. 6-8] [§3.4] [§4]
# Input:  predicted overlap; yaw logits (360,); true overlap; true bin
# Output: scalar loss

def pair_loss(overlap_pred, yaw_logits, overlap_true, yaw_true):
    # 1. Overlap term: scaled sigmoid of the absolute error  [eq. 7] [§4]
    err = abs(overlap_pred - overlap_true)
    loss = sigmoid(24.0 * (err + 0.25) - 12.0)
    # 2. Yaw term only for pairs with overlap above 30%  [§3.4]
    if overlap_true > 0.3:
        # One positive bin, all others negative  [§3.4] [eq. 8]
        target = zeros(360)
        target[yaw_true] = 1.0
        loss = loss + 5.0 * bce_sum(yaw_logits, target)  # alpha = 5
    return loss

# Ground-truth overlap for a pair of scans  [eq. 3] [§3.2] [§4]
# Input:  scans 1 and 2 as 3D points; poses T_w1, T_w2 (LiDAR frame)
# Output: overlap in [0, 1]

def gt_overlap(points1, points2, T_w1, T_w2):
    # 1. Keep only points within 75 m of the sensor  [§4]
    near1 = points1[norm(points1) < 75.0]
    near2 = points2[norm(points2) < 75.0]
    # 2. Reproject scan 2 into the frame of scan 1  [§3.2]
    T_12 = inverse(T_w1) @ T_w2
    near2_in_1 = transform(near2, T_12)
    # 3. Count pixels whose 3D points agree within eps = 1 m  [eq. 3] [§4]
    matched = count_within_tolerance(near2_in_1, near1, eps=1.0)
    # 4. Divide by the smaller number of valid pixels  [eq. 3]
    return matched / min(valid(near2_in_1), valid(near1))

Hyperparameters

Results

Author-reported. Setup: models trained on KITTI odometry sequences 03–10 with sequence 02 for validation; KITTI sequence 00 for the loop-closure test; Ford Campus sequence 00 as an unseen dataset §4. Baselines: Histogram [Röhling et al., §2], M2DP §2, and SuMa's heuristic §4.1. Hardware: Intel i7-8700 at 3.2 GHz with GTX 1080 Ti §4.7.

Loop-closure comparison tab. 2

Variants tab. 3

Yaw error without ICP, KITTI 00 tab. 4 (rows marked * are the OREOS authors' numbers)

Ablation on input cues tab. 5

Runtime §3.2 §4.7

The exhaustive overlap-with-orientations baseline takes about 1.2 s per pair, and the full network about 17 ms per pair with depth and normals only §3.2. The paper reports that it stays online because only candidates inside the pose-uncertainty search space are evaluated §4.7.

Qualitative and other figures. Precision–recall curves for KITTI and Ford fig. 5. A SLAM trajectory on KITTI 02 with and without OverlapNet loops [fig. 6]. Recall against the number of candidates on KITTI 00, where OverlapNet is best for small candidate counts and OREOS and LocNet++ slightly higher for large ones fig. 7 §4.3. Yaw error against overlap, with error near 1 degree above 90% overlap fig. 8 §4.4. ICP error against overlap, with and without the predicted yaw as initial guess fig. 9 §4.6.

Paper vs code

Official repository: github.com/PRBonn/OverlapNet at 4188c3a, MIT licence LICENSE.txt. The paper links it §1.

• Loss weights are reversed. Paper: L = L_O + α·L_Y with α = 5, so the yaw term has weight 5 eq. 6 §4. Code: overlap weight 5.0, orientation weight 1.0 training.py.
• Learning-rate schedule. Paper: 1e-3 with decay 0.99 per epoch §4. Code: epoch 0 uses 1e-4 (0.1 × initial), then 1e-3 × 0.99^(epoch−1) training.py.
• Optimiser. Not stated in the paper. Code uses Adagrad training.py. The config's momentum is logged only, not passed to the optimiser training.py training.py.
• Yaw-training threshold. Paper: train yaw only where overlap > 30% §3.4. Code: threshold 0.7 by default training.py training.py. The threshold does not remove low-overlap pairs from the yaw loss. It sets their 360-bin target to all zeros training.py. The repository's evaluation script also tests yaw only where overlap > 0.7 testing.py, so the 0.7 value appears in two places in the code. The paper does not mention 0.7.
• Yaw loss weighting. Paper: plain binary cross-entropy eq. 8. Code: weighted_cross_entropy_with_logits with a positive weight equal to the number of bins (360) training.py.
• Overlap ground truth. Paper: fraction of pixels whose 3D points lie within ε = 1 m, divided by the smaller valid count eq. 3. Code: compares range values |Δr| < 1 m over all reference scans and divides by the valid pixels of the current scan only com_overlap_yaw.py.
• Distance cut-off. Paper: points within 75 m for overlap ground truth §4. Code: the projection defaults to max_range = 50 m, and the ground-truth, depth and normal scripts all call it with that default utils.py utils.py com_overlap_yaw.py.
• Validation split. Paper: sequence 02 for validation §4. Code: split_train_val holds out a random 10% of pairs from each sequence split_train_val.py. No sequence-level split is provided.
• Headline model. Paper: the best model on KITTI uses all four cues §4.1. Code: the default network.yml disables intensity and semantics network.yml. The pretrained weights in the README are for the geometry-only model [README: Demo 2].
• Covariance search. Paper: among the 10 best-overlap candidates, the one nearest in Mahalanobis distance §4.1. Code (demo 3): takes the highest-overlap candidate inside a 2-D 3σ ellipse, above a 0.3 threshold. It also skips frames closer than 100 scans and less than 50 m away in travelled distance demo3_lcd.py demo3_lcd.py demo3_lcd.py. The top-10 and Mahalanobis step is not in the repository.
• SLAM integration. SuMa is not in the repository. The README points to SuMa++ for online preprocessing [README: How to use].
• Matches. The 11 leg convolutions and the delta-head shapes match Table 1 generateNet.py. Input 64 × 900, 1° yaw bins, and the overlap-loss constants s = 24, a = 0.25, b = 12 also match training.py network.yml.

Pitfalls

• Overlap is not symmetric in general. The paper says so in its concept section §3.1. Define which scan is the query, and keep that order in training and inference.
• Eq. 3 needs poses. Ground truth uses dataset poses §3.4, so label quality depends on pose accuracy. The network itself never sees poses.
• Yaw label rounding. The code's label index is an integer truncation of −yaw·180/π + 180 degrees. Decoding is 180 − argmax infer.py. The round trip rounds the yaw up to the next whole degree, not to the nearest. Fix the sign convention and rounding in your own pipeline and test it on a known rotation.
• Yaw is reliable only at high overlap. Depth-only yaw has std 25.32° tab. 5. The paper's ~1° figure applies above 90% overlap §4.4.
• Yaw test subset. The code evaluates yaw only where overlap > 0.7 testing.py. Compare numbers only on the same subset.
• Pose priors change the headline. The 0.96 AUC uses pose covariance, while the 0.87 AUC does not tab. 2 tab. 3. Compare like with like.
• Overlap threshold. The 30% threshold defines a true loop for evaluation §4.1. The paper also says ICP works around 30% overlap §4.6. Both are choices, not tuned constants.
• Worst case matching. Head matching can take 630 ms when all candidates are scored §4.7. Online use relies on the pose-uncertainty gate §4.7 §3.6.
• Delta head output. The paper's table lists only "Dense". The code uses a sigmoid output generateNet.py. The first delta-head convolution is linear, and the later ones use ReLU generateNet.py.
• Correlation head name. The class is called NormalizedCorrelation2D, but the head calls it with normalize='none' generateNet.py. Do not assume L2 normalisation.
• Loss reduction. The paper writes the yaw term as a sum over bins eq. 8. The code returns a per-bin tensor and relies on the framework to reduce it, which for Keras may be a mean. Check the reduction in your framework before comparing loss magnitudes training.py.
• Circular padding width. The paper says the padding copies the row to double its width §3.3.3. The code pads by 180 columns on one side and 179 on the other, giving width 719 rather than 720 NormalizedCorrelation2D.py RangePadding2D.py RangePadding2D.py.
• Depth units and range clipping. The code keeps only returns closer than 50 m, and feeds raw depth without normalisation by default gen_depth_data.py. Points beyond the cut-off become "no data".
• Not stated in the paper: batch size, optimiser, weight initialisation, PCA fit data, the activation of the leg, the number of training pairs, and the 3-channel semantic PCA setup.

Lineage

• Builds on: SuMa, the surfel-based SLAM system into which OverlapNet is integrated §3.5. RangeNet++ provides the semantic probabilities §3.3. A siamese network design is cited for the two legs §3.3 (not compiled here). ICP is the registration step that follows the yaw estimate §1 §4.6.
• Compared against: Histogram [Röhling et al.], M2DP [He et al.], SuMa's heuristic, FPFH+RANSAC, OREOS, and MLP-only and delta-only variants of the paper's own network §4.1 §4.4 tab. 2–4.
• Superseded by: not known from content in this wiki.