OverlapTransformer: An Efficient and Rotation-Invariant Transformer Network for LiDAR-Based Place Recognition

In plain words

A car with a spinning laser scanner needs to recognise a street it has driven down before, even when it comes back from the other direction. This paper turns each laser sweep into a short list of numbers, a fingerprint of the scene. Two sweeps taken at the same place should get fingerprints that are close together.

The network is designed so that turning the car's heading only moves pixels around inside the scan. The fingerprint is then built in a way that ignores where the sweep starts, so a rotated view of the same street gives nearly the same fingerprint. A small attention block lets distant parts of the scan relate to each other, which helps tell similar-looking places apart.

The network is trained with pairs of scans that either do or do not overlap, labelled from the vehicle's known poses. Its per-scan cost is low enough for online use, as the Results section shows. The authors also release a new LiDAR dataset recorded in a city with reverse-direction and long-term revisits.

TL;DR

• Input is a single LiDAR range image, one channel of depth per pixel; output is a 256-d descriptor, retrieved by Euclidean nearest neighbour §III §III-C.
• Yaw invariance comes from the architecture: convolutions are only vertical, there is no padding, no dropout, and NetVLAD pools over the columns, which are permutation-invariant §III-A §III-C.
• Trained on KITTI sequences with overlap labels (positive if overlap > 0.3) and a triplet loss §III-D §V-A.
• Author-reported: highest KITTI loop-closure scores in all four metrics tab. 2; 1.37 ms descriptor extraction and 0.44 ms search per scan tab. 4.
• Before implementing, read the Paper vs code and Pitfalls sections: the printed triplet loss, the transformer ordering and the training sequences all differ from the official code.

Problem

Loop closure and global localisation need to find an earlier place from the current LiDAR scan, quickly and under rotation. Handcrafted descriptors have trade-offs: the histogram method gives fast retrieval, while Scan Context is discriminative but slow to match §II tab. 4. Learned descriptors such as PointNetVLAD and NDT-Transformer work on local point-cloud maps, and PointNetVLAD is not yaw-invariant §II §III-C. OverlapNet uses several input cues and needs normal estimation, and its search time is far higher §II §V-F tab. 4. The paper asks for a method that uses only depth from a single scan, is yaw-invariant, runs online, and generalises to a new LiDAR and environment without fine-tuning §I §V-B.

Method

The paper uses IEEE section numbers (I to VI); subsections are written III-A and so on. Figure 2 gives the full pipeline fig. 2.

1. Range image §III-A eq. 1. Each 3-D point is projected onto a grid of H beams by W columns. The column comes from the azimuth and the row from the elevation, with the vertical field of view split into an upward and a downward part eq. 1. Each pixel stores the range. Paper settings: 64 beams on KITTI, 32 on Haomo, width 900 §V-A.

2. Yaw is a column shift §III-A eq. 2 eq. 3. A yaw rotation θ of the scan equals a horizontal circular shift s of the range image by s = w(1 − θ/π)/2 columns eq. 2. The same shift is written on the range image as an identity, R C_s = Π(R_θ P) eq. 3. The encoder output then shifts with the input §III-A.

3. Range image encoder (RIE) §III-A. A stack of convolutions that compress only the height and never the width, so the width stays a column index. No padding and no dropout, which keeps the shift relation exact at every layer §III-A §V-A. The paper calls this a modified OverlapNet leg but does not list its layers. The official code does, and the layers are in Hyperparameters.

4. Transformer module (TM) §III-B eq. 4 eq. 5. The encoder's columns are the tokens. One transformer block (multi-head self-attention, feed-forward network, layer normalisation) mixes them §III-B. Self-attention is softmax(QKᵀ/√d_k)V eq. 4. The paper writes the block's output as a function of the concatenation of encoder and attention features eq. 5. Attention with no position code is shift-equivariant, which is the argument in §III-B. The official code does not follow eq. 5 exactly; see Paper vs code.

5. Global descriptor generator (GDG) §III-C. NetVLAD with K = 64 clusters over the W column vectors §V-A. NetVLAD is permutation-invariant, so a circular column shift leaves its output unchanged §III-C. The 1-D output is the 256-d descriptor §III-C eq. 14.

6. Retrieval §III-C. Descriptors of database scans are stored. A query descriptor's nearest neighbours by Euclidean distance are the candidate places §III-C.

7. Training labels §III-D eq. 15. For a query scan R_q and reference scan R_r, the reference is reprojected into the query frame with ground-truth poses to get R'_r. The overlap is the fraction of pixels whose ranges agree within δ eq. 15. A pair is positive if overlap > 0.3 and negative otherwise §III-D §V-B.

8. Loss §III-D eq. 16. Per query: k_p positives and k_n negatives. The positive term uses the farthest positive, so the query is pulled toward the hardest positive. Every negative is pushed away §III-D. The paper's margin is α = 0.5 with k_p = k_n = 6 §V-A. Eq. 16 is printed without a per-negative hinge; the code uses one. See Paper vs code.

Pseudocode for the forward pass follows the official KITTI model. The ordering after the transformer is the code's, not eq. 5's.

# OverlapTransformer forward pass, KITTI model  [§III] [fig. 2]
# Input:  range_image (1, 64, 900)   # (channels, H, W), depth only
# Output: descriptor (256,), unit length

def describe(range_image):
    # Range image encoder: convolutions over height only, no padding
    # [§III-A]. Layer list in Hyperparameters
    # [code: modules/overlap_transformer.py#L42-L62]
    features = range_image_encoder(range_image)  # (128, 1, 900)
    # 1x1 convolution to 256 channels, then ReLU
    # [code: modules/overlap_transformer.py#L113-L114]
    local = relu(conv1x1(features, out_channels=256))  # (256, 1, 900)
    # One transformer block over the 900 columns, 4 heads, FFN 1024
    # [§III-B]. No positional encoding is added
    # [code: modules/overlap_transformer.py#L70-L71]
    attended = transformer_block(local, heads=4, ffn_dim=1024)
    # Concatenate the two maps, then mix to 1024 channels
    # [code: modules/overlap_transformer.py#L123-L124]
    mixed = relu(conv1x1(concat(local, attended), out_channels=1024))
    # L2-normalise each column over channels
    # [code: modules/overlap_transformer.py#L125]
    mixed = l2_normalize(mixed, axis=channels)
    # NetVLAD over the columns, 64 clusters, output 256
    # [§III-C] [code: modules/overlap_transformer.py#L86-L88]
    vlad = netvlad(mixed, clusters=64, out_dim=256)
    # Final L2 normalisation of the descriptor
    # [code: modules/overlap_transformer.py#L127]
    return l2_normalize(vlad)

Triplet loss for one query, as the training code computes it loss.py. The printed eq. 16 is the same idea without the hinge.

# Triplet loss for one query  [§III-D] [eq. 16]
# Input:  query (256,), positives (k_p, 256), negatives (k_n, 256)
# Output: scalar loss; margin alpha = 0.5  [§V-A]

def triplet_loss(query, positives, negatives, margin=0.5):
    # Squared distance from the query to each positive  [§III-D]
    pos_dists = [sum_sq(query - p) for p in positives]
    # The farthest positive, the hardest one  [§III-D]
    hardest = max(pos_dists)
    loss = 0.0
    for neg in negatives:
        # One hinge per negative: zero once it is far enough away
        # [code: modules/loss.py#L39-L41]
        loss += max(0.0, margin + hardest - sum_sq(query - neg))
    # Summed over negatives, then averaged over one query
    # [code: modules/loss.py#L43-L52]
    return loss

Overlap label, following the paper's definition and the code's denominator com_overlap_yaw.py.

# Overlap between a query scan and a reference scan  [§III-D] [eq. 15]
# Input:  query_range (H, W), reference_range (H, W) in the query frame
# Output: overlap in [0, 1]; delta = 1.0 m on KITTI, 1.2 on Haomo  [§V-A]

def overlap(query_range, reference_range, delta):
    # Pixels where the reference has a return
    # [code: tools/utils/com_overlap_yaw.py#L48]
    reference_valid = reference_range > 0
    # Pixels whose two ranges agree within delta  [eq. 15]
    agree = reference_valid & (abs(reference_range - query_range) < delta)
    # Denominator: valid pixels of the query only (paper: min of both)
    # [eq. 15] [code: tools/utils/com_overlap_yaw.py#L36]
    return count(agree) / count(query_range > 0)

Hyperparameters

Paper values come first. The table then gives what the official code uses, and marks what neither states.

KITTI range image encoder, as the official code builds it. Every convolution has no bias. Each is followed by a ReLU. The code creates BatchNorm modules, but the forward pass never calls them overlap_transformer.py overlap_transformer.py.

The Haomo model is a separate file with a different encoder. Its forward pass uses eight convolutions, not eleven. The rest matches the KITTI head overlap_transformer_haomo.py overlap_transformer_haomo.py.

Results

Author-reported. Hardware for timing: Intel i7-11700K and RTX 3070 §V-F. Training on KITTI sequences only, with no fine-tuning on Ford Campus §V-B.

Loop-closure detection tab. 2. KITTI is sequence 00, with a 100-scan exclusion. Ford Campus is sequence 00, without training on it.

The paper calls its Ford Campus recall@1% "on par" with Scan Context §V-B tab. 2. The baselines' setup is not described beyond the reference list §V-B.

Haomo place recognition §V-C fig. 4 fig. 5 fig. 6. Results are given as recall curves in the figures only. No numbers appear in the text. Query 1-3 against database 1-1 (same direction), against 1-2 (reverse), and query 2-2 against database 2-1 (long-term).

Yaw-invariance test §V-D fig. 7. KITTI queries rotated in steps of 30° against the original database, with recall@1 as the metric. Results are given only as a plot. The text says the method is not affected by pure yaw rotation and that OverlapNet, NDT-Transformer-P and PointNetVLAD degrade quickly §V-D.

Ablation on the transformer module §V-E tab. 3. Haomo database 1-2, query 1-3, recall@N.

The paper uses one transformer block for all other experiments because of speed §V-E.

Runtime §V-F tab. 4. Descriptor extraction includes LiDAR preprocessing. Search is top-1 against 2000 reference scans, averaged over ten runs. FAISS is used for methods that need no ad-hoc similarity function §V-F.

The paper states 730 Hz, which matches 1/1.37 ms §V-F tab. 4.

Haomo dataset §IV tab. 1. Collected in Beijing with a 32-beam LiDAR. Sequences 1-1 and 1-2 cover the same route in opposite directions. Sequence 1-3 is the query for both. Sequences 2-1 and 2-2 are a longer route on different dates. Figure 1 shows the reverse loop fig. 1.

Paper vs code

Official repository: haomo-ai/OverlapTransformer at 4dd0646. Copyleft (GPL-3.0), so this card cites lines and paraphrases; it quotes no code. The owner is the authors' company, HAOMO.AI [author affiliations, title page]. The repo has no pretrained weights and no Ford Campus code. Its README links the Haomo scans and poses README.md. The dataset's license is not stated there, and this card does not redistribute it.

• Training sequences. Paper: KITTI 03–10 for training §V-B. Code: 03–09 config.yml.
• Triplet loss. Printed eq. 16 multiplies the positive term by k_p and has no hinge eq. 16. The code clamps each negative's term at zero and sums over negatives, with no k_p factor. Training calls it with lazy=False, which sums over negatives loss.py training_overlap_transformer_kitti.py. The code's lazy=True branch averages the per-negative terms instead of summing them loss.py.
• Transformer ordering. Eq. 5 applies the feed-forward and layer norm to the concatenation of encoder and attention features eq. 5. The code runs the transformer block on the encoder output alone, then concatenates the block's output with that encoder output, then applies a 1×1 convolution to 1024 channels and a ReLU overlap_transformer.py. The paper describes no 1024-channel step; it appears only as the NetVLAD input size §V-A.
• Normalisation. The code L2-normalises the feature map before NetVLAD and the descriptor after it overlap_transformer.py. The published text does not mention this.
• Context gating. NetVLAD uses gating in the code overlap_transformer.py. The paper does not mention it.
• Relative location. The introduction says the transformer embeds the relative location of features §III. The code adds no positional encoding overlap_transformer.py. Attention without positions is the mechanism that keeps the shift argument in §III-B exact. This is inference from the code, not a claim the authors make.
• Overlap denominator. Eq. 15 divides by the smaller of the two valid pixel counts eq. 15. The code divides by the query's valid count only com_overlap_yaw.py com_overlap_yaw.py.
• Loop-closure ground truth. The code also requires candidates within 30 m and an index gap above 100 gen_gt_data.py. The paper gives only overlap > 0.3 and the 100-scan exclusion §V-B.
• Haomo query sampling. Paper: one scan per ten query scans §V-C. Code: every fifth test_haomo_topn_prepare.py. Table 1 lists 1350 queries for 13500 scans, which matches the paper's ten tab. 1.
• Haomo training data. The paper says the database is downsampled and overlaps are computed for all scans §V-C. The repo has no script that makes the Haomo training pairs; it loads a .npy file that is not shipped config_haomo.yml.
• Haomo ground truth. The config names a Haomo ground-truth file with overlap 0.3 and tau 1.2 config_haomo.yml. The file is not in the repo and no script in it generates it.
• Ford Campus. The paper reports Ford Campus results §V-B. The test scripts cover only KITTI 00 and Haomo.
• Yaw sweep. The 30° sweep of §V-D has no script in the repo. The only rotated input is a 180° column roll, whose descriptor is computed but not used for retrieval test_kitti00_prepare.py.
• Runtime. The README says under 4 ms per frame in Python and under 2 ms in C++ README.md. The paper reports 1.37 ms on an RTX 3070 tab. 4.
• Title. The arXiv title reads "Rotation-Invariant"; the paper's own title and the README read "Yaw-Angle-Invariant".
• Haomo model. The paper describes one encoder for both datasets §III-A §V-A. The repo has two: the Haomo one is shallower and has a different layer list (see Hyperparameters) overlap_transformer_haomo.py.

Pitfalls

• Printed triplet loss is unbounded. Taken literally, eq. 16 has no hinge, so pushing negatives apart drives the loss without bound. The code's per-negative hinge with margin 0.5 is what keeps training stable loss.py. Implement the hinge.
• Margin depends on normalisation. The descriptor is L2-normalised overlap_transformer.py. Squared distances between unit vectors are at most 4, so margin 0.5 is a fixed fraction of that range. If you drop the final normalisation, the margin changes meaning. This is inference from the code.
• Input scale is not stated. The paper does not give the depth unit or scale. The KITTI reader loads 8-bit grey PNGs and feeds the values as floats without rescaling read_samples.py. The depth generator can normalise but defaults to off gen_depth_data.py gen_depth_data.py. The Haomo reader loads .npy floats read_samples_haomo.py. Check the scale of your own data.
• Pixel collisions. Several points can land in one pixel. The code writes points in decreasing depth order, so in practice the nearest point is the one kept utils.py. The paper does not say which point wins.
• Yaw resolution. Eq. 2 means one column is 2π/w radians. With w = 900 that is 0.4°, and a 180° shift is exactly 450 columns eq. 2 test_kitti00_prepare.py. Rotations that are not multiples of 0.4° cannot sit exactly on the grid, which is the discretisation error the paper mentions §III-A.
• Cost of attention. Each transformer block attends over 900 tokens with 4 heads, so each head builds a 900 by 900 matrix overlap_transformer.py. The ablation shows the cost grows with more blocks: 0.86 ms without the module, 1.37 ms with one tab. 3.
• One query per step. The training loop does one optimiser step per query, with up to 12 references, and never batches queries training_overlap_transformer_kitti.py. Expect slow, noisy training. The paper does not state the batch size.
• Learning rate is tiny. The optimiser uses 5e-6 on KITTI and 1e-6 on Haomo, with a 0.9 decay every five epochs training_overlap_transformer_kitti.py training_overlap_transformer_haomo.py. These values are not in the paper.
• Epoch count. The loop starts at 1 and stops before 100, so training from scratch runs 99 passes training_overlap_transformer_kitti.py.
• Which checkpoint. Every epoch is saved, and validation prints a top-1 rate after each one training_overlap_transformer_kitti.py. The paper does not say which checkpoint it reports.
• Haomo starts from a checkpoint. The Haomo trainer sets resume = True and loads a file that is not in the repo training_overlap_transformer_haomo.py. Its starting weights are not documented, so a clean run needs a decision about initialisation.
• Shuffled picks. Pairs are reshuffled each epoch read_all_sets.py. The six positives and six negatives taken per query come from that shuffled list, so the choice changes between epochs. The rest are unused in that step.
• Empty positives. The KITTI loop skips queries with no negatives but not those with no positives training_overlap_transformer_kitti.py. The Haomo loop skips both training_overlap_transformer_haomo.py. Copy the stricter check.
• Stale model for Haomo. The Haomo model file is a different network from the KITTI one (see Hyperparameters). Do not reuse the KITTI weights or layer list for Haomo.
• Ignored arguments. The Haomo trainer accepts batch_size but never uses it training_overlap_transformer_haomo.py training_overlap_transformer_haomo.py.
• Scope of the claims. Loop-closure results are on KITTI 00 and one Ford sequence, with author-run baselines. Haomo results are figures only. Treat the generalisation claim as limited to the sequences reported.

Lineage

• Builds on: OverlapNet (Chen et al., 2021), whose overlap supervision and loop-closure protocol this paper follows §III-D §V-B. NetVLAD (Arandjelovic et al., 2016), whose pooling the GDG uses §III-C. The Transformer (Vaswani et al., 2017) §III-B. NDT-Transformer (Zhou et al., 2021), which the paper is inspired by in using attention on LiDAR §III-B. PointNetVLAD (Uy et al., 2018), for permutation-invariant LiDAR pooling §III-C.
• Compared against: Histogram (Röhling et al., 2015), Scan Context (Kim et al., 2018), LiDAR Iris (Wang et al., 2020), PointNetVLAD (Uy et al., 2018), OverlapNet (Chen et al., 2021), NDT-Transformer-P (Zhou et al., 2021), MinkLoc3D (Komorowski et al., 2021) tab. 2 tab. 4.
• Follow-ups named by the code README, not compiled here: SeqOT, CVTNet, EINet README.md.