---
kind: concept
slug: local-feature-matching
title: Local feature matching
summary: >-
  Finding points in two inputs that show the same scene point. Detect-and-describe pipelines, or dense matching without a detector (LoFTR, RoMa). In 3D, RoPS describes mesh patches, PREDATOR and PRNet pick points when scans overlap little, and Deep Global Registration and PointDSC filter matches. FeudalNav uses SuperGlue matches to steer navigation.
tags: [local-feature-matching, detector-free, transformer, visual-localization, relative-pose, 3d-vision, local-descriptor, point-cloud-registration, correspondence-filtering, partial-overlap, visual-navigation, dense-matching, frozen-foundation-features, spatial-consistency]
papers: ["2104.00680", "2305.15404", "1304.3192", "2011.13005", "2004.11540", "2103.05465", "1910.12240", "2402.12498"]
related: [self-attention, linear-attention, optimal-transport, graph-neural-network, metric-learning, convolutional-neural-network, iterative-closest-point, unsupervised-keypoint-detection, self-supervised-learning, hierarchical-reinforcement-learning, imitation-learning, dense-correspondence, gaussian-process, outlier-rejection, spectral-methods, attention-mechanism]
status: draft
updated_at: 2026-11-01
---

## Overview

Local feature matching pairs points in two inputs that show the same point in
the scene. The usual pipeline detects keypoints in each input, describes each
one, and matches the descriptors [arxiv:2104.00680 §1]. Detector-free methods
drop the detection step and predict matches over dense feature maps. Seven of
the eight variants below produce correspondences, or a pose built from them, that
a pose or localisation solver can consume [arxiv:2104.00680 §1]
[arxiv:2305.15404 §3.3] [arxiv:1304.3192 §6] [arxiv:2011.13005 §4.1]
[arxiv:2004.11540 §3.3] [arxiv:2103.05465 §3.5] [arxiv:1910.12240 §3.2]. The
eighth uses learned matches inside a navigation policy, to choose waypoints and
to decide when to stop [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §4.3].

The LoFTR card reports that detectors find too few repeatable points under
poor texture, repetitive patterns, viewpoint or illumination change and motion
blur. Indoors, low-texture regions can cover most of the view
[arxiv:2104.00680 §1]. The PREDATOR card argues that, for low-overlap scans,
the bottleneck is where interest points are chosen, not how each point is
described [arxiv:2011.13005 §1].

Each variant rests on one card, so each comparison rests on one paper's
author-reported results. This page covers eight methods and not the full
literature.

## Variants

### Detector-free, coarse-to-fine with Transformers (LoFTR)

The matcher runs in two stages. The coarse stage works on the 1/8 grid. A CNN
with a feature pyramid gives coarse and fine maps, a 2-D sine encoding is added
once, and the coarse features go through alternating self and cross attention
[arxiv:2104.00680 §3.1] [arxiv:2104.00680 §3.2]. The attention uses a linear
kernel, φ(x) = elu(x) + 1, so the cost is linear in the number of positions
[arxiv:2104.00680 §3.2]. See [linear attention](/concepts/linear-attention).

The coarse matches come from one of two confidence heads. The dual-softmax head
multiplies a softmax over each axis of the score matrix. The optimal-transport
head runs Sinkhorn iterations with a learned score for unmatched positions
[arxiv:2104.00680 §3.3]. See [optimal transport](/concepts/optimal-transport).
A match is kept when it is a mutual nearest neighbour and its confidence is at
least 0.2 [arxiv:2104.00680 §3.3] [arxiv:2104.00680 §3.6].

The fine stage crops a 5×5 window around each coarse match on the 1/2 map, runs
one fine self and cross round, and takes the expectation of a heatmap as the
sub-pixel position [arxiv:2104.00680 §3.4] [arxiv:2104.00680 §3.6]. The
variance of the heatmap is reported as the match's uncertainty
[arxiv:2104.00680 §3.5]. Card: [LoFTR: Detector-Free Local Feature Matching with Transformers](/papers/2104.00680).

Author-reported results, all from the card's tables:

- Homography on HPatches, 1K matches: LoFTR-DS reaches 65.9 AUC@3px against
  53.9 for SuperPoint with SuperGlue [arxiv:2104.00680 tab. 1].
- Indoor relative pose on ScanNet: LoFTR-DS reaches 40.8 AUC@10°, against
  33.81 for SuperPoint with SuperGlue [arxiv:2104.00680 tab. 2].
- Outdoor relative pose on MegaDepth: LoFTR-DS reaches 69.19 AUC@10°, against
  61.16 for SuperPoint with SuperGlue [arxiv:2104.00680 tab. 3].
- Visual localisation on Aachen Day-Night v1.1, local-feature track, night:
  LoFTR-DS is about level with SuperPoint with SuperGlue [arxiv:2104.00680 tab. 4].

### Frozen foundation features with a Transformer matcher and robust refinement (RoMa)

RoMa is a dense matcher with a coarse-to-fine design. It keeps the structure of
DKM, which has no card here, and changes the encoders, the coarse decoder and
both losses [arxiv:2305.15404 §3.1] [arxiv:2305.15404 §4.2]. The coarse
features are the patch tokens of a frozen DINOv2 ViT-L/14, with no class token.
The fine features come from a VGG19 pyramid at strides 1, 2, 4 and 8
[arxiv:2305.15404 §3.2] [arxiv:2305.15404 app. B].

A Gaussian-process encoder and a 5-block Transformer decoder predict, for each
coarse pixel, a distribution over 64×64 anchors and one matchability logit
[arxiv:2305.15404 §3.3] [arxiv:2305.15404 eq. 8]. The coarse warp is the
arg-max anchor refined by a soft-argmax over its neighbours. Five convolutional
refiners then move the warp from coarse to fine, with gradients detached between
them [arxiv:2305.15404 §3.1] [arxiv:2305.15404 app. B]. See
[Gaussian process](/concepts/gaussian-process) and [dense correspondence](/concepts/dense-correspondence).

The coarse stage is trained as classification over the anchor nearest the
ground truth, plus a matchability term. Each refiner is trained with a
generalised Charbonnier loss, because the authors argue that the refinement
distribution is locally unimodal while the coarse one is often multimodal
[arxiv:2305.15404 §1] [arxiv:2305.15404 §3.4]. The paper adds the two parts
with no extra weighting, and it does not state the weight on the matchability
term [arxiv:2305.15404 §3.4]. Training needs dense supervised correspondences
from MVS depth or RGB-D data [arxiv:2305.15404 §4.2]. Card: [RoMa: Robust Dense Feature Matching](/papers/2305.15404).

Author-reported results, all from the card's tables:

- WxBS v1.1, mAA at 10 px: 80.1, against 58.9 for DKM and 55.4 for LoFTR
  [arxiv:2305.15404 tab. 4].
- IMC2022 hidden test set, mAA@10: 88.0, against 83.8 for ASpanFormer, the best
  prior entry in the table [arxiv:2305.15404 tab. 3].
- MegaDepth-1500, AUC@10°: 76.7, against 74.9 for DKM and 69.2 for LoFTR
  [arxiv:2305.15404 tab. 5].
- ScanNet-1500, AUC@10°, with the model trained on ScanNet: 53.4, against 50.7
  for DKM and 40.8 for LoFTR [arxiv:2305.15404 tab. 6].
- Ablation on the authors' validation pairs: 100-PCK at 1 px falls from 17.0
  for the retrained DKM to 13.1 for RoMa [arxiv:2305.15404 tab. 2]. These
  pairs come from MegaDepth scenes that may overlap the test set, so the
  ablation may not be a clean held-out result [arxiv:2305.15404 §4.1]
  [arxiv:2305.15404 §4.3].

Cost: at 560×560, batch 8, on an RTX6000, RoMa takes 198.8 ms per pair against
186.3 ms for DKM, about 7% more [arxiv:2305.15404 §4.5]. The paper gives no
memory figure, and no RoMa timing against LoFTR, so compare on your own
hardware.

### Detector-based pipelines (the baselines)

The paper compares against detect-then-match pipelines such as SuperPoint with
SuperGlue, D2-Net, R2D2, DISK and ORB with GMS [arxiv:2104.00680 §4.1]
[arxiv:2104.00680 §4.2]. No card exists for these methods, so this page uses
only the numbers that the LoFTR paper reports for them.

### Hand-designed 3D surface descriptors (RoPS)

RoPS recognises 3D objects in scenes with clutter and occlusion. It works on
mesh triangles [arxiv:1304.3192 §3.1]. For each feature point it builds a local
reference frame (LRF) from the scatter matrix of the triangles inside a sphere
of radius r. Each axis is sign-fixed by a weighted first-moment test, and the
second axis follows from a right-handed cross product
[arxiv:1304.3192 §3.1] [arxiv:1304.3192 §3.2].

The descriptor rotates the neighbouring points about each LRF axis, projects
them onto three planes, and counts points in a 5×5 grid per projection. From
each normalised grid it keeps four central moments and the entropy. With three
rotations per axis, that gives 135 numbers [arxiv:1304.3192 §4.1]
[arxiv:1304.3192 tab. 2]. Recognition proposes each pose from a single
correspondence, then clusters the proposals and refines them with ICP
[arxiv:1304.3192 §6]. The single-correspondence design depends on LRF
repeatability, which the paper measures only on its tuning set
[arxiv:1304.3192 §3.3]. Card: [Rotational Projection Statistics for 3D Local Surface Description and Object Recognition](/papers/1304.3192).

Author-reported results, all from the card's text and tables:

- Bologna, 45 scenes, noise 0.1 to 0.5 mesh resolution: 100% at every noise
  level [arxiv:1304.3192 §7.1].
- UWA, 50 scenes, Rhino excluded: 98.9% overall. Recognition is 100% up to 80%
  occlusion and 93.1% at 85% [arxiv:1304.3192 §7.2].
- Queen's: 95.4% on the 80-scene set and on the 55-scene subset
  [arxiv:1304.3192 tab. 6].
- Ca' Foscari, 18 models: 99% precision and 96% recall
  [arxiv:1304.3192 §7.4] [arxiv:1304.3192 tab. 7].

### Learned overlap-aware point selection for 3D registration (PREDATOR)

PREDATOR registers two point clouds that share only a small surface. The two
encoders share weights. An overlap-attention block in the bottleneck lets the
two clouds exchange information, and the decoder outputs a descriptor per point
together with two separate per-point scores, overlap and matchability
[arxiv:2011.13005 §3.3] [arxiv:2011.13005 §3.4]. The descriptor is 32-dimensional
on 3DMatch [arxiv:2011.13005 tab. 7].

At test time, interest points are drawn without replacement, with probability
proportional to overlap times matchability. RANSAC then estimates the transform
[arxiv:2011.13005 §4.1] [arxiv:2011.13005 tab. 1]. Training uses ground-truth
correspondences for three losses: a circle loss on the descriptors, and
binary cross-entropy on the overlap and matchability scores
[arxiv:2011.13005 §3.5].

Author-reported results:

- 3DLoMatch (10-30% overlap), registration recall at 5000 samples: 59.8%,
  against 40.1% for FCGF and 37.2% for D3Feat [arxiv:2011.13005 tab. 2].
- 3DMatch at 5000 samples: 89.0%, against 85.1% for FCGF
  [arxiv:2011.13005 tab. 2].
- Sampling on 3DLoMatch at 5000 samples: top-k over the overlap-matchability
  product gives a higher inlier ratio than the probabilistic choice (33.7%
  against 26.7%) but lower registration recall (58.5% against 59.8%)
  [arxiv:2011.13005 tab. 1].

Cost: 30 epochs on 3DMatch at batch size 1 [arxiv:2011.13005 app. A.3]. About
271 ms per fragment pair on a GTX 1080 Ti, of which about 191 ms is data
loading [arxiv:2011.13005 tab. 11]. Card: [PREDATOR: Registration of 3D Point Clouds with Low Overlap](/papers/2011.13005).

### Learned inlier filtering with a differentiable pose solver (Deep Global Registration)

This variant keeps the detect-describe-match front end and changes what is done
with its output. Each point gets a fully convolutional descriptor (FCGF, from
prior work with no card here). Each point in the first scan is matched to its
nearest neighbour in feature space [arxiv:2004.11540 §3.1] [arxiv:2004.11540 §3.2].
A U-Net-shaped 6-D convolutional network then scores every putative match, with
the match written as the point pair in R^6, and predicts an inlier probability
[arxiv:2004.11540 §3.2] [arxiv:2004.11540 fig. 2].

Scores below a clip threshold are set to zero, and the rest enter a closed-form
weighted Procrustes solve [arxiv:2004.11540 §3.3]. A robust refinement on SE(3)
follows, with the matches held fixed [arxiv:2004.11540 §4.2]. A safeguard
checks the fraction of weight that survives the clip. When that fraction is
below τ_s = 0.05, the method falls back to RANSAC or branch-and-bound
[arxiv:2004.11540 §4]. Card: [Deep Global Registration](/papers/2004.11540).

Author-reported results on 3DMatch, 5 cm voxels, on a GTX 1080Ti
[arxiv:2004.11540 §5.1]:

- Recall 91.3%, TE 7.34 cm, RE 2.43°, in 1.21 s. RANSAC with 8M iterations
  reaches 74.9% in 4.55 s, and FGR reaches 42.7% in 0.31 s
  [arxiv:2004.11540 tab. 1].
- Without the safeguard the recall is 85.2% [arxiv:2004.11540 tab. 1].
- On KITTI, with 30 cm voxels, the method reaches 96.9% recall and 98.0% with
  ICP. The FCGF row of the same table reaches 98.2% [arxiv:2004.11540 tab. 3].

Its timings include feature extraction [arxiv:2004.11540 tab. 1].

### Learned spatial consistency with seeded spectral fits (PointDSC)

PointDSC also filters putative matches from a fixed descriptor, but it judges
each match against the others rather than on its own. Two correspondences that
are both correct keep the same distance between their two points in both scans.
A nonlocal attention block uses that rule: feature similarity is multiplied by
a length-consistency term, which is zero when the length changes by more than
σ_d [arxiv:2103.05465 §3.2] [arxiv:2103.05465 eq. 1] [arxiv:2103.05465 eq. 2].
An MLP on the output gives each match an initial confidence, and non-maximum
suppression picks well-spread seeds, capped at 10% of the input at test time
[arxiv:2103.05465 §3.3] [arxiv:2103.05465 §4].

For each seed, the k = 40 nearest neighbours in feature space, not in 3-D, form
a subset [arxiv:2103.05465 §3.4] [arxiv:2103.05465 §4]. A compatibility matrix
built from the length term and a learned feature term has a leading eigenvector
whose entries serve as inlier weights. A weighted SVD fit then gives one
rigid hypothesis per seed [arxiv:2103.05465 §3.4] [arxiv:2103.05465 eq. 3]
[arxiv:2103.05465 eq. 5]. The hypothesis with the most matches inside the
inlier threshold τ is kept [arxiv:2103.05465 §3.5] [arxiv:2103.05465 eq. 6].
The result is refined by reweighting the inliers, for at most 20 rounds
[arxiv:2103.05465 app. Post-refinement].

Training labels each match as an inlier from the true pose. The loss sums a
binary cross-entropy on the initial confidence and a squared error on the
compatibility targets, which are 1 only when both matches are inliers
[arxiv:2103.05465 §3.6] [arxiv:2103.05465 eq. 7] [arxiv:2103.05465 eq. 8]
[arxiv:2103.05465 eq. 9]. The paper uses Adam with learning rate 1e-4, 100
epochs, batch size 16 and 1,000 sampled matches per pair, with λ = 3
[arxiv:2103.05465 §4]. Card: [PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency](/papers/2103.05465).

Author-reported results, from the card's tables:

- 3DMatch with FCGF, 5 cm voxels, τ = 10 cm: registration recall 93.28%,
  against 91.30% for DGR and 91.50% for RANSAC with 100k iterations
  [arxiv:2103.05465 tab. 1].
- Same setting, 0.09 s per pair against 1.36 s for DGR and 5.50 s for RANSAC
  with 100k iterations [arxiv:2103.05465 tab. 1]. The text gives about 60 times
  against RANSAC with 100k iterations [arxiv:2103.05465 §5.1].
- 3DMatch with FPFH, same model and not fine-tuned: 78.50%, against 69.13% for
  DGR and 73.57% for RANSAC with 100k iterations [arxiv:2103.05465 tab. 1].
- KITTI with FPFH, re-trained, TE below 60 cm and RE below 5°: 98.20%, against
  89.37% for RANSAC with 100k iterations and 77.12% for DGR re-trained. Time is
  0.31 s against 13.7 s [arxiv:2103.05465 tab. 2].
- 3DLoMatch: the appendix reports FCGF rising from 35.7 to 52.0 when PointDSC is
  added [arxiv:2103.05465 app. 3DLoMatch].
- Ablation on 3DMatch: the spatial term lifts the classifier-head recall from
  83.30% to 88.17% [arxiv:2103.05465 tab. 4]. Removing the spectral loss lowers
  the full model from 93.28% to 92.61% [arxiv:2103.05465 tab. 5].

Cost: 89.5 ms per 3DMatch pair with about 5,000 matches, of which 62.0 ms is
the nonlocal encoder. The timing comes from an 8-core Xeon E5-2620 with a GTX
1080 Ti [arxiv:2103.05465 app. Time complexity]. The attention is a full N-by-N
matrix in every block, so cost grows quickly with the match count.

Limits: gains shrink with fewer than about 500 matches [arxiv:2103.05465 app. 3DLoMatch].
Large symmetric surfaces such as walls and floors can produce 90° or 180°
rotation errors when outlier clusters dominate [arxiv:2103.05465 app. Qualitative Results]
[arxiv:2103.05465 fig. 7].

### Learned soft keypoint matching with iterated SVD (PRNet)

PRNet aligns two partial point clouds without labelled correspondences. Each
pass embeds both clouds with a DGCNN, then passes the two embeddings through
one Transformer encoder and one decoder so that each cloud sees the other
[arxiv:1910.12240 §3.2]. It takes the k points with the largest feature norm
in each cloud as keypoints, and the paper states no supervision for them
[arxiv:1910.12240 eq. 9] [arxiv:1910.12240 §4.3]. A value head predicts a
temperature from the pooled features. Gumbel-Softmax at that temperature gives
a keypoint-to-keypoint matrix that is one-hot in the forward pass, with a
straight-through gradient [arxiv:1910.12240 eq. 10] [arxiv:1910.12240 eq. 11].
The matched keypoints give a rigid motion through the closed-form SVD fit
[arxiv:1910.12240 eq. 2] [arxiv:1910.12240 eq. 8]. The source is moved and the
pass repeats, with three passes in the paper's training setting and a discount
factor below 1 on later passes [arxiv:1910.12240 eq. 7] [arxiv:1910.12240 eq. 12].
The iterative loop is the same idea as [iterative closest point](/concepts/iterative-closest-point),
with learned keypoints and matching in place of nearest neighbours.

Training is self-supervised. Each pair is a partial crop of one object, moved
by a random rigid motion whose true value serves as the label [arxiv:1910.12240 §4.1].
The loss sums a motion term, a cycle-consistency term and a feature-alignment
term [arxiv:1910.12240 eq. 13] [arxiv:1910.12240 eq. 14] [arxiv:1910.12240 eq. 15].
Matching is sampled at test time too, so the released code's predictions depend
on the random seed, and the paper does not say whether it averaged over
samples [code: model.py#L441]. Card: [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240).

Author-reported results, all on synthetic motions:

- ModelNet40, unseen point clouds: rotation MSE 10.235 for PRNet against
  45.005 for DCP-v2, and the best value in all eight columns of the table
  [arxiv:1910.12240 tab. 1].
- Unseen categories: rotation MSE 24.857 against 95.431 for DCP-v2. FGR has
  lower rotation MAE (1.952 against 2.329) and translation MAE (0.007 against
  0.015) [arxiv:1910.12240 tab. 2].
- Gaussian noise: rotation MSE 18.691 against 47.378 for DCP-v2
  [arxiv:1910.12240 tab. 3].
- Real scans: qualitative only. The model was fine-tuned for 10 epochs on 100
  synthetic examples per scan [arxiv:1910.12240 §4.2].
- Keypoint choice: random keypoints give rotation MAE 1.689, against 1.454 for
  the norm-based choice [arxiv:1910.12240 supp. (a)].
- Timing per pair at 1024 points: 0.073 s for PRNet against 0.024 s for DCP-v2
  [arxiv:1910.12240 supp. Efficiency].

### Learned matches as goal checks in image-goal navigation (FeudalNav)

This variant does not estimate a pose. An agent is shown a goal image and must
walk to the place it shows. The paper uses SuperGlue, a learned matcher on
detected keypoints, in three places [arxiv:2402.12498 §4.1] [arxiv:2402.12498 §4.2]
[arxiv:2402.12498 §4.3]:

- A view joins the cluster of earlier views whose match confidence is highest,
  if that confidence is above ψ = 0.7, and otherwise starts a new cluster
  [arxiv:2402.12498 §4.1] [arxiv:2402.12498 §5.2].
- When the current view matches the goal image with confidence above β = 0.7,
  the mean of the matched keypoints replaces the predicted waypoint
  [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §5.2].
- Between two environment steps, a crop around the waypoint is matched again to
  re-locate it. The agent stops when the goal-match confidence is above β and the
  depth at the centre of the matched keypoints is at most 1 m
  [arxiv:2402.12498 §4.3].

The card does not measure matching accuracy on its own. It also leaves the
confidence aggregation over keypoints undefined [arxiv:2402.12498 §4.2], and it
does not say how the centre used for the stopping depth is computed
[arxiv:2402.12498 §4.3]. The reported results are navigation success and SPL on
Gibson, averaged over difficulty bands. Stacked FeudalNav reaches 40.50 success
on curved paths, against 23.97 for NRNS without noise [arxiv:2402.12498 tab. 2].
Card: [Feudal Networks for Visual Navigation](/papers/2402.12498).

## When to use which

The table gives the constraints the cards' results address. All numbers are
author-reported, and each row compares only the numbers from its own card.

| Situation | Use | Why |
|---|---|---|
| Indoor scenes with large plain or repeated areas | Detector-free LoFTR | Higher ScanNet and HPatches scores than the detector-based baselines in [arxiv:2104.00680 tab. 1] and [arxiv:2104.00680 tab. 2] |
| Outdoor scenes with large viewpoint change | Detector-free LoFTR, DS head | MegaDepth AUC@10° of 69.19 for DS against 61.16 for SuperPoint with SuperGlue [arxiv:2104.00680 tab. 3] |
| Lower latency is needed | DS head over OT head | 116 ms for DS against 130 ms for OT at 640×480 on an RTX 2080 Ti [arxiv:2104.00680 §3.6]. DS is also ahead at 5° and 10° on ScanNet [arxiv:2104.00680 tab. 2] |
| The lowest latency is needed and some accuracy can be traded | Smaller model (1/16 coarse, 1/4 fine) | 104 ms, but ScanNet AUC@10° drops to 34.82 [arxiv:2104.00680 tab. 6] |
| Only MegaDepth training data is available for indoor use | Expect lower indoor accuracy | OT trained on MegaDepth reaches 33.62 AUC@10° on ScanNet against 40.39 when trained on ScanNet [arxiv:2104.00680 tab. 2] |
| Night-time visual localisation with a full pipeline | LoFTR-OT or SuperPoint with SuperGlue | Full HLoc at night: LoFTR-OT leads at 0.25 m (78.5 against 77.0) and trails at 1.0 m (99.0 against 100.0) [arxiv:2104.00680 tab. 4] |
| Large changes in lighting, season or viewpoint between two images, and dense matches are wanted | RoMa | WxBS 80.1 against 58.9 for DKM and 55.4 for LoFTR [arxiv:2305.15404 tab. 4]. In the same paper's tables, MegaDepth-1500 AUC@10° is 76.7 against 69.2 for LoFTR [arxiv:2305.15404 tab. 5] |
| Dense matching without dense supervised correspondences | Not RoMa as published | Training needs them [arxiv:2305.15404 §4.2] |
| Dense matching where per-pair time is the constraint | Measure before choosing RoMa | RoMa is about 7% slower than DKM at 560×560 [arxiv:2305.15404 §4.5]. The card gives no RoMa timing against LoFTR, and the LoFTR latency above was measured on other hardware |
| Training budget is limited | Plan for a large run | LoFTR on ScanNet takes about 24 h on 64 GTX 1080Ti GPUs [arxiv:2104.00680 §3.6]. PREDATOR runs 30 epochs on 3DMatch at batch size 1 [arxiv:2011.13005 app. A.3]. PRNet runs 100 epochs, about 10 hours on four Tesla V100 GPUs [arxiv:1910.12240 supp. Training Protocol] |
| 3D objects with meshes, and a fixed descriptor is wanted | RoPS | Descriptor computed from fixed formulas [arxiv:1304.3192 §4.1]. 100% on Bologna and 98.9% on UWA [arxiv:1304.3192 §7.1] [arxiv:1304.3192 §7.2]. Parameters were tuned on six models [arxiv:1304.3192 §3.3] |
| Only raw point clouds, no meshes | PREDATOR | RoPS needs mesh triangles [arxiv:1304.3192 §3.1]. The paper meshed its point-cloud scenes first, and its results depend on that step [arxiv:1304.3192 §7.3] |
| Scan pairs with 10-30% overlap and labelled training pairs | PREDATOR | 59.8% against 40.1% for FCGF on 3DLoMatch [arxiv:2011.13005 tab. 2] |
| Scan pairs with high overlap | PREDATOR still leads, by a smaller margin | 89.0% against 85.1% for FCGF on 3DMatch [arxiv:2011.13005 tab. 2] |
| Partial views with unknown correspondences, and synthetic motions are easy to generate | PRNet | Trained without labelled correspondences [arxiv:1910.12240 §4.3]. Rotation MSE 10.235 against 45.005 for DCP-v2 on ModelNet40 [arxiv:1910.12240 tab. 1]. These results are on synthetic motions only, and the card has no 3DMatch or real-scan numbers [arxiv:1910.12240 §4.2] |
| Real scans, with no per-scan fine-tuning | Not PRNet as published | The paper fine-tunes per scan, with 100 synthetic examples each, and gives no numeric real-scan result [arxiv:1910.12240 §4.2] |
| Putative matches from a fixed descriptor, many of them outliers, and a GPU for training and inference | Learned filter with weighted Procrustes (Deep Global Registration) | 91.3% against 74.9% for RANSAC with 8M iterations on 3DMatch, and 1.21 s against 4.55 s [arxiv:2004.11540 tab. 1]. The network is trained on 3DMatch [arxiv:2004.11540 §5.1] |
| Putative matches from FCGF or FPFH, many of them outliers, a single rigid transform wanted, and the released setting for your data is known | PointDSC | 93.28% against 91.30% for DGR on 3DMatch with FCGF, and 0.09 s against 1.36 s [arxiv:2103.05465 tab. 1]. The DGR card gives 1.21 s for DGR on the same benchmark, so compare the two DGR rows with care [arxiv:2004.11540 tab. 1] |
| Outdoor scans with FPFH, re-trained on the target data, and a fixed time budget | PointDSC | 98.20% against 89.37% for RANSAC with 100k iterations, and 0.31 s against 13.7 s [arxiv:2103.05465 tab. 2]. The released KITTI config uses a different σ_d from the paper (see Contradictions) |
| Speed matters more than recall | FGR, if its recall is acceptable | 0.31 s on 3DMatch, but 42.7% recall [arxiv:2004.11540 tab. 1] |
| Outdoor LiDAR at 30 cm voxels | Compare before choosing the filter | The FCGF row reaches 98.2% recall on KITTI, against 96.9% for the filter and 98.0% with ICP [arxiv:2004.11540 tab. 3] |
| Image-goal navigation in unseen indoor scenes, with no odometry and no RL, and the policy must confirm when the goal is reached | Learned matches as goal checks (FeudalNav) | A confident goal match replaces the waypoint and gates stopping [arxiv:2402.12498 §4.2] [arxiv:2402.12498 §4.3]. Curved-path success 40.50 against 23.97 for NRNS without noise, on Gibson [arxiv:2402.12498 tab. 2]. The matching step itself is not measured, and the confidence is not defined [arxiv:2402.12498 §4.2] |
| Image-goal navigation with straight paths, where SPL matters | Compare before choosing | Stacked FeudalNav's straight-path SPL is 30.74, against 41.49 for NRNS without noise [arxiv:2402.12498 tab. 2] |

## Implementation sketch

Pseudocode for the LoFTR inference path, with the coarse and fine steps. The
shapes and constants come from the paper, and the code defaults are noted where
the paper does not state them. The RoPS card gives pseudocode for its LRF,
descriptor and recognition loop, and the PREDATOR card gives pseudocode for
overlap attention, its losses and the overlap-weighted sampling. The Deep
Global Registration card gives pseudocode for the whole filter-and-solve
pipeline. The common core of that family is: score each putative match, zero
the weak scores, then solve a weighted pose [arxiv:2004.11540 alg. 1]. The
PointDSC card gives pseudocode for seeded hypotheses and for its loss. It
differs by fitting one pose per seed, not one pose overall
[arxiv:2103.05465 §3.4] [arxiv:2103.05465 §3.6]. The PRNet card gives
pseudocode for one pass, its SVD fit and its discounted training loss
[arxiv:1910.12240 §3.2].

```pseudocode
# Coarse-to-fine dense matching, one image pair  [arxiv:2104.00680 §3.1-§3.4]
def match(image0, image1):
    # Coarse map at 1/8 and fine map at 1/2 of the image side  [§3.1]
    coarse0, fine0 = backbone(image0)
    coarse1, fine1 = backbone(image1)
    # One 2-D sine encoding per image, then flatten to token sequences  [§3.2]
    coarse0 = flatten(coarse0 + pos_encoding(coarse0))
    coarse1 = flatten(coarse1 + pos_encoding(coarse1))
    # Alternate self and cross linear attention, N_c = 4 times  [§3.2] [§3.6]
    for kind in ["self", "cross"] * 4:
        if kind == "self":
            coarse0 = attend(coarse0, coarse0)
            coarse1 = attend(coarse1, coarse1)
        else:
            coarse0 = attend(coarse0, coarse1)
            coarse1 = attend(coarse1, coarse0)  # reads updated coarse0
    # Dual-softmax confidence over all coarse pairs; tau not stated  [§3.3]
    # tau: not stated in the paper; code uses 0.1  [code: src/config/default.py#L30]
    scores = coarse0 @ coarse1.T / tau
    conf = softmax_rows(scores) * softmax_cols(scores)
    # Keep mutual nearest neighbours with confidence >= 0.2  [§3.3] [§3.6]
    coarse_matches = mutual_nearest(conf) & (conf >= 0.2)
    points = []
    for (i, j) in coarse_matches:
        # 5x5 window around each coarse match on the 1/2 map  [§3.4] [§3.6]
        w0 = crop(fine0, centre=i, size=5)
        w1 = crop(fine1, centre=j, size=5)
        # One fine self and cross round, then a heatmap from the window centre  [§3.6]
        w0, w1 = fine_round(w0, w1)
        heat = softmax(w1 @ w0[centre] / sqrt(d_fine))
        # Sub-pixel offset: expectation of the heatmap  [§3.4]
        points.append((pos(i), pos(j) + expectation(heat)))
    return points
```

## Contradictions

Disagreements between papers, and within the cards' papers, each side cited.

- **Cross-family numbers.** The seven variants are scored on different tasks:
  image pose, homography and warp accuracy (LoFTR, RoMa), recognition rate on object scenes
  (RoPS), registration recall on scan pairs (PREDATOR), registration recall
  with TE and RE thresholds (Deep Global Registration), rotation and
  translation error on synthetic ModelNet40 pairs (PRNet), and navigation
  success and SPL on Gibson (FeudalNav). No card reports
  another card's benchmark, so the numbers above are not comparable across
  variants [arxiv:2104.00680 tab. 2] [arxiv:2305.15404 tab. 4]
  [arxiv:1304.3192 tab. 6] [arxiv:2011.13005 tab. 2] [arxiv:2004.11540 tab. 1]
  [arxiv:2103.05465 tab. 1] [arxiv:1910.12240 tab. 1] [arxiv:2402.12498 tab. 2].
  PointDSC's registration recall uses the same TE and RE thresholds as DGR,
  but its KITTI rows use a different setting from the DGR card's KITTI row
  [arxiv:2103.05465 §5.2] [arxiv:2004.11540 tab. 3].
- **DGR numbers in two papers.** PointDSC's table gives DGR 91.30% recall,
  7.48 cm TE, 2.40° RE and 1.36 s on 3DMatch with FCGF [arxiv:2103.05465 tab. 1].
  The DGR card gives 91.3% recall, 7.34 cm, 2.43° and 1.21 s on the same
  benchmark [arxiv:2004.11540 tab. 1]. The cards do not say why the timing and
  error differ. Compare the two papers' DGR rows with that in mind.
- **PointDSC paper and released code.** The paper sets σ_d to 60 cm for
  outdoor scenes [arxiv:2103.05465 §4]. The KITTI release config and test script
  use 1.2 m [code: snapshot/PointDSC_KITTI_release/config.json#L31]
  [code: evaluation/test_KITTI.py#L167]. The paper also sets λ = 3, while the
  released configs use 1.0 for both weights [arxiv:2103.05465 eq. 10]
  [code: snapshot/PointDSC_3DMatch_release/config.json#L14-L15]. The 3DMatch
  release runs 50 epochs, against 100 in the paper [code: snapshot/PointDSC_3DMatch_release/config.json#L19].
  Reimplementing from the paper alone may not reproduce the table.
- **RoMa paper and released code.** The paper sets the Charbonnier scale
  c = 0.03 [arxiv:2305.15404 §3.4]. The training script sets c = 1e-4 on
  normalised grid coordinates [code: experiments/train_roma_outdoor.py#L219-L220].
  The paper's kernel has inverse temperature 10 [arxiv:2305.15404 app. A]. The
  code's T = 0.2 gives an inverse temperature of 5 under the form the code uses
  [code: romatch/models/model_zoo/roma_models.py#L153]
  [code: romatch/models/matcher.py#L187-L205]. The training split and the
  augmentation also differ: the paper uses DKM's split, and the code uses
  LoFTR's split with horizontal flips [arxiv:2305.15404 §4.2]
  [code: experiments/train_roma_outdoor.py#L199-L212]. The training script is
  marked untested, and as written it passes an argument that the model
  constructor does not take [code: experiments/train_roma_outdoor.py#L159-L165]
  [code: romatch/models/encoders.py#L29-L30]. Reimplementing from the paper alone
  may not reproduce the table.
- **PRNet paper and released code.** The code divides the matching logits by
  sqrt(d_k), while the paper's logits are unscaled [arxiv:1910.12240 eq. 10]
  [code: model.py#L438]. The code clamps the predicted temperature to
  [0.01, 100], a range the paper does not state [arxiv:1910.12240 §3.2]
  [code: model.py#L411]. For the backward pass, the paper uses the DCP softmax
  gradient [arxiv:1910.12240 §3.2], while the code uses PyTorch's Gumbel-Softmax
  straight-through gradient [code: model.py#L441]. Reimplementing from the paper
  alone may not reproduce the table.
- **RoPS baselines use its own frame.** For spin image, NormHist, LSP and
  THRIFT the paper replaces their LRF with its own, so the comparison tests the
  descriptor and the new LRF together [arxiv:1304.3192 §7.1].
- **RoPS abstract against body.** The abstract gives 96.0% on Ca' Foscari. The
  body reports 99% precision and 96% recall, and the abstract figure matches
  the recall rather than the precision [arxiv:1304.3192 §7.4]
  [arxiv:1304.3192 tab. 7]. This page quotes the body.
- **PREDATOR headline numbers.** The arXiv abstract in our metadata gives 89%,
  while the LaTeX source gives 90.6%. The table gives 89.0% at 5000 samples and
  90.6% at 1000 samples on 3DMatch [arxiv:2011.13005 tab. 2]. This page quotes
  the 5000-sample value.
- **PREDATOR paper and released code.** The paper adds the matchability loss
  after a few epochs. The released code keeps its weight at zero until
  validation recall passes 0.3 [arxiv:2011.13005 §3.5]
  [code: lib/trainer.py#L247-L251]. The code also computes the matchability
  loss only on overlap points [code: lib/loss.py#L131-L143], while the paper
  sums over all points [arxiv:2011.13005 eq. 11].
- **Ablation against the main table.** The ablation text says OT matching is
  used, but the Full row does not match the OT row of the main ScanNet table
  [arxiv:2104.00680 tab. 6] [arxiv:2104.00680 tab. 2]. Compare ablation rows
  only with each other.
- **InLoc claim against its table.** The text says LoFTR-OT outperforms all
  published methods on InLoc. In the table, SuperPoint with SuperGlue scores
  higher at DUC1 0.25 m and DUC2 0.5 m [arxiv:2104.00680 §4.3]
  [arxiv:2104.00680 tab. 5].
- **Border setting.** The ScanNet result depends on the border-removal setting.
  The released code's note gives 40.8 without it, and 39.5 with it
  [code: configs/loftr/indoor/scannet/loftr_ds_eval_new.py#L1-L11].
- **Paper and released code (LoFTR).** The released code differs from the paper
  in the optimiser, the coarse loss, the fine-loss weighting and the positional
  encoding [arxiv:2104.00680 §3.2] [arxiv:2104.00680 §3.5]
  [arxiv:2104.00680 §3.6] [code: src/config/default.py#L104-L113]. Reimplementing
  from the paper alone may not reproduce its numbers.
- **Speed claims in the Deep Global Registration paper.** The caption of its
  timing table says the method is as fast as FGR. The text says it is about
  twice as fast as RANSAC with 2M iterations [arxiv:2004.11540 §5.1]. The table
  gives 1.21 s for the method, 0.31 s for FGR and 1.39 s for RANSAC with 2M
  iterations [arxiv:2004.11540 tab. 1]. This page quotes the table.
- **ICP in the 3DMatch row.** The paper does not say whether its 3DMatch row
  includes ICP. The released code runs point-to-point ICP after every
  registration [code: core/deep_global_registration.py#L317-L322], and its
  README gives the paper's 3DMatch figures [code: README.md#L66]. The KITTI
  row differs slightly: the paper gives 3.46 cm TE for "Ours + ICP", and the
  README gives 3.14 cm [arxiv:2004.11540 tab. 3] [code: README.md#L67].

## Papers

- [LoFTR: Detector-Free Local Feature Matching with Transformers](/papers/2104.00680): a detector-free matcher with a coarse dense stage and a sub-pixel fine stage, using dual-softmax or optimal-transport heads.
- [RoMa: Robust Dense Feature Matching](/papers/2305.15404): a dense matcher with frozen DINOv2 coarse features, VGG19 fine features, an anchor-classification Transformer decoder and a robust refinement loss.
- [Rotational Projection Statistics for 3D Local Surface Description and Object Recognition](/papers/1304.3192): a hand-designed 3D surface descriptor with a sign-fixed local reference frame and a 135-number descriptor, recognised through single correspondences.
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](/papers/2011.13005): overlap attention with learned overlap and matchability scores, used to choose points for registering low-overlap scans.
- [Deep Global Registration](/papers/2004.11540): a 6-D convolutional inlier filter over putative matches, a differentiable weighted Procrustes solve and a robust SE(3) refinement, with a RANSAC fallback.
- [PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency](/papers/2103.05465): a nonlocal attention block with a length-consistency term, seeded neural spectral matching and one weighted rigid fit per seed, used to prune outlier correspondences.
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240): self-supervised keypoint selection and Gumbel-Softmax matching, with a closed-form SVD pose applied over repeated passes.
- [Feudal Networks for Visual Navigation](/papers/2402.12498): SuperGlue matches used to cluster views, to replace a waypoint with a goal match and to decide when to stop, inside a three-level navigation policy.