LoFTR: Detector-Free Local Feature Matching with Transformers

In plain words

Matching two photos of the same scene usually starts by finding distinctive corners or blobs in each photo and then pairing them up. On a blank wall or a repeating tile pattern there are few such spots, so the pairing breaks down. This paper skips the spot-picking step. It looks at every location in both images and decides which location in the other image it corresponds to.

Each location gets a description that depends on the whole scene. An attention layer lets every location look at every other location, both in its own image and in the other image, so a plain patch of wall can be recognised by where it sits relative to nearby edges. To keep the work affordable, the method first matches on a coarse grid, keeps the confident pairs, and then zooms into a small neighbourhood around each pair to place it precisely.

The approach is aimed at the places where spot-based matching struggles, such as indoor rooms with large plain or repetitive areas. Good matches there are what camera-pose and localisation pipelines need.

TL;DR

LoFTR is a detector-free local feature matcher. It builds dense matches between two images from Transformer features instead of from detected keypoints. Coarse matches come from a dual-softmax confidence with a mutual-nearest-neighbour filter, then each match is refined to sub-pixel accuracy inside a 5x5 window. The paper reports gains over detector-based and detector-free baselines on HPatches, ScanNet and MegaDepth. Before reimplementing, read "Paper vs code": the released code differs from the paper in the optimiser, the coarse loss, the fine-loss weighting and the positional encoding.

Problem

Detector-based pipelines (detect, describe, match) need interest points that repeat between the two images. Detectors often find too few of them under poor texture, repetitive patterns, viewpoint change, illumination change and motion blur. The problem is most visible indoors, where low-texture or repetitive regions can cover most of the view §1.

Earlier detector-free methods that match dense CNN features have a limited receptive field, so they cannot tell apart regions that look alike locally. The authors argue that a large receptive field is needed for those regions §1.

Full self-attention over all positions costs O(N²) in the number of positions. The authors call it impractical even after the CNN has reduced the grid, which is why they use a linear variant §3.2. At 640×480 input the coarse 1/8 grid has 80×60 = 4,800 positions (our arithmetic from the stated resolutions).

Method

LoFTR runs in two stages. A coarse stage (1/8 of the image side) finds candidate correspondences with Transformer features. A fine stage (1/2 of the image side) refines each candidate to a sub-pixel position §3.1 §3.4.

1. Local feature CNN. A ResNet with a feature pyramid (FPN) produces a coarse map at 1/8 and a fine map at 1/2 of the image side §3.1. The official code takes a single grayscale channel resnet_fpn.py.

2. Positional encoding and LoFTR module (coarse). A 2-D sinusoidal encoding, extended from DETR, is added once to the coarse features of each image, which are then flattened to sequences §3.2. The module alternates self-attention (an image attends to itself) and cross-attention (an image attends to the other image). The paper uses this alternation N_c = 4 times §3.2 §3.6. The attention is linear: the softmax kernel is replaced by φ(x) = elu(x) + 1, which lets the sum over positions be reordered and cost O(N) §3.2.

3. Coarse matching. The features are compared with a scaled dot product. The paper turns the scores into a confidence matrix in one of two ways §3.3:

• Dual softmax: the product of a softmax over the other image's positions and a softmax over the first image's positions.
• Optimal transport: a Sinkhorn solver with three iterations and a learned "dustbin" score for unmatched positions §3.3 §3.6.

Coarse matches are pairs whose confidence is at least θ_c = 0.2 and that are mutual nearest neighbours §3.3 §3.6.

4. Coarse-to-fine refinement. For each coarse match (i, j), the method locates the match on the 1/2 fine maps, crops a w×w window (w = 5) around each side, and runs a smaller LoFTR module on the two windows, with N_f = 1 §3.4 §3.6. The fine features of the window are concatenated with the coarse feature of the match, which the paper describes as upsampled coarse features §3.6. The centre feature of window 0 is correlated with all features of window 1, giving a heatmap. The sub-pixel position is the expectation of that heatmap §3.4. The total variance of the heatmap is the match's uncertainty §3.5.

5. Supervision. The coarse loss is a negative log-likelihood over the ground-truth coarse pairs §3.5. Ground truth comes from camera poses and depth maps: two 1/8-grid cells are a ground-truth pair when they are mutual nearest neighbours after reprojection §3.5. The fine loss is an L2 distance between the predicted position and the position warped with the ground-truth pose and depth, weighted by the inverse of the heatmap variance. The gradient does not flow through that weight. Windows where the warped point falls outside the window are skipped §3.5. The total loss is the sum of the two terms §3.5.

The pseudocode below gives the inference path. Shapes are for one image pair on a 640×480 input; the channel counts are not stated in the paper and are taken from the code defaults default.py.

# LoFTR inference for one image pair  [§3.1] to [§3.4]
# Input:  image0, image1: grayscale images (1, H, W)
# Output: matches (point in image0, sub-pixel point in image1)

def loftr(image0, image1):
    # 1. Local feature CNN: coarse map at 1/8, fine map at 1/2  [§3.1]
    coarse0, fine0 = backbone(image0)       # (d, H/8, W/8), (d_f, H/2, W/2)
    coarse1, fine1 = backbone(image1)
    # 2. One sine encoding per image, then flatten to sequences  [§3.2]
    coarse0 = flatten(coarse0 + sine_encoding(coarse0))   # (L, d)
    coarse1 = flatten(coarse1 + sine_encoding(coarse1))   # (S, d)
    # 3. Alternate self and cross linear attention, N_c = 4 times  [§3.2]
    # Each attend() call is attention plus a feed-forward block, see
    # "Paper vs code" [code: src/loftr/loftr_module/transformer.py#L91-L99]
    for layer_name in ["self", "cross"] * 4:
        if layer_name == "self":
            coarse0 = attend(coarse0, coarse0)
            coarse1 = attend(coarse1, coarse1)
        else:
            coarse0 = attend(coarse0, coarse1)
            coarse1 = attend(coarse1, coarse0)
    # 4. Dual-softmax confidence over all coarse pairs  [§3.3]
    # Code divides the dot product by d and by the temperature 0.1
    # [code: src/loftr/utils/coarse_matching.py#L109-L119]
    scores = matmul(coarse0, transpose(coarse1)) / (d * 0.1)   # (L, S)
    confidence = row_softmax(scores) * column_softmax(scores)  # (L, S)
    # 5. Keep pairs above theta_c = 0.2 that are mutual nearest  [§3.3]
    # Code also clears a 2-cell border (BORDER_RM = 2), not in the paper
    matches = mutual_nearest(confidence) & (confidence > 0.2)
    coarse_matches = list_of_pairs(matches)               # (i, j) pairs
    # 6. Crop a 5x5 fine window around each coarse match  [§3.4] [§3.6]
    for (i, j) in coarse_matches:
        window0 = crop(fine0, centre=i, size=5)           # (25, d_f)
        window1 = crop(fine1, centre=j, size=5)           # (25, d_f)
        # Coarse feature of the match, projected and joined to every
        # window position
        # [code: src/loftr/loftr_module/fine_preprocess.py#L49-L58]
        window0 = merge(window0, project(coarse0[i]))
        window1 = merge(window1, project(coarse1[j]))
        # 7. One fine self and cross round on the windows  [§3.6]
        window0, window1 = fine_transformer(window0, window1)
        # 8. Heatmap: centre of window0 against all of window1  [§3.4]
        heatmap = softmax(window1 @ window0[centre] / sqrt(d_f))  # (25,)
        # 9. Sub-pixel offset: expectation of the heatmap  [§3.4]
        # Code scales the offset by (W // 2) * 2 pixels
        # [code: src/loftr/utils/fine_matching.py#L69]
        offset = expectation(heatmap)                      # in [-1, 1]^2
        point1 = centre_of(j) + offset * 2 * (5 // 2)
        # Uncertainty: total variance of the heatmap  [§3.5]
        variance = total_variance(heatmap)
    return matches

Hyperparameters

Results

Author-reported. The setups are as follows. HPatches: 52 illumination and 56 viewpoint sequences, one reference paired with five images, LoFTR trained on MegaDepth, OpenCV RANSAC for homography §4.1. ScanNet: 1500 test pairs from SuperGlue's list, training on 230M pairs, essential-matrix RANSAC §4.2. MegaDepth: 1500 pairs from two validation scenes §4.2. Aachen Day-Night v1.1 and InLoc through HLoc §4.3. The baselines and their defaults are given in §4.1 and §4.2.

Homography estimation, HPatches tab. 1. AUC of the corner error (%), at 3, 5 and 10 px. Max 1K matches for LoFTR.

Relative pose, ScanNet (indoor) tab. 2. Pose AUC (%) at 5°, 10°, 20°. DRC-Net and LoFTR-OT† are models trained on MegaDepth (†).

Relative pose, MegaDepth (outdoor) tab. 3. Pose AUC (%) at 5°, 10°, 20°. The text reports LoFTR-DS as 61% above DRC-Net and 13% above SuperGlue at AUC@10°; these match the DS row against the rows below §4.2.

Visual localisation, Aachen Day-Night v1.1 tab. 4. Accuracy (%) at (0.25 m, 2°) / (0.5 m, 5°) / (1.0 m, 10°). Night only for the local-feature track, where the day column is not reported.

Visual localisation, InLoc tab. 5. Accuracy (%) at (0.25 m, 10°) / (0.5 m, 10°) / (1.0 m, 10°), for the DUC1 and DUC2 query sets.

Ablation, ScanNet, OT matching tab. 6. Pose AUC (%) at 5°, 10°, 20°.

The paper's text describes the ablation as using OT matching, but the Full row does not match the OT row of tab. 2. See Pitfalls.

Paper vs code

Code: the official repository at code_commit (df7ca80), Apache-2.0.

• Optimiser and learning rate. Paper: Adam, lr 1e-3, batch 64 on ScanNet §3.6. Code: AdamW with weight decay 0.1 and a canonical learning rate of 6e-3 at batch 64, scaled linearly by world size times per-GPU batch default.py train.py [code: src/optimizers/init.py#L10-L12]. The outdoor config uses 8e-3 with warm-up loftr_ds.py.
• Learning-rate warm-up and schedule. The paper states neither. The default is a linear warm-up over 4800 steps from 0 and MultiStepLR with γ = 0.5; the indoor configs use milestones up to epoch 29 default.py loftr_ds.py lightning_loftr.py.
• Coarse loss. Paper: NLL over ground-truth pairs with no focal term §3.5. Code default: focal loss with α = 0.25 and γ = 2 default.py, applied only to ground-truth positives when dual-softmax is used with sparse supervision, so unmatched cells receive no negative term loftr_loss.py loftr_loss.py.
• Fine loss. Paper: distance (norm, not squared), weighted by 1/σ² where σ² is the total variance §3.5. Code: squared distance, weighted by 1/(σx + σy) (sum of per-axis standard deviations), normalised to mean 1 and detached loftr_loss.py fine_matching.py.
• Fine-level context. Paper: coarse features are upsampled and concatenated with the fine features §3.6. Code: a linear projection of the single coarse vector at the match, copied to all 25 window positions, then concatenated and merged fine_preprocess.py. No spatial upsampling of the coarse map is visible in the code path.
• Positional encoding. Paper: DETR-style 2-D sine, added once, formula in the supplement §3.2. Code keeps two variants behind TEMP_BUG_FIX; the default is the corrected one, and the comment says the original buggy version is kept for backward compatibility with released models position_encoding.py default.py. The buggy variant is in loftr_ot.py.
• Border removal. Paper: not mentioned. Code: removes a 2-cell border of coarse matches by default default.py coarse_matching.py. The ScanNet evaluation config sets it to 0 and states that this matches the paper's 40.8 AUC@10 loftr_ds_eval_new.py.
• Dual-softmax scaling. Paper: a score scaled by 1/τ, with no value for τ §3.3. Code: both features are divided by √d before the dot product, then the result is divided by τ = 0.1 coarse_matching.py default.py.
• Cross-attention update order. Paper: "interleave" the self and cross layers §3.2. Code: image 0 is updated first, and image 1's cross-attention then reads the updated image-0 features transformer.py.
• Feed-forward block. Paper: not described beyond the Transformer encoder §3.2. Code: a two-layer MLP with hidden width 2d and ReLU, no bias, plus LayerNorm after attention and after the MLP, with residual connections transformer.py. The config keys D_FFN are defined default.py but not read.
• Backbone. Paper: "modified ResNet-18" §3.6. Code: single-channel 7×7 stem with stride 2, three residual stages of two blocks each, and a bilinear-upsampling FPN that returns the 1/8 and 1/2 maps resnet_fpn.py.
• Sinkhorn variant. Paper: OT with three iterations §3.6. Code: imports superglue.py at run time and stops with "download superglue.py first!" when it is absent coarse_matching.py. No file with that name is in the cloned repository.
• ScanNet training overlap. Paper: pairs with overlap 0.4 to 0.8 §4.2. Code: minimum 0.4 is applied scannet.py. This file shows no maximum.
• Coarse sampling in training. Paper: not mentioned. Code: subsamples candidate coarse matches to a fraction of the batch (0.2 indoor, 0.3 outdoor) and pads them with ground-truth pairs at confidence 0, with a floor of 200 coarse_matching.py default.py loftr_ds.py.
• Fine supervision window. Paper: skip a pair whose warped point leaves the window §3.5. Code: keeps pairs whose ∞-norm offset is below 1.0 in normalised window units loftr_loss.py default.py.
• Pose RANSAC threshold. Paper: RANSAC on the essential matrix, threshold not stated §4.2. Code: 0.5 px default.py.
• MegaDepth training overlap. Paper: not stated. Code: 0.0 megadepth_trainval_840.py.

Pitfalls

• Equations are not numbered in the source. All displayed equations in the method use the starred environment, so they have no number in the PDF. Cite them by section, for example §3.3, and do not invent equation numbers.
• Supplement not in our source. The paper defers the positional-encoding formula, the full training details, the timing analysis and the visual localisation protocol to a supplement. We did not have it, so those values are "not stated" here. The code covers some of them, and the table above says which.
• Missing overview figure. The method section refers to an overview figure. The source contains its file (figures/sys-arch.tex), but no \input includes it. Figure numbers in this card are counted from the included figures: teaser fig. 1, attention layer fig. 2, self/cross attention fig. 3, qualitative matches fig. 4, qualitative attention fig. 5. If the compiled PDF numbers them differently, follow the PDF.
• Ablation does not match the table. The ablation text says OT matching is used §4.4, and the Full row is "N_c = 4, N_f = 1". Its values (20.06 / 40.8 / 57.62) differ from the OT row of tab. 2 (21.51 / 40.39 / 57.96) at 5° and 10°. The 10° and 20° values match the DS row (40.8 / 57.62). Treat the ablation as internally inconsistent and compare its rows only with each other.
• Text claim against InLoc table. The text says LoFTR-OT outperforms all published methods on InLoc §4.3. In tab. 5, HLoc + SuperPoint + SuperGlue is higher at DUC1 (0.25 m) and DUC2 (0.5 m), so read the claim against the table.
• Linear attention numerics. The code divides the values by the sequence length before the reduction and multiplies back afterwards, to avoid fp16 overflow. It also adds ε = 1e-6 to the normaliser. Neither appears in the paper's formula linear_attention.py linear_attention.py.
• Coordinate scale in the fine stage. The heatmap offset lives in normalised window units. The code converts it to pixels by multiplying by the window half-width and by the fine-to-image scale, which is 2 at 640 px input. Getting this factor wrong shifts every fine match fine_matching.py fine_matching.py.
• Variance is clamped. Per-axis variances are clamped at 1e-10 before the square root, and the weight in the fine loss is detached fine_matching.py loftr_loss.py.
• Borders are zero-padded. Window extraction pads with zeros, so windows near the image border contain zero-valued features fine_preprocess.py.
• Empty match sets. With no coarse match, the fine stage returns empty results at inference. During training the code asserts that this cannot happen fine_matching.py.
• Dual-softmax leaves negatives unsupervised. Cells that are not ground-truth pairs get no training signal under dual-softmax with sparse supervision. Only the OT variant has a negative term loftr_loss.py.
• Which positional encoding to load. The code keeps two variants, and the paper does not say which one produced its reported numbers. Use the checkpoint's own configuration. The ScanNet evaluation config for the corrected encoding is in loftr_ds_eval_new.py.
• Border sensitivity. The repository says the border-removal bug fix moves the ScanNet AUC@10 from 40.8 to 39.5, which is a small change. Note which setting you use when comparing with the paper loftr_ds_eval_new.py.
• Timing. The paper reports 116 ms for dual-softmax and 130 ms for OT at 640×480 on an RTX 2080 Ti, and 104 ms for the smaller variant §3.6 §4.4. These are author measurements on one GPU.

Lineage

• Builds on: SuperGlue (Sarlin et al., 2019) for Transformer matching with self and cross attention, and for optimal transport §1 §3.2 §3.3. Linear Transformer (Katharopoulos et al., 2020) for the linear attention kernel §3.2. DETR (Carion et al., 2020), linked as DETR, for the sinusoidal positional encoding §3.2. Dual-softmax matching from neighbourhood consensus networks (Rocco et al., 2018) and DISK (Tyszkiewicz et al., 2020) §3.3. Correlation-based sub-pixel refinement (Wang et al., 2020) §3.4.
• Compared against: detector-based D2-Net, R2D2, DISK, SuperPoint with SuperGlue, ORB with GMS, ContextDesc, PointCN and OANet; detector-free Sparse-NCNet and DRC-Net §4.1 §4.2.
• Superseded by: none recorded in this wiki yet.