Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed

In plain words

Matching two photos means finding the same physical points in both, such as a corner of a building seen from two angles. Camera pose estimation, 3D reconstruction and image search all depend on it. Detector-free matchers skip the step of picking keypoints first. They compare the images as a whole, which helps on plain walls and under large changes of viewpoint, but they are slow.

This paper keeps that approach and attacks the slow part. Most of the work happens on a coarse grid of image cells. The authors observe that attention over every cell repeats information that neighbouring cells already share, so they merge neighbouring cells before attention and keep only the strongest ones. A cheaper scoring step is used at inference, and the final positions are refined in two steps: first to a single pixel, then to a fraction of a pixel.

TL;DR

Efficient LoFTR is a semi-dense image matcher. Its optimized model is faster than LoFTR, and its full model is more accurate than AspanFormer on the MegaDepth pose benchmark, both author-reported tab. 1. It merges neighbouring tokens before attention, drops the coarse dual softmax in its optimized model, and refines matches in two stages §3. Use it when you need dense-ish matches at low latency. Read Paper vs code and Pitfalls first: the released configs and scripts differ from the paper in several places.

Problem

Detector-free matching runs a transformer over the whole coarse feature map of both images. Its cost grows with the number of tokens, and the authors call LoFTR's efficiency limited for that reason §1. Linear attention reduces the cost but weakens the model §3.2. QuadTree attention reduces cost through multi-scale steps, which adds latency §1. The paper assumes that the attention weights of neighbouring queries are similar, and that most weight falls on a few keys §3.2.

Method

The pipeline has four parts §3:

1. Feature extraction §3.1. A RepVGG-style backbone is trained with multi-branch blocks and converted to a single branch at inference by re-parameterisation. It produces coarse features at 1/8 resolution and fine features at 1/4 and 1/2 app. B.1.
2. Aggregated attention §3.2. Coarse features of the two images pass through four rounds of self- and cross-attention §4.1. Queries are aggregated by a strided depthwise convolution, and keys and values by max-pooling, both with a 4x4 window. Attention runs on the reduced token set. The output is upsampled back to the coarse grid and fused with the input. Rotary position encoding (RoPE) is used in self-attention only §3.2 app. B.2.
3. Coarse matching §3.3. A dense score matrix between all coarse tokens is built. The full model applies a dual softmax (softmax over both axes, multiplied) and keeps pairs above a threshold that are mutual nearest neighbours. The optimized model skips the softmax and uses the raw scores.
4. Subpixel refinement §3.4. Fine feature patches are cropped around each coarse match. Stage one correlates the two patches densely, takes the best pixel pair, and keeps one match per coarse match. Stage two correlates a 3x3 neighbourhood in image B with the stage-one pixel and takes an expectation, which gives a subpixel position.

Key equations as written in the paper:

\operatorname{VanillaAttention}(Q,K,V) = \operatorname{softmax}(QK^T)V eq. 1

\operatorname{LinearAttention}(Q,K,V) = \phi(Q)(\phi(K)^T\phi(V)), \quad \phi(\cdot)=\operatorname{elu}(\cdot)+1 eq. 2

f'_i = \operatorname{Conv2D}(f_i), \quad f'_j = \operatorname{MaxPool}(f_j) eq. 3

\mathcal{L}_{c} = -\frac{1}{N}\sum_{(\tilde{i},\tilde{j})\in\{\mathcal{M}_c\}_{gt}} \log \mathcal{S}(\tilde{i},\tilde{j}) eq. 4

Eq. 3 is the paper's aggregation step; the paper says a stride of s shrinks the token count by s squared §3.2. The total loss is a weighted sum, L = L_c + alpha L_f1 + beta L_f2, with L_c the coarse term, L_f1 the first fine stage and L_f2 the second §3.5.

The pseudocode below is our paraphrase of the paper and of the released inference code. Citations give the source of each step.

# Efficient LoFTR inference  [§3]
# Input:  image_a, image_b (1, H, W), grayscale
#         [code: src/loftr/backbone/repvgg.py#L161]
# Output: matched pixel positions in both images, (M, 2) each

def efficient_loftr(image_a, image_b):
    # Single-branch CNN after re-parameterisation  [§3.1] [app. B.1]
    # Coarse maps at 1/8; fine maps at 1/4 and 1/2 are kept as skips
    coarse_a, coarse_b, skips_a, skips_b = backbone(image_a, image_b)
    # Four rounds of self-attention then cross-attention  [§3.2] [§4.1]
    for kind in ["self", "cross"] * 4:
        coarse_a, coarse_b = aggregated_layer(coarse_a, coarse_b, kind)
    # Score matrix, dual softmax, mutual nearest neighbours  [§3.3]
    coarse_matches = match_coarse(coarse_a, coarse_b)
    # Fine features: coarse maps fused with the 1/4 and 1/2 skips  [§3.4]
    # [code: src/loftr/loftr_module/fine_preprocess.py#L49-L60]
    fine_a, fine_b = fuse_fine(coarse_a, coarse_b, skips_a, skips_b)
    # Two-stage correlation around each coarse match  [§3.4]
    return refine_two_stage(fine_a, fine_b, coarse_matches)

# Aggregated attention layer  [§3.2] [eq. 3]
# feats_x, feats_src: (C, H/8, W/8); aggregation is 4x4 per side

def aggregated_layer(feats_x, feats_src, kind):
    # Queries: strided depthwise 4x4 conv merges neighbouring tokens  [eq. 3]
    queries = depthwise_conv_4x4(feats_x)
    # Keys and values: 4x4 max-pool keeps the salient tokens  [eq. 3]
    pooled = max_pool_4x4(feats_src)
    keys = project_keys(pooled)
    values = project_values(pooled)
    if kind == "self":
        # Rotary position encoding on self-attention only  [§3.2] [app. B.2]
        queries, keys = rope(queries), rope(keys)
    # Vanilla softmax attention on the reduced token set  [eq. 1]
    attended = attention(queries, keys, values)
    # Upsample to the coarse grid and fuse with the input  [§3.2]
    # [code: src/loftr/loftr_module/transformer.py#L80-L89]
    attended = upsample_bilinear(attended, factor=4)
    return feats_x + feed_forward(concat(feats_x, attended))

# Coarse matching  [§3.3]
# feats_a: (L, C); feats_b: (S, C) coarse tokens of each image

def match_coarse(feats_a, feats_b, skip_softmax):
    # Dense scores, scaled by a temperature
    # [code: src/loftr/utils/coarse_matching.py#L97-L112]
    scores = dense_dot(feats_a, feats_b) / temperature  # (L, S)
    if skip_softmax:
        # Optimized model: raw scores, no softmax  [§3.3]
        confidence = scores
    else:
        # Dual softmax over both token axes  [§3.3]
        # [code: src/loftr/utils/coarse_matching.py#L122]
        confidence = softmax(scores, axis=1) * softmax(scores, axis=0)
    # Keep pairs above the threshold that are mutual nearest neighbours  [§3.3]
    keep = confidence > threshold
    keep = keep and is_row_max(confidence) and is_col_max(confidence)
    return index pairs where keep is true

# Two-stage subpixel refinement  [§3.4]
# patch_a: 8x8 fine features at one coarse match in image A
# patch_b: 10x10 window in image B; its centre 8x8 is used in stage one

def refine_two_stage(patch_a, patch_b, sliced_channels=8):
    # Stage one: dense correlation over all but the last 8 channels  [§3.4]
    # [code: src/loftr/utils/fine_matching.py#L51-L59]
    scores = dense_dot(patch_a[..., :-8], patch_b[..., :-8])
    scores = dual_softmax(scores)  # [code: fine_matching.py#L57]
    # Best pixel pair per coarse match  [§3.4]
    # [code: src/loftr/utils/fine_matching.py#L128-L131]
    pixel_a, pixel_b = argmax(scores)
    # Stage two: 3x3 window in B around pixel_b, correlated with the
    # last 8 channels  [§3.4] [code: src/loftr/utils/fine_matching.py#L92-L99]
    local_scores = local_dot(patch_a[pixel_a][-8:],
                             window_3x3(patch_b, pixel_b))
    weights = softmax(local_scores / 10.0)  # temperature 10 in the configs
    # Expectation gives the subpixel offset of B; A stays at pixel_a  [§3.4]
    return pixel_a, pixel_b + expectation(weights)

# Training loss  [§3.5] [eq. 4] [§4.1]
# gt: coarse and fine ground truth from depth and pose

def training_loss(scores, fine_scores, offsets, gt):
    # Coarse term: log-likelihood in the paper (eq. 4); focal in the code
    # [code: src/losses/loftr_loss.py#L56-L60] [code: src/config/default.py#L51]
    loss_coarse = focal(scores, gt.coarse)
    # First fine stage: same form over each fine patch  [§3.5]
    loss_fine = focal(fine_scores, gt.fine)
    # Second stage: squared error, only where the GT offset is inside
    # one fine pixel (infinity norm)  [§3.5]
    # [code: src/losses/loftr_loss.py#L134-L148]
    loss_sub = squared_error(offsets, gt.offset)
    # Weights alpha = 1.0 and beta = 0.25  [§4.1]
    return loss_coarse + 1.0 * loss_fine + 0.25 * loss_sub

Hyperparameters

Values not given by the paper are marked "not stated". Code values are from the released configs and scripts at the cited commit, not from the paper.

Results

Author-reported. Setup: models trained on MegaDepth only, evaluated on MegaDepth (1500 pairs from Sacre Coeur and St. Peter's Square) and ScanNet §4.2. Pose AUC at 5, 10 and 20 degrees §4.2. Time is per image pair at 640x480 on one RTX 3090; slashes give FP32 / mixed precision tab. 1.

Table 1. Relative pose estimation, MegaDepth and ScanNet tab. 1

Bold in the source marks the best per column, and the authors' ScanNet results favour AspanFormer and SP + SG tab. 1. The paper states that its full model beats AspanFormer on all MegaDepth metrics and is about 2 times faster than AspanFormer §4.2.

Table 2. Homography estimation, HPatches (AUC of corner reprojection error) tab. 2

Table 3. Visual localization, InLoc, DUC1 / DUC2 at (0.25 m, 2°) / (0.5 m, 5°) / (1.0 m, 10°) tab. 3

Table 4. Visual localization, Aachen v1.1, day / night at the same thresholds tab. 4

Table 5. Ablation on MegaDepth, times for a 1200x1200 pair tab. 5

The supplement adds ablations on RoPE versus sinusoidal encoding, aggregation window, input resolution, linear attention after aggregation, ScanNet, and a RANSAC study with OpenCV and LO-RANSAC app. C. Those tables were not transcribed here. In the RANSAC study the gap to LightGlue widens with a tuned naive RANSAC, and with LO-RANSAC the authors report every method improves and their method stays ahead of LightGlue app. C. The resolution study names 1184x1184 as the default MegaDepth size and 640x640 as fastest app. C.

Paper vs code

Code: zju3dv/efficientloftr at commit 07e9c14 (shallow clone), LICENSE "Project Registration License (PRL) v1.0". This is not MIT, Apache-2.0 or BSD, so we cite the code by path and line only and quote none of it.

• Batch and hardware. Paper: batch 16 on 8 V100 §4.1. Released training script: 4 GPUs x batch 5 = 20 eloftr_outdoor.sh.
• Learning rate. Paper: initial 4e-3 §4.1. Code: canonical 8e-3 eloftr_full.py, scaled by total batch over 64 train.py. For the released script that is 8e-3 x 20/64 = 2.5e-3. For the paper's batch 16 it would be 2e-3. Neither equals 4e-3.
• Coarse loss. Paper: log-likelihood (eq. 4) §3.5. Code: focal loss with alpha 0.25 and gamma 2, the default, not overridden in the configs default.py loftr_loss.py.
• Stage-one matching. Paper: mutual nearest neighbours on the patch score matrix, then top-1 per coarse match §3.4. Code: a single argmax of the dual-softmax patch matrix fine_matching.py fine_matching.py. No mutual-nearest check appears in the fine path; the grep for "mutual" in src/ hit only the coarse matcher and the supervision code.
• Softmax at inference. Paper: drop the coarse dual softmax in the optimized model §3.3. Code: SKIP_SOFTMAX removes only the coarse softmax coarse_matching.py. The fine stage still applies a dual softmax fine_matching.py.
• Optimized threshold. Config: 25.0 eloftr_optimized.py. The released optimized outdoor evaluation passes --thr 20 outdoor_opt_auc.sh.
• Test resolution. Paper: MegaDepth longest edge 1200 §4.2. Config: 832 megadepth_test_1500.py. Released optimized outdoor script: --megasize 1152 outdoor_opt_auc.sh.
• Indoor coarse threshold. Full config: 0.2 eloftr_full.py. Released indoor evaluation: --thr 0.1 indoor_full_auc.sh.
• RANSAC repeats. Released scripts run RANSAC five times, following the RoMa protocol outdoor_opt_auc.sh indoor_full_auc.sh. The paper's main text does not mention repeats.
• Subpixel loss. Paper: L2 over the final subpixel matches against all ground-truth fine matches §3.5. Code: only pairs whose GT offset lies within the infinity-norm threshold (default 1.0) loftr_loss.py default.py.
• Stage-two details not in the paper. Code: stage two uses only the last 8 channels, with a temperature of 10 in the configs default.py eloftr_full.py. The paper names neither.
• No discrepancy found in the aggregation window (4x4) or the four rounds of self and cross attention.

Pitfalls

• Speed claims depend on settings. The "about 2.5x faster than LoFTR" claim in the abstract matches the optimized model in mixed precision (27.0 ms vs 66.2 ms) tab. 1. In FP32 the optimized model takes 35.6 ms, about 1.9 times faster tab. 1. The table also mixes precisions per row, so compare like with like.
• Optimized threshold is a raw score. The coarse threshold of 25 applies to scores after division by the temperature (0.1) and with no softmax coarse_matching.py. Reusing 0.2 from the full model is wrong, and the paper states no value at all.
• Dual softmax is still needed for training. The paper says the softmax helps learn discriminative features and can be dropped at inference §3.3. The code always computes the fine-stage dual softmax fine_matching.py.
• Feature scaling happens twice. Coarse features are divided by sqrt(C) inside coarse matching coarse_matching.py and again before the fine path loftr.py. The paper does not describe either step.
• Reparameterisation is a required step. The backbone trains multi-branch and is converted to one branch before inference §3.1 loftr.py. The paper calls the conversion lossless §3.1, so skipping it should change speed, not accuracy. This is our inference; the paper reports no such ablation.
• Aggregation details are ambiguous. The paper calls the query branch a strided depthwise convolution and the keys a max-pool eq. 3. The code adds LayerNorm after each pooling and fuses the output by concatenation with a residual MLP transformer.py. The paper says only that the upsampled output is "fused" with the input §3.2.
• Stride description in the supplement. The supplement says the first stage has stride 1 and the next three have stride 2 app. B.1. In the code the first block has stride 2 and the next stage has stride 1 repvgg.py, which still gives 1/2, 1/4 and 1/8 outputs. Follow the code for implementation.
• Input is grayscale. The first block takes one channel repvgg.py. The paper does not say this.
• Border removal and sampling. The code removes two border cells from coarse matches default.py and samples 30 percent of coarse pairs for training eloftr_full.py. The paper mentions neither.
• Empty coarse matches. If no coarse match passes, the code returns the coarse coordinates as fine matches fine_matching.py. Downstream code must not assume subpixel positions.
• Resolution changes results. The paper's resolution study says larger inputs raise accuracy and slow matching app. C. The released test scripts use sizes other than the paper's (see Paper vs code), so reproduced numbers may not match Table 1.
• Training recipe is not fully given. The paper omits the epoch count, schedule, weight decay and warm-up. The released script fills these in eloftr_outdoor.sh but at a different batch size from the paper.
• Generalisation. One model is trained on MegaDepth and reused for every dataset and task §4.1. The paper reports no per-dataset retraining.
• Known failure mode. The paper reports failures on scenes with strong repeated structure, where global context is missing app. E.
• Licence of the code. The PRL permits use free for research, but a company project must register first. Check that before reusing any code beyond citing it by path and line.

Lineage

• Builds on: LoFTR, the detector-free matcher whose attention, correlation and refinement this paper modifies §1 §3.
• Changes from LoFTR: aggregated attention on reduced tokens §3.2; RepVGG backbone §3.1; two-stage refinement §3.4; optional dual-softmax removal §3.3.
• Compared against: SuperPoint with NN, SuperGlue and LightGlue; DRC-Net, QuadTree, MatchFormer, AspanFormer, TopicFM; DKM and RoMa §4.2 tab. 1.
• Superseded by: not known from content already in this wiki.