RoMa: Robust Dense Feature Matching

In plain words

Matching two photos of the same scene means finding, for many pixels in the first photo, the spot in the second photo that shows the same point. It is hard when the views differ a lot in lighting, scale, viewpoint or texture. Older matchers learn their features from scratch on a small set of 3D-labelled photos, so they overfit.

RoMa keeps a large general-purpose image model (DINOv2) frozen. Its features are robust but blurry, so RoMa adds sharper features from a small convolutional network and uses them for the final, pixel-accurate positions. The part that picks rough matches is a Transformer that scores a grid of candidate spots for each pixel, which lets it hold more than one plausible answer. The training loss treats the rough stage and the fine stage differently, because they face different kinds of uncertainty.

The authors report large gains over earlier dense matchers on hard benchmarks, most of all on WxBS.

TL;DR

RoMa is a dense two-view feature matcher that combines frozen DINOv2 coarse features, VGG19 fine features, a Gaussian-process encoder and a Transformer decoder that predicts anchor probabilities. Training uses regression-by- classification for the coarse matches and a robust Charbonnier regression for refinement. Author-reported results set new state-of-the-art numbers on IMC2022, WxBS, MegaDepth-1500, ScanNet-1500, MegaDepth-8-Scenes and InLoc. It is a good starting point if you need dense matching under strong appearance change. Reproducing it needs the paper's training data and the pitfalls below.

Problem

• Dense matchers trained with 3D supervision get specialised coarse features, but the labelled data is small, so models overfit and fail on scenes unlike the training set §1.
• Frozen backbones pretrained on ImageNet classification are weak for matching out of the box (EPE 87.6 for VGG19, 60.2 for RN50) tab. 1 §3.2.
• DINOv2 features are much more robust (robustness 85.6% vs 57.5% for RN50) but they have only stride-14 resolution, so they cannot localise a match on their own tab. 1 §3.2.
• The baseline DKM uses one non-robust regression loss for both coarse and refinement stages. The authors argue that the coarse conditional is often multimodal and the refinement conditional is locally unimodal, so they need different losses §1 §3.4.
• Assumptions: supervised dense correspondences from MVS depth (MegaDepth) or RGB-D (ScanNet) are available for training §4.2. The authors list the limited amount of supervised data and the indirect training objective as limitations [limitations, items a and b].

Method

RoMa is a coarse-to-fine dense matcher, built on DKM's structure. The steps below follow the paper's order.

1. Features. The feature encoder is split into a coarse part and a fine part, F_{\text{coarse}} and F_{\text{fine}} §3.2 eq. 1. The coarse part is DINOv2 (ViT-L/14), frozen throughout training. Only its patch tokens are used, with no class token. These tokens are linearly projected to 512 dimensions with batch norm §3.2 app. B. The fine part is a VGG19 feature pyramid taken from the layer before each 2×2 max-pool, at strides 1, 2, 4 and 8, with 64, 128, 256 and 512 channels. These are projected to 9, 64, 256 and 512 channels app. B. The authors chose VGG19 for fine features after an ablation showed it beats ResNet50 there, even though VGG19 is the weaker coarse feature §3.2 tab. 2, setups II–III.

2. Global matcher. A Gaussian-process match encoder works on the coarse features and outputs 512-dimensional embeddings, using an exponential cosine kernel §3.1 app. B. A Transformer match decoder then reads the concatenated DINOv2 projection and the GP output (1024 dimensions) app. B. It has 5 ViT blocks, 8 heads, hidden size 1024 and MLP size 4096, with no positional encodings §3.3 app. B. Per coarse pixel, it outputs K+1 logits: one per anchor for K = 64 \times 64 anchors laid out uniformly over the image grid, plus one matchability logit §3.3. The conditional match distribution is a mixture of uniform bases centred on those anchors eq. 8 §3.3.

To turn this into a warp, RoMa takes the arg-max anchor for each coarse pixel, then applies a local soft-argmax over that anchor and its four neighbours §3.3 eq. 9 is the ToWarp equation in §3.3.

3. Refiners. Five convolutional refiners work at strides {1, 2, 4, 8, 14} app. B. Each takes the stacked feature maps, a local correlation volume around the previous warp and a linear encoding of that warp. It predicts a residual warp and a certainty logit §3.1 app. B. Refinement runs from coarse to fine, and the gradients between refiners are detached. The warp is upsampled bilinearly between scales §3.1 (following DKM).

4. Loss. The total loss is a coarse term plus a sum of fine terms, with no extra weighting between them §3.4.

• Coarse term: a KL divergence to a theoretical match model reduces to a negative log-probability of the anchor closest to the ground-truth target, plus a weighted term for matchability, implemented as binary cross-entropy §3.4. The weight \lambda on the matchability term is named but its value is not stated.
• Fine terms, for scales i \in \{0,1,2,3\}: each refiner outputs the mean \mu of a generalised Charbonnier distribution with \alpha = 0.5. The loss has the form (\lVert \mu - x^B \rVert^2 + s)^{1/4} with s = 2^i c and c = 0.03 §3.4. Its gradients act like L2 locally and decay with distance fig. 4 §3.4.

Pseudocode for the inference pass:

# RoMa inference  [§3.1] [§3.2] [§3.3] [app. B]
# Input:  image_a, image_b  (3, H, W), H and W multiples of 14 [code]
# Output: warp (H_c, W_c, 2), certainty (H_c, W_c)

def match(image_a, image_b):
    # Frozen DINOv2 patch tokens, no class token  [§3.2] [eq. 7]
    coarse_a = project_512(frozen_dinov2(image_a))  # (512, H_c, W_c)
    coarse_b = project_512(frozen_dinov2(image_b))
    # Fine pyramid from VGG19 at strides 1, 2, 4, 8  [§3.2] [app. B]
    fine_a = fine_pyramid(image_a)
    fine_b = fine_pyramid(image_b)
    # Gaussian-process match encoder, 512 dimensions  [§3.1] [app. B]
    match_features = gp_encode(coarse_a, coarse_b)
    # Transformer decoder without positional encodings  [§3.3] [app. B]
    # Logits over K = 64 * 64 anchors plus one matchability logit
    anchor_logits = transformer_decode(
        coarse_a, coarse_b, match_features)
    # Coarse warp: arg-max anchor, then local soft-argmax over
    # its four neighbours  [§3.3] [eq. 9]
    warp, certainty = decode_coarse(anchor_logits)
    # Refine coarse to fine; gradients detached, warp upsampled
    # bilinearly between scales  [§3.1] [app. B]
    for scale in [14, 8, 4, 2, 1]:
        warp, certainty = refine(
            fine_a[scale], fine_b[scale], warp, certainty)
    return warp, certainty

Pseudocode for the training loss:

# Training loss  [§3.4]
# Coarse term: classification on the nearest anchor, plus
# lambda-weighted matchability term  [eq. 8] [§3.4]
def coarse_loss(anchor_probs, matchability, gt_anchor, gt_match, lam):
    # gt_anchor: index of the anchor nearest the ground-truth target
    # gt_match:  1 where a ground-truth correspondence exists
    class_term = -log(anchor_probs[gt_anchor])
    match_term = binary_cross_entropy(matchability, gt_match)
    return class_term + lam * match_term  # lam not stated

# Fine term at refinement scale i: generalised Charbonnier with
# alpha = 0.5 and s = 2^i * c, c = 0.03  [§3.4]
def refine_loss(warp_mean, gt_warp, scale_index):
    # squared distance between predicted mean and ground truth
    sq_dist = sum((warp_mean - gt_warp) ** 2, axis=-1)
    s = (2 ** scale_index) * 0.03
    # minimised form; see Pitfalls on the sign  [§3.4]
    return (sq_dist + s) ** 0.25

# Total loss: coarse term plus fine terms for i = 0..3,
# unweighted  [§3.4]
def total_loss(coarse_term, fine_warps, gt_warp, gt_scalars, lam):
    total = coarse_loss(*coarse_term, lam)
    for scale_index, warp_mean in enumerate(fine_warps):
        total = total + refine_loss(warp_mean, gt_warp, scale_index)
    return total

Key equation for the coarse conditional eq. 8 §3.3:

p_{\text{coarse},\theta}(x^B \mid x^A) = \sum_{k=1}^{K} \pi_k(x^A)\,\mathcal{B}_{m_k}

with K = 64 \times 64, \mathcal{B} uniform and m_k the anchor coordinates §3.3.

Hyperparameters

Values marked "paper" are stated in the paper. Values marked "code" appear only in the official repository at the commit in the front matter and are not in the paper.

Results

All numbers are author-reported. Setup: MegaDepth and ScanNet training data, the paper's RoMa model at 560×560, compared with the baselines the paper lists in each table §4. Ablation numbers come from a validation set the authors built from random MegaDepth pairs, retrained DKM included §4.1.

Frozen features on MegaDepth (linear layer and kernel nearest-neighbour matcher, 448×448) tab. 1 §3.2:

Ablation (validation set; 100-PCK ↓ at 1, 3, 5 px) tab. 2 §4.1:

IMC2022 (hidden test set, mAA@10 ↑) tab. 3 §4.3:

WxBS (v1.1, mAA at 10 px ↑, MAGSAC++ in OpenCV) tab. 4 §4.3:

MegaDepth-1500 (AUC @5°/10°/20° ↑) tab. 5 §4.3:

ScanNet-1500 (AUC ↑; model trained on ScanNet) tab. 6 §4.3:

MegaDepth-8-Scenes (AUC ↑) tab. 7 §4.3:

InLoc (% localised at (0.25 m, 2°) / (0.5 m, 5°) / (1.0 m, 10°) ↑, HLoc pipeline) tab. 8 §4.4:

Runtime (560×560, batch 8, RTX6000, per pair) §4.5: DKM 186.3 ms, RoMa 198.8 ms, about 7% more.

Paper vs code

Code: the official repository Parskatt/RoMa, linked from the abstract. It is MIT-licensed, copyright Johan Edstedt, who is an author. Commit 77f8d68803526dcddfd9b7a46bc76125bdc25f15, cloned shallow. Config values below are the repository's, not the paper's.

• Robust loss scale. Paper: c = 0.03 and s = 2^i c §3.4. Code: the training script sets c = 1e-4 train_roma_outdoor.py. The loss computes cs = c * scale and then reg_loss = cs**a * ((x/cs)**2 + 1)**(a/2) robust_loss.py. With a = 0.5 this equals (x^2 + (2^i c)^2)^{1/4}, so the code's s is the square of the paper's. The code's pixel thresholds use 2/512 per pixel robust_loss.py, so its coordinates are in the normalised grid units that get_gt_warp produces. The paper gives no coordinate units.
• Matchability weight. Paper: no scaling between the losses needs tuning §3.4. Code: the certainty BCE is multiplied by ce_weight = 0.01 in the coarse and fine terms train_roma_outdoor.py robust_loss.py robust_loss.py robust_loss.py robust_loss.py.
• Kernel temperature. Paper: inverse temperature 10 app. A app. B. Code: T = 0.2 roma_models.py, and the kernel computes \exp((\cos - 1)/T) matcher.py. Under that form the code's inverse temperature is 5, not 10. Check before you implement.
• Training split. Paper: the same split as DKM, pairs from MegaDepth and ScanNet with test scenes excluded §4.2. Code (outdoor script): MegaDepth train_loftr split, min_overlap 0.01 and 0.35, horizontal-flip augmentation, scene weighting with alpha = 0.75 train_roma_outdoor.py.
• Fine-level supervision mask. Not in paper. Code: at scales up to 8, fine supervision is masked to pixels whose error at the previous scale is below a local window robust_loss.py.
• Refiner naming. Paper: refiners at strides {1, 2, 4, 8, 14} app. B. Code: keys "16", "8", "4", "2", "1" roma_models.py. The "16" key holds the DINOv2 patch-14 level encoders.py, so the labels match the paper's stride-14 refiner by position, not by name.
• Refiner dimensions. Consistent. The code's roma_model gives internal dims 1377, 1137, 569, 144 and 24, the same as the paper roma_models.py app. B. The padded variant roma_model_pad defaults to 1384, 1144, 576, 144 and 24 instead roma_models.py.
• Decoder. Consistent: 5 blocks, width 1024, 8 heads, no positional encodings, MLP ratio 4 (MLP 4096) roma_models.py block.py.
• Frozen DINOv2. Consistent: the forward pass runs DINOv2 under torch.no_grad() encoders.py.
• Coarse resolution at evaluation. Not stated in the paper. The outdoor eval script builds the model with coarse_res = 672 and upsample_res = 1344 eval_roma_outdoor.py. The indoor eval uses "medium" (560) roma_indoor.py.
• Certainty attenuation. Not in paper. attenuate_cert subtracts the coarse certainty from the finest certainty when set matcher.py. It defaults to True in roma_model roma_models.py and is False in the training script train_roma_outdoor.py. I did not trace how the flag changes the final output.
• Training script status. The script warns it is not tested train_roma_outdoor.py. As written, get_model passes use_vgg=True to CNNandDinov2 train_roma_outdoor.py, whose constructor does not take that argument encoders.py. It would raise a TypeError unless the call is changed.
• Indoor model. The indoor script interleaves MegaDepth and ScanNet loaders in training roma_indoor.py. The paper says only that the ScanNet-evaluated model was trained on the ScanNet training set §4.2.

Pitfalls

• Version. This card follows arXiv v2 (LaTeX source, CVPR 2024 template, updated 2023-12-11) [main.tex]. The paper's appendix is the source of the architecture details app. B.
• Sign inconsistency in the paper. The log-likelihood is written as -(\lVert\mu - x\rVert^2 + s)^{1/4} §3.4, and the KL divergence then writes the same expression with its own minus sign inside the negative log §3.4. Read literally, the two lines disagree. The minimised form is the positive term +(\lVert\mu - x\rVert^2 + s)^{1/4}, as in the code robust_loss.py.
• Charbonnier constant and units. Paper: c = 0.03, no coordinate units. Code: c = 10^{-4} on normalised grid coordinates, with the squared scale term. Pick one convention and test it on a held-out pair set. Don't copy one constant over the other.
• Coarse anchors and matchability. Per coarse pixel the decoder emits 4097 logits (64×64 + 1). At 560×560 the coarse grid is 40×40 (560/14), so each pair carries about 6.6 million coarse logits. The paper does not give the memory cost. The paper gives no runtime or memory figure beyond the 7% comparison §4.5.
• Balanced sampling is under-specified. The paper says only "balanced sampling" with 10,000 matches §4.3 app. F, following DKM. The code adds the following settings: certainties above 0.05 are set to 1 matcher.py; an expansion factor of 4 before resampling matcher.py; a KDE with standard deviation 0.1 matcher.py; and a minimum of about 10 neighbours matcher.py. Use the code's values if you need the reported numbers.
• Unstated training details. Not in the paper: the fine encoder's initialisation (the code uses ImageNet VGG19-BN); batch-norm handling; optimizer; schedule; number of steps; gradient clipping; the value of \lambda; and the refiner kernel size. Each one can change the result. The paper says only that the training setup follows DKM §4.2.
• Training split. The paper says DKM's split §4.2. The released training script uses LoFTR's split with different overlap windows and horizontal flips train_roma_outdoor.py. Expect different numbers if you use the wrong split.
• Ablation validation scenes. The paper says the validation pairs come from MegaDepth scenes "[0015,0022]" §4.1. This could be a range that includes the MegaDepth-1500 test scenes 0015 and 0022 §4.3. The paper does not say how that overlap is handled. Ablation numbers may not be clean held-out results.
• Headline percentages. The 36% on WxBS is the relative mAA gain over DKM (80.1 vs 58.9) tab. 4. The 26% on IMC2022 is a relative error reduction against ASpanFormer, the best prior entry at 83.8 tab. 3 §4.3. Against DKM (83.1), the reduction in error would be about 29% (arithmetic on the table).
• Supervision. Training needs dense supervised correspondences from MVS depth or RGB-D. The authors name this as a limitation [limitations, item a]. The loss approximates the KL term with discrete known correspondences §3.4.
• Joint training. Coarse matcher and refiners are trained jointly, but gradients are detached between them, so two-stage training is possible §4.2.
• Ablation baseline. The DKM row in the ablation is retrained by the authors, not the published DKM model §4.1 tab. 2.
• Figures. Descriptions only. Fig. 1 is a teaser, Fig. 2 the method overview, Fig. 3 the scale-space motivation, Fig. 4 the Charbonnier gradient shape §1 §3.4. Appendix figures show frozen-feature matches and WxBS qualitative comparisons app. A app. C.

Lineage

• Builds on: DKM (Edstedt et al., CVPR 2023; "DKM: Dense Kernelized Feature Matching for Geometry Estimation" in the bibliography). RoMa keeps DKM's Gaussian-process encoder, convolutional refiners, training setup and balanced sampling, and changes the coarse encoder, coarse decoder, coarse loss and refinement loss §3.1 §4.2 §4.3. It does not have an arXiv link in this wiki yet.
• Uses DINOv2 (Oquab et al., 2023) as the frozen coarse encoder §3.2.
• Uses the generalised Charbonnier distribution from Barron (2019) for the refinement loss §3.4.
• Compared against: DKM, LoFTR, ASpanFormer, PDC-Net+, CasMTR, PMatch, ASTR, PATS, LightGlue, SuperGlue, SiLK, MatchFormer, QuadTree, DISK, SuperPoint [tabs. 3–8].
• Superseded by: not stated in the paper or the repository at this commit.