Dense correspondence

Overview

Dense correspondence gives, for a pixel in a source image, the pixel in a target image that shows the same point. Reconstruction, tracking, segmentation and editing all need it 2306.03881 §1. The usual route is a network trained on annotated pairs, which are costly to collect 2306.03881 §2. Optical flow, rectified stereo and posed-image depth ask the same matching question, and one model can answer all three 2211.05783 §1. Inside an editing tool, the same question appears as tracking: a handle the user placed must be found again after each change to the image 2305.10973 §3.3.

Points on 3D surfaces raise the same question. NDF matches query points of a rigid body to a demonstrated pose through learned descriptors 2112.05124 §II-B. BEHAVE predicts, for points near a person and an object, their correspondence on a standard body model, and fits both models to them 2204.06950 §4.1. Here "correspondence" means a point on a model surface, not a pixel in another image.

The image matchers share one skeleton: compute a feature for every pixel, score every candidate in the other image, then turn the scores into a match. The variants differ in where the features come from and how the scores become a match. DIFT takes the nearest neighbour of features read from a pretrained diffusion denoiser 2306.03881 §3. UniMatch takes a softmax-weighted average over all target pixels, with no learned matching parameters 2211.05783 §3.1. RoMa predicts a distribution over a 64 × 64 anchor grid for each coarse pixel, then refines the warp 2305.15404 §3.3. DragGAN takes the nearest neighbour, but only inside a small square around the handle, using features of a StyleGAN2 generator 2305.10973 §3.3.

The baselines in the same tables are self-supervised image encoders (DINO, OpenCLIP), used without fine-tuning, and supervised or weakly supervised matchers 2306.03881 §5.1 tab. 1. They are comparison points here, not variants.

Variants

Diffusion features on a Stable Diffusion backbone (DIFT_sd)

A real image is noised to level t and passed once through the denoiser. The output of one upsampling block is kept as the feature map. The backbone is Stable Diffusion 2-1, trained on LAION 2306.03881 §5.1. The paper describes noising the image 2306.03881 §4.2; the code encodes it with the VAE and noises the latent instead dift_sd.py.

On SPair-71k this is the strongest row without labels: 52.9 PCK per image and 59.5 per point 2306.03881 tab. 1 tab. 2. Its settings for the semantic tasks are t = 261 and block 1, at 768 × 768 2306.03881 app. C. Card: Emergent Correspondence from Image Diffusion.

Diffusion features on an ADM backbone (DIFT_adm)

The same procedure with the Ablated Diffusion Model, trained on ImageNet without labels 2306.03881 §5.1. Its settings for the semantic tasks are t = 101 and block 4, at 512 × 512 2306.03881 app. C. It is the stronger option on video and is weaker on SPair-71k: 46.3 PCK per image and 52.0 per point 2306.03881 tab. 1 tab. 2.

The paper chooses t and n by grid search per task 2306.03881 §4.2 app. C. For SPair-71k the search is on that set, and the same values are reused for PF-WILLOW and CUB 2306.03881 §5.2.

Label propagation for video

For video, a label propagation scheme passes labels from earlier frames to later ones. The propagation uses a softmax temperature, a top-k filter and a spatial radius, with a set number of previous frames 2306.03881 §6.2. The DAVIS settings are 28 previous frames and a radius of 15 pixels [arxiv:2306.03881 app. C]. The paper reports that features alone do not reproduce the video numbers; the propagation is part of the method 2306.03881 §6.2.

Transformer matcher for flow, stereo and depth (UniMatch)

A weight-shared CNN gives 1/8-resolution features for each image. Six Transformer blocks, each with self-attention, cross-attention and a feed-forward layer, then enhance both feature maps, with fixed sine and cosine positions 2211.05783 §3.2. Attention runs in local windows, shifted by half a window, not over all pixels 2211.05783 tab. 2.

The matching layer has no parameters. Every pixel is correlated with the other view, a softmax over all targets gives a distribution, and its expected coordinate gives flow, disparity or depth 2211.05783 §3.1 eq. 1–8. The feature network and the propagation layer are shared across tasks; only this matching step is task-specific 2211.05783 §3.1. Pixels with no clear match, such as occluded ones, take values from a self-attention layer over the features 2211.05783 §3.3. Optional refinements add a 1/4-resolution matching step and RAFT-style local iterations. The final models use 6 local steps for flow, 3 for stereo and 1 at 1/8 for depth 2211.05783 §3.4.2 §4.5.

On Sintel, GMFlow with one refinement beats RAFT with 31 refinements on training EPE: 1.08 against 1.41 on clean and 2.48 against 2.69 on final. The times are 151 ms against 170 ms on a V100 at 436 × 1024 2211.05783 tab. 3. For stereo, 1D horizontal cross-attention is faster and more accurate than 2D: Things EPE 1.22 at 50 ms against 1.25 at 61 ms 2211.05783 §3.2 tab. 6.

Starting from a flow model helps the other tasks. For stereo on Things, EPE with no fine-tuning is 2.58 from flow and 75.24 from random initialisation; after 100K steps it is 1.00 against 1.11. For depth on ScanNet, Abs Rel with no fine-tuning is 0.198 against 0.536, and 0.063 against 0.069 after 100K steps 2211.05783 tab. 10. The paper finds no gain in the other direction 2211.05783 §4.4. Card: Unifying Flow, Stereo and Depth Estimation.

Frozen foundation features with a coarse-to-fine matcher (RoMa)

RoMa is a coarse-to-fine matcher built on DKM's structure 2305.15404 §3.1. Its coarse features are the patch tokens of a frozen DINOv2 ViT-L/14, projected to 512 channels 2305.15404 §3.2 app. B. Its fine features come from a VGG19 pyramid at strides 1, 2, 4 and 8 2305.15404 app. B. The authors chose VGG19 for this role after an ablation 2305.15404 §3.2 tab. 2.

A Gaussian-process match encoder and a 5-block Transformer decoder with no positional encodings then predict, for each coarse pixel, logits over 64 × 64 anchors plus a matchability logit 2305.15404 §3.3 app. B. Five convolutional refiners at strides 1, 2, 4, 8 and 14 move the warp from coarse to fine, with gradients detached between them 2305.15404 §3.1 app. B.

The coarse stage is trained as classification on the anchor nearest the target, plus a matchability term. The fine stage uses a generalised Charbonnier loss with α = 0.5. The two are summed with no extra weights [arxiv:2305.15404 §3.4]. Training needs dense supervised correspondences from MVS depth (MegaDepth) or RGB-D (ScanNet) 2305.15404 §4.2.

Author-reported results. On WxBS, mAA at 10 px is 80.1, against 58.9 for DKM and 55.4 for LoFTR 2305.15404 tab. 4. On the IMC2022 hidden test set, mAA at 10 is 88.0, against 83.1 for DKM and 83.8 for ASpanFormer 2305.15404 tab. 3. On MegaDepth-1500, AUC at 5° is 62.6, against 60.4 for DKM 2305.15404 tab. 5. In the ablation, the DKM row is retrained by the authors, not the published model 2305.15404 §4.1 tab. 2. The per-pair runtime at 560 × 560 is 198.8 ms against 186.3 ms for DKM, on an RTX6000 2305.15404 §4.5. Card: RoMa: Robust Dense Feature Matching.

Nearest-neighbour tracking in GAN features (DragGAN)

DragGAN edits a StyleGAN2 image by moving user-chosen handle points to target points. Each iteration takes one optimiser step on the latent code. Then each handle moves to the pixel inside a square around it whose block-6 feature is nearest, in L1 distance, to the feature the handle had at the start 2305.10973 §3.3 eq. 2. The square's half-width is 12 pixels at 512 px 2305.10973 §3.4. No tracker network is used, and the same features also drive the motion 2305.10973 §3.2.

Swapping the tracker changes the result. With RAFT tracking, the one-point face-landmark distance is 13.43, and with PIPs it is 2.98, against 2.44 for the full method 2305.10973 tab. 1. The authors report that the other trackers drift off the handle 2305.10973 §4.1 fig. 6. These numbers measure the whole edit on face landmarks, not the tracker alone 2305.10973 §4.2. Card: Drag Your GAN.

Point descriptor fields for pose search (NDF)

NDF gives each 3D point a descriptor. The descriptor is the concatenated activations of an occupancy network, so the network is trained by 3D reconstruction and needs no keypoint labels 2112.05124 §II-A eq. 4. Vector Neurons make the descriptor SE(3)-equivariant: rotating and moving the object moves the descriptors with it 2112.05124 §II-A eq. 5–7. A rigid body is represented by a set of query points, and its pose descriptor is the set of their descriptors 2112.05124 §II-B eq. 10. To find a pose, an optimiser starts from random poses and minimises the L1 distance to the demonstration's descriptor 2112.05124 §II-B eq. 11.

The paper trains one model across mugs, bowls and bottles on 100,000 simulated objects 2112.05124 §III. Author-reported simulation results: with ten demonstrations, overall success on unseen upright objects is 0.88 for mugs, 0.91 for bowls and 0.87 for bottles, and 0.58 to 0.78 for arbitrary poses. The Dense Object Nets baseline reaches 0.00 to 0.45 on the same tasks 2112.05124 tab. I. With one demonstration on upright mugs, overall success is 0.46, against 0.32 for Dense Object Nets 2112.05124 tab. IV. Real-robot results are qualitative only 2112.05124 §IV-C. Card: Neural Descriptor Fields.

Body and object correspondence from multi-view depth (BEHAVE)

BEHAVE tracks a person and a rigid object with a known template mesh, frame by frame, from four calibrated RGBD cameras 2204.06950 §3 §4. A voxel-based network predicts, at each sampled 3D point, the unsigned distance to the human and to the object, a correspondence onto the SMPL body model, and the object orientation 2204.06950 §4.1 §4.2. The fit combines a body term, an object term and a contact term that pulls touching object vertices onto their predicted body points 2204.06950 eq. 1 eq. 5.

Author-reported: SMPL vertex error is 4.99 cm and object vertex error 21.20 cm. The single-image PHOSA baseline gets 13.73 cm and 34.73 cm [arxiv:2204.06950 tab. 2]. Removing the contact prediction leaves SMPL error at 4.96 cm and object error at 21.28 cm, so the authors describe the contact term's effect as qualitative 2204.06950 tab. 3 §5.4. The reference fits are pseudo-ground-truth, so these errors are relative to them 2204.06950 §3. Card: BEHAVE.

When to use which

Timings are in the paper's appendix: DIFT_sd takes 203 ms per 768 × 768 image on one A6000 and DIFT_adm 110 ms per 512 × 512 image. The DINO and OpenCLIP times are at other sizes, so treat the comparison as the paper's setup only 2306.03881 app. B.

Implementation sketch

The DIFT core is one denoiser pass per noise copy, then nearest-neighbour matching. The UniMatch core is a softmax over all target pixels, with no learned parameters, so both can be written in a few lines. The DragGAN tracker is the DIFT matcher restricted to a square window around each handle, with an L1 distance 2305.10973 §3.3 eq. 2. Its pseudocode is on the card.

# Extract one feature map from a real image  [arxiv:2306.03881 §4.2] [app. C]
# Input:  image in [-1, 1]; time step t; block index n; copies K
# Output: feature map (C, h, w)

def extract_features(image, t, n, copies):
    # Each copy gets its own noise draw, so repeat the image  [app. C]
    batch = repeat(image, copies)
    # Noise every copy to level t with the forward process  [§4.2]
    noisy = add_noise(batch, t)
    # One denoiser pass; keep the output of upsampling block n  [app. C]
    acts = unet_up_blocks(noisy, t)[n]
    # Average over copies to get one feature map  [§4.2]
    return mean(acts, axis=0)

# Match each source point to the target image  [§3] [eq. 1]
# Input:  source and target feature maps; source points
# Output: matched target point for each source point

def match(src_feat, tgt_feat, points):
    # Read the source feature at each point, bilinear interpolation  [§3]
    queries = sample_bilinear(src_feat, points)
    # Cosine similarity with every target location  [§3]
    scores = cosine(tgt_feat, queries)
    # Nearest neighbour is the highest-scoring target location  [eq. 1]
    return argmax(scores, over=target_locations)

# Soft matching for flow, stereo or depth  [arxiv:2211.05783 §3.1] [eq. 1-4]
# Input:  features of both views (C, H, W); pixel grid (H, W, 2)
# Output: flow (H, W, 2) at 1/8 resolution

def soft_match_flow(feat0, feat1, grid):
    # Correlate every pixel of view 0 with every pixel of view 1  [eq. 1]
    corr = flatten(feat0).T @ flatten(feat1) / sqrt(C)   # (HW, HW)
    # Softmax over all targets  [eq. 2]
    prob = softmax(corr, axis=-1)
    # Expected target coordinate, a weighted average  [eq. 3]
    expected = prob @ flatten(grid)                       # (HW, 2)
    # Flow is expected target minus own coordinate  [eq. 4]
    return reshape(expected - flatten(grid), (H, W, 2))

Practical notes for DIFT, from the paper and its code:

• Noise copies. The default is eight copies. Each copy is a full denoiser pass 2306.03881 app. B. Dropping the averaging costs DIFT_sd 1.6 points per point on SPair-71k (59.5 to 57.9) 2306.03881 app. B.
• Prompts. For Stable Diffusion, SPair-71k uses "a photo of a [class]" and CUB uses a bird prompt. An empty prompt costs about 2 points on SPair-71k for DIFT_sd (59.5 to 57.6) 2306.03881 app. C.
• Output grid. Features are at 1/8 or 1/16 of the input size. The code upsamples the feature map to image size before matching, so matches land on an image-size grid while the features stay coarse 2306.03881 app. C eval_spair.py.
• Keypoint sampling. Bilinear sampling conventions differ across frameworks. Check the align-corners default before porting the code eval_hpatches.py.
• Seeds. Each run draws new noise, so the features change between runs. Fix the seed when outputs must be reproducible 2306.03881 app. B.

Practical notes for UniMatch:

• Propagation projections. The paper's propagation layer has no learned projections 2211.05783 §3.3. The released code applies learned query and key projections before the dot product attention.py attention.py. A code comment says they were kept so the released models need not be retrained. Implementing the paper's version will not reproduce the released weights.
• Stereo variant. The released stereo scripts use windowed, shifted 1D cross-attention, not the full-width 1D variant of the paper gmstereo_scale2_regrefine3_train.sh transformer.py.
• Global matching is quadratic. A 384 × 512 training crop gives a 48 × 64 grid, so 3,072 positions and about 9.4 M scores per image pair gmflow_scale2_train.sh matching.py. The paper does not quantify the memory.
• Flow loss exclusion. The code drops pixels with ground-truth flow of 400 px or more from the loss flow_loss.py. The paper does not say this.

Practical notes for RoMa:

• Charbonnier scale. The paper gives c = 0.03 with no coordinate units 2305.15404 §3.4. The training script sets c = 1e-4 on normalised grid coordinates train_roma_outdoor.py robust_loss.py. Pick one convention and test it on held-out pairs; do not copy one constant over the other.
• Kernel temperature. The paper gives an inverse temperature of 10 2305.15404 app. A app. B. The code sets T = 0.2 [code: romatch/models/model_zoo/roma_models.py#L153], which is an inverse temperature of 5 in the code's form matcher.py.
• Coarse memory. The decoder emits 4,097 logits per coarse pixel. At 560 × 560 the coarse grid is 40 × 40, so each pair carries about 6.6 M coarse logits 2305.15404 §4.2 app. B. The paper gives no memory figure.
• Training script. The script warns that it is untested. As written it passes use_vgg=True to a constructor that does not take it, so it raises a TypeError train_roma_outdoor.py encoders.py.

Practical notes for NDF, from the paper and its code:

• Descriptor normalisation. The paper's descriptor is a plain concatenation 2112.05124 eq. 4. The released code L2-normalises it per query point when features are returned, and the paper does not say whether its numbers used this vnn_occupancy_net_pointnet_dgcnn.py.
• Pose search is under-specified. The number of random starts, the number of steps and the learning rate are not stated in the paper 2112.05124 §II-B. The released code uses 10 starts and 250 steps by default optimizer.py. Results depend on these settings, and the search is non-convex.
• Scale unit. Query-point scale is reported from 0.1 to 10.0, and the unit is not defined 2112.05124 tab. III. The code multiplies inputs by 10 before the network vnn_occupancy_net_pointnet_dgcnn.py, so the same scale value means something different there.

Practical notes for BEHAVE:

• Loss weights are missing. The three terms of the fit 2204.06950 eq. 1 and the regularisers have no stated weights, so an implementation has to pick them 2204.06950 eq. 2.
• Contact threshold is missing. The paper uses the same symbol ε for the orientation filter (2 cm), for the network contact test, and for the dataset's 2 cm label. It gives no value for the network contact test 2204.06950 §3 §4.2 §4.3.
• Orientation convention. The nine network outputs are reshaped to a 3 × 3 matrix, but the paper does not say row-major or column-major, and it does not say how a reflection is avoided 2204.06950 §4.2.

Contradictions

Within the DIFT paper, the text and the tables describe the same SPair-71k result in different ways. The introduction says DIFT_sd stays on par with the best supervised method 2306.03881 §1. The per-image table puts it 2.4 points below SCorrSAN 2306.03881 tab. 1. The paper also reports that DIFT_sd beats the best supervised method in 9 of 18 categories 2306.03881 §5.2.

The DIFT paper and its code disagree on several points. The paper defines PCK per image as the average of per-image values 2306.03881 §5.2. The code averages one value per image pair eval_spair.py. The paper's table uses the code's version, so per-pair and per-picture numbers should not be compared. The paper's DAVIS setting uses 28 previous frames [arxiv:2306.03881 app. C], while the code caps the context at 10 frames for one video eval_davis.py.

The UniMatch paper gives two sizes for GMDepth: 7.3 M parameters and 40 ms in one comparison 2211.05783 tab. 8, and 4.7 M parameters and 17 ms in another 2211.05783 tab. 9. The paper does not say which configuration each uses.

Across the six papers, no benchmark and metric pair is reported by more than one card. RoMa reports pose AUC on ScanNet and UniMatch reports depth on ScanNet, which are different tasks. DragGAN reports landmark distance on face edits, which none of the others measure. NDF reports simulated grasp and place success, and BEHAVE reports body and object vertex error on its own dataset. So no numerical conflict between the papers is known, and their numbers should not be read as a ranking across families.

NDF's introduction says overall success is above 85 per cent. That holds for upright objects only; with arbitrary poses the table gives 0.58 to 0.78 2112.05124 §I tab. I. Quote the table, not the headline.

Papers

• Emergent Correspondence from Image Diffusion: training-free correspondence from a pretrained diffusion denoiser, with per-task settings and results on SPair-71k, PF-WILLOW, CUB, HPatches and DAVIS.
• Unifying Flow, Stereo and Depth Estimation: one Transformer matcher with parameter-free softmax matching for optical flow, rectified stereo and posed-image depth, with cross-task transfer results.
• RoMa: Robust Dense Feature Matching: a coarse-to-fine matcher on frozen DINOv2 features with a Transformer anchor decoder and robust refinement, with results on WxBS, IMC2022 and MegaDepth.
• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold: point-based GAN editing whose tracker is a windowed nearest-neighbour search in StyleGAN2 features, with face-landmark and reconstruction results.
• Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation: equivariant point descriptors from an occupancy network, used for few-shot pose search, with simulated pick-and-place results.
• BEHAVE: Dataset and Method for Tracking Human Object Interactions: per-frame tracking of a body and a known object from multi-view depth, with predicted body correspondences and contact, and a non-commercial dataset.