PREDATOR: Registration of 3D Point Clouds with Low Overlap

In plain words

Two scans of the same room, taken from different spots, often show only part of the same surfaces. Lining them up is easy when most of the room is shared and hard when only a small patch is. Earlier systems were tested mostly on pairs with a lot of shared area, and they break down when the shared part is small.

The authors' idea is to make the two scans "look at each other" before picking points to match. Each scan gets a summary of its shape, and the two summaries exchange information. The network then learns which parts of each scan are likely to be in the shared area and which points are distinctive enough to match reliably. Only those points are used to find the alignment.

The paper argues that the bottleneck is not mainly the description of each point, but choosing where to look. It also introduces a new, harder test set made from pairs with little overlap.

TL;DR

PREDATOR is a pairwise 3D point-cloud registration network with a cross-attention "overlap attention" block in its bottleneck §3.3. It predicts a per-point overlap score and a matchability score, and samples interest points in proportion to their product §4.1. The authors report 59.8% registration recall on the new 3DLoMatch set (10-30% overlap) against 40.1% for the best prior method, and 89.0% on 3DMatch at 5000 samples tab. 2. Use it when your scan pairs can share little area; check the deviations in "Paper vs code" before you reimplement the loss.

Problem

Existing benchmarks evaluate only pairs with at least 30% overlap, so the field optimised for that regime §1 §3.1. Below 30% overlap, the registration recall of the best methods drops quickly §1 fig. 2. The authors trace this to feature-point sampling rather than descriptor quality: a large gain appears when interest points are drawn mostly from the shared region §1 fig. 2.

Recovering the transform requires enough overlap, defined by the ratio in eq. 1 exceeding a threshold tau. The paper treats tau > 0.3 as the prior practice and tau > 0.1 as its low-overlap regime §3.1 eq. 1.

Method

The network has three modules §3. It is a two-stream encoder-decoder with shared weights. The default backbone uses residual blocks with KPConv-style point convolutions, and the architecture is described as backbone-agnostic §3.

1. Shared encoder. Each cloud is voxel-downsampled and passed through ResNet-like blocks and strided convolutions. The output is a set of superpoints with latent features of width b §3.2. The superpoint count depends on the spatial extent of each input, so the two clouds can have different counts §3.2.
2. Overlap attention (bottleneck). Three steps §3.3:
   • A graph neural network runs separately on each cloud. Superpoints are linked to their k nearest neighbours in Euclidean space, and each update takes a max over neighbours of a learned function of the centre feature and the neighbour offsets eq. 2. The update is applied twice with separate parameters, and the outputs are combined in a final linear layer eq. 3.
   • A cross-attention block, with four parallel heads, connects every superpoint of one cloud to every superpoint of the other eq. 4 eq. 5. It is applied in both directions with the same weights. The co-contextual feature is the GNN feature plus an MLP of the query and the attention message eq. 6.
   • A second within-cloud GNN with its own parameters refines the conditioned features. Linear projections then give an overlap logit per superpoint. Soft correspondences eq. 7 weight the other cloud's overlap to give a cross-overlap score.
3. Decoder. The conditioned features are concatenated with the overlap and cross-overlap scores. The decoder uses nearest-neighbour upsampling, linear layers and encoder skip connections §3.4. It outputs a 32-dimensional descriptor per point and two separate per-point scores: overlap and matchability §3.4. The paper keeps the two apart so that a point can be unambiguous but outside the overlap, or inside the overlap with an ambiguous descriptor §3.4.

Training is end to end, with three losses against ground-truth correspondences §3.5:

• Circle loss on the descriptors, using positive radius r_p and safe radius r_s for feature-space pairs eq. 8.
• Overlap loss: binary cross-entropy on each point's overlap label, with weights inversely proportional to label frequency eq. 9 eq. 10.
• Matchability loss: binary cross-entropy against labels generated on the fly. A point is matchable if its nearest neighbour in feature space lies within r_m of its ground-truth position eq. 11 eq. 12.

The paper weights the three terms equally §3.5. It adds the matchability loss only after a few epochs, once more than 30% of interest points can be matched correctly §3.5. The code gates it on validation recall instead (see "Paper vs code").

At test time, interest points are sampled without replacement, with probability proportional to the product of overlap and matchability scores (the "prob. (om)" strategy). RANSAC then estimates the transform §4.1 tab. 1. The paper finds this sampling gives the best registration recall §4.1.

Pseudocode: overlap attention

# Overlap attention in the bottleneck  [§3.3]
# Input:  feats_p (N', b), feats_q (M', b); coords_p, coords_q
# Output: conditioned feats, overlap logits, cross-overlap logits

def overlap_attention(coords_p, coords_q, feats_p, feats_q):
    # Within-cloud GNN over k-NN graphs  [§3.3] [eq. 2-3]
    feats_p = gnn(coords_p, feats_p)
    feats_q = gnn(coords_q, feats_q)
    # Cross-attention, 4 heads, both directions  [§3.3] [eq. 4-6]
    feats_p = feats_p + cross_attention(feats_p, feats_q)
    feats_q = feats_q + cross_attention(feats_q, feats_p)
    # Second within-cloud GNN, separate weights  [§3.3]
    feats_p = gnn_second(coords_p, feats_p)
    feats_q = gnn_second(coords_q, feats_q)
    # Per-superpoint overlap logits, linear projection  [§3.3]
    overlap_p = linear(feats_p)                  # (N',)
    overlap_q = linear(feats_q)                  # (M',)
    # Soft correspondences give cross-overlap scores  [§3.3] [eq. 7]
    weights_pq = softmax(feats_p @ feats_q.T / temperature)
    weights_qp = softmax(feats_q @ feats_p.T / temperature)
    cross_p = weights_pq @ overlap_q             # (N',)
    cross_q = weights_qp @ overlap_p             # (M',)
    return feats_p, feats_q, overlap_p, overlap_q, cross_p, cross_q

Pseudocode: training loss and test-time sampling

# Training loss for one pair  [§3.5]
# Labels come from ground-truth correspondences

def training_loss(points_p, points_q, feats_p, feats_q, scores):
    # Overlap label: ground-truth match within r_o  [§3.5] [eq. 10]
    labels_overlap = overlap_labels(points_p, points_q)
    # Overlap BCE, inverse-frequency weights  [§3.5] [eq. 9]
    loss_overlap = weighted_bce(scores.overlap, labels_overlap)
    # Matchability label: feature-space NN within r_m  [§3.5] [eq. 12]
    labels_match = matchability_labels(points_p, points_q, feats_p, feats_q)
    # Matchability BCE  [§3.5] [eq. 11]
    loss_match = weighted_bce(scores.match, labels_match)
    # Circle loss, radii r_p and r_s  [§3.5] [eq. 8]
    loss_circle = circle_loss(points_p, points_q, feats_p, feats_q)
    # Paper weights the three terms equally  [§3.5]
    return loss_circle + loss_overlap + loss_match

# Test-time interest-point sampling  [§4.1]
# Code: scripts/evaluate_predator.py#L20-L36
# Input:  overlap (N,), matchability (N,), n_points
# Output: indices of n_points sampled points

def sample_interest_points(overlap, matchability, n_points):
    # Score is the product of overlap and matchability  [§4.1]
    scores = overlap * matchability
    # Sample without replacement, prob. proportional to score  [§4.1]
    probs = scores / scores.sum()
    return choice(len(scores), size=n_points, replace=False, p=probs)

Equations

Overlap ratio used to define positive pairs eq. 1:

\frac{1}{N}\,\Big|\Big\{\, i \;:\; \big\lVert \bar{T}(p_i) - \mathrm{NN}(\bar{T}(p_i), Q) \big\rVert_2 \le v \,\Big\}\Big| \;>\; \tau eq. 1

Here v is a tolerance that depends on point density, determined after voxel downsampling §3.1.

Within-cloud GNN update, applied twice eq. 2:

x_i^{(k+1)} = \max_{(i,j)\in\mathcal{E}} h_\theta\big(\mathrm{cat}[\,x_i^{(k)},\; x_j^{(k)} - x_i^{(k)}\,]\big) eq. 2

Here h_theta is a linear layer, instance normalisation and LeakyReLU, and the max is element-wise over neighbours §3.3.

GNN output eq. 3:

x_i^{\mathrm{GNN}} = h_\theta\big(\mathrm{cat}[\,x_i^{(0)},\, x_i^{(1)},\, x_i^{(2)}\,]\big) eq. 3

Keys, values and queries eq. 4:

k_j = W_k\, x_j^{\mathrm{GNN}}, \qquad v_j = W_v\, x_j^{\mathrm{GNN}}, \qquad s_i = W_s\, x_i^{\mathrm{GNN}} eq. 4

Message, with attention weights a_ij = softmax(s_i^T k_j / sqrt(b)) eq. 5:

m_{i\leftarrow} = \sum_{j:(i,j)\in\mathcal{E}} a_{ij}\, v_j eq. 5

Here E is the bipartite edge set between the two clouds, and b is the feature width. Each superpoint is connected to all superpoints of the other cloud §3.3.

Co-contextual feature eq. 6:

x_i^{\mathrm{CA}} = x_i^{\mathrm{GNN}} + \mathrm{MLP}\big(\mathrm{cat}[\,s_i,\; m_{i\leftarrow}\,]\big) eq. 6

Here the MLP is described as three fully connected layers, with instance normalisation and ReLU after the first two §3.3.

Soft correspondence and cross-overlap score eq. 7:

\tilde{o}_i^{P'} = w_i^{\top}\, o^{Q'}, \qquad w_{ij} = \mathrm{softmax}_j\!\Big(\tfrac{1}{t}\,\langle f_i^{P'},\, f_j^{Q'}\rangle\Big) eq. 7

Here t is a temperature parameter; the paper gives no value §3.3. As t tends to 0, the soft assignment becomes nearest-neighbour §3.3.

Circle loss for the P side, with beta_p = gamma (d - Delta_p), beta_n = gamma (Delta_n - d), Delta_p = 0.1, Delta_n = 1.4, and L_c = (L_c^P + L_c^Q) / 2 eq. 8:

\mathcal{L}_c^{P} = \frac{1}{n_p}\sum_{i=1}^{n_p} \log\Big[\, 1 + \sum_{j\in\mathcal{E}_p} e^{\beta_p^j\,(d_i^j - \Delta_p)} \cdot \sum_{k\in\mathcal{E}_n} e^{\beta_n^k\,(\Delta_n - d_i^k)} \Big] eq. 8

Here d is the feature-space L2 distance, E_p are positives within radius r_p, and E_n are negatives beyond radius r_s §3.5.

Overlap loss, with L_o = (L_o^P + L_o^Q) / 2 eq. 9:

\mathcal{L}_o^{P} = \frac{1}{|P|}\sum_{i=1}^{|P|} \bar{o}_{p_i}\log o_{p_i} + (1-\bar{o}_{p_i})\log(1-o_{p_i}) eq. 9

Overlap label eq. 10:

\bar{o}_{p_i} = \begin{cases} 1, & \lVert \bar{T}(p_i) - \mathrm{NN}(\bar{T}(p_i), Q) \rVert_2 < r_o \\ 0, & \text{otherwise} \end{cases} eq. 10

Matchability loss, with L_m = (L_m^P + L_m^Q) / 2 eq. 11:

\mathcal{L}_m^{P} = \frac{1}{|P|}\sum_{i=1}^{|P|} \bar{m}_{p_i}\log m_{p_i} + (1-\bar{m}_{p_i})\log(1-m_{p_i}) eq. 11

Matchability label, generated on the fly eq. 12:

\bar{m}_{p_i} = \begin{cases} 1, & \lVert \bar{T}(p_i) - \mathrm{NN}_F(p_i, Q) \rVert_2 < r_m \\ 0, & \text{otherwise} \end{cases} eq. 12

Inlier ratio and registration RMSE, as defined in the appendix app. A.1:

\mathrm{IR} = \frac{1}{|\mathcal{K}_{ij}|}\sum_{(p,q)\in\mathcal{K}_{ij}} \big[\, \lVert \bar{T}(p) - q \rVert_2 < \tau_1 \,\big], \qquad \tau_1 = 10\,\text{cm} app. A.1

\mathrm{RMSE} = \sqrt{\frac{1}{|\mathcal{H}^{*}_{ij}|}\sum_{(p,q)\in\mathcal{H}^{*}_{ij}} \lVert T(p) - q \rVert_2^2} app. A.1

A pair counts as registered when RMSE < 0.2 app. A.1.

Hyperparameters

Dataset-dependent loss and voxel settings tab. 6 app. A.3:

Here n_p is the number of negative pairs in the circle loss app. A.3, and gamma is the circle-loss scale.

Encoder and decoder widths tab. 7:

Training settings app. A.3:

Other values:

• Circle-loss margins: Delta_p = 0.1, Delta_n = 1.4 §3.5.
• Cross-attention: four heads §3.3.
• Within-cloud GNN: k = 10 nearest neighbours; bottleneck width 256; layer order self, cross, self indoor.yaml. The text gives only the head count and the paper's k; the width and layer order come from the code.
• Overlap-attention feature width: not stated in the paper text (code: 256).
• Temperature t of the soft assignment: not stated §3.3. The code learns it (see Paper vs code).
• Learning-rate warm-up: not stated.
• RANSAC settings for evaluation: not stated in the paper. Code defaults are in "Paper vs code".

Results

All numbers are author-reported on the paper's own benchmarks. Registration recall (RR) is the main metric §4.1.

3DMatch and 3DLoMatch, registration recall (%) across sample counts tab. 2:

The 3DLoMatch gain over the closest competitor is 15.5 to 19.7 percentage points across sample counts §4.1. The 3DMatch maximum of 90.6 is at 1000 samples tab. 2.

Sampling strategies on the PREDATOR model tab. 1:

Here "om" is the product of overlap and matchability scores. The authors note that top-k sampling finds the most inliers, but they cluster and do not give reliable transforms §4.1. Probabilistic sampling gives the best RR §4.1 fig. 7.

Ablation of the overlap-attention parts tab. 3. The paper does not state the sample count for this table:

Adding the overlap scores alone raises RR by 1.5 points on 3DMatch and 3.9 on 3DLoMatch §4.1. Adding the overlap-attention module to FCGF raises 3DLoMatch RR at 250 samples from 26.8 to 55.9 tab. 10 (author-reported).

ModelNet40 and ModelLoNet (RANSAC with 450 points) tab. 4. ModelLoNet has 53.6% average overlap, versus 73.5% for ModelNet §4.2:

On the full ModelNet set, RPM-Net has lower error than PREDATOR (prob. (om)) on all three metrics. In the low-overlap ModelLoNet set, PREDATOR (prob. (om)) has the lowest RRE of the four rows, and the paper highlights this §4.2 tab. 4.

odometryKITTI tab. 5. The protocol follows D3Feat, with pairs at most 10 m apart and ICP-refined ground truth §4.3:

Runtime per fragment pair in ms, averaged over 1623 3DMatch test pairs, on a GTX 1080 Ti, voxel size 2.5 cm tab. 11 app. A.7:

The paper says the data loader dominates, since neighbour indices are precomputed before the forward pass app. A.7.

Other reported results:

• Relative overlap: discarding points with predicted overlap below 0.5 almost doubles overlap on 3DLoMatch (133% increase) and raises 3DMatch overlap by more than 50% on average §4.1 fig. 6.
• Direct registration baselines: PRNet and RPM-Net did not train to reasonable results on 3DMatch. RPM-Net with an FCGF feature extractor reached 54.9% RR, against 85.1% for FCGF with RANSAC §4.1.

Paper vs code

Code: ShengyuH/OverlapPredator at commit 8c78f125fc58d62ad7d149adf1fed43ed54937e4, MIT LICENSE. The paper's abstract links it as the code release.

• Epochs and LR decay. The paper gives 30 epochs for 3DMatch and says the learning rate is decayed "by 0.05" per epoch app. A.3. The code sets 40 epochs indoor.yaml and gamma 0.95 indoor.yaml. ModelNet and KITTI use gamma 0.99 modelnet.yaml kitti.yaml. The paper's 200 and 150 epochs for those datasets match the code modelnet.yaml kitti.yaml.
• Matchability loss schedule. The paper adds it after "few epochs", once more than 30% of interest points match §3.5. The code keeps its weight at 0 and sets it to 1 only when validation feature-match recall exceeds 0.3 trainer.py. The config sets w_saliency_loss: 0.0 indoor.yaml.
• Loss weights. The paper says the three losses are weighted equally §3.5. The code uses weight 1 for the circle and overlap losses indoor.yaml, plus the gated matchability weight described above.
• Matchability coverage. Eq. 11 sums over all points of P eq. 11. The code computes the matchability BCE only on the ground-truth overlap points, the indices of matched superpoints loss.py. Non-overlap points get no matchability label.
• Cross-overlap input. Eq. 7 uses overlap scores that the text calls probabilities §3.3 eq. 7. The code takes the softmax-weighted average of the other cloud's raw bottleneck logits, before any sigmoid architectures.py. Sigmoid is applied only to the final decoder outputs architectures.py.
• Temperature. The paper gives no value for t §3.3. The code makes it learnable as exp(epsilon) + 0.03, with epsilon initialised to -5, so it starts near 0.037 architectures.py architectures.py. Features are L2-normalised before the inner product, which the paper does not state architectures.py architectures.py.
• Descriptor normalisation. The decoder's output descriptors are L2-normalised architectures.py. The paper does not say so §3.4.
• Cross-attention scale. Eq. 5 divides by sqrt(b) eq. 5. The code divides by the per-head width, since it uses four heads gcn.py gcn.py.
• Cross-attention MLP. The paper describes a three-layer MLP §3.3. The code's MLP has two linear layers, 2b to 2b to b, with norm and ReLU after the first gcn.py gcn.py. The code also concatenates the raw query feature, not W_s x gcn.py, whereas eq. 6 uses s_i eq. 6.
• Circle loss weights. The paper gives beta = gamma (d - Delta) §3.5 eq. 8. The code clips the weights at zero and detaches them loss.py. It also keeps only correspondences within r_p minus 1 mm loss.py and randomly keeps at most n_p of them loss.py.
• Overlap labels. The code's labels come from correspondences within r_o indoor.py loss.py. This matches eq. 10.
• Circle loss source. The paper footnotes the circle loss as "added to the repository after publication, not mentioned in the paper" §3.5. Treat the code as the reference for this loss.
• Table 7 first-conv width. The table's first-conv feature dimension is half the code's first_feats_dim for all three datasets tab. 7. The text does not define the column, so this may be a convention difference rather than an error.
• Evaluation RANSAC. The paper does not state RANSAC settings. The 3DMatch evaluation script calls ransac_pose_estimation with defaults: distance threshold 0.05 and ransac_n 3 evaluate_predator.py benchmark_utils.py. The KITTI tester uses distance 0.3 and ransac_n 4 tester.py.
• Thresholds. The RMSE threshold of 0.2 m app. A.1 matches the code's err2=0.2 default benchmark.py. The 10 cm inlier distance app. A.1 matches the default in get_inlier_ratio benchmark_utils.py. The FMR threshold of 0.05 matches the evaluation script's default evaluate_predator.py.

Pitfalls

• Missing minus sign in the loss equations. Eqs. 9 and 11 are written as log-likelihood sums, without a leading minus sign eq. 9 eq. 11. Minimise the negative.
• Loss gating. The matchability term is zero until the validation recall passes 0.3. A reimplementation that adds it from the start will train differently from the code trainer.py.
• Matchability on overlap points only. The code leaves non-overlap points unsupervised for matchability loss.py. This affects the sampling distribution, since the paper's sampling multiplies by matchability §4.1.
• Temperature and normalisation are not in the paper. Eq. 7 and the descriptor inner products depend on t and on whether features are normalised. The code uses a learnable t and L2 normalisation architectures.py.
• Cost is quadratic in superpoints. Cross-attention is O(n^2) in memory, where n is the superpoint count. The paper says n depends on voxel size, strided-conv count and scene extent, and that large scenes can be split into parts §4.3. Batch size is 1 because of memory app. A.3.
• Two numbers for the headline result. The arXiv abstract in our metadata (v3) says "more than 20%" and "89%". The LaTeX source abstract says "more than 15 percent points" and "90.6%". 89.0% is the 5000-sample value and 90.6% the 1000-sample value tab. 2. Cite the tables.
• Split sizes disagree in the paper. §4.1 gives 46 training, 8 validation and 8 test scenes §4.1. The appendix says the official benchmark splits 54 for training and 8 for testing app. A.2.
• Overlap-bug footnote. A bug in the official 3DMatch overlap computation puts a few pairs (under 7%) in both 3DMatch and 3DLoMatch §4.1.
• D3Feat numbers are starred. The released D3Feat code does not reproduce the paper's KITTI numbers, so D3Feat* is marked with a footnote §4.3 tab. 5.
• Sampling matters more than descriptors. Top-k sampling finds the most inliers, but its RR is below prob. (om) at every sample count on both datasets tab. 1. Choose the sampling strategy as carefully as the network.
• Timing hides the data loader. About 191 of the 271 ms overall time is data loading, which is precomputation of neighbour indices, not the network itself tab. 11 app. A.7.

Lineage

• Builds on: KPConv point convolutions for the encoder §3 §3.2; the D3Feat codebase, whose KPConv implementation the model reuses app. A.4; the Transformer multi-head attention §3.3; the cross-attention block of SuperGlue §3.3; the message-passing formulation §3.3; the dynamic graph CNN for the within-cloud GNN §3.3; and the circle loss, following D3Feat §3.5.
• Compared against: 3DSN, FCGF and D3Feat on 3DMatch and 3DLoMatch §4.1 tab. 2; RPM-Net and DCP-v2 on ModelNet40 and ModelLoNet §4.2 tab. 4; 3DFeat-Net, FCGF and D3Feat* on odometryKITTI §4.3 tab. 5.
• Superseded by: not known from content already in this wiki.