Deep Global Registration

In plain words

Two 3D scans of the same room, taken from different places, overlap only partly. To stitch them into one model you must find the single rigid move (a turn plus a shift) that lines them up. A simple method guesses which points match between the scans, but many of those guesses are wrong, and a few bad matches can throw the whole alignment off.

This paper's idea is to score each guessed match before trusting it. A network looks at the whole set of guesses together, treats each guess as a point in a six-number space (the coordinates of both ends), and learns which ones sit on a consistent geometric shape. Good matches get a high weight, bad ones a low weight. A weighted fit then computes the move in one closed-form step, and a final gradient-based polish makes it more precise.

The method is aimed at real, messy scans with noise and little overlap, where earlier learned methods failed. When the weights say too little of the scans agree, the system falls back to a slower but sturdier classical search.

TL;DR

• Three modules: a 6-D convolutional network that predicts inlier probabilities for correspondences §3.2, a differentiable Weighted Procrustes solver for the pose §3.3, and a robust SE(3) optimiser that refines the pose §4.
• Pairwise registration on 3DMatch: 91.3% recall, TE 7.34 cm, RE 2.43° at 1.21 s, versus 74.9% for the best classical baseline in the table tab. 1. Author-reported.
• Requires voxel-downsampled point clouds and a GPU for the network. A safeguard RANSAC path handles low-overlap pairs.
• The paper's text leaves several values open (threshold, learning rate, iteration count, loss weights). The official code fixes them; see Paper vs code.

Problem

Pairwise registration aligns two partial scans. Accuracy and robustness here set the quality of a whole reconstruction §3. Classical global registration pairs feature matches with robust estimators such as RANSAC or FGR §1 tab. 1. Earlier end-to-end learned methods use globally pooled features or a singly stochastic matrix. The paper argues that these lose local geometry and that the stochastic matrix assumes every point has a counterpart, which fails for partially overlapping scans §1, §5.1.

Assumptions: the scans are real RGB-D or LiDAR data at a known or pre-chosen voxel size (5 cm indoors, 30 cm outdoors) §5.1, §5.3. Ground-truth transforms come from RGB-D reconstruction pipelines §5.1.

Method

The pipeline has four stages.

1. Downsample. Each scan is voxel-downsampled so that one point remains per voxel §5.1. The paper uses 5 cm for indoor scans §5.1 and 30 cm for KITTI §5.3.
2. Features and putative matches. Each point gets a feature from FCGF, a fully convolutional descriptor of 16 to 32 dimensions §3.1. Each point in the first scan is matched to its nearest neighbour in feature space §3.2.
3. Inlier probabilities. Each match (x_i, y_j) is a point in six-dimensional space, [x_i; y_j] in R^6. A U-Net-shaped 6-D convolutional network with residual blocks predicts an inlier probability for every match §3.2 fig. 2. Training uses a binary cross-entropy loss over all matches eq. 1.
4. Pose. Weights below a clip threshold are set to zero, and the rest go into a Weighted Procrustes solve. This is the closed-form minimiser of the weighted squared error between transformed first-scan points and second-scan points §3.3 eq. 2–4. The solution is differentiable through the weights, so the paper also shows an end-to-end loss on the rotation and translation §3.3 eq. 5–7.
5. Robust refinement. The rotation is parameterised by six numbers using the continuous 6-D representation of Zhou et al., with a Gram–Schmidt-like map to a rotation matrix §4.1 eq. 8. The pose is refined by gradient descent on a weighted robust loss (Huber) over the fixed set of matches. The matches are not recomputed in the inner loop §4 §4.2 eq. 9.
6. Safeguard. The fraction of weight mass that survives the clip is a check on whether the solve is stable. If the fraction is below τ_s = 0.05, the system falls back to RANSAC or branch-and-bound §4 alg. 1.

Pseudocode of the whole pipeline follows alg. 1:

# Pairwise registration  [alg. 1]
# Input:  scans X (N_x, 3), Y (N_y, 3)
# Output: rotation R (3, 3), translation t (3,)

def register(points_x, points_y):
    # Per-point features from a fully convolutional network  [§3.1]
    feats_x = feature_net(points_x)              # (N_x, D), D = 16 to 32
    feats_y = feature_net(points_y)              # (N_y, D)

    # Putative matches: nearest neighbour in feature space  [§3.2]
    matches = nearest_neighbour(feats_x, feats_y)   # (M, 2) index pairs

    # Inlier probability per match, from the 6-D point (x_i, y_j)  [§3.2]
    probs = inlier_convnet(points_x, points_y, matches)   # (M,)

    # Zero out weak weights  [§4.2]
    weights = where(probs > clip_thresh, probs, 0.0)      # (M,)

    # Stability check: surviving weight mass against tau_s  [§4]
    valid_fraction = sum(weights) / len(matches)
    if valid_fraction < tau_s:
        # Fall back to RANSAC or branch-and-bound  [§4] [alg. 1]
        return safeguard_search(points_x, points_y, matches)

    # Closed-form initial pose  [§3.3]
    rot, trans = weighted_procrustes(points_x, points_y, matches, weights)

    # Robust refinement on SE(3), matches and weights held fixed  [§4.2]
    rot, trans = refine_pose(rot, trans, points_x, points_y, matches, weights)
    return rot, trans

The closed-form step is the weighted Procrustes solve. The paper states it as a theorem about the rotation via an SVD and a centring matrix §3.3, Theorem 1. The official code uses the standard centred form, which should give the same rotation. Weights are normalised to sum to one registration.py.

# Weighted Procrustes for SE(3)  [§3.3] [eq. 3] [eq. 4]
# Input:  src (M, 3), dst (M, 3), weights (M,) with weights >= 0
# Output: rotation (3, 3), translation (3,)

def weighted_procrustes(src, dst, weights):
    # Normalise weights to sum to one  [§3.3]
    weights = weights / (sum(weights) + eps)     # (M,)

    # Weighted centroids of both point sets  [§3.3]
    centroid_src = sum(weights * src)            # (3,)
    centroid_dst = sum(weights * dst)            # (3,)

    # Weighted cross-covariance of the centred points  [§3.3]
    cross = (dst - centroid_dst).T @ (weights * (src - centroid_src))  # (3, 3)

    # Singular value decomposition  [§3.3] [theorem 1]
    left, _, right_t = svd(cross)

    # Flip the last axis if the determinant is negative (no reflections)  [§3.3]
    sign = det(left) * det(right_t)
    rotation = left @ diag(1, 1, sign) @ right_t

    # Translation maps the source centroid to the target centroid  [§3.3]
    translation = centroid_dst - rotation @ centroid_src
    return rotation, translation

The 6-D rotation representation is built from two vectors in R^3. The first is normalised, the second is orthogonalised against it and normalised, and the third is their cross product §4.1 eq. 8:

# Continuous 6-D rotation representation  [§4.1] [eq. 8]
# Input:  a1 (3,), a2 (3,)
# Output: rotation matrix (3, 3)

def rotation_from_6d(a1, a2):
    # Normalise the first column  [§4.1]
    col1 = a1 / norm(a1)                         # (3,)

    # Remove the component along col1 from a2, then normalise  [§4.1]
    col2 = a2 - dot(col1, a2) * col1             # (3,)
    col2 = col2 / norm(col2)                     # (3,)

    # The third column completes a right-handed frame  [§4.1]
    col3 = cross(col1, col2)                     # (3,)
    return stack_columns(col1, col2, col3)       # (3, 3)

The refinement minimises a robust energy over the rotation parameters and the translation. Each match contributes its Huber residual, scaled by its weight §4.2 eq. 9:

# Robust SE(3) refinement  [§4.2] [eq. 9]
# Input:  initial rot (3, 3), trans (3,), matches, weights (M,)
# Output: refined rot (3, 3), trans (3,)

def refine_pose(rot, trans, src, dst, matches, weights):
    # Start from the Procrustes pose; the first two columns give a1, a2  [§4.1]
    a1, a2 = rot[:, 0], rot[:, 1]

    for step in range(max_iter):   # max_iter is not stated in the paper
        # Residuals of the matched points under the current pose  [§4.2]
        rotation = rotation_from_6d(a1, a2)
        moved = src[matches[:, 0]] @ rotation.T + trans
        residual = dst[matches[:, 1]] - moved

        # Weighted Huber loss over all matches  [§4.2]
        loss = sum(weights * huber(residual)) / sum(weights)

        # First-order update of a1, a2 and trans  [§4.2]
        # The paper allows SGD, Adam or higher-order methods; code uses Adam
        a1, a2, trans = gradient_step(loss, a1, a2, trans)
    return rotation_from_6d(a1, a2), trans

Figure 1 is a teaser on a 3DMatch pair where RANSAC, FGR and DCP fail on the harder pair fig. 1. Figure 3 shows the method's global registration on all eight 3DMatch test scenes fig. 3. Figure 4 plots recall against rotation and translation thresholds fig. 4. Figure 5 breaks recall, TE and RE down per scene fig. 5.

Hyperparameters

Values in the paper are cited to the paper. Values marked "code" appear only in the official code, at the commit in the front matter. The paper does not state them.

Results

Author-reported. Setup: 3DMatch test set (8 scenes), pairwise registration, 5 cm voxels, Intel i7-7700 CPU and GTX 1080Ti GPU; learned baselines trained on the 3DMatch training split §5.1 tab. 1. Time includes feature extraction tab. 1.

Pairwise registration, 3DMatch tab. 1

Source for every row: tab. 1. The paper gives the recall curves for varying thresholds in fig. 4 and per-scene values in fig. 5.

Multi-way registration, Augmented ICL-NUIM, ATE in cm tab. 2

Source: tab. 2. The multi-way pipeline is the open-source one of Zhou et al., with its pairwise stage replaced by this method §5.2. The caption notes that BAD-SLAM's loop closure succeeded only in Living room 2 tab. 2.

Outdoor LiDAR, KITTI test split tab. 3

Source: tab. 3. Thresholds 0.6 m and 5° tab. 3. The paper uses voxel size 30 cm §5.3.

Paper vs code

The official code is at chrischoy/DeepGlobalRegistration, MIT licence, commit 1f4871d. The paper links it in the introduction §1. Differences found:

• Weight clipping is applied to probabilities, not logits. The paper defines φ(w) = I[w > τ]·w and calls the network outputs "logit scores" §4.2. The code applies the sigmoid first, then sets every probability below 0.05 to zero, and the same is done in training and validation deep_global_registration.py trainer.py trainer.py. The paper does not state τ.
• Safeguard threshold has a floor. The paper uses τ_s = 0.05 on the ratio of surviving weight to the number of correspondences §4. The code compares the sum of clipped weights against max(200, 0.05 × |M|) deep_global_registration.py. For small match sets, the 200 floor decides the branch.
• Safeguard search. The paper calls for RANSAC or branch-and-bound §4. The code runs correspondence-based RANSAC with 80,000 iterations, distance 2 × voxel size, and a fixed cap of 4,000,000 in the criteria deep_global_registration.py deep_global_registration.py.
• ICP runs by default. The paper describes ICP only as the "+ ICP" variant on KITTI tab. 3. The code sets use_icp = True and runs point-to-point ICP after every registration, including the 3DMatch path deep_global_registration.py deep_global_registration.py. The test script builds the same class test_3dmatch.py. The paper does not say whether its 3DMatch "Ours" row includes ICP. The released README lists TE 7.34 cm and RE 2.43° for the 3DMatch model, the same figures as the paper's "Ours" row README.md.
• Negative sign in the BCE equation. Equation 1 is written as a sum of log-probabilities without a minus sign eq. 1. The code minimises the mean binary cross-entropy with logits loss.py trainer.py. Read eq. 1 as the negative of this objective.
• Loss terms and weights. The paper says the final loss is a weighted sum of rotation, translation and BCE losses §3.3 without weights. The code's defaults are 1 for each term config.py. The code also keeps only batch entries with more than 10 valid weight mass trainer.py and does not apply that filter in the paper.
• Inlier features. The paper's 6-D ConvNet takes correspondence coordinates §3.2. The code's default feature is a constant 1, so the 6-D position of each match is the only input config.py deep_global_registration.py. The coordinate variant exists as an option deep_global_registration.py.
• Refinement stopping rule. The break counter increments whenever the loss changes by less than the ratio, and it is never reset. Early stopping therefore triggers after 20 such iterations in total, not 20 in a row registration.py.
• Refinement optimiser. The paper names SGD or Adam as options §4.2. The code uses Adam with learning rate 0.1, an exponential decay of 0.999 per step, and a maximum of 1000 iterations registration.py registration.py.
• Huber loss scale. The code divides each residual by 2 × voxel size before the Huber function, and normalises by the weight sum loss.py deep_global_registration.py. The paper gives no scale.
• Positive-pair search. The paper defines ground-truth matches by a distance threshold τ §3.2 and does not state its value. The code sets the matching search size to the voxel size times a multiplier, which is 4 in the training script (0.2 m at 5 cm) base_loader.py train_3dmatch.sh. The code default is 1.5 config.py. I did not trace this value through the matching function.
• Kernel sizes and feature network. The paper gives no kernel sizes. The training script uses 7 for the FCGF network's first layer train_3dmatch.sh. The config default is 3 for both networks config.py config.py.
• KITTI numbers. The paper's "Ours + ICP" row reads TE 3.46 cm tab. 3. The README's KITTI model row reads TE 3.14 cm and RE 0.14° README.md. I could not confirm that the two are the same setting.

No discrepancy found in the rotation error definition: both use arccos((trace − 1)/2) §5.1 metrics.py.

Pitfalls

• Labels in the text are swapped. Section 5.1 says TE is defined by Eq. eq:rte and RE by eq:rre §5.1. In the paper's own equations, eq:rte is the rotation error and eq:rre is the translation error §3.3. Use the definitions, not the labels.
• The table caption overstates speed. Table 1's caption says the method is "as fast as FGR" tab. 1. The times are 1.21 s and 0.31 s. The text also says the method is "about twice as fast as RANSAC with 2M iterations" §5.1, while the table gives 1.21 s and 1.39 s. Use the table values.
• Claims on KITTI. FCGF has the highest recall on KITTI (98.2%), and "Ours + ICP" is at 98.0% tab. 3. Do not read the abstract's claim of outperforming state-of-the-art as holding on recall there.
• Multi-way results are mixed. The method is best on two of four scenes and has the best average rank tab. 2. Office 1 is best for ElasticFusion, and Living room 2 is best for RANSAC.
• Numerically sensitive steps. The SVD of the cross-covariance may fail on degenerate input. The code catches the runtime error and falls back to the safeguard deep_global_registration.py. The rotation error uses arccos with arguments clamped to ±0.999, so errors near 0° and 180° are flattened metrics.py.
• Unstated safeguard behaviour. The paper does not say what happens if the safeguard RANSAC also fails. The code returns its result, with ICP applied after it deep_global_registration.py.
• Cost. The paper's timings include feature extraction, and the experiments ran on a GTX 1080Ti GPU tab. 1 §5.1. The 6-D network is trained on 3DMatch §5.1. The FCGF features come from prior work §3.1, and the paper does not describe their training here. It states no CPU-only timing.
• Generalisation claim. The multi-way experiments use networks trained only on 3DMatch and not fine-tuned on the other datasets §5.2. The results show cross-dataset transfer on these scenes only.
• Learned baselines. The paper reports that DCP, PRNet and 3DRegNet did not reach reasonable performance in its setup, and that PointNetLK was fine-tuned for 400 epochs §5.1. Their rows in tab. 1 are therefore weak baselines.

Lineage

• Builds on FCGF features for correspondence descriptors §3.1 and on the differentiable Procrustes of Wang et al. (DCP), which the paper replaces with the weighted version §3.3.
• Uses Zhou et al.'s continuous 6-D rotation representation for the refinement §4.1.
• Compared against FGR, RANSAC, Go-ICP, Super4PCS, ICP variants, DCP, PRNet, 3DRegNet and PointNetLK on 3DMatch tab. 1, and against FGR, RANSAC, FCGF on KITTI tab. 3.
• Superseded: not stated in this wiki's content.