GRNet: Gridding Residual Network for Dense Point Cloud Completion

In plain words

A 3D scanner, such as a LiDAR or a depth camera, often returns a cloud of points that is missing whole sides of an object. GRNet is a neural network that guesses the missing shape and returns a complete cloud with many more points.

Point clouds are awkward for neural networks. The points have no order and no fixed grid, so the tools that work well on images do not apply directly. GRNet first drops the points onto a regular grid of small cubes. Each point spreads its weight over the corners of the cube it sits in. Ordinary 3D convolutions can then read the grid and learn what shape surrounds each spot.

The network turns the grid back into points. For each new point, it looks up features from the nearby grid cells and moves the point to its final position. Training compares predicted points with the true ones using a standard distance between two point sets. The paper also defines a loss that compares the two clouds on the grid. The released training code does not use that loss; see Pitfalls.

TL;DR

GRNet completes an incomplete point cloud in one forward pass. Three grid layers (Gridding, Gridding Reverse and Cubic Feature Sampling) let standard 3D convolutions process unordered points. The paper reports the best overall Chamfer distance on ShapeNet and Completion3D, and the best consistency on KITTI cars. The paper and the official code disagree in several places, listed under Paper vs code. The paper's own numbers were not reproduced for this card.

Problem

Partial clouds from depth sensors and LiDAR are sparse and incomplete §1. The paper groups earlier approaches by their cost. Voxelising a cloud into binary cells loses geometric information and is not differentiable as a reconstruction target §1 §3.2. MLP encoders with max pooling do not model the connectivity and neighbour context of points §1 §2. Graph methods build neighbourhoods with k-nearest neighbours, which depend on point density §1. SPLATNet and InterpConv assume that output points coincide with input points, so the paper says they cannot be used for completion §2. The paper also argues that Chamfer-trained networks tend to output a mean shape §1 §3.7.

Method

GRNet works coarse-to-fine §3.1 fig. 1. The five components are Gridding §3.2, a 3D CNN §3.3, Gridding Reverse §3.4, Cubic Feature Sampling §3.5 and an MLP §3.6. The first three produce a coarse cloud, the last produces the dense cloud.

1. Gridding §3.2. Each point of the input cloud (in the unit cube) is spread over the 8 vertices of the cell it lies in. The grid has 64 vertices per axis §4.7 tab. 5. The paper's grid value at a vertex sums the trilinear weight of each neighbouring point and divides by the number of neighbours §3.2 eq. 2 eq. 3. Vertices with no neighbour get value 0. The released code does not divide by the count; see Paper vs code.

   w_i = \sum_{p \in \mathcal{N}(v_i)} \frac{w(v_i, p)}{|\mathcal{N}(v_i)|} §3.2 eq. 2

   w(v_i, p) = (1 - |x_i^v - x|)(1 - |y_i^v - y|)(1 - |z_i^v - z|) §3.2 eq. 3

2. 3D CNN §3.3 fig. 2. The encoder has four blocks. Each block has a 3D convolution with 4^3 filters and padding 2, batch normalisation, leaky ReLU and max pooling with kernel 2^3. Output channels are 32, 64, 128 and 256. Two fully connected layers (2048, then 16384) follow the encoder. The decoder has four transposed convolutions with batch normalisation and ReLU. Encoder and decoder maps of the same size are connected (U-Net style). The output is a grid W' eq. 4. The paper's decoder description (stride 1, padding 2) cannot reach 64^3; see Paper vs code.

3. Gridding Reverse §3.4 eq. 5. Each grid cell gives one point. Its coordinate is the weighted mean of the 8 vertex coordinates, with weights taken from the output grid W'. A cell whose weights sum to zero gives no point.

   p_i^c = \frac{\sum_{\theta \in \Theta^i} w_\theta' v_\theta}{\sum_{\theta \in \Theta^i} w_\theta'} §3.4 eq. 5

4. Cubic Feature Sampling §3.5 eq. 6 tab. 6. 2,048 coarse points are sampled at random. For each point, the feature vectors of the 8 vertices of its cell are read from three decoder maps: 128 channels at 8^3, 64 at 16^3 and 32 at 32^3. The 8 vertices of each map are concatenated, giving 1,792 values per point §3.5. The paper gives the gradient of these features with respect to point coordinates as zero app. A.3.

   f_i^c = [f_{\theta_1^i}^v, f_{\theta_2^i}^v, \dots, f_{\theta_8^i}^v] §3.5 eq. 6

5. MLP §3.6 eq. 7. The MLP takes the coarse cloud and its features and predicts residual offsets. Each coarse point is copied r = 8 times, the copies are offset, and the result has 16,384 points. The MLP's four layers have output sizes 1792, 448, 112 and 24.

   P^f = {\rm MLP}(F^c) + {\rm Tile}(P^c, r) §3.6 eq. 7

6. Training objective. The paper defines the Gridding Loss, an L1 distance between the grids of predicted and true clouds §3.7 eq. 8. The paper does not state the full training objective. For evaluation it uses Chamfer distance §4.2 eq. 9 and F-Score §4.2 eq. 10.

   \mathcal{L}_{Gridding}(W^{pred}, W^{gt}) = \frac{1}{N_G^3} \sum ||W^{pred} - W^{gt}|| §3.7 eq. 8

# GRNet forward pass  [§3.1] [fig. 1]
# Input:  partial_cloud (B, 2048, 3), coordinates in the unit cube
# Output: coarse_cloud (B, 2048, 3), dense_cloud (B, 16384, 3)

def grnet(partial_cloud):
    # Gridding: spread each point over the 8 corners of its cell  [§3.2]
    grid_in = gridding(partial_cloud, resolution=64)  # (B, 64, 64, 64)

    # Encoder: four blocks, each halves the resolution  [§3.3]
    enc_32 = encoder_block_1(grid_in)                 # (B, 32, 32, 32, 32)
    enc_16 = encoder_block_2(enc_32)                  # (B, 64, 16, 16, 16)
    enc_8 = encoder_block_3(enc_16)                   # (B, 128, 8, 8, 8)
    enc_4 = encoder_block_4(enc_8)                    # (B, 256, 4, 4, 4)

    # Bottleneck: fully connected 2048 and 16384, added back to enc_4  [§3.3]
    code = relu(linear(flatten(enc_4), 2048))         # (B, 2048)
    dec_4 = reshape(relu(linear(code, 16384)), (-1, 256, 4, 4, 4))
    dec_4 = dec_4 + enc_4

    # Decoder: each step doubles the resolution and adds the encoder map
    # of the same size  [§3.3]  (stride 2 needed; see Paper vs code)
    dec_8 = transposed_block_7(dec_4) + enc_8         # (B, 128, 8, 8, 8)
    dec_16 = transposed_block_8(dec_8) + enc_16       # (B, 64, 16, 16, 16)
    dec_32 = transposed_block_9(dec_16) + enc_32      # (B, 32, 32, 32, 32)
    dec_64 = transposed_block_10(dec_32) + grid_in    # (B, 1, 64, 64, 64)

    # Gridding Reverse: one point per cell, from its 8 corners  [§3.4]
    cell_points = gridding_reverse(dec_64)            # up to 63^3 points

    # Coarse cloud: 2,048 points sampled at random  [§3.5]
    coarse_cloud = random_sample(cell_points, 2048)

    # Cubic feature sampling from three decoder maps  [§3.5] [tab. 6]
    feats_32 = cubic_sample(coarse_cloud, dec_32)     # (B, 2048, 256)
    feats_16 = cubic_sample(coarse_cloud, dec_16)     # (B, 2048, 512)
    feats_8 = cubic_sample(coarse_cloud, dec_8)       # (B, 2048, 1024)
    point_feats = concat(feats_32, feats_16, feats_8) # (B, 2048, 1792)

    # MLP: 8 offsets per coarse point, added to 8 copies of it  [§3.6]
    offsets = mlp(point_feats)                        # (B, 2048, 24)
    offsets = reshape(offsets, (-1, 16384, 3))
    dense_cloud = tile(coarse_cloud, 8) + offsets     # (B, 16384, 3)
    return coarse_cloud, dense_cloud

# Cubic feature sampling for one coarse point  [§3.5] [app. A.3]
# Input:  point (3,) in grid coordinates, feature_map (C, t, t, t)
# Output: features (8 * C,)

def cubic_sample(point, feature_map):
    # Indices of the 8 corners of the cell that contains the point  [app. A.3]
    corners = cell_corners(point, feature_map.size)   # 8 index triples
    # Concatenate the channel vectors of the 8 corners  [§3.5] [eq. 6]
    return concat([feature_map[:, c] for c in corners])

# Training loss as released  [code: core/train.py#L112-L115]
# Chamfer distance is the mean squared nearest-neighbour distance,
# taken in both directions  [§4.2] [eq. 9]

def training_loss(coarse_cloud, dense_cloud, gt_cloud):
    sparse_loss = chamfer(coarse_cloud, gt_cloud)
    dense_loss = chamfer(dense_cloud, gt_cloud)
    return sparse_loss + dense_loss

Hyperparameters

Results

Author-reported. Setup: ShapeNet with the PCN splits and 16,384 ground-truth points §4.1. Completion3D is evaluated on the online leaderboard using the checkpoint with the lowest validation Chamfer distance, and 1,184 test objects §4.5. KITTI baselines were fine-tuned on ShapeNetCars §4.6.

The paper says GRNet is best on ShapeNet Chamfer and F-Score §4.4 and on Completion3D §4.5. On KITTI it is not best on FD or MMD tab. 9.

Paper vs code

Code is at commit fd3e483bc182b3456a82a5fdecef213b5defe96d, MIT licence.

• Gridding values are not averaged. Paper eq. 2 divides each vertex sum by the number of neighbours §3.2 eq. 2. The CUDA kernel adds the trilinear products to the vertex without a count gridding.cu. The vertex value therefore grows with local point density.
• Decoder stride. The paper says the transposed convolutions use stride 1 and padding 2 §3.3. With kernel 4 that maps a size-n input to n-1, so the paper's 4 to 64 path is impossible. The code uses stride 2 and padding 1 grnet.py, and its comments give the sizes 4, 8, 16, 32, 64 grnet.py.
• Skip connections are sums. The paper says "U-net connections" §3.3. The code adds each encoder map to the decoder map of the same size grnet.py.
• Coarse cloud contains the input points. The paper samples 2,048 points from the coarse cloud §3.5. The code first concatenates the 2,048 partial input points, then drops all-zero rows, then samples 2,048 of the union grnet.py grnet.py. The sparse loss therefore includes the input points. If fewer than 2,048 non-zero points remain, sampling is with replacement grnet.py.
• Cubic feature coordinates have an offset. The supplementary formula uses floor and ceil of (t/2) times the coordinate, with no offset app. A.3. Coordinates lie in [-1, 1], so that formula gives negative indices. The code maps them to [0, t] by adding t/2 [code: extensions/cubic_feature_sampling/init.py#L38-L39].
• Empty-cell rule. The paper skips a cell only when its weights sum to zero §3.4. The code skips when the sum is below 1e-6 gridding_reverse.cu, and skips every cell on the first index plane gridding_reverse.cu.
• Gridding Loss is not in training. The paper defines the Gridding Loss §3.7 eq. 8 and says it recovers details §1. The code builds the loss object train.py but never calls it. The training loss is the sum of two Chamfer distances train.py. The test loop also builds it and does not call it test.py.
• Gridding Loss grid differs. The code uses a grid spanning the bounding box of both clouds plus one cell on each side, not a fixed N_G^3 grid [code: extensions/gridding_loss/init.py#L69-L70]. Each vertex stores 8 values, one per cube corner gridding_distance.cu. The L1 loss averages over all of them [code: extensions/gridding_loss/init.py#L97-L98].
• Chamfer distance matches. Eq. 9 is the mean squared nearest-neighbour distance in both directions §4.2 eq. 9. The code computes the same [code: extensions/chamfer_dist/init.py#L42].
• Metric scale. Tables use x10^4 for L2 Chamfer tab. 1. The code logs the same quantity times 1000 metrics.py, so a logged 0.2723 is the paper's 2.723.
• Metrics not implemented. The appendix L1-norm Chamfer app. B and the FD and MMD metrics app. D are not in the repository. The Metrics class contains only F-Score and Chamfer metrics.py.
• Completion3D subsampling is absent. The paper subsamples the completed clouds to 2,048 points for the benchmark §4.5. The inference script writes the 16,384-point dense cloud without subsampling inference.py.
• KITTI back-transform is absent. The paper transforms completed clouds back to the world frame §4.6. The repository normalises the input to the box frame data_transforms.py and writes the output in that frame inference.py. No inverse transform is in the repository.
• Training views and augmentation. The paper does not state how many partial views each model has §4.1. The config sets 8 renderings per model, one chosen at random at each load during training and view 0 at test config.py data_loaders.py. Training also applies random mirror flips in x and z data_transforms.py data_loaders.py. The paper mentions neither.
• Layer list. The paper gives four MLP output sizes (1792, 448, 112, 24) §3.6. The code's first layer is 1792 to 1792, followed by 448, 112 and 24 grnet.py. The last layer has no activation in the code.

Pitfalls

• The training objective is not in the paper. Only the released code defines it: an unweighted sum of Chamfer distances on the coarse and dense clouds train.py. Implement that, and say so when comparing.
• Ablation tab. 7 cannot be reproduced from the released code. The code has no switch that adds the Gridding Loss, so the reported effect of that loss is not something the repository lets you test. The paper does not say which training loop produced tab. 7.
• Use stride 2 in the decoder. The stated stride 1 gives no upsampling path to 64^3. The code's stride 2 does grnet.py.
• Eq. 2 without the count is what the code does. If you follow the paper's averaging, your grid values differ from the code's gridding.cu.
• Output points can be input points. The coarse cloud and the sparse loss include the input points grnet.py. Dense outputs are offsets from those points grnet.py, so they inherit the input's coordinates.
• Random sampling at test time. The input is subsampled at random and so is the coarse cloud, with no fixed seed data_loaders.py grnet.py. Benchmark numbers vary from run to run unless you set the seed yourself.
• Empty rows are dropped. Any point that is exactly (0, 0, 0) is removed, not only the zero-filled rows of the grid output grnet.py.
• Inputs must lie in the unit cube. The gridding kernel computes cell indices without a range check gridding.cu. Points outside [-1, 1]^3 index outside the grid. The paper's ground truth lies in (-1, 1) app. A.2, and the paper states no input normalisation.
• Cubic feature sampling gives zero features outside the grid cubic_feature_sampling.cu. Its gradient with respect to coordinates is zero, as the paper states app. A.3. Coordinates therefore learn only through the Gridding Reverse path and the offsets.
• Cost is hidden in the parameter count. At 64^3 the whole network has about 76.70 M parameters and a backward pass of 100 ms at batch size 1 on a TITAN Xp tab. 5. The released checkpoint is 306.8 MB README.md, which matches 76.7 M float32 values.
• Coarse-point count is not reported. The appendix says the coarse cloud usually has 3,000 to 4,000 points app. E. The code does not count them, and the number depends on how many cells get non-zero weight.
• Mirror augmentation is not in the paper. Turn it off if you want to match the paper's stated setup exactly; the paper does not say whether it used one.

Lineage

• Compared against: AtlasNet, PCN, FoldingNet, TopNet and MSN on ShapeNet and Completion3D §4.4 tab. 1 tab. 3.
• Builds on: Chamfer distance, as used by PSGN §3.7 §4.2; a 3D encoder-decoder with U-Net connections §3.3; Adam §4.3.
• Related in the paper's discussion: graph-based networks (DGCNN, DCG) and convolutions on discrete grids (PointCNN, Hua et al.) §2.
• Superseded by: not known from content on this wiki.