Deep Parametric Continuous Convolutional Neural Networks

In plain words

A normal image filter assumes the pixels sit on a regular grid, so it can slide over them. A 3D scan from a lidar or depth sensor does not: its points are spread unevenly through space. This paper defines the filter as a small neural network that can be asked for a value at any relative position between two points. Because it is not tied to a grid, the same layer works on scattered points and can produce its output at new positions, which also lets it combine many points into fewer ones.

The authors stack these layers into deep networks with the usual parts (batch normalisation, ReLU, residual links) and train them directly on raw points. They test the idea on indoor room scans, on lidar sweeps from driving, and on estimating how each lidar point moved between two sweeps.

TL;DR

The paper proposes parametric continuous convolution: the kernel is an MLP of the relative offset between points, so the operator is defined over the whole continuous space and works on non-grid data. Layers are stacked into residual networks for point-wise segmentation and lidar motion estimation. Author-reported results beat the compared baselines on the indoor and driving benchmarks, and the outdoor data is in-house and not stated to be released. No official code was found.

Problem

Discrete convolutions need data on a grid. Point clouds have no grid, so existing options are to voxelise the space (memory-heavy, most cells empty), use graph networks (hard to make deep or to generalise), or use point-wise networks that aggregate with pooling such as PointNet §2. The authors want an operator that is a true convolution over continuous coordinates, with kernel values defined for any offset, and that can take input and output points that differ §1 §3.2.

Continuous convolution in its exact form needs an integral that is analytically tractable, which real non-parametric features and sparse sampled points are not §3.1. The paper therefore uses a Monte-Carlo estimate over the sampled points §3.1.

Method

The core operator uses a learnable kernel and a sum over sampled support points. The authors list these design choices:

1. Kernel as an MLP. The kernel is g(z; θ) = MLP(z; θ) with z the relative position in R^D. The MLP is argued to be a universal approximator of continuous functions, and polynomials are rejected because low-order ones are not expressive and high-order ones are numerically unstable in back-propagation §3.1.
2. Monte-Carlo estimate. The continuous convolution is approximated by an average over sampled points y_i, which gives h(x) ≈ Σ_i (1/N) f(y_i) g(x − y_i) §3.1. This display is not numbered in the paper.
3. Layer. Each output channel k at output location i sums over input channels d and support points j, weighting f_{d,j} by the kernel value g_{d,k}(y_i − x_j) §3.2. The inputs are features, support locations and output locations, which may differ. The paper's index naming is inconsistent here; see Pitfalls §3.2.
4. Locality window. g(z) = MLP(z) · w(z), where w is the indicator of the K nearest neighbours or of a ball of radius r §3.3.
5. Factorisation for cost. The full weight tensor W ∈ R^{N×|S|×F×O} is split into a linear map W1 ∈ R^{F×O} and a kernel tensor W2 ∈ R^{N×|S|×O} produced by the MLP. Only N·|S|·O kernel values are computed and stored, not N·|S|·F·O §3.3.
6. Inference speed-up. At inference the batch-norm layer is merged into the MLP's final fully connected layer; the authors report a 3× speed-up from this §3.3.
7. Special cases. A regular 2D grid gives ordinary convolution; a Gaussian kernel over position and feature gives a bilateral filter; neighbour-vertex support gives a first-order spatial graph convolution §3.3.

The deep network stacks these layers. Residual connections between layers are reported as critical for convergence, and batch norm, non-linearities and pooling over the support domain are allowed §3.2. Training is end-to-end by back-propagation through the kernel MLP §3.2.

# Parametric continuous convolution layer  [§3.2] [§3.3]
# Inputs:  features (N, F), support_locs (N, D), output_locs (M, D)
# Output:  out (M, O)

def continuous_conv(features, support_locs, output_locs, kernel_mlp, W1):
    # Offsets from every output point to every support point  [§3.1]
    offsets = output_locs[:, None, :] - support_locs[None, :, :]  # (M, N, D)
    # Locality window: K nearest support points or those within radius r  [§3.3]
    window = nearest_or_radius_mask(offsets)                  # (M, N)
    # Kernel value per point pair and output channel, from the MLP  [§3.3]
    kernel = kernel_mlp(offsets) * window[..., None]          # (M, N, O)
    # Linear mixing of input channels, shared over point pairs  [§3.3]
    mixed = features @ W1                                     # (N, O)
    # Weighted sum over support points, as in the Monte-Carlo estimate  [§3.1]
    out = (kernel * mixed[None, :, :]).sum(axis=1)            # (M, O)
    return out

# Indoor semantic-segmentation head  [§4.1]
# Input:  point_features (N, 128) from the last continuous conv layer
# Output: logits (N, 13)

def segmentation_head(point_features, fc):
    # Global room context: max over all points  [§4.1]
    global_feature = point_features.max(axis=0)               # (128,)
    # Repeat the global feature for every point  [§4.1]
    tiled = broadcast_to(global_feature, point_features.shape)  # (N, 128)
    # Concatenate to 256 values per point, then classify  [§4.1]
    joined = concatenate([point_features, tiled], axis=1)     # (N, 256)
    # Final linear layer; softmax over the 13 classes  [§4.1]
    return fc(joined)                                         # (N, 13)

Hyperparameters

Only the values below are stated. Learning rate, batch size, number of epochs, learning-rate schedule and the neighbour count K (or radius r) used for the accuracy experiments are not stated anywhere in the paper.

Results

All numbers are author-reported. Each result is shown with its setup and the baseline it is compared against.

• Indoor (Stanford large-scale 3D indoor scene dataset, 13 classes) §4.1: Ours PCCN reaches 58.27 mIOU and 67.01 mAcc, against SEGCloud at 48.92 and 57.35 and PointNet at 41.09 and 48.98 tab. 1. The text reports gains of 9.3% mIOU and 9.6% mAcc over the best baseline §4.1.

• Driving (in-house Velodyne-64 scans, North American cities) §4.2:

  Model	pACC	mIOU	Model size	Source
  PointNet	91.96	38.05	20.34 MB	tab. 2
  3D-FCN	94.31	49.28	74.66 MB	tab. 2
  Ours PCCN	94.56	46.35	9.34 MB	tab. 2
  Ours 3D-FCN+PCCN	95.45	58.06	74.67 MB	tab. 2

• Lidar flow (same driving data) §4.3: the input is two consecutive lidar sweeps, and the output is a 3D motion vector for each point of the first sweep §4.3. 3D-FCN has EPE 8.161 cm, 25.92% outliers at 10 cm and 7.12% at 20 cm. 3D-FCN+PCCN has 7.810 cm, 19.84% and 5.97% tab. 3. The text states about 20% fewer outliers §4.3.

• ModelNet40 classification app. E: Ours 88.9% against PointNet 89.2% and PointNet++ 91.9% tab. 4. The authors call it comparable to PointNet.

• Runtime §4.2: an 8-layer model with 32 dims and 50 neighbours runs a forward pass in 33 ms, with a KD-tree neighbour search of 28 ms, so 61 ms end to end on a GTX 1080 Ti.

• KITTI generalisation §4.2 app. A: trained on North American driving data and tested on KITTI without retraining. Only qualitative results are shown for this; no metric is reported.

Paper vs code

No official code.

Searched: the TeX sources contain no code or project link [raw/src]; the arXiv abstract page lists no repository or project page; GitHub searches for "PCCN", "continuous convolution", "parametric continuous convolution", and the full title found no repository owned by the authors; the first author's GitHub account and an organisation search in uber-research found nothing relevant. Third-party repositories with similar names were not used.

Pitfalls

• Dataset sizes disagree within the paper. The introduction states 223 billion outdoor points §1; the experiments state 137 billion outdoor and 629 million indoor §4; the conclusion states up to 200 billion §5. Do not quote one as a fact without this caveat.
• Index names in the layer formula are inconsistent. §3.1 defines support points as y_j and output points as x_i, but the layer formula §3.2 sums over x_j ∈ S and y_i ∈ O, the reverse of the definitions. Implement the verbal definition: output i, support j.
• Kernel output dimension is not explicit. §3.1 writes g: R^D → R, a scalar, yet the factorised form in §3.3 needs O kernel values per point pair. The paper does not state how the MLP's output is shaped; the reading above (one output per output channel) follows from the factorisation.
• Normalisation is not stated. The Monte-Carlo estimate has a 1/N factor §3.1, but the layer formula §3.2 has none. Whether the code or the authors normalise by the number of neighbours is not stated.
• Neighbour count and radius. The locality window is K nearest neighbours or radius r §3.3, but K or r is only given for the runtime benchmark (50 neighbours) §4.2. The values used for training are not stated.
• Training details are missing. Only the optimiser (Adam), the loss, and the layer counts are stated. Learning rate, batch size, epochs and schedule are not stated for any model.
• Residual links. The paper says residual links between continuous conv layers are critical for convergence §3.2. Where they sit (every layer or every block) is not stated; the driving network says "residual" §4.2.
• The PCCN-only driving model is weaker on mIOU. Ours PCCN has 46.35 mIOU against 49.28 for 3D-FCN tab. 2. It is about 8× smaller §4.2, but the accuracy gain comes from the combined 3D-FCN+PCCN model.
• Baselines were modified. PointNet was adapted to the driving data and trained from scratch, and the point-rotation layer was removed because it hurt validation accuracy §4.2. The 3D-FCN baseline was trained from scratch with the authors' own voxelisation §4.2.
• Runtime model is unclear. The text says 8 layers §4.2, while the driving PCCN has 16 layers §4.2. The benchmarked model is not identified.
• Relative reduction in outliers. The text says "approximately 20%" §4.3. The table gives about 23% at 10 cm and about 16% at 20 cm, computed from tab. 3.
• Data and code are not public. The driving and flow data are in-house and labelled by in-house annotators §4.2 §4.3. No data release is stated. Ground-truth flow combines GPS and IMU, ICP alignment to pre-scanned geometry, and tracklet-based object poses, and assumes rigid objects app. B. Results on these data cannot be reproduced from public sources.
• Venue. The arXiv comment reads "Accepted by CVPR 2018", but the arXiv submission is dated 2021-01-17 [arXiv metadata]. Cite the arXiv id.
• Special-case claims are not tested. The bilateral-filter and graph- convolution special cases §3.3 are stated analytically; the experiments do not test them.

Lineage

• Builds on: Monte-Carlo integration for the sampled estimate §3.1; the universal approximation property of MLPs for the kernel §3.1.
• Compared against: PointNet, PointNet++ and SEGCloud §4.1 §2; 3D-FCN and voxel-based networks §4.2 §2; MVCNN, VoxNet, ECC and others for classification app. E.
• Related: SchNet, a concurrent work using a similar parametric function of relative positions with shallow Gaussian kernels §2; spatial graph convolutions, which the paper recovers as a special case §3.3.