Continuous convolution

Overview

A standard convolution slides a fixed grid of learned weights over its input, so it needs the data to sit on a grid. Point clouds from lidar or depth sensors do not. A continuous convolution removes the grid: the kernel is a function of the relative position between two points, so it can be evaluated for any offset. The output is a sum over sampled support points, weighted by that kernel 2101.06742 §3.1. The kernel is learned as a small multi-layer perceptron, which lets the same layer take scattered inputs and produce outputs at new positions 2101.06742 §3.1 2101.06742 §3.2.

The only card for this concept is one paper, which uses the operator in a deep residual network for indoor and driving point clouds. The design space below therefore has one variant. Other approaches to point data, such as voxel grids and graph convolutions, are named there only as the paper's comparisons and special cases.

Variants

Parametric continuous convolution (MLP kernel)

The kernel is an MLP of the offset z between points. The authors argue that an MLP is a universal approximator for continuous functions and that polynomial kernels are either not expressive enough or unstable to train 2101.06742 §3.1. Locality is added by multiplying the kernel with a window that is 1 for the K nearest neighbours or for points within a radius r 2101.06742 §3.3. For cost, the layer computes one kernel value per point pair and output channel, not one per input-output channel pair, and mixes input channels with a shared linear map 2101.06742 §3.3. At inference the batch-norm layer is merged into the final fully connected layer of the kernel MLP, which the authors report as a 3x speed-up 2101.06742 §3.3. Card: Deep Parametric Continuous Convolutional Neural Networks.

Two networks are built from this layer. A 13-class indoor model stacks 8 continuous convolution layers with 32 hidden dimensions and a global max-pooled feature concatenated to each point 2101.06742 §4.1. A driving model stacks 16 residual layers in one configuration and 7 layers on top of a 3D-FCN in another 2101.06742 §4.2 2101.06742 tab. 2.

When to use which

The paper compares its operator with the alternatives on accuracy and cost. The numbers are author-reported and come from the paper's own setups, so they are not directly comparable to other benchmarks.

Implementation sketch

Language-agnostic pseudocode of the common core. The neighbour window and the kernel MLP are the parts the paper leaves open, see Pitfalls on the card.

# Parametric continuous convolution layer  [arxiv:2101.06742 §3.2] [§3.3]
# Inputs:  features (N, F), support_locs (N, D), output_locs (M, D)
# Output:  out (M, O)

def continuous_conv(features, support_locs, output_locs, kernel_mlp, W1):
    # Offsets from each output point to each support point  [§3.1]
    offsets = output_locs[:, None, :] - support_locs[None, :, :]  # (M, N, D)
    # Locality window: K nearest support points or those within radius r  [§3.3]
    window = nearest_or_radius_mask(offsets)                  # (M, N)
    # One kernel value per point pair and output channel, from the MLP  [§3.3]
    kernel = kernel_mlp(offsets) * window[..., None]          # (M, N, O)
    # Linear mixing of input channels, shared across point pairs  [§3.3]
    mixed = features @ W1                                     # (N, O)
    # Weighted sum over support points, the Monte-Carlo estimate  [§3.1]
    out = (kernel * mixed[None, :, :]).sum(axis=1)            # (M, O)
    return out

Contradictions

None known across papers: only one card exists. Within the paper, the layer formula 2101.06742 §3.2 reverses the index names defined in §3.1, and the text gives different dataset sizes in its introduction, experiments and conclusion 2101.06742 §1 2101.06742 §4 2101.06742 §5.

Papers

• Deep Parametric Continuous Convolutional Neural Networks: introduces parametric continuous convolution with an MLP kernel and applies it to indoor segmentation, driving segmentation and lidar motion estimation.