FPNN: Field Probing Neural Networks for 3D Data

In plain words

A 3D object is usually stored as a grid of small cubes, and the standard approach slides a small 3D filter over that grid, as image networks slide a filter over pixels. This gets very expensive as the grid gets finer, and most cubes are empty, so much of the work is wasted.

The paper's alternative is to place a small number of "sensors" in the space around the object. Each sensor is a short line of points, and each point reads the value of a distance field (how far that spot is from the surface) at its location. The network learns two things: what each sensor should measure, and where each of its points should sit. The sensors therefore move towards the informative parts of the shape instead of scanning every cube. The result is fed into ordinary fully connected layers that name the object's category. The authors report that this is cheaper than 3D convolution while keeping accuracy competitive on a 3D classification benchmark.

TL;DR

• FPNN classifies 3D shapes given as distance fields, replacing 3D convolutions with "field probing" layers §3.2.
• Each probing filter is a set of learnable points: their positions and the weights attached to them are both trained §3.2.
• Author-reported ModelNet40 top-1 accuracy: 88.4% with four FC layers and normal fields, below the 88.8% of the SubvolSup+BN baseline cited from [qi2016volumetric] tab. 1 tab. 6.
• Use it as a reference for field sampling and learned sampling locations. Do not use it as a current classifier: the authors point to PointCNN in the repository README.

Problem

Voxel-based 3D CNNs have cubic cost in voxel resolution, and researchers typically use 30 × 30 × 30 grids. Surface-based occupancy grids are sparse at high resolution, so most convolutions multiply zeros, and long-range context needs larger kernels §1 fig. 1. The paper assumes the input can be turned into a 3D field whose gradients can be computed §3.1 §4.6.

Method

The paper has no numbered equations and no algorithm block. Equations are inline and cited by subsection; the pseudocode below is our restatement of §3.2.

Steps:

1. Build the input field. Convert a mesh or point cloud to an occupancy grid, take a distance transform of the occupied cells, and store the resulting distance field on an R × R × R grid. Values between grid points come from trilinear interpolation. Normal fields are the normalized gradient of the distance field §3.1 fig. 3. Other fields are allowed if their gradients can be computed §3.1.
2. Initialize the probing filters. Split the space into G × G × G cells and place P filters in each. Each filter is a segment that points in a random direction, with its length drawn from the range set by l_low and l_high (default [0.2, 0.8] · R), and its centre placed at random inside its cell. Weights start with Xavier initialization §3.2 "Initialization" fig. 4. G and P are not stated.
3. Sensor layer. For each probing point, read the field at its location. The layer's parameters are the point locations. Its gradient with respect to location is computed numerically from the field, and that gradient moves the points during training §3.2 "Sensor Layer".
4. Gaussian layer. Apply an element-wise Gaussian to the sensor values, so samples near the surface get large values and distant samples get small ones. The authors say that without it training does not converge well [§3.2 "Gaussian Layer"].
5. DotProduct layer. For each filter, take the dot product of its weights and its signals, giving one value per filter. Weights are not shared across locations §3.2 "DotProduct Layer".
6. Head and training. Batch normalization and ReLU follow the probing layers. Then come either one FC classifier (1-FC) or three 1024-wide FC layers with dropout and a classifier (4-FCs). Training uses softmax cross-entropy over 40 classes §4.3 §4.2 fig. 5.

# Field probing layer, forward pass  [§3.2]
# Input:  field (R, R, R, T), distance and/or normal channels
# Params: locations (C, N, 3), weights (C, N, T)
# Output: features (C,), one value per probing filter

def field_probing_forward(field, locations, weights):
    # 1. Sensor layer: read the field at each probing point  [§3.2]
    raw_signals = trilinear_sample(field, locations)       # (C, N, T)
    # 2. Gaussian layer: emphasise values near the surface  [§3.2]
    signals = exp(-raw_signals ** 2 / (2 * sigma ** 2))    # (C, N, T)
    # 3. DotProduct layer: one weighted sum per filter  [§3.2]
    features = sum(signals * weights, axis=(1, 2))         # (C,)
    return features, raw_signals

# Gradient of the loss with respect to probing point locations  [§3.2]
# Input:  upstream gradient per filter, weights, raw sensor values,
#         field gradient at each probing point (computed numerically)
# Output: gradient for each location, (C, N, 3)

def location_gradient(grad_features, weights, raw_signals, field_grad):
    # Upstream gradient for each signal, from the DotProduct layer  [§3.2]
    grad_signals = grad_features[:, None, None] * weights  # (C, N, T)
    # Chain rule through the Gaussian, g'(x) = -x g(x) / sigma^2  [§3.2]
    grad_signals = grad_signals * gaussian_slope(raw_signals, sigma)
    # Chain rule through the Sensor layer, using the field gradient  [§3.2]
    return sum(grad_signals[..., None] * field_grad, axis=2)  # (C, N, 3)

# One classification training step, FPNN with 4 FC layers  [§4.2] [§4.3]
# Input:  distance fields (B, R, R, R, 1) with random rotations  [§4.2]
# Output: scalar softmax cross-entropy over 40 classes

def training_step(fields, labels):
    # 1. Probing features per sample, (B, C)  [§3.2 "FPNN Architecture"]
    features = field_probing_forward_batch(fields)
    # 2. Batch norm, ReLU, dropout, then the 1024-wide FC layers  [§4.3]
    hidden = fc_stack_with_dropout(features)               # (B, 1024)
    # 3. Softmax over the 40 class logits  [§4.2] [§4.3]
    return cross_entropy(classifier(hidden), labels)

Key formulas, quoted in our notation:

g(x) = e^{-\frac{x^2}{2\sigma^2}}, \qquad g'(x) = -\frac{x\, e^{-x^2/(2\sigma^2)}}{\sigma^2} §3.2 "Gaussian Layer"

v_c = \sum_{n=1}^{N} \sum_{t=1}^{T} p_{c,n,t}\, w_{c,n,t} §3.2 "DotProduct Layer"

Complexity: the field probing layer costs O(C × N × T). A 3D convolution costs O(K³ × C × S³) §3.2 "Complexity". The typical values are C = 1024, N = 8, T = 4 against K = 6, C = 48, S = 12 for the 3D CNN.

Hyperparameters

Results

Author-reported. Setup: ModelNet40, 9,843 training and 2,468 test models using the authors' split; random rotations for train and test; single test view §4.2 §4.3. Baselines are numbers the paper cites from [WU_CVPR15_3D], [Maturana_IROS15_VoxNet] and [qi2016volumetric].

Top-1 accuracy, ModelNet40 tab. 1:

Robustness to perturbations, top-1 accuracy tab. 2. R = random horizontal rotation; R15 adds ±15° tilt; T0.1 and T0.2 are translations; S is scaling; R45 is a larger rotation. Perturbations are applied to train and test:

Filter length range tab. 3:

Field resolution tab. 4:

Generalizability, with ModelNet40 split by category name into two halves of 20 classes each tab. 5 §4.4:

Comparison with two methods cited from [qi2016volumetric] tab. 6:

Other results, in words only: the 1-FC probing model reaches 85.0% and the paper calls this better than the cited 3D CNNs at 83.0% [WU_CVPR15_3D] and 83.8% [Maturana_IROS15_VoxNet] §4.3. Timing: the paper reports that probing cost does not grow with input resolution, with a slight rise at higher resolution. Timing numbers exist only in the figure fig. 6 §4.1. The 1-FC accuracy grows with resolution from 16³ to 64³, and the Gaussian σ near 10% of object size is best; a smaller σ costs about 1% §4.3. The t-SNE view of features is qualitative fig. 7.

Paper vs code

Checked against the official repository at code_commit d418f6e3c0fd69370e23b0116862fc764aafca1b (github.com/yangyanli/FPNN). The repository holds the training configurations and field generators. The Caffe layer implementation (backends/caffe, a submodule to yangyanli/caffe) is not in the shallow clone, so the Sensor, DotProduct and Gaussian layers are not verified against code. We checked the configs only; the README has no statement on the layers.

• Weight initialization. Paper: Xavier for probing weights §3.2 "Initialization". Code: Gaussian with std 0.05 and bias 1 for the DotProduct layer train_val.prototxt. Hidden FC layers use Gaussian std 0.05 train_val.prototxt.
• Learning-rate schedule. Paper: learning rate 0.01, schedule not stated §4.3. Code: multiply by 0.1 every 40,000 iterations solver.prototxt.
• Filter length range. Paper default: [0.2, 0.8] · R §3.2 "Initialization". Shipped configs: min_span 0.1, max_span 0.4 train_val.prototxt. Table 3 has no 0.1–0.4 column tab. 3. The units are not stated in either place.
• Gaussian scope. Paper: element-wise Gaussian on the Sensor values §3.2 "Gaussian Layer". Multi-field code: the normals are sliced off and only the distance channel is passed through the Gaussian train_val.prototxt train_val.prototxt train_val.prototxt.
• +NF training. Paper: one training description for all settings §4.3. Code +NF config: batch 64 with 4 augmentations (256 per step), 160,000 iterations, step at 80,000 train_val.prototxt solver.prototxt.
• Normal fields. Paper: normals are the normalized gradient of the distance field §3.1. Code: a GradientField layer computes them from the un-rotated data, then Transform3D rotates them with the distance field train_val.prototxt train_val.prototxt. Whether the rotation also rotates the 3-vector components is inside the uncloned layer code; not verified.
• Probing-layer parameter decay. Paper: silent on decay for locations. Code: decay_mult 0 on the probing layer train_val.prototxt.
• "Without FP" control. Paper: trains "without/with updating the field probing layer parameters" §4.3, without naming which parameters. Code: lr_mult 0 on the probing layer train_val.prototxt. The DotProduct layer is still trained in that config train_val.prototxt.
• Rotation axis. Paper: "random horizontal rotation" §4.2. Code: rotation about z only train_val.prototxt. This matches if z is the vertical axis, which the config does not state.
• Dropout ratio. Paper: not stated §4.3. Code: 0.5 train_val.prototxt.
• Matches. C = 1024, N = 8, T = 4 §3.2 "Complexity" match num_filter 1024 and len_filter 8 train_val.prototxt. The 256 × 4 = 1024 batch matches §4.3 footnote train_val.prototxt train_val.prototxt. 80,000 iterations, momentum 0.9 and decay 0.0005 match solver.prototxt. Weights are not shared, matching share_weights: false train_val.prototxt.
• Evaluation. The solver evaluates on the test set every 200 iterations solver.prototxt. The paper does not say which checkpoint it reports.

Pitfalls

• The baseline is not "no probing layer". Table 1's "w/o FP" column means the probing layers were not updated §4.3 tab. 1. The paper does not say whether the DotProduct weights were frozen as well. The shipped control config freezes only the probing layer's own parameters train_val.prototxt, so a reimplementation should state which of the two it froze.
• Gains are small. With four FC layers the gain from updating probing layers is 0.9 points (86.6 to 87.5) §4.3 tab. 1. The paper says so itself.
• Cross-protocol comparisons. The footnote says the split used in [WU_CVPR15_3D] was at most 80/20 per category, which the authors call too small and unlike their own split §4.2 footnote. The text still compares the 83.0% of [WU_CVPR15_3D] and the 83.8% of [Maturana_IROS15_VoxNet] directly with 85.0% §4.3. Table 6 numbers are cited from [qi2016volumetric] and are not re-run here tab. 6.
• Accuracy depends on augmentation. Training and testing both use random rotations about one axis §4.2. Numbers are not comparable with unaugmented benchmarks, and the paper's own baseline numbers may use a different protocol.
• Filter-length units. The paper's default is [0.2, 0.8] · R, while the shipped configs use 0.1–0.4 train_val.prototxt. The units (grid cells or a fraction of the field) are not stated anywhere in the repository we could read.
• Gaussian σ units. The paper gives σ as about 10% of object size, and the code gives sigma = 8 for 64³ fields train_val.prototxt. If σ is in grid cells, that is 12.5% of R. "Object size" is not defined.
• Padding and the Gaussian. Out-of-grid values are not specified in the paper. The code pads with 100 train_val.prototxt. If the padding and σ use the same units, the Gaussian maps that padding to about e^-78, which is effectively zero. This is our arithmetic, not the paper's.
• Sign convention. The distance field's sign is not stated §3.1. The Gaussian is even in x, so the sign does not affect the distance branch. It does flip the normal fields §3.1.
• Normal fields divide by a norm. The normal formula divides by the gradient magnitude l §3.1. The paper does not say what happens where the gradient vanishes, so an implementation needs a guard.
• Numerical gradients for locations. The paper computes the field gradient numerically from trilinear interpolation §3.2 "Sensor Layer". The step size is not stated. The paper does not say whether probing points are clamped to the grid when they move out of it.
• Arithmetic error in the complexity claim. The paper writes 1024 × 8 × 4 ≈ 1.83% × 6³ × 48 × 12³ §3.2 "Complexity". By the paper's own numbers, 32,768 / 17,915,904 ≈ 0.18%. This is our arithmetic. The saving is about ten times larger than the printed percentage.
• Timing is not like for like. The convolution timing uses cuBLAS kernels, while the probing layers use "naive parallelism" §4.1. The paper says so. The figure values are not in the text fig. 6.
• Retrieval claim is qualitative. The paper says probing features support shape retrieval §4.3. Evidence is one t-SNE figure fig. 7, with no retrieval metric.
• Hyperparameter selection. σ and the span were chosen "empirically" §4.3. The paper does not say whether this used a validation split. Accuracies are on the test split.
• Code cannot be verified for the core layers. Without backends/caffe, the forward and backward code of the probing layers is unverified. Do not treat the configs as a complete reference implementation.
• No license file. The repository has no LICENSE. Cite it by path and line only; do not copy code.
• Successor. The repository README says PointCNN outperforms FPNN on ModelNet40 and recommends it [README.md]. We did not check that claim.

Lineage

• Builds on: convolutional neural networks §2; voxel 3D CNNs of [WU_CVPR15_3D] and [Maturana_IROS15_VoxNet], named by the paper and not cards in this wiki; the sparse-coding view, as an interpretation of the probing weights §2.
• Compared against: the 3D CNN numbers of [WU_CVPR15_3D] and [Maturana_IROS15_VoxNet], and SubvolSup+BN and MVCNN-MultiRes from [qi2016volumetric] tab. 6.
• Superseded by: the repository README points to PointCNN (github.com/yangyanli/PointCNN) as the stronger successor. It is not in this wiki.