Point-NeRF: Point-based Neural Radiance Fields

In plain words

Classic neural rendering methods such as NeRF store a whole scene inside one network and train that network from scratch for every new scene. This is slow, and much of the work goes into empty air that is never seen.

Point-NeRF instead scatters many small learned "dots" in space, each carrying a short feature vector that describes the surface nearby. Dots are placed where the scene probably has surfaces, so most of the empty air is skipped. A network can predict a first set of dots directly from a few photos, which gives a usable scene immediately. A short per-scene training run then refines the dots, removes stray ones, and adds dots where the first guess had holes.

To render a picture, the method shoots rays through the image, looks only at dots near each ray, blends their features into color and density, and sums the result. Because everything is differentiable, the same rendering loss trains the dots and the networks together.

TL;DR

• Radiance field = neural point cloud with per-point features and a confidence in [0, 1] §3.
• A cross-scene network (MVS depth plus a 2D CNN) gives an initial point cloud in a feed-forward pass §4.1.
• Per-scene optimization adds point pruning (confidence below 0.1) and point growing, which fills holes that an external point cloud such as COLMAP leaves §4.2.
• Author-reported: NeRF-level quality on NeRF Synthetic in about 40 minutes of per-scene optimization (20K steps) §6.2.

Problem

Per-scene neural radiance fields such as NeRF need long training, mainly because one global MLP is fitted from scratch and many samples fall in empty space §1. Generalizable models that predict a field from images need no per-scene fitting, but the feed-forward result alone is lower in quality: on DTU, Point-NeRF's feed-forward PSNR is below IBRNet's §6.1 tab. 1. Point-based neural rendering is fast, but earlier methods rasterize point features and post-process them with 2D CNNs, which limits quality §1. Point clouds from reconstruction tools such as COLMAP contain holes and outliers, and simple per-point optimization cannot repair large holes §4.2.

The method assumes posed multi-view images with known camera parameters §4.1 and, for the feed-forward stage, a pre-trained MVS depth network §5.

Method

Representation (§3). A scene is a point set. Each point i has a position p_i, a feature f_i and a confidence γ_i in [0, 1]. To shade a 3D location x along a ray, the method takes the K nearest points within a radius R §3 app. H. Each point's feature is passed through a small MLP F together with the offset x − p_i, so the point describes the local scene in its own frame §3 eq. 3. These features are blended with inverse-distance weights scaled by confidence §3 eq. 4, and a radiance MLP R turns the blended feature and the view direction into color §3 eq. 5. Density is computed the same way: a per-point MLP T gives a density per point, and the blend uses the same weights §3 eq. 6 eq. 7. Pixels are formed by standard volume rendering along the ray over M shading samples §3 eq. 1.

Generating the initial field (§4.1). For each reference view, a MVSNet-style network builds a plane-swept cost volume from the reference view and two neighboring views, regresses a depth probability volume, and outputs a depth map as the probability-weighted sum of plane depths. The depth map is unprojected to points §4.1. Each point's confidence is read by trilinear sampling of the depth probability volume at the point §4.1. Point features come from a 2D CNN run on each image; the features at three resolutions are concatenated and sampled at the projected point §4.1 [app. G]. Points from all views are merged §4.1. The MVSNet part is first pre-trained with ground-truth depth; the whole chain is then trained end to end with an L2 rendering loss only §5.

Optimizing per scene (§4.2). Starting from the initial field, or from an external point cloud, the point features, confidences and MLPs are optimized with the rendering loss plus a sparsity term on confidences §4.2 eq. 12. The sparsity term pushes each confidence toward 0 or 1 §4.2 eq. 11. Every 10K iterations the method prunes points with confidence below 0.1 §4.2. It also grows points: for each ray it takes the shading sample with the largest opacity §4.2 eq. 10; if that opacity is above a threshold and the sample is far from every existing point, a new point is placed there §4.2. The paper does not state the two thresholds.

# Radiance of one ray  [§3] [eq. 1]
# Input:  origin (3,), direction (3,), points (N, 3), feats (N, C),
#         confs (N,), deltas (M,)
# Output: color (3,)

def render_ray(origin, direction, points, feats, confs, deltas):
    # 1. Sample M shading locations along the ray  [§3] [eq. 1]
    shading_locs = sample_along_ray(origin, direction, M)
    densities = []
    radiances = []
    for loc in shading_locs:
        # 2. Take the K nearest points within radius R  [§3] [app. H]
        nbr_idx = query_neighbors(loc, points, K, R)
        # 3. Per-point MLP on feature and offset  [§3] [eq. 3]
        local_feats = F(feats[nbr_idx], loc - points[nbr_idx])
        # 4. Inverse-distance weights times confidence  [§3] [eq. 4] [eq. 7]
        inv_dist = 1 / norm(points[nbr_idx] - loc)
        weights = confs[nbr_idx] * inv_dist / sum(inv_dist)
        # 5. Blend features, then regress radiance  [§3] [eq. 4] [eq. 5]
        agg_feat = sum(weights[:, None] * local_feats)
        radiances.append(R_mlp(agg_feat, direction))
        # 6. Per-point density, blended with the same weights  [§3] [eq. 6]
        point_sigma = T_mlp(local_feats)
        densities.append(sum(weights * point_sigma))
    # 7. Volume-render the samples  [§3] [eq. 1]
    return composite(densities, radiances, deltas)

Key equations (numbering counted from the source, see Pitfalls):

\mathbf{C} = \sum_{j=1}^{M} \tau_j \left(1 - \exp(-\sigma_j \Delta_j)\right) \mathbf{r}_j, \quad \tau_j = \exp\left(-\sum_{t=1}^{j-1} \sigma_t \Delta_t\right) eq. 1

(\sigma, \mathbf{r}) = \text{Point-NeRF}(x, d, p_1, f_1, \gamma_1, \dots, p_K, f_K, \gamma_K) eq. 2

f_{i,x} = F(f_i, x - p_i) eq. 3

f_x = \sum_i \gamma_i \frac{w_i}{\sum_i w_i} f_{i,x}, \quad w_i = \frac{1}{\lVert p_i - x \rVert} eq. 4

\mathbf{r} = R(f_x, d) eq. 5

\sigma_i = T(f_{i,x}) eq. 6

\sigma = \sum_i \sigma_i \gamma_i \frac{w_i}{\sum_i w_i} eq. 7

\{p_i, \gamma_i\} = G_{p,\gamma}(I_q, \Phi_q, I_{q_1}, \Phi_{q_1}, \dots) eq. 8

\{f_i\} = G_f(I_q) eq. 9

\alpha_j = 1 - \exp(-\sigma_j \Delta_j), \quad j_g = \arg\max_j \alpha_j eq. 10

\mathcal{L}_{\text{sparse}} = \frac{1}{|\gamma|} \sum_i \left[\log \gamma_i + \log(1 - \gamma_i)\right] eq. 11

\mathcal{L}_{\text{opt}} = \mathcal{L}_{\text{render}} + a\,\mathcal{L}_{\text{sparse}}, \quad a = 2\times10^{-3} eq. 12 §5

Optimization loop and the pruning/growing schedule:

# Per-scene optimization  [§4.2] [§5] [eq. 11] [eq. 12]
# Every 10K iterations, prune weak points, then grow points.

def optimize_scene(points, rays, steps):
    for step in range(steps):
        # 1. Render a batch of rays and compare with ground truth  [§5]
        loss_render = l2(render_batch(points, rays), gt_colors)
        # 2. Confidence sparsity; eps is not stated in the paper  [eq. 11]
        loss_sparse = mean(log(confs) + log(1 - confs))
        # 3. Total loss with a = 2e-3  [eq. 12] [§5]
        loss = loss_render + 2e-3 * loss_sparse
        update(points, mlps, loss)
        if step > 0 and step % 10000 == 0:
            # 4. Drop points with confidence below 0.1  [§4.2]
            points = points[confs >= 0.1]
            # 5. Add candidates from rays  [§4.2] [eq. 10]
            points = grow_points(points, rays)
    return points

# Growing candidates from one ray  [§4.2] [eq. 10]
# T_opacity and T_dist are not stated in the paper.

def grow_candidate(samples, sigmas, deltas, points):
    # 1. Opacity of each shading sample  [§4.2] [eq. 10]
    alphas = 1 - exp(-sigmas * deltas)
    best = argmax(alphas)
    # 2. Distance from that sample to its nearest point  [§4.2]
    dist = min_distance(samples[best], points)
    # 3. Grow if the sample is opaque and far from points  [§4.2]
    if alphas[best] > T_opacity and dist > T_dist:
        return samples[best]
    return None

Hyperparameters

Values are from the paper where stated. Code values are from the official per-scene scripts (dev_scripts/w_n360/hotdog.sh, the NeRF Synthetic script used as the reference below) and are not part of the paper.

Results

Author-reported. All numbers below are from the paper's tables; setups are given per table. The paper reports PSNR / SSIM / LPIPS unless noted.

DTU, test views, trained on DTU tab. 1. Setting: novel-view synthesis on the DTU test scenes, with per-scene fine-tuning for 1K and 10K iterations §6.1.

NeRF Synthetic, trained on DTU only tab. 2. Setting: eight object scenes with 360-degree cameras, per-scene optimization §6.2.

The paper says 20K steps take about 40 minutes and that this is at least 30 times faster than NeRF's 20+ hours §6.2.

Tanks and Temples and ScanNet tab. 3. Five Tanks and Temples scenes and two ScanNet scenes, the same selections as NSVF. Metrics are PSNR / SSIM / LPIPS (Alex).

Pruning and growing ablation tab. 4. Ship and Hotdog scenes, PSNR / SSIM / LPIPS (VGG).

Other numbers: the appendix repeats the NeRF Synthetic per-scene breakdown app. C and the Tanks and Temples and ScanNet breakdowns app. D app. E. Random instead of image-extracted point features gives 25.44 PSNR at 20K and 32.01 at 200K steps, against 30.71 and 33.77 for the full model app. A. Authors report a version with MVSNet points filtered by background color at 33.25 mean PSNR on NeRF Synthetic app. J.

Paper vs code

Code is at Xharlie/pointnerf (commit 1117541468cf2792c272be0fe1e7a0ce7bbceed4). Its owner matches the first author's project page, which the paper links. The repository is licensed CC BY-NC-SA 2.0; see Pitfalls.

• Sparsity weight. Paper sets a = 2e-3 for all experiments §5 eq. 12. Every per-scene script sets sparse_loss_weight=0 hotdog.sh col_ship.sh. The code's sparse term is also a different function: it weights an absolute term |1 - exp(-2 * conf)| by the blend weights, not the log form of eq. 11 base_rendering_model.py.
• Growing rule. Paper grows at the sample with largest opacity when the opacity is above T_opacity and the sample is far from its nearest point §4.2 eq. 10. The probing code in probe_hole chooses rays that miss the points and sit next to the rendered mask, then keeps the max-opacity sample if its opacity exceeds prob_thresh=0.7 train_ft.py hotdog.sh. I found no nearest-point distance check in that path train_ft.py. Grown confidences are multiplied by prob_mul=0.4 hotdog.sh train_ft.py.
• Distance weighting. Paper uses w = 1/‖p − x‖ §3 eq. 4. The default kernel in the code, linear, uses 1 / (axis_w0 · sqrt(dx² + dy²) + axis_w1 · |dz|) point_aggregators.py. Scripts keep the default kernel. This is a different distance.
• Feature width. Paper gives 59 channels §5. Per-scene scripts use 32 hotdog.sh. The README's DTU training recipe uses 63, described as "as in paper", and 32, described as "better for per-scene optimization" README.md.
• COLMAP confidence. Paper sets the initial confidence of COLMAP points to 0.3 app. F. Per-scene COLMAP scripts use default_conf=0.15 col_ship.sh.
• Prune interval. Paper prunes every 10K iterations §4.2. The scripts set prune_iter=10001 hotdog.sh, which is the same schedule offset by one step.
• Pruning threshold. Paper: confidence below 0.1 §4.2. Code keeps points with confidence ≥ 0.1 neural_points.py. No discrepancy found.
• Neighbors. Paper: 8 nearest neighbors app. H. Code: K=8 hotdog.sh. No discrepancy found.
• Learning rate. Paper: Adam, 5e-4 §5. Per-scene scripts: lr=5e-4 hotdog.sh. No discrepancy found, but the code also sets a separate plr=2e-3 hotdog.sh that the paper does not mention.
• MVS filtering. Paper, appendix J: for NeRF Synthetic the MVSNet points are filtered with the alpha channel, and a second setting uses background color. The README confirms alpha-channel filtering for the MVSNet point cloud README.md and adds a background-color variant. The per-scene scripts use depth_conf_thresh=0.8 and geo_cnsst_num=5 hotdog.sh hotdog.sh, values the paper does not state.

Pitfalls

• Growing thresholds are missing. T_opacity and T_dist are symbols in §4.2 with no values, and the code uses a different growing procedure (see Paper vs code). An implementation must pick its own values and should say so.
• Sparsity loss details. Eq. 11 has no epsilon. Taking log of confidences at exactly 0 or 1 is undefined, so clamp them. The paper does not say how. The shipped scripts set the weight to 0, so the sparsity term may not be used in the released results at all.
• Equation and section numbers were counted from the LaTeX source. Check them against the PDF before citing. The appendix numbering in this card follows the appendix order in the source.
• Two blend normalizations. Eq. 4 and eq. 7 both normalize by Σ w_i. Use the same normalization for features and density. Confidence multiplies inside the sum, not in the denominator.
• Neighborhood query cost. The paper says the shading samples are placed only in grid cells near points, so empty space is skipped app. H. This makes the neighbor query central to speed. The paper does not give a query radius value, so measure what your radius covers.
• Initialization source differs. ScanNet results initialize points from dataset depth images, not from the MVS network. The appendix says this explicitly app. J, and it matters for any comparison on ScanNet.
• Timing claim conflict. Table 1 lists NeRF at 10 hours tab. 1, while the text says 20+ hours §6.2 and §7. Both are from the paper; treat the speed-up factor as author-reported.
• Background handling. Point-NeRF uses only local radiance and has no background model, so unbounded or outside-in scenes are not handled. The paper says so app. J.
• Feed-forward quality. Without per-scene optimization, the method is below IBRNet and MVSNeRF on DTU PSNR §6.1 tab. 1.
• Code license. The official code is CC BY-NC-SA 2.0. Reading it to check details is allowed for this card, but no verbatim code is quoted here, and implementers should not copy it into commercial products.
• Figures. Fig. 1 is a teaser that renders a field from generated points and from COLMAP points. Fig. 2 shows the feed-forward pipeline and the per-ray shading with K neighbors fig. 2. Fig. 3 shows the gradient paths for initialization and per-scene optimization fig. 3. Fig. 4 compares renders with and without pruning and growing on Hotdog fig. 4. Fig. 5 shows growing from 1000 COLMAP points on Chair fig. 5. Fig. 6 shows DTU qualitative results fig. 6 and Fig. 7 NeRF Synthetic results fig. 7. No figure is reproduced on this page.

Lineage

• Builds on: NeRF for volume rendering with ray marching §1 §3; MVSNet for the cost-volume depth network §4.1; PointNet-style aggregation of point sets §3; the grid point query inspired by CAGQ app. H; the confidence sparsity loss of Lombardi et al. 2019 §4.2.
• Compared against: PixelNeRF, MVSNeRF, IBRNet, NeRF, NSVF, NPBG, NV and SRN tab. 1 tab. 2 tab. 3 app. D app. E.
• Superseded by: not stated in this wiki.