Plenoxels: Radiance Fields without Neural Networks

In plain words

Given a set of photos of an object or a scene taken from known camera positions, the goal is to build a 3D model that can show the scene from new viewpoints. The recent standard for this, Neural Radiance Fields (NeRF), stores the scene inside a neural network and needs many hours or days of GPU time to train.

Plenoxels skip the network. The scene is stored as a sparse grid of small 3D cells. Each occupied cell keeps a transparency value and a few numbers that describe how its colour changes with the viewing angle. To draw a pixel, the method walks along the camera ray, blends the colours of the cells it passes through, and compares the result with the real photo. The grid values are then nudged to reduce that error, with a small penalty that keeps neighbouring cells smooth.

Training starts with a coarse grid. Empty cells are removed, and the remaining cells are split into smaller ones to add detail. The idea is that the blending step, not a neural network, is what makes this kind of reconstruction work. The result is training that is much faster than NeRF's and can run on a single GPU.

TL;DR

• A grid-based alternative to NeRF: the scene is a sparse voxel grid with opacity and degree-2 spherical harmonic colour per occupied voxel corner, optimised directly from calibrated images without any neural component §3 §3.2.
• Author-reported: about 11 minutes per bounded synthetic scene on one Titan RTX, against roughly one day for NeRF, at comparable quality (31.71 vs 31.85 PSNR for JAXNeRF on the 8 NeRF synthetic scenes) §1 tab. 2.
• Works on unbounded scenes by adding a normalised-device-coordinate mode for forward-facing scenes and a multi-sphere background for 360° scenes §3.6 §4.2 §4.3.
• Use it when you need fast per-scene radiance field fitting and can run CUDA code; do not copy its numbers without the setup, since they are author-reported and the JAXNeRF baseline was rerun by the authors §4.1 app. D.

Problem

Neural radiance fields reconstruct photorealistic views from calibrated images, but training takes more than a day on one GPU, and rendering one image needs about 30 seconds in the original setting. Later work sped up rendering, but single-GPU training still took hours §1. The authors aim to cut training time by about two orders of magnitude while keeping NeRF-level quality, with the method running on one GPU §1 §4.

The method assumes calibrated camera poses and images with the usual volume rendering model. It is tuned per dataset, with fixed regularisation weights and fixed learning rates across scenes §3.7 §5.

Method

The model is a sparse voxel grid. Each occupied voxel corner stores an opacity value and a vector of spherical harmonic (SH) coefficients for each colour channel. Opacity and colour at any point are obtained by trilinear interpolation of the stored corner values, and colour is evaluated at the viewing direction §3 fig. 2.

Rendering §3.1. Colour of a ray is computed with the volume rendering formula used in NeRF, which integrates over samples along the ray eq. 1. The authors note that the formula is approximate, since it assumes single scattering and constant values between samples, but it is differentiable, so each training ray gives a gradient §3.1.

\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \bigl(1 - \exp(-\sigma_i \delta_i)\bigr) \mathbf{c}_i, \qquad T_i = \exp\Bigl(-\sum_{j=1}^{i-1} \sigma_j \delta_j\Bigr) \qquad [\text{eq. 1}]

Here \sigma_i is the opacity of sample i, \mathbf{c}_i its colour and \delta_i the distance to the next sample §3.1 eq. 1.

Sparse grid §3.2. Instead of an octree, the authors keep a dense array of indices. Each cell is either empty or points into a separate table that holds the values of occupied voxels only §3.2 app. B. The paper uses degree-2 spherical harmonics: 9 coefficients per colour channel, 27 per voxel. Degree 2 is used because PlenOctrees reported little benefit from higher degrees §3.2. Unlike PlenOctrees, which treats the values as constant inside each voxel, this model interpolates them trilinearly, which the authors find important for optimisation §3.2 §3.3. All coefficients are optimised directly, with no special initialisation and no pretraining §3.2.

Interpolation §3.3. Opacity and colour at each sample are interpolated from the nearest 8 stored voxels. Trilinear interpolation improves both the effective resolution and the optimisation, and it is more stable than nearest neighbour interpolation over a range of learning rates §3.3 tab. 1.

Coarse to fine §3.4. The grid starts dense at a low resolution, is optimised, is pruned, and then each remaining voxel is split in half in every dimension. The new values are initialised by trilinear interpolation, and the same mechanism can resize the grid to any resolution §3.4. Pruning uses the PlenOctrees method: a voxel is removed when its maximum rendering weight over all training rays is below a threshold, or the threshold is applied to the density value in each voxel instead §3.4. Because trilinear interpolation would otherwise erode surfaces, a voxel is removed only if it and its neighbours are all unoccupied, so a dilation step runs first §3.4.

# Render one ray with the NeRF volume rendering formula  [§3.1] [eq. 1]
# Input:  sample positions along the ray, step lengths
# Output: RGB colour of the ray

def render_ray(grid, origin, direction, positions, deltas):
    # Accumulated colour and transmittance so far  [eq. 1]
    color = zeros(3)
    transmittance = 1.0
    for i in range(len(positions)):
        # Trilinear interpolation of opacity and SH coefficients from the
        # 8 nearest stored corners  [§3.3]
        sigma, sh = trilinear(grid, positions[i])
        # Colour: SH basis at the view direction, weighted by the
        # coefficients; negative values clipped to zero  [app. B]
        c = clip_nonnegative(sum_sh(sh, direction))
        # Share of the light absorbed at sample i  [eq. 1]
        alpha = 1.0 - exp(-sigma * deltas[i])
        color = color + transmittance * alpha * c
        # Light that continues past sample i  [eq. 1]
        transmittance = transmittance * (1.0 - alpha)
    return color

# Coarse-to-fine upsampling step  [§3.4] [app. B]
# Input:  current grid, weight threshold, target resolution
# Output: grid at the target resolution with only kept voxels

def prune_and_upsample(grid, threshold, target_resolution):
    # Score each occupied voxel by its largest rendering weight over the
    # training rays  [§3.4]
    scores = max_weight_per_voxel(grid, training_rays)
    # Keep voxels above the threshold, then dilate the kept mask so a voxel
    # is removed only if it and its neighbours are unoccupied  [§3.4]
    keep = dilate(scores > threshold)
    # Resample the kept grid to the target resolution by trilinear
    # interpolation  [§3.4]
    return resample(grid, keep, target_resolution)

Optimisation §3.5. The loss is the mean squared error over rendered ray colours plus a total variation (TV) term weighted by \lambda_{TV} eq. 2. The TV term penalises differences between neighbouring voxel values in all three axes, separately for opacity and each SH channel §3.5. It is evaluated on a random sample of voxels each step, and the MSE on a random batch of rays §3.5. The optimiser is RMSProp, chosen because a second-order method would need a Hessian that is too large to form and invert at this scale §3.5. The learning rate schedule follows JAXNeRF and Mip-NeRF, with a separate initial rate for opacity and for SH coefficients §3.5 app. B.

\mathcal{L} = \mathcal{L}_{recon} + \lambda_{TV}\,\mathcal{L}_{TV} \qquad [\text{eq. 2}]

The MSE term averages the squared error between rendered and ground-truth colours over the rays in a batch. The TV term is the average over voxels of the square root of the summed squared differences along the three axes, with each difference normalised by the grid resolution §3.5 app. B. The exact normalisation in the supplement is relative to a resolution of 256 app. B.

Unbounded scenes §3.6. Forward-facing scenes use the same grid with normalised device coordinates from NeRF §3.6 §4.2. For 360° scenes, a background model of 64 concentric spheres stores RGB and opacity on an equirectangular voxel grid, interpolated trilinearly within and between spheres. The spheres are spaced linearly in inverse radius from 1 to infinity, and the inner scene is scaled to fit in the unit sphere §3.6 app. B.

Regularisation §3.7. TV is used on all scenes §3.7 fig. 3. Real forward-facing and 360° scenes also get a sparsity loss, a Cauchy-type term on opacity taken from SNeRG eq. 3. The 360° scenes add a beta-distribution loss on the accumulated foreground transmittance, taken from Neural Volumes eq. 4 §3.7.

\mathcal{L}_s = \lambda_s \sum_{i,k} \log\bigl(1 + 2\,\sigma(\mathbf{r}_i(t_k))^2\bigr) \qquad [\text{eq. 3}]

\mathcal{L}_{\beta} = \lambda_{\beta} \sum_{\mathbf{r}} \bigl(\log T_{FG}(\mathbf{r}) + \log(1 - T_{FG}(\mathbf{r}))\bigr) \qquad [\text{eq. 4}]

Here \sigma(\mathbf{r}_i(t_k)) is the opacity of sample k on ray i, and T_{FG}(\mathbf{r}) is the accumulated foreground transmittance of ray \mathbf{r} §3.7 eq. 3 eq. 4.

# Plenoxel optimisation loop  [§3.5] [app. B]
# Input:  calibrated images, rays, resolution schedule, step count
# Output: sparse grid of opacity and SH coefficients

def optimise(rays, schedule, steps):
    # Start with a dense grid at the coarsest resolution  [§3.4] [app. B]
    grid = dense_grid(schedule.resolutions[0])
    for step in range(steps):
        # Sample a batch of rays; 5000 per step  [app. B]
        batch = sample_rays(rays, count=5000)
        # Render colours and compare with the ground truth  [§3.1] [eq. 2]
        predicted = render(grid, batch)
        loss = mean_squared_error(predicted, batch.colors)
        # TV on a random 1% of voxels per step  [§3.5] [app. B]
        loss = loss + schedule.lambda_tv * total_variation(grid, 0.01)
        # Sparsity and beta losses, where the scene type uses them  [§3.7]
        loss = loss + extra_regularisers(grid, batch, schedule)
        # RMSProp update with separate learning rates  [§3.5] [app. B]
        grid = rmsprop_step(grid, loss, schedule.learning_rates(step))
        # Upsample after the scheduled steps  [§3.4] [app. B]
        if step in schedule.upsample_steps:
            grid = prune_and_upsample(grid, schedule.next_resolution())
    return grid

Hyperparameters

Values are from the paper and its supplement app. B unless marked as code. The paper does not give the sampling step size, the initial opacity or the background learning rates. Those rows come from the official code and are labelled as such.

Results

Author-reported, on one Titan RTX GPU §1. The baselines are the NeRF results reported in the NeRF paper (for Neural Volumes) and the authors' own JAXNeRF reruns, which fixed a pixel-centering bug §4.1 app. D. Forward-facing and 360° columns are also author-reported. Per-scene numbers are in app. D.

Main comparison tab. 2:

The LLFF row has no training time in the table, because LLFF pretrains an MPI-predicting network that is amortised across scenes tab. 2. The NeRF rows for the synthetic and forward-facing settings are commented out in the source tab. 2; the synthetic NeRF numbers (31.01 / 0.947 / 0.081) appear only in the view-count ablation tab. 3.

Ablations (synthetic scenes, averaged over the 8 scenes):

The learning rate schedule and optimiser ablations are in app. C. On the synthetic scenes they move PSNR by up to about 0.4 dB (schedule) and 1.2 dB (optimiser). On the forward-facing scenes, removing TV on opacity costs about 1.0 dB of PSNR and removing TV on SH about 0.5 dB app. C fig. 3.

Timing (rounded in the main text): 11 minutes per synthetic scene, 24 minutes per forward-facing scene and 27 minutes per 360° scene tab. 2. The per-scene means are 11.1, 24.2 and 27.3 minutes app. D. The authors report good quality after the first epoch, which takes under 1.5 minutes §4; fig. 5 shows the synthetic scenes after one epoch, at an average of 1 minute 20 seconds.

Paper vs code

Official code: https://github.com/sxyu/svox2 at commit ee80e2c4df8f29a407fda5729a494be94ccf9234, BSD-2-Clause. The paper's project page at alexyu.net/plenoxels links this repository, and the owner is the first author. The repository holds a CUDA package (svox2/) and training scripts (opt/). The configs for synthetic, forward-facing and 360° scenes match the paper's supplement on the values checked here. The differences below are where the code goes beyond or differs from the text.

• Colour offset. The paper defines colour as the SH combination of the coefficients §3.2. The renderer adds 0.5 to that sum, then clips at zero svox2.py. The offset is not in the paper.
• Opacity clip. The paper uses opacity directly in eq. 1 §3.1. The renderer applies ReLU to the interpolated opacity before the exponential svox2.py.
• Weight threshold. The paper gives a threshold of 0.256 for synthetic scenes app. B. The code's default is 0.0005 × 512, and it divides that value by the z-resolution at each upsample opt.py opt.py. The effective threshold therefore depends on the grid's z-resolution, which the paper does not say.
• Dilation. The paper describes one dilation step §3.4 without a count. The training code calls dilation with dilate=2 opt.py, and the function applies that many dilation passes svox2.py.
• TV epsilon. The paper's TV term has no epsilon §3.5. One CUDA kernel adds 1e-5 under the square root loss_kernel.cu.
• Beta-loss epsilon. The backward pass of the beta loss adds 1e-3 to a denominator render_lerp_kernel_cuvol.cu. The paper does not state this numerical detail.
• Background learning rates and background threshold. The paper states that the background uses plain exponential decay without the delay app. B. The code default sets background opacity from 3 to 3e-3 and colour from 0.1 to 5e-6 opt.py, and a background density threshold of 1 + 1e-9 opt.py. The paper gives none of these values.
• Forward-facing background brightness. The forward-facing and 360° configs set background_brightness to 0.5 and 1.0 llff.json tnt.json. The paper does not state this value.
• Sparsity gradient. Eq. 3 matches the gradient in the render kernel, 4σ / (1 + 2σ²) render_lerp_kernel_cuvol.cu.
• TV sampling. The paper says 1% of voxels per step app. B. The code default is 0.01 opt.py, and it applies TV to contiguous chunks of the pointer grid for speed app. B opt.py. Contiguous chunks are not uniformly random voxels.

Pitfalls

• Author-reported numbers and comparisons. Times and PSNR come from the authors' own runs, on one GPU. JAXNeRF was rerun with a pixel-centring fix, and its times were measured on four GPUs and multiplied by four app. D. Treat the speed comparison as indicative, not a controlled benchmark.
• Inconsistent LPIPS for the same setting. The synthetic-scene headline reports LPIPS 0.049 in the main table tab. 2 and 0.050 in the ablation tables tab. 3 tab. 4. The per-scene mean is 0.049 app. D.
• Regularisation is tuned per dataset. TV weights were fixed per dataset and not per scene. The paper says a sweep per scene would likely improve results §5 app. B. Expect to retune when you move to new data.
• TV is switched off after the first upsample on synthetic scenes. This is a deliberate choice; the authors suspect it is because the synthetic scenes have 100 training views and little noise app. B. Forward-facing and 360° runs keep TV on.
• Sparse gradients. Within the first 1 to 2 minutes, fewer than 10% of voxels get non-zero gradients §3.8 fig. 4. This is why the implementation is fast, but it also means most voxels get no updates for long periods.
• Hidden memory cost. The SH coefficients, their gradients and the RMSProp averages all have the same shape as the stored values app. B. The code caps grid elements at 44,000,000, sized for 22 GB opt.py.
• CUDA is required. The optimisers and the TV term assert a CUDA build svox2.py. The paper also describes a slower, higher-level JAX implementation §3.8; the cloned repository's training path depends on the CUDA package.
• Interactive rendering is not the same as fast rendering. The authors say their implementation is not optimised for rendering and reach about 15 fps §1. A PlenOctree conversion is suggested for faster rendering §1.
• Background model is simpler than the foreground. The 360° background uses only RGB, with no SH, and no coarse-to-fine schedule app. B.
• Unstated values to fill in. The sampling step, the initial opacity and the background learning rates are not in the paper. The code defaults above are the only available values, and a reimplementation should either match them or record the change.
• Scope of the claim. The paper argues that differentiable volume rendering, not the neural network, is what makes NeRF work §1 §5. That is the paper's interpretation of its own results, not an established result.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020), which supplies the volume rendering formula and the forward-facing normalised device coordinates §3.1 §3.6; PlenOctrees (arXiv:2103.14024), which supplies the spherical harmonic representation and the pruning rule §3.2 §3.4; Neural Volumes, which supplies the beta loss §3.7.
• Compared against: Neural Volumes, JAXNeRF, LLFF and NeRF++ tab. 2.
• Superseded by: not known from content already in this wiki.