Empty space skipping

Overview

When a renderer samples points along each ray, most of them often lie in empty air and contribute nothing to the pixel. Empty space skipping (ESS) stores a coarse map of which regions of the scene are occupied, and does not evaluate samples that fall into regions marked empty. Only the remaining samples reach the network 2103.13744 §3.4. The map is a uniform occupancy grid with one bit per cell. Its resolution is 16 times the resolution of the tiny-network grid, so at most 256 cells per axis 2103.13744 §3.4 2103.13744 §4.1.

Two cards cover this concept. KiloNeRF is the grid-based variant. DONeRF reaches the same goal, fewer network evaluations on empty space, without an occupancy grid: it places a few samples near surfaces that a learned oracle predicts. The paper names NSVF's self-pruning voxels as ESS-style work 2103.03231 §2; NSVF's paper is at arXiv:2007.11571.

The KiloNeRF paper uses ESS in two places: during fine-tuning and during inference 2103.13744 §3.4. It pairs ESS with early ray termination (ERT), which stops sampling a ray once its remaining transmittance drops below a threshold. ERT applies at inference only 2103.13744 §3.4. NSVF, the baseline the paper compares against, is also reported to use ESS and ERT 2103.13744 §4.1.

Variants

Occupancy grid from a trained density field (KiloNeRF)

The occupancy grid is extracted from a trained NeRF teacher, not from the images. Each occupancy cell is filled by querying the teacher's density on a 3x3x3 sub-grid inside the cell. The cell is marked occupied if any of those densities exceeds a threshold of 10 2103.13744 §3.4 2103.13744 §4.1. The grid is then fixed for fine-tuning and rendering. Card: KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs.

The paper reports the effect of ESS and ERT together on a single scene. On Lego, NeRF with ESS and ERT renders in 788 ms, against 56185 ms for the baseline NeRF, a factor of 71. The tiny networks give a further factor of about 36 2103.13744 tab. 2 2103.13744 §4.3. The paper does not report ESS alone without ERT, so the separate contribution of ESS is not known from the card.

Early ray termination as a companion step

ERT is not ESS, but the paper uses it in the same render loop. A ray leaves the active set once its transmittance falls below 0.01. Samples are evaluated in passes ordered by distance from the camera, so terminated rays drop out between passes 2103.13744 §3.4 2103.13744 §4.1. Samples are taken stratified and equidistant, at most 384 per ray 2103.13744 §3.4 2103.13744 §4.1.

Depth oracle instead of a grid (DONeRF)

DONeRF keeps no occupancy grid. A single MLP, the depth oracle, is evaluated once per ray. It outputs one score per depth class. The scores are read as a piecewise-constant density, and samples are placed at uniform steps of its inverse CDF, so most of them fall where the oracle expects a surface 2103.03231 §3 2103.03231 §5.1. A shading network then evaluates only the placed samples, 2 to 16 per ray 2103.03231 §3. The oracle is trained against a per-ray depth target that is blurred in image and depth space, and the blur is not applied at inference 2103.03231 §5.1. Card: DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.

Author-reported, over six Blender scenes at 400×400: mean PSNR is 31.14 dB at 4 samples per ray and 29.50 dB at 2, against 30.52 dB for NeRF at 256 samples 2103.03231 tab. 2. The paper's timing breakdown covers 2 samples only: 51.3 ms per 800×800 frame on an RTX 3090 with TensorRT in half precision, of which 34.9 ms is shading 2103.03231 §6.1. The paper gives no wall-clock time for NeRF, so no measured speed-up against NeRF is reported 2103.03231 §6.3.

When to use which

Implementation sketch

Pseudocode in our own words. The occupancy build runs once, before training or rendering. The render loop is the common core: samples are filtered by the grid, evaluated, composited, and rays are dropped when they become opaque.

# Build the occupancy grid once from a trained density field  [§3.4]
def build_occupancy(density_fn, bbox_min, bbox_max, cells, sub=3, tau=10.0):
    # one bit per cell; cells are 16x finer than the network grid  [§4.1]
    occ = zeros(cells, dtype=bool)
    for c in all_cells(cells):
        # query a sub x sub x sub grid of points inside the cell  [§3.4]
        pts = sub_grid_points(c, sub, bbox_min, bbox_max)
        # the cell is occupied if any density exceeds tau  [§3.4] [§4.1]
        occ[c] = max(density_fn(pts)) > tau
    return occ

# Render with ESS and ERT  [§3.4] [§3.5]
def render(rays, field, occ, max_samples=384, eps=0.01):
    # per-ray state for the compositing of eq. 2-4  [eq. 2] [eq. 3]
    color = zeros(num_rays, 3)
    trans = ones(num_rays)
    live = all_rays
    # visit samples nearest-first, in passes  [§3.4]
    for k in range(0, max_samples, pass_size):
        if not live:
            break
        pts = samples_for(live, k, pass_size)
        # ESS: drop samples whose occupancy cell is empty  [§3.4]
        pts = pts[occ[cell_index(pts)]]
        # evaluate only what is left, grouped by network  [§3.5]
        rgb, sigma = field(sort_by_cell(pts))
        color, trans = composite(color, trans, rgb, sigma)
        # ERT: retire rays whose transmittance fell below eps  [§3.4]
        live = live[trans[live] >= eps]
    return color

Where variants differ: whether a grid is used at all (DONeRF replaces the occupancy test with one oracle evaluation per ray, and the DONeRF paper card has its own sampler pseudocode), the grid source (a teacher model in KiloNeRF), the sub-sampling used to fill a cell, the threshold, and whether ESS runs during training. The papers' values for these are in the Variants section.

Contradictions

Speed-up figures are not comparable across the two papers. KiloNeRF reports measured render times against NeRF on a GTX 1080 Ti 2103.13744 tab. 1. DONeRF's speed-up range is a ratio of FLOP per pixel, not a measured time, and the paper reports no NeRF wall-clock time 2103.03231 tab. 2 2103.03231 §6.3. The DONeRF paper also gives different FLOP-ratio ranges in different places, including §6.3 and the table caption 2103.03231 §6.3 2103.03231 tab. 2. Neither figure should be quoted as a like-for-like speed-up.

Papers

• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs: uses an occupancy grid from a teacher NeRF, with ESS during fine-tuning and inference and ERT at inference, for a reported 1258-2167x render speed-up over NeRF.
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks: a per-ray depth oracle places 2 to 16 shading samples without an occupancy grid; author-reported 31.14 dB mean PSNR at 4 samples per ray.