DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

In plain words

A NeRF draws a picture by asking a neural network about many points along every camera ray. That is too slow for real-time uses such as games or VR. DONeRF adds a small helper network, the depth oracle. It looks at a ray once and says where the surfaces probably are, and the main network then checks only a few points near those places. The helper picks depth ranges instead of giving an exact distance, and its training targets are blurred so that it errs toward checking a little extra empty space. Points are spaced on a logarithmic scale and pulled toward the centre of the view area, so near and far parts both get a sensible share of samples. On the paper's own test scenes, the result is close to NeRF quality with a handful of samples per ray.

TL;DR

• DONeRF replaces NeRF's dense ray marching with two MLPs: a depth oracle that runs once per ray and classifies depth bins, and a shading MLP that marches only 2 to 16 samples placed around the oracle's peaks §3.
• Author-reported: mean PSNR 31.14 dB at 4 samples per ray, against 30.52 dB for NeRF at 256 samples, over six Blender scenes at 400×400 tab. 2.
• Training needs a reference depth per pixel. Without it (a dense NeRF supplies the depth), the paper reports a loss of about 1 dB §6.4; the averages in tab. 2 give gaps of 0.8 to 1.5 dB.
• The paper is CC BY 4.0. The official code is CC BY-NC 4.0, so this card does not use it.

Problem

• NeRF takes 64 coarse samples and then 64 plus 128 fine samples, which the paper counts as 256 network evaluations per ray app. E. The introduction estimates that a 1440×1600, 90 Hz VR headset would need about 37 petaFLOPS at that rate §1.
• The number of samples per ray dominates the cost of NeRF-style rendering §4.
• Local sampling around a known surface can reach nearly the same quality with 2 samples per ray, but it needs ground-truth depth at inference, which is usually not available §4, Conclusion.
• NeRF handles small scenes well but large scenes need splitting into several networks, which multiplies memory and evaluation cost §1.
• Assumptions: static synthetic scenes, a view cell that bounds where the camera may be, and training-time depth §1, §6.2. Sampled poses were limited to 30° pitch and 20° yaw from the initial direction, a limit the authors chose to stay comparable with plane-based methods §6.2.

Method

The pipeline has five stages: unify the ray descriptions inside a view cell, work in a non-linear depth space, let an oracle estimate sample importance, warp the sample positions toward the view cell, and shade with a small MLP using few samples per ray fig. 1.

View cell. A bounding box with a primary orientation and a maximum viewing angle. Training rays start inside the box and stay within the angle bounds §3, fig. 2. Per-scene box sizes are listed in app. C; their units are not stated.

Ray unification. Each ray origin is mapped onto a sphere that surrounds the view cell, so that identical rays get identical oracle inputs §5.2, fig. 9. The paper does not state how the sphere radius is set.

Non-linear depth. Samples are spaced by a logarithmic remapping of depth between the near and far distances §4, eq. 3. The oracle's classes use the same logarithmic spacing §5.2.

Space warp. After sampling, each position relative to the view cell centre is scaled by an inverse square root of its distance, so background points are pulled closer before positional encoding §4, eq. 4–5, fig. 3.

Depth oracle. A single MLP evaluated once per ray. It takes the unified ray and a set of 3D positions placed at the centres of its depth classes, which are fed to the first layer §5.2. It outputs one score per class and is trained as a classifier, with binary cross-entropy against a filtered target §5.1, §6.1. The oracle has no skip connections and no positional encoding app. D. The number of classes is not stated for the main model; the 16-class example in fig. 7 is a visualisation.

Classified depth target. For a ray whose first surface lies at depth d_s, the class containing d_s is set to 1 and all others to 0 §5.1, eq. 6. Two filters then soften this target:

• An image-space filter takes, for each class, the maximum over neighbouring rays of their value minus a distance penalty that grows to 1 at the window edge §5.1, eq. 7, fig. 7. The window is K×K.
• A depth-axis filter applies a triangular kernel of size Z, sums the weighted neighbours and clips the result at 1 §5.1, eq. 8, fig. 6.

The filters apply only to training targets; nothing is filtered at inference §5.1.

Sample placement. The oracle's scores are treated as a piecewise-constant probability density along the ray. Its cumulative distribution is inverted, and samples are taken at uniform steps of the cumulative value §5.1, fig. 8. This follows NeRF's coarse-to-fine step, with the oracle's output replacing NeRF's opacity-derived density §5.1.

Shading network. Eight layers of 256 units with one skip connection that forwards the encoded direction to the last layer app. D, fig. 13. It receives the warped and encoded positions and returns colour and opacity per sample, and the samples are accumulated along the ray §3.

Losses. Mean squared error on RGB and binary cross-entropy on the oracle's classes §6.1. The shading network also gets an opacity term that is zero when the accumulated opacity is at least 1 and otherwise penalises the squared shortfall §6.1, eq. 9. The total is alpha times the MSE plus beta times the opacity term §6.1, eq. 10. The opacity term was added because low sample counts produced dark artefacts on black backgrounds §6.1.

Training order. The ablation trains the oracle first, then the shading network on the oracle's predictions §5.3. Each network trains for 300,000 iterations, and the checkpoint with the lowest validation loss is used §6.1.

Without ground-truth depth. Depth maps are extracted from a dense NeRF with 12 layers of 512 units and 128 samples per ray, for all training poses, and then used as if they were ground truth §6.1.

# DONeRF inference for one ray  [§3] [§4] [§5]
# Input:  origin (3,), direction (3,), view cell centre c, d_max
# Output: colour (3,)

def render_ray(origin, direction, c, d_max, num_samples):
    # 1. Map the ray start onto a sphere around the view cell  [§5.2]
    unified_origin, unified_dir = unify_ray(origin, direction, c)
    # 2. One oracle evaluation scores every depth class  [§5.1] [§5.2]
    #    Classes are log-spaced; their number is not stated.
    class_scores = oracle(unified_origin, unified_dir)      # (C,)
    # 3. Read the scores as a piecewise-constant density and take
    #    samples at uniform steps of its inverse CDF  [§5.1] [fig. 8]
    #    Depths are on the log-spaced class axis  [§5.2]
    depths = sample_inverse_cdf(class_scores, num_samples)  # (N,)
    # 4. Place the samples along the ray  [§4] [eq. 1]
    points = origin + depths * direction                     # (N, 3)
    # 5. Warp each offset from the view cell centre, then encode
    #    [§4] [eq. 2, eq. 4-5]
    features = encode(warp_offset(points - c, d_max))        # (N, F)
    # 6. Shading MLP gives colour and opacity per sample  [§3] [app. D]
    colours, opacities = shading_mlp(features, direction)
    # 7. Accumulate along the ray as in NeRF  [§3]
    return composite(colours, opacities)


# Radial warp used before encoding  [eq. 5]: v / sqrt(|v| * d_max)
def warp_offset(offset, d_max):
    return offset / sqrt(norm(offset) * d_max)

# Oracle training target for one ray  [§5.1] [eq. 6-8]
# Inputs: reference depth d_s of the first surface, class bounds d_z
# Output: soft class target (C,) for the binary cross-entropy loss

def oracle_target(ray_index, depth_map, bounds, K, Z):
    # 1. One-hot class of the first surface along this ray  [eq. 6]
    hard = one_hot_class(depth_map[ray_index], bounds)      # (C,)
    # 2. Spread each class to neighbouring rays with a distance
    #    penalty, keeping the maximum  [eq. 7]
    spread = neighbourhood_max(hard, ray_index, K)          # (C,)
    # 3. Smooth along depth with a triangle kernel of size Z,
    #    then clip at 1  [eq. 8]
    smooth = triangle_filter_clipped(spread, Z)             # (C,)
    return smooth

# Shading loss  [§6.1] [eq. 9-10]; alpha = 1.0, beta = 10.0
# Inputs: predicted and ground-truth RGB, per-sample opacities delta_i
# Output: scalar loss

def shading_loss(pred_rgb, gt_rgb, deltas, alpha, beta):
    # 1. Mean squared error on RGB  [§6.1]
    mse = mean((pred_rgb - gt_rgb) ** 2)
    # 2. Accumulated opacity per ray, sum of delta_i  [eq. 9]
    total_opacity = sum(deltas)                             # (rays,)
    # 3. Zero when opacity reaches 1, else the squared shortfall
    opacity_term = where(total_opacity >= 1, 0, (total_opacity - 1) ** 2)
    # 4. Reduction over rays is not stated; a mean is assumed here
    opacity_loss = mean(opacity_term)
    # 5. Weighted sum  [eq. 10]
    return alpha * mse + beta * opacity_loss

Hyperparameters

Results

Author-reported. Setup: six Blender scenes (Bulldozer, Forest, Classroom, San Miguel, Pavillon, Barbershop) at 400×400 §6, app. C. The NeRF baseline is a PyTorch port run with DONeRF's network layout; NSVF, LLFF and NeX use the authors' released code, with NeX's suggested parameters for an 11 GB memory budget app. E. Timings are on one RTX 3090 with TensorRT in half precision §6.1. Metrics are PSNR and FLIP §6.

Table 1: ablation of depth-oracle configurations, averaged over all scenes tab. 1. Columns are the number of samples per ray N. SD is a single depth output; SD unified adds ray unification; K, Z and I are the neighbourhood size, the depth-filter size and the number of oracle inputs.

The paper attributes about 0.6 dB to ray unification, 0.6 to 1.5 dB to classification, about 1 dB to multiple oracle inputs, 1.3 to 1.8 dB to the neighbourhood filter and 0.1 dB to the depth filter §5.3. The main model is K-5 Z-5 I-128. At N=8 and N=16, K-9 Z-1 I-128 is marginally higher in PSNR tab. 1.

Table 2: main comparison tab. 2. Each cell is PSNR / FLIP (FLIP lower is better). Storage is in MiB and the cost is in MFLOP per pixel.

Headline claims from the text: DONeRF-4 matches or beats NeRF on average with 4 samples against 256 §6.3; DONeRF-16 is up to 7 dB above NeRF at the same resolution §6.3; and the paper claims 48× to 78× fewer FLOP per pixel than NeRF at equal or better quality tab. 2 caption. The per-scene breakdown in the appendix tables is summarised here rather than reproduced: tab. 3 to tab. 6 give oracle ablations per scene at N=2 to 16, and tab. 7 to tab. 10 give the sampling-strategy ablations per scene.

Timing §6.1. At 800×800 with 2 samples, the stages take 0.21 ms (ray unification and first features), 12.64 ms (oracle), 3.46 ms (warp and second features), 34.9 ms (shading) and 0.09 ms (colour), for 51.3 ms in total, or about 20 frames per second. Shading time grows roughly linearly: 34.9, 65.4, 136.4 and 270.4 ms for 2, 4, 8 and 16 samples. The paper states that shading is the main cost.

Paper vs code

No code comparison was made. The authors' repository facebookresearch/DONERF is linked from the introduction §1. Its LICENSE.md is Attribution-NonCommercial 4.0, which excludes commercial use, and wikillm carries advertising. This card is therefore written from the paper alone; no code, configuration or script from the repository is cited or quoted. The commit is recorded in the frontmatter for reference.

Pitfalls

• K=1 is undefined in eq. 7. The radial term divides by floor(K/2), which is zero for K=1. The ablation includes K-1 rows tab. 1, but the paper never says how the unfiltered case is computed. Treat K=1 as no image-space filter, and note that this is an inference, not a statement in the paper.
• Eq. 7 can go negative, but the result is not. Distant neighbours get a negative value, while the centre ray (distance 0) contributes its own class value, which is never negative. So the maximum stays at or above 0 and matches the paper's "0 contribution" wording. This follows from the equation; the paper does not spell it out.
• Eq. 8 is not normalised. The triangle weights sum to more than 1 when several classes overlap, and the clip at 1 is the only control. For a single one-hot target and Z=5, the neighbours get 2/3 and 1/3 of the peak, derived from the kernel shape.
• Sample count versus evaluation count. Eq. 1 uses i from 0 to N, which gives N+1 depth values, while the paper speaks of N samples. The paper does not say how the endpoints are handled.
• Log remapping depends on units. Eq. 3 adds 1 inside the logarithm, so the result depends on the scene's length unit. Near and far distances and the view-cell size units are not stated, so the mapping cannot be reproduced from the paper alone.
• Warp at the view-cell centre. Eq. 5 divides by sqrt(|p| · d_max). At p = 0 the expression is undefined, and the paper does not address this.
• Opacity δ_i is undefined. Eq. 9 uses per-sample opacities δ_i without defining them; they are presumably NeRF's standard accumulation weights. The reduction over rays in the opacity loss is also not stated; the pseudocode assumes a mean.
• Oracle frozen or not during shading training? §5.3 says the shading network is trained "using the oracle's predictions" but does not say whether the oracle's weights are then frozen.
• Ground-truth depth is a one-value-per-ray target. The main model uses the first surface only. The paper says that transparent surfaces and mirrors are an issue for this target and that multiple surface depths would be a straightforward extension §6.4, Conclusion. The noGT variant also outperforms the GT variant for Pavillon and Barbershop at higher sample counts §6.4.
• Baselines are not identical. The NeRF baseline is a PyTorch port with the same 8×256 layout as DONeRF, not the original NeRF network, and uses 2048 samples per iteration against DONeRF's 4096 app. E, §6.1. The comparison is therefore at matched capacity but not matched training recipe.
• Speed-up figures disagree across the paper. The abstract says up to 48×, the introduction 24× to 98×, §6.3 15× to 78×, and the table caption 48× to 78× fewer FLOP. The table's FLOP column reproduces the §6.3 range (211.42 / 14.29 ≈ 15, 211.42 / 2.70 ≈ 78). These are FLOP ratios, not measured speed-ups. The paper reports no NeRF wall-clock time.
• Timing numbers come from different resolutions. The teaser reports 22 ms per frame at 400×400 for DONeRF-4 at 4.35 MFLOP per pixel. §6.1 reports 51.3 ms at 800×800 for 2 samples. The paper gives no timing table at 400×400.
• Dataset assets have their own licences. The scenes come from Blender demo files and other third-party sources app. C and are not restated here. Check their licences before reusing the scenes or the paper's datasets.
• Gaps versus NeRF depend on sample count. Averaged over the scenes, DONeRF-2 (29.50 dB) and DONeRF-2-noGT (28.26 dB) are below NeRF (30.52 dB), while DONeRF-4 (31.14 dB) and the larger counts are above it tab. 2. Read the per-count rows before quoting "beats NeRF".

Lineage

• Builds on: NeRF (Mildenhall et al., 2020), whose coarse-to-fine sampling DONeRF replaces with a single oracle evaluation §5.1, app. E.
• Compared against: NeRF, NeRF with log+warp sampling, NSVF in three sizes, LLFF and NeX (with and without an MPI) §6.3, app. E.
• Related sample-reduction work named in the paper: empty-space skipping by self-pruning voxels in NSVF §2, and multiple smaller networks in DeRF §2, §4.
• Superseded by: not known from content already in this wiki.