Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

In plain words

A neural radiance field can draw a still object from new camera angles, but it is slow to draw. A faster older trick stores the answers in a tree of small cubes, so the computer only looks things up instead of running a network for every pixel. That trick works for still objects. Moving people or objects are harder, because each cube's answer changes from frame to frame.

The authors store, for each cube, a short list of wave-like numbers that describe how the cube's density and colour change over time. Like a compressed audio clip, a few of these numbers capture the overall motion, so the tree stays small. They also build the tree faster by letting a generalizable network predict the cube values from a few camera views, then blending the predictions from many views. A final short tuning pass sharpens the result. The paper reports real-time playback of these moving scenes from new viewpoints.

TL;DR

• Extends PlenOctree (a tree of precomputed radiance values) to dynamic scenes by storing Fourier coefficients per leaf instead of per-frame values §4.1.
• Builds the tree in minutes with a coarse-to-fine fusion of a generalizable NeRF (IBRNet-style) prediction, instead of days of per-scene training §3, §4.2.
• Fine-tunes the coefficients directly with an L2 image loss §4.3, eq. 13.
• Reports 100 fps at 800×800 and the best PSNR, SSIM, MAE and LPIPS against four dynamic-scene baselines on the authors' datasets §5.1, tab. 1, tab. 2.
• Limitations the authors state: dense input views, larger storage and GPU memory than MLP methods, and poor handling of large motions § Discussion.

Problem

Free-viewpoint video for telepresence and VR needs fast generation and real-time rendering §1. NeRF is accurate but too slow to render in real time, and PlenOctree-style trees fix rendering speed for static scenes [§1, §2]. Dynamic methods of the time, such as deformation-based NeRFs and Neural Body, need long training and rendering times §2, §5.1, tab. 2. The paper also notes that the octree structure of a moving object changes from frame to frame, which breaks a direct per-frame encoding §4.2.

Method

The method has two parts. Section 3 builds a static PlenOctree quickly from a generalizable NeRF. Section 4 extends that tree to a time-varying one by replacing each leaf's per-frame values with Fourier coefficients.

Static stage: generalized PlenOctree fusion §3

A PlenOctree leaf stores a density σ and spherical-harmonic (SH) coefficients z, and the colour seen from a direction (θ, φ) is a sigmoid of a weighted sum of SH basis functions §3, eq. 1. The paper's problem is that measuring σ and z for each leaf is slow §3.

1. Coarse tree from 6 views. Initialise a coarse tree with N³ voxel leaves §3. The generalizable network Ψ predicts images for 6 sparse views around the object, computed from multi-view images and silhouettes §3. A shape-from-silhouette visual hull limits which leaves are kept §3.

2. Per-leaf view-independent values. For each leaf inside the hull, the method samples directions (θ, φ) and queries Ψ for view-dependent σ(θ, φ) and colour c(θ, φ). These become view-independent values by taking an expectation: σ = E[σ(θ, φ)] and z = E[SH(θ, φ) · c(θ, φ)] §3, eq. 2.

3. Fine fusion over 100 dense views. Render 100 dense views from the coarse tree. For each view i, query Ψ only where the transmittance T_i > 1e-3 §3. Update each leaf with weights T_i §3, eq. 3, eq. 4:

   σ_i = (W_{i-1} σ_{i-1} + T_i σ_i) / (W_{i-1} + T_i) eq. 3

   z = (W_{i-1} z_{i-1} + T_i · SH(θ_i, φ_i) · c(θ_i, φ_i)) / (W_{i-1} + T_i) eq. 4

4. Weight and count. Keep an update count C_i = C_{i-1} + 1 and a weight W_i = ((C_i − 1) / C_i) W_{i-1} + T_i / C_i §3, eq. 5.

5. Prune. Remove leaves with σ < 1e-3 §3.

The paper says this fusion replaces gradient descent for the refinement, and that a static scene reaches a PlenOctree within 60 seconds §3.

# Fine-stage fusion of one leaf over dense views  [§3] [eq. 3] [eq. 4] [eq. 5]
# sigma: density, z: SH coefficients; W: running weight; C: update count
# W_0 and C_0 are not stated; the initial values here are a reading.

def fuse_leaf(leaf, dense_views):
    sigma = coarse_sigma(leaf)                  # value from step 2  [§3]
    z = coarse_z(leaf)                          # value from step 2  [§3]
    weight = not_stated                         # W_0, not stated  [§3]
    count = not_stated                          # C_0, not stated  [§3]
    for view in dense_views:                    # 100 dense views  [§3]
        T_i = transmittance(view, leaf)
        # Query psi only where T_i > 1e-3  [§3]
        if T_i <= 1e-3:
            continue
        sigma_new, z_new = psi_predict(view, leaf)  # view i  [§3]
        # Weighted running average of density  [eq. 3]
        sigma = (weight * sigma + T_i * sigma_new) / (weight + T_i)
        # Weighted running average of SH coefficients  [eq. 4]
        z = (weight * z + T_i * z_new) / (weight + T_i)
        # Update count, then weight  [eq. 5]
        count = count + 1
        weight = (count - 1) / count * weight + T_i / count
    # Drop leaves whose density is below 1e-3  [§3]
    return sigma, z

Fourier representation of time §4.1

Each leaf stores, instead of one value per frame, the Fourier coefficients of its density over time and of its SH coefficients over time §4.1, eq. 6.

• Density at frame t is recovered from n1 coefficients with a real inverse transform §4.1, eq. 7. The basis at index i is a cosine for even i and a sine for odd i, with frequency set by i and T §4.1, eq. 8.
• Each SH element z_{m,l} in R³ has its own n2 coefficients on the same basis §4.1, eq. 9. Colour is then computed from these SH values as in §3, eq. 1.
• Density and colour are real-valued and the basis is differentiable, so the coefficients can be fine-tuned by back-propagation §4.3.

Building the Fourier tree §4.2

1. Build a PlenOctree for every frame t = 1..T with the static fusion above §4.2. The per-frame octrees are not guaranteed to match, because the object moves §4.2.
2. Make the structure identical across frames. Take the union of all frames' leaf structures. A leaf that is divided in some frame is split in the union, and the new children copy the value of the parent leaf §4.2.
3. For every leaf of the union tree, run a forward transform over the T frames to get k^σ and k^z §4.2, eq. 10, eq. 11, eq. 12. The forward basis carries a 1/T factor eq. 12.

Fine-tuning §4.3

Fine-tune the coefficients by minimising a sum of squared errors between rendered and ground-truth images over all frames t and views i §4.3, eq. 13. The paper says this is shorter than fine-tuning a Fourier NeRF-SH network, because the Fourier PlenOctree is explicit §4.3.

# Fourier encoding of one leaf over T frames  [§4.2] [eq. 10-12]
# sigma_t: (T,) density per frame; z_t: (T, K, 3) SH values, K = (lmax+1)^2
# n1, n2: coefficient counts; both are fixed per run  [§5.2]

def fourier_encode(sigma_t, z_t, n1, n2):
    k_sigma = zeros(n1)                         # (n1,)
    k_z = zeros(n2, K, 3)                       # (n2, K, 3)
    for t in range(1, T + 1):                   # frames 1..T  [§4.2]
        for i in range(n1):
            k_sigma[i] += sigma_t[t] * dft_basis(i, t, T)  # [eq. 10]
        for i in range(n2):
            k_z[i] += z_t[t] * dft_basis(i, t, T)          # [eq. 11]
    return k_sigma, k_z

def dft_basis(i, t, T):
    # 1/T * cos(i*pi*t/T) if i is even, 1/T * sin((i+1)*pi*t/T) if odd
    # [eq. 12]
    if i % 2 == 0:
        return cos(i * pi * t / T) / T
    return sin((i + 1) * pi * t / T) / T

# Decoding one leaf at frame t  [§4.1] [eq. 7-9]
# The inverse basis has no 1/T factor  [eq. 8]

def decode_leaf(k_sigma, k_z, t, T):
    sigma = 0
    for i in range(n1):                         # [eq. 7]
        sigma += k_sigma[i] * idft_basis(i, t, T)
    z = zeros(K, 3)
    for i in range(n2):                         # [eq. 9]
        z += k_z[i] * idft_basis(i, t, T)
    # Colour from z via the SH sum and sigmoid  [§3, eq. 1]
    return sigma, z

def idft_basis(i, t, T):
    # cos(i*pi*t/T) if i is even, sin((i+1)*pi*t/T) if odd  [eq. 8]
    if i % 2 == 0:
        return cos(i * pi * t / T)
    return sin((i + 1) * pi * t / T)

Hyperparameters

Results

Author-reported. Setup: five real datasets at 2048×1536 and five synthetic datasets at 1080×1080; PC with one RTX 3090; baselines trained on the same data; the paper gives no per-dataset breakdown and no standard deviations §5, §5.1.

Comparison on rendering accuracy tab. 1

Training and rendering speed tab. 2

Fourier dimensions tab. 3

Fine-tuning time tab. 4

Representation variants under a 24 GB limit tab. 5

The paper's own summary is that the Fourier form keeps quality with much less storage than the unlimited no-DFT variant, and that 10 minutes of fine-tuning removes most blur §5.2, fig. 8, fig. 9.

Paper vs code

No official code.

Searched: the arXiv metadata has no code link or comment; the LaTeX source has no URL besides the template; no project page is linked. A GitHub repository search for "fourier plenoctree", "FourierPlenOctree", "plenoctree", "dynamic plenoctree" and "free-view video" with the authors' affiliation (ShanghaiTech) found no repository by the authors. The hits for "plenoctree" are the original PlenOctree project and unrelated tools, not this method.

Pitfalls

• Many training details are missing. Optimizer, learning rate, iterations, batch size, Ψ architecture, the coarse grid size N, the number of frames per dataset and the dataset names are not stated §3, §4.3, §5. An implementer cannot reproduce the fine-tuning from the text alone.
• "Ours" PSNR differs between tables. Tab. 1 reports 35.21 tab. 1, tab. 3 reports 36.21 for n1=31, n2=5 tab. 3, tab. 4 reports 32.93 for 10-minute fine-tuning tab. 4, and tab. 5 reports 32.15 for the Fourier variant tab. 5. The paper does not say which setting or dataset each table uses.
• Test split may not be held-out sequences. The quantitative test keeps 10% of captured views for testing §5.1. The abstract speaks of "unseen dynamic scenes" [abstract]. Whether test views are unseen cameras, unseen frames or both is not stated.
• Speed claim lacks a baseline timing. The "3000 times faster than NeRF" figure §5 has no NeRF timing in the paper; only the 100 fps rate at 800×800 is reported.
• Reconstruction time is ambiguous. tab. 2 and tab. 5 give 2 hours tab. 2, tab. 5. The paper does not say whether this includes the octree fusion, the per-frame stage or fine-tuning, and §3 gives 60 seconds only for a static scene §3.
• Sampling of directions. The paper samples θ and φ uniformly in [0, 2π] each §3. That is uniform in the angle parameters, not over the sphere. Whether the expectation in eq. 2 is weighted to compensate is not stated.
• Overloaded symbols. Eq. 3 uses σ_i on both sides, the new prediction and the running value §3. Read σ_i on the right as the view-i prediction and the left as the updated value.
• Initial values not stated. W_0 and C_0 for the fusion are not given §3, eq. 5.
• Frame index base. The forward sum runs over t = 1..T §4.2, eq. 10, while the inverse basis is written with a frame index t without stating the base §4.1, eq. 7. A 0-based frame index shifts every frequency by one frame; check the round trip on the training frames.
• Index range notation. The sums are written for i = 0..n−1, but the text says 0 ≤ i ≤ n2 §4.1. Use n2 entries.
• Normalisation asymmetry. The forward transform has a 1/T factor §4.2, eq. 12, and the inverse does not §4.1, eq. 8. Round-trip reconstruction is not reported.
• Basis frequencies. Index i = 0 is the constant term. For i ≥ 1, the cos and sin indices pair up at the same frequency (for example i = 1 and i = 2 both use 2π/T). Each cos/sin pair therefore shares one frequency, so the n1 = 31 count is not a count of distinct frequencies §4.1, eq. 8.
• Union of structures. The copy rule for leaves that were split in only some frames is described only briefly §4.2. Whether a parent's value is copied into all frames or only the frames where it was a leaf is not stated.
• Visual hull dependence. The octree structure is initialised from the union of visual hulls, which the authors call inefficient § Discussion. Large motions break this assumption § Discussion.
• Storage and memory cost. Fourier coefficients grow with the number of coefficients and with motion speed and the number of frames § Discussion. The n1 = 31, n2 = 11 setting needs about twice the storage of n2 = 5 tab. 3.
• Dangling figure reference. The Method section refers to parts (a) and (b) of the pipeline figure, which is Fig. 2 in the Related work section §3, §4, fig. 2.

Lineage

• Builds on: NeRF (volume rendering with an MLP) §2; PlenOctree, which stores spherical-harmonic values in an octree and provides the leaf representation §2, §3, eq. 1; generalizable NeRF networks such as IBRNet, which provide Ψ §3; volumetric fusion in the style of KinectFusion §3.
• Compared against: Neural Body, NeuralVolumes, ST-NeRF and iButter §5.1.
• Superseded by: not stated in this card's sources.