Level-of-detail rendering

Overview

Level-of-detail (LoD) rendering draws a scene at a resolution that matches how much of the screen each part covers. Far or small parts are shown at lower detail, so each frame costs less than drawing the full scene at full detail. In the hierarchy method, far or small parts are replaced by fewer, coarser primitives 2406.12080 §1, §4.2. In the progressive NeRF, each detail level is a separate output of one network 2112.05504 §3.3.

The problem it solves is concrete. Radiance fields and 3D Gaussian splatting keep the whole scene in memory and draw every primitive. Large captures exceed GPU memory and render too slowly 2406.12080 §1. An LoD scheme addresses both by drawing only what the current view needs.

The design choices of the hierarchy method, which is the most detailed method on this page 2406.12080 §4, §5, §6:

1. Building parents. Merge a node's children into one primitive whose mean, covariance and spherical harmonics are weighted averages of the children's 2406.12080 §4.1.
2. Tree shape. Build a binary tree top-down by median cuts on the longest axis of each node's bounding box 2406.12080 §4.1.
3. Pruning. Remove nodes whose parent adds little detail over its children 2406.12080 §5.2.
4. Choosing a cut per view. Draw a node when its projected size meets a pixel target and its parent's does not 2406.12080 §4.2.
5. Switching levels. Blend each drawn node with its parent over a transition, so the switch does not pop 2406.12080 §4.2.
6. Tuning parents. Optimise interior nodes so they look right when they stand in for many children 2406.12080 §5.1.
7. Scaling to very large captures. Split the scene into chunks, train each chunk alone, then consolidate them into one structure 2406.12080 §6.

The progressive NeRF makes a different set of choices, listed under Variants: how scales are assigned to images, how each stage adds capacity, and how the loss treats each output 2112.05504 §3.3.

Variants

Two methods have paper cards here. They solve the same problem with different representations, so their numbers are not directly comparable.

Merged Gaussian hierarchy

A tree of 3D Gaussians, cut per view by projected size and blended across levels. The design choices are items 1 to 7 above. Card: Merged-Gaussian tree for multi-km street scenes.

The paper names Block-NeRF, Mega-NeRF and BungeeNeRF as related large-scene work and says they do not handle street-level data at this scale 2406.12080 §2. Its own comparisons are with several splatting and NeRF baselines on single chunks, and with Mega-NeRF on Mill 19 [arxiv:2406.12080 §8.2]. See Contradictions.

Progressive multi-scale NeRF

A neural radiance field that gives one output per detail level from one network. Images are split into four scales by camera distance, one new scale per 2x zoom-out 2112.05504 §3.3. Training starts with the most distant scale. Each later stage adds one residual block and one output head. Head L is supervised by every image whose scale is at most L, so the far views shape every head and the close views shape only the higher ones [arxiv:2112.05504 §3.3, eq. 3]. Card: BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering.

The paper's argument is that training one NeRF on all scales at once biases it towards low frequencies, which blurs close views, and that training each scale alone leaves seams between scales 2112.05504 §3.2, fig. 2. The card does not describe a per-view rule for picking a head at render time beyond the output being one per level.

When to use which

Each row is a constraint and the choice the cited results support. The numbers are author-reported.

Implementation sketch

Common core of the hierarchy method: a parent merge, a cut per view, and the blend weight for one drawn node. Granularity is the projected size of a node's leaf bounding box, in pixels 2406.12080 §4.2. The ellipsoid surface used in the merge weight has no formula in the paper 2406.12080 §4.1.

# Merge K children into one parent  [eq. 8] [eq. 3] [eq. 4] [eq. 9]
# Input:  children with mean, cov, sh, opacity
# Output: parent with mean, cov, sh, falloff

def merge(children):
    # Unnormalised weight: opacity times ellipsoid surface  [eq. 8]
    raw = [c.opacity * ellipsoid_surface(c.cov) for c in children]
    total = sum(raw)
    w = [r / total for r in raw]         # normalised, sums to 1
    # Weighted mean, then covariance including child spread  [eq. 3, eq. 4]
    mean = sum(wi * c.mean for wi, c in zip(w, children))
    cov = sum(
        wi * (c.cov + outer(c.mean - mean, c.mean - mean))
        for wi, c in zip(w, children)
    )
    # Weighted SH, same weights  [eq. 9]
    sh = sum(wi * c.sh for wi, c in zip(w, children))
    # Falloff is the unnormalised sum over the parent surface  [§4.1]
    falloff = total / ellipsoid_surface(cov)
    return Gaussian(mean, cov, sh, falloff)

# Choose the cut for one view  [§4.2]
# Input:  tree, camera, pixel target tau
# Output: nodes to draw, each blended with its parent

def select_cut(tree, camera, tau):
    cut = []
    for node in tree.nodes:
        # Node meets the target  [§4.2]
        fine = granularity(node, camera) <= tau
        # Parent is too coarse, or the node is the root  [§4.2]
        coarse_parent = (
            node.parent is None
            or granularity(node.parent, camera) > tau
        )
        if fine and coarse_parent:
            cut.append(node)
    return cut

# Blend weight and child opacity start  [eq. 10] [eq. 12] [§5.2]
# Input:  node, its parent, tau, parent falloff, K children
# Output: weight t, opacity start alpha'

def blend_weight(node, camera, tau):
    # Position of tau between node and parent  [eq. 10]
    # Which end is t = 0 is not stated  [§4.2]
    e_n = granularity(node, camera)
    e_p = granularity(node.parent, camera)
    return (tau - e_n) / (e_p - e_n)

def opacity_start(alpha_parent, k):
    # Start value so the children match the parent's colour  [eq. 12] [§5.2]
    return 1 - (1 - alpha_parent) ** (1 / k)

Points to check in an implementation of the hierarchy:

• The falloff clamp, the blend direction and the rotation renormalisation are not settled by the paper; see Contradictions and the card's pitfalls 2406.12080 §4.1, §4.2, §7.3.
• The tree is pruned with a loop over pixel targets, doubling each round 2406.12080 §5.2. Which image dimension sets the upper bound is not stated.
• Chunk training uses the 3DGS optimiser with a depth prior and per-image exposure 2406.12080 §6.2, app. C. The loss for the hierarchy stage is not stated 2406.12080 §5.1.

The progressive NeRF needs two pieces that are not in the hierarchy: the sum of residual heads, and the loss that masks rays by scale.

# Output of head L: base output plus residuals of blocks 2..L  [eq. 2]
# Input:  base output, residual outputs of blocks 2..Lmax, head index L
# Output: colour and density of head L (before rendering)

def head_output(base_out, residuals, L):
    out = base_out
    # residuals[l] is the residual of block l + 2, for colour and density
    for res in residuals[: L - 1]:
        out = out + res
    return out

# Training loss at stage L: each head sees rays up to its scale  [eq. 3] [§3.3]
# Input:  per-head predictions, targets, scale id of each ray, stage L
# Output: scalar loss

def stage_loss(preds, targets, scale_ids, L):
    loss = 0.0
    for l in range(1, L + 1):
        keep = scale_ids <= l        # rays with scale index at most l
        loss += sum_squared_error(preds[l][keep], targets[keep])
    return loss

Points to check in an implementation of the progressive NeRF:

• Stage count, the 2x spacing between scales and the number of images per scale are set by hand in the authors' code; the paper calls the number of scales "relatively flexible" 2112.05504 §3.3. Changing the count changes the schedule and the outputs.
• Each stage uses 100k iterations with Adam at learning rate 5e-4, batch 2,048 rays and a reset optimiser 2112.05504 §4. The paper does not say how a new stage's weights are initialised, and the learning-rate decay is exponential with no rate given 2112.05504 §4.
• The paper's density and colour activations follow the original NeRF 2112.05504 §3.3. Where the code's activations sit relative to the residual sum is not specified in the paper.

Contradictions

• Falloff clamp. The text says the blended falloff is clamped to 1 2406.12080 §4.1. The implementation section says the rasteriser clamps the blend at 0.99 and zeroes the opacity gradient when it does 2406.12080 §7.3. Both statements come from the same paper. The two limits differ only for blended values between 0.99 and 1, and the paper does not say which one applies.
• Street-level scope of the progressive NeRF. The hierarchy paper says Block-NeRF, Mega-NeRF and BungeeNeRF do not handle street-level data at the scale it targets 2406.12080 §2. The progressive NeRF paper trains on images whose closest scale is at full resolution, and evaluates on city scenes captured at a range of camera distances 2112.05504 §3.3, §4. The progressive NeRF card gives no scene extent in metres or kilometres, so the two claims cannot be compared on scale. This page does not resolve it.

Papers

• Merged-Gaussian tree for multi-km street scenes (arXiv:2406.12080): builds a tree of merged 3D Gaussians for multi-km street scenes, selects a cut by projected size, blends across levels, and consolidates independently trained chunks.
• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (arXiv:2112.05504): trains a NeRF in stages from the most distant scale to the closest, adding one residual block and output head per scale, so one model gives several detail levels.