A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets

In plain words

Gaussian splatting draws a scene as a cloud of soft blobs and renders it fast. A street-level capture of several kilometres needs far more blobs than a computer can hold or draw at interactive speed.

This paper does two things. First, it cuts the city into blocks and trains each block on its own, so no single training run has to hold everything. Second, it stacks the blobs into a tree. Each parent blob stands in for its children. When the camera is far away, the renderer draws the parents; when it is close, it draws the children. The parent and child blobs are blended over a short transition so the switch does not pop. The parents are also tuned afterwards, so they look right when they stand in for many children.

For an engineer the useful parts are the tree construction, the rule that picks which level to draw, the transition blend, and the chunk pipeline. The card lists the values the paper gives and marks the places where the paper leaves a value out or where its equations and text disagree.

TL;DR

The paper builds a Gaussian-splatting level-of-detail hierarchy for very large outdoor captures (up to several kilometres and tens of thousands of images). Each chunk is trained independently, its leaves are merged into a binary tree, the tree is pruned, and interior nodes are optimised. A cut through the tree is chosen per view from projected size, and levels blend smoothly. The authors report real-time rendering of their captured scenes on consumer-grade GPUs. Official code exists under a non-commercial licence, so this card does not use it (see Paper vs code).

Problem

Radiance fields and 3D Gaussian splatting (3DGS) keep the whole scene in memory and draw every primitive. For large captures this exceeds GPU memory and is too slow to render §1. Sparse capture from a vehicle or a helmet rig also makes the original 3DGS densification rule and its training recipe unreliable §6.2, app. C. Existing large-scene NeRF methods either do not reach real-time rates or do not scale to this size §2.

Method

The pipeline has three stages §1, fig. 1:

1. Train a coarse scaffold on the full data with no densification and fixed positions §6.1.
2. Train each chunk independently with the 3DGS optimiser, with depth supervision, maximum-gradient densification and exposure compensation §6.2, app. C.
3. Build a tree per chunk, compact it, optimise its interior nodes, and consolidate all chunks into one scene-level structure §4.1, §5, §6.3.

Rendering then picks a cut through the tree per view and blends across level boundaries §4.2.

Interior nodes: merging Gaussians

Each node is a 3D Gaussian with mean, covariance, SH coefficients and a value called falloff (opacity for leaves) §3.1, §4.1.

Given children with means \mu_i and covariances \Sigma_i, the parent's mean and covariance are weighted moments with weights w_i normalised to sum to 1 eq. 3, eq. 4, §4.1. The unnormalised weight is w'_i = o_i S_i, the opacity times the ellipsoid surface of the child eq. 8, §4.1. The paper derives it from matching the parent's projected contribution to the sum of the children's, under near-isotropy and low perspective distortion eq. 5–eq. 8, §4.1.

SH coefficients are merged with the same normalised weights eq. 9, §4.1. Falloff is \sum_i w'_i / S_p, which can exceed 1 §4.1, fig. 2. The paper keeps this value as the parent's falloff and clamps the blending weight to 1 §4.1.

Building the tree

Build an axis-aligned bounding-box (AABB) tree top-down. The root box encloses all leaves, each enlarged to three times its size §4.1. Split each node by a binary median cut on the longest axis of the box, using the projected means §4.1. Then merge the children bottom-up §4.1.

After merging, orientation matching rotates each child's axes to minimise the rotation relative to its parent. The paper uses an exhaustive search for this §4.1.

Cut selection

For a view and a target granularity \tau_\epsilon (in pixels), the granularity \epsilon(n) of a node is the projected size of the largest side of its leaf-bounding box §4.2, fig. 3. A node is in the cut when it meets \tau_\epsilon and its parent does not §4.2. Because a parent's box is never smaller than a child's, each branch is cut either at one node or not at all §4.2.

Smooth level switching

Each cut node n with parent p gets a weight from its own and its parent's granularity eq. 10, §4.2. Position, colour and SH are interpolated linearly with that weight §4.2. Scale and rotation are interpolated separately; rotation is linear, not slerp §4.2. Opacity needs a start value so that, at the start of a transition, the children blend to the same colour as the parent §4.2, fig. 4.

For two children, that start value is \alpha' = 1 - (1 - \alpha_p)^{1/2} eq. 12, §4.2. The blend is \alpha(t) = t\,\alpha_i + (1 - t)\,\alpha' eq. 13, §4.2. For K children the paper gives \alpha' = 1 - (1 - \alpha_p)^{1/K} §5.2.

# Cut selection for one view  [§4.2] [eq. 10]
# Input:  tree (nodes with parent links), camera, tau (pixels)
# Output: list of nodes whose blend the renderer draws

def select_cut(tree, camera, tau):
    cut = []
    for node in tree.nodes:
        # Projected size of the node's leaf-bounding box  [§4.2]
        fine_enough = granularity(node, camera) <= tau
        # A root has no parent; treat it as coarse  [§4.2]
        parent_coarse = (
            node.parent is None
            or granularity(node.parent, camera) > tau
        )
        if fine_enough and parent_coarse:
            cut.append(node)
    return cut

# Interpolation weight and child opacity start  [§4.2] [eq. 10] [eq. 12]
# Input:  node n, its parent p, tau, parent falloff alpha_p, K children
# Output: weight t in the transition, and the start value alpha'

def transition_weight(node, tau, camera):
    # t_n = (tau - eps(n)) / (eps(p) - eps(n))  [eq. 10]
    eps_n = granularity(node, camera)
    eps_p = granularity(node.parent, camera)
    return (tau - eps_n) / (eps_p - eps_n)

def child_opacity_start(alpha_p, num_children):
    # alpha' = 1 - (1 - alpha_p)^(1/K)  [eq. 12] [§5.2]
    return 1 - (1 - alpha_p) ** (1 / num_children)

# Merge child Gaussians into a parent  [§4.1] [eq. 3, eq. 4, eq. 8, eq. 9]
# Input:  children (Gaussians with mean, cov, sh, opacity)
# Output: parent Gaussian (mean, cov, sh, falloff)

def merge(children):
    # Unnormalised weights w'_i = o_i * S_i  [eq. 8] [§4.1]
    raw = [c.opacity * ellipsoid_surface(c.cov) for c in children]
    total = sum(raw)
    # Normalised weights w_i  [§4.1]
    weights = [r / total for r in raw]
    # Parent mean: weighted average of child means  [eq. 3]
    mean = sum(w * c.mean for c, w in zip(children, weights))
    # Parent covariance: weighted spread plus child covariances  [eq. 4]
    cov = sum(
        w * (c.cov + outer(c.mean - mean, c.mean - mean))
        for c, w in zip(children, weights)
    )
    # Parent SH: weighted average, same weights  [eq. 9]
    sh = sum(w * c.sh for c, w in zip(children, weights))
    # Parent falloff: unnormalised sum over parent surface  [§4.1]
    parent_surface = ellipsoid_surface(cov)  # formula not stated
    falloff = total / parent_surface
    return Gaussian(mean, cov, sh, falloff)

Hierarchy optimisation

Interior nodes are rendered like leaves, so they can be optimised §5.1. Each step draws a random training view and a random granularity. The granularity is \tau = \tau_{max}^{\xi}\,\tau_{min}^{1-\xi} with \xi drawn from [0, 1) §5.1. The render is at full resolution, with cut selection and interpolation active §5.1. Leaves are frozen with a stop-gradient §5.1, §7.3. Only the positions, shapes, colours and opacities of interior nodes change §5.1, §7.3.

Compaction

Before optimisation, the tree is pruned to avoid nodes whose parent is barely larger than its children §5.2:

# Sparsify the tree so each kept parent covers enough detail  [§5.2]
# Input:  tree with leaves, training views, tau_min, tau_max (pixels)
# Output: pruned tree

def compact(tree, views, tau_min, tau_max):
    # Leaves are always kept  [§5.2]
    mark_relevant(tree.leaves)
    tau = tau_min  # 3 px, minimum primitive extent  [§5.2]
    while tau < tau_max:  # tau_max: half the image resolution  [§5.2]
        # Union of cuts over all training views at this tau  [§5.2]
        cut = union(select_cut(tree, v, tau) for v in views)
        # Bottom-most nodes of that union are the highest-detail kept  [§5.2]
        keep = bottom_most(cut)
        # Remove nodes between those and already marked nodes  [§5.2]
        remove_between(tree, keep, mark_relevant)
        tau = 2 * tau  # double the granularity each round  [§5.2]
    return tree

Chunks, depth and densification

Chunks are 50 × 50 m for walking captures and 100 × 100 m for vehicle captures §6. A camera belongs to a chunk if it is inside the chunk bounds, or within twice the bounds and sees more than 50 SfM points inside the bounds §6.1.

The scaffold has a skybox of 100,000 primitives on a sphere ten times the scene diameter §6.2. Outside a chunk, only opacity and SH of the scaffold are optimised §6.2.

Densification uses the maximum of the screen-space positional gradient rather than the mean §6.2.

Depth regularisation uses a monocular depth model rescaled to the COLMAP inverse depth in each image app. C.1. The loss is an absolute difference between rendered and prior depth app. C.1. Its weight decays exponentially from 1 to 0.01 over the per-chunk optimisation app. C.1.

Exposure is a per-image 3 × 4 affine transform, initialised to identity and optimised with Adam app. C.2.

Consolidation deletes a primitive that belongs to chunk i but lies outside it when it is closer to another chunk j §6.3. It then builds one scene root over all chunk trees §6.3.

Hyperparameters

Not stated: the training loss used for hierarchy optimisation §5.1; number of iterations for each stage; learning rates for the hierarchy stage; densification thresholds and interval §6.2; the depth-loss weight and decay rate app. C.1; the mask used for pixels in the depth prior app. C.1; and the ellipsoid surface formula §4.1.

Results

Author-reported. Setup: one chunk per scene for competitor comparisons, one V100 for training, one A6000 for rendering. Test images are every 50th image alphabetically from each rig camera §8.2. Exposure is disabled in the single-chunk comparison §8.2.

Single chunk, leaves and \tau levels tab. 2:

The full table in tab. 2 also gives \tau_1 and \tau_3 rows and the Mip-NeRF 360, Instant-NGP and F2-NeRF baselines.

Mega-NeRF Mill 19 tab. 3: on Building, Mega-NeRF 20.93 / 0.504 / 0.547 (PSNR / LPIPS / SSIM) against 21.52 / 0.297 / 0.723 for Ours opt (\tau_2). On Rubble: 24.06 / 0.516 / 0.553 against 24.64 / 0.284 / 0.755. The authors report 3 hours on two (Building) or four (Rubble) V100s against 27–30 hours on eight V100s for Mega-NeRF §8.2.

Mip-NeRF 360, single chunk §8.2: 3DGS averages 29.11 dB and Ours 28.96 dB over the dataset. The \tau_1, \tau_2 and \tau_3 hierarchies average 28.86 / 28.87, 28.05 / 28.52 and 24.82 / 27.25 dB (basic / optimised).

Full scene runtime tab. 5 (average per frame; "#Render" is Gaussians on chip):

The text reports 30+ FPS on average at \tau_1 and about 60 FPS at \tau_2 across the full paths §8.2.

Storage: after consolidation, scene sizes are 6, 17, 27 and 88 GB, about 68% larger than non-hierarchical 3DGS files §8.1.

Ablation, averages over four scenes tab. 6:

The text says depth supervision does not improve per-image metrics. Its benefit is qualitative on roads and in view extrapolation §8.3, fig. 11.

Paper vs code

Official code exists: the project page links graphdeco-inria/hierarchical-3d-gaussians, an Inria organisation, and the author block lists Inria affiliations. The repository is at commit 596dc7081ee74e57a66502f4746328911f763b34. Its LICENSE.md is an Inria licence that grants research use only and forbids commercial use without consent. It also says the original 3DGS licence applies to the parts built on 3DGS. wikillm publishes with advertising, so this card does not use the repository's code. Only the LICENSE.md was read. The card is written from the paper alone, as for other non-commercial repositories in this wiki.

Because the code was not read, no paper-code comparison has been made. A reader with a research licence should check at least the hierarchy-optimisation loss, the densification thresholds and the depth-loss weight, which the paper leaves unstated.

Pitfalls

• The falloff clamp has two values. The text says blended values are clamped to 1 §4.1. The implementation section says the rasteriser clamps at 0.99 and zeroes the opacity gradient when that happens §7.3. Pick one and test it on a parent that stands in for many children.
• Ellipsoid surface is not defined. The weight w'_i uses the ellipsoid surface S_i eq. 8, §4.1. The paper gives no formula. Its general statement, that the surface is proportional to the square root of the projected covariance determinant, does not fix the constant or the 3D definition.
• Orientation matching is under-specified. The paper says an exhaustive search minimises relative rotation §4.1, but not what is searched: which axis permutations and signs, or whether scale is kept fixed.
• Interpolation direction is not stated. Eq. 10 gives t_n and eq. 13 gives \alpha(t), but the paper does not say which end of the transition is t = 0 or t = 1 §4.2. Check at the start of a transition that the blended colour equals the parent's colour, as the paper requires §4.2, fig. 4.
• Rotation interpolation is linear, not slerp. Linear interpolation of quaternions gives non-unit rotations unless the result is renormalised. The paper does not say whether it renormalises §4.2.
• Opacity start value uses two different rules. For two children it is 1 - (1-\alpha_p)^{1/2} eq. 12. For K children it is 1 - (1-\alpha_p)^{1/K} §5.2. Both are stated; implement the general one.
• \tau_{max} is ambiguous. "Half the image's resolution" does not say which dimension, or whether it is measured before or after any downsampling §5.2.
• "Bottom-most" nodes in compaction is loosely defined. The paper calls the lowest nodes in the union of cuts a cut, but does not say how ties and partial branches are handled §5.2.
• The hierarchy-optimisation loss is not stated. Chunk training follows the 3DGS optimiser with depth and exposure changes §6.2, app. C. The paper does not give the loss used when optimising interior nodes §5.1.
• Depth prior definitions are ambiguous. The median t(\mathbf{D}) is written over \mathbf{D}, but the scale s(\mathbf{D}) sums over SfM indices only app. C.1. It is not clear whether the median is also taken over SfM pixels.
• Depth loss weight and decay rate are missing. The paper gives the start and end values of the weight, not its rate or the exact schedule app. C.1.
• Densification uses the maximum gradient, but no threshold is given §6.2. The interval and threshold must be taken from the 3DGS defaults or tuned.
• Image counts disagree across the paper. BigCity has 38,235 images in tab. 1 but 28,145 in the caption of fig. 8. The introduction says 5,800 to 28,000 images §1, while §7.2 says 5,800 to 40,000.
• Chunk counts are not mapped to scenes. The text gives 4, 11, 22 and 46 chunks in an order that matches the scene order only loosely §8.1. Check against the dataset before using them as a benchmark.
• Table 2 has a likely copy error. The Mip-NeRF 360 row has identical values (20.95 / 0.442 / 0.731) for Campus and BigCity tab. 2. Treat that row with caution.
• Optimisation gains at \tau_2 are small. The optimised \tau_2 PSNR gains over the unoptimised hierarchy are +0.57, +0.40, +0.17 and +0.08 dB on the four scenes tab. 2. The larger gains are at \tau_3, which the text also emphasises §8.2.
• Rendering can be slower than 3DGS. At \tau_1 or leaves, the extra weight computation and the \texttt{pow} in the blend cost time §8.2, tab. 4. The real speed-up appears only at coarser cuts such as \tau_2.
• Cut update and cleanup are asynchronous. The cut is updated every two frames and cleaned up every 100 frames §6.3. These intervals are the paper's and are not justified.
• Memory cost. Each Gaussian takes 400 bytes in the implementation, 69% more than 3DGS, and the theoretical minimum is 284 bytes §8.2.
• Pose estimation is expensive. The hierarchical COLMAP mapper gives poses that are not good enough alone. The paper adds exhaustive matching for the chunk cameras, retriangulation, bundle adjustment and Procrustes alignment, which takes 30 minutes to 5 hours on a V100 16 GB per chunk app. B.
• Training time is per stage, but the scope is unclear. The text says coarse, chunk and hierarchy optimisation each take about one hour on one V100 §8.1. It does not say whether this is per chunk or per scene.
• Data and code are not yet public. The paper says the source and capture data will be released §1. A footnote says dataset release is pending a Data Protection Officer's approval §2.
• The EWA reference key is misleading. The text attributes the anti-aliasing filter to Yu et al., but the bibliography key in the source is geiger §5.1.
• Mega-NeRF comparison uses different inputs. It uses pixSFM poses from the Mega-NeRF release, with points triangulated by COLMAP and metric scale applied afterwards §8.2. It is not the same pipeline as the paper's own.
• Masking. Humans and licence plates are removed, and moving vehicles are found by an SfM-point density test. The test is described as effective but not accurate app. A.

Lineage

• Builds on: 3D Gaussian Splatting, arXiv:2308.04079 §3.1. The rasteriser, densification and SH parameters are kept §6.2, §7.1.
• Anti-aliasing: EWA filtering, as cited in §5.1.
• Depth prior: DPT, Ranftl et al. app. C.1.
• Compared against: Mip-NeRF 360, Instant-NGP, F2-NeRF and 3DGS on single chunks §8.2, and Mega-NeRF on Mill 19 §8.2.
• Related large-scene work: Block-NeRF, Mega-NeRF and BungeeNeRF, which the paper says cannot handle street-level data at this scale §2.
• Superseded by: not stated in this wiki.