HexPlane: A Fast Representation for Dynamic Scenes

In plain words

Most methods that rebuild a moving scene from several cameras use one large neural network. You give it a place and a moment in time, and it returns a colour. That works, but every pixel needs many network calls, so rendering is slow and training can take days.

HexPlane keeps the scene in a different form. It stores six flat grids of learned numbers, like six sheets of paper. Three sheets cover space only, and three cover space combined with time. To get the details of one point at one moment, you look up its position on each sheet, multiply the paired readings, and join the results. A very small network then turns that summary into a colour. Images are drawn by blending colours along each camera ray, as in NeRF.

The authors report that this matches the image quality of the slow method on their benchmark while training about a hundred times faster. The results are their own measurements, not independent ones.

TL;DR

• HexPlane replaces the per-point MLP of dynamic-NeRF methods with six learned 2D feature planes: three spatial (XY, XZ, YZ) and three spatio-temporal (ZT, YT, XT). §3.3 eq. 4
• A point's feature is six bilinear lookups, fused and projected, then decoded to colour by a tiny MLP and trained with volume rendering and an MSE loss. §3.3 §3.4 eq. 5 eq. 7
• Author-reported: on Plenoptic Video, PSNR 29.47 on flame salmon versus DyNeRF's 29.58, at 12 h versus 1344 h of training. tab. 1
• Author-reported: on D-NeRF (monocular, synthetic), PSNR 31.04 without a deformation field. tab. 2
• Read Pitfalls before implementing: the paper leaves several values open, and the released code differs from the paper in a few places.

Problem

Dynamic-scene methods built on NeRF either query a large MLP for every sample or deform a canonical scene with another MLP. Both need many network calls per pixel, and training a single dynamic scene can take up to days of GPU time. §1

Explicit grids would be faster, but a dense 4D grid scales with the fourth power of the resolution. A naive 4D volume needs O(N³TF) memory. The paper gives 48 GB for a float32 RGB volume with N=512 and T=32. §3.1

Multi-view capture gives only one or a few views per time step, so each time step cannot be modelled on its own. The representation has to share information across time. §3.1

The method assumes a bounded scene, with coordinates normalised into a box, or normalised device coordinates (NDC) for forward-facing scenes. §4.1 [supp. Training Details] It does not assume deformation fields or category priors. §1 §4.4 The authors state that the representation is limited to bounded scenes. §4.3

Method

The method has four parts: a low-rank factorisation of a 4D volume (§3.1), a shared time basis (§3.2), the six-plane HexPlane representation (§3.3), and the rendering and training setup (§3.4).

1. Factorised 4D volume. A 3D volume is written as a sum of vector-matrix outer products, following TensoRF. Three groups of R₁, R₂, R₃ components use the planes XY, XZ, YZ with vectors on the remaining axes. §3.1 eq. 1 With R = R₁+R₂+R₃ ≪ N, memory falls from O(N³TF) to O(RN²T). §3.1
2. Shared time basis. The paper first writes a time-t volume as a weighted sum of shared basis volumes, with weights f(t). §3.2 eq. 2 It then shares the low-rank components across these basis volumes, which gives eq. 3. §3.2 eq. 3 The authors report that the basis volumes were still too costly, so only small R_t fits in GPU memory. §3.2
3. Time weights. The paper tried sinusoidal and MLP time functions. It reports the sinusoidal version as inflexible and hard to optimise, and an MLP as slow. It uses a learned piecewise-linear function instead, obtained by linear interpolation along the first axis of a learned T×R matrix. §3.2
4. HexPlane. The vector-times-time term is replaced by a plane that depends on a spatial axis and time, such as the ZT plane. Applying the same change to every term gives six learned planes, paired as XY with ZT, XZ with YT and YZ with XT. §3.3 eq. 4 Each pair is a spatial plane and a spatio-temporal plane whose two axes are orthogonal. §3.3
5. Query. A point (x, y, z, t) is read from each plane by bilinear interpolation. Within each pair the two readings are multiplied element-wise. The three products are concatenated and multiplied by a learned matrix V, which is a linear projection to the output width. §3.3 eq. 5 eq. 6 Querying uses six bilinear interpolations and one vector-matrix product. §3.3
6. Two HexPlanes. One HexPlane gives the opacity of a point, and a second gives the appearance feature. The appearance feature is queried only for points with high opacity. §3.4 HexPlane_Base.py
7. Colour and rendering. A tiny MLP takes the appearance feature and the view direction and returns RGB. Colours are composited along rays with volume rendering. §3.4 An MLP-free variant stores spherical-harmonic coefficients in the feature instead. §3.4 §4.3 tab. 6
8. Loss. The loss is the mean squared error between rendered and target ray colours, plus λ_reg times a regulariser. §3.4 eq. 7 The regularisers are total variation (TV) on the planes and a depth smoothness term from RegNeRF. §3.4 See Paper vs code for what the release implements.
9. Coarse-to-fine grids. The spatial grid starts coarse and doubles in resolution at scheduled iterations. §3.4 [supp. Training Details]
10. Emptiness voxel. A coarse 3D voxel marks regions with no density, and samples inside empty cells are skipped. The voxel stores the maximum opacity over time for each cell. §3.4 HexPlane_Base.py
11. Ray sampling for multi-view video. Three stages sample rays: key frames weighted by how far pixels deviate from the global median, then all frames with the same weighting, then a single frame where the sampling weight follows the change between nearby frames. [supp. Training Details] trainer.py

Pseudocode. The first block is the query (eq. 5 and 6). The second is one training step. Symbols are described in the comments; shapes are given where they help.

# Query one HexPlane for a batch of spacetime points  [§3.3] [eq. 5]
# Input:  points (B, 4), columns (x, y, z, t) normalised to [-1, 1]
# Output: features (B, F)
def query_hexplane(points, planes, spacetime_planes, basis):
    # Each pair pairs a spatial plane with a spatio-temporal plane  [§3.3]
    for k in range(3):
        # Bilinear read of the spatial plane at its two spatial axes  [§3.3]
        spatial = bilinear(planes[k], points[spatial_axes[k]])
        # Bilinear read of the spatio-temporal plane at (t, third axis)  [§3.3]
        temporal = bilinear(spacetime_planes[k], points[time_axes[k]])
        # Element-wise product inside the pair (fusion one)  [§3.3] [eq. 4]
        pair_features[k] = spatial * temporal
    # Concatenate the three pairs (fusion two)  [§3.3]
    fused = concatenate(pair_features)
    # Project with the learned matrix V to the output width  [§3.3] [eq. 6]
    return basis(fused)

# One training step on a batch of rays  [§3.4] [eq. 7]
# Input:  rays (B, 6) as origins and directions; targets (B, 3); times (B, 1)
# Output: loss (scalar)
def training_step(rays, targets, times):
    # Sample S points per ray, jittered inside the near-far range  [§3.4]
    points, depths = sample_along_rays(rays, times)
    # Opacity from the density HexPlane  [§3.4] [code: config/nv3d.yaml#L15]
    sigma = density_activation(density_head(query_hexplane(points, density_planes)))
    # Appearance feature only where the ray weight is above the threshold  [§3.4]
    feature = query_hexplane(points[visible], app_planes)
    # Tiny MLP maps feature and view direction to RGB  [§3.4]
    rgb = app_mlp(feature, view_dirs)
    # Alpha-composite the colours along each ray  [§3.4]
    color = volume_render(sigma, rgb, depths)
    # Photometric mean squared error  [§3.4] [eq. 7]
    loss = mean((color - targets) ** 2)
    # Total variation on all six planes, weight lambda  [§3.4] [supp. Training Details]
    loss = loss + lambda_tv * total_variation(all_planes)
    return loss

Key equations, in the paper's notation.

\mathbf{D}(x,y,z,t) = (\mathbf{P}^{XYR_1}_{xy\bullet}\odot\mathbf{P}^{ZTR_1}_{zt\bullet})\mathbf{V}^{R_1F} + \dots eq. 5

[\mathbf{P}^{XYR_1}_{xy\bullet}\odot\mathbf{P}^{ZTR_1}_{zt\bullet};\ \dots]\,\mathbf{V}^{RF} eq. 6

\mathcal{L} = \frac{1}{|\mathcal{R}|}\sum_{\mathbf{r}\in\mathcal{R}}\|C(\mathbf{r})-\hat{C}(\mathbf{r})\|_2^2 + \lambda_{\text{reg}}\mathcal{L}_{\text{reg}} eq. 7

Hyperparameters

Values marked "paper" come from the paper or its supplement. Values marked "code" come from the official release at code_commit; the paper does not state them. Plenoptic settings are from config/nv3d.yaml, and D-NeRF settings from config/dnerf_slim.yaml.

Results

All numbers are author-reported. They come from the paper's own runs, and the baseline numbers are copied from the DyNeRF paper, not re-run. §4.1 [supp. License] The paper does not ship the baseline models.

Setup. Plenoptic Video: multi-view GoPro capture, 21 cameras at 2028×2704 and 30 FPS, 19 ten-second videos per scene, 18 used for training and 1 for evaluation. The paper says the scenes are highly specular, translucent and transparent, with topology changes and flames. §4.1 The paper trains HexPlane on one 16 GB V100 with a batch of 4096 rays and the same 650k iterations, which it says is six times fewer samples than DyNeRF; DyNeRF used 8 V100s for a week with a batch of 24576. §4.1 D-NeRF: monocular synthetic objects with 360° views; HexPlane's D-NeRF times were re-measured on one 2080 Ti. §4.1

Table 1, Plenoptic Video, flame-salmon scene and averages (author-reported). Table 1 in the paper, with the -all rows giving the average over public scenes except the unsynchronised one. tab. 1

† is a HexPlane trained for 100k iterations in three stages of 10k, 50k and 40k. [supp. Training Details] The paper says the same-iteration HexPlane beats DyNeRF on every metric except PSNR. §4.1 This holds in the table: 29.470 vs 29.581 PSNR, and better D-SSIM, LPIPS and JOD. tab. 1

The paper's memory claim: HexPlane takes about 200 MB for the whole model. §4.1

Table 2, D-NeRF (author-reported, monocular, no deformation field). tab. 2

The paper also notes that its D-NeRF training time is re-measured on the same GPU for all methods. §4.1 The paper says HexPlane "distinctly outperforms" the other methods on D-NeRF even without a deformation field. §4.1 Table 2 does not support that wording against TiNeuVox-B, which has higher PSNR (32.67 vs 31.04) and equal SSIM and LPIPS. HexPlane is faster than the deformation-based methods in the table. tab. 2

Table 3, factorisation (D-NeRF, ablation). tab. 3

The paper reports that larger R improves quality up to a point, and that an overly large R overfits and hurts novel views. §4.2

Table 4, plane designs (D-NeRF, ablation). tab. 4

Table 5, feature fusion (D-NeRF, complete version in the supplement). Fusion one combines each pair; fusion two combines the three pairs. [supp. Fusion Ablations] The "opacity with MLP" columns use an MLP on an 8-dimensional opacity feature, as in the cited chan2022efficient setup. [supp. Fusion Ablations]

The paper's reading: Multiply-Concat is best. Multiply-Sum and Sum-Multiply stay close to it, while Sum-Sum and Sum-Concat fail without an MLP. §4.3 tab. 5

Table 6, MLP-free spherical harmonics (D-NeRF). tab. 6

Qualitative results with no numbers. Synthesis on Plenoptic scenes fig. 3 and a comparison with DyNeRF's published images fig. 4. The paper says the DyNeRF model is not public, so the comparison matches frames visually. §4.1 Grid-resolution results show that a higher space grid helps and that a shorter time grid barely changes quality. §4.2 fig. 5 Non-rectangular coordinates for unbounded scenes fig. 6 and monocular iPhone captures fig. 7 are shown without numeric scores. §4.3 §4.4

Paper vs code

Code: Caoang327/HexPlane at fc99b61da94fc406702d9e080ca5e37f5d7dcfad, MIT licence. It is linked from the authors' project page. LICENSE

Matches: the Plenoptic basis counts (48/24/24 and 24/12/12), the planes' learning rate (0.02) and MLP learning rate (0.001), Adam β, batch of 4096 rays, 650k iterations split 300k/250k/100k, γ and α values, TV temporal-to-spatial ratio of 2, and the D-NeRF bounding box and iteration count. [code: config/nv3d.yaml] config.py [code: config/dnerf_slim.yaml#L22-L27]

Differences:

• Depth smoothness from RegNeRF. The paper says it uses the RegNeRF depth smoothness term. §3.4 The trainer's depth loss compares rendered depth with supplied depth, and it is off unless depth_loss is set. [code: hexplane/render/trainer.py#L372-L379] config.py The RegNeRF patch sampler (get_random_pose) exists in the D-NeRF dataset class, but I found no call to it in the trainer. dnerf_dataset.py None of the released configs enables a depth term. Treat the paper's RegNeRF claim as not implemented in the release.
• Emptiness updates. The paper says the emptiness voxel is computed at 50k and 100k. [supp. Training Details] The Plenoptic config updates it at 50k, 100k, 150k and 220k. nv3d.yaml The D-NeRF config updates at 4k, 8k and 10k, where the paper says 4k and 10k. dnerf_slim.yaml
• Bounding box on z. The paper's NDC bounds are z from 0.0 to 1.0. [supp. Training Details] The config uses z from −1.0 to 1.0. [code: config/nv3d.yaml#L26-L27]
• Starting resolution, D-NeRF. The paper starts from 32³. [supp. Training Details] The D-NeRF config sets N_voxel_init: 262144, which is 64³. [code: config/dnerf_slim.yaml#L5]
• Grid growth. The paper says the grid goes from 64³ to 512³. [supp. Training Details] The config uses aligned upsampling, N → 2N−1 per axis. nv3d.yaml trainer.py Three doublings of a 64-cell axis give 505, not 512. The real per-axis sizes follow from the non-square grid in N_to_reso and are printed at run time. [code: hexplane/render/util/util.py#L25-L29] HexPlane_Base.py
• Time weight during training. The TV weight is multiplied by a learning-rate decay factor at every step. trainer.py The paper writes λ_reg without a schedule.
• Density activation. The paper does not state one. The Plenoptic config uses ReLU. nv3d.yaml The default is softplus with a −10 shift. config.py HexPlane_Base.py
• Released D-NeRF model. The D-NeRF config uses HexPlane_Slim, which fixes multiply-then-concat fusion and a plain density. HexPlane_Slim.py config.py That matches the paper's best-scoring fusion (Table 5), but the paper does not name the slim variant.
• Per-scene LPIPS arrows. The per-scene D-NeRF table in the supplement marks LPIPS with ↑, but lower is better, as in the main table. [supp. Per-scene results]

No discrepancy found for: the emptiness voxel's maximum over time (paper and code both take the maximum), the choice of reading the appearance feature only where opacity is high, and the DyNeRF baseline numbers (they are copied from the DyNeRF paper). §3.4 HexPlane_Base.py

Pitfalls

• Unstated appearance network. The paper says only "tiny MLP". The release uses a feature width of 27, a 3-layer MLP of width 128, and a sigmoid output. config.py config.py HexPlane_Base.py Positional encodings are 0 in the Plenoptic config and 2 by default. nv3d.yaml
• Activation of density. ReLU in the Plenoptic config, softplus with a −10 shift by default. Choose one and say which you chose. nv3d.yaml
• Non-square grids. Plane sizes are set from the scene extent, not one cube. [supp. Training Details] A square implementation will not match the paper's sizes.
• Empty-space mask is dilated. Before thresholding, the emptiness voxel is max-pooled with a 3×3×3 kernel, which keeps more samples than the paper's description suggests. The threshold is 1e-3. HexPlane_Base.py
• Background. forward uses a white background by default. [code: hexplane/model/HexPlane_Base.py#L423] HexPlane_Base.py The paper does not state the background.
• Speed-up is not like-for-like. The 112× and 672× figures compare HexPlane on one V100 at 650k or 100k iterations with DyNeRF's published 8-GPU, week-long run. §4.1 tab. 1 Reproduce the training time before quoting it.
• Dataset licence. The Plenoptic Video dataset is CC BY-NC 4.0, and the paper says so. [supp. License] The paper's own licence (CC BY 4.0) does not extend to it. This card reproduces only published numbers, not the dataset. Do not ship the dataset in a commercial product.
• Failure modes. Fast motion and very few observations degrade results. [supp. Failure Cases] The paper reports colour jitter and artifacts that need stronger regularisation. [supp. Limitations]
• Expected cost. About 200 MB for the trained model. §4.1 Training takes 12 h at 650k iterations on one V100, per the paper. tab. 1
• Open design choices the paper leaves. Basis initialisation, learning-rate decay ratio, key-frame interval (30 in the release), and the spherical re-parameterisation for unbounded scenes. For the last, the paper gives r = 1/√(x²+y²+z²) and samples r in [0, 1] [supp. / §4.3]; I did not check it against code, because the release has no spherical config.

Lineage

• Builds on: TensoRF (arXiv:2203.09517) for the factorised volume and the coarse-to-fine scheme §3.1 §3.4; NeRF for volume rendering and NDC §3.4 §4.1; explicit and hybrid methods for static scenes, which the introduction cites as its starting point. §1
• Compared against: DyNeRF (Li et al., CVPR 2022, "Neural 3D Video Synthesis From Multi-View Video") on Plenoptic Video; D-NeRF, T-NeRF and TiNeuVox (arXiv:2205.15285) on D-NeRF; Neural Volumes and LLFF on Plenoptic Video. tab. 1 tab. 2
• Superseded by: not known from content already in this wiki.