K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

In plain words

A radiance field is a function that takes a point in a scene and a viewing direction and returns how much material is there and what colour it looks like from that direction. New photographs are rendered by asking that function many times along each camera ray. A moving scene needs one more input, time, so the function lives in four dimensions. Storing its values in a full four-dimensional grid would take far too much memory.

K-Planes stores the values on flat sheets instead of in a cube. Every pair of coordinates gets its own two-dimensional sheet. A still 3D scene gets three sheets (x–y, x–z, y–z). A 4D video gets six: those three plus three sheets that pair time with each space axis. To evaluate a point, the model reads the value on each sheet at that point's position and multiplies the six readings together. Multiplying, rather than adding, lets a single point be picked out precisely. A small learned function then turns the combined number into a colour and a density.

The split between space-only and space-time sheets keeps the model easy to inspect. A time sheet that stays at 1 everywhere means that part of the scene does not move, so it falls back on the plain spatial sheets. Each sheet can be looked at, or penalised, on its own. The authors report that this compact, explicit model trains quickly and reaches quality competitive with recent methods on the benchmarks they tried.

TL;DR

K-Planes represents a d-dimensional radiance field with the \binom{d}{2} planes that cover every pair of dimensions: tri-planes for static 3D scenes and hex-planes for dynamic 4D scenes §3 §3.1. Plane features are multiplied together and decoded by a linear decoder with a learned colour basis, or by a small MLP in a hybrid variant §3.1 §3.3. Author-reported results are close to TensoRF-class baselines on static scenes and to recent hybrid methods on dynamic scenes, at training times of 33 to 52 minutes for most settings tab. 3. Use it when you want an explicit, inspectable model. Check the baseline setups before comparing numbers, because the paper itself flags differing SSIM implementations and partial baseline coverage §4 tab. 3.

Problem

Storing a 4D volume directly is prohibitively expensive because of the curse of dimensionality. Factorisations of 3D volumes for static radiance fields exist but do not extend easily to higher dimensions §1. Most radiance-field models also use black-box MLP components, and the authors seek a white-box alternative that remains competitive §1. The authors estimate that a direct representation at their resolution would need more than 300 GB, against about 200 MB for their factorisation §1.

Method

Factorisation. A d-dimensional scene uses k = \binom{d}{2} planes, one for each pair of dimensions. The 4D instance has three space-only planes (xy, xz, yz) and three space-time planes (xt, yt, zt), each of shape N \times N \times M, where M is the feature length §3.1. A 4D point (i, j, k, \tau) is normalised to [0, N) and projected onto each plane. Each projection is read by bilinear interpolation eq. 1 §3.1. The six feature vectors are combined by an elementwise product (Hadamard product) to give one length-M vector eq. 2 §3.1.

Why multiply. Prior tri-plane work adds plane features. The authors show that multiplication can pick out a single spatially localised signal, which addition cannot §3.1 (figure "Addition versus Hadamard product" in §3.1). In an ablation on the static Lego scene, multiplication improves the explicit model's PSNR from 28.78 to 35.29 and the hybrid model's from 34.80 to 35.75 tab. 2.

Multiscale. Each scale has its own set of planes, at spatial resolutions that are multiples of a base resolution. The time dimension is not replicated at multiple scales §3.2. The per-scale feature vectors are concatenated before decoding §3.2.

Priors. Three regularisers are defined in the text:

• Spatial total variation with an L2 penalty on the space-only planes, and on the spatial axes of the space-time planes §3.2 eq. 3.
• A second-difference penalty along time on the space-time planes §3.2 eq. 4.
• An L1 penalty pulling the space-time planes toward 1, the multiplicative identity, which are initialised at 1 §3.2 eq. 5.

Linear decoder (explicit model). The feature vector is turned into colour with a learned basis that replaces spherical harmonics. A small MLP maps the view direction \bm{d} to three length-M basis vectors, one for each of red, green and blue. Colour is the dot product of the features with each channel's basis, followed by a sigmoid eq. 6 §3.3. Density uses a separate view-independent basis b_\sigma and an exponential activation with truncated gradient eq. 7 §3.3.

Hybrid decoder (MLP model). A first MLP g_\sigma maps the features to density and to an extra feature vector \hat f. A second MLP g_{RGB} maps \hat f and an embedded view direction to colour eq. 8 §3.3. The paper writes the view embedding as \gamma(\bm{d}) without defining it in the text; see Paper vs code.

Global appearance. For photo collections, each training image gets an M-dimensional appearance vector. It is fed only to the colour decoder, so it can change colour but not geometry. The paper uses a 32-dimensional appearance code §3.3 §4.3.

Optimisation details. Forward-facing scenes use normalised device coordinates. Unbounded scenes use an \ell_\infty version of the Mip-NeRF 360 contraction §3.4. A two-stage proposal sampler uses a small K-Planes density model trained with a histogram loss §3.4. For multi-view video, the authors sample training rays in proportion to colour variation over a ±25-frame window once the static part has converged §3.4.

Volume rendering follows NeRF's quadrature app. Volumetric rendering eq. 9.

# Feature lookup for one 4D point  [§3.1] [eq. 1] [eq. 2]
# Input:  point (x, y, z, t) scaled to the grid range; planes per scale
# Output: one feature vector, concatenated across scales

def query_features(point, planes_per_scale):
    # Each scale has its own six planes  [§3.2]
    scale_features = []
    for planes in planes_per_scale:
        # Bilinear read on every plane at its projected coordinates  [eq. 1]
        plane_values = [
            bilinear_sample(plane, project(point, pair))
            for plane, pair in zip(planes, PLANE_PAIRS)
        ]
        # Multiply across the planes, not add  [eq. 2] [§3.1]
        scale_features.append(elementwise_product(plane_values))
    # Concatenate across scales before decoding  [§3.2]
    return concatenate(scale_features)

# Linear decoder with a learned colour basis  [§3.3] [eq. 6] [eq. 7]
# Input:  features (F,), view direction (3,)
# Output: density (scalar), colour (3,)

def decode_linear(features, direction, basis_mlp, sigma_basis):
    # The MLP outputs one length-F basis per colour channel  [eq. 6]
    basis = basis_mlp(direction).reshape(3, -1)  # (3, F)
    colour = sigmoid(basis @ features)  # (3,)
    # Density has no view dependence  [eq. 7]
    density = truncated_exp(dot(features, sigma_basis))
    return density, colour

# Volume rendering along one ray  [app. Volumetric rendering] [eq. 9]
# Input:  densities (N,), colours (N, 3), distances to next sample (N,)
# Output: pixel colour (3,)

def render_ray(densities, colours, deltas):
    # Optical depth accumulated before each sample  [eq. 9]
    depth_per_sample = densities * deltas
    depth_before = cumsum(depth_per_sample) - depth_per_sample
    transmittance = exp(-depth_before)
    # Absorption by each sample  [eq. 9]
    absorption = 1 - exp(-depth_per_sample)
    weights = transmittance * absorption
    return sum(weights[:, None] * colours, axis=0)

Hyperparameters

The paper states few values in its text and points to the released configs for the rest app. Model hyperparameters. Rows whose source reads "code only" are not stated in the paper; they come from the config files and code at the commit above.

Results

Author-reported. Setup: NVIDIA A30 GPU for timings; metrics are averages over each dataset's scenes unless noted tab. 3. Phototourism uses MS-SSIM tab. 3. Baseline numbers are taken from the cited works, so the paper's own SSIM footnote applies §4.

The paper reports that the explicit and hybrid models are close on the NeRF benchmark and that the hybrid model is slightly better on LLFF §4. On D-NeRF it beats D-NeRF on quality and training time, but does not beat TiNeuVox or V4D §4. On Phototourism it is below NeRF-W on PSNR. The figure caption notes slightly lower resolution than NeRF-W, while training takes about 35 GPU minutes against NeRF-W's roughly 384 GPU hours app. Per-scene results tab. 3. The DyNeRF and LLFF baselines report only the flame-salmon video, so the averages are not like-for-like tab. 3 footnote.

Paper vs code

Official code: github.com/sarafridov/K-Planes, linked from the paper and owned by the first author's account, at commit b4cea7212cd5cf2d33889d9b6795873d1e0cd919. Licence BSD-3-Clause LICENSE.

• Total variation scope. The paper applies spatial TV to the three space-only planes and to the spatial axes of the space-time planes §3.2 eq. 3. The code applies a 2D TV to the same three spatial planes, then to every plane including the space-time ones regularization.py, so the space-time planes receive TV along time as well. The constants also differ: the paper normalises by \lvert C\rvert n^2 eq. 3, and the code uses a per-plane mean with a factor of two regularization.py.
• Time smoothness normalisation. The paper normalises the sum by \lvert C\rvert n^2 eq. 4. The code takes a mean over the squared second differences regularization.py.
• Sparse transients. The paper sums the L1 norms over the three space-time planes eq. 5. The code takes a mean of \lvert 1 - P\rvert per plane and sums across planes regularization.py.
• Undocumented losses. The paper's text does not describe a distortion loss or a reconstruction loss. The code uses MSE on RGB base_trainer.py base_trainer.py and a distortion loss with weight 0.001 in the NeRF and DyNeRF configs nerf_explicit.py regularization.py.
• Hybrid direction encoding. The paper writes the hybrid colour MLP's input as \gamma(\bm{d}) §3.3 eq. 8. The code uses a degree-4 spherical-harmonics encoding of the view direction kplane_field.py kplane_field.py.
• IST timing. The paper says importance sampling runs "during the last portion of optimization" after the static scene has converged §3.4. The DyNeRF configs switch at step 50,000 out of 120,001 (explicit) or 90,001 (hybrid) dynerf_explicit.py dynerf_explicit.py dynerf_hybrid.py, so the switch happens around 42% and 56% of training, not in a final portion. The switch itself is in video_trainer.py.
• "Pure PyTorch." The abstract claims a pure PyTorch implementation and the introduction says no custom CUDA kernels §1. The code imports tiny-cuda-nn for the MLPs and the spherical-harmonics encoding, and the requirements list it kplane_field.py kplane_field.py requirements.txt.
• Time-smoothness weight. The appendix ablation on Jumping Jacks picks 0.01 app. Time smoothness regularizer. The DyNeRF configs use 0.001 dynerf_explicit.py. The paper does not state the DyNeRF value.
• Per-dataset settings the paper does not state. Learning rate (0.01 vs 0.02 for LLFF), feature length (16 for LLFF and DyNeRF hybrid), decoder depth (4 vs 1 hidden layer), and scales differ across configs llff_explicit.py nerf_explicit.py trevi_explicit.py.

Pitfalls

• Coordinate convention. The paper normalises to [0, N) §3.1. The code maps coordinates to [-1, 1] and samples with align_corners=True and border padding kplane_field.py interpolation.py interpolation.py. The paper does not say which corner convention it assumes. Using the other convention can shift samples by up to half a grid cell (derived here, not discussed in the paper).
• Initialisation scale. The paper gives the time planes' initial value (1) but not the spatial planes'. The code draws them uniformly from [0.1, 0.5] kplane_field.py. Derived here, not stated in the paper: at initialisation a feature is a product of three values in [0.1, 0.5] and three ones, so it lies in [0.001, 0.125]. Learning rate and initialisation interact through this product.
• Multiplicative coupling. Each feature is a product over six planes, so one plane's scale multiplies into every feature. The paper's own ablation shows a large gap between multiplication and addition for the linear decoder §3.1 tab. 2. The paper does not analyse how this interacts with the learning rate or the regulariser weights.
• Linear decoder input. The basis MLP takes the raw view direction, not an encoding; the encoding line is commented out kplane_field.py kplane_field.py.
• Appearance embedding. Initialised with a standard normal kplane_field.py. At test time the paper optimises a new per-image embedding on the left half of each test image §4.3. The code optimises it with Adam at learning rate 0.1 phototourism_trainer.py trevi_explicit.py.
• Scheduler quirk. The warm-up-cosine schedule uses num_cycles=0.999, so the cosine never quite reaches its zero point lr_scheduling.py. The residual is tiny, but a reimplementation with exact cosine will differ slightly.
• Costs the abstract hides. Training times are on one A30 for K-Planes and on different hardware for the baselines, some of which took days on eight GPUs tab. 3 app. Per-scene results. Memory grows with M and with the number of scales; the paper's own table shows 132M parameters at M=128 on Lego app. Feature length.
• Scale choices and results. The multiscale ablation on Lego shows that dropping the highest resolution reduces parameters but lowers quality app. Multiscale.
• Metric comparability. Some prior work uses MipNeRF's SSIM and some uses scikit-image, which gives higher values. The paper says it used the baselines' implementation where possible §4 footnote. Do not compare SSIM across rows without checking.
• DyNeRF baseline coverage. The DyNeRF and LLFF baseline values refer to the flame-salmon video only tab. 3 footnote.

Lineage

• Builds on: tri-plane factorisation with additive combination (Hadamard product is the change) §3.1; spherical-harmonic and TV-regularised plane models such as Plenoxels, Plenoctrees and TensoRF, which the paper cites for the SH decoder and TV regulariser §3.3 §3.2; NeX, for the learned basis §3.3; Instant-NGP, for multiscale features and the hybrid MLP decoder §3.2 §3.3; Mip-NeRF 360, for contraction, proposal sampling and the histogram loss §3.4; DyNeRF, for the importance sampling strategy §3.4; NeRF-W, for the appearance code §3.3. None of these has a card in this wiki yet.
• Compared against: TensoRF, Plenoxels, I-NGP, D-NeRF, TiNeuVox, V4D, DyNeRF, MixVoxels-L, NeRF-W and LearnIt tab. 3.
• Superseded by: not known from the content of this wiki.