Explicit feature grids (dense voxels and factorised planes)

Overview

An explicit feature grid keeps a field's values in arrays and reads them by interpolation, instead of computing them with a black-box network. The K-Planes authors contrast this with radiance-field models that rely on MLP components 2301.10241 §1. A dense grid grows exponentially with the number of input dimensions. The K-Planes authors estimate that a direct 4D representation at their resolution would need more than 300 GB, against about 200 MB for their factorised version 2301.10241 §1. HexPlane gives a different estimate for a 4D volume at its own settings, 48 GB for float32 RGB at 512³ spatial and 32 time steps 2301.09632 §3.1, so the two figures are not comparable. In most of the papers below the arrays are free parameters fitted to one scene. In one, a network predicts them for each input 2312.00113 §3.2.

Four forms appear on this page. A dense 3D voxel grid is read directly, with no factorisation 2212.00774 §4.2. Factorised planes use one plane per pair of input axes, so a d-dimensional input uses C(d, 2) planes 2301.10241 §3 2301.09632 §3.3. K-Planes and HexPlane fit those planes to a scene, and combine them differently. K-Planes multiplies all planes together 2301.10241 §3.1. HexPlane multiplies within each complementary pair, concatenates the three products and projects them linearly 2301.09632 §3.3. In both cases the product, not a sum, is the core choice 2301.10241 §3.1 2301.09632 tab. 5. The fourth form comes from the event-camera paper. It uses the same three axis pairs for an (x, y, t) input, but a network computes the planes once per window from an event volume and a frame 2312.00113 §3.2.

Plane sets follow the input dimension. A static 3D scene uses three planes (xy, xz, yz). A 4D video adds three space-time planes (xt, yt, zt), giving six 2301.10241 §3.1 2301.09632 §3.3. The event-camera paper's three-dimensional input gives the three planes x–y, x–t and y–t 2312.00113 §3.2. In K-Planes the time planes start at 1 and an L1 penalty pulls them toward 1 2301.10241 §3.2 2301.10241 eq. 5. Because of that, a region that does not move keeps its time planes near 1 and is carried by the space planes alone. This follows from the initialisation and penalty; the paper presents it as the reason the split is easy to inspect 2301.10241 §1.

Variants

What is shared is that the field's values are stored in arrays and read by interpolation. The variants differ in where the arrays come from, how the grid is laid out, how the reads are combined, and how the feature becomes colour and density.

Dense voxel grid (no factorisation)

A single 3D array of features, read by trilinear interpolation along each ray. Score Jacobian Chaining optimises a voxel radiance field of this kind, with a 100³ grid over [-1, 1]³ and four feature channels in the Stable Diffusion latent space 2212.00774 app. A3 2212.00774 §5.2. The paper uses the grid to lift a 2D diffusion model to 3D without 3D data 2212.00774 §4. Its reported cost is about 25 minutes and 9 GB of GPU memory per object on one A6000, author-reported 2212.00774 app. A3. The paper does not compare the dense grid with a factorised one, so it gives no memory comparison.

Full product over all planes (K-Planes)

Every plane is read at the point's projection, and the reads are multiplied elementwise across all planes 2301.10241 §3.1 2301.10241 eq. 2. The product then goes to one of two decoders, described below. Card: K-Planes.

Explicit decoder (linear learned basis)

The feature vector is turned into colour by a dot product with a learned basis, which the paper uses in place of spherical harmonics. A small MLP maps the view direction to three basis vectors, one per colour channel. Colour is a sigmoid of those dot products. Density uses a separate view-independent basis and a truncated exponential 2301.10241 §3.3 2301.10241 eq. 6 2301.10241 eq. 7.

Hybrid decoder (small MLP)

A first MLP maps the features to density and to an extra feature vector. A second MLP maps that vector and an encoded view direction to colour 2301.10241 §3.3 2301.10241 eq. 8. The text writes the direction encoding as γ(d) without defining it, so an implementation needs the released code or a choice of its own.

Paired products, concatenated and projected (HexPlane)

Each spatial plane is paired with the complementary space-time plane (XY with ZT, XZ with YT, YZ with XT). Each pair is multiplied elementwise, the three products are concatenated, and a learned matrix projects the result to the feature width 2301.09632 §3.3 2301.09632 eq. 5 2301.09632 eq. 6. Two such plane sets are used: one for opacity and one for appearance, and the appearance set is queried only where opacity is high 2301.09632 §3.4. Colour comes from a tiny MLP that takes the appearance feature and the view direction 2301.09632 §3.4. The paper does not give the MLP's size. Card: HexPlane.

Planes predicted per window (event-camera decompression)

Here the arrays are not fitted to a scene. A U-Net turns an event volume and the initial frame into planes at four scales, each with x–y, x–t and y–t planes 2312.00113 §3.2 2312.00113 suppl. Model. A point is decoded by sampling the three planes bilinearly, multiplying the three samples elementwise, and concatenating the products over scales 2312.00113 §3.2. A three-layer convolutional decoder then maps these features to RGB 2312.00113 suppl. Model. Because the planes come from the network, the model runs once per window and can decode any number of times 2312.00113 §3. The supplement gives 24 feature channels, 8 per plane 2312.00113 suppl. Model. The released code uses a different channel layout for the time planes, so check the card's Paper vs code before reusing either. The paper validates queries at its protocol times only. Queries at other times are not verified 2312.00113 §4.1. Card: Event-based Continuous Color Video Decompression from Single Frames.

When to use which

Implementation sketch

The plane lookup is common to the two plane forms. The two forms differ only in how the reads are combined. Planes are keyed by the axes they read, so a plane for axes (0, 1) is the xy plane. The decoder after the lookup is the part that differs; see Variants.

# Planes for d input axes: one per pair of axes  [arxiv:2301.10241 §3]
# Output: list of axis pairs, C(d, 2) entries

def plane_pairs(d):
    return list(combinations(range(d), 2))

# Full product over all planes (K-Planes)  [arxiv:2301.10241 §3.1] [eq. 2]
# Input: point (d,) scaled to the grid range; planes keyed by axis pair
# Output: feature vector (M,)

def query_full_product(point, planes):
    # Bilinear read on each plane at the point's projection  [eq. 1]
    reads = [
        bilinear_sample(planes[pair], point[list(pair)])
        for pair in plane_pairs(len(point))
    ]
    # Elementwise product across all planes, not a sum  [eq. 2] [§3.1]
    return prod(reads, axis=0)

# Paired products, concatenated and projected (HexPlane)  [arxiv:2301.09632 §3.3]
# Axis indices into (x, y, z, t); each spatial pair is matched with its complement
PAIRS = [((0, 1), (2, 3)), ((0, 2), (1, 3)), ((1, 2), (0, 3))]

# Input: point (4,) scaled to the grid range; planes keyed by axis pair
# Output: feature vector (F,); V is a learned (3M, F) matrix  [eq. 6]

def query_paired(point, planes, V):
    pair_features = []
    for spatial, complement in PAIRS:
        # Two bilinear reads per pair, multiplied elementwise  [eq. 5]
        a = bilinear_sample(planes[spatial], point[list(spatial)])
        b = bilinear_sample(planes[complement], point[list(complement)])
        pair_features.append(a * b)
    # Concatenate the three pair products, then project  [eq. 6]
    return V @ concatenate(pair_features)

Initialisation: K-Planes gives the initial value of its time planes (1) but does not state the spatial-plane initial value 2301.10241 §3.2. Because a full-product feature is a product over six planes, the scale of each plane multiplies into every feature. The paper does not analyse how this interacts with the learning rate or the regulariser weights 2301.10241 §3.1.

The planes-predicted-per-window variant 2312.00113 §3.2 uses the same bilinear reads and the same product within each scale. Its planes come from a network for each window, and its decoder is a convolution 2312.00113 suppl. Model. The sketch above does not cover that decoder.

Contradictions

The papers do not agree on the best way to combine planes for dynamic scenes. K-Planes multiplies all six planes and reports that product as its main change over additive tri-plane work 2301.10241 §3.1. HexPlane finds that multiplying within each complementary pair and then concatenating outperforms the full product on D-NeRF, with and without an MLP 2301.09632 tab. 5. The two sets of experiments use different benchmarks and decoders, so this is a difference in the reported best choice rather than a direct conflict. Neither paper runs the other's combination on its own main benchmark.

Memory figures also differ. K-Planes estimates more than 300 GB for a direct 4D representation at its resolution 2301.10241 §1. HexPlane estimates 48 GB for a float32 RGB volume at 512³ spatial and 32 time steps 2301.09632 §3.1. Both papers report about 200 MB for their trained factorised model 2301.10241 §1 2301.09632 §4.1. The dense-volume figures use different resolutions and are not comparable.

Differences between each paper and its released code are listed on its card: for K-Planes, the scope of total variation, the timing of importance sampling and the claim of a pure PyTorch implementation [card: 2301.10241]; for HexPlane, the RegNeRF depth term, the emptiness-update schedule and the starting grid size for D-NeRF [card: 2301.09632]; for SJC, the score scale and the number of noise samples [card: 2212.00774]; for the event-camera paper, the time-plane channel layout and the reported numbers against its README [card: 2312.00113].

Papers

• Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation: optimises a dense voxel radiance field through a 2D diffusion score, with noise-averaged score estimation.
• HexPlane: A Fast Representation for Dynamic Scenes: six feature planes for space and time, with paired products concatenated and projected before a tiny MLP.
• K-Planes: Explicit Radiance Fields in Space, Time, and Appearance: factorises a d-dimensional radiance field into one plane per pair of axes, combined by product, with explicit and hybrid decoders.
• Event-based Continuous Color Video Decompression from Single Frames: a network predicts x–y, x–t and y–t planes per window from events and one frame, then fuses them with flow-splatted frames.