MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes

In plain words

A neural radiance field stores a 3D scene as a learned function and draws new views by sampling that function along each camera ray. Those models are good at photorealism but slow to draw, and the fast variants usually need a lot of GPU memory, which breaks down for large scenes.

MERF keeps the scene in two cheap-to-read forms at once. A coarse 3D grid holds the empty-versus-solid structure, and three flat, high-resolution grids, one per axis plane, hold the fine detail. The model is trained in a compressed form that is set up to match what the viewer will read later. Before shipping, the trained model is converted, or "baked," into a plain lookup format, so the browser only has to read values and blend them along each ray.

Large outdoor scenes raise one more problem: the space outside the camera's area must still be represented. MERF bends the coordinates of distant space into a bounded box in a way that keeps straight rays straight, so the viewer can skip empty space quickly. The result is real-time rendering in a web browser. The authors report that its quality beats the other real-time methods they compare against.

TL;DR

MERF is a radiance-field representation for real-time view synthesis of large unbounded scenes in a browser §1. It combines a low-resolution sparse 3D grid with three high-resolution 2D planes and trains that mix through a hash-encoded MLP with quantization-aware training §4.1, §5.1, §5.2. A baking step marks occupied voxels and writes a block-sparse, PNG-encoded model that a WebGL shader renders §5.3, §6. Author-reported results on mip-NeRF 360 show the best quality among real-time methods for a given memory budget §8, tab. 1, tab. 3.

Problem

Volumetric radiance fields need many evaluations per ray. Compact MLP models are slow to query, and fast sparse-voxel models use large amounts of graphics memory §1. Methods that do run in real time for large scenes have not been shown at high resolution §1, §2. Baking a slow model into a fast one usually loses quality through a training/rendering mismatch, and fine-tuning the baked model does not scale to large scenes because gradients need far more memory than rendering §1. Training also needs more than twelve times the video memory of rendering §5.1. Prior real-time methods that restrict the network to planes or polygons lose quality on large unbounded scenes §2.

Method

MERF has one radiance field with two parameterizations: a volume used for training and a baked grid used for rendering §1. Both encode the same function §1.

Rendering model. Colour along a ray is the sum of weighted sample colours, with weights from the usual volume-rendering quadrature §3, eq. 1. MERF uses SNeRG's deferred shading: diffuse colour and a feature vector are accumulated along the ray, and one small MLP adds view dependence once per ray [§3, eq. 2, eq. 3].

\mathbf{C} = \mathbf{C}_d + h(\mathbf{C}_d, \mathbf{F}, \mathbf{d}) eq. 3

Scene parameterization (step 1). Each location holds eight channels: density, three diffuse colour channels and four view-dependence features §4.1. A low-resolution 3D grid with L per axis and three high-resolution planes with R per axis are summed at a query point §4.1. Trilinear reads come from the grid and bilinear reads from each plane fig. 2.

\mathbf{t}(x, y, z) = \mathbf{V}(x, y, z) + \mathbf{P}_x(y, z) + \mathbf{P}_y(x, z) + \mathbf{P}_z(x, y) eq. 5

Nonlinearities after interpolation (step 2). Activations are applied after the sum, which the authors say raises representational power §4.1. The density uses an exponential, and colour and features use a logistic sigmoid eq. 6.

\sigma_{\text{density}} = \exp(\tilde{\tau}), \quad \mathbf{c}_d = \sigma(\tilde{\mathbf{c}}_d), \quad \mathbf{f} = \sigma(\tilde{\mathbf{f}}) eq. 6

# Query the scene representation at one point  [§4.1]
# Input:  x (3,) contracted position in [-2, 2]^3
# Output: density (1,), diffuse rgb (3,), view feature (4,)
def query_scene(x):
    # Trilinear read of the low-resolution grid, L = 512  [§4.1]
    grid_value = trilinear(sparse_grid, x)            # (8,)
    # Bilinear reads from each high-resolution plane, R = 2048  [§4.1]
    plane_value = sum(bilinear(plane, x) for plane in planes)  # (8,)
    # Sum the two reads, then split the channels  [eq. 5]
    raw = grid_value + plane_value                    # (8,)
    # Nonlinearities applied after interpolation  [eq. 6]
    density = exp(raw[0])
    diffuse_rgb = sigmoid(raw[1:4])
    view_feature = sigmoid(raw[4:8])
    return density, diffuse_rgb, view_feature

Contraction for unbounded scenes (step 3). Mip-NeRF 360's spherical contraction curves straight rays, which makes ray-box intersection hard §4.2. MERF instead uses a piecewise-projective map. Inside the unit L-infinity cube it is the identity. Outside, the coordinate with the largest magnitude is mapped by a projective transform and the others are divided by the largest magnitude §4.2, eq. 7. The map has seven regions, and straight rays stay piecewise straight, so standard ray-AABB tests apply in contracted space §4.2.

\text{contract}(\mathbf{x})_j = \begin{cases} x_j & \|\mathbf{x}\|_\infty \le 1 \\ x_j / \|\mathbf{x}\|_\infty & x_j \ne \|\mathbf{x}\|_\infty > 1 \\ (2 - 1/|x_j|)\, x_j/|x_j| & x_j = \|\mathbf{x}\|_\infty > 1 \end{cases} eq. 7

# Piecewise-projective contraction  [§4.2] [eq. 7]
def contract(x):
    # Inside the L-infinity unit cube the map is the identity  [eq. 7]
    if max(abs(x)) <= 1:
        return x
    # The largest-magnitude coordinate j gets a projective map  [eq. 7]
    j = argmax(abs(x))
    scale = max(abs(x))
    # The other coordinates are divided by the largest magnitude  [eq. 7]
    y = x / scale
    y[j] = sign(x[j]) * (2 - 1 / scale)
    return y

Training with hash-encoded grids (step 4). Instead of storing the grid and planes directly, MERF parameterizes them with one MLP that uses a multi-resolution hash encoding §5.1, app. B. The MLP is queried at the virtual grid corners, and its outputs are interpolated with the same bilinear or trilinear weights the renderer uses §5.1. A proposal network with hash encoding, following mip-NeRF 360, places samples near surfaces §5.1, app. B.

Quantization-aware training (step 5). The eight channels are squashed into (0, 1), rounded to one byte with a straight-through gradient, then mapped to [-m, m]. The authors use m = 14 for density and m = 7 for colour and features §5.2.

\tilde{\mathbf{t}}' = 2m \cdot q\big(\sigma(\tilde{\mathbf{t}})\big) - m eq. 8

q(x) = x + \text{stop-grad}\!\left(\frac{\lfloor 255\, x + \tfrac{1}{2} \rfloor}{255} - x\right) eq. 9

# Simulated 8-bit storage of one grid channel during training  [§5.2]
# m = 14 for density, m = 7 for diffuse colour and features  [§5.2]
def simulate_storage(t, m):
    # Squash to (0, 1) with a sigmoid  [eq. 8]
    u = sigmoid(t)
    # Round to 255 steps; the gradient passes straight through  [eq. 9]
    u_rounded = floor(255 * u + 0.5) / 255
    u_quantized = u + stop_gradient(u_rounded - u)
    # Map (0, 1) to [-m, m]  [eq. 8]
    return 2 * m * u_quantized - m

Baking (step 6). After training, MERF renders all training rays and keeps the weighted points they produce §5.3. A voxel is occupied if one of its eight corners comes from a point whose volume-rendering weight and alpha are both above 0.005. Alpha is computed with the step size of the real-time renderer, not the training step, which culls more aggressively §5.3. Voxels the proposal network never samples are not stored §5.1, §5.3. The three planes and the grid are then written out, the grid in a block-sparse layout after SNeRG, with max-pooled occupancy grids and PNG-encoded textures §5.3.

# Occupancy from training rays for baking  [§5.3]
# Threshold 0.005 on both weight and alpha  [§5.3]
def bake_occupancy(train_rays, threshold):
    # No voxel is occupied before the rays are rendered  [§5.3]
    occupied = zeros(grid_shape, dtype=bool)
    for ray in train_rays:
        # Weighted points from the training renderer  [§5.3]
        for point, weight in weighted_samples(ray):
            # Alpha uses the real-time renderer's step size  [§5.3]
            alpha = 1 - exp(-density(point) * renderer_step(point))
            if weight > threshold and alpha > threshold:
                # Mark the voxels around the point  [§5.3]
                occupied[neighbours_of(point)] = True
    return occupied

Real-time renderer (step 7). A WebGL fragment shader ray-marches a hierarchy of occupancy grids, coarse to fine, and jumps across empty blocks using the ray-AABB test of the contraction §6. Density and appearance are stored as separate textures, and appearance is read only where the density gives non-zero opacity §6. The deferred MLP is unrolled and laid out for linear memory access, which the authors report as a 4x speed-up §6.

Key equations are eq. 3, eq. 5 to eq. 9. Architecture details are in app. B.

Hyperparameters

Results

Author-reported. Setup: mip-NeRF 360 dataset, five outdoor and four indoor scenes; unless stated otherwise, metrics average the five outdoor scenes §7. The hardware for the real-time comparison is a MacBook (M1) at 1280x720 and an RTX 3090 at 1920x1080 §7.3.

Quality on mip-NeRF 360 tab. 1

Ablations, outdoor average tab. 2

Speed and memory tab. 3. The table gives VRAM and DISK without units.

Figure 4 plots PSNR against VRAM for MERF at several grid resolutions, SNeRG++ and Mobile-NeRF, and an ablation without the 3D grid fig. 4. Figures 5 to 7 are qualitative comparisons fig. 5 fig. 6 fig. 7.

Paper vs code

Official code: google-research/google-research, subdirectory merf/, linked from the authors' project page. It is Apache-2.0 (root LICENSE; the MERF subdirectory has no LICENSE of its own), cloned at commit 83f1e7b59e762648c953de751d31f75720683622. Copyright headers in the code read 2026, so the code may be newer than the paper's experiments. Checked configs, models, baking, training loop, losses and the webviewer shader.

Discrepancies found:

• Density activation. Paper: density is exp of the raw value eq. 6. Code: exp(x - 1) math.py [code: merf/internal/models.py#L192]. Exp input is capped at 88 [code: merf/internal/math.py#L43-L45].
• Regularization. Paper: weight decay 0.1 on grid values app. A. Code: a penalty with multiplier 0.03 on the hash parameters of the density MLP and the proposal MLP, averaged over parameters configs.py train_utils.py. With alpha = 2 the penalty is the quadratic case of the robust loss math.py.
• Loss terms. Paper states none of these. Code uses a Charbonnier data loss train_utils.py, a mip-NeRF 360 interlevel loss with weight 1.0 train_utils.py, a distortion loss with weight 0.01 train_utils.py, and a sparsity term with weight 0.01 on 2^17 random points in [-2, 2]^3 [code: merf/internal/train_utils.py#L238-L256] train_utils.py.
• Gradient clipping. Norm clip at 0.001 is on by default [code: merf/internal/configs.py#L123] train_utils.py. Paper states none.
• Training-time alpha culling. The code zeroes density whose alpha, using the voxel size as step, is below a threshold models.py. The threshold is 0 until step 10000 and rises log-linearly to 0.005 at step 20000 configs.py. Paper describes culling only at bake time §5.3.
• Occupancy levels. Paper: filter sizes 16, 32, 128 §6. Code builds factors 8, 16, 32, 64, 128 and renders with the factor-16 grid [code: merf/train.py#L429-L440]. The shader declares five occupancy levels [code: merf/webviewer/fragment.glsl#L488-L497].
• Termination. Paper: stop below transmittance 2e-4 §6. Shader: stop when visibility is at most 1/255 fragment.glsl. Appearance is read only when alpha exceeds 0.5/255 fragment.glsl.
• Larger steps when occluded. The shader multiplies the step by up to 8 when visibility is low fragment.glsl. This is on unless quality is high index.js. The paper does not mention it.
• Shared MLP. Paper: grids parameterized by MLPs §5.1. Code: one MLP produces both the grid and plane values models.py.
• Batch and memory. Paper: batch 2^16 pixels app. A. Code: 2^16 with eight gradient-accumulation steps by default configs.py configs.py. The README gives the paper's hardware as eight V100 GPUs with 16 GiB each, about two hours per scene [code: merf/README.md#L64-L70].

Agreements: 25000 steps, Adam betas and epsilon, learning-rate endpoints, 20 hash levels with 2^21 entries, the proposal network settings, quantization ranges, R = 2048 and L = 512 merf.gin, and the baking threshold of 0.005 train.py. Contraction matches [code: merf/internal/coord.py#L53-L73] §4.2.

Pitfalls

• Unstated hash-encoding details. Features per level, grid size range and initialization are not in the paper. Code uses 2 features per level, grid sizes 16 to 8192, init range 1e-4 / 10, and output scale 10 [code: merf/internal/hash_encoding.py#L77-L83] hash_encoding.py hash_encoding.py.
• Density shift. A port that follows eq. 6 literally will not match the released models. Follow the code's exp(x - 1) if reproducing them.
• Train/bake mismatch. The near-lossless baking result tab. 2 is produced by code that also culls during training, a step the paper does not describe. An implementation without that step may not reproduce tab. 2.
• Gradient clipping at 0.001 is a strong constraint on update size and is not stated in the paper.
• Renderer differences. The paper's termination (2e-4) and occupancy levels differ from the shader. The larger-steps shortcut trades quality for speed.
• Memory versus batch. The README says a batch of 2^16 leads to out-of-memory errors on a single GPU, so the code accumulates eight steps by default [code: merf/README.md#L64-L70].
• Claims not in the tables. The paper says "one fifth" of memory versus SNeRG++ §7.3. Table 3 gives 524 against 4571 (about 11%, roughly one eighth). The conclusion's "31.6% (MSE)" §8 is not in any table.
• Units. VRAM and DISK in tab. 3 have no units.
• "On par" with SVS §7.1 is an author statement. SSIM is 0.616 against 0.662 tab. 1.
• Indoor averaging. The paper does not state how indoor metrics are averaged.
• Numbering. Equation, table and figure numbers here follow LaTeX source order. Check against the compiled PDF before citing.
• Trade-off in limitations. The view-dependence model is evaluated once per ray, so semi-transparent objects are not modelled faithfully. The small MLP may not scale to much larger scenes or complex reflections §7.4.

Lineage

• Builds on: SNeRG for deferred shading and baking §3; mip-NeRF 360 for the contraction idea and the proposal sampler §3, §4.2, §5.1; Instant-NGP for the multi-resolution hash encoding §5.1; Plenoxels for the voxel grid and for the fine-tuning idea it is contrasted with §1, §5.1; triplane representations for the planes §2. None of these have cards in this wiki, so they are named, not linked.
• Compared against: NeRF, NeRF++, Stable View Synthesis (SVS), mip-NeRF 360, Instant-NGP, Deep Blending, Mobile-NeRF and an improved SNeRG (SNeRG++) §7.
• Superseded by: not stated in this wiki.