FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

In plain words

A neural radiance field (NeRF) learns a 3D scene from photos and can then draw it from new camera angles. With only a handful of photos it tends to memorize the photos instead: it gets the training views right and invents walls or floating blobs close to the camera in the views it has not seen.

The authors show that a large part of the problem comes from fine detail being switched on too early. NeRF's inputs are passed through sine and cosine waves of increasing frequency. If the fine waves are visible from the first training step, the network locks onto noise. The fix is a schedule: at the start only the coarse waves are visible, and finer ones are revealed step by step as training goes on.

The second idea covers the gaps that a few photos leave. Where no training photo sees a surface, the model tends to put density right in front of the camera. The paper penalizes density in the samples nearest the camera.

Both changes are small additions to a plain NeRF. Neither needs a pre-trained model or extra rendering passes.

TL;DR

Plain NeRF overfits badly with few input views. FreeNeRF gates the positional encoding so that only low frequencies are visible at the start and then linearly reveals higher ones, and adds an occlusion penalty on near-camera density. Both are cheap and need no extra rendering or pre-training. The authors report state-of-the-art PSNR on Blender, DTU and LLFF, with training time 1.02–1.04× the baseline tab. 4.

Problem

Under few-shot inputs (3, 6 or 9 views in the experiments), NeRF overfits the training images and fails to recover scene geometry for novel views §3.2 fig. 1. Earlier fixes either pre-train on large multi-view datasets, add external depth or semantic supervision, or render extra patches from novel poses for regularization, and these come with extra computation or pipeline complexity §1 §2.

Method

The method adds two regularizers to a NeRF baseline. Both are applied during training only. The paper implements them on two baselines: DietNeRF's codebase (NeRF) for Blender and RegNeRF's codebase (mip-NeRF) for DTU and LLFF §4.1.

Preliminaries. NeRF maps a 3D point and viewing direction to density and colour with an MLP. Its inputs are first concatenated with a sine/cosine encoding of frequencies up to 2^{L-1} eq. 1 eq. 2, and colour is obtained by volume rendering the samples along each ray eq. 3 §3.1.

Frequency regularization §3.2.

1. Form the encoded input: raw coordinates plus their L encoded entries eq. 2.
2. Multiply each entry by a visibility weight \alpha_i(t, T, L), where t is the current step and T the last step of the schedule eq. 4 eq. 5. The weight is 1 for the first tL/T + 3 entries, a fractional value for the next three, and 0 beyond that eq. 5.
3. Apply the same kind of mask to the view-direction encoding. The paper says both coordinates and directions are regularized §4.1, but does not give the direction encoding length.
4. Schedule: at the start only the raw input and a few low bands are visible. Visibility grows by three entries (one 3-bit band) per step of the schedule and reaches full length at T §3.2. The paper calls this a "frequency curriculum" §4.3.

Occlusion regularization §3.3.

1. Take the densities \sigma_k of the K samples along each ray, ordered near to far.
2. Set a binary mask m_k = 1 for the first M samples and 0 for the rest.
3. Penalize the mean of \sigma_k m_k over the K samples eq. 6, with weight 0.01 §4.1 app. B.2.
4. The penalty is applied to the outputs of the second (fine) stage of the NeRF app. B.2.

# Visibility mask for the encoded inputs  [eq. 5]
# Inputs: step t, last regularized step T, encoded length L (96 for x)
# Output: alpha, one weight in [0, 1] per encoded entry (L + 3,)
def frequency_mask(t, T, L):
    # Progress through the schedule, in units of encoded entries  [eq. 5]
    progress = t * L / T
    alpha = []
    for i in range(1, L + 4):
        # Entries up to progress + 3 are fully visible  [eq. 5]
        if i <= progress + 3:
            alpha.append(1.0)
        # The next band of three is faded by the fractional part  [eq. 5]
        elif i <= progress + 6:
            alpha.append(progress - floor(progress))
        # Higher frequencies stay hidden  [eq. 5]
        else:
            alpha.append(0.0)
    return alpha

# Masked encoding of one input vector  [eq. 2] [eq. 4]
# Inputs: raw vector x (3,), step t, last step T
# Output: encoded vector (L + 3,)
def masked_encoding(x, t, T, L):
    # Concatenate the raw input with its sine/cosine encoding  [eq. 2]
    encoded = concat(x, positional_encoding(x, L))  # (L + 3,)
    # Scale each entry by its visibility weight  [eq. 4]
    return encoded * frequency_mask(t, T, L)

# Occlusion penalty for a batch of rays  [eq. 6] [§3.3]
# Inputs: densities (B, K) ordered near to far; reg_range = M
# Output: scalar penalty
def occlusion_loss(densities, reg_range):
    # Mask is 1 for the M samples nearest the camera, 0 for the rest  [§3.3]
    index = arange(K)
    mask = (index < reg_range).astype(densities)  # (K,)
    # Average over the K samples, then over the batch  [eq. 6]
    return mean(densities * mask)

# One training step  [§3.2] [§3.3] [§4.1]
# Inputs: a batch of rays and target pixels, step t, schedule end T
# Output: scalar training loss
def training_loss(rays, pixels, t, T, M):
    # Render with masked encodings; keep the fine-stage densities  [app. B.2]
    rgb, densities = render(rays, encoder=masked_encoding, t=t, T=T)
    # Photometric term: the loss form is not stated in the paper text
    photometric = reconstruction_loss(rgb, pixels)
    # Occlusion term, weight 0.01  [§4.1]
    occlusion = occlusion_loss(densities, reg_range=M)
    return photometric + 0.01 * occlusion

Key equations, as in the paper:

\gamma_L(\mathbf{x}) = [\sin(\mathbf{x}), \cos(\mathbf{x}), \dots, \sin(2^{L-1}\mathbf{x}), \cos(2^{L-1}\mathbf{x})] \quad [\text{eq. 1}]

\mathbf{x}' = [\mathbf{x}, \gamma_L(\mathbf{x})] \quad [\text{eq. 2}]

\gamma_L'(t,T;\mathbf{x}) = \gamma_L(\mathbf{x}) \odot \boldsymbol{\alpha}(t,T,L) \quad [\text{eq. 4}]

\mathcal{L}_{occ} = \frac{1}{K}\sum_{k=1}^{K}\sigma_k\, m_k \quad [\text{eq. 6}]

Hyperparameters

Paper values are cited to the paper; code values to the repository at code_commit. Blender configs are not in this repository, so Blender values are paper-only.

Results

Author-reported. Each row is a single run. Setups follow DietNeRF (Blender, 8 views) and RegNeRF (DTU and LLFF) §4.1. Metric values are the paper's.

Blender, 8 views, PSNR / SSIM / LPIPS tab. 1:

DTU, object PSNR (background masked), 3 / 6 / 9 views tab. 2:

DTU, full-image PSNR: FreeNeRF 18.02 / 22.39 / 24.2 for 3 / 6 / 9 views tab. 2.

LLFF, PSNR / SSIM / LPIPS, 3 / 6 / 9 views tab. 3:

Training-time multiplier relative to each baseline tab. 4: FreeNeRF 1.02× (Blender), 1.04× (DTU), 1.04× (LLFF). DietNeRF 2.8× (Blender). RegNeRF 1.69× (DTU) and 1.98× (LLFF).

Ablations on DTU and LLFF, 3 views, batch size 1024 unless noted tab. 5: removing the occlusion loss drops DTU-3 to 17.40 (from 19.81 at M=10); the distortion loss of Barron et al. (2022) gives 15.14 on DTU-3. Varying the frequency schedule end T gives a PSNR and LPIPS trade-off fig. 7 §4.3.

Paper vs code

The official repository is Jiawei-Yang/FreeNeRF, linked from the project page and owned by the first author. Checked at code_commit.

• Frequency mask ramp. Paper eq. 5 uses L, the encoded length of coordinates (96 for 16 bands) eq. 5. The code builds a 99-entry mask and moves a pointer of 33 \cdot t/T + 1 band-units from the start math.py train.py. The code reveals entries about 3% faster than eq. 5 with L = 96. It also reaches full visibility when the pointer is clipped at 33, about 0.97T, while the paper's formula reaches it at T.
• 9-view schedule. Main text says 20% §4.1; appendix and code say 50% app. A.1 llff9_freenerf.gin dtu9_freenerf.gin.
• Occlusion mask. The paper defines the mask by sample index up to M §3.3. The code uses the first reg_range sample indices math.py. Both match, but the penalized distance depends on sample spacing, which the paper does not state.
• Occlusion stage. Paper says the second stage app. B.2. Code uses the last ray-history entry, which is the fine stage train.py. Consistent.
• Weights, M values, LR, clipping, batch size, iterations. Match the paper's stated values llff3_freenerf.gin dtu3_freenerf.gin.
• Density noise 1.0. Set in the configs llff3_freenerf.gin but not mentioned in the paper.
• Ablation vs main tables. The ablation's default at batch size 4096 gives DTU-3 20.20 and LLFF-3 19.73 tab. 5, while the main tables give 19.92 and 19.63 tab. 2 tab. 3. The paper does not explain the difference.
• Overhead text vs table. Text says baselines are 1.62–2.8× §4.2, but the lowest baseline value in tab. 4 is 1.69×.
• Blender. No Blender config in this repository, so the Blender numbers could not be checked against code.
• License. The repository has no LICENSE file (GitHub reports none). Code is cited for facts only; nothing is quoted.

Pitfalls

• Schedule for 9 views. The main text (20%) and the appendix and code (50%) disagree. Use 50%, which matches the code.
• Mask ramp. Implement it as in the code if you want to match the released numbers. Following eq. 5 literally gives a slightly slower reveal.
• Mask on view directions. Paper says directions are masked too but gives no encoding length. The code uses the 27-entry direction encoding with degree 4 models.py models.py.
• Numerical clipping. In the code the mask is clipped to [10^{-8},\,1-10^{-8}] for stability math.py. The paper does not mention it.
• Occlusion penalty is index-based. M counts samples, not distance. Its effect depends on how the samples are spaced and on the near bound. The paper says its penalty differs from changing the near bound §4.3, and the DTU-3 result drops sharply when M is large (11.02 at M=25) tab. 5.
• Black/white prior shape. The code thresholds the mean ray colour and applies the penalty over occ_wb_range columns math.py. The indexing implies a 2-D array, but its exact shape was not checked. Verify before implementing this part.
• Photometric loss. Its form is not stated in the paper text, and the card does not fill it in. Check the code if you need an exact match.
• Batch size. Main results use 4096 rays; ablations use 1024. Mixing them changes the numbers, as the ablation above shows.
• Depth evaluation. There is no ground-truth depth. The "depth error" in app. A.2 is measured against depth from mip-NeRF trained on all views, so it measures agreement with that model, not accuracy.
• Known failures. The occlusion term over-regularizes some DTU scenes (an incomplete white desk) and does not remove distant floaters app. A.1 fig. A.3. Longer frequency schedules raise PSNR but can lower LPIPS §4.3.
• Timing. The 1.02–1.04× overhead is wall-clock on the authors' setup, not a FLOP count tab. 4.
• Baseline numbers. Blender and DTU baselines (except mip-NeRF and RegNeRF reproductions) are copied from the DietNeRF and RegNeRF papers §4.1.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020) for the baseline encoding and volume rendering §3.1; mip-NeRF (Barron et al., 2021) as the DTU and LLFF baseline §4.1; DietNeRF's codebase for Blender §4.1.
• Related frequency schedules: coarse-to-fine frequency schedules in Nerfies (Park et al., 2021) and BARF (Lin et al., 2021). The paper distinguishes its work by targeting few-shot failure modes §3.2.
• Compared against: DietNeRF, RegNeRF, PixelNeRF, MVSNeRF, SRF and NV, plus mip-NeRF and simplified NeRF §4.2 tab. 1 tab. 2 tab. 3.
• Superseded by: not known from content already in this wiki.