Frequency regularization

Overview

A neural network fed raw coordinates through a sinusoidal encoding can learn high-frequency noise before it has learned the coarse shape. With few training images this is the failure that matters: the model fits the training views and invents geometry, such as blobs near the camera, in views it has not seen 2303.07418 §3.2 fig. 1. Frequency regularization fixes the timing, not the model. It gates the encoding so that only the low frequencies are visible at first and the higher ones are revealed as training proceeds 2303.07418 §3.2 eq. 4. The invariant is that the full encoding is visible at the end of the schedule, so the final model has the same inputs as a plain NeRF; only the order in which they are learned changes. For the general problem, see Few-shot learning.

Variants

Linearly revealed mask on the NeRF positional encoding

Multiply every entry of the encoded input (the raw coordinates plus their sine/cosine bands) by a visibility weight. The weight is 1 for the bands that are fully visible at step t, a fractional value for the next three entries, and 0 for the rest. The visible prefix grows linearly with t, at L/T entries per step, so the encoding is fully visible at the schedule end T 2303.07418 §3.2 eq. 5. The same kind of mask is applied to the view-direction encoding 2303.07418 §4.1. Card: FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization.

• Blender, 8 views, PSNR: 24.259 for FreeNeRF against 23.147 for DietNeRF and 13.931 for the reproduced NeRF baseline 2303.07418 tab. 1.
• DTU, object PSNR, 3 / 6 / 9 views: 19.92 / 23.25 / 25.38, against 18.50 / 22.18 / 24.88 for the reproduced RegNeRF baseline 2303.07418 tab. 2.
• LLFF, PSNR, 3 views: 19.63, against 18.84 for the reproduced RegNeRF baseline 2303.07418 tab. 3.
• Training-time multiplier over each baseline: 1.02x (Blender), 1.04x (DTU and LLFF) 2303.07418 tab. 4. These are wall-clock figures on the authors' setup.
• The paper pairs this mask with a separate penalty on density near the camera 2303.07418 §3.3. That penalty is not frequency regularization, and this page does not cover it.

The paper distinguishes this approach from the coarse-to-fine frequency schedules in Nerfies and BARF, saying it targets few-shot failure modes 2303.07418 §3.2. Neither has a card in this wiki, so they are not described here.

When to use which

Schedule end T: the paper's default is 90% of training for 3 views and 70% for 6 views 2303.07418 §4.1. For 9 views the paper gives two values, see Contradictions. The authors report that a longer schedule gives higher PSNR but lower LPIPS, and that fine-tuning the trained model can recover LPIPS 2303.07418 §4.3. The per-T results are only in a figure, so the page does not quote them 2303.07418 fig. 7.

Attribution is limited. The paper's ablation removes the occlusion penalty 2303.07418 tab. 5, and the text does not give numbers for a run without frequency regularization; the paper shows that run only in a figure 2303.07418 fig. 7. So the page cannot say how much of the gain comes from the mask alone.

Implementation sketch

The core is one mask multiplied into the encoded input. The paper summarizes the schedule as a single line of code 2303.07418 §3.2.

# Visibility weight for each encoded entry at step t  [arxiv:2303.07418 eq. 5]
# Inputs: step t, schedule end T, encoded band count L
# Output: list of L + 3 weights in [0, 1]
def visibility(t, T, L):
    # Progress along the schedule, in encoded entries  [arxiv:2303.07418 eq. 5]
    progress = t * L / T
    fraction = progress - floor(progress)
    weights = []
    for i in range(1, L + 4):
        # Entries up to progress + 3 are fully visible  [arxiv:2303.07418 eq. 5]
        if i <= progress + 3:
            weights.append(1.0)
        # The next three entries are faded in  [arxiv:2303.07418 eq. 5]
        elif i <= progress + 6:
            weights.append(fraction)
        # Higher frequencies stay hidden  [arxiv:2303.07418 eq. 5]
        else:
            weights.append(0.0)
    return weights

# Masked encoding of one input vector  [arxiv:2303.07418 eq. 4]
# Inputs: raw vector x, step t, schedule end T
# Output: encoded vector of length L + 3
def masked_encoding(x, t, T, L):
    # Raw input followed by its sine/cosine bands  [arxiv:2303.07418 eq. 2]
    encoded = concat(x, positional_encoding(x, L))
    # Scale each entry by its visibility weight  [arxiv:2303.07418 eq. 4]
    return encoded * visibility(t, T, L)

Where implementations may differ:

• The mask applies to view directions too, but the paper does not give the direction encoding length 2303.07418 §4.1.
• The paper states the mask with L as the encoded length of the coordinates. The paper card records how the released code's ramp differs slightly from eq. 5; see the paper card.
• The paper's photometric loss form is not given in its text, so the loss in a training step must come from the baseline you build on.

Contradictions

• Schedule end for 9 views. The main text gives 20% of training 2303.07418 §4.1. The appendix gives 50% 2303.07418 app. A.1. The two values are not reconciled in the paper, so check it before picking one. The paper's 9-view comparison reports an object PSNR of 25.59 at a 90% schedule and 25.38 at a 50% schedule 2303.07418 app. A.1.

Papers

• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization: introduces the linearly revealed positional-encoding mask, with an occlusion penalty on near-camera density as a second regularizer.