InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering

In plain words

A neural radiance field (NeRF) learns a 3D scene from photographs taken from known camera positions, then draws new views of it. With only a handful of photos, the learned scene is often noisy and blurred, and it can memorise the training photos instead of the scene's shape. Along each camera ray, a clean scene should have its density concentrated at one or two spots, where the ray meets a surface, and close to zero everywhere else.

InfoNeRF adds a penalty that pushes each ray's density toward this concentrated shape. It measures how spread out the density is along the ray, using an entropy formula from information theory, and it can do this even for rays from camera positions that have no photo, because it needs no pixel colours for them. A second penalty asks that two nearby camera positions, slightly rotated from each other, produce similar density along their rays. This keeps the scene from collapsing into a trivial answer when all photos look alike.

The idea plugs into existing NeRF variants with a few extra rays per training step. The authors report clearly better few-shot results than plain NeRF on the synthetic benchmark and on a human-body dataset.

TL;DR

• Few-shot NeRF with 3 to 4 views, trained from scratch, improves on NeRF by adding two regularisers to the usual RGB loss.
• Ray entropy minimisation (sparser opacity along each ray) is the main gain; a KL term between rotated near-views helps mainly on narrow-baseline data such as DTU.
• Author-reported: 18.65 PSNR vs 15.93 for NeRF on Realistic Synthetic 360 at 4 views tab. 1.
• The official code has several settings that differ from the paper, most importantly the shipped synthetic config turns the KL term off. Read "Paper vs code" before reimplementing.

Problem

Few-shot NeRF fits a scene from very few calibrated views. The paper argues that two failure modes follow: reconstruction is inconsistent because the views leave geometry under-constrained §4.1, and the model overfits to seen images and converges to degenerate solutions, especially when the training views are close together §4.1. Plain NeRF therefore produces noise and blur, and it fails when the viewpoint shifts slightly §4.1. The paper also notes that NeRF cannot use rays from unseen images, since it needs their ground-truth colours §4.2.

Method

Background (NeRF). A MLP maps a 3D point and a viewing direction to a density σ and an RGB colour. A ray colour is a quadrature sum over samples of transmittance times opacity times colour §3. Points are sampled in two stages: uniformly first, then by importance sampling from the first stage's opacity §3. The baseline RGB loss is the mean squared error over seen-ray pixels §3.

Normalised ray density. For each sample i on ray r, the opacity is α_i = 1 − exp(−σ_i δ_i), with δ_i the sample spacing. The paper uses α rather than σ so that the irregular spacing δ_i is accounted for. The ray density is p_i = α_i / Σ_j α_j §4.2.

Ray entropy. The ray entropy is H(r) = −Σ_i p_i log p_i §4.2. The paper says this costs almost nothing extra, because σ and δ are already computed for volume rendering §4.2. The log base is not stated.

Disregarding non-hitting rays. A ray is masked out of the entropy term unless its cumulative density Q(r) = Σ_i α_i exceeds a threshold ε. The paper motivates this by low-entropy rays that hit nothing §4.2. The value of ε is not stated.

Entropy loss. The loss averages M(r) · H(r) over the seen rays R_s and the unseen rays R_u, normalised by |R_s| + |R_u| §4.2. The unseen rays come from randomly sampled poses of the training-set images, which have no ground-truth colour §4.2. The paper's text describes R_u as "randomly sampled unseen images" §4.2; the code draws from a different pool (see "Paper vs code").

Information-gain reduction (KL) loss. For an observed ray r, sample a ray r̃ from a slightly rotated camera pose and penalise D_KL(P(r) ‖ P(r̃)), the KL divergence between the two ray densities p §4.3. The rotation is drawn from −5° to +5° §4.3.

Overall objective. L_total = L_RGB + λ₁ L_entropy + λ₂ L_KL §4.4. L_RGB uses seen rays only. The λ values are not stated §4.4.

# InfoNeRF training step  [§4.2] [§4.3] [§4.4]
# Input:  seen rays R_s with colours; unseen rays R_u; a near pose per ray
# Output: scalar loss

def infonerf_loss(seen_rays, unseen_rays, near_rays, targets):
    # Volume-render all rays and keep per-sample opacities  [§3]
    colours, opacities = render(seen_rays, unseen_rays, near_rays)

    # Seen-ray photometric loss only  [§4.4]
    rgb_loss = mse(colours[seen], targets)

    # Normalised density per ray, p_i = alpha_i / sum_j alpha_j  [§4.2]
    density = opacities / sum(opacities, over_samples)

    # Ray entropy H(r) = -sum_i p_i log p_i  [§4.2]
    entropy = -sum(density * log(density), over_samples)

    # Keep only rays with total opacity Q(r) above epsilon  [§4.2]
    keep = cumulative_opacity(opacities) > epsilon

    # Average over seen and unseen rays  [§4.2]
    entropy_loss = mean(keep * entropy over seen and unseen rays)

    # KL between the density of r and of its slightly rotated copy  [§4.3]
    kl_loss = kl(density_of(seen rays), density_of(near_rays))

    return rgb_loss + lambda_1 * entropy_loss + lambda_2 * kl_loss  # [§4.4]

Architecture and sampling. The regularisers are tested on top of NeRF, and also plugged into PixelNeRF and mipNeRF §5.4, app. E. The network sizes, positional encoding and sample counts are not stated in the paper. The code uses NeRF-style defaults (see Hyperparameters).

Hyperparameters

Few-shot training views: 4 of 100 random views per Realistic Synthetic scene, resampled over 5 runs §5.1, tab. 1. ZJU-MoCap: 4 uniformly spaced views of 23 cameras §5.1. DTU: 3 of 49 views per scene, 15 scenes, 46 test views §5.1.

Results

Author-reported. Setup: Realistic Synthetic 360° (8 scenes), 4 training views, 200 test views per scene §5.1. Mean ± std over 5 runs tab. 1 caption.

Per-scene PSNR in tab. 2 has InfoNeRF in bold on all 8 scenes. The supplementary SSIM and LPIPS breakdowns are referenced in the text §5.3; their values are not reproduced here. The paper also reports that the entropy term alone helps PSNR on DTU but lowers SSIM §5.4. Qualitative comparisons are in fig. 1 and app. G.

Paper vs code

Code: repository mjmjeong/InfoNeRF at commit 4eac3a34096380803a436acc9197001b69bdfdbc. Its README links this paper's arXiv page and names the same three authors, and the owner name matches the first author's given name. The paper's abstract points to a project website that the repository also links; that site was not reachable at compile time. The repository has no LICENSE file, so it is cited by path and line only. Code paths below are relative to the repository root. The paper's project website was not reachable when this card was compiled.

1. Loss weights. The paper gives no λ₁, λ₂ or ε values §4.4, §4.2. The code uses λ₁ = 1e-3 on the synthetic config lego.txt and λ₁ = 2e-3, λ₂ = 2e-5, ε = 0.01 on DTU scan114.txt. The default ε in the parser is 0.1 run_nerf.py.

2. The shipped synthetic config does not use the KL term. The paper's method includes L_KL, and Table 1 is for InfoNeRF §5.3. The only synthetic config, lego, does not set smoothing, which defaults to off lego.txt; run_nerf.py. Reproducing Table 1 from this config therefore trains entropy only. The paper's ablation says the KL term matters less on synthetic data §5.4.

3. Learning-rate decay. The paper says ×0.1 every 250,000 iterations §5.2. The code default matches (lrate_decay = 250 run_nerf.py), but the synthetic config sets 500 lego.txt, which is 500,000 steps run_nerf.py.

4. Which rays enter the entropy term. The paper averages the entropy over R_s ∪ R_u §4.2. In the code, computing_entropy_all defaults to false, so the entropy is taken only over rays after the first N_rand in the batch loss.py. Those are the N_entropy unseen rays and, when smoothing is on, the near-pose rays run_nerf.py. The seen-view RGB rays are excluded.

5. Source of the unseen-ray poses. The paper says the unseen rays come from randomly sampled unseen images §4.2. The code draws the pose index with np.random.choice(len(images)) run_nerf.py. In the Blender loader, images holds all splits, not only the training views load_blender.py. Only i_train is used for RGB, so this pool includes poses of test frames. The code uses no test colours, but the poses are drawn from them. This is a reading of the code, not something I ran.

6. KL direction. The paper defines the loss as D_KL(P(r) ‖ P(r̃)) §4.3. The code calls KLDivLoss(p.log(), q) with p from the first half of the batch and q from the second half loss.py. PyTorch's convention makes this KL(q ‖ p), with the near-pose ray as the reference distribution. The two directions differ, so match the code if you want the published numbers.

7. Accumulated opacity for the mask. The paper defines Q(r) = Σ α_i §4.2. The code thresholds acc, which is the sum of the rendering weights weights = alpha · transmittance [code: run_nerf_helpers.py#L427-L429; utils/loss.py#L44, #L64]. These are not the same quantity.

8. Entropy log base and normalisation. The paper does not state the log base. The code uses log2 with an epsilon of 1e-10 by default loss.py. The normaliser is the mean over the rays in the batch that pass the entropy slice, not |R_s| + |R_u| as written loss.py.

9. Coarse-network RGB loss. The paper's L_RGB has one term §3, §4.4. The code adds the coarse network's RGB loss, which NeRF also does run_nerf.py. The paper does not mention it.

10. Near-pose rotation. The paper draws the rotation from −5° to +5° §4.3. The code samples each axis uniformly in ±5° generate_near_c2w.py. The default mode also rotates the camera position about the origin [code: utils/generate_near_c2w.py#L43-L50; run_nerf.py#L573]. The paper says only "rotating the camera pose".

11. Equation for T_i. In §3 the paper writes the transmittance as T_i = Σ_j σ_j δ_j. This drops the exponential. The code uses the product of (1 − α_j) run_nerf_helpers.py. Use the product form.

12. DTU coverage. The paper reports 15 DTU scenes §5.1. The repository ships one DTU config, scan114, with fixed training views 22, 25 and 28 scan114.txt.

13. Text vs caption, ray-count ablation. The text says the images were halved §5.4. The table caption says the 4-view setting tab. ablation_ray_num. The code does not reproduce this ablation.

14. Synthetic training views. The paper samples 4 views at random per scene and reports 5 runs §5.1, tab. 1. The shipped config fixes train_scene = [26, 86, 2, 55] lego.txt. Only the Lego scene is provided.

15. Other code-only settings. The code adds a centre-crop warm-up for 500 iterations with a crop fraction of 0.5 lego.txt, and DTU uses raw noise of std 1.0 scan114.txt. The paper states neither.

No discrepancies were checked against the project website, which did not resolve.

Pitfalls

• Extra rays per step. The code batch is the seen rays plus N_entropy unseen rays, and with smoothing also the two near-pose sets. That is roughly 2× to 4× the rays per step of plain NeRF with the paper's settings run_nerf.py. The paper calls the entropy cost "negligible", and that holds per ray, not per step §4.2.
• ε is critical. The mask uses it, and the paper gives no value. The code uses 0.1 by default and 0.01 on DTU. A wrong value changes which rays get pushed toward sparsity.
• Scale of the λ values. λ₁ and λ₂ are not given in the paper. The code values differ by dataset, so a single setting will not transfer.
• Sign and direction of KL. Implementing D_KL(P(r) ‖ P(r̃)) as written in the paper gives different gradients from the code's direction (see Paper vs code, item 6).
• Entropy-only can hurt SSIM. On DTU, the entropy term alone raises PSNR but lowers SSIM §5.4. The paper attributes this to overfitting to seen views and uses KL to correct it.
• Synthetic result depends on the KL term. The shipped synthetic config turns KL off, so runs from it do not match the full method's ablation.
• Standard deviations. Only the synthetic table reports them, from 5 runs. DTU and ZJU-MoCap numbers are single values tab. 4, human_results. The synthetic std is also large for some scenes (e.g. NeRF on Lego ± 4.53).
• Comparisons are not like-for-like. PixelNeRF is pretrained on DTU with dense views, so its numbers use a dataset prior §5.3. NB uses an SMPL body model §5.3. InfoNeRF does not.
• Calibration assumed. The method needs calibrated poses for both seen and unseen rays. The paper lists calibrated cameras as a limitation app. H.
• Robustness. The authors say the method is more vulnerable to adversarial attacks app. H. This is their statement, not a measured result.
• Benefit fades with more views. Gains over NeRF shrink as the number of training views grows, and saturate beyond about 8 views on Realistic Synthetic §5.4, app. D.

Lineage

• Builds on: NeRF §3; the sampling and opacity formulation of volume rendering §3. Entropy in the Shannon sense, from Shannon 1948 §4.2.
• Compared against: NeRF, DietNeRF §5.3, PixelNeRF §5.3, Neural Volume and Neural Body on ZJU-MoCap §5.3. Plug-in tests with PixelNeRF and mipNeRF app. E.
• Superseded by: not stated in the paper. No later version is recorded in this wiki.