Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

In plain words

A neural radiance field (NeRF) is a learned 3D model of one scene. It is trained so that its renderings match the photos it was shown, and it can then draw the scene from new camera positions. The catch is that it needs many photos. With only a handful, it can fit the photos by putting wrong geometry right in front of the cameras, and the views from other angles come out broken.

This paper's fix is to also check the scene from camera positions that have no photo. For such a view, the model cannot compare pixels, because there is no matching photo. Instead it compares meaning: a pretrained image model (CLIP) turns both the new view and a real photo into a short summary vector, and the training nudges the two summaries to agree. The same object should still count as the same object from the side or from above, even when the pixels differ.

The method keeps the usual NeRF training and adds this meaning check on top, and a short extra phase that uses only the pixel comparison refines the detail. On synthetic object scenes with eight photos, the authors report clearly better perceptual scores than plain NeRF. The approach also works on real photos of objects when a model was first trained on other scenes, and it can fill in parts of an object that no photo showed.

TL;DR

DietNeRF trains a NeRF from few posed photos with the standard pixel loss eq. 3 plus a semantic consistency loss §4.1. The semantic term renders a random, unseen pose and maximises the cosine similarity between the frozen CLIP ViT embedding of that render and the embedding of a random training image [alg. 1] eq. 5. Author-reported, on 8-view Realistic Synthetic scenes: 23.147 PSNR and 74.9 FID for the semantic-loss model, against 14.934 PSNR and 228.1 FID for NeRF; a 50k-iteration pixel-only fine-tune reaches 23.591 PSNR and 72.0 FID tab. 1. The paper also reports single-view fine-tuning of pixelNeRF on DTU tab. 2 and reconstruction of an unseen side of the Lego scene tab. 3.

Problem

With few views, NeRF's pixel loss is only applied at the training poses. It can therefore reach a low training error with degenerate geometry, such as placing density near the training cameras. Novel views then show artifacts §3. Regularising the architecture (single MLP, lower positional frequency, lower learning rate) removes the worst artifacts but also removes fine detail §3.

Two further limits are named. First, a NeRF trained per scene has no prior knowledge, so it cannot complete regions that were never observed. Second, multi-view generalising models trained on many scenes are often blurry or restricted to one object category §1. The paper's stated assumption is that large single-view datasets such as ImageNet or CLIP's web data provide the prior, because an image encoder's features for different views of one object are similar §4.2 fig. 3.

Method

Base representation. The scene is a NeRF f_θ that maps a 3D position and a viewing direction to a colour and a density §2. Each pixel is the volume rendering integral along its camera ray eq. 1. The paper uses NeRF's coarse and fine networks with hierarchical sampling §2. The released code applies the same loss to both networks run_nerf.py.

Pixel loss. Standard MSE over sampled rays from the observed views eq. 2 eq. 3. In practice a batch of rays is drawn from the training images §3.

Semantic consistency loss. Let φ be a frozen image encoder (CLIP ViT, or an ImageNet ViT for the ablation). The loss for a rendered image Î and a training image I is written as a scaled squared distance between embeddings eq. 4. For unit-norm embeddings it reduces to a cosine-similarity term up to a constant and a rescaling that is absorbed into λ eq. 5 §4.1. The paper says this term is minimised. The code minimises the negative cosine similarity, which pushes the two embeddings together run_nerf.py.

Pose sampling. For bounded 360° scenes the rendered pose is drawn uniformly from the upper hemisphere, with a radius drawn from a bounded range §4.3. For unbounded forward-facing scenes the paper interpolates between three known training poses with random weights §4.3. The released configs use only the hemisphere sampler, with the angle and radius ranges given under Hyperparameters load_blender.py. The interpolation branch exists in the code run_nerf.py but no released config uses it.

Efficiency. The semantic render uses about 15 to 20% of the rays of a full-resolution image, on a strided grid over the whole image plane §4.4. The loss is applied only every k iterations, with k between 10 and 16 working well; the paper calls this lazy regularisation §4.4. Backpropagation through rendering uses rematerialisation of intermediate activations §4.4.

Training procedure. Algorithm 1 summarises the loop alg. 1. The target embeddings are computed once. Each iteration adds the MSE term, and every K iterations it adds the semantic term for one random training target and one random pose. The paper then fine-tunes with the MSE alone for 20 to 70k iterations to recover detail §4.4.

# Train one DietNeRF on one scene  [alg. 1]
# views: list of (image, pose) observed by the user
# phi: frozen image encoder, returns one embedding vector per image
def train_dietnerf(views, phi, sample_pose, K, lam, num_iters):
    # Embed every observed image once, before training  [alg. 1] [§4.1]
    target_embeddings = [phi(image) for image, _ in views]
    nerf = init_radiance_field()
    for step in range(num_iters):
        # Pixel loss on a batch of rays from the observed views  [eq. 3]
        rays, target_colors = sample_rays(views)
        loss = mse(render(nerf, rays), target_colors)
        if step % K == 0:
            # Render an unseen pose at reduced resolution  [§4.4]
            pose = sample_pose()
            rendered = render_low_res(nerf, pose)
            # Compare meaning, not pixels, with a random target  [§4.1]
            target = random_choice(target_embeddings)
            # Minimise the negative cosine  [eq. 5] [code: run_nerf.py#L1064]
            loss = loss - lam * cosine(target, phi(rendered))
        update(nerf, loss)
    return nerf

Hyperparameters

Simplified NeRF (the baseline tuned per scene) uses the values in tab. 6. The table lists them per scene; the released configs set the same values, for example the Lego config 312_blender_lego_8views_tune.txt.

Results

Author-reported. Setup: Realistic Synthetic scenes with eight random training views; metrics computed on 25 test views per scene; FID and KID use all 3200 ground-truth images against 200 renders app. A tab. 1. Baselines are NeRF, Neural Volumes (NV) and Simplified NeRF, all trained by the authors.

DTU single-view synthesis (pixelNeRF checkpoint, one source view). NeRF and pixelNeRF PSNR, SSIM and LPIPS are copied from the pixelNeRF paper tab. 2. The paper does not report FID for NeRF here.

Unseen side of the Lego scene, 14 views of one half tab. 3:

Ablation on the Lego scene, 8 views, encoder choice tab. 4:

Fine-tune length, Realistic Synthetic, 8 views tab. 5:

Per-scene PSNR, SSIM and LPIPS are in tab. 7 (appendix). FID and KID are not reported per scene.

Paper vs code

The code is at ajayjain/DietNeRF, commit ae5d7b4b314d9e4ba66f1fd422d1265f187e4182. The owner is ajayjain, matching first author Ajay Jain, and the paper's project page links the repository. The repository has no LICENSE file, so its code cannot be quoted; the snippets above are paraphrases with citations.

• Sign of the semantic term. Eq. 5 is written as a positive dot product and the text says the term is minimised §4.3 eq. 5. The code minimises the negative cosine run_nerf.py run_nerf.py.
• Loss weight. The paper gives no numeric λ. The released 8-view configs use 0.1 for both networks 250_blender_chair_8views_ctr.txt, while the argument default is 0.2 run_nerf.py.
• Which token is embedded. The paper says "final activations" of the ViT §1 §4.1. The code takes token 0 (the class token) of the last transformer block's output run_nerf.py run_nerf.py. The CLIP wrapper returns that output before the final layer norm and the projection model.py.
• Coarse network. The paper's algorithm has one semantic term alg. 1. The code adds a second term for the coarse network with its own weight run_nerf.py.
• Pose sampler ranges. Angles and radius are in the configs, not in the paper 250_blender_chair_8views_ctr.txt.
• Batching. The released configs set no_batching = True. Each iteration then draws one random training image and 1024 rays from it run_nerf.py 250_blender_chair_8views_ctr.txt.
• Mixed precision. The paper says mixed precision is used §4.4. The flag defaults to off run_nerf.py and the DietNeRF configs do not set it. The released 8-view runs therefore appear to use full precision.
• Simplified NeRF, Hotdog. Table 6 gives max frequency 2^3 for Hotdog. The config sets multires = 5, which gives max frequency 2^4 under the code's definition 345_blender_hotdog_8views_simplest.txt run_nerf_helpers.py.
• DTU. The README says the DTU experiments are "coming soon" README.md. The pixelNeRF fine-tuning code and configs are not in the repository.
• Code fork. The repository vendors a fork of OpenAI's CLIP in CLIP/ with a wrapper clip_utils.py. The fork is MIT-licensed LICENSE.

The paper's other stated values match the code: Adam, 200k iterations, a consistency interval of 10 to 16 and a 15 to 20% ray budget run_nerf.py run_nerf.py. CLIP's normalisation constants are in clip_utils.py; the paper does not state them.

Pitfalls

• Sign. Implement the semantic term as minus the cosine similarity times λ, as the code does. Minimising the cosine as written in eq. 5 would push the embeddings apart.
• λ and the schedule are not in the paper. Use 0.1 and K = 10 to match the released runs. The ablation shows results only for the fine-tune length.
• Which layer and token. Use the class token of the last transformer block, taken before the final layer norm and projection, as the code does. The paper's words could also mean the projected embedding.
• Resolution. The semantic render is 168 x 168 and is resized bilinearly to 224 before the encoder run_nerf.py. Training targets are resized from 400 x 400 to 224 run_nerf.py. These resizes are not in the paper.
• Ray jitter. The strided grid starts at a random non-integer offset, so sampled rays fall between pixels run_nerf_helpers.py run_nerf_helpers.py.
• Pose ranges exist only in the code. The paper says only that the radius is bounded. Use the config ranges if you want to match the released results.
• Cost. Backpropagation through rendering is memory intensive §4.4. The semantic term adds a backward pass through the renderer, although only every K-th iteration pays for it. The paper uses gradient checkpointing and mixed precision to reduce memory §4.4, but the released configs do not enable mixed precision (see Paper vs code).
• Fine-tune length is inconsistent. The text says 20k to 70k §4.4, the ablation covers 10k to 200k tab. 5, and the released script uses 50k. Use 50k.
• Unseen-side claim. The text says DietNeRF gives "quantitatively more accurate colors" in the unseen region §5.3. The base model has lower PSNR and SSIM than Simplified NeRF in tab. 3; its LPIPS is slightly better (0.157 against 0.160). Only the fine-tuned variant beats Simplified NeRF on all three metrics.
• PSNR gain. The text says the full pipeline gives "+8.5 PSNR" over NeRF §5.1. The table difference is 23.591 - 14.934 = 8.657 tab. 1.
• Simplified NeRF is tuned per scene. The baseline's settings were chosen per scene by hand until the geometry converged, which the paper calls a fair comparison app. A tab. 6. Matching it needs the same per-scene search.
• Random view subsets. The 8 training views are sampled at random. The code seeds NumPy at 0, so a rerun samples the same views only if the data order is unchanged run_nerf.py run_nerf.py.
• FID and KID are not per-scene metrics. They need a large sample; the paper computes them over all scenes and excludes them from the 14-view experiment §5.3 app. A.
• Author-reported results. The DTU baselines come from the pixelNeRF paper tab. 2. Released code covers only the Realistic Synthetic experiments README.md.

Lineage

• Builds on NeRF (Mildenhall et al., 2020) for the representation and volume rendering §2.
• Uses CLIP (Radford et al., 2021) as the frozen encoder; ImageNet ViT models are the ablation alternative §4.1 tab. 4.
• Fine-tunes pixelNeRF (Yu et al., 2020) for single-view synthesis on DTU §5.2.
• The lazy-regularisation idea is credited to StyleGAN2 §4.4.
• Compared against NeRF, Neural Volumes, Simplified NeRF and pixelNeRF §5. IBRNet, which needs at least 50 views, and GRAF are discussed in related work only §7.