pixelNeRF: Neural Radiance Fields from One or Few Images

In plain words

A normal radiance-field model learns one scene from many photos, and it has to be trained again from scratch for every new scene. When only one or two photos exist, it has almost nothing to work with. This paper trains one network on many scenes instead. The network learns what objects typically look like, so it can fill in the unseen side of an object from a single picture.

The trick is to attach the picture's local colour features to every point in space that projects onto them. The network then asks, for each point along a camera ray, "what does the picture say about this spot?" and combines that with the position and viewing direction. Because the points are measured in the input camera's frame, the same network works for a single object, several objects or a real scene. It needs only ordinary photos with known camera positions, not 3D shapes or masks, and a new view comes out in one forward pass with no extra fitting.

TL;DR

pixelNeRF predicts a neural radiance field from one or a few posed images in a single forward pass, after training across many scenes. It uses fully convolutional ResNet34 features, sampled per point by projection, and pools views inside the MLP, so more input views can be used at test time without retraining. The paper reports better few-view results than SRN and DVR on ShapeNet, and it runs on DTU with three views §5 (author-reported).

Problem

NeRF needs many posed images and a long optimisation per scene, and without image features it generalises poorly when few views exist §1. Earlier few-view methods mostly use a canonical object frame or a global latent code, which fails on scenes with several objects or no shared canonical frame, and which loses local detail §1 tab. 1.

The authors' setting is: known camera intrinsics and relative poses for the inputs, multi-view training images only, and no 3D supervision §1 §4.2. The problem is harder than single-scene NeRF because the network must learn scene priors, not just geometry consistency §3.

Method

The model has two parts: a fully convolutional image encoder E, which produces a pixel-aligned feature grid W = E(I), and a NeRF MLP f, which returns density and colour from a point, a view direction and the local feature §4. The query point and direction are expressed in the input view's camera frame, not in a canonical frame §4 §1.

1. Single-image case §4.1 eq. 3. The input image is encoded once. For each query point x on a target ray, the point is projected to the input image with known intrinsics, and the feature is bilinearly interpolated at that pixel §4.1. The MLP receives the positional encoding of x (6 frequencies), the view direction d (not encoded), and the feature. The feature enters the MLP as a residual, as described under Hyperparameters §5 app. B.1.

2. Encoder §5 app. B.1. A ResNet34 pretrained on ImageNet supplies a feature pyramid taken before the first four pooling stages. Each level is bilinearly upsampled and concatenated into a 512-channel feature per pixel. For 64×64 inputs the first pooling layer is skipped so the features keep more resolution app. B.1 sn64.conf.

3. Feature conditioning in the MLP §5. The MLP is a fully connected ResNet with 5 blocks and width 512, following DVR app. B.1. The image feature is added through one learned linear layer per block, in the style of AdaIN and SPADE §5. The view direction goes in at the start of the network §4.1.

4. Multiple views §4.2 eq. 4 eq. 5 eq. 6. Each input view i gets its own transform. The point and direction are mapped into that camera frame with the world-to-camera pose P(i) eq. 4, and features are looked up in that view's grid §4.2. The first layers f1 run separately per view, giving V(i) eq. 5. The views are averaged with a permutation-invariant pool ψ, then the final layers f2 predict density and colour from the average eq. 6. The authors place the pooling after the third ResNet block app. B.1. With one view, f = f2 ∘ f1 §4.2.

5. Volume rendering and loss §3 eq. 1 eq. 2. A pixel is rendered by the NeRF volume integral eq. 1, and training minimises the squared RGB error between the rendered and target pixel over all rays eq. 2. The code uses the standard discrete alpha-compositing form and renders both a coarse and a fine pass nerf.py.

6. Sampling app. B.1. Per ray, 64 stratified uniform samples are taken for the coarse pass. The fine pass takes 16 importance samples from the coarse weights and 16 more drawn from a normal distribution (SD 0.01) around the coarse expected depth app. B.1 default.conf.

# Single-image pixelNeRF: colour and density at one query point  [§4.1] [eq. 3]
# Input:  feature_grid (512, H/2, W/2); query x (3,); direction d (3,)
# Output: rgb (3,), sigma (scalar)

def query_point(feature_grid, x, d):
    # Project x onto the input image with known intrinsics  [§4.1]
    # The code negates the camera z axis in this projection
    # [code: src/model/models.py#L130-L131]
    # [code: src/model/models.py#L206-L210]
    pixel = project_to_image(intrinsics, x)                  # (2,)
    # Bilinear interpolation of the pixel-aligned feature  [§4.1]
    feature = bilinear(feature_grid, pixel)                  # (512,)
    # Positional encoding of x, 6 frequencies; direction left unencoded
    # [app. B.1] [code: conf/default.conf#L20-L28]
    position_code = positional_encoding(x, num_freqs=6, omega=1.5)
    # Input to the first linear layer: encoding and view direction
    hidden = linear_in(concatenate(position_code, d))
    # Five residual blocks of width 512  [§5] [app. B.1]
    for block_index in range(5):
        # Code adds the feature to blocks 0 to 2 only; the paper text says
        # each block  [code: src/model/resnetfc.py#L111] [#L174-L175]
        if block_index < 3:
            hidden = hidden + linear_feature[block_index](feature)
        hidden = resnet_block[block_index](hidden)
    rgb, sigma = linear_out(relu(hidden))
    return rgb, sigma

# Multi-view pixelNeRF: pool per-view features before the last blocks
# [§4.2] [eq. 4] [eq. 5] [eq. 6]
# Input:  images (N, 3, H, W); poses P(i) world to camera; query x, d
# Output: rgb, sigma

def query_views(feature_grids, poses, x, d):
    # Map the point and direction into each input camera frame  [eq. 4]
    per_view = []
    for view in range(N):
        x_view = pose_rotation[view] @ x + pose_translation[view]  # (3,)
        d_view = pose_rotation[view] @ d                           # (3,)
        feature = bilinear(feature_grids[view], project(x_view))   # (512,)
        # Early layers run separately for each view  [eq. 5]
        per_view.append(first_layers(x_view, d_view, feature))     # (512,)
    # Average over views, then the remaining layers  [eq. 6] [app. B.1]
    pooled = mean(per_view)                                        # (512,)
    return last_layers(pooled)

# Per-ray compositing and loss  [§3] [eq. 1] [eq. 2]
# Input:  sigmas (K,), colours (K, 3), step lengths (K,), target (3,)
# Output: rgb (3,), loss (scalar)

def render_and_loss(sigmas, colours, deltas, target):
    # Opacity of each segment from density and its length  [eq. 1]
    alphas = 1 - exp(-relu(sigmas) * deltas)                   # (K,)
    # Light reaching each sample: product of earlier (1 - alpha)
    transmittance = exclusive_cumprod(1 - alphas)              # (K,)
    weights = alphas * transmittance                           # (K,)
    rgb = sum(weights[:, None] * colours, axis=0)              # (3,)
    # Squared error against the target pixel  [eq. 2]
    # The code averages over rays and adds the coarse pass to the fine pass
    # [code: src/model/loss.py#L100-L102] [code: conf/default.conf#L77-L78]
    return rgb, mean((rgb - target) ** 2)

Hyperparameters

Values are from the paper unless the source column says code. Where the paper gives none, the table says "not stated".

Results

Author-reported. All numbers are from the paper's tables. Metrics are PSNR and SSIM, with LPIPS where stated.

Category-specific, SRN protocol, 128×128 tab. 2 §5.1.1. Each cell is PSNR / SSIM. * marks methods that do not need canonical poses at test time.

Baseline numbers for TCO, GRF, dGQN and ENR are copied from their papers. SRN for cars was re-run on an updated dataset app. B.2.1.

Category-agnostic, 13 ShapeNet categories, 64×64 tab. 4 §5.1.2. Means over the 13 categories, one input view, 23 target views. Per-category values are in tab. 4.

Unseen categories and multiple objects tab. 5 §5.2.

Real car photos fig. 8 are shown qualitatively only; they are masked with PointRend and no metric is reported §5.2.

Ablations, abbreviated 25-view protocol on chairs tab. 3 §5.1.1. These numbers do not match tab. 2: they use the shorter protocol.

DTU real scenes fig. 9 fig. 10 §5.3. One model trained on 88 scenes renders held-out scenes from three views without test-time optimisation [fig. 9]. Per-scene NeRF baselines are optimised separately for 1, 3, 6 and 9 views, and the PSNR quantiles are plotted in fig. 10. Each NeRF takes about 14 hours, and the paper gives 60 of them §5.3 app. B.2.6. The DTU numbers exist only in fig. 10; no numeric table is given.

Paper vs code

Official code: sxyu/pixel-nerf, owned by Alex Yu, an author. The paper's abstract points to the project page alexyu.net/pixelnerf, which links the repository as its Code link [abstract]. The repository is pinned at commit 91a044bd and licensed BSD-2-Clause LICENSE. The project page also says that parts of the implementation come from kwea123/nerf_pl, a third-party repository; we did not use that repository and quote no code from either. Paths below are relative to the repository root at code_commit.

• Feature residual. Paper: "a residual at each layer" and "at the beginning of each ResNet block" §4.1 §5. Code: n_lin_z = min(combine_layer, n_blocks) with combine_layer = 3, so the feature is added to blocks 0, 1 and 2 only, and blocks 3 and 4 get no feature resnetfc.py resnetfc.py default_mv.conf. The single-view and multi-view configs both inherit this.
• Multi-object frequency. Paper: ω = 2.0 for two-object scenes app. B.1. Code: conf/exp/multi_obj.conf does not override freq_factor, so it inherits 1.5 from conf/default.conf default.conf multi_obj.conf. We could not check which value produced the paper's numbers.
• Loss. Paper: squared error summed over rays, with no coarse term eq. 2. Code: MSE averaged over rays, with the coarse and fine losses each weighted 1.0 loss.py default.conf. The coarse term is an implementation choice that the paper does not describe.
• Fine samples. Paper: 16 importance plus 16 depth samples app. B.1. Code: n_fine = 32 with n_fine_depth = 16, giving 16 importance samples default.conf. Consistent.
• Positional frequency. Paper: sin(2^k ω x) with k from 0 to 5 app. B.1. Code: frequencies ω · 2^k with the input kept code.py code.py. Consistent.
• Background. Paper: white for ShapeNet, black for DTU app. B.1. Code: white_bkgd = True by default and False in the DTU config default.conf dtu.conf. Consistent.
• Learning rate. Paper: 1e-4, schedule not stated app. B.2. Code: 1e-4 with exponential decay factor 1.0, i.e. none args.py trainer.py. Consistent.
• Config defaults. conf/default.conf sets 3 MLP blocks, but every experiment config includes conf/default_mv.conf, which sets 5 default.conf default_mv.conf. Use the default_mv values, which match the paper.
• Projection sign. The code negates the camera z axis and flips the y focal length models.py models.py. The paper says only "known intrinsics" §4.1. Implementers need the sign convention from the code.

No discrepancy was found in the encoder width, the 512-channel feature, the image-space projection, or the view-direction handling. The code search (step 3) found one official repository: the project page's link, owned by an author.

Pitfalls

• Feature residual. The paper text says the feature enters at every block, but the released code adds it at three of five blocks. Follow the code if you want to reproduce the reported numbers, and say which you chose.
• Tables are not comparable across protocols. The ablation's full model (23.43 PSNR, 1-view) differs from the main Table 2 entry (23.72) because the ablation uses 25 target views instead of 251 tab. 3 §5.1.1. Do not mix the two.
• Baseline comparability. The category-specific comparison follows SRN's protocol. Its cars baseline was re-run on a corrected dataset, and the authors say the SRN paper's original cars numbers used buggy renderings app. B.2.1. TCO, GRF, dGQN and ENR numbers come from their own papers and cannot be re-checked here §5.1.1.
• Input-view selection. Results depend on the views chosen as input. The authors fix "informative" views shared across methods: view 64 (or 64 and 128) for SRN chairs and cars, views 25, 22, 28, 40, 44, 48, 0, 8, 13 (prefix) for DTU, and a random view per object for category-agnostic app. B.2 §5.1.2.
• Ray bounds are manual. The authors tune near and far bounds by hand per dataset and call scale invariance an open problem §6 app. B.1. The DTU bounds are not stated, and the "input" entry in the bounds list is ambiguous. Set bounds from the dataset you use, not from this card.
• DTU split and baselines. The standard MVSNet split overlaps between train and test, so the authors made their own split app. B.2.6. DTU numbers therefore cannot be compared with other papers on the standard split. Five NeRF baselines did not converge with 6 or 9 views and were initialised from 3-view weights instead app. B.2.6.
• Cost. Rendering is slow, as in NeRF, and runtime grows linearly with the number of input views §6. Training costs 5 to 6 days on one GPU per model app. B.2.
• Scope limits. Backgrounds are not synthesised; the sim-to-real demonstration masks them out with PointRend §5.2. The paper handles only known intrinsics and poses and only "wide baseline" real data from a controlled capture §6.
• Feed-forward is not free. Test-time views are fixed and informative, so the reported quality depends on how those inputs were chosen app. B.2.

Lineage

• Builds on: NeRF, for volume rendering and the positional encoding §3 app. B.1. The MLP design follows DVR app. B.1. The feature conditioning follows AdaIN and SPADE §5.
• Compared against: SRN and DVR on ShapeNet §5; SoftRas (grayscale) in the category-agnostic setting §5.1.2; NeRF on DTU §5.3; TCO, dGQN, GRF and ENR on the category-specific benchmark tab. 2.
• Superseded by: not stated in this wiki.