Neural Reflectance Fields for Appearance Acquisition

In plain words

Taking a photo of an object from many angles gives you a view of it, but not enough to show it under a different lamp. This paper trains a small neural network that stands in for the whole scene. For each point in space the network says how dense the material is, which way its surface faces, and how shiny and how bright it is. A renderer draws the scene by following rays through that description, so the result can be viewed from new positions.

The photos are taken with a phone camera and its own flash, so the light always sits next to the camera. That is easy to capture, but it hides how the scene looks under other lamps. The renderer adds a second light path toward the lamp, so the trained model can then be lit from any direction and still cast shadows. The authors also show that the result can be dropped into a standard physically based renderer next to ordinary 3D models.

TL;DR

• A Neural Reflectance Field is an MLP that maps a 3D point to density, normal and microfacet BRDF parameters, with no view or light input §4.1.
• It is trained per scene, from collocated flash photos, by differentiating through a reflectance-aware ray marcher §4.2 §5.
• A precomputed adaptive transmittance volume lets it render shadows under new point-light positions §4.3.
• Reported results are visual only: no quantitative metric is given, and the comparisons are against a mesh-based method and a voxel-based volume method §6.

Problem

Acquiring an object's appearance so it can be shown under new viewpoints and new lights normally means reconstructing geometry and reflectance. Current reconstruction is error-prone, so renders carry visible artifacts §1. Mesh pipelines that start from multi-view stereo often fail on low-texture, specular or thin structures §6. Discrete voxel volumes with per-voxel BRDFs can do relighting, but their resolution is fixed by memory, so fine textures and sharp boundaries are lost §1 §6.

Earlier neural ray-marching methods such as NeRF model view-dependent radiance for view synthesis only and do not represent lighting, so they cannot relight §1 §3.

Assumptions: the input is an unstructured set of images of a static scene, captured with the camera and a point light at the same position, with known camera poses from structure from motion §1 §5.

Method

The representation and the renderer are described together. The network is trained through the renderer, so every step below must be differentiable.

1. Network output. An MLP with 14 fully connected layers and ReLU maps the encoded 3D position to a (4+m)-dimensional vector: volume density σ (1-D), normal n (3-D) and m reflectance parameters §4.1. The paper's reflectance model is a microfacet BRDF with a Lambertian diffuse albedo (3-D) and a GGX specular term with a roughness (1-D) §5 §3.2. The output therefore has 8 values in total §5.

2. Positional encoding. Each coordinate is mapped with sine and cosine features at octave frequencies, using W = 10 levels §4.1 eq. 10.

3. No view or light input. The network takes only the position. Viewing, lighting and shading are computed by the renderer, not learned §4.1.

4. Reflectance-aware rendering equation. Radiance at the camera is the integral, along the camera ray, of view transmittance, light transmittance, density, the BRDF and the light intensity, under a single point light and one bounce of direct illumination §3.1 eq. 4 eq. 5 eq. 6.

5. Discrete ray marching. The integral is summed over N samples along the ray §3.2 eq. 7:

   \hat{L}(c,\omega_o) = \sum_{j=0}^{N} T_c(x_j)\,T_l(x_j)\, \bigl(1-e^{-\sigma(x_j)\Delta t_j}\bigr)\, f_r(x_j)\, L_l(x_j) eq. 7

   The view transmittance is the inclusive sum shown in eq. 8:

   T_c(x_j) = \exp\Bigl(-\sum_{k=0}^{j}\sigma(x_k)\,\Delta t_k\Bigr) eq. 8

   The light transmittance has the same form along a second ray toward the light eq. 9. This second ray is what makes general relighting expensive, so it is avoided during training (step 6) and approximated at inference (step 8).

6. Collocated training. For flash images, the light and the camera coincide, so T_l equals T_c and no extra light ray is marched §3.2 §4.2. Training therefore needs only one ray per pixel.

7. Loss and coarse-to-fine sampling for camera rays. Two networks (coarse and fine) are trained per scene §4.2. The coarse network is evaluated at N1 stratified samples, one random point per bin along the ray. Each sample gets a visibility weight eq. 11, the weights are normalised into a piecewise-constant distribution, and N2 more samples are drawn from it. The fine network renders the final value from all N1 + N2 samples §4.2. The loss is an L2 term on the coarse and fine colours, plus a regulariser on the view transmittance that pushes it towards 0 or 1 for clean backgrounds eq. 12 §5.

8. Adaptive transmittance volume for light. At inference, a virtual image plane sits in front of the point light. A ray is marched through each pixel with the same coarse-to-fine scheme, using the coarse network to choose samples and then the fine network to store light transmittance at those samples. The result is a volume that concentrates samples where the coarse network sees visible structure §4.3 fig. 3.

9. Shading at inference. At each camera-ray sample, the light transmittance is linearly interpolated from the nearest precomputed samples. Camera rays again use coarse-to-fine sampling §4.3.

10. Variants and export. For fur, the normal in the model is replaced by a fibre tangent and a classical fur model is used §5 fig. 6. For Monte Carlo rendering, the network is evaluated on a 512 × 512 × 512 grid to make a volume, which is then used in Mitsuba §6 fig. 9.

# Render one camera ray  [eq. 7] [eq. 8]
# Input:  points x_0..x_N along the ray, step sizes dt_j
# Output: RGB radiance

def render_ray(points, steps, light_radiance):
    # Density, normal and reflectance from the network  [§4.1]
    density, normal, reflectance = network(points)
    # Per-sample opacity from the density  [eq. 7]
    alpha = 1 - exp(-density * steps)
    # View transmittance as written: inclusive sum over k <= j  [eq. 8]
    view_trans = exp(-cumsum(density * steps))
    # Light transmittance: equal to view_trans when collocated  [§4.2] [eq. 9]
    light_trans = light_transmittance(points, view_trans)
    # Shading with the microfacet BRDF  [§3.1] [§5]
    shading = brdf(points, normal, reflectance)
    radiance = sum(view_trans * light_trans * alpha * shading * light_radiance)
    return radiance

# Coarse-to-fine sampling for one camera ray  [§4.2] [eq. 11]
# Input:  ray interval [t_near, t_far]; N1 = 64, N2 = 128
# Output: N1 + N2 sample positions, sorted by depth

def sample_ray(t_near, t_far, coarse_net, fine_net):
    # One random point per bin  [§4.2]
    coarse_t = stratified_samples(t_near, t_far, bins=N1)
    coarse_density = coarse_net(positions(coarse_t))
    # Visibility weight per sample, from the coarse network  [eq. 11]
    weights = view_trans(coarse_density) * (1 - exp(-coarse_density * dt))
    # Draw N2 samples from the normalised weights  [§4.2]
    fine_t = sample_from_weights(coarse_t, weights / sum(weights), N2)
    return sort(concatenate(coarse_t, fine_t))

# Training loss for one batch of rays  [eq. 12] [§5]
# Input:  batch of 50 x 50 collocated pixel rays, target colours
# Output: scalar loss

def training_loss(rays, target_rgb, beta=1e-4):
    # Coarse and fine colour per ray  [§4.2]
    coarse_rgb, fine_rgb, view_trans = render_both(rays)
    # L2 terms on both networks  [eq. 12]
    data = sum(norm2(coarse_rgb - target_rgb)) \
        + sum(norm2(fine_rgb - target_rgb))
    # Push view transmittance towards 0 or 1  [eq. 12] [§5]
    regulariser = sum(log(view_trans) + log(1 - view_trans))
    return data + beta * regulariser

Hyperparameters

Results

Author-reported. Setup: real scenes captured with a phone flash, one network per scene. Baselines were run by their own authors on the same inputs §6. The paper reports no quantitative metric and no table of numbers, so the table below is qualitative.

Limitations stated by the authors: slight blur where detail is very dense (fur, face), dark floaters from background regions seen by several views (usually fixed with a 3D bounding-box mask), and some flicker in relit video from inconsistent adaptive samples across frames §6.

Paper vs code

No official code. The source has no code link, and the arXiv abstract page lists none. Searches on GitHub for the title and method names found no repository that belongs to the authors. No project page appears in the source.

Pitfalls

• Output naming. §5 says the network outputs a 1-D "transmittance" and §4.1 says it outputs a volume density σ. Implement one scalar density, because the regulariser in eq. 12 is applied to view transmittance computed from that density eq. 8. The paper does not describe a separate transmittance head.

• Inclusive transmittance. eq. 8 sums up to and including sample j. The alpha of sample j eq. 7 is multiplied by that transmittance, so the sample's own density attenuates its own contribution. Standard alpha compositing uses the sum below j. The paper does not discuss the difference, and no code is available. Implement eq. 8 as written and note the choice.

• Positional-encoding range. The text calls W = 10 the highest frequency level, but the formula eq. 10 uses exponents 0 to W−1, so the top frequency is 2^9 π. Whether the raw coordinates are also concatenated to the encoding is not stated.

• Log regulariser at the boundaries. log T and log(1 − T) diverge at T = 0 and T = 1, and the paper does not give a clamp or epsilon. Use a small epsilon and check for NaN, because the term is minimised exactly where it becomes unbounded.

• Loss norm and colour space. The norm in eq. 12 is not stated to be per-channel RGB, and the paper does not say whether captured values were linearised before the L2 loss. Both change the loss scale and the learned highlights.

• Coarse-to-fine sampling details. The distribution in eq. 11 is piecewise constant, but the paper does not say how samples are placed inside each bin, how the bin edges are defined, or how step sizes Δt are computed from the fine samples.

• Light transmittance lookup. §4.3 says the nearest sampled points are interpolated linearly. It does not say whether this is 1-D along each light ray or 3-D, or how neighbours are found. The source also contains commented-out text describing a different lookup. That text is not part of the published paper and this card does not use it.

• Precomputation cost. The memory and time of the transmittance volume depend on the image-plane resolution and the sample counts, which are not stated. Expect a per-scene precomputation step before any relit render.

• Transmittance volume and video flicker. Adaptive samples may differ across frames. The authors recommend more samples, which costs memory and time §6.

• Shadow-free training. The network sees only collocated images with no cast shadows §4.3. Shadows under new light come from the learned density combined with the precomputed transmittance volume, not from training images that contain shadows.

• Volume-only export. The Mitsuba composition step discretises the field to 512³ voxels, which limits detail §6. The paper states this and says the limit can be removed by implementing the network in the renderer.

• BRDF citation. §3.2 cites Karis 2013 for the analytic BRDF. §4.1 and §5 cite Walter 2007 for the microfacet GGX model. The paper does not explain the difference.

• Scene-specific training. Each network is trained on one scene, and the paper reports about 2 days on 4 GPUs §5. There is no generalisation to other scenes.

• Figure and equation numbers. These were counted from the LaTeX source, because the source contains no compiled PDF. Check them against the published PDF.

Lineage

• Builds on: NeRF, arXiv:2003.08934, for ray marching and positional encoding, which the paper adapts to reflectance rather than radiance §1 §3. Classical volume-rendering and shading models: Max 1995 and Kniss et al. 2003 §3.1.
• Compared against: the mesh-based flash-photography method of Nam et al. 2018, and the voxel-based deep reflectance volumes of Bi et al. 2020 §6.
• Superseded by: not stated in the paper, and no later card in this wiki covers it.