wikillm

Paper · 2021-11-30 · cs.CV, cs.GR

NeRFReN: Neural Radiance Fields with Reflections

Models a scene as transmitted and reflected NeRFs, combined as C = C_t + beta * C_r with a learned reflection fraction beta. Depth-smoothness and bidirectional-depth priors plus warm-up training steer the split. Author-reported: PSNR above NeRF on 3 of 6 reflective scenes (art1, bookcase, mirror), slightly below on the rest; mirror-like scenes need user masks.

Neural radiance field (NeRF) and variantsNovel view synthesisview-synthesisreflectionsneural-radiance-fielddepth-estimation

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A neural radiance field (NeRF) is a learned 3D model that can draw a scene from new camera angles. It works well for ordinary shine, but glass and mirrors break it. A mirror shows a second world. NeRF tries to explain that second world as real objects standing behind the glass. The depth then comes out wrong, and the picture turns blurry when you move to the side and the reflection should vanish.

NeRFReN splits the scene into two parts: what you see through the glass, and what the glass reflects. Each part gets its own small neural 3D model. The final picture is the see-through part plus the reflected part, scaled by how reflective each spot is. A few extra rules keep the split sensible. The see-through surface is pushed to be smooth. The reflected part is pushed to look like a thin shell that a viewer sees from only a few angles. Training starts gently so the two parts do not swallow each other. The hardest mirrors also need a few user-labelled masks that mark where the reflection is.

TL;DR

  • The scene is modelled as a transmitted NeRF and a reflected NeRF, combined additively with a learned reflection fraction per point §3.2 eq. 7.
  • Three aids make the split identifiable: a depth-smoothness prior on the transmitted depth, a bidirectional depth consistency prior on the reflected depth, and a warm-up schedule on their weights §3.3 §3.4.
  • Author-reported: PSNR is higher than NeRF on art1, bookcase and mirror and slightly lower on art2, art3 and tv; the depth benefit is shown only qualitatively tab. 1 §4.2.
  • Mirror-like scenes need user masks (4 for mirror, 1 for tv) to decompose correctly §4.1 app. A.4.2.
  • Official PyTorch code exists, but the repository has no LICENSE file. This card cites code by path and line only and quotes none.

This card condenses the paper. It is not endorsed by the authors.

Problem

NeRF takes the view direction as input, which lets it reproduce only low-frequency view effects such as highlights §3.1. Under strong reflections NeRF instead models the reflected content as translucent geometry at a virtual depth. Expected depth is then biased between the real and virtual surfaces, and renderings blur when the reflected objects are visible only from some viewpoints §1 §3.1 fig. 1.

Splitting a scene into transmitted and reflected parts without supervision is under-constrained. Degenerate solutions include one part explaining everything while the other stays empty, both parts explaining everything, and partial mixtures §1. The method assumes (1) the reflection fraction depends only on the transmitted part, (2) the transmitted depth is locally smooth because reflectors are mostly planar, and (3) the reflected part needs only simple geometry because it is seen from limited angles §1. These hold for planar or nearly planar reflectors §1.

Method

The method has four parts: a recap of NeRF §3.1, the two-field formulation with its network §3.2 fig. 2, two geometric priors §3.3 fig. 3, and a warm-up training schedule §3.4. An optional interactive setting with user masks is described in §3.5.

Base NeRF recap §3.1

NeRF maps a position x and a view direction d to a density sigma and a colour c. Along a ray, with depth samples t_i and spacing delta_i = t_{i+1} - t_i, the colour is the weighted sum of sample colours. The weights come from opacity and accumulated transmittance eq. 2 eq. 3:

\hat C(r)=\sum_i \omega_i\,c_i,\qquad \omega_i=T_i\bigl(1-e^{-\sigma_i\delta_i}\bigr),\qquad T_i=\exp\Bigl(-\sum_{j<i}\sigma_j\delta_j\Bigr) eq. 2 eq. 3

NeRF is trained with a photometric loss eq. 4. The expected termination depth is the weight-averaged sample depth eq. 5.

\mathcal L_{pm}=\lVert \hat C-C\rVert_2 \qquad\qquad t^*(r)=\sum_i \omega_i\,t_i eq. 4 eq. 5

Two fields and additive composition §3.2

NeRFReN keeps two fields. The transmitted field has density sigma^t and colour c^t, and it is conditioned on the view direction. The reflected field has density sigma^r and colour c^r, and it is not conditioned on the view direction §3.2 app. A.1.1. A reflection fraction alpha is predicted per point from the transmitted branch §3.2 fig. 2.

Each field is volume-rendered with eq. 2 to give a transmitted colour and a reflected colour. The per-pixel reflection fraction beta is the transmitted weights accumulated against alpha eq. 6. The final pixel is the transmitted colour plus beta times the reflected colour eq. 7.

\beta(r)=\sum_i T_i(\sigma^t)\bigl(1-e^{-\sigma^t_i\delta_i}\bigr)\,\alpha_i eq. 6

\hat C=\hat C(r;\sigma^t,c^t)+\beta(r)\,\hat C(r;\sigma^r,c^r) eq. 7

Pseudocode
# Render one ray through both fields  [§3.2] [eq. 2] [eq. 6] [eq. 7]
# Input:  ray r with depth samples t_1..t_N; transmitted and reflected fields
# Output: pixel colour C_hat, transmitted depth t_star

def render_ray(ray, transmitted_field, reflected_field):
    # Query the transmitted field with the view direction  [§3.2]
    sigma_t, colour_t, alpha = transmitted_field(points, direction)
    # Query the reflected field without the view direction  [§3.2]
    sigma_r, colour_r = reflected_field(points)
    # Point weights of the transmitted ray  [eq. 3]
    weights_t = point_weights(sigma_t, deltas)  # (N,)
    # Point weights of the reflected ray  [eq. 3]
    weights_r = point_weights(sigma_r, deltas)  # (N,)
    # Reflection fraction: transmitted weights times alpha  [eq. 6]
    beta = sum(weights_t * alpha)
    # Colour of each field  [eq. 2]
    C_t = sum(weights_t * colour_t)
    C_r = sum(weights_r * colour_r)
    # Additive composition  [eq. 7]
    C_hat = C_t + beta * C_r
    # Depth is taken from the transmitted field  [§4.2] [eq. 5]
    t_star = sum(weights_t * t_values)
    return C_hat, t_star

The network has a shared positional-encoded trunk, then a separate transmitted branch (density, alpha, direction-conditioned colour) and a separate reflected branch (density and colour, no direction) §3.2 fig. 2 app. A.1.1 fig. A1. Layer counts and widths are given only in the figure in the paper; the code's values are listed under Hyperparameters.

Geometric priors §3.3

Depth smoothness on the transmitted depth eq. 8. Pixels in 8-connected neighbourhoods are penalised for depth differences, weighted down where colour changes strongly, so that depth edges are preserved. It is applied to small patches sampled during training §3.3 app. A.1.3.

\mathcal L_d=\sum_p\sum_{q\in\mathcal N(p)}\omega(p,q)\,\lVert t^*(p)-t^*(q)\rVert_1,\qquad \omega(p,q)=\exp\bigl(-\gamma\lVert C(p)-C(q)\rVert_1\bigr) eq. 8

Bidirectional depth consistency (BDC) eq. 9 eq. 10. A backward depth is computed by accumulating from the far end of the ray eq. 9. The BDC loss is the L1 difference between the forward and backward reflected depths eq. 10. It pushes the reflected density along each ray to be unimodal with small variance, a thin shell-like geometry. Zero BDC can also come from an empty ray, which the authors argue cannot happen because the transmitted part alone cannot reproduce the input images under the smoothness prior §3.3 fig. 3.

\overleftarrow{t}^*=\sum_i \overleftarrow{\omega}_i\,t_i,\qquad \overleftarrow{T}_i=\exp\Bigl(-\sum_{j>i}\sigma_j\delta_j\Bigr) eq. 9

\mathcal L_{bdc}=\sum_r\lVert \overrightarrow{t}^*(r)-\overleftarrow{t}^*(r)\rVert_1 eq. 10

Warm-up training §3.4

The total loss is the photometric loss plus the two priors eq. 11. The weights are small at the start and then raised and lowered over training. Early in training the viewing direction is also masked, so that the reflected part cannot absorb the transmitted highlights §3.4 app. A.1.2 fig. A2.

\mathcal L=\mathcal L_{pm}+\lambda_d\mathcal L_d+\lambda_{bdc}\mathcal L_{bdc} eq. 11

Pseudocode
# One training step: photometric loss plus the scheduled priors  [eq. 11]
# Input:  batch of 4x4 patches (rays); global step and epoch counters
# Output: scalar loss to minimise

def training_loss(patches, targets, step, epoch):
    # Render every ray of the batch through both fields  [§3.2]
    C_hat, t_trans, t_refl_fwd, t_refl_bwd = render_patches(patches)
    # Photometric term  [eq. 4]
    loss = photometric(C_hat, targets)
    # Depth smoothness on the transmitted depth  [eq. 8] [app. A.1.3]
    loss += lambda_d(step) * smoothness(t_trans, targets)
    # Bidirectional depth consistency on the reflected depth  [eq. 10]
    loss += lambda_bdc(epoch) * l1(t_refl_fwd, t_refl_bwd)
    # Early anchor of the transmitted colour to the input  [eq. A.1]
    if step < 1000:
        loss += 0.01 * anchor(transmitted_colour(patches), targets)
    return loss

Interactive setting §3.5

For textureless reflectors such as mirrors the unsupervised split often fails. The user then supplies binary masks for a few training images, with 1 for reflective pixels and 0 for non-reflective pixels. An L1 loss ties the predicted beta to the mask §3.5 eq. 12. The paper does not state this loss weight.

\mathcal L_{\beta}=\sum_p\lVert \hat\beta(p)-\beta(p)\rVert_1 eq. 12

Hyperparameters

Values the paper states are cited to the paper. Values the paper does not state are marked, and the code's value is given with its citation. Code citations are to the official repository at the commit in the frontmatter.

NameValueWhereNotes
optimiserAdamapp. A.1.3beta1 0.9 and beta2 0.999 not stated in paper; code uses beta1 0.9 and beta2 0.999 nerfren_model.py train_options.py
initial learning rate5e-4app. A.1.3
learning-rate scheduleexponential decay to 5e-6 "in the last 20 epochs"app. A.1.3code: exp policy, 40 epochs, 20 decay epochs, final 5e-6 train_options.py; see Pitfalls
epochs40app. A.1.3
patch size4 x 4app. A.1.3patches are needed for the smoothness prior
rays per stepnot statedapp. A.1.3code: batch of 128 patches, i.e. 2048 rays train_art1.sh
image resolutionnot stated—code: 648 x 432 train_art1.sh
samples per ray64 coarse and 64 fine for NeRFReN; 128 fine for NeRFapp. A.1.3coarse count for NeRF not stated; code uses 64 and 64 nerfren_model.py
lambda_d (depth smoothness)0.01, then 0.1 from step 1k, then 0.01 from step 5kapp. A.1.3code: 0.01 start for art2, art3, bookcase; 0.1 start for art1, tv, mirror train_art2.sh train_art1.sh
gamma in omeganot stated§3.3code divides colour difference by bilateral_gamma = 0.1 criterions.py nerfren_model.py
lambda_bdc1e-4, then 0.05, then 1e-4 at epoch 12, then 0 at epoch 15app. A.1.3paper says 0.05 at epoch 20; code uses epoch 10 train_art1.sh
lambda_init (transmitted anchor)0.01 for the first 1k stepsapp. A.1.3code: trans_lowpass 0.01 until step 1000 train_art1.sh train_art1.sh
lambda_beta (smoothness of beta)1e-4app. A.1.3
lambda_beta_mask (interactive)not stated§3.5code: 0.1, set to 0 from epoch 10 in art1, tv, mirror train_art1.sh train_art1.sh
viewing-direction maskingmasked before epoch 10app. A.1.3code scripts: ramp from epoch 9 to 10 train_art1.sh; code defaults 10 to 20 nerfren_model.py
reflected depth smoothnessnot in paper—code: 1e-10 at start, 0.01 from epoch 15 train_art1.sh
coarse-fine alignmentnot in paper—code: 0.1 until step 5000, art1 and mirror only train_art1.sh
masks per scenetv 1, mirror 4§4.1code also gives art1 one mask train_art1.sh
transmitted branchone extra layer, direction-conditioned colour, outputs density, alpha and colour§3.2 fig. 2code: trunk depth 5, width 256, skip at trunk layer index 4, D_sep 1 networks.py
reflected branchone extra layer, colour without direction§3.2 fig. A1code: no direction input for reflected colour by default networks.py
position and direction encodingnot stated in textfig. A1code: degree 10 for position, 4 for direction nerfren_model.py
density activationnot stated—code: ReLU before rendering nerfren_model.py rendering.py
colour and beta activationnot stated—code: sigmoid for both networks.py networks.py
photometric lossL2 normeq. 4code uses mean squared error; see Paper vs code

Results

Author-reported. Setup: six self-captured forward-facing scenes with glass and mirror reflections, the RFFR dataset introduced by the paper. Images are split into train and test sets; PSNR, SSIM and LPIPS are computed on the test images. Baselines are NeRF and NeRF-D, a NeRF variant with the depth-smoothness term §4 tab. 1. Hardware and training time are not stated. The paper also reports NeRF and NeRFReN on eight LLFF scenes, with NeRFReN slightly below NeRF on average app. A.3 tab. A1.

Table 1 values, copied from the paper:

ScenePSNR NeRFPSNR NeRF-DPSNR NeRFReNSSIM NeRFSSIM NeRF-DSSIM NeRFReNLPIPS NeRFLPIPS NeRF-DLPIPS NeRFReN
art136.4836.4939.490.9750.9710.9780.0290.0270.029
art242.6341.5842.330.9740.9680.9720.0590.0700.067
art341.1839.8041.030.9730.9650.9700.0530.1170.072
bookcase29.5029.7830.180.8830.8760.8900.0780.1360.100
tv33.4733.0233.240.9600.9540.9550.0540.0630.061
mirror32.2232.3333.300.9440.9390.9470.0660.0720.071

The paper's bold and underline marks in Table 1 are not reproduced here. From the numbers, NeRFReN has the highest PSNR on art1, bookcase and mirror. NeRF has the highest PSNR on art2, art3 and tv. NeRF-D has the lowest LPIPS on art1, and NeRF has the lowest LPIPS on the other five scenes tab. 1.

Table A1, LLFF PSNR (author-reported) tab. A1:

Methodfernflowerfortresshornsleavesorchidsroomtrexaverage
NeRF26.0131.7033.2332.9222.9821.3837.0432.0329.66
NeRFReN25.3629.9533.6031.5922.7521.8937.5230.6429.16

The depth results are qualitative only: the paper shows depth maps against NeRF and NerfingMVS and calls the NeRFReN depth physically correct §4.2 fig. 5 fig. 6. The interactive results on tv and mirror are shown with 1 and 4 masks §4.1 fig. 4 app. A.4.2 fig. A7. Reflection removal and reflection substitution are shown qualitatively §4.4 fig. 9.

Paper vs code

Code: https://github.com/bennyguo/nerfren at commit 382f07fb033d5d9c61500772518bd98e71197443. The repository has no LICENSE file, and the README gives no licence statement. The code_license field therefore reads "none". Code is cited here by path and line only, and no snippet is quoted.

Differences between the paper and the code, each with both citations:

  1. Photometric loss. The paper uses an L2 norm eq. 4. The code uses mean squared error nerfren_model.py criterions.py.
  2. Depth-smoothness weight. The paper uses exp(-gamma times the colour difference) eq. 8, with gamma not stated. The code divides the colour difference by bilateral_gamma, default 0.1 criterions.py nerfren_model.py.
  3. Reflected depth smoothness. The paper applies depth smoothness to the transmitted depth only §3.3. The code also has a reflected-depth smoothness term, 1e-10 at the start and 0.01 from epoch 15 in all six scripts train_art1.sh nerfren_model.py.
  4. BDC weight schedule. The paper says lambda_bdc rises to 0.05 at epoch 20 app. A.1.3. The code rises at epoch 10 train_art1.sh. The paper's own sequence (decrease at 12, then 15, then a rise at 20) is internally inconsistent.
  5. Initial lambda_d. The paper gives 0.01 app. A.1.3. The code gives 0.1 for art1, tv and mirror train_art1.sh and 0.01 for art2, art3 and bookcase train_art2.sh.
  6. Masks per scene. The paper uses masks only for tv (1) and mirror (4), and says the other scenes are inferred from images alone §4.1. The code also gives art1 one mask with weight 0.1 train_art1.sh.
  7. Mask loss is switched off. The paper describes the mask loss as supervision with no schedule §3.5. The code sets its weight to 0 from epoch 10 in art1, tv and mirror train_art1.sh.
  8. Transmitted anchor. The paper compares each transmitted ray with the input colour, as a per-ray L2 term over the first 1k iterations app. A.1.3. The code compares patch-averaged transmitted colour with patch-averaged target colour, using MSE nerfren_model.py.
  9. Coarse-fine alignment. The code has a term that aligns the fine outputs with the detached coarse outputs, weight 0.1 until step 5000, in art1 and mirror. The paper does not mention it nerfren_model.py train_art1.sh.
  10. BDC normalisation. The paper sums the L1 difference over rays eq. 10. The code uses the mean over all elements nerfren_model.py criterions.py.
  11. Beta smoothness. The paper's smoothness is summed over 8-connected neighbours app. A.1.3. The code averages four directional differences criterions.py.
  12. Activations. The paper states none. The code applies ReLU to density before rendering rendering.py nerfren_model.py, a sigmoid to colour networks.py, and a sigmoid to beta networks.py.
  13. Output clamp. The code clamps the composed colour to [0, 1] rendering.py. The paper does not mention a clamp.
  14. Direction masking. The paper masks the viewing direction before epoch 10 app. A.1.3. The code unmasks frequency bands linearly between the start and end epochs. Defaults are 10 to 20 nerfren_model.py nerfren_model.py. The scripts use 9 to 10, so every band is open at epoch 10 train_art1.sh.
  15. Near and far. The paper does not state them. The code rescales depths by 0.75 times the nearest bound, converts rays to normalised device coordinates, and uses near 0 and far 1 rffr_dataset.py rffr_dataset.py.
  16. NeRF-D. The paper describes NeRF-D as NeRF with the depth-smoothness term §4. The repository has no NeRF-D configuration. The depth-smoothness options in the NeRF model default to 0 nerf_model.py. The NeRF-D setting used for Table 1 is therefore not reproducible from the repository.

Pitfalls

  • Schedules mix units. The paper calls some schedule points iterations and others epochs without defining the start. In the code, lambda_d, the transmitted anchor and the coarse-fine term use the global iteration count nerf_model.py. The BDC weight, the mask weight and direction masking use the epoch counter, which starts at 1 train.py nerf_model.py.
  • Learning-rate end point. With exponential decay, 40 epochs, 20 decay epochs and a final rate of 5e-6, the code keeps 5e-4 for epochs 1 to 20 and decays from epoch 21. The last epoch then runs at about 6.2e-6, not 5e-6, if the scheduler steps once per epoch. This follows from the scheduler formula, not from a stated value train_options.py networks.py.
  • Strong priors from the start collapse the split. With strong geometric weights and no warm-up, the reflected field converges to empty, and with view direction unmasked early the reflection is explained by view dependence and blurs app. A.4.1 fig. A6. The warm-up schedule is required, and its exact numbers are in the table above.
  • Masks are not optional for mirrors. Without user masks the network cannot separate transmitted from reflected geometry for mirror or tv. Four masks give fair results for mirror and one suffices for tv app. A.4.2 fig. A7 §4.1.
  • Glass is not located exactly. For art2 and art3 the opaque surface behind the glass is taken as the reflector. The authors say this is not strictly correct but does not hurt view synthesis, because composition happens in image space §4.1 fig. 4.
  • Depth is not physically checked. Depth is taken from the transmitted field §4.2. The paper has no quantitative depth metric, and its depth claims rest on figures fig. 5 fig. 6.
  • Stated limits. The method does not handle curved reflectors, several non-coplanar reflectors, view-dependent reflected content, or Fresnel-driven view dependence of beta. The authors note the assumptions may fail when reflected objects are themselves reflective or under strong Fresnel effect §5 app. A.2 fig. A3 §3.2 footnote.
  • Compute is not matched by sample count alone. The fine network is evaluated twice per sample, once for each field. The comparison uses 64 fine samples for NeRFReN and 128 for NeRF, which the authors present as comparable cost app. A.1.3.
  • Sampling has no stated count. The paper does not give the number of rays per step. The code draws 128 random 4 x 4 patches per step, with replacement over all training patches, and treats an epoch as the dataset length divided by 16 rffr_dataset.py rffr_dataset.py [code: data/init.py#L68-L73].
  • Notation collisions. L_beta names both the interactive mask loss eq. 12 and the smoothness of beta eq. A.2. The transmitted-anchor equation writes the input image with the hat symbol used for rendered colour eq. A.1. Read the symbols by their definitions.
  • Mask format. The code reads masks from the alpha channel of PNG files under refl_masks/, resized with nearest-neighbour rffr_dataset.py. The paper says only "binary masks".

Lineage

  • Builds on: NeRF (Mildenhall et al., cited in the paper as mildenhall2020nerf) for the base model §3.1. The image formulation follows the layer decomposition of Sinha et al. (2012), but replaces their piecewise-planar geometry with NeRF §2.
  • Compared against: NeRF, and NeRF-D, a NeRF variant with the depth-smoothness term §4 tab. 1. Also NerfingMVS (Wei et al., 2021) on depth §4.2 fig. 6.
  • Related: NeRF-based inverse rendering methods (NeRD and NeRFactor), which assume opaque surfaces and are not suited to complex reflections §2. Concurrent work that concentrates densities at a known depth (Urban Radiance Fields, Dense Depth Priors) §3.3.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “NeRFReN: Neural Radiance Fields with Reflections” by Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, Song-Hai Zhang (arXiv:2111.15234), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/bennyguo/nerfren at commit 382f07fb033d (none (no LICENSE file in repository)). Any quoted code is shown under that license.