Depth-supervised NeRF: Fewer Views and Faster Training for Free

In plain words

NeRF builds a 3D scene from photos by learning how much stuff and what colour sits at every point in space. With only a handful of photos it often gets the shape wrong: the training photos look right, but the geometry between the cameras is nonsense, so new viewpoints come out blurry or broken. Training is also slow, often hours to days per scene.

The software that works out where each camera was also produces a sparse cloud of 3D points as a by-product. Standard NeRF training ignores those points. This method uses them as free hints. For each point seen in a photo, it tells NeRF roughly how far along that camera ray the surface should be, and it trusts the hint less when the point was poorly located. The extra signal makes the learned shape more plausible from fewer photos, and the authors report that it also reaches the same quality in fewer training steps. It is an add-on, so it can be attached to other NeRF-style methods.

TL;DR

DS-NeRF adds a depth term to NeRF training, using the sparse 3D points that COLMAP already produces when it estimates camera poses. Each keypoint's projected depth becomes a Gaussian target for where a ray should stop, with the width set by reprojection error. The authors report better novel views from 2 to 10 training views on NeRF Real and Redwood, and 2-3x fewer iterations to reach NeRF's peak PSNR. The released default loss is a plain depth MSE, not the paper's distribution loss, and the released code gives keypoint rays no RGB target; where paper and code differ, implement the code and document the choice.

Problem

NeRF is ill-posed with few views: it can render the training views well while learning wrong geometry, which then fails on test views §1 fig. 2. Even with many views, training takes hours to days per scene at moderate resolution on one GPU §1. Standard NeRF pipelines use COLMAP poses but discard its sparse 3D points; without depth supervision NeRF must solve the 3D correspondence problem implicitly §1. The method assumes the poses come from structure-from-motion (SfM), and it is only as good as the SfM depth §3.2 app. A.

Method

Volumetric rendering §3.1. A colour ray is the integral of density and colour along the ray, weighted by transmittance. NeRF approximates this integral with samples eq. 1. The reconstruction loss over rendered colours is the mean squared error eq. 2.

\hat{\mathbf{C}} = \int_{0}^{\infty} T(t)\,\sigma(t)\,\mathbf{c}(t)\,dt, \qquad T(t) = \exp\!\Big(-\int_{0}^{t}\sigma(s)\,ds\Big) eq. 1

\mathcal{L}_{\mathrm{Color}} = \mathbb{E}_{\mathbf{r}\in\mathcal{R}(\mathbf{P})} \big\lVert \hat{\mathbf{C}}(\mathbf{r}) - \mathbf{C}(\mathbf{r}) \big\rVert_2^2 eq. 2

Ray-termination distribution §3.1. The paper defines h(t) = T(t)\sigma(t) as the probability that a ray terminates at distance t. The scene lies between near and far bounds, and the far bound is treated as an opaque wall so that h integrates to one §3.1 app. B.1. The derivation assumes every ray eventually hits an opaque surface, \int_a^\infty \sigma = \infty for all a \ge 0 app. B.1. The ideal h for a pixel with closest-surface depth \mathbf{D} is a delta at \mathbf{D}, and the empirical variance of h falls as NeRF is trained on more views §3.1 fig. 3.

Keypoint depth §3.2. SfM returns 3D keypoints and, for each camera j, the keypoints it sees. Project keypoint \mathbf{x}_i with camera \mathbf{P}_j; its re-projected z value is the keypoint depth D_{ij}. Its uncertainty \hat\sigma_i is the average reprojection error over the views that saw it §3.2. The paper models the first surface along the ray as \mathcal{N}(D_{ij}, \hat\sigma_i) and minimises the KL divergence to h_{ij}. Dropping constants gives the depth loss, stated in the paper as an integral and then discretised over samples t_k §3.2. The paper does not define \Delta t_k.

\mathcal{L}_{\mathrm{Depth}} = \mathbb{E}_{\mathbf{x}_i\in X_j}\Big[ -\int \log h(t)\,\exp\!\Big(-\frac{(t-D_{ij})^2}{2\hat\sigma_i^2}\Big)\,dt\Big] \approx \mathbb{E}_{\mathbf{x}_i\in X_j}\Big[ -\sum_k \log h_k\,\exp\!\Big(-\frac{(t_k-D_{ij})^2}{2\hat\sigma_i^2}\Big)\,\Delta t_k\Big] §3.2

The total loss is \mathcal{L} = \mathcal{L}_{\mathrm{Color}} + \lambda_D\,\mathcal{L}_{\mathrm{Depth}}, where \lambda_D is a hyper-parameter whose value the paper does not give §3.2.

# DS-NeRF training loss for one step  [§3.2] [eq. 2]
# Input:  color_rays (N_rgb rays), target_rgb (N_rgb, 3)
#         depth_rays (N_depth rays), target_depth (N_depth,), sigma_hat
# Output: scalar loss

def training_loss(color_rays, target_rgb, depth_rays, target_depth, sigma_hat):
    # Render colours for the pixel rays  [§3.1] [eq. 1]
    rgb_pred = render_color(color_rays)                 # (N_rgb, 3)
    # Render termination weights h_k along the keypoint rays  [§3.1]
    weights, t_vals = render_termination(depth_rays)    # (N_depth, K)
    # Colour reconstruction term  [eq. 2]
    loss_color = mean((rgb_pred - target_rgb) ** 2)
    # Depth term: keypoint depth as a Gaussian belief  [§3.2]
    loss_depth = depth_distribution_loss(
        weights, t_vals, target_depth, sigma_hat)
    # lambda_D is not given in the paper; released configs use 0.1
    # [code: configs/fern_2v.txt#L15]
    return loss_color + lambda_depth * loss_depth

# Negative log-likelihood of keypoint depth under h  [§3.2]
# Input: weights (K,), t_vals (K,), scalar target_depth, scalar sigma_hat

def depth_distribution_loss(weights, t_vals, target_depth, sigma_hat):
    # Gaussian belief over the first-surface depth, per sample  [§3.2]
    belief = exp(-((t_vals - target_depth) ** 2) / (2 * sigma_hat ** 2))
    # Sample spacing; the paper leaves dt_k undefined, the release uses
    # t_{k+1} - t_k  [code: loss.py#L77-L80]
    spacing = sample_spacing(t_vals)                    # (K,)
    # Discretised integral, computed per ray  [§3.2] [code: loss.py#L87]
    return -sum(log(weights + 1e-5) * belief * spacing)

# Depth term as released by default: squared error of expected depth
# [code: run_nerf.py#L989-L990]
# expected depth = sum_k w_k * t_k  [code: run_nerf_helpers.py#L380]

def released_depth_loss(depth_pred, target_depth):
    return mean((depth_pred - target_depth) ** 2)

# Depth target of a keypoint seen in camera j  [§3.2]
# [code: load_llff.py#L371]
# Input: point_xyz (3,), cam_center (3,), cam_z_axis (3,)

def keypoint_depth(point_xyz, cam_center, cam_z_axis):
    # Project the 3D point onto the camera's viewing axis  [§3.2]
    return cam_z_axis @ (point_xyz - cam_center)

Ray batch §3.2 app. B.2. Keypoint rays are separate from pixel rays in each batch; the paper says a portion of the training rays is allocated to keypoints and that RGB supervision is interpolated for them. The release splits the batch 50/50 by default and does not supervise keypoint rays with RGB (see Paper vs code).

Hyperparameters

Values come from the paper where stated. Otherwise they come from the released LLFF/NeRF-Real configs and defaults, and the table says which.

Results

Author-reported. Setups: NeRF Real has 8 forward-facing scenes evaluated at 2, 5 and 10 training views §4.1. DTU has 15 test scenes at 3, 6 and 9 views tab. 2. Redwood has 5 RGB-D sequences at 2, 5 and 10 views app. B.5. The baselines are LLFF, NeRF, pixelNeRF, metaNeRF, IBRNet and MVSNeRF §4.2. The headline is DS-NeRF with the KL-form loss, which the release does not run by default (see Paper vs code).

NeRF Real tab. 1. Values are PSNR ↑ / SSIM ↑ / LPIPS ↓ at 2 / 5 / 10 views.

Setup: DS-NeRF uses the NeRF Real training subsets with COLMAP keypoints §4.1. The headline is 20.2 PSNR at 2 views versus 13.5 for NeRF tab. 1. The paper also says the depth-supervised pixelNeRF and IBRNet finetunes improve, while MVSNeRF does not §4.3 tab. 1.

DTU tab. 2. Values are PSNR ↑ / SSIM ↑ / LPIPS ↓ at 3 / 6 / 9 views.

DS-NeRF is behind the DTU-pretrained pixelNeRF and metaNeRF at 3 views, which the paper attributes to DTU priors §4.3 tab. 2. At 9 views, NeRF's SSIM (0.82) is above DS-NeRF's (0.81) tab. 2.

Redwood-3dscan tab. 3. Values are PSNR ↑ / SSIM ↑ / LPIPS ↓ at 2 / 5 / 10 views.

Setup: the RGB-D variant replaces COLMAP supervision with dense depth derived from 3DMatch on RGB-D input, scaled to the COLMAP poses by a least-squares fit §4.1 §4.3. At 10 views, NeRF's SSIM (0.82) is above DS-NeRF's (0.81) tab. 3.

Depth error tab. 4. Percent error of rendered depth against reference depth from dense MVS (NeRF Real) or RGB-D fusion (Redwood).

The error uses a least-squares scale and shift a, b that align the COLMAP keypoint depth to the MVS depth on training views eq. 3 app. B.6:

\min_{a,b}\sum_{p\in\mathcal{P}}\big(a\,D_{\mathrm{SfM}}(p) - b - D_{\mathrm{MVS}}(p)\big)^2 eq. 3

Then \mathrm{Err}(\hat D) = \lVert a\hat D - b - D_{\mathrm{MVS}}\rVert^2 for each test view eq. 4 app. B.6. The dense MVS depth is used only for evaluation app. B.6.

\mathrm{Err}(\hat D) = \big\lVert a\,\hat D - b - D_{\mathrm{MVS}}\big\rVert^2 eq. 4

Four-fold NeRF Real tab. 5, appendix. Trained on 6 scenes and tested on 2, for each of 4 splits. PSNR / SSIM / LPIPS at 2 / 5 / 10 views.

The paper uses this setting to show that metaNeRF and pixelNeRF, when their priors come from NeRF Real scenes instead of DTU, do not reach DS-NeRF's quality app. C.1 tab. 5.

Training speed §4.5 fig. 7. DS-NeRF reaches a given test PSNR in 2-3x fewer iterations than NeRF on NeRF Real. At 2 views NeRF is unable to match DS-NeRF's PSNR at all §4.5. Per iteration, DS-NeRF takes about 362.4 ms and NeRF about 359.8 ms on a single RTX A5000 §4.5. The paper estimates that at 5 views DS-NeRF reaches NeRF's peak PSNR about 13 hours sooner in wall-clock time §4.5.

Ablations fig. 6 app. C.3. Replacing the KL depth loss with an MSE between SfM depth and rendered depth lowers the 2-view NeRF Real PSNR from 20.2 to 19.5 tab. 1. Keeping only 20%, 50% or 100% of keypoints at 5 views gives 21.5, 22.2 and 22.6 PSNR, against 18.2 for NeRF app. C.3.

Paper vs code

Code: dunbar12138/DSNeRF at commit cacad3c86086f5749970d277437e55ba243037e9. The authors' project page (https://www.cs.cmu.edu/~dsnerf/) links this repository. The repository's LICENSE is MIT, but its copyright line names bmild (2020), the NeRF codebase's authors LICENSE. The GitHub API reports MIT.

1. Depth loss form (main discrepancy). The paper's headline is the KL/NLL distribution loss §3.2 tab. 1. The released LLFF configs set depth_loss = True and colmap_depth = True but not sigma_loss, so the depth term is the plain squared error between expected depth and keypoint depth, weighted by 0.1 fern_2v.txt run_nerf.py. The paper's MSE ablation §4.2 appears to match this default. The distribution loss is implemented as SigmaLoss in loss.py, but it runs only with --sigma_loss run_nerf.py run_nerf.py loss.py.

2. Uncertainty. The paper uses \hat\sigma_i, the average reprojection error, in the Gaussian term §3.2. The distribution loss in the release does not use it: SigmaLoss is called with its default err=1, a constant run_nerf.py loss.py. The weighted-MSE option uses a different weight, w = 2\exp(-(\mathrm{err}/\overline{\mathrm{err}})^2) with err the COLMAP point error and \overline{\mathrm{err}} its mean load_llff.py run_nerf.py. That is a heuristic, not the paper's Gaussian.

3. RGB on keypoint rays. The appendix says RGB supervision is interpolated for keypoint rays so that depth and colour come from one ray march app. B.2. In the release a keypoint ray carries origin, direction, depth and weight, with no RGB run_nerf.py. RGB loss is computed only on the N_rgb pixel rays run_nerf.py.

4. Ray split. The paper says "a portion of the training rays" app. B.2. The release fixes 50% by default run_nerf.py run_nerf.py.

5. Coarse network loss. The paper states one loss. The release adds the coarse network's colour MSE with weight 1 run_nerf.py.

6. Depth data source. The paper derives keypoint depth from COLMAP §3.2. The colmap_llff path the LLFF configs use loads train_depths.npy load_llff.py, and this repository does not generate that file. The COLMAP depth generator, load_colmap_depth, is used only in the llff path and rescales depth by bd_factor=0.75 load_llff.py run_nerf.py. The units of the released data are therefore not reproducible from this code alone.

7. Missing configs and data paths. The release has only eight LLFF/NeRF-Real configs (*_2v.txt) . No DTU or Redwood configs are included. The code has no 3DMatch or Redwood loader; load_sensor_depth exists in load_llff.py but nothing calls it load_llff.py. The README's examples name configs/fern_dsnerf.txt, which does not exist README.md.

8. Depth-ray sampling. The keypoint-ray iterator restarts when it is exhausted run_nerf.py. The paper does not describe this.

Pitfalls

• Pick one depth loss and say which. An implementer who follows the paper gets the KL form with an undefined \sigma scale; one who runs the release gets a plain MSE. The paper's headline numbers come from the KL form. Only the released MSE default is run by the released configs.

• \sigma is a variance or a standard deviation, depending on where you read. The text calls \hat\sigma_i a variance §3.2, but the formula uses 2\hat\sigma_i^2. The release uses 2 * err with err=1 loss.py loss.py. Use one convention and check that the belief is about the size of the depth error.

• Scale must be consistent. The Gaussian width is in scene units. Keypoint depths, near and far bounds, and the reprojection-error weights must all use the same scale run_nerf.py load_llff.py. Keypoints outside the bounds are dropped load_llff.py.

• Last interval is 1e10 wide. Both raw2outputs and SigmaLoss pad the last spacing with 1e10 run_nerf_helpers.py loss.py. This implements the paper's far-bound opaque-wall assumption §3.1, but in SigmaLoss the padded term is multiplied by the distance. Check its numeric size when porting.

• Log floor. The release computes log(weights + 1e-5) loss.py, which floors each per-sample log at about −11.5 (computed). Weights far below 1e-5 then pass almost no gradient through the log. The cumulative product also adds 1e-10 run_nerf_helpers.py.

• Speed claims. "2-3x faster" means fewer iterations to a PSNR threshold §4.5. Per-iteration cost is higher, about 0.7% by the paper's numbers §4.5. The wall-clock saving depends on the hardware and iteration count.

• Costs hidden in "free". The method needs SfM on every training set, including a COLMAP run per scene and camera poses. For DTU the paper uses ground-truth calibrated poses to get keypoints §4.1 app. B.3. The depth is only as good as SfM and the Gaussian assumption; the paper lists both as limitations app. A.

• Keypoint count. There are about 1615 keypoints per view at 2 views app. C.3, but each depth batch draws 2048 keypoint rays. The iterator reshuffles and restarts, so many keypoints are reused within a batch cycle run_nerf.py.

• Missing values to set yourself. The paper gives no learning rate, iteration count, ray count, MLP size, λ_D or near/far values for NeRF Real. The LLFF-config values above are the only specified ones. Take them from the release and mark them as code-derived.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020), the base model here §1. Its arXiv id is not in this paper's bibliography, and it has no card in this wiki yet. Also COLMAP SfM (Schönberger and Frahm, 2016) for poses and keypoints §3.2.
• Compared against: LLFF (Mildenhall et al., 2019) §4.2 tab. 1; pixelNeRF §4.2; metaNeRF §4.2; IBRNet §4.2; MVSNeRF §4.2. RGB-D depth uses 3DMatch (Zeng et al., 2017) §4.3.
• Superseded by: none stated in the wiki content.