NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

In plain words

Given a few dozen photos of an object taken from known camera positions, we want a clean 3D surface mesh of it. One family of neural methods models the surface directly and renders only the point where each camera ray hits it. Those give crisp surfaces but need a cut-out mask of the object in every photo and tend to get stuck on objects with holes, thin parts or sudden depth jumps. Another family (NeRF and its relatives) models a fuzzy "fog" of density and blends colors along the whole ray. That trains robustly, but the fog has no precise surface, so meshes pulled out of it are noisy.

NeuS keeps the robust fog-style rendering but makes the fog come from a signed distance function: a network that tells, for any point, how far it is from the surface, positive outside and negative inside. The surface is where that distance is zero. The trick is in how distance is turned into fog. The obvious conversion makes the rendered color come mostly from a point slightly in front of the true surface, so the learned surface ends up in the wrong place. The authors derive a different conversion whose rendering weight peaks right on the surface (for locally flat surfaces) and still lets the nearest surface hide the ones behind it.

A single learned "sharpness" number controls how thick the fog layer around the surface is; it starts blurry and tightens during training. The result is a mesh you can extract directly from the zero level of the network, trained from photos alone, with or without masks.

TL;DR

NeuS represents geometry as an MLP signed distance function f plus a view-dependent color MLP, and trains both by volume rendering against input images. Instead of plugging an SDF-derived density straight into NeRF's w = T\sigma (which peaks before the surface), it defines an opaque density \rho from the logistic CDF \Phi_s of the SDF so that, discretised, \alpha_i = \max((\Phi_s(f_i) - \Phi_s(f_{i+1}))/\Phi_s(f_i), 0). The weight is occlusion-aware and its maximum lies on the zero-level set to first order. A single trainable scalar s (inverse standard deviation) sharpens the surface over training. Author-reported mean Chamfer distance on 15 DTU scans: 0.77 with masks, 0.84 without, best among IDR, NeRF, COLMAP and UNISURF in both settings.

Problem

Surface-rendering methods such as DVR and IDR need foreground masks, get trapped in local minima and struggle with severe self-occlusion, thin structures and abrupt depth changes, because gradients only flow through the single ray-surface intersection §1 fig. 1. Volume-rendering methods such as NeRF optimise robustly but their density field lacks surface constraints, so extracted level-set meshes are noisy §1 §4.2. Naively using an SDF-derived density in standard volume rendering introduces a systematic depth bias: the rendering weight is maximal before the ray reaches the surface §3.1 fig. 2 app. B. Assumptions: posed images, region of interest inside a unit sphere §4.1.

Method

Representation §3.1. Two MLPs: f:\mathbb{R}^3\to\mathbb{R} (SDF) and c:\mathbb{R}^3\times\mathbb{S}^2\to\mathbb{R}^3 (color from position and view direction). The surface is the zero-level set:

\mathcal{S} = \{ \mathbf{x} \in \mathbb{R}^3 \mid f(\mathbf{x}) = 0 \} eq. 1

The S-density is \phi_s(f(\mathbf{x})) with the logistic density \phi_s(x) = s e^{-sx}/(1+e^{-sx})^2, the derivative of the sigmoid \Phi_s(x) = (1+e^{-sx})^{-1}. Its standard deviation is 1/s (as written in the paper), and s is trainable; 1/s is expected to approach 0 as training converges §3.1.

Rendering §3.1. For a ray \mathbf{p}(t) = \mathbf{o} + t\mathbf{v}:

C(\mathbf{o}, \mathbf{v}) = \int_0^{+\infty} w(t)\, c(\mathbf{p}(t), \mathbf{v})\, dt eq. 2

Two requirements on w §3.1:

1. Unbiased: w has a local maximum at the surface intersection t^* with f(\mathbf{p}(t^*))=0.
2. Occlusion-aware: for two points with equal SDF, the nearer one gets the larger weight.

Three candidate weights:

• Naive solution eq. 3: w(t) = T(t)\sigma(t) with \sigma(t) = \phi_s(f(\mathbf{p}(t))) and T(t) = \exp(-\int_0^t \sigma(u)du). Occlusion-aware but biased: dw/dt(t^*) = -T(t^*)\phi_s(0)^2 < 0, so the peak is before the surface §3.1 app. B.
• Straightforward construction eq. 4: normalised S-density w(t) = \phi_s(f(\mathbf{p}(t))) / \int_0^{\infty}\phi_s(f(\mathbf{p}(u)))du. Unbiased but not occlusion-aware: two surfaces on a ray get blended equally §3.1.
• NeuS eq. 5: w(t) = T(t)\rho(t), T(t) = \exp(-\int_0^t \rho(u)du) with the opaque density below.

Deriving \rho §3.1 eq. 6–10. For a single plane infinitely far from the camera, f(\mathbf{p}(t)) = -|\cos\theta|(t - t^*), and eq. 4 reduces to w(t) = |\cos\theta|\,\phi_s(f(\mathbf{p}(t))) eq. 6. Setting this equal to T(t)\rho(t) = -dT/dt and integrating gives T(t) = \Phi_s(f(\mathbf{p}(t))) eq. 7 eq. 8, hence

\rho(t) = \frac{-\frac{d\Phi_s}{dt}(f(\mathbf{p}(t)))}{\Phi_s(f(\mathbf{p}(t)))} eq. 9

For multiple intersections the numerator becomes negative where the SDF increases along the ray (exiting a surface), so it is clipped:

\rho(t) = \max\left(\frac{-\frac{d\Phi_s}{dt}(f(\mathbf{p}(t)))}{\Phi_s(f(\mathbf{p}(t)))}, 0\right) eq. 10

Expanded with the chain rule: \rho(t) = \max\left(\frac{-(\nabla f(\mathbf{p}(t))\cdot\mathbf{v})\,\phi_s(f(\mathbf{p}(t)))}{\Phi_s(f(\mathbf{p}(t)))}, 0\right) app. A.

Theorem 1 §3.1 app. B: if a ray enters a smooth surface from outside, f decreases monotonically on [t_l, t_r] \ni t^*, and the surface is locally planar (\nabla f fixed), then w from eq. 5 and eq. 10 is maximal at t^* on that interval. The proof makes no assumption about surfaces before t_l, so it covers multiple intersections. A second-order analysis (curved surface) gives a peak offset \Delta_t = O(s^{-2}) for NeuS vs O(s^{-1}) for the naive solution, obtained by numerically solving the stationarity conditions app. C fig. 9.

Discretisation §3.1. With n samples t_1 < \dots < t_n:

\hat{C} = \sum_{i=1}^{n} T_i \alpha_i c_i, \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j) eq. 11

\alpha_i = 1 - \exp\left(-\int_{t_i}^{t_{i+1}} \rho(t)\,dt\right) eq. 12

\alpha_i = \max\left(\frac{\Phi_s(f(\mathbf{p}(t_i))) - \Phi_s(f(\mathbf{p}(t_{i+1})))}{\Phi_s(f(\mathbf{p}(t_i)))}, 0\right) eq. 13

The closed form is exact (no quadrature) because the integrand of eq. 12 is the derivative of -\ln\Phi_s(f(\mathbf{p}(t))) on entering segments and 0 on exiting ones app. A. In the implementation, \alpha_i uses SDF at section points \mathbf{q}_i = \mathbf{o} + t_i\mathbf{v} and color c_i is evaluated at mid-points \mathbf{p}_i = \mathbf{o} + \frac{t_i+t_{i+1}}{2}\mathbf{v} app. D.1 fig. 10. (The released code differs; see Paper vs code.)

Training §3.2. Each iteration samples a batch of m pixels/rays from one image, with color C_k and optional mask M_k\in\{0,1\}; n points per ray:

\mathcal{L} = \mathcal{L}_{color} + \lambda\mathcal{L}_{reg} + \beta\mathcal{L}_{mask} eq. 14

\mathcal{L}_{color} = \frac{1}{m}\sum_k \mathcal{R}(\hat{C}_k, C_k), \quad \mathcal{R} = \text{L1} eq. 15

\mathcal{L}_{reg} = \frac{1}{nm}\sum_{k,i}\left(\|\nabla f(\hat{\mathbf{p}}_{k,i})\|_2 - 1\right)^2 eq. 16

\mathcal{L}_{mask} = \text{BCE}(M_k, \hat{O}_k), \quad \hat{O}_k = \sum_{i=1}^{n} T_{k,i}\alpha_{k,i} eq. 17

\mathcal{L}_{reg} is the Eikonal term §3.2. The trainable s is optimised jointly with the networks §3.2.

Hierarchical sampling §3.2 app. D.1. One network only (no NeRF-style coarse/fine pair). Sample 64 points uniformly, then 4 rounds of importance sampling, 16 new points each (128 total). Round i computes its sampling PDF from NeuS weights with a fixed s = 32 \times 2^i rather than the learned s; the final rendering uses the learned s. In the w/o-mask setting, 32 extra points outside the unit sphere feed a NeRF++ background model.

# NeuS forward pass for a batch of rays  [§3.1] [§3.2] [app. D.1]
# Input:  rays_origin (m, 3), rays_dir (m, 3), unit-length directions
# Output: rendered colors (m, 3), opacity sums (m,), eikonal loss

def render_rays(rays_origin, rays_dir):
    # Ray segment inside the unit sphere  [§4.1]
    near, far = intersect_unit_sphere(rays_origin, rays_dir)
    # Coarse samples: 64 uniform depths  [app. D.1]
    depths = uniform_samples(near, far, 64)
    sdf = sdf_net(rays_origin + depths * rays_dir)
    # 4 rounds of importance sampling with a fixed s  [app. D.1]
    for round_index in range(4):
        # Paper: s = 32 * 2^i (released code uses 64 * 2^i)  [app. D.1]
        fixed_s = 32 * 2 ** round_index
        weights = neus_weights(sdf, depths, fixed_s)
        new_depths = sample_from_pdf(depths, weights, 16)
        depths = sort(concat(depths, new_depths))
        sdf = sdf_net(rays_origin + depths * rays_dir)
    # Final render with the learned s  [§3.2]
    section_points = rays_origin + depths * rays_dir
    mid_points = rays_origin + midpoints(depths) * rays_dir
    sdf_section = sdf_net(section_points)
    normals = grad_sdf(mid_points)
    feature = sdf_net_feature(mid_points)
    colors = color_net(mid_points, rays_dir, normals, feature)
    # Discrete opacity from section-point SDFs  [eq. 13] [app. D.1]
    cdf = sigmoid(learned_s * sdf_section)
    cdf_prev = cdf[:, :-1]
    cdf_next = cdf[:, 1:]
    alpha = clamp_min((cdf_prev - cdf_next) / cdf_prev, 0)
    # Transmittance and weights as in NeRF  [eq. 11]
    transmittance = cumprod_exclusive(1 - alpha)
    weights = transmittance * alpha
    color = sum(weights * colors)
    opacity = sum(weights)
    # Eikonal regulariser on sampled points  [eq. 16]
    eikonal = mean((norm(normals) - 1) ** 2)
    return color, opacity, eikonal

def training_loss(color, opacity, eikonal, target_rgb, mask):
    # L1 color + lambda * Eikonal + beta * BCE mask  [eq. 14] [eq. 15] [eq. 17]
    color_loss = mean(abs(color - target_rgb))
    mask_loss = binary_cross_entropy(opacity, mask)
    return color_loss + lam * eikonal + beta * mask_loss

Mesh extraction: the paper does not describe it beyond taking the zero-level set eq. 1. The code runs marching cubes on -f at threshold 0 over the object bounding box renderer.py renderer.py.

Hyperparameters

Results

Author-reported. Chamfer distance (lower is better) on 15 DTU scans (49 or 64 images, 1600×1200, masks from IDR), measured as in IDR and UNISURF §4.1 §4.2. IDR (w/ mask) and NeRF, UNISURF (w/o mask) scores are copied from the IDR and UNISURF papers; COLMAP uses trim=0 §4.2 tab. 1.

Bold marks the best per setting as in tab. 1. Columns 2–4: w/ mask; columns 5–8: w/o mask.

Ablation on one DTU scene (shown as a skull; scan id not stated), Chamfer distance and MAE between predicted and ground-truth SDF on points sampled uniformly in the bounding sphere §4.3 fig. 6:

Without Eikonal or geometric init, Chamfer stays close to the full model but the field is no longer a valid SDF; random init also produces axis-aligned artifacts attributed to positional-encoding spectral bias §4.3 app. E fig. 13.

NeRF baseline threshold (Chamfer, DTU) tab. 2:

Rendering speed app. E tab. 3: one 1600×1200 image takes about 320 s on a 2080Ti with default volume rendering, about 60 s when sphere tracing first locates the surface and only nearby points are sampled. Mean PSNR / SSIM over the 15 scans: 31.97 / 0.840 default vs 30.85 / 0.820 with sphere tracing.

Novel view synthesis (10% of DTU images held out, w/o mask) app. E tab. 4: mean PSNR 28.55 vs NeRF 29.31; mean SSIM 0.820 vs 0.826, i.e. comparable to NeRF, slightly lower.

Per-scan values from tab. 3 and tab. 4:

Qualitative only: 7 BlendedMVS scenes (31–143 images, 768×576) and two self-captured thin objects with 32 images, where NeuS recovers thin parts and abrupt depth changes that IDR and NeRF miss §4.1 §4.2 §4.3 fig. 4 fig. 5 fig. 8. A metal rabbit with textureless regions is shown as a failure case app. E fig. 16.

Paper vs code

Code: official repository by the first author (MIT), checked at 6e0a2309. Configs: wmask.conf (w/ mask), womask.conf (w/o mask, NeRF++ background), thin_structure.conf, womask_dtu_large_roi.conf.

• Importance-sampling s. Paper: fixed s = 32\times2^i in round i app. D.1. Code: 64 * 2**i renderer.py.
• Where SDF is evaluated for α. Paper: SDF at section points \mathbf{q}_i, \mathbf{q}_{i+1} app. D.1. Code: one SDF and gradient evaluation at the mid-point, then section-point SDFs are extrapolated linearly, sdf ± cos * dist / 2, with cos derived from \nabla f\cdot\mathbf{v} renderer.py. The importance-sampling step uses finite differences of neighbouring SDF samples instead, takes the minimum of the current and previous slope, and clips it to [-10^3, 0] renderer.py.
• Cosine annealing (not in the paper). In w/o-mask training the slope used for α is blended from -\text{relu}(-\cos/2 + 1/2) to -\text{relu}(-\cos) over the first 50k iterations (anneal_end = 50000), keeping it nonzero early on renderer.py exp_runner.py womask.conf. The w/ mask config sets anneal_end = 0, so only the second form is used.
• α clipping and epsilons. Code computes ((prev_cdf - next_cdf + 1e-5) / (prev_cdf + 1e-5)).clip(0, 1) rather than the bare max(·, 0) of eq. 13 renderer.py.
• Parameterisation of s. Paper: "trainable parameter" §3.1. Code: s = \exp(10\cdot v), v initialised to 0.3, clipped to [10^{-6}, 10^6] fields.py renderer.py.
• Eikonal points. Paper averages over all nm samples eq. 16. Code averages only over samples with \|\mathbf{p}\| < 1.2 renderer.py.
• Color loss with masks. Paper averages L1 over all m rays eq. 15. With mask_weight > 0 the code multiplies the color error by the mask, so background pixels do not contribute exp_runner.py. The BCE input is clipped to [10^{-3}, 1-10^{-3}] exp_runner.py.
• Positional encoding of position goes only into the SDF MLP; the color MLP receives raw 3-D position fields.py womask.conf. The paper's wording leaves this open app. D.1.
• Background inside the sphere. With NeRF++ enabled, samples outside the unit sphere take the background model's α and color instead of NeuS's renderer.py.
• Last segment length. The final sample's segment length is the constant 2 / n_samples, not a distance to the next sample renderer.py renderer.py.

Pitfalls

• The sign convention matters. f > 0 outside and < 0 inside app. B; α is nonzero only where the SDF decreases along the ray. Flipping the sign makes the object invisible. The geometric init (positive mean on the last layer, bias −0.5) starts f as roughly a sphere of radius 0.5 with this convention fields.py.
• Unbiasedness is first-order only. It holds for locally planar surfaces; for curved surfaces the offset shrinks as O(s^{-2}), so it relies on s growing during training app. C. A single global s is used for all locations, which the authors list as a limitation §5.
• Don't evaluate α at mid-points. Eq. 13 needs SDF at both ends of each segment. Using one SDF per sample with the naive 1 - \exp(-\sigma\delta) formula reintroduces the bias analysed in app. B.
• Division by \Phi_s(f_i) is near zero deep inside the object; add an epsilon as the code does renderer.py.
• Second-order gradients. The color MLP takes \nabla f and the Eikonal loss penalises it, so the SDF must be differentiated with create_graph=True at every sample fields.py. This, plus 128 samples per ray, is why training takes 14–16 h for 300k steps on one GPU §4.1.
• Cameras must be normalised so the object fits the unit sphere §4.1; near/far come from intersecting each ray with that sphere dataset.py.
• Unstated loss weights. λ and β are not given in the paper; use the code values (0.1, and 0.1 or 0) womask.conf.
• Rendering is slow. About 320 s per 1600×1200 image at default sampling app. E. Sphere tracing speeds it up to about 60 s at a cost of about 1.1 dB mean PSNR tab. 3.
• Textureless regions remain ambiguous and can fail §5 fig. 16.
• Baselines are mixed-source. Several numbers in tab. 1 are copied from the IDR and UNISURF papers, not rerun §4.2.

Lineage

• Builds on: NeRF (arXiv:2003.08934) for volume rendering, discretisation and hierarchical sampling; IDR (Yariv et al., 2020) for the SDF/color network architecture, weight normalisation, L1 color loss and data format; IGR (arXiv:2002.10099) for the Eikonal term; SAL (Atzmon and Lipman, 2020) for geometric initialisation; NeRF++ (arXiv:2010.07492) for the background model.
• Compared against: IDR, NeRF, COLMAP, and the concurrent UNISURF (arXiv:2104.10078), which instead uses an occupancy field.
• Superseded by: no later work with a card in this wiki yet.