SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views

In plain words

Turning a few photos of an object into a 3D surface is hard when there are only two or three of them. Most neural methods for this need many photos and a long training run for each new object, and they fall apart when the photos are sparse.

SparseNeuS learns, across many scenes, what a plausible surface looks like given a handful of photos with known camera positions. For a new scene it builds a coarse 3D grid of image features, uses it to guess the rough shape, then refines the shape with a finer grid placed only near that rough surface. Colours are not invented from scratch. The network picks how much to trust each input photo at each point, and blends those photos' colours. An optional short fine-tuning pass on the new scene sharpens the surface, and it ignores regions where the photos disagree, such as where something is hidden from one of the cameras.

TL;DR

• Input: two or three posed photos of an unseen scene, no 3D ground truth.
• Output: a closed surface stored as a signed distance function, extracted at the zero level set §3.1.
• Generic inference needs no per-scene training. Per-scene fine-tuning for 10k iterations gives the best reported DTU numbers §5.1.
• Author-reported DTU Chamfer distance: 1.64 generic and 1.27 fine-tuned, against 1.52 for COLMAP tab. 1.

Problem

Dense multi-view stereo needs many overlapping views, and with sparse views weak textures, noise and reflections leave gaps in the matches §1. Earlier neural surface methods depend on a large number of views and run a slow optimisation per scene, so they do not generalise to new scenes §1. The paper targets surfaces from sparse views with both properties: generalisation and few inputs §1, §5.1.

Method

The pipeline has three parts: geometry reasoning, appearance prediction and per-scene fine-tuning §3.

1. Geometry encoding volume §3.1. A bounding box that covers the region of interest is defined in the camera frame of the centred input image and split into regular voxels §3.1. Each input image goes through a 2D feature network (a Feature Pyramid Network §4). Each voxel is projected into every view and its feature is read by interpolation. The per-voxel variance across views is the cost volume B. A sparse 3D U-Net-like CNN turns B into the geometry encoding volume M §3.1 eq. 1.

2. Surface prediction §3.1. An MLP f_θ takes the positionally encoded 3D point and the interpolated feature M(q), and outputs the signed distance s(q). The surface is the zero level set of s.

3. Cascade §3.1 §4. Two volumes are built. The coarse one (96³) gives the global shape. The fine one (192³) is built only where the coarse surface lies; the paper says far-away vertices are discarded, which cuts memory and time. Training is two-stage: the coarse networks train first, then the fine networks train while the coarse ones are frozen §4.

4. Blending weights §3.2. A location q is projected into every input view to read a colour and a feature F_i(q). The feature is concatenated with the mean and variance of the features across views and passed through a small MLP. A second MLP f_c takes that feature, M(q) and the view-direction difference Δd_i = d − d_i, and outputs a weight per view. The weights are normalised with a softmax across views. The colour of q is the weighted sum of the projected colours §3.2.

5. Pixel-based blending §3.2 eq. 2. Colours and SDF values are taken at 64 coarse and 64 fine samples per ray app. §2. SDF is turned into density with the NeuS conversion, and volume rendering accumulates the per-sample colours §3.2 eq. 2. The density conversion parameters are described by reference to NeuS only.

6. Patch-based blending §3.2 app. §1. The local normal at q is the gradient of the SDF, n_q = ∇s(q). A small set of points is sampled on the local plane through q and its normal. A homography induced by that plane maps the points into each input view, so their colours are read by interpolation. They reuse the blending weights of q, so only one weight query is needed. The patch colours are volume-rendered the same way as pixel colours §3.2 eq. 2. The patch is 5×5 §4.

7. Generic training losses §3.4 app. §2. Training uses no 3D ground truth. The total loss is a weighted sum of a colour term, an Eikonal term eq. 5 and a sparseness term eq. 6, with α = 0.1 and β = 0.02 eq. 4. In generic training the consistency-aware log terms are removed §3.4.

8. Per-scene fine-tuning §3.3. The 2D feature network and the 3D CNN are dropped. The fine volume and the SDF MLP are kept and optimised directly. The blending CNN is replaced by a tiny MLP f_c′ whose inputs are PE(q), PE(d), the normal n_q, s(q) and M(q) §3.3. The colour loss is the consistency-aware form in eq. 3. Its opacity-weighted pixel L1 and patch NCC terms are gated by two log terms on the accumulated opacity O(r). Defaults are λ0 = 0.01 and λ1 = 0.015 app. §2. The fine-tuning uses only the three input images §4.

The consistency-aware loss above is written in the paper's form. The released code does not implement its log terms; see Paper vs code.

Pseudocode for the geometry and blending forward pass. This is the paper's formulation, not the released code, and it omits the cascade and the training losses.

# Build a geometry encoding volume for one level  [§3.1] [eq. 1]
# Input:  feature maps F (one per view), projections pi, voxel centres V
# Output: geometry encoding volume M, one feature per voxel

def build_geometry_volume(feature_maps, projections, voxel_centres):
    # Project each voxel into every view and read its feature  [§3.1]
    projected = []
    for view in range(num_views):
        projected.append(interpolate(feature_maps[view],
                                     projections[view](voxel_centres)))
    # Matching cost: variance of the projected features across views  [eq. 1]
    cost = variance(projected, axis_views)                    # (V, C)
    # Sparse 3D CNN aggregates the cost into geometry features  [§3.1]
    return sparse_unet(cost)                                  # (V, C')

# Signed distance and colour at one sample on a ray  [§3.1] [§3.2]
# Input:  point q (3,), ray direction d (3,), volume M, feature maps F
# Output: signed distance s, colour c

def query_point(q, d, volume, feature_maps, projections):
    # SDF from the positionally encoded point and its volume feature  [§3.1]
    sdf = sdf_mlp(positional_encoding(q), interpolate(volume, q))
    # Feature of q in each view, plus mean and variance across views  [§3.2]
    view_feats = [interpolate(feature_maps[v], projections[v](q))
                  for v in range(num_views)]
    shared = mean_and_variance(view_feats)
    # Per-view weight from feature, volume feature and direction change  [§3.2]
    logits = []
    for v in range(num_views):
        refined = tiny_mlp(concat(view_feats[v], shared))
        delta_d = d - view_direction[v]
        logits.append(blend_mlp(refined, interpolate(volume, q), delta_d))
    # Softmax over views, then a weighted sum of the projected colours  [§3.2]
    weights = softmax(logits)
    colours = [sample_colour(v, projections[v](q)) for v in range(num_views)]
    colour = sum(weights[v] * colours[v] for v in range(num_views))
    return sdf, colour

Key equations. The volume-rendering weights are cumulative transmittance times per-sample opacity, applied to pixel or patch colours §3.2 eq. 2. The consistency-aware colour loss is written in eq. 3 and the total loss in eq. 4.

U(r)=\sum_{i=1}^{M} T_{i}\left(1-\exp(-\sigma_{i})\right) u_{i},\quad T_{i}=\exp\left(-\sum_{j<i}\sigma_{j}\right) eq. 2

\mathcal{L}_{total} = \mathcal{L}_{color} + \alpha\,\mathcal{L}_{eik} + \beta\,\mathcal{L}_{sparse} eq. 4

Hyperparameters

Values are as stated in the paper unless the Where column says otherwise. Code values are read from the official repository at code_commit; code paths are relative to the repository root.

Results

Author-reported. Setup: DTU, 15 test scans, three input views per set, two sets per scan, IDR's foreground masks for evaluation, Chamfer distance of predicted meshes against ground-truth point clouds §4 §5.1. The table's unit is not stated in the paper. Lower is better tab. 1.

Ablations on one three-view set of each of the 15 test scans §5.2 tab. 2:

Per-scene optimisation from scratch, with and without learned priors, on the scene in fig. 6: 1.98 without priors and 1.65 with them. The text does not name the scene §5.2 fig. 6.

Qualitative: generic results are smoother than MVSNeRF's on DTU fig. 3; fine- tuned results are more complete than COLMAP's on weak-texture objects and cleaner than NeuS's, which the paper reports as distorted with three views §5.1 fig. 3. On BlendedMVS, which is never used for training, the generic model is compared visually with MVSNeRF, COLMAP and NeuS §5.1 fig. 4. These comparisons are visual only; no BlendedMVS table is given.

Paper vs code

Official code: xxlong0/SparseNeuS (linked from the project page and the paper's abstract footnote), commit ce3b7ef35e5c737ed2505d8cc54ee22ee0ed6415, MIT licence. Paths are relative to the repository root.

• Generic training length. Paper: 150k coarse plus 150k fine iterations §4. Code: end_iter = 200000 in both lod configs, with a cosine schedule and no milestone decay, although learning_rate_milestone is parsed general_lod0.conf exp_runner_generic.py.
• Learning rate. Paper: not stated §4. Code: 2e-4, with a cosine decay general_lod0.conf.
• SDF MLP width. Paper: 256 app. §2. Code: hidden_dim = 128 general_lod0.conf.
• Generic blending MLP. Paper: a three-layer, 256-wide MLP for the fine-tune blender app. §2. Code generic rendering head: first hidden layer 64, then 32 rendering_network.py. The fine-tune blender was not checked.
• Patch size. Paper: 5 × 5 §4. Code fine-tune: h_patch_size = 3, giving a 7 × 7 window finetune.conf. The NCC uses a Gaussian-weighted window with σ = 1.5 ncc.py ncc.py. The paper does not describe the window weighting.
• Patch error selection. Paper: sums the patch NCC error eq. 3. Code keeps only the smallest 70% of patch errors per batch color_loss.py. This selection is not in the paper.
• Patch loss weight. Paper: patch term has weight 1 in eq. 3. Code: 0.2 finetune.conf.
• Consistency-aware loss. Paper: opacity-weighted pixel and patch terms, plus λ0 and λ1 log terms on O(r) eq. 3 app. §2. Code: the λ log terms are absent. The visibility arguments are read by the argument parser and commented out in the trainer exp_runner_finetune.py trainer_finetune.py. The fine-tune loss instead uses a binary cross-entropy between the accumulated weight sum and a foreground mask, weighted 0.5 trainer_finetune.py trainer_finetune.py finetune.conf.
• Foreground masks in fine-tuning. Not in the paper. The README says the user should remove backgrounds with a segmentation model, and calls this "a bit difference from original paper" [README.md].
• Generic colour loss. Paper: opacity-weighted pixel and patch terms, with the log terms removed §3.4. Code: plain L1 terms with no opacity weight and no patch term trainer_generic.py trainer_generic.py trainer_generic.py.
• Pixel loss scale. Fine-tune: sum of absolute error over RGB color_loss.py. Generic: mean over elements, which averages RGB trainer_generic.py. For equal per-channel error, the fine-tune colour term is three times the generic one.
• Sparseness term. Paper: one exponential term over sampled points eq. 6. Code: the average of two exponential terms, one over random samples trainer_generic.py. In fine-tuning, the random-sample term uses 10× τ trainer_finetune.py.
• Loss schedule. Not in the paper. Code applies a linear ramp to the sparseness and Eikonal weights. For the coarse level, the ramp runs from 0 to 50k iterations; for the fine level it runs from 15k to 30k trainer_generic.py. The foreground-background term is off before 50k iterations trainer_generic.py.
• Foreground-background term. Not in the paper. Code adds an L1 term that pushes weight sums to zero where the mask is zero, weight 0.01, active after 50k iterations trainer_generic.py general_lod0.conf. The paper mentions black background only as an optional prior §4.
• Training views. Paper: not stated for generic training. Code: nviews = 5 general_lod0.conf.
• Denoising. Paper: intensity threshold 10, dilation and erosion, drop components smaller than 10% of the image app. §2. Code: loads pre-made masks from Masks/<scan>_train/ and zeroes masked pixels dtu_general.py. The threshold algorithm is not in the repository. Where no mask file exists, the code uses an all-ones mask, so nothing is cleaned.
• Fine-tune learning rate. Paper: not stated. Code: geometry parameters are frozen for the first 500 iterations, then decay to half exp_runner_finetune.py.
• Sparseness weight comment. The lod1 config comment says 0.002, but both configs set 0.02 general_lod1.conf.
• Variance init. Not stated in the paper. Code: 0.2 general_lod0.conf.

Pitfalls

• Equation numbers. The source numbers the equations in printed order as: 1 (cost volume, §3.1), 2 (volume rendering), 3 (consistency-aware colour loss), 4 (total loss), 5 (Eikonal), 6 (sparseness). The appendix refers to the consistency loss as "Eq. 6", the total loss as "Eq. 7" and the sparseness term as "Eq. 10" app. §2. Those appendix numbers do not match the source order. Check which numbers the printed PDF shows before citing.
• Log terms in eq. 3. As printed, the log terms are unbounded below in O(r), so the sign convention in eq. 3 is unclear. The paper's stated purpose is to drive opacity to zero in inconsistent regions. The released code has no such terms, so this cannot be settled from the code.
• Tables 1 and 2 disagree on the same method. Fine-tuned cascaded volumes score 1.27 in tab. 1 but 1.21 in tab. 2. tab. 2 is evaluated on one set of three images per scan, not the full protocol. Use tab. 1 for headline comparisons.
• Generic result versus COLMAP. The generic mean (1.64) is worse than COLMAP's (1.52) in tab. 1. The "large margin" claim in §5.1 is against the learned-field baselines and the generic results, not COLMAP. Only the fine-tuned result beats COLMAP.
• Fine-tuning relies on foreground masks. The paper's fine-tune uses only the input images §4. The released fine-tune loss includes a mask term trainer_finetune.py finetune.conf, and the README tells the user to produce masks with a segmentation model first. Plan for that step before fine-tuning.
• Closed surfaces only. The representation is a signed distance field, so the output is a closed surface §6. Open surfaces are listed as future work §6.
• Patch blending is expensive per point. A naive k × k patch needs k² ray queries. The paper avoids this with a local plane and homography §3.2 app. §1, but the sampled-plane point count is not stated.
• Sample count for the plane. The number of points sampled on the local plane is not stated app. §1.
• Sparse-view claims for COLMAP. The paper runs COLMAP on three images. It does not state COLMAP's settings §5.1.
• Denoising is not reproducible from the repo. The threshold and area steps in app. §2 must be reproduced from the paper's description. The masks used by the code come from the authors' download [README.md].
• Dataset download. The DTU training set and the prepared test set are hosted on personal cloud shares, linked from the README [README.md]. Check availability before planning a run.
• Timing claims. The 20-minute figure is for fine-tuning 10k iterations §5.1. Generic inference time is not stated in the text.

Lineage

• Builds on: the variance cost volume of MVSNet and MVSNeRF §3.1; SDF volume rendering from NeuS §3.2; the Eikonal regulariser of Gropp et al. §3.4.
• Compared against: PixelNeRF, IBRNet and MVSNeRF (generic) and IDR, NeuS, VolSDF, UniSurf and COLMAP (per scene or classic MVS) §5.1.
• Superseded by: not stated in this card's sources.