UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

In plain words

Given photos of an object taken from known camera positions, we want a 3D surface of that object. One family of neural methods models the surface directly and only looks at the single point where each camera ray hits it. These give sharp surfaces, but they need a cut-out mask of the object in every photo and they can get stuck without one. Another family, the one behind NeRF, models a fuzzy cloud of density that light passes through and blends colours along the whole ray. That trains without masks, but the cloud has no crisp surface, so surfaces pulled out of it are noisy.

UNISURF uses one model for both views of the problem. It finds where each ray first hits the surface, then blends colours over a small band of points around that hit. The band starts wide, so the model can find the rough shape without masks, and narrows as training goes on, so the surface sharpens. The result is an accurate surface from photos alone, close to the mask-based method IDR.

TL;DR

UNISURF represents geometry as a neural occupancy field and renders it with one formula that gives both surface and volume rendering §4.1. Early in training, a wide sampling band around the surface lets volume rendering shape the model without masks; the band shrinks over training §4.3. On 15 DTU scans the authors report a mean Chamfer distance of 1.02 with no masks and no trimming, close to IDR's 0.90 with masks tab. 1. The paper's code is public under MIT, but several details of the released code differ from the paper (see Paper vs code).

Problem

Surface renderers such as DVR and IDR need pixel-accurate object masks for all views and a suitable initialisation, and they only get gradients where a ray meets the surface §1 §3. Volume renderers like NeRF need no masks, but the density level set they yield is non-smooth and contains artefacts because the radiance field is not constrained enough in ambiguous regions §1 fig. 3. The authors target solid, non-transparent objects; they state that fog or smoke are out of scope §4 §6.

Method

Occupancy field. The shape is an occupancy network o_\theta(\mathbf{x}): \mathbb{R}^3 \to [0,1], and the surface is the set of points where o_\theta = 0.5 §3 eq. 1. A colour field is learned jointly §3 eq. 2. The paper keeps NeRF's ray-integral view but restricts it to solid objects: with discrete occupancy the ray colour is the colour of the first occupied sample §4.1 eq. 6.

One formula for both renderings. The authors write NeRF's alpha blending eq. 3–eq. 5 and substitute the occupancy o_\theta for the alpha values §4.1. Volume rendering sums weights over N samples along the ray eq. 7. Surface rendering evaluates the colour field once at the surface point \mathbf{x}_s found by root-finding eq. 8. The colour field takes the point, the surface normal \mathbf{n}, a feature vector \mathbf{h} from the geometry network, and the view direction \mathbf{d} §4.1.

Normals. The normal is the normalised gradient of the occupancy field, computed with double backpropagation §4.2 eq. 12.

Loss. Reconstruction is an \ell_1 colour loss over the rays in the batch eq. 10, plus a smoothness term eq. 11 that pulls the normal at a surface point towards the normal at a nearby random point. The paper writes the total as \mathcal{L}_{rec} + \lambda \mathcal{L}_{reg} eq. 9 but does not state \lambda §4.2.

Sampling schedule. Each ray's sample window is [t_s - \Delta, t_s + \Delta] around the surface depth t_s. Samples are stratified inside that window eq. 13. \Delta_k decays exponentially from \Delta_{max} towards \Delta_{min} with iteration k and decay rate \beta eq. 14 §4.3. Samples between camera and surface are added to reduce free-space artefacts, and rays that miss the surface are sampled over their full length §4.3. The authors show that as \Delta \to 0 and N \to \infty, volume rendering approaches surface rendering; the proof is in the supplement, which is not in our source §4.1 §4.3.

Inference. Meshes are extracted with MISE, starting at 64^3 and upsampling 3 times with no gradient refinement §4.4. Novel views can be rendered either way. Surface rendering is faster, and the authors report similar images fig. 5.

# Unified rendering of one ray  [§4.1] [eq. 7] [eq. 8]
# Input:  points (N, 3) ordered from camera outward; view_dir (3,)
# Output: colour (3,)

def render_ray(points, view_dir, occupancy_net, colour_net):
    # Occupancy o_i in [0, 1] at each sample  [§4.1] [eq. 6]
    occupancy = occupancy_net(points)                 # (N,)
    # Visibility: no occupied sample lies in front of sample i  [eq. 7]
    visibility = exclusive_cumprod(1.0 - occupancy)   # (N,)
    # Weight of sample i is its occupancy times its visibility  [eq. 7]
    weights = occupancy * visibility                  # (N,)
    # Normal from the gradient of the occupancy field  [§4.2] [eq. 12]
    normals = normalise(gradient(occupancy_net, points))  # (N, 3)
    # Colour at each sample, conditioned on normal and features  [§4.1]
    features = occupancy_net.features(points)         # (N, 256)
    colours = colour_net(points, normals, features, view_dir)  # (N, 3)
    # Weighted sum over samples gives the pixel colour  [eq. 7]
    return sum(weights[:, None] * colours, axis=0)    # (3,)

# Training step on one batch of rays  [§4.2] [§4.3] [§4.4]
# Input:  rays for M = 1024 pixels of one view; observed colours (M, 3)
# Output: scalar loss

def training_step(rays, observed, iteration):
    # First crossing of each ray with the surface, no gradient  [§4.4]
    surface_t = ray_march_surface(rays, occupancy_net, threshold=0.5)
    # Half-width of the sample window, shrinking with iteration  [eq. 14]
    delta = max(delta_max * exp(-iteration * beta), delta_min)
    # 64 samples inside the window, 32 in free space  [§4.4] [eq. 13]
    samples = sample_window(surface_t, delta, n_inside=64, n_free=32)
    # Volume-rendered colour of each ray  [eq. 7]
    rendered = render_rays(samples, colour_net, occupancy_net)
    # L1 colour loss over the rays  [eq. 10]
    colour_loss = l1_sum(rendered, observed)
    # Normals at surface points and at jittered neighbours  [eq. 11]
    surface_points = samples_at_surface(samples)
    neighbours = surface_points + small_uniform_offset()
    normal_loss = l2_norm(normal(surface_points) - normal(neighbours))
    # Weighted sum; lambda is not stated in the paper  [eq. 9]
    return colour_loss + lam * normal_loss

Hyperparameters

Values are the paper's unless marked as code. Where the paper gives no value, the cell says "not stated". The code column is what the released DTU configs and model code use, cited by path and line.

Results

Author-reported. Setup: 15 DTU scans §5.2 with 49 to 64 images each at 1200×1600 and the official "Surface" Chamfer evaluation, applied to meshes cleaned with the masks §5.2. Each model is fit to one scene §4.4. Baselines are COLMAP with trim 7 and trim 0, NeRF with density threshold 50, and IDR with masks §5.1. Lower Chamfer is better. Bold numbers in the paper are best among methods without masks or trimming tab. 1.

Among methods without masks or trimming, UNISURF has the lowest Chamfer on 12 of 15 scans tab. 1. COLMAP with trim 7 is best on the mean, but the authors note its meshes are not watertight and incomplete §5.3.

Ablation on the DTU Buddha scene (Chamfer distance, lower is better) fig. 7:

Volume versus surface rendering for novel views fig. 5:

The paper reports BlendedMVS and SceneNet scenes only qualitatively: NeRF shows more noise than UNISURF, and COLMAP shows artefacts on uniform regions. IDR did not converge on any of these scenes §5.4 fig. 8. The LLFF results are in the supplement, which is not in our source §5.

Paper vs code

Code: official repository autonomousvision/unisurf at commit 224efc73d50acf422edacaebcaf59a8b8ba6d1ff, MIT licence LICENSE. Each difference below cites both sides.

• Sample window, DTU. Paper: Δ_max 1.0 and Δ_min 0.05 §4.4. Code: 2.0 and 0.1 in every DTU config scan0114.yaml. Decay β matches.
• Learning-rate schedule. Paper: decay after 200k and 400k iterations §4.4. Code: MultiStepLR at epochs 4000 and 8000 with gamma 0.5, stepped once per epoch scan0114.yaml train.py train.py. Each epoch passes over the per-image dataset, which has n_views = 64 entries per scan scan0114.yaml dataloading.py, so the code's milestones fall at about 256k and 512k iterations if an epoch is 64 iterations. This is inferred from the code, not stated by the paper.
• Training length. Paper: 450k iterations §4.4. Code: an unbounded loop that stops only when the user stops it train.py. The released checkpoint is model_400000.pt default.yaml.
• Colour-loss normalisation. Paper: a sum over rays eq. 10. Code: an \ell_1 sum over all ray channels divided by the number of rays losses.py.
• Regulariser normalisation. Paper: a sum over surface points eq. 11. Code: the mean over surface points, weighted by 0.005 losses.py scan0114.yaml.
• Normal fed to the colour network. Paper: the normalised normal \mathbf{n} §4.1 eq. 12. Code: the raw, unnormalised gradient of the occupancy logit. The normalisation line is commented out network.py.
• Free-space samples. Paper: samples drawn randomly between camera and surface, with no start iteration given §4.3. Code: 32 extra points, jittered within bins, added only after iteration 5000 rendering.py rendering.py.
• Network depth. Paper: 8-layer occupancy network, 4-layer colour network §4.4. Code: occupancy has 9 linear maps (8 hidden plus output), colour has 5 linear maps (4 hidden plus output) network.py network.py.
• Occupancy sharpness. Paper: o_\theta is a direct probability §3. Code: sigmoid of −10 times the network output network.py.
• Background model. The paper uses a background model for scenes with complex backgrounds §5.4. The code has a model_bg argument that is never built ("coming soon" in the docstring), and the training script does not pass one rendering.py rendering.py.
• Numerical constants. Code adds 10^{-6} to 1-\alpha in the transmittance product rendering.py rendering.py and 10^{-5} to the normal's norm rendering.py. Paper gives neither.
• View direction sign. Code feeds the direction pointing from the surface back to the camera (negative ray direction) rendering.py. Paper's \mathbf{d} is the viewing direction, with its sign not stated §4.1.
• Matches. Samples 64 and 32, 256 proposals, 8 secant steps, M = 1024, Adam at 1e-4, β = 1.5e-5, MISE settings: paper and code agree.

Pitfalls

• Supplement missing. The paper defers the volume-to-surface limit proof, the density-threshold validation for NeRF, the background model, LLFF results and the limitations to a supplement §4.1 §5.1 §5.4 §5.5 §6. None of it is in our source. Values only found there are "not stated" in this card.
• Hidden λ and ε. The regulariser weight and the neighbour offset are not stated in the paper. The code uses 0.005 and ±0.005 per axis scan0114.yaml rendering.py.
• Two window ranges. The paper's Δ values differ from the code's DTU configs §4.4 scan0114.yaml. Which one produced the reported table is not stated.
• Schedule units. Paper milestones are in iterations; the code's are in epochs. If you reimplement the code's schedule, the decay points depend on the dataset's view count.
• Surface rendering alone is not enough. In the authors' ablation, surface rendering without volume rendering does not converge fig. 7. The sampling schedule is described as critical for capturing detailed geometry §4.3.
• Free-space artefacts. Surfaces remain less accurate in rarely observed regions, and in overexposed or textureless areas §6. The paper cites a smoothness prior as the remedy and leaves a learned prior for future work §6.
• Mask-based baselines are not like-for-like. IDR uses masks, COLMAP trim 7 removes geometry, and COLMAP trim 0 is watertight but worse. Compare the columns that match your setting, as the paper's bold convention does tab. 1.
• Cost. Each ray needs 256 occupancy evaluations for the proposal search, 8 secant steps and 64 to 96 samples for colour §4.4 rendering.py. The figure reports 178 s for volume and 84 s for surface rendering, on a V100, for the view shown fig. 5. Training time is not stated.
• Solid objects only. The formulation assumes occupancy in {0, 1} and cannot represent transparent or semi-transparent material §4.1 §6.
• View count. The paper's DTU setup uses 49 to 64 views per scene and one scene per fit §4.4 §5.2. Behaviour with fewer views is not stated.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020) for volume rendering with alpha blending, and the occupancy and implicit-surface models of Mescheder et al. and Niemeyer et al. (DVR), with IDR (Yariv et al., 2020) as the surface renderer it extends §1 §3. No arXiv ids are given in the source, so none are linked.
• Related card: NeuS also turns a neural SDF into a volume-rendering weight to fit surfaces from posed images.
• Compared against: COLMAP (Schönberger et al., 2016), NeRF, IDR §5.1.
• Superseded by: not stated in this wiki's content.