Multi-view reconstruction

Overview

Given images taken from known camera poses, the goal is a 3D surface of the object or scene. The usual recipe is to fit a scene representation so that rendering it reproduces the input photos, then read the surface out of that representation. The fitting variants train with a photometric loss against the input images 2106.10689 §3.2 2104.10078 §4.2 2403.17888 §5. The two One-2-3-45 variants are a looser fit for this page: they take one photo, they fit no per-scene representation, and they do not need the camera poses to be given. One-2-3-45 does estimate the input's elevation from the photo, and needs no per-shape optimisation 2306.16928 §1 §3.4.

The variants differ in what the representation is:

• Neural implicit surface with volume rendering. A network gives a value at every point, a signed distance (NeuS) or an occupancy (UNISURF), and the surface is read from it. NeuS is the card for the signed-distance form 2106.10689 §3.1. UNISURF is the card for the occupancy form 2104.10078 §3.
• Explicit oriented primitives. The scene is a set of flat 2D Gaussian disks, each with a normal, rendered by exact ray-disk intersection. 2D Gaussian Splatting is the card here [arxiv:2403.17888 §4.1] §4.2.
• Generated views with a feed-forward SDF network. A diffusion model draws 32 posed views from one photo, and a cost-volume network predicts a signed distance and colour in one pass. One-2-3-45 is the card here 2306.16928 §3.3 §4.1.
• Generated views and volume diffusion. A diffusion model draws six views from one photo, and a second diffusion model predicts a signed distance and colour volume from them. One-2-3-45++ is the card here 2311.07885 §3.
• Fragment volumes from posed video. A network predicts a sparse signed distance volume for each fragment of 9 key frames, and a recurrent cell fuses each fragment with the earlier ones. NeuralRecon is the card here 2104.00681 §3.1 §3.2. Its input is a video, and its camera poses come from an external SLAM system 2104.00681 §3.

A further route, rendering a density field with plain NeRF and thresholding it, is covered only as a baseline in the NeuS card: its level-set meshes are noisy 2106.10689 §1 §4.2.

One variant does not reconstruct an unknown surface at all. It tracks a person and a rigid object whose mesh is known in advance, from several calibrated RGBD cameras, and estimates only the pose of each 2204.06950 §3 §4. It is listed here because it shares the multi-view setting, not because it recovers geometry.

One variant is not learned at all. Classical bundle block adjustment, tuned for UAV surveys, estimates camera poses and tie points from the images and is judged against surveyed check points rather than a rendered mesh 2610.01438 §2.2 §2.3. It is a survey-accuracy route, not a surface route, and its numbers cannot be ranked against the Chamfer and F-score figures below.

Variants

Neural SDF with unbiased volume rendering (NeuS)

The representation is an MLP signed distance f and a view-dependent color MLP. The surface is the zero level set of f 2106.10689 §3.1 eq. 1. Naively converting the SDF into NeRF density puts the peak of the rendering weight in front of the surface, which biases the geometry. NeuS instead builds an opaque density from a logistic CDF of f, so the weight peaks on the surface for locally planar patches and occluded surfaces are still hidden 2106.10689 §3.1 app. B. The author-reported result on 15 DTU scans is mean Chamfer 0.77 with object masks and 0.84 without 2106.10689 tab. 1. Card: NeuS.

Cost: about 14 h (with masks) or 16 h (without) on one RTX 2080 Ti for 300k iterations 2106.10689 §4.1 app. D.1. Rendering one 1600×1200 image takes about 320 s with default sampling 2106.10689 app. E.

Neural occupancy field with one rendering formula (UNISURF)

The representation is an occupancy MLP o_\theta with values in [0,1], and the surface is the set where o_\theta = 0.5 2104.10078 §3 eq. 1. Each ray is rendered two ways with one formula. Surface rendering colours the first crossing with the surface, found by root-finding. Volume rendering blends colours over samples in a window around that crossing, with occupancy standing in for alpha 2104.10078 §4.1 eq. 7 eq. 8. The window's half-width shrinks exponentially during training, from a wide start to a narrow end, so the model can fit a rough shape without masks at first 2104.10078 §4.3 eq. 14. Normals come from the gradient of the occupancy field 2104.10078 §4.2 eq. 12.

The author-reported DTU mean Chamfer is 1.02 with no masks and no trimming, against 0.90 for IDR with masks. Among the methods without masks or trimming, UNISURF is lowest on 12 of 15 scans 2104.10078 tab. 1. The authors note that COLMAP with trim 7 has a lower mean (0.65), but its meshes are not watertight 2104.10078 §5.3. Card: UNISURF.

The paper states no training time. For one view, the authors report 178 s for volume rendering and 84 s for surface rendering on one V100 [arxiv:2104.10078 fig. 5]. The formulation assumes solid objects with occupancy in {0, 1}, so it cannot represent transparent or semi-transparent material [arxiv:2104.10078 §4.1] §6.

Explicit oriented Gaussian disks (2DGS)

Each primitive is a flat 2D Gaussian with a center, two tangent vectors and two scales, so it has a well-defined normal 2403.17888 §4.1. Pixels are shaded by intersecting the pixel ray with the disk plane in closed form, which stays exact at grazing angles 2403.17888 §4.2. Two extra losses shape the geometry: a depth distortion loss that pulls splats together along each ray, and a normal consistency loss that aligns splat normals with the normal of the rendered depth map 2403.17888 §5. The mesh is a TSDF fusion of the rendered median depth maps 2403.17888 §6.1. The author-reported DTU mean Chamfer is 0.80 after 30k iterations, 10.9 min on one RTX 3090, against 1.96 for 3D Gaussian Splatting 2403.17888 tab. 1. Card: 2D Gaussian Splatting. It is also listed under the Gaussian splatting concept.

Image quality is slightly below 3DGS: Mip-NeRF 360 outdoor PSNR 24.34 against 24.64 2403.17888 tab. 4.

Generated views with a feed-forward SDF network (One-2-3-45)

The input is one photo, and nothing is fitted per shape. Zero123 is used frozen to render 8 anchor views around the object, and each anchor gets 4 nearby views 10 degrees apart, so 32 source views in total 2306.16928 §4.1. The camera elevation of the photo is not given. It is estimated from the four nearby views of the first anchor: features are matched with LoFTR, and the candidate elevation with the smallest reprojection error is kept. The estimate takes under a second per shape 2306.16928 §3.4 app. elevation. On 1,700 random renders the author-reported median elevation error is 5.4 degrees 2306.16928 §4.3 fig. 7. A cost-volume network based on SparseNeuS then predicts an SDF and a colour field in one pass, with an SDF-based volume rendering step that follows NeuS 2306.16928 §3.3. Marching cubes gives the mesh 2306.16928 §3.3. Card: One-2-3-45.

Author-reported F-score 83.5 and CLIP similarity 78.1, averaged over 20 Objaverse and 20 GSO shapes, in about 45 s on one A100 [arxiv:2306.16928 tab. 1] §4.2. About 5 s of that is the reconstruction module [arxiv:2306.16928 §4.2]. The training data is the LVIS subset of Objaverse, 46k models 2306.16928 §4.1 app. training. The reconstruction module is trained on Zero123 predictions as source views and ground-truth renders as targets, and the authors call mixing ground-truth and predicted views during training essential 2306.16928 §3.3 §4.3.

Generated views and volume diffusion (One-2-3-45++)

The input is one RGB image, and nothing is fitted per scene. A fine-tuned Stable Diffusion 2 draws six views at fixed poses, tiled into one image 2311.07885 §3.1 fig. 3. Two 3D diffusion stages then predict the shape: a 64³ occupancy shell, with each occupied voxel split into eight to form a 128³ sparse grid that holds an SDF and a colour per voxel [arxiv:2311.07885 §3.2]. Image features are projected onto each voxel from the generated views 2311.07885 §3.2 fig. 4. The 3D stages are trained on rendered views, not on generated ones, and the paper reports that training on generated views lowers 3D IoU 2311.07885 §3.2 tab. 4. A colour field is then fitted to the views with the geometry fixed 2311.07885 §3.3. Card: One-2-3-45++.

Author-reported F-Score 93.6 and CLIP similarity 81.0 on 1,030 GSO shapes, in about 60 s 2311.07885 tab. 1. These are not Chamfer distances on DTU, so they cannot be ranked against the numbers above. Training used 16 GPUs for about 10 days for the 2D stage and 8 A100 GPUs for about 10 days per 3D stage 2311.07885 §3.1 §3.2.

Fragment volumes from posed video (NeuralRecon)

The input is a monocular video, and the camera poses are taken as given from an external SLAM system 2104.00681 §3. A key frame is taken when the camera has moved and turned past thresholds, and each window of 9 key frames forms a fragment 2104.00681 §3.1 §4.3. Image features are back-projected into a sparse 3D grid for the fragment. Three levels of sparse 3D convolution then predict an SDF and an occupancy per voxel, and voxels judged empty are removed 2104.00681 §3.2 §3.3. A ConvGRU fuses each fragment with a hidden state from earlier fragments, in place of running-average TSDF fusion 2104.00681 §3.2 §4.3. Marching Cubes runs on the global volume at every step 2104.00681 §3.2. Nothing is fitted per scene. Card: NeuralRecon.

Author-reported: 30 ms per key frame on one RTX 2080 Ti 2104.00681 §4.2 tab. 4, about 10 times faster than Atlas §4.2. On ScanNet with the Atlas split, its single-layer F-score is 0.562, against 0.558 for COLMAP 2104.00681 tab. 1. Its double-layer F-score is 0.516 tab. 1.

Known object template from multi-view RGBD (BEHAVE)

The input is depth and colour from four Kinect cameras placed around a recording volume 2204.06950 §3. Each frame is processed on its own 2204.06950 §4. The points of the person and of the object are voxelised, and a 3D CNN encodes them into a feature grid. At each query point, three decoders predict an unsigned distance to the human and to the object surface, a correspondence onto the SMPL body model, and a nine-value object orientation 2204.06950 §4.1 §4.2. A per-frame fit then moves the body model and the object's rigid pose. It minimises a sum of three terms: the body fit, the object fit, and a contact term that pulls object vertices onto the body where both predicted distances are small 2204.06950 eq. 1 §4.3. No weights are given between these terms eq. 1. Card: BEHAVE.

Author-reported vertex errors on the BEHAVE recordings are 4.99 cm for the body and 21.20 cm for the object, against 13.73 cm and 34.73 cm for PHOSA, a single-image method 2204.06950 tab. 2. The paper reports no runtime, and the training schedule and optimiser are not stated 2204.06950 §4 §5.3. The dataset and the code are both non-commercial (card). The card's Pitfalls section records the dataset's licence terms.

Classical bundle adjustment with tuned settings (UAV photogrammetry)

No learned model is involved. The study runs commercial photogrammetry software (Agisoft Metashape) on ten UAV surveys of one 220 ha site, and varies eight processing settings across all 768 combinations 2610.01438 §2.1 §2.2 tab. 1. Each combination runs at least two full bundle block adjustments. The second run is preceded by removing images with clearly wrong exterior orientation, and each variant is ranked by the 3D RMSE at check points 2610.01438 §2.2 §2.3. The factors with the largest effect are the additional self-calibration terms, at least five ground control points, and camera positions from PPK, post-processing against the team's own static base station 2610.01438 §2.2 §3.1 fig. 6. Card: The Impact of Processing Parameters on High-Accuracy Measurements in UAV Photogrammetry.

Author-reported: the best variant reaches 16 mm 3D RMSE at check points, and the worst reaches 303 mm 2610.01438 abstract. Every variant in the top 10 % had self-calibration on and more than one ground control point, and PPK was used in over 90 % of them 2610.01438 §3.1. With the tuned settings, the median absolute systematic vertical error of the deformation pipeline falls from 14 mm to 7 mm, and the random error stays at 6 to 7 mm 2610.01438 §3.2 tab. 5.

When to use which

Implementation sketch

The three fitting variants share a loop. They differ in the render step and the surface read-out. Shapes and sampling counts are omitted. The two generated-view variants and NeuralRecon do not use this loop. The One-2-3-45++ one is sketched after the settings below.

# Common loop for both variants  [arxiv:2106.10689 §3.2] [arxiv:2403.17888 §5]
# Input: posed images, camera intrinsics and extrinsics
# Output: a surface (mesh) read from the trained representation

def train(images, cameras, steps):
    # representation: SDF MLP (NeuS), occupancy MLP (UNISURF) or splat set (2DGS)
    model = init_representation(cameras)
    for step in range(steps):
        # pick one image and a batch of its pixels  [arxiv:2106.10689 §3.2]
        # [arxiv:2104.10078 §4.4]
        view, target = sample_view(images, cameras)
        # render colour plus a geometric quantity for each pixel
        # NeuS: volume-rendered colour and opacity, Eikonal term
        # UNISURF: surface or volume colour over a sampling window
        # 2DGS: alpha-blended colour, depth, normal, distortion
        pred = render(model, view)
        # colour loss plus the variant's geometry losses
        loss = colour_loss(pred, target) + geometry_losses(model, pred)
        update(model, loss)
    return model

def extract_mesh(model, cameras):
    # NeuS: marching cubes on the zero level set of the SDF
    #       [arxiv:2106.10689 §3.1] [code: models/renderer.py#L380-L385]
    # UNISURF: marching cubes on occupancy 0.5, from MISE  [arxiv:2104.10078 §4.4]
    # 2DGS: render median depth for all views, then TSDF fusion
    #       voxel 0.004, truncation 0.02  [arxiv:2403.17888 §6.1]
    if model.kind == "sdf":
        return marching_cubes(model.sdf, level=0)
    if model.kind == "occupancy":
        return marching_cubes(model.occupancy, level=0.5)
    depths = [render_median_depth(model, c) for c in cameras]
    return tsdf_fuse(depths, cameras, voxel=0.004, trunc=0.02)

Practical settings that differ by variant:

• NeuS: 300k iterations 2106.10689 §4.1, 512 rays per batch, 128 samples per ray 2106.10689 app. D.1. The Eikonal and mask weights are not stated in the paper 2106.10689 eq. 14.
• UNISURF: 450k iterations, 1024 pixels per iteration, 64 samples inside the window and 32 in free space, 8-layer occupancy network of width 256 2104.10078 §4.4 eq. 13. The window starts at half-width 1.0 and ends at 0.05, with decay rate 1.5e-5 2104.10078 §4.4. The released DTU configs use 2.0 and 0.1 instead scan0114.yaml, and the paper does not say which one produced Table 1. The regulariser weight \lambda is not stated in the paper 2104.10078 eq. 9.
• 2DGS: 30k iterations, depth-distortion weight 1000 for bounded scenes, normal weight 0.05 2403.17888 §5. The distortion and normal losses start at iterations 3000 and 7000 in the released code. The paper does not state this warm-up (see the card's Paper vs code section).
• One-2-3-45: 8 anchor views with 4 nearby views each, 10 degrees apart 2306.16928 §4.1 app. elevation. The elevation search is coarse in 10-degree steps, then fine in 1-degree steps within ±10 degrees app. elevation. Training is 300k iterations in the paper and 200k in the released config, with Eikonal weight 0.1 and sparsity weight 0.02 2306.16928 app. training one2345_lod_train.conf. Training took about 6 days on 2 A10 GPUs 2306.16928 app. training.
• One-2-3-45++: six views, a 64³ occupancy grid, then a 128³ sparse grid with eight children per occupied voxel 2311.07885 §3.1 §3.2. The sampler and step count, guidance scale and occupancy threshold are not stated 2311.07885 §3.2.
• NeuralRecon: fragments of 9 key frames, 3 levels, 4 cm finest voxel and 12 cm truncation 2104.00681 §4.3 §3.3. Its key-frame test uses 0.1 m and 15 degrees, joined by "and" in the paper 2104.00681 §3.1 §3.3. The coarse voxel sizes, level loss weights and optimiser are not stated in the paper 2104.00681 §3.2 §3.3. The NeuralRecon fragment loop does not fit the sketch above, so it is left to its card.
• Classical bundle adjustment (UAV survey): the first pass uses AC off, tie-point accuracy 1 px, camera accuracy 10 degrees and marker accuracy 0.5 px, and the final pass uses the enhanced configuration of 2610.01438 §3.2. Its settings are Metashape names, and the tie-point filtration thresholds in fig. 2 are not a general rule 2610.01438 §2.2. The paper's pipeline is a sequence of adjustments and filtering, not the sketch above, so it is left to its card.

# One-2-3-45++ inference only  [arxiv:2311.07885 §3.1] [§3.2] [§3.3]
# Input: one RGB image; output: textured mesh. No per-scene fitting.

def image_to_mesh(image):
    # Six views at fixed poses, tiled into one image  [§3.1]
    views = split_tile(generate_multi_view(image))
    # Stage 1: denoise a 64^3 occupancy shell from Gaussian noise  [§3.2]
    occupancy = denoise_occupancy(noise_64, views)
    # Split each occupied voxel into 8 for a 128^3 sparse grid  [§3.2]
    sparse_grid = subdivide_occupied(occupancy)
    # Stage 2: denoise SDF and colour on the sparse grid  [§3.2]
    sdf, colour = denoise_sparse(sparse_grid, views)
    mesh = marching_cubes(sdf, colour)             # [§3.2]
    # Fit a colour field to the views with the geometry fixed  [§3.3]
    return refine_texture(mesh, views)

Contradictions

• The benchmarks differ across the variants. NeuS, UNISURF and 2DGS report DTU Chamfer distance 2106.10689 tab. 1 2104.10078 tab. 1 2403.17888 tab. 1. One-2-3-45 reports F-score and CLIP similarity on Objaverse and GSO 2306.16928 tab. 1, and so does One-2-3-45++ 2311.07885 tab. 1. NeuralRecon reports a 3D F-score on ScanNet 2104.00681 tab. 1. The UAV study reports 3D RMSE at surveyed check points 2610.01438 §2.3. BEHAVE reports vertex error against pseudo-ground-truth body and object fits on its own recordings 2204.06950 tab. 2 §3. None of these numbers measures the others.
• Tie-point filtration has no measured gain for the best settings, but its guidance recommends it. The paper finds that the filtration step has no significant effect on the best variants, yet it cites USGS guidance that recommends the step. The enhanced configuration turns it off 2610.01438 §3.2 §4 fig. 6. The paper does not resolve this, so this page does not either.
• NeuralRecon compares single-layer with double-layer meshes. Its headline F-score, 0.562, is single-layer, and the Atlas comparison sets it against Atlas's double-layer 0.499. Its own double-layer result is 0.516 2104.00681 tab. 1 §4.1. The ablation table names no layer setting, and its best row is also 0.516 2104.00681 tab. 5.
• One-2-3-45 has two sets of numbers. The One-2-3-45 paper reports F-score 83.5 and CLIP similarity 78.1 over 20 Objaverse and 20 GSO shapes, at an F-score threshold of 0.05 2306.16928 tab. 1 §4.2. The One-2-3-45++ paper reports One-2-3-45 at F-score 90.4 and CLIP similarity 70.8 on 1,030 GSO shapes, with the F-score threshold not stated 2311.07885 tab. 1 §4.1. The datasets and metrics differ, so neither set can be checked against the other.
• One-2-3-45 depth loss. The paper uses a depth weight of 1 and calls depth supervision essential 2306.16928 app. training §4.3. The released training config sets it to 0.0 one2345_lod_train.conf. The paper does not say which setting produced its model, so this page does not resolve it.
• One-2-3-45++ text and its own table disagree. The text says it surpasses other methods in PSNR, LPIPS and mask IoU, but its table has SyncDreamer with lower LPIPS (0.095 against 0.110) and higher mask IoU (0.894 against 0.878) 2311.07885 §4.3 tab. 5.
• The NeuS mean is reported at two values. The NeuS card gives 0.77 with masks and 0.84 without 2106.10689 tab. 1. The 2DGS card lists NeuS at 0.84 and does not state the mask setting 2403.17888 tab. 1. Its per-scan NeuS values match the NeuS paper's no-mask column, so the 0.84 is most likely the no-mask figure, but the 2DGS paper does not say so.
• Resolution differs between the two reports. NeuS evaluates on DTU at 1600×1200 2106.10689 §4.1. 2DGS evaluates at 800×600 2403.17888 §6.2. The Chamfer numbers are therefore not measured on identical inputs.
• Depth used for the 2DGS normal loss. The paper uses the median depth 2403.17888 §5. The authors' code defaults to expected depth for unbounded scenes, and the paper does not mention this switch 2403.17888 app. B.

Papers

• NeuS: unbiased volume rendering for an SDF, with an opaque density derived from a logistic CDF.
• UNISURF: a neural occupancy field rendered by one formula for surface and volume, without object masks.
• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields: oriented 2D Gaussian disks with exact ray-splat intersection, depth-distortion and normal losses, TSDF meshing.
• One-2-3-45: one photo to a 360-degree textured mesh through 32 Zero123 views, an estimated input elevation and a feed-forward SDF cost-volume network.
• One-2-3-45++: one image to a textured mesh through six generated views and two sparse-volume diffusion stages.
• NeuralRecon: real-time TSDF reconstruction from monocular video, one fragment of 9 key frames at a time, with a ConvGRU that fuses fragments.
• BEHAVE: Dataset and Method for Tracking Human Object Interactions: per-frame tracking of a person and a known object from four RGBD cameras, with a learned distance and correspondence field and a contact term.
• The Impact of Processing Parameters on High-Accuracy Measurements in UAV Photogrammetry: 768 bundle-adjustment settings variants on ten UAV surveys, ranked by 3D RMSE at check points, with a deformation pipeline on top.