--- kind: concept slug: multi-view-stereo title: Multi-view stereo summary: >- Estimating depth from calibrated photos by building a cost volume over depth hypotheses. Four variants: UniMVSNet (slice-plus-offset read-out, author-reported first on DTU); Pseudo-LiDAR++ (depth-trained stereo with sparse LiDAR correction, for 3D detection); Point-NeRF (MVS-seeded neural point cloud); MVSplat (one-pass Gaussians from a plane-sweep volume). tags: [3d-reconstruction, depth-estimation, multi-view, novel-view-synthesis, stereo-depth, 3d-object-detection] papers: ["2201.01501", "2201.08845", "2403.14627", "1906.06310"] related: [multi-view-reconstruction, monocular-depth-estimation, focal-loss, cost-volume, neural-radiance-field, feed-forward-reconstruction, gaussian-splatting, graph-label-propagation, object-detection] status: draft updated_at: 2026-10-31 --- ## Overview Given several photos of a scene with known camera poses, multi-view stereo (MVS) estimates a depth map for a reference view. Learning-based methods warp the features of the other views onto a set of depth hypotheses, build a 3D cost volume, regularise it, and read depth from the result [arxiv:2201.01501 §3.1] [arxiv:2403.14627 §3.1]. The read-out step is where the variants differ. Some papers take the expectation over the hypotheses (regression), and some take the best hypothesis (classification). Regression is indirect and can overfit, and classification cannot give a depth between two hypotheses [arxiv:2201.01501 §1]. MVS is also used inside novel-view synthesis, where the depth is only a means to a render. Point-NeRF uses an MVS-style network to place its first points [arxiv:2201.08845 §4.1], and MVSplat uses a cost volume to place its Gaussians [arxiv:2403.14627 §3.1]. In driving, a stereo depth map is turned into a point cloud for LiDAR-style 3D detectors [arxiv:1906.06310 §1]. The four variants below cover depth estimation, this detection use, and the two rendering uses. ## Variants ### Unified depth representation (UniMVSNet) Each pixel gets a target over the M hypotheses that is zero everywhere except in the slice that contains the ground truth. Inside that slice the target holds a proximity value between 0 and 1 [arxiv:2201.01501 §3.2]. Training uses a Unified Focal Loss, which extends focal loss to continuous targets and scales the error by the size of the positive target [arxiv:2201.01501 §3.3]. At inference the network picks the slice with the largest value and adds an offset inside it [arxiv:2201.01501 §3.2]. The network is a coarse-to-fine cost-volume pipeline with three stages [arxiv:2201.01501 §4]. Card: [Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation](/papers/2201.01501). The paper's ablation on DTU attributes its gain mostly to the representation and the loss. Adaptive aggregation and finer ground truth give smaller additional gains [arxiv:2201.01501 tab. 3]. These are author-reported results from one paper. ### Depth-trained stereo with sparse LiDAR correction (Pseudo-LiDAR++) A stereo network re-indexes its matching volume by depth instead of disparity, using 1 m bins from 1 m to 80 m, and regresses depth as the softmax-weighted sum of the bins. It is trained with a smooth-L1 loss on depth rather than on disparity [arxiv:1906.06310 §3]. The paper argues that a fixed disparity error costs much more depth at long range, so a disparity loss spends its effort on nearby objects [arxiv:1906.06310 §3] [arxiv:1906.06310 eq. 5]. A graph-based step then fixes depths at a few LiDAR points and solves the other depths so that each point stays a weighted sum of its nearest neighbours [arxiv:1906.06310 §4] [arxiv:1906.06310 alg. 1]. The corrected depth is back-projected to points, and 3D detectors trained from scratch on those points are the downstream task [arxiv:1906.06310 §5.1]. Card: [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](/papers/1906.06310). On KITTI validation, for cars with PointRCNN at moderate difficulty and IoU 0.7, AP_BEV rises from 56.0 for pseudo-LiDAR with PSMNet depth to 64.0 with the stereo network alone, and to 76.9 with the 4-beam correction [arxiv:1906.06310 tab. 1]. The 4-beam input is simulated by thinning 64-beam scans, not recorded with a 4-beam sensor [arxiv:1906.06310 §5.1]. Most ablations use PointRCNN, and the paper finds that the depth loss and the cost volume contribute differently across detectors [arxiv:1906.06310 §5.2] [arxiv:1906.06310 app. D.1]. These are author-reported results from one paper. ### MVS depth as the seed of a neural point field (Point-NeRF) For each reference view, an MVSNet-style network builds a plane-swept cost volume from the reference view and two neighbours, regresses a depth probability volume, and outputs depth as the probability-weighted sum of the plane depths. The depth map is unprojected to points, and each point's confidence is read from the probability volume [arxiv:2201.08845 §4.1]. The MVS part is pre-trained with ground-truth depth. The whole chain is then trained with a rendering loss only [arxiv:2201.08845 §5]. Per-scene optimisation then prunes low-confidence points and grows points into holes [arxiv:2201.08845 §4.2]. Card: [Point-NeRF: Point-based Neural Radiance Fields](/papers/2201.08845). Without per-scene optimisation, the paper's feed-forward result on DTU is below IBRNet and MVSNeRF (23.89 PSNR against 26.04 and 26.63) [arxiv:2201.08845 tab. 1]. Per-scene optimisation is what closes that gap, and the paper reports about 40 minutes for 20K steps on NeRF Synthetic [arxiv:2201.08845 §6.2]. ### Plane-sweep cost volume for feed-forward Gaussians (MVSplat) Two posed views are encoded at 1/4 resolution with a shallow CNN and a multi-view transformer [arxiv:2403.14627 §3.1]. For each view, the other view's features are warped onto 128 inverse-depth planes [arxiv:2403.14627 §4.1]. Each warped feature is correlated with the view, and the correlations are averaged over the other views to form the cost volume [arxiv:2403.14627 §3.1]. A U-Net refines the volume. A softmax over the planes gives a coarse depth as an expectation, and a second light U-Net adds a residual depth [arxiv:2403.14627 §3.1]. Each pixel's depth is unprojected to a Gaussian centre [arxiv:2403.14627 §3.2]. The model is trained with an L2 and LPIPS rendering loss, without depth supervision [arxiv:2403.14627 §3.3]. Card: [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](/papers/2403.14627). The paper reports 26.39 PSNR on RealEstate10K at 0.044 s per scene with 12.0M parameters, against 25.89 PSNR at 0.104 s for pixelSplat [arxiv:2403.14627 tab. 1]. Timing includes encoding and rendering, and the hardware is not stated [arxiv:2403.14627 tab. 1]. ## When to use which The rows follow the constraints each paper states. The Why column cites the paper's own numbers or design choices. | Situation | Use | Why | |---|---|---| | Depth maps are the output and you have ground-truth depth to train on | UniMVSNet | Unified target and Unified Focal Loss are built from ground-truth depth [arxiv:2201.01501 alg. 1]; reports first place on DTU Overall 0.315 mm [arxiv:2201.01501 tab. 1] | | Need a sub-hypothesis depth value without giving up a classifier-style head | UniMVSNet | The slice-plus-offset read-out is the paper's proposal [arxiv:2201.01501 §3.2] | | Render novel views from two or a few posed images, with no per-scene fitting | MVSplat | One forward pass of 0.044 s, author-reported [arxiv:2403.14627 tab. 1] | | No depth labels, only posed images | MVSplat | Trained with photometric losses only [arxiv:2403.14627 §3.3] | | Depth labels exist but the MVS stage cannot be pre-trained for your data | Check first | Point-NeRF pre-trains its MVS part with ground-truth depth [arxiv:2201.08845 §5] | | Per-scene optimisation budget is available and a good first point cloud helps | Point-NeRF | Quality rises from 30.71 PSNR at 20K steps to 33.31 at 200K on NeRF Synthetic [arxiv:2201.08845 tab. 2] | | Memory is tight | Reduce planes first, then check | At 128 planes and 1/4 resolution the cost volume is 128 × 64 × 64 per view at 256 × 256 [arxiv:2403.14627 §3.1] [arxiv:2403.14627 app. C]; the paper does not test other plane counts | | Stereo cameras for 3D object detection in driving, with LiDAR available for training data | Pseudo-LiDAR++ | The paper's main gap is for faraway objects [arxiv:1906.06310 §1]; depth ground truth is projected from 64-beam LiDAR [arxiv:1906.06310 §5.1] | | Stereo pair for detection and no LiDAR at run time | Pseudo-LiDAR++, stereo only | The stereo network alone reaches AP_BEV 64.0 against 56.0 for the baseline [arxiv:1906.06310 tab. 1]; LiDAR is still needed for training [arxiv:1906.06310 §5.1] | | A few LiDAR beams can be added at run time | Pseudo-LiDAR++ with correction | AP_BEV 76.9 in the author's setup [arxiv:1906.06310 tab. 1], but the beams are simulated [arxiv:1906.06310 §5.1] | | Need to compare with DTU or Tanks and Temples numbers | UniMVSNet | Those benchmark results are reported only in its tables [arxiv:2201.01501 tab. 1] [arxiv:2201.01501 tab. 2] | | Need to compare with KITTI detection numbers | Pseudo-LiDAR++ | Its AP_BEV and AP_3D results on KITTI validation and test are in its tables [arxiv:1906.06310 tab. 1] [arxiv:1906.06310 tab. 2] | | Need to reproduce exact DTU ground truth | Check first | The finer ground truth is described but not in the repository the card references [arxiv:2201.01501 app. B] | ## Implementation sketch The core is shared across the papers: warp, correlate, regularise, then read depth. The variants differ in the read-out, in what they supervise, and in what they do with the depth map afterwards. ```pseudocode # Shared core: warp, correlate, regularise, read depth # [arxiv:2201.01501 eq. 1] [arxiv:2403.14627 eq. 2-4] [arxiv:2201.08845 §4.1] def depth_from_cost_volume(ref_feat, src_feats, planes, cam): # Warp each source feature onto every depth plane [arxiv:2201.01501 eq. 1] warped = [warp(f, plane, cam) for f in src_feats for plane in planes] # Correlate with the reference and average over the source views # [arxiv:2403.14627 eq. 3-4] cost = mean(ref_feat * w / sqrt(C) for w in warped) # (D, H, W) # Regularise the volume with a 3D or 2D U-Net [arxiv:2201.08845 §4.1] cost = regularise(cost) prob = softmax(cost, over=planes) # Read-out, regression: expectation over the planes [arxiv:2403.14627 eq. 6] # Pseudo-LiDAR++ uses the same form over its depth bins [arxiv:1906.06310 §3] depth_reg = sum(prob * planes, over=planes) # Read-out, classification: best plane [arxiv:2201.01501 eq. 5] depth_cls = planes[argmax(prob, over=planes)] return depth_reg, depth_cls ``` Where the variants differ: - **Read-out.** Pseudo-LiDAR++, Point-NeRF and MVSplat use the regression form [arxiv:1906.06310 §3] [arxiv:2201.08845 §4.1] [arxiv:2403.14627 §3.1]. UniMVSNet replaces the probability volume with per-hypothesis sigmoids and decodes a slice and an offset [arxiv:2201.01501 §3.2]. - **Supervision.** UniMVSNet is trained on ground-truth depth [arxiv:2201.01501 alg. 1]. Pseudo-LiDAR++ is trained on depth projected from 64-beam LiDAR [arxiv:1906.06310 §5.1]. Point-NeRF uses depth for its MVS pre-training only [arxiv:2201.08845 §5]. MVSplat uses no depth [arxiv:2403.14627 §3.3]. - **Output.** UniMVSNet outputs a depth map [arxiv:2201.01501 alg. 2]. Pseudo-LiDAR++ back-projects its corrected depth map to points for a 3D detector [arxiv:1906.06310 §5.1]. Point-NeRF unprojects depth to points for a neural point field [arxiv:2201.08845 §4.1]. MVSplat unprojects depth to Gaussian centres [arxiv:2403.14627 §3.2]. - **Post-processing.** Pseudo-LiDAR++ adds a sparse-anchor correction after the network. It is a solve over the point cloud with no learned parameters [arxiv:1906.06310 §4]. None of the other variants has this step. The UniMVSNet target and decoder are given in full below. ```pseudocode # Build the unified target for one pixel from its ground-truth depth # [arxiv:2201.01501 alg. 1] def make_target(depth_gt, hyps): # hyps: M increasing depth values; slice i spans [hyps[i], hyps[i+1]) target = zeros(M) for i in range(M - 1): lo, width = hyps[i], hyps[i + 1] - hyps[i] # Only the slice that contains the ground truth is non-zero if lo <= depth_gt < lo + width: # Proximity: 1 at the lower edge, falling to 0 at the top target[i] = 1 - (depth_gt - lo) / width return target ``` ```pseudocode # Decode depth from the network's per-slice outputs in (0, 1) # [arxiv:2201.01501 alg. 2] def decode(unity, hyps): # Pick the slice with the largest output i = argmax(unity) width = hyps[i + 1] - hyps[i] # The offset is the complement of the output, scaled by the slice width return hyps[i] + (1 - unity[i]) * width ``` ```pseudocode # Per-stage loss sketch [arxiv:2201.01501 §3.3] [arxiv:2201.01501 eq. 9] def stage_loss(unity_hat, target, gamma, alpha_neg): # Positive entries are weighted by their relative error # Negative entries are weighted by their predicted value # Both are focal-scaled by gamma; negatives also get alpha_neg # Exact scale functions are in the paper's appendix [app. A] ... ``` Two details in the loss are not fully specified in the paper: how the per-pixel loss is averaged over hypotheses, and which normaliser to use when no slice contains the ground truth [arxiv:2201.01501 §3.4] [arxiv:2201.01501 eq. 10]. ## Contradictions No two papers report incompatible numbers for the same setup. The four evaluate different tasks: depth on DTU and Tanks and Temples [arxiv:2201.01501 tab. 1] [arxiv:2201.01501 tab. 2], 3D detection on KITTI [arxiv:1906.06310 tab. 1], novel-view synthesis on NeRF Synthetic [arxiv:2201.08845 tab. 2], and novel-view synthesis on RealEstate10K [arxiv:2403.14627 tab. 1]. Their numbers are not comparable. Whether depth supervision helps is open. Pseudo-LiDAR++ trains its stereo network on depth and reports a gain for PointRCNN from the depth loss [arxiv:1906.06310 tab. 3]. Point-NeRF pre-trains its MVS part with ground-truth depth [arxiv:2201.08845 §5]. MVSplat trains without any depth and reports that reflective surfaces such as glass and windows are unreliable [arxiv:2403.14627 §3.3] [arxiv:2403.14627 §5]. The cards do not compare these choices on one setup. The read-out is also disputed. UniMVSNet argues that regression is indirect and can overfit, and that classification cannot give a depth between two hypotheses [arxiv:2201.01501 §1]. Pseudo-LiDAR++ and MVSplat use regression [arxiv:1906.06310 §3] [arxiv:2403.14627 §3.1]. The cards report no head-to-head test of the two read-outs on one setup. ## Papers - [Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation](/papers/2201.01501): unified depth representation with a Unified Focal Loss. Reports first place on DTU and Tanks and Temples. - [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](/papers/1906.06310): depth-trained stereo network, then a graph-based correction from a few LiDAR points, feeding 3D detectors. - [Point-NeRF: Point-based Neural Radiance Fields](/papers/2201.08845): MVS-style network seeds a neural point cloud, refined per scene with pruning and growing. - [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](/papers/2403.14627): plane-sweep cost volume predicts per-pixel Gaussians from two posed views in one pass.