Multi-view stereo

Overview

Given several photos of a scene with known camera poses, multi-view stereo (MVS) estimates a depth map for a reference view. Learning-based methods warp the features of the other views onto a set of depth hypotheses, build a 3D cost volume, regularise it, and read depth from the result 2201.01501 §3.1 2403.14627 §3.1. The read-out step is where the variants differ. Some papers take the expectation over the hypotheses (regression), and some take the best hypothesis (classification). Regression is indirect and can overfit, and classification cannot give a depth between two hypotheses 2201.01501 §1.

MVS is also used inside novel-view synthesis, where the depth is only a means to a render. Point-NeRF uses an MVS-style network to place its first points 2201.08845 §4.1, and MVSplat uses a cost volume to place its Gaussians 2403.14627 §3.1. In driving, a stereo depth map is turned into a point cloud for LiDAR-style 3D detectors 1906.06310 §1. The four variants below cover depth estimation, this detection use, and the two rendering uses.

Variants

Unified depth representation (UniMVSNet)

Each pixel gets a target over the M hypotheses that is zero everywhere except in the slice that contains the ground truth. Inside that slice the target holds a proximity value between 0 and 1 2201.01501 §3.2. Training uses a Unified Focal Loss, which extends focal loss to continuous targets and scales the error by the size of the positive target 2201.01501 §3.3. At inference the network picks the slice with the largest value and adds an offset inside it 2201.01501 §3.2. The network is a coarse-to-fine cost-volume pipeline with three stages 2201.01501 §4. Card: Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation.

The paper's ablation on DTU attributes its gain mostly to the representation and the loss. Adaptive aggregation and finer ground truth give smaller additional gains 2201.01501 tab. 3. These are author-reported results from one paper.

Depth-trained stereo with sparse LiDAR correction (Pseudo-LiDAR++)

A stereo network re-indexes its matching volume by depth instead of disparity, using 1 m bins from 1 m to 80 m, and regresses depth as the softmax-weighted sum of the bins. It is trained with a smooth-L1 loss on depth rather than on disparity 1906.06310 §3. The paper argues that a fixed disparity error costs much more depth at long range, so a disparity loss spends its effort on nearby objects 1906.06310 §3 1906.06310 eq. 5. A graph-based step then fixes depths at a few LiDAR points and solves the other depths so that each point stays a weighted sum of its nearest neighbours 1906.06310 §4 1906.06310 alg. 1. The corrected depth is back-projected to points, and 3D detectors trained from scratch on those points are the downstream task 1906.06310 §5.1. Card: Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving.

On KITTI validation, for cars with PointRCNN at moderate difficulty and IoU 0.7, AP_BEV rises from 56.0 for pseudo-LiDAR with PSMNet depth to 64.0 with the stereo network alone, and to 76.9 with the 4-beam correction 1906.06310 tab. 1. The 4-beam input is simulated by thinning 64-beam scans, not recorded with a 4-beam sensor 1906.06310 §5.1. Most ablations use PointRCNN, and the paper finds that the depth loss and the cost volume contribute differently across detectors 1906.06310 §5.2 1906.06310 app. D.1. These are author-reported results from one paper.

MVS depth as the seed of a neural point field (Point-NeRF)

For each reference view, an MVSNet-style network builds a plane-swept cost volume from the reference view and two neighbours, regresses a depth probability volume, and outputs depth as the probability-weighted sum of the plane depths. The depth map is unprojected to points, and each point's confidence is read from the probability volume 2201.08845 §4.1. The MVS part is pre-trained with ground-truth depth. The whole chain is then trained with a rendering loss only 2201.08845 §5. Per-scene optimisation then prunes low-confidence points and grows points into holes 2201.08845 §4.2. Card: Point-NeRF: Point-based Neural Radiance Fields.

Without per-scene optimisation, the paper's feed-forward result on DTU is below IBRNet and MVSNeRF (23.89 PSNR against 26.04 and 26.63) 2201.08845 tab. 1. Per-scene optimisation is what closes that gap, and the paper reports about 40 minutes for 20K steps on NeRF Synthetic 2201.08845 §6.2.

Plane-sweep cost volume for feed-forward Gaussians (MVSplat)

Two posed views are encoded at 1/4 resolution with a shallow CNN and a multi-view transformer 2403.14627 §3.1. For each view, the other view's features are warped onto 128 inverse-depth planes [arxiv:2403.14627 §4.1]. Each warped feature is correlated with the view, and the correlations are averaged over the other views to form the cost volume [arxiv:2403.14627 §3.1]. A U-Net refines the volume. A softmax over the planes gives a coarse depth as an expectation, and a second light U-Net adds a residual depth 2403.14627 §3.1. Each pixel's depth is unprojected to a Gaussian centre 2403.14627 §3.2. The model is trained with an L2 and LPIPS rendering loss, without depth supervision 2403.14627 §3.3. Card: MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images.

The paper reports 26.39 PSNR on RealEstate10K at 0.044 s per scene with 12.0M parameters, against 25.89 PSNR at 0.104 s for pixelSplat 2403.14627 tab. 1. Timing includes encoding and rendering, and the hardware is not stated 2403.14627 tab. 1.

When to use which

The rows follow the constraints each paper states. The Why column cites the paper's own numbers or design choices.

Implementation sketch

The core is shared across the papers: warp, correlate, regularise, then read depth. The variants differ in the read-out, in what they supervise, and in what they do with the depth map afterwards.

# Shared core: warp, correlate, regularise, read depth
# [arxiv:2201.01501 eq. 1] [arxiv:2403.14627 eq. 2-4] [arxiv:2201.08845 §4.1]

def depth_from_cost_volume(ref_feat, src_feats, planes, cam):
    # Warp each source feature onto every depth plane  [arxiv:2201.01501 eq. 1]
    warped = [warp(f, plane, cam) for f in src_feats for plane in planes]
    # Correlate with the reference and average over the source views
    # [arxiv:2403.14627 eq. 3-4]
    cost = mean(ref_feat * w / sqrt(C) for w in warped)  # (D, H, W)
    # Regularise the volume with a 3D or 2D U-Net  [arxiv:2201.08845 §4.1]
    cost = regularise(cost)
    prob = softmax(cost, over=planes)
    # Read-out, regression: expectation over the planes  [arxiv:2403.14627 eq. 6]
    # Pseudo-LiDAR++ uses the same form over its depth bins  [arxiv:1906.06310 §3]
    depth_reg = sum(prob * planes, over=planes)
    # Read-out, classification: best plane  [arxiv:2201.01501 eq. 5]
    depth_cls = planes[argmax(prob, over=planes)]
    return depth_reg, depth_cls

Where the variants differ:

• Read-out. Pseudo-LiDAR++, Point-NeRF and MVSplat use the regression form 1906.06310 §3 2201.08845 §4.1 2403.14627 §3.1. UniMVSNet replaces the probability volume with per-hypothesis sigmoids and decodes a slice and an offset 2201.01501 §3.2.
• Supervision. UniMVSNet is trained on ground-truth depth [arxiv:2201.01501 alg. 1]. Pseudo-LiDAR++ is trained on depth projected from 64-beam LiDAR 1906.06310 §5.1. Point-NeRF uses depth for its MVS pre-training only 2201.08845 §5. MVSplat uses no depth 2403.14627 §3.3.
• Output. UniMVSNet outputs a depth map 2201.01501 alg. 2. Pseudo-LiDAR++ back-projects its corrected depth map to points for a 3D detector 1906.06310 §5.1. Point-NeRF unprojects depth to points for a neural point field 2201.08845 §4.1. MVSplat unprojects depth to Gaussian centres 2403.14627 §3.2.
• Post-processing. Pseudo-LiDAR++ adds a sparse-anchor correction after the network. It is a solve over the point cloud with no learned parameters 1906.06310 §4. None of the other variants has this step.

The UniMVSNet target and decoder are given in full below.

# Build the unified target for one pixel from its ground-truth depth
# [arxiv:2201.01501 alg. 1]
def make_target(depth_gt, hyps):
    # hyps: M increasing depth values; slice i spans [hyps[i], hyps[i+1])
    target = zeros(M)
    for i in range(M - 1):
        lo, width = hyps[i], hyps[i + 1] - hyps[i]
        # Only the slice that contains the ground truth is non-zero
        if lo <= depth_gt < lo + width:
            # Proximity: 1 at the lower edge, falling to 0 at the top
            target[i] = 1 - (depth_gt - lo) / width
    return target

# Decode depth from the network's per-slice outputs in (0, 1)
# [arxiv:2201.01501 alg. 2]
def decode(unity, hyps):
    # Pick the slice with the largest output
    i = argmax(unity)
    width = hyps[i + 1] - hyps[i]
    # The offset is the complement of the output, scaled by the slice width
    return hyps[i] + (1 - unity[i]) * width

# Per-stage loss sketch [arxiv:2201.01501 §3.3] [arxiv:2201.01501 eq. 9]
def stage_loss(unity_hat, target, gamma, alpha_neg):
    # Positive entries are weighted by their relative error
    # Negative entries are weighted by their predicted value
    # Both are focal-scaled by gamma; negatives also get alpha_neg
    # Exact scale functions are in the paper's appendix [app. A]
    ...

Two details in the loss are not fully specified in the paper: how the per-pixel loss is averaged over hypotheses, and which normaliser to use when no slice contains the ground truth 2201.01501 §3.4 2201.01501 eq. 10.

Contradictions

No two papers report incompatible numbers for the same setup. The four evaluate different tasks: depth on DTU and Tanks and Temples 2201.01501 tab. 1 2201.01501 tab. 2, 3D detection on KITTI 1906.06310 tab. 1, novel-view synthesis on NeRF Synthetic 2201.08845 tab. 2, and novel-view synthesis on RealEstate10K 2403.14627 tab. 1. Their numbers are not comparable.

Whether depth supervision helps is open. Pseudo-LiDAR++ trains its stereo network on depth and reports a gain for PointRCNN from the depth loss 1906.06310 tab. 3. Point-NeRF pre-trains its MVS part with ground-truth depth 2201.08845 §5. MVSplat trains without any depth and reports that reflective surfaces such as glass and windows are unreliable 2403.14627 §3.3 2403.14627 §5. The cards do not compare these choices on one setup.

The read-out is also disputed. UniMVSNet argues that regression is indirect and can overfit, and that classification cannot give a depth between two hypotheses 2201.01501 §1. Pseudo-LiDAR++ and MVSplat use regression 1906.06310 §3 2403.14627 §3.1. The cards report no head-to-head test of the two read-outs on one setup.

Papers

• Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation: unified depth representation with a Unified Focal Loss. Reports first place on DTU and Tanks and Temples.
• Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving: depth-trained stereo network, then a graph-based correction from a few LiDAR points, feeding 3D detectors.
• Point-NeRF: Point-based Neural Radiance Fields: MVS-style network seeds a neural point cloud, refined per scene with pruning and growing.
• MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images: plane-sweep cost volume predicts per-pixel Gaussians from two posed views in one pass.