MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images

In plain words

Turning a few photos of a place into a 3D scene you can view from new angles usually means fitting a model to that one scene, which is slow. MVSplat is a neural network that skips the fitting. You give it two photos taken from known camera positions, and in one pass it outputs a 3D point cloud of small soft blobs (called Gaussians) that can be drawn from any new viewpoint.

The core idea is old-fashioned stereo vision, built into the network. For each pixel, the model tests many possible distances from the camera and checks how well the two photos agree at each distance. Where they agree, there is probably a surface, so the pixel's blob is placed at that distance. The network then refines these depth estimates and predicts the rest of each blob's shape and colour.

This matters because earlier feed-forward methods predicted the blob positions more loosely, which left noisy floating blobs and weaker geometry. MVSplat places the blobs using matching evidence instead, and it is also smaller and faster than the closest prior method, pixelSplat.

TL;DR

• Feed-forward 3D Gaussian Splatting from sparse posed views: two input images, one pass, no per-scene optimisation at test time §1 §3.
• Core component is a plane-sweep cost volume over depth candidates, built from cross-view features, then refined by a U-Net §3.1.
• Depth is unprojected to Gaussian centres, and other Gaussian parameters are predicted alongside it §3.2.
• Trained with an L2 plus LPIPS rendering loss, with no depth supervision §3.3.
• Author-reported: best PSNR/SSIM/LPIPS on RealEstate10K and ACID, with about 22 fps and 12.0M parameters tab. 1 §1.

Problem

Per-scene optimisation of neural radiance fields is expensive, and several NeRF-based methods have high memory cost or slow rendering §1. The closest prior feed-forward Gaussian method, pixelSplat, predicts a probability distribution over depth for each view. The paper argues that such a distribution is hard to learn from image features alone, so its geometry is noisy and its good geometry needs an extra fine-tuning stage with depth regularisation §1 §4.2.

Assumptions: input views with known intrinsics and extrinsics, a near and far depth range per view, and training data of multi-view scenes with posed frames §3 §4.1. The paper evaluates at 256×256 resolution §4.1.

Method

The model maps K posed images to one Gaussian per pixel per view. Each Gaussian has a position, an opacity, a covariance and spherical-harmonic colour coefficients eq. 1 §3.

1. Features. A shallow residual CNN extracts 4× downsampled features per view, and a multi-view transformer with local window attention exchanges information across views §3.1 app. C.
2. Cost volume. For view i, the features of every other view are warped onto D depth candidates sampled uniformly in inverse depth. Each warped feature is compared with view i by a scaled dot product, the correlations are averaged over the other views, and the result is a cost volume of D planes per view. High similarity marks a likely surface point §3.1 eq. 2 eq. 3 eq. 4 §1.
3. Cost refinement. A lightweight 2D U-Net takes the feature and cost concatenation and outputs a residual added to the cost. Cross-view attention is inserted at the U-Net's lowest resolution §3.1 eq. 5.
4. Coarse depth. A softmax over the depth planes gives a per-pixel distribution. The coarse depth is the expectation over the candidates §3.1 eq. 6. The per-pixel maximum of this distribution is kept as a matching confidence §3.2.
5. Depth refinement. A second, very light U-Net takes the images, the upsampled features, the coarse depth and the confidence, and predicts a residual depth that is added to the coarse depth. It also uses cross-view attention at its lowest resolution §3.1 app. C.
6. Gaussian heads. Two conv-layer heads predict the opacity and the covariance and colour parameters. The covariance comes from scales and a quaternion. The colour comes from spherical-harmonic coefficients. The paper's inputs are the image features, the refined cost volume and the images §3.2 (the code's inputs differ, see Paper vs code).
7. Centres. Each pixel's depth is unprojected along its camera ray to a world-space point, and these points become the Gaussian centres §3.2.
8. Loss. Only rendering losses on target images: an L2 term with weight 1 and an LPIPS term with weight 0.05 §3.3.

# Forward pass: K posed views -> Gaussians for every pixel of every view  [§3]
# Input:  images (K, 3, H, W); intrinsics (K, 3, 3); extrinsics (K, 4, 4)
# Output: Gaussians: means, covariances, colors, opacities

def predict_gaussians(images, intrinsics, extrinsics, near, far):
    # 1. Cross-view-aware features at 1/4 resolution  [§3.1]
    features = multiview_transformer(cnn(images))  # (K, C, H/4, W/4)

    # 2. Cost volume over D inverse-depth planes  [§3.1] [eq. 2-4]
    cost = plane_sweep_cost(features, intrinsics, extrinsics, near, far)
    # (K, D, H/4, W/4)

    # 3. Refine the cost: U-Net output plus a skip term  [§3.1] [eq. 5]
    refine_in = concat(features, cost)  # (K, C + D, H/4, W/4)
    cost = unet_refine(refine_in) + skip_conv(refine_in)  # (K, D, H/4, W/4)

    # 4. Per-pixel distribution over the depth planes  [§3.1] [eq. 6]
    probs = softmax(depth_head(cost), over=planes)  # (K, D, H/4, W/4)

    # 5. Coarse depth as the expectation over the planes  [§3.1] [eq. 6]
    coarse_depth = sum(probs * depth_planes, over=planes)  # (K, H/4, W/4)
    coarse_depth = upsample_bilinear(coarse_depth, scale=4)  # (K, H, W)

    # 6. Matching confidence: maximum probability per pixel  [§3.2]
    confidence = upsample(max(probs, over=planes), scale=4)  # (K, 1, H, W)

    # 7. Full-resolution features from the CNN upsampler  [§3.1]
    proj_feat = upsampler(concat(features, cnn_features))  # (K, C', H, W)

    # 8. Depth refinement U-Net on images, features, depth, confidence  [§3.1]
    refined = refine_unet(concat(images, proj_feat, coarse_depth, confidence))

    # 9. Residual depth is added to the coarse depth  [§3.1]
    depth = coarse_depth + residual_head(refined)  # (K, 1, H, W)

    # 10. Opacity: sigmoid of a head on the refinement output  [§3.2]
    opacities = sigmoid(opacity_head(refined))  # (K, 1, H, W)

    # 11. Centers: unproject each pixel's depth along its camera ray  [§3.2]
    means = unproject(depth, intrinsics, extrinsics)  # (K, H, W, 3)

    # 12. Covariance from scales and a quaternion; color from SH  [§3.2]
    covariances = build_covariance(scales, quaternions)  # (K, H, W, 3, 3)
    colors = spherical_harmonics(sh_coeffs)  # (K, H, W, 3, D_sh)

    return means, covariances, colors, opacities

Key equations, as written in the paper:

f_\theta: \{(I^i, P^i)\}_{i=1}^{K} \mapsto \{(\mu_j, \alpha_j, \Sigma_j, c_j)\}_{j=1}^{H \times W \times K} eq. 1

F^{j \to i}_{d_m} = \mathcal{W}(F^j, P^i, P^j, d_m) eq. 2

C^{i}_{d_m} = \frac{F^i \cdot F^{j \to i}_{d_m}}{\sqrt{C}} eq. 3

C^{i} = [C^{i}_{d_1}, C^{i}_{d_2}, \dots, C^{i}_{d_D}] eq. 4

\tilde{C}^{i} = C^{i} + \Delta C^{i} eq. 5

V^{i} = \mathrm{softmax}(\hat{C}^{i})\, G, \quad G = [d_1, \dots, d_D] eq. 6

Here P are camera projection matrices, K is the number of input views, D is the number of depth candidates, C is the channel dimension and G is the vector of depth candidates §3.1. The paper gives no value for C; the code uses 128 costvolume.yaml.

Hyperparameters

Values in the first two columns come from the paper where it states them, and the Notes column flags where the code differs or is the only source.

Results

Author-reported. Setup: trained on RealEstate10K, two context views, three target views per test scene, 256 × 256 §4.1 tab. 1. Training used a single A100 GPU §4.1. The paper does not state the hardware used for the timing column. Timing includes encoding and rendering tab. 1.

Table 1: comparisons with the state of the art tab. 1.

The paper says the MuRF numbers come from the authors re-training the official code, while the other baselines are taken from the pixelSplat paper §4.2.

Table 2: cross-dataset generalisation, trained on RealEstate10K only tab. 2.

Table 3: ablations on RealEstate10K tab. 3. "base" is the model without depth refinement. The final model is "base + refine".

Other reported results (appendix tables, author-reported):

• Table A app. A: pixelSplat with its own depth branch: 25.89 PSNR, 0.858 SSIM, 0.142 LPIPS. With the paper's cost volume depth in place of that branch: 26.63, 0.875, 0.122.
• Table B app. A: Epipolar Transformer 0.055 s, 26.09 PSNR; Swin Transformer 0.038 s, 26.12 PSNR.
• Table C app. A: one Gaussian per pixel, render 0.0015 s, 26.39 PSNR; three per pixel, render 0.0023 s, 26.54 PSNR.
• Table D app. A: pixelSplat with DINO init, 25.89 PSNR; MVSplat with UniMatch init, 26.39; random init at 300K steps, 26.01; random init at 450K steps, 26.29.
• Few-view DTU, trained on two views and tested on three context views: 14.30 PSNR, 0.508 SSIM, 0.371 LPIPS, against pixelSplat's 12.52, 0.367, 0.585 §4.2.

Qualitative results are in fig. 3 fig. 4 fig. 5 fig. 6 and in the supplement app. B.

Paper vs code

Code: official repository donydchen/mvsplat at commit 01f9a28edb5eb68416e7e63b01f8d90c3bdfbf01, MIT licence. The paper links the same repository §4.1 and the authors' project page p. 1. The paper's headline setup matches the RealEstate10K experiment config. Differences:

• Cost volume resolution. Paper: the cost volume is upsampled to full resolution by a CNN upsampler, and the softmax of eq. 6 runs on that full resolution volume §3.1 eq. 6. Code: the cost volume and softmax stay at 1/4 resolution. The coarse disparity is bilinearly upsampled, and the CNN upsampler is applied to features instead depth_predictor_multiview.py.
• Cost-refinement skip path. Paper: the U-Net residual is added to the initial cost volume §3.1 eq. 5. Code: the residual is added to a 1×1 conv of the U-Net input (features concatenated with cost), not to the raw cost depth_predictor_multiview.py.
• Expectation domain. Paper: the softmax weights the depth candidates themselves eq. 6. Code: it averages inverse depths (disparities) and takes the reciprocal later depth_predictor_multiview.py depth_predictor_multiview.py.
• Depth refinement residual. Paper: the residual is a depth §3.1. Code: the residual is added in disparity space and the sum is clamped to [1/far, 1/near] depth_predictor_multiview.py.
• Opacity. Paper: two convolution layers predict opacity from the matching confidence §3.2. Code: the maximum probability is only an input to the depth-refinement U-Net. Opacity is the sigmoid of the second output channel of a conv head on the refinement output depth_predictor_multiview.py depth_predictor_multiview.py. The map_pdf_to_opacity mapping is the identity when initial and final are both 0, which is the default encoder_costvolume.py costvolume.yaml.
• Inputs to the covariance and colour heads. Paper: refined cost volume, image features and the images §3.2. Code: the depth-refinement output, the images and the upsampled features depth_predictor_multiview.py.
• Sub-pixel centre offset. Not in the paper. Code shifts each ray by sigmoid(offset) − 0.5 of a pixel before unprojection encoder_costvolume.py.
• Scale parameterisation. Not in the paper. Code maps scales to [0.5, 15] and multiplies by depth and a camera-dependent factor, so Gaussian size grows with depth gaussian_adapter.py.
• Depth-refinement width. Paper: "similar configuration" to the cost-volume U-Net, which keeps 128 channels app. C. Code: 32 channels in the RealEstate10K experiment re10k.yaml.
• LPIPS schedule. Paper: weight 0.05, no schedule given §3.3. Code: LPIPS applies from step 0 in the RealEstate10K experiment re10k.yaml. The default is 150,000 lpips.yaml. The code's LPIPS network is VGG loss_lpips.py.
• Learning-rate schedule. Paper gives no learning rate or schedule. Code: Adam, peak 2e-4, one-cycle cosine with 1% warm-up model_wrapper.py main.yaml.
• Pretrained weights. Paper: "publicly available UniMatch pretrained weight" app. C. Code points to a specific checkpoint file (named after a GMDepth-scale1 variant) that is not in the repository costvolume.yaml. The paper does not name the variant.
• Two-view pose. Code computes the reference pose as the inverse of the target extrinsic times the reference extrinsic. The comment says this reproduces the paper's numbers exactly depth_predictor_multiview.py.
• Default config vs experiment config. Code defaults (32 planes, 64 depth-refinement channels) are not the paper's settings. Use the RealEstate10K experiment file re10k.yaml.
• Timing numbers. The text in §4.2 gives encoder 0.043 s against pixelSplat 0.102 s, while Table 1 gives 0.044 s and 0.104 s §4.2 tab. 1.

No discrepancies were found in: the feature-extractor layer count, the 4× downsampling, the window split, the 128 depth planes for the experiments, the batch size of 14, the 300K steps, the near and far bounds, the 256 × 256 input and the L2 + LPIPS loss weights.

Evaluation index files are in the repository evaluation_index_re10k.json, which fixes the test scenes and views.

Pitfalls

• Scale of the cost volume. With 128 planes at 1/4 resolution, the cost volume is (K, 128, H/4, W/4) per scene, which is 128 × 64 × 64 per view at 256 × 256 §3.1 app. C. This is the main memory cost of the method.
• Undocumented normalisation. Warping uses normalised grid coordinates with align_corners=True, pixel coordinates are divided by (w − 1), and intrinsics are scaled to pixel units from normalised values. Mismatched conventions silently break the cost volume depth_predictor_multiview.py depth_predictor_multiview.py.
• Projection clamp. The projected depth is clamped at 1e-3 before the division. The paper does not discuss points behind the camera depth_predictor_multiview.py depth_predictor_multiview.py.
• Batch layout. The depth predictor works in (view × batch) order, not (batch × view), and the code warns about it depth_predictor_multiview.py. A reimplementation that mixes the two will produce wrong correlations.
• Ablations are not final-checkpoint numbers. The "w/o cross-view attention" variant overfits, so the paper reports its best checkpoint, not its final one §4.3 tab. 3 app. A.
• Geometry without depth supervision. The model is trained with photometric loss only. The paper reports that reflective surfaces such as glass and windows are unreliable §5 app. A.
• Generalisation depends on the training set. Cross-dataset results come from RealEstate10K training only. The paper calls the training diversity insufficient for in-the-wild scenes §5 tab. 2.
• Count of Gaussians. The model outputs one Gaussian per pixel per view, so two views at 256 × 256 give 131,072 Gaussians per scene §3 app. A. Rendering more Gaussians per pixel improves quality and slows rendering app. A tab. C.
• Reported speed is an encoder-plus-render time on unstated hardware. Compare with the table caption before quoting 22 fps §1 tab. 1.
• Text and table numbers disagree slightly on encoder time and render time for pixelSplat §4.2 tab. 1.

Lineage

• Builds on: pixelSplat (Charatan et al., CVPR 2023), the feed-forward Gaussian baseline whose protocol the paper follows §4.1 §4.2. Not linked: no arXiv id in the paper's bibliography.
• Builds on: UniMatch for the backbone initialisation app. C costvolume.yaml.
• Uses: Swin Transformer local window attention §3.1 app. C.
• Compared against: pixelNeRF, GPNR, AttnRend, MuRF and pixelSplat §4.2 tab. 1.
• Superseded by: not stated in this wiki.