DepthSplat: Connecting Gaussian Splatting and Depth

In plain words

Turning a few photos of a scene into a 3D model can be done with 3D Gaussian splatting, which represents the scene as many small soft blobs that can be drawn from any viewpoint. Fast feed-forward methods predict those blobs in a single pass through a network. They place each blob using a depth estimate that comes from comparing the photos with each other. That comparison fails on textureless walls, shiny surfaces and parts of the scene that only one photo shows. Separate models estimate depth from a single photo very well, but depths from different photos do not agree with each other in scale.

DepthSplat combines the two ideas. Its depth network compares the photos with each other, and it also receives features from a large pretrained depth model that works on single images. The predicted depth is turned into 3D positions for the blobs. A second small network sets each blob's remaining properties, such as its size, orientation, colour and opacity.

The same system can be trained with the photos alone: it renders the blobs from held-out viewpoints and compares them with the real images. That signal on its own pre-trains the depth network, and the authors report that this helps later depth tasks. They also report fast reconstruction from many input views.

TL;DR

• Feed-forward Gaussian splatting from posed images. Per-pixel depth comes from a two-branch network: a multi-view cost volume plus features from a pretrained Depth Anything V2 ViT §3 fig. 2.
• Depth is unprojected to Gaussian centres. A second head sets opacity, scale, rotation and colour §3.4.
• Training uses MSE plus 0.05 times LPIPS on rendered novel views. That signal alone pre-trains the depth model without depth labels §3.5 §4.3.
• Author-reported: RealEstate10K 2-view PSNR 27.47 for the large model, against 26.39 for MVSplat tab. 6. Twelve views at 512 x 960 take 0.6 s §1.
• The official MIT code does not include the depth-supervised loss from eq. 1 or the depth datasets, so the depth benchmark numbers cannot be reproduced from it mse.yaml.

Problem

Feed-forward Gaussian splatting from sparse views, such as pixelSplat and MVSplat, places each Gaussian with multi-view feature matching. Matching struggles with occlusions, textureless regions and reflective surfaces §1. Monocular depth models are robust on in-the-wild images but their depths do not stay consistent in scale across views §1. At 512 x 960, MVSplat runs out of memory with 6 or 12 input views. The authors contrast this with their lower lowest feature resolution (1/8 against 1/4) and local rather than pairwise global cross-view attention app. B. The paper also proposes pre-training depth from posed multi-view images without geometry labels §1 §4.3.

Method

Input: N images with projection matrices. Output: a depth map for each image and one Gaussian per pixel per view, giving H x W x N Gaussians §3.

The depth model has two branches §3 fig. 2. The multi-view branch encodes each image with a lightweight shared ResNet at 1/s resolution and passes the features through a multi-view Transformer with six stacked self- and cross-attention layers §3.1. For more than two views, each reference view attends to its two nearest views by camera distance, and its cost volume uses the same two neighbours §3.1. Cost volumes come from plane-sweep stereo over D depth candidates, using dot-product correlation averaged over the selected views §3.1. The monocular branch takes features from the Depth Anything V2 ViT (patch size 14, output at 1/14 resolution). These are bilinearly resized to the cost-volume resolution §3.2.

The depth regressor concatenates the monocular features with the cost volume, runs a 2D U-Net and applies a softmax over the D candidates. The depth is the weighted average over candidates §3.3. A second scale at twice the resolution searches a narrow window around the upsampled coarse depth, using a smaller cost volume. A DPT head then upsamples to full resolution §3.3.

Each pixel's depth is unprojected with its camera to give the Gaussian centre. A second head, a DPT head according to the paper, takes the image, depth and features and predicts the remaining Gaussian parameters §3.4. The Gaussians are rendered with the standard splatting operation §3.4.

# DepthSplat forward pass  [§3] [fig. 2]
# Input:  images (B, V, 3, H, W); intrinsics (B, V, 3, 3);
#         extrinsics (B, V, 4, 4), camera to world; near, far (scalars)
# Output: depth (B, V, H, W); one Gaussian per pixel per view

def forward(images, intrinsics, extrinsics, near, far):
    # ImageNet mean and std normalisation  [code: mv_unimatch.py#L206-L214]
    images = normalise(images)
    # Shared ResNet-style CNN at 1/s resolution, s = 4 or 8  [§3.1] [§4]
    cnn_feats = cnn(images)                 # (B, V, C, H/s, W/s)
    # Six self- and cross-attention layers across views  [§3.1]
    # Cross-attention uses the 2 nearest views  [§3.1]
    mv_feats = mv_transformer(cnn_feats, extrinsics)
    # Depth Anything V2 ViT (patch 14), resized to H/s x W/s  [§3.2]
    mono_feats = resize(vit(images), (H / s, W / s))
    # D candidates. The paper says uniform in depth [§3.1]. The code
    # spaces them uniformly in inverse depth  [code: mv_unimatch.py#L426-L435]
    candidates = linspace_inverse(near, far, D)     # D = 128 in code
    # Plane-sweep cost volume: warp each neighbour onto every candidate,
    # dot product scaled by 1/sqrt(C), average over neighbours  [§3.1]
    # [code: mv_unimatch.py#L502-L505]
    cost = plane_sweep(mv_feats, candidates, intrinsics, extrinsics)
    # Concatenate cost volume and monocular features  [§3.3]
    # The code also concatenates the CNN and multi-view features
    # [code: mv_unimatch.py#L512-L514]
    logits = unet(concat(cost, mono_feats))         # (B, V, D, H/s, W/s)
    # Softmax over candidates, then the expectation over candidates  [§3.3]
    # Candidates are inverse depths in the code, so this averages 1/depth
    probs = softmax(logits, over=candidates)
    inv_depth = sum(probs * candidates)
    # Second scale at 2x resolution: search a narrow window of candidates
    # around the upsampled coarse depth, then fuse again  [§3.3]
    inv_depth = refine_second_scale(inv_depth, ...)
    # Full resolution: bilinear upsample plus a learned DPT residual,
    # clamped to [1/far, 1/near]  [§3.3] [code: mv_unimatch.py#L545-L563]
    inv_depth = clamp(upsample(inv_depth) + residual, 1 / far, 1 / near)
    depth = 1 / inv_depth
    # Gaussian centres: unproject each pixel with its depth and camera  [§3.4]
    # The code also shifts each centre by up to half a pixel
    # [code: encoder_depthsplat.py#L289-L292]
    means = unproject(depth, intrinsics, extrinsics)
    # Second head fed with image, depth, features and match confidence
    # The paper says DPT head [§3.4]; the code uses plain 3x3 convs
    # [code: encoder_depthsplat.py#L110-L129]
    raw = gaussian_head(concat(images, depth, feats, match_conf))
    opacity = sigmoid(raw[..., 0])  # [code: encoder_depthsplat.py#L278]
    # Scale, rotation and SH coefficients  [code: gaussian_adapter.py#L62-L82]
    scales, rotations, sh = split(raw[..., 1:])
    return depth, Gaussians(means, opacity, scales, rotations, sh)

# Training objectives  [§3.5] [eq. 1] [eq. 2]
# Depth term: paper only. Not in the released loss code (see Paper vs code).
def depth_loss(depth, gt_depth):
    # L1 and x/y gradient terms, weights alpha = beta = 20  [eq. 1] [§3.5]
    # The text says inverse depth; eq. 1 writes depth
    return 20 * l1(depth, gt_depth) + 20 * grad_l1(depth, gt_depth)

# Rendering term, the one used for all released experiments  [eq. 2]
def render_loss(rendered, target):
    # Code averages over views; eq. 2 sums over the M novel views
    # [code: loss_mse.py#L44]  [code: loss_lpips.py#L54-L57]
    return mse(rendered, target) + 0.05 * lpips(rendered, target)

Key equations:

L_{\mathrm{depth}} = \alpha\,|D_{\mathrm{pred}} - D_{\mathrm{gt}}| + \beta\,|\partial_x D_{\mathrm{pred}} - \partial_x D_{\mathrm{gt}}| + \beta\,|\partial_y D_{\mathrm{pred}} - \partial_y D_{\mathrm{gt}}| eq. 1

L_{\mathrm{gs}} = \sum_{m=1}^{M} \left( \mathrm{MSE}(I^m_{\mathrm{render}}, I^m_{\mathrm{gt}}) + \lambda\,\mathrm{LPIPS}(I^m_{\mathrm{render}}, I^m_{\mathrm{gt}}) \right) eq. 2

Eq. 1 is the depth term; it has an unmatched closing parenthesis in the source, which is dropped here. Eq. 2 sums over M novel views rendered in one pass.

Hyperparameters

Results

Author-reported. Setup: RealEstate10K at 256 x 256 with 2 context views §4.4 tab. 6; DL3DV at 256 x 448 §4.4 tab. 7; a RealEstate10K-trained model evaluated on DL3DV and ACID §4.4 tab. 8; ScanNet depth with base models trained on RealEstate10K and DL3DV §4.4 tab. 5.

The model sizes are small (ViT-S, 1-scale), base (ViT-B, 2-scale) and large (ViT-L, 2-scale) §4.1. The large row of Table 6 matches the ViT-L 2-scale row of Table 1, so it is the large model. Table 7 does not state its variant.

Other reported results: the best monocular backbone for view synthesis is Depth Anything V2, although Depth Anything V1 has a slightly lower depth Abs Rel tab. 2. Concatenation fusion gives Abs Rel 8.46 and attention fusion gives 8.40 tab. 3. Hardware for the 0.6 s figure is not stated in the text read. Table 1 times are on one A100 for two 256 x 256 views tab. 1.

Paper vs code

Code location. The paper gives github.com/cvg/depthsplat as the code and training scripts §4. The clone is at commit 2dad25a7b9ba7c5537aba08463ef88d8511a06be, under the MIT licence with the copyright line of the first author LICENSE. Line references below are to that commit.

Depth candidates. Paper: uniform in depth between the near and far planes §3.1. Code: uniform in inverse depth, with candidates set between 1/far and 1/near mv_unimatch.py. The code also computes its expectation over these inverse depths mv_unimatch.py.

Cost volume. Paper: dot-product correlation §3.1. Code: the correlation is divided by the square root of the channel count, and then averaged over the neighbour views mv_unimatch.py.

Regressor inputs. Paper: cost volume and monocular features are concatenated §3.3. Code: cost volume, CNN features, multi-view features and monocular features are concatenated mv_unimatch.py. The regressor is a convolution, GroupNorm, a U-Net with cross-view self-attention, and a convolution, plus a 1 x 1 residual path mv_unimatch.py mv_unimatch.py. The paper describes only a 2D U-Net §3.3.

Full-resolution upsampling. Paper: a DPT head upsamples to full resolution §3.3. Code: the DPT output is added as a residual to a bilinear upsampling of the inverse depth, then clamped to the near and far range mv_unimatch.py.

Gaussian head. Paper: an additional DPT head takes image, depth and features §3.4. Code: a plain conv stack, conv then GELU then conv, applied after a conv regressor, with the match confidence also as input encoder_depthsplat.py encoder_depthsplat.py.

Gaussian centres. Paper: the centre is the unprojected depth §3.4. Code: each centre is also offset by sigmoid(x) - 0.5 in pixel units, so up to half a pixel encoder_depthsplat.py.

Gaussian parameters. Paper: not stated §3.4. Code: opacity is a sigmoid encoder_depthsplat.py; scale is softplus(x - 4) clamped to [1e-10, 3]; rotations are normalised quaternions; SH degree is 2; the DC term is initialised from the input RGB gaussian_adapter.py. The last layer's weights for the scale block are zeroed at initialisation encoder_depthsplat.py.

Depth loss. Paper: eq. 1 with alpha = beta = 20, supervising depth §3.5. Code: the loss modules are MSE and LPIPS only mse.yaml lpips.yaml, and the dataset loaders are RealEstate10K and DL3DV only dataset_re10k.py dataset_dl3dv.py. No depth-supervised training is in the release, and no TartanAir, ScanNet, VKITTI2 or KITTI loader was found. The depth benchmark training cannot be reproduced from the code. Inference-only depth output is available inference_depth.sh.

Rendering loss. Paper: sum over M views eq. 2. Code: MSE and LPIPS are averaged over batch and views loss_mse.py. This changes the overall scale by M, not the ratio between terms. LPIPS uses a VGG network loss_lpips.py; the paper does not name the network.

Intermediate supervision. Not in the paper. Code: rendering loss is also applied to the coarse-scale depth. Each intermediate loss is weighted by 0.9 raised to (number of intermediate levels minus its index), or by 0.9 when there is one level model_wrapper.py model_wrapper.py.

Optimiser. Paper: AdamW with cosine schedule §4. Code: OneCycle with cosine annealing and 1% warm-up of max steps model_wrapper.py. The warm_up_steps value is not used by the optimiser main.yaml.

Stem. Paper: stride-2 3 x 3 convolutions set the downsampling factor §3.1. Code: a 7 x 7 stride-2 first convolution, then residual stages backbone.py.

Training scale. Paper: DL3DV fine-tune on total batch 4 for 100K steps §4. The script uses batch 1 per GPU, 2 nodes, and its comment says 8 GPUs dl3dv_depthsplat_train.sh.

Model for high resolution. Appendix C calls the high-resolution model "small", with ViT-S for the monocular branch and one scale for the multi-view branch app. C. The released 256 x 448 script trains the base model (ViT-B, 2 scales) dl3dv_depthsplat_train.sh.

Config defaults. The experiment configs say 300,001 steps and batch 14 per GPU re10k.yaml re10k.yaml, but the scripts override them with 150K steps and batch 8 re10k_depthsplat_train.sh.

Pitfalls

• Candidate spacing. Depth candidates are uniform in inverse depth in the code, not in depth as the paper says. This gives more candidates to near depths. Eq. 1 writes depth while the text says inverse depth. The released code has no such loss.
• Expectation in inverse depth. The softmax expectation averages inverse depth in the code. Averaging depth directly, as the paper's wording suggests, gives different depths where the distribution is wide.
• Unstated inputs. The paper does not give the near and far values, the number of candidates D, the view sampling for training, the image normalisation, or the weight decay. The code values are in the table above.
• Hidden extra input to the Gaussian head. The match confidence, the maximum of the softmax over candidates, is an input to the Gaussian head in the code encoder_depthsplat.py. The paper does not list it.
• Sub-pixel offset. Gaussian centres are shifted by up to half a pixel. Without this, positions snap to pixel centres encoder_depthsplat.py.
• Pretrained learning rate by name. Any parameter whose name contains "pretrained" gets the lower learning rate, which is the monocular ViT model_wrapper.py. Renaming that module silently changes the learning rate.
• Pretrained weights. The encoder constructor first loads DINOv2 through torch.hub, and the Depth Anything V2 checkpoint is loaded later from the config mv_unimatch.py re10k_depthsplat_train.sh. The checkpoint decides the final weights.
• Patch shim. Images are cropped to multiples of shim_patch_size times the downscale factor, which is 16 by default and 64 for DL3DV encoder_depthsplat.py patch_shim.py dl3dv.yaml. Non-multiple sizes lose pixels at the edges.
• Matching neighbours. With more than three views, neighbours are chosen by camera-centre distance only; rotation is ignored encoder_depthsplat.py. The paper says "camera distance" without further detail §3.1.
• Intermediate depth. Coarse-scale depths are only upsampled and supervised during training mv_unimatch.py. They are not returned at test time.
• Memory and time. Cost volumes scale with D times the number of neighbour views at each resolution. Table 1 gives 38 M to 354 M parameters, and times from 0.050 s to 0.079 s for two views tab. 1. No peak-memory figure was found in the sections read.
• Model variant per table. Table 6 uses the large model tab. 1; Table 5 uses the base model §4.4; the variant behind Table 7 is not stated. Compare numbers only within the same variant.
• Speed claim. The 0.6 s figure for 12 views at 512 x 960 has no hardware stated in the text read §1.

Lineage

• Builds on: MVSplat, the feed-forward Gaussian method that uses a cost volume without monocular features §1 §4.4; UniMatch for the matching backbone and depth loss weights §3.1 §3.5; Depth Anything V2 for the monocular features §3.2; cascade matching for the hierarchical second scale §3.3; 3D Gaussian splatting for rendering §3.4.
• Compared against: pixelNeRF, GPNR, AttnRend, MuRF, pixelSplat, MVSplat and TranSplat on RealEstate10K tab. 6; MVSplat on DL3DV and ACID tab. 7 tab. 8; DeMoN, BA-Net, DeepV2D, NeuralRecon, DRO and UniMatch on ScanNet depth tab. 5.
• Superseded by: not known from content already in this wiki.