SparseGS: Sparse View Synthesis using 3D Gaussian Splatting

In plain words

3D Gaussian Splatting builds a scene out of many small, semi-transparent blobs and draws them quickly. It works well when there are many photos. With only a dozen photos of a park or a room, the blobs can end up in the wrong place: stray specks, called floaters, hang in front of the scene, and from a new angle the background can fall apart.

SparseGS keeps the same blob model and adds ideas that keep it honest with few photos. It takes a rough depth picture of each training photo from a depth-prediction model and nudges the blobs to agree with it, without needing the two depth scales to match exactly. It also changes how the depth of a pixel is read off the blobs, so the depth follows the blobs that matter most while the others still get a learning signal. For some unseen viewpoints it asks an image diffusion model to smooth out grainy artifacts, and it warps training photos slightly to make new nearby views. Once the scene is fitted, it finds blobs that sit at the wrong depth and deletes them.

The aim is better pictures from new angles, using few input photos, with training and real-time rendering that stay practical.

TL;DR

SparseGS is a training pipeline for 3D Gaussian Splatting when only 12 (Mip-NeRF360) or 3 (forward-facing scenes) input images are available. It combines a monocular depth prior with Pearson-correlation losses, a softmax-scaled depth renderer, a Stable Diffusion score-distillation loss and depth-warped views, and then prunes floaters. The paper reports the highest PSNR on Mip-NeRF360 among the methods it compares, at 12 and 24 views, but not on LLFF or DTU. The official code is public under a non-commercial licence, and it disagrees with the paper on several loss weights and schedules (see Paper vs code).

Problem

Plain 3DGS needs dense training views to reconstruct geometry. With sparse inputs it produces artifacts such as floaters and background collapse at unseen viewpoints §1 [abstract]. The paper argues that alpha-blended depth can let gradients reach the wrong Gaussians, which hurts quality §1 §3.1. The target settings are 360-degree unbounded scenes with 12 input images and forward-facing scenes with as few as 3 [abstract] §1. Mip-NeRF360 evaluations use 12 or 24 views §4.1.

Method

The pipeline keeps 3DGS and adds three parts: a depth-correlation loss against a monocular depth prior, an unseen-viewpoint regulariser built from SDS and depth warping, and a floater-pruning step §3. Figure 2 shows the pipeline, with the base 3DGS parts in grey and the new parts in colour fig. 2.

1. Three depth renderings from the same Gaussians §3.1. Alpha-blended depth sums over every Gaussian on the ray, weighted by transmittance and opacity eq. 3. Mode-selected depth takes the depth of the single Gaussian with the largest weight w_i = T_i α_i eq. 4. Softmax-scaled depth is a log of a weighted average of depths, with weights w_i·exp(β·w_i) eq. 5. As β goes to 0 it approaches the log of alpha depth, and as β goes to infinity it approaches the log of mode depth eq. 6 eq. 7. The paper keeps the normalising sums from the forward pass and uses accumulators in the backward pass, so the cost stays linear rather than quadratic in the number of Gaussians per ray app. A.

2. Patch-based depth correlation loss §3.2 eq. 8. A pre-trained monocular depth model supplies pseudo-ground-truth depth for each training view. The paper uses Marigold for this §3.2. The softmax depth is compared with that prior by one minus the Pearson correlation on N randomly sampled non-overlapping square patches. A global Pearson term over the whole image is added so patch edges do not show seams. Pearson correlation ignores scale and shift, which suits a monocular model that predicts relative depth §3.2.

3. Score Distillation Sampling on unseen views §3.3.1 eq. 9 eq. 10 app. B fig. 4. Random cameras are sampled on an elliptic cylinder fitted to the training camera positions. The cylinder's axes come from the 90th percentile of displacements from the scene centre, and its height from the 10th to 90th percentile of camera heights. Each camera looks at the scene centre and uses the average up vector app. B. The rendered view is noised, Stable Diffusion 2.1 predicts the noise, and the loss compares that prediction with the noise that was added. The gradient flows back to the Gaussians eq. 9 eq. 10. The paper reports that this removes high-frequency artifacts and leaves the scene structure unchanged fig. 4.

4. Depth warping loss §3.3.2 eq. 11 eq. 12 app. B. The monocular depth of a training view is rescaled to the minimum and maximum of that view's alpha-blended depth. The training image is then re-projected to nearby camera poses eq. 11. The loss is an L1 difference on the valid warped pixels, using a warp mask eq. 12. The paper says the warp views are rotations about the average up vector of the training cameras, and this is how it avoids the large view changes of the SDS samples app. B.

5. Floater pruning §3.4 app. C fig. 5. For each training view, Δ is the relative difference between mode-selected and alpha-blended depth. Views with floaters show bimodal histograms of Δ. A dip-test statistic is computed per view and averaged over views. The mean sets a percentile cut-off, written as a·e^(b·D̄) with a = 0.97 and b = −7.5 app. C. Pixels above the cut-off are flagged. For each flagged pixel, the Gaussians up to and including the mode Gaussian are deleted fig. 5 app. C.

6. Schedules §4.1. The depth correlation loss converges after about 3k iterations. SDS is enabled after two-thirds of the iterations, once most of the geometry is stable. Floater pruning runs once, after training finishes.

The base 3DGS losses, optimiser and densification are not restated in the paper §4.1.

# Depth from the Gaussians along one pixel's ray  [§3.1]
# weights: w_i = T_i * alpha_i, front to back   depths: d_i   beta = 5 [§4.1]
def softmax_depth(weights, depths, beta):
    # Weight each Gaussian by w_i * exp(beta * w_i)  [eq. 5]
    scaled = weights * exp(beta * weights)
    # Log of the weighted mean depth  [eq. 5]
    return log(sum(scaled * depths) / sum(scaled))

# Depth-correlation term for one training view  [§3.2]
def depth_loss(softmax_depth, prior_depth, patch_size, num_patches):
    # Sample non-overlapping square patches at random  [§3.2]
    patches = sample_patches(softmax_depth, prior_depth, patch_size, num_patches)
    # One minus Pearson correlation per patch, averaged  [eq. 8]
    local = mean(1 - pearson(p_depth, p_prior) for p_depth, p_prior in patches)
    # Global term so that patch edges do not show seams  [§3.2]
    global_term = 1 - pearson(softmax_depth, prior_depth)
    # Both weights are 0.15 in the paper  [§4.1]
    return 0.15 * local + 0.15 * global_term

# One training iteration  [§3, §4.1]
def training_step(iteration, total_iters, view, prior, rng):
    # Photometric loss on the training view; the base 3DGS loss, not restated
    loss = photometric_loss(render(view), view.image)
    # Depth-correlation term against the monocular prior  [§3.2]
    loss += depth_loss(softmax_depth(render(view)), prior, patch, n)
    # After two-thirds of training, SDS on a random unseen view  [§3.3.1]
    if iteration > 2 / 3 * total_iters:
        # SDS weight 5e-4  [§4.1]
        loss += 5e-4 * sds_loss(render(random_ellipse_view(rng)))
    # Masked L1 on a depth-warped view  [§3.3.2] [app. B]
    # Warp weight 0.05  [§4.1]
    loss += 0.05 * masked_l1(render(warped_view), warped_target)
    return loss

# Floater pruning, run once after training  [§3.4] [app. C]
def prune_floaters(gaussians, train_views, a=0.97, b=-7.5):
    # Mean dip statistic of the Δ histograms over the training views  [app. C]
    avg_dip = mean(dip_test(delta(v)) for v in train_views)
    # Percentile cut-off from the scene-wide dip  [app. C]
    cutoff = a * exp(b * avg_dip)
    for view in train_views:
        # Δ = (d_mode - d_alpha) / d_alpha per pixel  [app. C]
        diff = (mode_depth(view) - alpha_depth(view)) / alpha_depth(view)
        # Flag pixels above the cut-off  [app. C]
        flagged = diff > percentile(diff, cutoff)
        for pixel in flagged:
            # Remove Gaussians up to and including the mode  [§3.4] [fig. 5]
            gaussians.remove(gaussians_upto_mode(view, pixel))

Hyperparameters

Values in the paper's own text and tables are listed first. Values that the paper leaves unstated are marked, and code values are given where the code sets them.

Results

Author-reported. Setup: Mip-NeRF360 with 12 and 24 input views, LLFF and DTU in the RegNeRF protocol, images downscaled to 1/4 §4.1. The paper does not state the hardware or the training time §4.1.

Mip-NeRF360 (unbounded), PSNR / LPIPS / SSIM tab. 1

"w/o MVS" means the initial point cloud comes from structure-from-motion without the multi-view-stereo step §4.2 tab. 1.

Forward-facing scenes, PSNR / LPIPS / SSIM tab. 2

Ablation on Mip-NeRF360, 12 views tab. 3

The text gives the same ablation as gains of 1.37 dB for softmax depth, 0.33 dB on average for UVR and 0.28 dB for pruning §4.3. See Pitfalls for the UVR figure.

Paper vs code

Code: the official repository ForMyCat/SparseGS, linked from the arXiv comments, at commit 95e7aef29c5562400d3b2b38cc7e90436a432b7c. Its LICENSE.md is the Gaussian-Splatting licence (Inria and MPII), which allows research use only. The card cites files and does not quote code.

• Loss weights differ. The paper sets the global Pearson, local Pearson, depth-warp and SDS weights to 0.15, 0.15, 0.05 and 5×10⁻⁴ §4.1. The example command in run.sh and the README uses 0.05, 0.15, 0.1 and 0.001 run.sh README.md. Code defaults are 0.0 for all four [code: arguments/init.py#L64-L65] [code: arguments/init.py#L79] [code: arguments/init.py#L83].
• Iteration count differs. The paper uses 10k iterations for 12 views §4.1. The example run for a 12-view scene uses 30 000 run.sh.
• Patch size and sampling differ. §3.2 says patch size is 1/100 of image resolution §3.2. §4.1 gives 64 px and 32 px §4.1. The example run uses --box_p 128 run.sh. The paper samples non-overlapping patches §3.2. The code draws random top-left corners, which can overlap, and uses p_corr = 0.5 of the grid cells loss_utils.py.
• Monocular depth model differs. The paper uses Marigold §3.2. The repository's depth preparation runs BoostingMonocularDepth README.md prepare_depth.py. Marigold does not appear in the code.
• Input resolution differs. The paper downscales to 1/4 §4.1. The example run passes -r 2, which the loader maps to half resolution run.sh camera_utils.py.
• Warp views differ. The paper makes four warp views per camera at −3°, −1.5°, +1.5° and +3° §4.1 app. B. The code makes two per camera at ±2.5° [code: scene/init.py#L102-L112].
• Warp views are sampled differently. On 4 in 10 iterations after the warp start, the code replaces the training view with a warp view and skips both Pearson losses on that iteration train.py train.py train.py. The paper does not describe this.
• Warp depth scale is fixed once. The code takes the min and max of alpha depth over all training views once, at the warp start iteration, and reuses them train.py cameras.py. The paper computes the scale from d^alpha and d^pt without saying when §3.3.2.
• SDS timestep and sampling. The paper does not state a timestep or a view probability. The code fixes the timestep from step_ratio (0.99 in the example run, clipped to 2%–98% of the scheduler's timesteps), uses guidance scale 100, and applies SDS on 10% of iterations after two-thirds sd_utils.py sd_utils.py train.py.
• Pruning timing differs. The paper prunes once after training §4.1 §3.4. The example run prunes at iteration 20 000 of 30 000, then keeps training run.sh train.py.
• Pruning statistic differs. Δ is defined with d^alpha in the denominator app. C. The code divides by the depth plus max(min depth, mean − 3σ) prune_utils.py. The dip test uses only positive Δ train.py.
• Pruning percentile differs. The paper gives a = 0.97 app. C. The code computes 100 × prune_perc × exp(−prune_exp × dip) with prune_perc = 0.98 in the example run, and clamps the result at 80 train.py train.py run.sh.
• Gaussian removal rule is extra. The code deletes Gaussians along each flagged pixel's ranked list, up to the mode, only where the Gaussian's response exceeds exp(−4) train.py [code: arguments/init.py#L74]. The paper says all Gaussians up to the mode are removed fig. 5 app. C.
• Consistent parts. β = 5 matches the code §4.1 [code: arguments/init.py#L98]. The SDS start at two-thirds matches §4.1 train.py. The elliptic-cylinder percentiles match app. B pose_utils.py. The softmax code subtracts 0.3 from w inside the exponential and adds 1e-8 inside the log forward.cu forward.cu. A constant factor in the exponent cancels in the ratio, so the value is the same as the paper's formula up to the epsilon.

Pitfalls

• Percentile units are ambiguous. Appendix C writes a percentile of a·e^(b·D̄) with a = 0.97, which only makes sense as a fraction. The code multiplies by 100 to get a percentile train.py. Use a fraction with a quantile function, and check the result against the 80 floor the code uses.
• Pruning statistic depends on the denominator. Different denominators give different Δ distributions, so cut-offs from the code do not transfer to the paper's definition, and the reverse holds too prune_utils.py.
• Unstated safety rules in the code. The 80 floor and the exp(−4) response threshold change which Gaussians get deleted, and the paper does not mention either train.py train.py.
• Patch size is given three ways. §3.2 says 1/100 of the resolution, §4.1 says 64 and 32 px, and the example code uses 128 px §3.2 §4.1 run.sh.
• UVR gain in the text does not match Table 3. The text gives an average of 0.33 dB for UVR §4.3. Table 3 gives 19.09 − 18.86 = 0.23 dB for that step tab. 3. The paper does not explain the difference.
• DTU PSNR is not a win. SparseGS scores 18.89 PSNR on DTU, below DNGaussian (18.91) and SparseNeRF (19.55) tab. 2. The text calls the method competitive there §4.2.
• Cost is not reported. The paper gives no GPU, no wall-clock training time and no per-iteration cost §4.1. The SDS step encodes a 512×512 render through the Stable Diffusion VAE and runs the U-Net with classifier-free guidance sd_utils.py sd_utils.py. The claim of fast training is not quantified in the paper.
• No variance is reported. The Results tables give single numbers, and the text gives no seeds or spread §4.
• Monocular depth quality matters twice. The depth prior drives the Pearson losses and the warp scale §3.2 §3.3.2. A different monocular model, as in the code, changes both, and the paper does not test that.
• Pruning sees only training views. The mask is computed from Δ on the training views app. C. The paper does not say what happens to floaters that are visible only from unseen viewpoints.
• Non-commercial code. The repository's licence forbids commercial use. Use the paper's equations and the method description. Do not copy code from the repository into a commercial system without a separate licence.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., cited as kerbl3Dgaussians §2). No card for it yet.
• Compared against: RegNeRF, SparseNeRF and Mip-NeRF 360 tab. 1 tab. 2. Concurrent few-shot 3DGS methods FSGS (arXiv:2312.00451) and DNGaussian (arXiv:2403.06912) tab. 1 tab. 2 [main.bib].
• Superseded by: not known from content already in this wiki.