Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images

In plain words

3D Gaussian Splatting builds a 3D scene out of many small soft blobs and learns them from photos. With only two to five photos, it fits those photos perfectly by putting blobs at the wrong distances from the camera. The result looks right from the training views and falls apart from others, with floating specks and broken shapes.

The paper adds a hint about the shape of the scene. A depth-estimating network guesses how far each pixel is from the camera. Its guesses are not in the right units, so the paper lines them up with the few sparse distance points that a standard camera-pose tool already produces. The training then also asks each blob's rendered distance to agree with this hint, and asks neighbouring pixels to have similar distances except across object edges. Training stops once the blobs drift too far from the hint. A few other settings are simplified because the usual ones assume many photos. The authors report that this gives clearly better images and shapes than the standard method when photos are scarce.

TL;DR

Few-shot 3DGS with a dense depth prior §3. Monocular depth from ZoeDepth §4.1 is scale- and offset-fitted per image to COLMAP points §3.1 eq. 4. Rendered depth gets an L1 loss §3.2 eq. 7, and an edge-masked smoothness loss adds §3.3 eq. 8. Three few-shot changes: max SH degree 1, early stop on a moving average of depth loss, no periodic opacity reset §3.4. Author-reported: on NeRF-LLFF with 2 to 5 views, mean PSNR is 15.94 to 18.74 against 12.25 to 16.17 for 3DGS tab. 1.

Problem

With few images, 3DGS overfits the training views and produces floating artefacts and wrong geometry §1 fig. 1. Sparse SfM points are too few to regularise every splat §1. Depth sensors would give dense depth, but they are impractical §1. Monocular networks give dense depth, but only up to scale and offset, and the scale mismatch between views creates geometry conflicts §1. The paper states that its target setting is few images with known camera poses from SfM §3 §4.1.

Method

Overview fig. 2. The method is 3DGS with an extra depth target, a smoothness term and three few-shot changes. The input is k images, with camera poses, intrinsics and a point cloud from SfM (COLMAP is the example) §3. The pipeline has four parts.

1. Sparse depth from SfM. Each SfM point is projected into each image, giving a sparse depth map per image §3 eq. 1 eq. 2. Pixels without a point are empty.

2. Dense depth prior. A pre-trained monocular depth network gives a dense depth D_dense = s·F(I) + t §3.1 eq. 3. The scale s and offset t are solved by a weighted least-squares fit to the sparse depth §3.1 eq. 4. The weight of each point is the reciprocal of its SfM reprojection error, normalised to [0, 1] §3.1. The adjusted depth D*_dense is the target for the depth loss §3.1. The paper calls this an "adjusted dense depth" and does not say whether s and t are fit once per image or once per scene; the equation is per image, so this card reads it that way.

3. Depth rendering. The rendered depth reuses the per-splat alpha and transmittance weights from colour rendering: D = Σ d_i · α_i · T_i §3.2 eq. 5 eq. 6. Here d_i is the splat's depth in the camera frame §3.2. The depth loss is the L1 distance between rendered depth and D*_dense §3.2 eq. 7.

4. Smoothness and training loss. Adjacent pixels should have similar depth, except where a Canny edge mask marks an edge §3.3 eq. 8. The total loss adds the 3DGS colour and D-SSIM terms with the depth and smoothness terms, each with its own weight §3.3 eq. 9.

# Fit the dense depth prior for one training image  [§3.1] [eq. 4]
# image: one training view; sparse_depth: SfM depth, empty where no point
# weights: per-point reliability in [0, 1]
def fit_dense_depth(image, sparse_depth, weights):
    # Monocular network output, relative up to scale and offset  [§3.1]
    raw = monocular_depth(image)
    # Least squares for scale and offset over the sparse pixels  [§3.1] [eq. 4]
    scale, offset = weighted_least_squares(raw, sparse_depth, weights)
    # Adjusted dense depth is the depth target for the depth loss  [§3.1]
    return scale * raw + offset

# Training loss for one iteration  [eq. 7] [eq. 8] [eq. 9]
# rendered, target: RGB images; depth_rendered: alpha-blended splat depth
# depth_target: adjusted dense depth; depths: depths for neighbour terms
def total_loss(rendered, target, depth_rendered, depth_target, depths):
    # 3DGS colour and D-SSIM terms, weighted  [eq. 9]
    loss = (1 - lambda_ssim) * l1(rendered, target)
    loss += lambda_ssim * dssim(rendered, target)
    # L1 depth term toward the adjusted dense depth  [eq. 7]
    loss += lambda_depth * l1(depth_rendered, depth_target)
    # Smoothness between neighbouring depths, off Canny edges  [eq. 8]
    loss += lambda_smooth * smoothness_off_edges(depths, canny_mask)
    return loss

Few-shot changes §3.4:

5. Spherical harmonics degree 1. The maximum SH degree is set to 1, to stop high-frequency colour coefficients from overfitting §3.4.

6. Early stop on depth loss. Colour loss drives the optimisation, and depth and smoothness act as guides. A moving average of the depth loss decides when to halt, once the splats start to leave the depth guide §3.4.

7. No periodic opacity reset. The paper removes 3DGS's periodic reset, because resetting opacity led to splats being removed or trapped in local optima §3.4.

# Early stop on the moving average of the depth loss  [§3.4]
# depth_losses: depth loss per iteration so far
def should_halt(depth_losses, window):
    # Average over the window; window size and threshold are not stated  [§3.4]
    average = moving_average(depth_losses, window)
    # Halt when the splats leave the depth guide; the test is not given  [§3.4]
    return average_is_diverging(average)

Preprocessing §4.1: all images of a scene go through COLMAP so that poses and points are consistent across k-shot subsets. The points used for depth fitting and as initial points for a k-shot run are those visible in the chosen views (see Pitfalls).

Hyperparameters

Results

Author-reported. Setup: NeRF-LLFF, eight scenes, k = 2, 3, 4, 5 views, ten seeds averaged, test on the held-out views §4.1. Baselines: 3DGS on the same k poses and filtered points, and an oracle that uses pseudo-ground-truth depth §4.1. Both the mean rows and the per-scene rows are copied as printed in tab. 1; the mean rows do not equal the averages of the scene rows (see Pitfalls).

Mean over eight scenes tab. 1

Per scene tab. 1

Ablation on Horns tab. 2. Only the 2-view and 5-view columns are reported.

Initialisation on Fortress tab. 3.

The text describes the qualitative results. On Fern at 2 views, 3DGS produces geometry that is "entirely erroneous" against its RGB match, while the method's depth looks plausible §4.2 fig. 3. Crops are shown in fig. 4. The oracle images are in fig. 5 §4.2. The paper says the method does better on indoor scenes (Fortress, Room, Fern) and worse on natural scenes (Orchids, Flower), and that Leaves fails for COLMAP §4.2.

Paper vs code

Official code exists at https://github.com/robot0321/DepthRegularizedGS. The paper's text gives no code link. The project page, linked from the abstract, has a "Code" link to this repository, and the repository is by the first author (the paper lists the same handle in the contact address). Searched: the arXiv abstract and comments, the project page, and the GitHub search API for "DepthRegGS" (no results). The repository's LICENSE is the Gaussian-Splatting License, which is non-commercial and research-only. So this card does not use the code: no code field, and nothing is cited or quoted from the code. The README warns that the depth rasteriser had errors and was changed after arXiv v1, so a reader who wants the code should not assume it matches the paper's tables.

Pitfalls

• Sparse-point rule impossible for k = 2. The paper selects points "visible in at least three out of the k cameras" §4.1. With k = 2 no point can meet that test as written, yet 2-view results are reported. The paper does not say whether the rule is applied to all train views or to some other set. Implementers need to decide and document the choice.
• Sparse depth source differs between sections. §3.1 projects all visible SfM points §3 eq. 2. The experiments use the filtered set §4.1. The card's reading is the filtered set for experiments.
• Weighting in the fit is ambiguous. Equation 4 places w(p) on the sparse depth term: ||w(p)·D_sparse(p) − D_dense(p; s, t)||² eq. 4. Multiplying the sparse depth by the weight, rather than the residual, is an unusual choice. The normalisation of w is not stated §3.1. Whether the fit is per image is implied, not stated.
• Empty sparse pixels. D_sparse is defined as p_z over [0, ∞] eq. 2. How pixels without a point are marked (zero, NaN, or excluded) is not stated. The fit must skip them.
• Monocular depth scale claims conflict. §1 says these networks give relative depth §1. §4.1 uses ZoeDepth trained on NYU Depth v2 and KITTI, which the paper does not describe as relative. The fit makes either reading work, but the checkpoint is not named.
• Notation clash in the rendering equation. T_i is the transmittance product in eq. 5 and the translation vector in d_i = (R_i p_i + T_i)_z §3.2. Also p is used for both a projected pixel eq. 1 and a 3D point in §3.2. Use separate names when implementing.
• Loss norms and weights are not given. The depth loss is written as a norm eq. 7, but the norm's order and whether it sums or averages over pixels are not stated. λ_depth, λ_smooth and λ_ssim are symbols only eq. 9. The depth term's size depends on the depth units and on these weights, so they are the main tuning risk.
• Smoothness details. The adjacency is not defined §3.3 eq. 8. The squared-difference form is given. The Canny thresholds are not stated, and the edge mask is applied to both depths of a pair §3.3.
• Few-shot changes count mismatch. The text says "we modify two optimization techniques" §3.4, then lists three: SH degree, early stop and reset removal. Treat all three as modifications.
• Early-stop rule is missing. The paper says it uses a moving average of the depth loss to halt when splats deviate §3.4. The window, threshold, direction and what stops (the whole optimisation or only some parts) are not stated.
• Reset rule not restated. The paper removes the periodic reset but does not say its interval or its exact 3DGS behaviour §3.4. An implementer who keeps the stock 3DGS schedule for other settings will get a different run.
• Oracle uses test views. The oracle's pseudo-ground-truth depth comes from optimising both train and test images §4.1. It is an upper bound that has seen the test views, not a method result.
• Table means do not match the scene rows. Averaging the eight scene rows of tab. 1 gives values different from the printed means in four cases I checked: 3DGS 2-view PSNR 12.28 vs 12.25 printed; ours 2-view PSNR 15.91 vs 15.94; ours 5-view PSNR 18.80 vs 18.74; 3DGS 5-view PSNR 16.18 vs 16.17. Use per-scene values when comparing.
• Not uniformly better. On Flower, 3DGS has higher PSNR at 4 and 5 views (19.71 vs 19.15; 21.39 vs 20.45). On Leaves, 3DGS has higher SSIM at 2 to 5 views and lower LPIPS at every view. On Orchids, 3DGS LPIPS is lower at every view. The mean claims hide these.
• Caption copy-paste. Table 2 and Table 3 captions repeat the same text about "supervising with sparse depth from COLMAP instead of dense depth", which does not describe Table 3 tab. 2 tab. 3. Table 2's caption says "evaluated in Horns" and Table 3's says "Fortress"; those match the scenes used.
• Ablation text overstates one claim. The text says the smoothness loss without depth "yields worse results compared to the baseline" §4.3. In the table, "w/o depth loss" at 2 views (11.49) is slightly above 3DGS on Horns (11.43), and only worse at 5 views (12.97 vs 14.75) tab. 2.
• Single-scene ablations. Table 2 is Horns, Table 3 is Fortress. The paper does not say whether these averages cover all ten seeds.
• Hidden preprocessing cost. Poses and points come from running COLMAP on all images of each scene, including test images §4.1. Consistent poses across k-shot subsets depend on it. In a truly few-shot deployment this full-scene step is not available.
• Baselines are narrow. Only 3DGS and the oracle are compared §4.2. No other few-shot method is compared.
• Depth model cost and domain. Each training image needs one monocular depth inference, and the authors say quality depends heavily on it, with worse results on natural scenes §5 §4.2. The paper gives no run time.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al.), the base representation and optimisation §3 §4.1.
• Uses: ZoeDepth (Bhat et al.) for dense monocular depth §4.1; COLMAP (Schönberger et al.) for SfM §3; the smoothness constraint is inspired by Godard et al. §3.3; Canny edge detection §3.3.
• Compared against: 3DGS on the same poses and points, and an oracle with pseudo-ground-truth depth §4.1 tab. 1.
• Superseded by: not recorded in the wiki.