FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting

In plain words

Gaussian splatting draws a scene as a cloud of soft blobs and renders it fast. It normally needs many photos. With only three photos from different places, the starting cloud is far too sparse, and the blobs learn to fit those three photos exactly and then look wrong from new angles.

FSGS makes the cloud grow where it is thin. Each blob looks at its nearest neighbours. Where a blob is far from its neighbours, the method places new blobs halfway along the links to them. The method also invents extra camera positions near the real ones and asks that the scene's depth look plausible from those positions too. The depth comes from a pretrained model that guesses depth from a single picture. Its guesses are used only for their shape, not their absolute scale.

For an engineer the useful parts are the growth rule, the extra views, and the depth loss. The card lists the values the paper gives and the places where the paper leaves a value out or the equations do not say which way to go.

TL;DR

FSGS is a Gaussian-splatting method for novel view synthesis from as few as three training views. It starts from a sparse COLMAP point cloud, adds Gaussians by proximity-guided unpooling, and regularises depth on training and synthesised pseudo views with a monocular depth prior. The authors report 20.31 dB PSNR on LLFF at 1/8 resolution with 458 FPS. Official code exists but is licensed for non-commercial research only, so this card does not use it (see Paper vs code).

Problem

Three-view novel view synthesis from 3D Gaussian Splatting (3D-GS) fails for two reasons. First, the 3D-GS densification rule adds Gaussians where view-space position gradients are large, which is noisy with few views, and 3D-GS depends on the quantity and accuracy of its initial SfM points §3.1. Second, with insufficient parallax the Gaussians can grow very large and overfit the training views §3.1, fig. 3. The paper targets real-time rendering, which NeRF-based few-shot methods do not reach §1.

Method

The pipeline runs in four parts §3, fig. 2. The numbering of equations follows the order in the LaTeX source.

1. Initialisation §3, app. Initialization. Camera poses and a sparse point cloud come from COLMAP using the training views only. Each point starts one Gaussian. Spherical-harmonic (SH) coefficients start at degree 0, the remaining SH coefficients and the rotation are set to 0, opacity starts at 0.1, and the scale is set to the average distance between points app. Initialization.

2. Proximity graph and unpooling §3.2, fig. 4, alg. 1.

• Build a directed graph: each Gaussian links to its K nearest neighbours by Euclidean distance between centres. K = 3 §3.2.
• Distances: d_{ij} = \lVert \mu_i - \mu_j \rVert §3.2. The neighbour set is D_i^K = K\text{-min}_{j \neq i}\, d_{ij} eq. 3.
• Proximity score: P_i = \frac{1}{K}\sum_{j=1}^{K} D_i^K, the mean distance to the K neighbours eq. 4.
• If P_i exceeds t_\text{prox}, place one new Gaussian at the midpoint of each edge from the source Gaussian to a destination neighbour §3.2. The new Gaussian copies the scale and opacity of the destination. Its rotation and SH coefficients are set to 0 §3.2.
• The graph is rebuilt after each densification or pruning step §3.2.

3. Pseudo views §3.3, eq. 5, alg. 1. A pseudo camera is made from two nearby training cameras. Its rotation is the average of their rotations (as a quaternion). Its location is the average of their locations plus noise, \mathbf{t}' = \mathbf{t} + \varepsilon with \varepsilon \sim \mathcal{N}(0, \delta) on the three translation degrees of freedom eq. 5. Pseudo views are used only after 2,000 iterations §4.1, alg. 1.

4. Depth correlation loss §3.3, eqs. 6–7. A pretrained monocular depth model predicts a depth map for each training image and for each rendered pseudo view §3.3, app. Training. The rendered depth is the alpha-blended z-buffer of the Gaussians eq. 7. The loss compares the two depth maps with the Pearson correlation eq. 6. Correlation is used because it ignores the unknown absolute scale of the prior §3.3. For Blender scenes, correlation is computed only on pixels whose depth is greater than 0 app. Training.

The total loss, with weights found by grid search §3.4, eq. 8:

\mathcal{L} = \lambda_1 \lVert C - \hat C \rVert_1 + \lambda_2\, \mathrm{D\text{-}SSIM}(C, \hat C) + \lambda_3 \lVert \mathrm{Corr}(D_\text{ras}, D_\text{est}) \rVert_1 eq. 8

with \lambda_1 = 0.8, \lambda_2 = 0.2, \lambda_3 = 0.05 §3.4. Algorithm 1 adds a fourth term, \lambda_4 times the Pearson loss on pseudo views, but the paper does not state \lambda_4 alg. 1.

# FSGS training loop  [alg. 1] [§3.4] [app. Training]
# Input:  training images, poses, SfM points; depth model E
# Output: Gaussians G that render novel views

def train(images, poses, sfm_points, depth_model):
    # 1. Monocular depth for each training image  [alg. 1]
    depth_priors = [depth_model(image) for image in images]

    # 2. Pseudo camera poses from pairs of training poses  [§3.3] [eq. 5]
    pseudo_poses = make_pseudo_poses(poses)  # count and delta not stated

    # 3. Start one Gaussian per SfM point  [app. Initialization]
    gaussians = init_gaussians(sfm_points)

    for iteration in range(10000):  # [§4.1]
        # 4. Photometric and depth loss on a training view  [eq. 8]
        view = sample_training_view()
        rgb, depth = render(gaussians, view)
        loss = 0.8 * l1(rgb, view.image)  # lambda_1 [§3.4]
        loss += 0.2 * dssim(rgb, view.image)  # lambda_2 [§3.4]
        prior = depth_priors[view.index]
        loss += 0.05 * depth_corr(depth, prior)  # sign: see Pitfalls

        # 5. Pseudo-view depth term, after 2000 iterations  [§4.1] [alg. 1]
        if iteration > 2000:
            pseudo = sample(pseudo_poses)
            rgb_p, depth_p = render(gaussians, pseudo)
            # The prior is computed from the rendered pseudo image  [alg. 1]
            prior_p = depth_model(rgb_p)
            # lambda_4 is not stated in the paper
            loss += lambda_4 * depth_corr(depth_p, prior_p)

        # 6. Densify, prune and unpool every 100 steps after step 500  [§4.1]
        if iteration > 500 and iteration % 100 == 0:
            # Remove Gaussians by the rule as written in alg. 1  (see Pitfalls)
            prune(gaussians)
            # Standard 3D-GS gradient-based densification  [§3.1] [alg. 1]
            densify(gaussians)
            # Score = mean distance to 3 nearest neighbours  [§3.2] [eq. 4]
            scores = mean_knn_distance(gaussians.means, k=3)
            for source in gaussians where scores > 10:  # t_prox = 10 [§4.1]
                for destination in neighbours(source):
                    # New Gaussian at the edge midpoint  [§3.2]
                    new = midpoint(source, destination)
                    new.scale = destination.scale
                    new.opacity = destination.opacity
                    new.rotation = 0  # convention not stated
                    new.sh = 0
                    add(gaussians, new)

        # 7. Opacity reset to 0.05 at 2000, 5000 and 7000  [app. Training]
        if iteration in (2000, 5000, 7000):
            gaussians.opacity = 0.05

        # 8. SH degree rises from 0 to 4, one step every 500 steps  [app.]
        gaussians.sh_degree = min(iteration // 500, 4)

        update(gaussians, loss)  # per-parameter learning rates [app. Training]
    return gaussians

Learning rates from app. Training: position 0.00016, SH 0.0025, opacity 0.05, scale 0.005, rotation 0.001. The paper states no learning-rate decay.

Hyperparameters

Results

Author-reported. Setup in §4.1. Test images on LLFF and Mip-NeRF360 are every eighth image. FPS was measured on the A6000 for all methods §4.1.

LLFF, 3 training views tab. 1

Gaussian counts on LLFF: 57,513 for FSGS against 63,219 for 3D-GS §4.2.

Mip-NeRF360, 24 training views tab. 2

Blender (8 views, 400×400) and Shiny (3 views) tab. 3

Training views, LLFF at 1/8 resolution app. Effects of Training Views, tab. 6. PSNR / SSIM / LPIPS at 3, 6 and 9 views:

Mobile-phone captures (iPhone 15 Pro, 3 training views, 3 scenes) app. FSGS on Mobile Phones Data, tab. 7. 1/8 FPS, PSNR, SSIM, LPIPS | 1/4 FPS, PSNR, SSIM, LPIPS:

Ablation on LLFF, 3 views, 1/8 resolution tab. 4

Depth estimator on LLFF, 1/8 resolution tab. 5. The table header says "FPS ↓", but the text reads higher FPS as better; the values below are as printed.

Headline numbers: 2,180× the FPS of SparseNeRF on LLFF at 1/8 §4.2, tab. 1; 4,142× on Mip-NeRF360 at 1/8 tab. 2; a 0.85 dB PSNR gain over SparseNeRF on Mip-NeRF360 §4.2.

Paper vs code

No code comparison was made. The authors' repository is VITA-Group/FSGS, linked from the project page; its LICENSE.md is the Gaussian-Splatting License, which permits research use only and prohibits commercial use without the licensors' consent. wikillm publishes with advertising, so this card uses the paper alone and does not read or cite the repository's code. Only the repository's LICENSE.md was read, and the code was not used. The paper's own appendix conflicts on the depth model (see Pitfalls), and the released code could settle that question for someone with a research licence.

Pitfalls

• Resolution order is inverted in the text. §4.1 says 4× and 8× downsampling give "504×378 and 1008×756 respectively". The table captions and columns put 504×378 at "1/8 Resolution" tab. 1, which is the correct reading for a 4032×3024 original. Use 1/8 = 504×378 and 1/4 = 1008×756.
• Mip-NeRF360 is not few-shot. It uses 24 training views §4.1. The "203 FPS" real-time figure in the introduction §1 is the Mip-NeRF360 1/4-resolution number tab. 2.
• The regularisation term's sign is unclear. Eq. 8 writes \lambda_3 \lVert \mathrm{Corr} \rVert_1. As written, minimising this drives the correlation to zero, the opposite of the stated aim of aligning depth structure §3.3. The paper never states the sign or whether a term such as 1 - \mathrm{Corr} was meant. Choose a form that is maximised at correlation 1, and test it on a short run.
• \lambda_4 is missing. Algorithm 1 uses a pseudo-view weight but the paper never gives its value alg. 1.
• The pseudo-view depth term uses the rendered image. Algorithm 1 computes the prior for a pseudo view from the rendered pseudo RGB image, not from ground truth alg. 1. The algorithm's pseudo-view Pearson term also indexes the rendered depth with i where the pseudo view is j. Read it as comparing the depth of the pseudo view with the prior of that same view.
• Depth model is stated two ways. The main text names DPT §3.3, §4.1. The appendix links the PyTorch Hub MiDaS v2 model and says the estimator outputs inverse depth app. Training. The DPT Large row in tab. 5 matches the main-table numbers, which suggests DPT Large was used. Inverse depth is compared with rendered z-depth eq. 7 with no conversion stated.
• Pruning rule as printed looks reversed. Algorithm 1 removes a Gaussian when \alpha_i > \varepsilon alg. 1. Removing high-opacity Gaussians is unusual, and the appendix's opacity resets suggest low-opacity removal was meant. \varepsilon is not given. IsTooLarge and NoProximity are not defined, and the proximity-triggered unpooling is named NoProximity in the algorithm but described by threshold in §3.2.
• Proximity threshold has no units. t_\text{prox} = 10 §4.1 is in scene units. It depends on the COLMAP scale of each scene, which the paper does not normalise. Expect to re-tune it when the scene scale changes.
• "Rotation set to 0" is not a valid quaternion. §3.2 and app. Initialization set rotation to 0. A zero quaternion is not a rotation. The convention (identity, (1,0,0,0), or a different ordering) is not stated.
• Opacity is reset to 0.05 but initialised at 0.1. app. Initialization and app. Training give different values. The paper does not say whether the new unpooled Gaussians keep the destination's opacity after resets.
• Gradient threshold and density controls are not given. The value t_\text{pos} for densification alg. 1 and the densify/split rule are not stated beyond "standard 3D-GS densification" §3.1. The paper gives no learning-rate decay.
• Pseudo-view count and noise are not given. The number of pseudo views and the variance \delta in eq. 5 are not stated §3.3.
• Blender depth loss masks pixels. Correlation is computed only where depth is greater than 0 app. Training. The paper does not say how the mask affects the other datasets.
• Baseline training recipe. For 3D-GS, the paper follows its original dense-view recipe §4.1. Whether the 3-view 3D-GS numbers used that recipe unchanged is not stated.
• Small text mismatches. §4.2 says 0.40 dB over FreeNeRF on Blender, but tab. 3 gives 24.64 − 24.26 = 0.38 dB. The mobile-phone claim of "over 0.75" dB app. equals 0.75 exactly.
• Rebuttal material is not in the paper. The LaTeX source includes a rebuttal file with full-resolution and 1/2-resolution LLFF results and extra baselines. It is not part of the published paper, so its numbers are not used here.
• Cost. Each step renders the pseudo view and runs the depth model on it, so pseudo views add a depth-model call per step after iteration 2,000. The paper reports the wall-clock times above but not the share of this cost.

Lineage

• Builds on: 3D Gaussian Splatting, arXiv:2308.04079 §3.1. FSGS keeps its rasteriser and its densification as a base, and adds unpooling, pseudo views and the depth term §3.
• Depth prior: DPT, Ranftl et al. 2021, arXiv:2103.13413 §3.3.
• Compared against: Mip-NeRF, DietNeRF, RegNeRF, FreeNeRF and SparseNeRF §4.2. Concurrent work SparseGS and DRGS is described only in the rebuttal and is not in the paper.
• Superseded by: not stated in this wiki.