Taming 3DGS: High-Quality Radiance Fields with Limited Resources

In plain words

3D Gaussian Splatting builds a 3D scene out of many small soft blobs. Its usual training keeps adding blobs wherever the picture still looks wrong, and it does not stop at a fixed count. So the final model can be very large, the memory use is hard to predict, and the training time varies from run to run. This makes it hard to run on a phone or in any setting with a fixed limit.

The paper asks for a fixed size instead. The user picks how many blobs the model may end up with, and the method adds blobs on a smooth schedule until it reaches that number. Which blobs get copied is decided by a score that mixes several signals: how strongly a blob is pushed during training, how visible it is, how transparent it is, how big it is, and how wrong the picture is where it sits. The authors also make the training code itself faster, so a run finishes in less time. They report that the result looks close to the standard method while using far fewer blobs and far less time.

TL;DR

Budgeted 3DGS: densification grows the Gaussian count along a parabolic target that ends at a user-set budget, with no heavy pruning §1 §3.2. New Gaussians are drawn by a score built from per-Gaussian and per-pixel terms over 10 sampled views §3.3. Backpropagation is parallelised over splats, and higher spherical-harmonic bands are updated every 16 iterations §4.1 §4.2. Author-reported: a 4-5x cut in model size and training time against 3DGS, with quality that is competitive or better [abstract] tab. 1.

Problem

Standard 3DGS has four costs. Its optimisation is hard to control, and it often produces redundant Gaussians, which leaves some regions under-reconstructed §1. Model size can differ by up to about one order of magnitude between scenes that start from the same number of input points §1. Downstream tasks that need a fixed-size input cannot use the representation §1. Training time varies strongly and does not track the much faster rendering §1 §2. The authors also report that backpropagation is the main bottleneck, followed by Adam updates, and that the Adam cost rises as the Gaussian count grows §4 fig. 3.

Method

Overview fig. 2. Structure-from-Motion points initialise the model. A fixed densification schedule then adds Gaussians. A score-based sampler chooses which Gaussians to copy, using a per-pixel saliency map. Densification runs less often than in 3DGS §3. High-opacity Gaussians are switched on mid-training §3.4. Separately, the training step is made faster §4.

1. Background. A Gaussian has a position, a covariance and an opacity. Its contribution at a point is given by eq. 1. Colour uses degree-3 spherical harmonics plus a base colour. Training minimises a combined L1 and SSIM loss over the rendered and ground-truth images §3.1.

G(x) = o\,e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}, \quad \Sigma = R\,S\,S^T R^T eq. 1

2. Growth schedule. The number of added Gaussians per step follows a parabola that starts from the SfM count and peaks at the budget §3.2 fig. 2b. Each step adds the gap between the current count and the accumulated target, so that removed low-opacity Gaussians do not leave the count short §3.2. The paper prints the curve as eq. 2. Read literally, that form does not match the text (see Pitfalls).

A(x) = \frac{B-S-2N}{N^2}\,x^2 + 2x + B eq. 2

Here N is the number of densification steps, B the budget and S the number of SfM points §3.2.

3. Saliency map per view. For view v, the per-pixel saliency combines the L1 error against the rendering and a Laplacian-filter edge map, restricted to a binary region-of-interest mask. Both weights are set to 0.5 in the experiments §3.3 eq. 3.

\mathbf{S}_v = \mathbb{1}_{ROI} \odot \left(\lambda_1 \mathcal{L}_1(v, r_v) + \lambda_2 E(v)\right), \quad v \in V eq. 3

4. Per-Gaussian score. For each of N views, the score function F is evaluated for every Gaussian and the results are summed. The inputs are the positional gradient, the covered pixel count, the pixel-to-centre distances, the saliency sum, the blending weight sum, depth, opacity and scale §3.3 eq. 4. The vector of Gaussian scores is eq. 5.

S_g = \sum_{i=1}^{N} F\!\left(\nabla_g, c_g^i, \mathbb{1}_g^i, \mathbf{D}_g^i, \mathbf{S}_v^i, \mathbf{B}_g^i, z_g^i, o_g, s_g\right) eq. 4

\mathbf{S}_G = [S_{g_1}, \dots, S_{g_M}]^T eq. 5

The terms of F and their coefficients are listed below. Each parameter's range is first normalised by median scaling to remove outliers, then multiplied by the view's photometric loss, and the results are summed with the coefficients §3.3.

5. Sampling and splitting. Given the target B of Gaussians to add, B primitives are drawn at random from all Gaussians, using \mathbf{S}_G as sampling weights §3.3. Algorithm 1 then applies a rule to each drawn Gaussian. If its positional gradient is above \mathcal{G}_t and its radius is above \mathcal{R}_t, it is split. If the gradient is above \mathcal{G}_t and the radius is at most \mathcal{R}_t, it is cloned alg. 1. Densification runs once every 500 iterations, one fifth of the 3DGS frequency §3.3.

# Densify one step toward the target count  [§3.3] [alg. 1]
# gaussians: current model; views: training views; target: budget step
def densify_step(gaussians, views, target):
    # Accumulate the score over N sampled views  [§3.3] [eq. 4]
    scores = zeros(len(gaussians))
    for view in sample_views(views, count=10):  # N = 10 [§3.3]
        # Saliency: weighted L1 error and edges, inside the ROI  [eq. 3]
        saliency = roi * (0.5 * l1_error(view) + 0.5 * edges(view))
        # Per-Gaussian terms collected while rendering this view  [§3.3]
        terms = gaussian_terms(gaussians, view, saliency)
        # Median-scaled terms, weighted, times the view's loss  [§3.3]
        scores += view.photometric_loss * weighted_sum(terms)
    # Draw the number of new Gaussians, weighted by score  [§3.3] [alg. 1]
    num_new = target - len(gaussians)
    drawn = sample_by_weight(gaussians, scores, num_new)
    for g in drawn:
        # Split large Gaussians, clone small ones  [alg. 1]
        if grad(g) > G_t and radius(g) > R_t:
            split(g)
        elif grad(g) > G_t and radius(g) <= R_t:
            clone(g)
    return gaussians

6. High-opacity Gaussians. At the midpoint of training, 15K iterations §3.4, the opacity activation is replaced by an absolute-value function. Blending weights are clamped to 1 from above during rendering §3.4. The paper reports that this raises PSNR in its ablation §5.3 tab. 2.

# Switch to high-opacity Gaussians at the training midpoint  [§3.4]
def enable_high_opacity(gaussians, iteration):
    # Midpoint of training, 15K iterations  [§3.4]
    if iteration != 15_000:
        return
    # Opacity activation becomes an absolute value  [§3.4]
    gaussians.opacity_activation = absolute_value
    # Blending weights are clamped to 1 from above in the renderer  [§3.4]

7. Faster backpropagation. The original backward pass maps threads to pixels, iterates over splats back to front, and adds each per-pixel gradient to the splat's total with atomic operations. Contention on those atomics slows the backward pass §4.1. The proposed backward pass maps threads to splats instead. Each thread keeps a per-splat state. Pixel states (transmittance and accumulated colour) are passed between threads. The forward pass stores a pixel state every N splats, so the backward pass can start from those checkpoints §4.1 fig. 3b. Tiles also skip the tail of their sorted list once the last contributor is known. Tighter culling of splats is applied in both passes, following Radl et al. §4.1.

# Backward pass parallel over splats, not pixels  [§4.1] [fig. 3b]
# Each tile keeps a per-splat state, passed between threads
def backward_tile(splats_sorted, pixel_checkpoints):
    # Start from checkpoints stored every N splats in the forward pass  [§4.1]
    for i in splats_back_to_front:
        # Receive the state of every pixel after blending splats 0..i-1  [§4.1]
        state = exchange_pixel_states(i, pixel_checkpoints)
        # Gradient portion of splat i for every pixel  [§4.1]
        grad[i] += gradient_portion(state, i)

8. Batched SH updates. Bands above the first are updated with one Adam step every 16 iterations. Those bands hold 48 of the 59 optimised per-Gaussian attributes §4.2. The paper attributes most Adam time to these bands §4.2 fig. 3a. The first band is updated every iteration.

9. Other speedups. The degree-0 SH term and higher bands are loaded from separate tensors in the rasteriser, which avoids a concatenation §4.2. The SSIM loss uses an 11x11 Gaussian kernel evaluated as two separable 1D convolutions, fused with the SSIM computation §4.2. The authors state these changes are equivalent to 3DGS, except for the batched SH update §4.2. They report 1.5-2x speedups over gsplat v1.0 §4.2.

10. Pixel-level control (demonstration). A binary region-of-interest mask can be given to the saliency term. In a face example, the mask comes from SegmentAnything and its weight is set to 10^3. The face reaches the same PSNR as 3DGS sooner and with far fewer Gaussians §3.3 fig. 6.

Hyperparameters

Results

Author-reported. Setup: three scene sets, the same train/test split as 3DGS §5.1. Top of tab. 1: budgeted runs. Bottom: runs matched to 3DGS's model size. Evaluation ran on an RTX A4500. Baseline times were measured on the same hardware or adjusted §5.

Tanks and Temples (2 scenes) tab. 1

MipNeRF-360 (9 scenes) tab. 1

Deep Blending (2 scenes) tab. 1

Ablations on Tanks and Temples, budgeted setting tab. 2. All rows yield the same Gaussian count.

The text states that removing the image loss or the score-based sampling "significantly harms quality", and that the original SH update frequency can add minor quality at up to a 50% speed cost §5.3. The per-splat backward ablation keeps the same quality and changes only the time tab. 2.

Headline claims from the abstract: budgeted runs reach 4-5x reductions in model size and training time, and more generous budgets surpass 3DGS quality [abstract] fig. 1.

Paper vs code

Official code exists at https://github.com/humansensinglab/taming-3dgs. The URL the paper gives for the optimisations, https://github.com/nullptr81/3dgs-accel, resolves to that repository §4.2. This card does not use the code. The repository's base code is derived from 3DGS, and the copied LICENSE_ORIGINAL.md is the Gaussian-Splatting License, which forbids commercial use. The MIT statement in the README covers only the performance optimisations on a separate branch, so the default branch cannot be cited under our licensing rules. Searched: the paper's text, the project page, and the GitHub repository and its branches. Nothing in this card is taken from the code.

Pitfalls

• Eq. 2 is internally inconsistent. Taken literally, eq. 2 gives A(0) = B, while the text says the curve starts at the SfM count S §3.2. At x = N it gives 2B - S, while the text says it peaks at B. The paper does not state a corrected form. Any implementation has to pick the curve by its endpoints (start at S, reach B at the last step) and say so.
• Budget multiplier. The paper does not say how the SfM point count is obtained, or whether it is counted before or after any filtering §5.2.
• Score normalisation is undefined. "Median scaling" is described but no formula is given §3.3. The paper also does not give the exact normalisation of F. Coefficients are given, but not how each term is scaled before weighting.
• Positional gradient. The paper calls it "the magnitude of the positional gradient" §3.3 without saying whether it is accumulated over iterations, averaged, or taken in screen space.
• Sampling details. The paper does not state whether the B draws are with or without replacement §3.3.
• Drawn Gaussians may not yield B new ones. Algorithm 1 splits or clones only the drawn Gaussians whose gradient passes \mathcal{G}_t alg. 1. The paper does not say how a shortfall is handled or whether it is recounted.
• Thresholds and windows missing. \mathcal{G}_t, \mathcal{R}_t, the densification start and end, and N are not given alg. 1 §3.2. Algorithm 1 uses radius for the split decision, but the text does not mention radius.
• "Purely constructive" is not defined precisely. The paper says it does not require "substantial pruning or culling" §1. It also says 3DGS prunes low-opacity Gaussians over time §3.2. Whether any pruning occurs in this method is not stated.
• High opacity. The paper does not say how existing opacity values are mapped when the activation changes at 15K §3.4. "Clamping blending weights to 1" does not say whether the clamp applies to per-Gaussian alpha or to the summed weight.
• Per-splat backward. The paper describes it without equations. The state checkpoint interval is given as N in the text §4.1 and as 32 in the figure caption fig. 3. The caption also states 32-splat buckets, one per CUDA warp. These details are GPU-specific.
• Batched SH update. The paper does not state how Adam's moment estimates are handled on skipped iterations §4.2. The ablation shows a speed cost of up to 50% when the batching is removed §5.3.
• SH learning rate. The paper says the SH rate is raised "four times" to 0.001 §5, but the baseline value is not stated, so the factor cannot be checked.
• Equivalence claim. The paper calls the speedups numerically equivalent to 3DGS, except the SH schedule §4.2. Table 2 supports this for the per-splat backward, which keeps the same PSNR and LPIPS tab. 2. The SSIM kernel claim has no separate test.
• Resolution and downscaling. Image resolution used in each scene set is not stated.
• Matched scenario budgets. The per-scene Gaussian budgets for the matched runs are not listed §5.2.
• Baseline times may be adjusted. Times for baselines were "obtained on the same hardware or adjusted" §5. Treat cross-method timing as author-reported.
• Unit of Peak #G. The table header gives no unit for Peak #G while #G is in units of 10^6 tab. 1.
• Released code not checked. The optimisations are not verified against released code; see Paper vs code.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al.), the base representation and training pipeline §3.1 §5.
• Compared against: Compressed 3DGS (C3DGS, Niedermayr et al.) §5.2; Compact-3DGS (R-VQ) §5.2; Papantonakis et al. §5.2; INGP-Big (Instant-NGP) §5.2; Mini-Splatting §5.2; Plenoxels §5.2; MipNeRF360 and Zip-NeRF §5.2; DISTWAR for backward timing §4.1 fig. 4; gsplat v1.0 §4.2.
• Superseded by: not recorded in the wiki.