Reducing the Memory Footprint of 3D Gaussian Splatting

In plain words

A 3D Gaussian Splatting scene is a cloud of millions of small fuzzy blobs. Each blob stores a position, a shape, an opacity and a set of numbers that describe how its colour changes when you look at it from different sides. The original method makes these scenes very large, often hundreds of megabytes, which is slow to download and heavy on a phone.

This paper shrinks the scene in three ways. It removes blobs that sit in crowded spots where the camera could not tell them apart anyway. It drops the view-dependent colour numbers for blobs whose colour barely changes with the viewing angle. And it rounds the remaining numbers to a small table of shared values, so each number is stored as a one-byte index. The result is much smaller on disk and renders somewhat faster, with image quality that the authors report as close to the original.

TL;DR

The paper analyses where 3DGS memory goes: the number of Gaussians, the spherical-harmonic (SH) colour coefficients, and the precision of the stored attributes §3. It proposes a resolution-aware pruning rule §4.1, a per-Gaussian choice of SH bands §4.2, and k-means codebooks with half-float storage §4.3. The authors report about 27 times smaller files and a 1.7 times rendering speedup on standard scenes, with PSNR changes between −0.32 and +0.16 dB §1 §6.1 tab. 1. Relevant if you need compact 3DGS assets for storage, streaming or a web viewer. Code is under a non-commercial licence.

Problem

The original 3DGS representation is large: the paper reports 700 MB to 1.2 GB for the scenes it presents §1. Each Gaussian carries 59 floats with three SH bands, and 45 of those (76 %) are SH coefficients §3 fig. 2. The paper argues that 3DGS densification often creates unnecessarily dense sets of primitives, that most surfaces are diffuse and need only the base colour, and that most attributes tolerate low precision, while positions do not §3. Results are for the standard 3DGS schedule of 30,000 iterations §6.2, with densification stopping at 15,000 §4.2. The SH degree is raised every 1000 iterations in the code train.py.

Method

The method has three components, applied in this order fig. 3 §4:

1. Redundant-primitive culling during training, every 1000 iterations. A Gaussian is redundant when many other Gaussians fall inside a sphere around it whose radius is set by the smallest pixel footprint of any training view that sees it §4.1. Half of the redundant ones, the lower-opacity half, are deleted. Separately, the lowest 3 % by opacity are deleted, with an opacity ceiling of 0.05 §4.1 fig. 4.
2. SH band adjustment once, at iteration 15,000 (when densification stops). Each Gaussian keeps 0, 1, 2 or 3 SH bands §4.2. A Gaussian whose colour barely varies between views is reduced to its base colour. Otherwise the lowest band whose colour stays close to the full-band colour is kept §4.2 eq. 1 eq. 2. The reported split for the main model is 89 %, 0.1 %, 2.7 % and 8.2 % of primitives at 0, 1, 2 and 3 bands §4.2.
3. Codebook quantisation after training. Attributes except position are clustered into 256-entry k-means codebooks, and each value is stored as a one-byte index. Positions and codebook entries are stored as 16-bit floats §4.3.

Section 5 describes the storage format. The file holds four vertex groups, one per SH band count (0 to 3), so a renderer can read a variable number of SH coefficients per Gaussian §5. Quantised data is decoded at load time, not during rendering §5.

Step 1: redundant-primitive culling §4.1

# Redundancy culling, run every 1000 iterations  [§4.1]
# Input:  centres (N, 3), shapes (N, 3, 3), opacity (N,), views
# Output: boolean mask (N,), True = delete

def redundancy_mask(centres, shapes, opacity, views):
    # Smallest world-space pixel footprint at each centre, over the
    # views that see it; the closest view gives the smallest one  [§4.1]
    pixel_size = min_pixel_footprint(centres, views)    # (N,)
    # Sphere radius: half the diagonal of a cube with face pixel_size  [§4.1]
    radius = pixel_size * sqrt(3) / 2                   # (N,)
    # The 30 nearest centres are the only candidates to test  [§4.1]
    candidates = knn(centres, k=30)                     # (N, 30)
    # Count primitives whose ellipsoid meets the sphere around each
    # centre; the centre's own primitive is counted too  [§4.1]
    sphere_score = count_hits(centres, shapes, candidates, radius)
    # Each primitive takes the smallest score of any sphere it meets  [§4.1]
    score = min_score_over_hit_spheres(sphere_score, candidates, shapes)
    # Threshold is mean + 1 std, but never below 3  [§4.1]
    threshold = max(mean(score) + std(score), 3)
    # Redundant primitives are those above the threshold  [§4.1]
    redundant = score > threshold
    # Of the redundant ones, delete the half with the lowest opacity  [§4.1]
    median_opacity = median(opacity[redundant])
    mask = redundant & (opacity < median_opacity)
    # Also delete the lowest 3 % by opacity, capped at opacity 0.05  [§4.1]
    low_opacity = min(quantile(opacity, 0.03), 0.05)
    mask = mask | (opacity < low_opacity)
    return mask

Step 2: SH band adjustment, at iteration 15,000 §4.2

# SH band adjustment, once at iteration 15,000  [§4.2]
# Input:  per Gaussian: colour per view, weight per view (average
#         transmittance), SH coefficients per band
# Output: number of bands to keep (0 to 3) and base colour

def adjust_sh_bands(colours, weights, sh_coeffs):
    # Weighted mean and spread of the colour across views  [§4.2] [eq. 1]
    mean_colour = weighted_mean(colours, weights)       # (3,)
    spread = weighted_variance(colours, weights, mean_colour)  # (3,)
    # Colour is the same from every view: no view dependence  [§4.2]
    # (the paper tests per channel; the code averages channels, see
    # Paper vs code 1)
    if mean(spread) < 0.04:
        # Store only the base colour, with no higher bands  [§4.2]
        return 0, mean_colour
    # Distance from the full colour when only the first q bands are used
    distances = zeros(3)
    for q in [0, 1, 2]:
        # Weighted average over views of the colour difference  [§4.2]
        distances[q] = weighted_mean_over_views(
            norm(colour_with_bands(sh_coeffs, q) - colour_full(sh_coeffs)),
            weights,
        )
    # Keep the lowest band whose distance is below 0.04  [§4.2]
    for q in [0, 1, 2]:
        if distances[q] < 0.04:
            return q, mean_colour
    # Otherwise keep all three bands  [§4.2]
    return 3, mean_colour

Step 3: codebook quantisation after training §4.3

# Codebook quantisation, after training  [§4.3]
# Input:  attribute values in 20 groups (see below)
# Output: per group a 256-entry codebook, and one 1-byte index per value

def quantise(groups):
    # Groups: opacity; the three scale components (one shared codebook);
    # rotation real and imaginary parts; base colour; and 15 SH groups,
    # each a 3-channel vector  [§4.3]
    for group in groups:
        # Cluster the values into 256 centres with k-means  [§4.3]
        centres = kmeans(group.values, clusters=256)
        # Each value stores the index of its nearest centre  [§4.3]
        indices = nearest_centre(group.values, centres)
        store(indices, centres)
    # Positions and codebook entries are stored as 16-bit floats  [§4.3]
    store_as_float16(positions)
    return

Hyperparameters

Values in the paper are cited to the paper. Where the paper does not state a value, the code is cited, and the row says so. Code paths are relative to the repository at code_commit.

Results

Author-reported. Setup: NVIDIA RTX A6000 on Linux, the Mip-NeRF360 full set, two Deep Blending scenes and two Tanks & Temples scenes, as in the 3DGS paper §6 tab. 1. "3DGS*" is the authors' own re-run of the public 3DGS code, not the numbers printed in the 3DGS paper §6.2 tab. 2. FPS counts only the CUDA rasteriser, excluding graphics-API overhead §6.

Per-scene results are in tab. 7 to tab. 9. Comparisons with unpublished concurrent methods are in tab. 5 and tab. 6. Ours is 29 MB on Mip-NeRF360 and 18 MB on Deep Blending there, against 29 MB and 25 MB for Compressed3D tab. 5. The authors note that on-disk savings do not carry over to VRAM at render time for their compressed format app. (this refers to Compressed3D, not to this method).

Paper vs code

The official code is at graphdeco-inria/reduced-3dgs, at commit dc905c13491a717ab30062b4828736e11a7c7907. The paper's footnote points to a project page §5, and that page links this repository. The repository extends the 3DGS code, and the 3DGS licence applies to the original parts LICENSE.md.

1. Base-colour test uses a different quantity. The paper's eq. 1 defines σ_c as a weighted sum of squared deviations, with no square root, and thresholds it at 0.04 §4.2 eq. 1. The code takes the square root of the weighted variance, then averages over the three colour channels, and tests that against the threshold gaussian_model.py. The CUDA kernel that returns the variance was not read, so the exact normalisation is not confirmed.
2. Band 0 is not tested in the distance step. The paper tests every q in {0, 1, 2} and picks the lowest that passes §4.2. The code loops over q from the active degree down to 1 only gaussian_model.py, so a Gaussian reaches zero bands only through the base-colour test.
3. Threshold units. The paper gives ε_cdist = 0.04 §4.2. The full-method configuration passes 6, which the code scales by √3/255 to about 0.041 full_eval.py train.py.
4. Loss weights and schedule are missing from the paper. The paper gives no weight for the opacity L1 term §4.1 or the SH sparsity term §4.2. The full configuration uses 0.001 for opacity full_eval.py and 0.1 for SH full_eval.py. Culling stops at 27,000 iterations of 30,000 when enabled train.py train.py. The paper does not say when culling stops.
5. Opacity-reset iterations skip culling. Before iteration 15,000, the code skips the culling step on opacity-reset iterations train.py. The paper is silent on this.
6. Self-count in the redundancy score. The paper's wording ("two other primitives" for a score of 3) implies the centre's own primitive is counted. The code adds 1 at the end of the count [code: scene/init.py#L142-L185]. Whether the k-NN list already holds the centre was not checked.
7. K-means details. The paper does not state the initialisation or stopping rule. The code seeds centres with random samples and runs at most 500 iterations at tolerance 1e-4 gaussian_model.py. The base colour uses tolerance 1e-3 gaussian_model.py.
8. Pixel footprint. The code computes the footprint per Gaussian centre, using only views where the centre lies inside the view's depth range [code: scene/init.py#L106-L140]. This matches the paper's description §4.1.

Pitfalls

• Licence. The code is under the Gaussian-Splatting License, which permits research and evaluation use only. Commercial use needs the licensors' consent LICENSE.md. Do not copy the code into a commercial product without that consent. This card quotes no code verbatim.
• Training memory does not drop. The pruning and SH steps run during training, but the authors say their memory peak is unchanged, so the saving is in disk storage and render-time memory §5. Budget training VRAM as for 3DGS.
• The 27× figure is not a plain average. The three per-dataset ratios in tab. 1 are 25.7, 27.6 and 34.8. The paper does not state how its 27× is computed §1 §6.1.
• Appendix sizes are not reproducible from the tables. The appendix gives 26 MB and 41 MB for the full and low variants as averages over three datasets app.. The per-dataset values in tab. 2 give different averages, and the averaging basis is not stated.
• Minimum over spheres. A primitive takes the smallest score of any sphere it meets, so one sparse neighbourhood protects it §4.1.
• Opacity cap. The low-opacity cut uses the lower of the 3 % quantile and 0.05 gaussian_model.py. When the quantile is above 0.05, fewer than 3 % are removed.
• FPS and speed. The 1.7× speedup is from the rasteriser alone. Per-scene ratios against 3DGS* in tab. 2 are about 1.6 to 1.9. The WebGL phone demo is one scene over Wi-Fi, not a benchmark §6.1.
• Quantised scenes are decoded at load time. The viewer decompresses on parse, and codebook lookups are not in the render loop §5. The authors expect rendering from codebooks to cut memory traffic further, but did not measure it §5. They also say VRAM at render time would fall only if the codebooks and half floats are integrated into the renderer §6.1.
• Per-stage gains. The text says each component cuts size about 3× §6.1. Table 1 gives Mip-NeRF360 steps of 744 → 339 MB, 339 → 102 MB and 102 → 29 MB, so the first step is about 2.2× and the others about 3.3× and 3.5× tab. 1.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., ACM TOG 2023). The paper's bibliography gives no arXiv id, so none is linked here §3. The paper uses the maximal-sampling-rate terminology of Mip-Splatting (Yu et al., 2023) §4.1.
• Compared against: Mip-NeRF360, Instant-NGP, Plenoxels, MeRF tab. 2. Unpublished preprints: EAGLES, Compact3D, Compressed3D (arXiv:2401.02436 per the paper's bibliography), Compact3DGS, LightGaussian tab. 5 tab. 6.
• Superseded by: not known from content in this wiki.