EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS

In plain words

3D Gaussian splatting draws a scene as a large cloud of small soft blobs. It renders fast and trains quickly, but a single scene can need millions of blobs, and each blob stores many numbers. That makes the saved scene large and the training run memory-hungry.

EAGLES keeps the same kind of blob cloud and makes it cheaper to store. Most of each blob's numbers, the colour and the rotation (and the opacity), are replaced by short lists of whole numbers. A small learned decoder turns those lists back into the real values. The method also starts training with small images and grows them to full size, which makes the early steps cheaper and the fit steadier. Finally, it removes the blobs that contribute least to the final picture.

The result the authors report is a scene that is roughly ten to twenty times smaller on disk, with about the same picture quality and shorter training. The card below lists the exact settings, and the places where the paper leaves a value out or its numbers do not agree with each other.

TL;DR

EAGLES compresses 3D Gaussian splatting (3D-GS) along three axes: quantized per-point attributes decoded by small learned decoders §4.1, a progressive render-scale schedule from 0.3 to 1.0 §4.2, and influence-based pruning that removes 15% of Gaussians at each stage §4.3 §5.1. The authors report 27.23 dB PSNR on Mip-NeRF360 with 54 MB storage against 734 MB for 3D-GS as reported in the original paper tab. 1. Their own rerun of 3D-GS gives 27.45 dB and 745 MB tab. 1. The official code is licensed for non-commercial research only, so this card does not use it (see Paper vs code).

Problem

Standard 3D-GS needs several million Gaussians per high-resolution scene. Each Gaussian holds position, scale, rotation, opacity and spherical-harmonic colour coefficients §1 §3. The colour and rotation coefficients make up more than 80% of the stored values for SH degree 4 §4.1. The authors report storage above 1 GB per scene and GPU training memory near 20 GB for several high-resolution scenes §1. Densification also adds many redundant Gaussians that slow training and rendering §4.3. Compression methods for radiance fields reached by then either trade away quality or need a lot of tuning, according to the related-work section §2.

Method

The method has three parts, shown in one overview figure fig. 2. Attribute quantization shrinks storage §4.1. Progressive training changes the render resolution during optimisation §4.2. Influence pruning removes Gaussians that barely affect the rendered views §4.3. The base 3D-GS training loss and densification rules stay in place, and the paper states that all other hyperparameters are 3D-GS defaults [supp. Hyperparameters].

Background (3D-GS as used). A Gaussian is evaluated with eq. 1. Its covariance is built from a scale and a rotation eq. 2. Projection to the image uses eq. 3. Pixel colour is an ordered alpha blend of the overlapping Gaussians eq. 4. Training minimises a weighted sum of L1 and SSIM terms eq. 5, with the SSIM weight set to 0.2 §3.

G(\boldsymbol{x}) = e^{-\frac{1}{2}\boldsymbol{x}^T\Sigma^{-1}\boldsymbol{x}} eq. 1

\Sigma = RSS^TR^T eq. 2

\Sigma' = JP\Sigma P^TJ^T eq. 3

C = \sum_{i\in\mathcal{N}}\boldsymbol{c}_i\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j) eq. 4

\mathcal{L} = (1-\lambda)\mathcal{L}_1+\lambda\mathcal{L}_\text{SSIM} eq. 5

1. Attribute quantization. Each quantized attribute (colour, rotation, opacity) has an integer latent vector and a decoder, an MLP that maps latents to attribute values §4.1. During training the paper keeps continuous latents, rounds them in the forward pass, and passes gradients straight through the rounding (straight-through estimator). The decoded attribute is then eq. 6 §4.1. Latents are trained end to end with the rest of the scene §4.1.

\boldsymbol{a} = D(STE(\widehat{\boldsymbol{q}})) eq. 6

Position, scale and the base colour (the degree-0 SH term) are not quantized, because the paper found them sensitive to initialisation and prone to large quality losses §4.1. After training, the latents are rounded to integers and stored with the decoder, using entropy coding §4.1. The paper does not name the entropy coder.

2. Latent initialisation. The 3D-GS point values are turned into initial latents by inverting the decoder. For a linear decoder this is a least-squares fit [supp. Hyperparameters]. The paper does not say how an MLP decoder is initialised.

3. Latent learning rate. The latents take the attribute's learning rate, multiply it by a scale factor, and then divide it by the size of the decoder weights (for a linear decoder) [supp. Hyperparameters]. The paper says this keeps training stable when that size is very large or very small. The scale factor is 1.0 for all three attributes in the paper's table tab. 6.

4. Opacity quantization. Opacity is quantized too, with a one-dimensional latent. The paper argues that the quantized opacity gets fewer extreme gradients than unquantized opacity, and that gradient steps move it through discrete bins, which acts as a soft regulariser against saturation §4.1 fig. 3. It says this removes many floaters §5.3 fig. 7.

5. Progressive training (coarse to fine). Training starts with the scene rendered at a small scale and increases the scale to full resolution over a schedule §4.2. The paper uses a start scale of 0.3 and an end scale of 1.0 on a cosine schedule, and applies the schedule over the first 70% of iterations; after that, training stays at full resolution §5.1. The render scale is applied by downsampling the training images §5.4. The paper argues this gives the Gaussians a better early fit, and that it renders fewer pixels early on, which lowers training time §4.2.

# Render scale for the training views  [§4.2] [§5.1]
# The paper gives no formula; this is one cosine reading of its schedule.
def render_scale(iteration, total_iterations):
    # The schedule covers the first 70% of iterations  [§5.1]
    progress = min(iteration / (0.7 * total_iterations), 1.0)
    # Start at 0.3, end at 1.0, on a cosine curve  [§5.1]
    return 1.0 - 0.7 * (1.0 + cos(pi * progress)) / 2.0

6. Influence pruning. For Gaussian i at pixel p, the paper defines the influence as its blending weight after the Gaussians in front of it eq. 7. The influence of Gaussian i on the whole scene is the sum over pixels eq. 7 §4.3. The weights come out of the rasteriser at no extra cost §4.3. They are accumulated over N iterations so that all training views count, and a fixed fraction of the Gaussians with the lowest totals is removed §4.3 §5.1.

W_{i,p} = \alpha_i T_i = \alpha_i\prod_{j=1}^{i-1}(1-\alpha_j), \quad W_i = \sum_p W_{i,p} eq. 7

# Influence pruning  [§4.3] [eq. 7] [§5.1]
# Per training step: add each Gaussian's pixel influence to a running total
def accumulate_influence(totals, alphas, transmittances):
    # alphas, transmittances: (G, P), one row per Gaussian  [eq. 7]
    per_gaussian = (alphas * transmittances).sum(axis=1)  # (G,)
    return totals + per_gaussian
# At each pruning stage: drop the lowest-influence fraction  [§5.1]
def prune_stage(gaussians, totals, fraction=0.15):
    # The fraction 0.15 is the paper's value per stage  [§5.1]
    count_drop = int(fraction * len(gaussians))
    keep = argsort(totals)[count_drop:]
    return gaussians[keep]

Hyperparameters

Results

Author-reported. Setup: three real-scene datasets, the train/test split above §5.1, the 3D-GS rows from the original 3D-GS paper unless marked with * (the authors' runs) §5.1. Measured on the authors' hardware, which is not stated §5.1.

The abstract claims 10 to 20 times less memory at comparable quality [abstract]. Against the original 3D-GS storage figure, Table 1 gives about 13.6 times on Mip-NeRF360 and about 14.2 times on Tanks and Temples tab. 1.

Mip-NeRF360, 9 scenes tab. 1:

Tanks and Temples, 2 scenes tab. 1:

Deep Blending, 2 scenes tab. 2:

Ablation on three scenes, one per dataset, with components added in order tab. 3. PSNR in dB, storage, number of Gaussians, FPS:

Progressive scaling variants on one Tanks and Temples scene (PSNR in dB) tab. 4:

Peak GPU memory during training and rendering tab. 5:

Other figures: the teaser and the visual comparisons fig. 1 fig. 5 fig. 6; the influence-pruning visual fig. 4; the resize-scale sweep on the Truck scene, averaged over three seeds fig. 8.

Paper vs code

The abstract and the project page both link an official repository: https://github.com/Sharath-girish/efficientgaussian. Its owner is the first author. Its LICENSE.md is the Gaussian-Splatting License, which the repository's own text describes as allowing research use only, and which forbids commercial use. This card therefore does not use the code, and the code fields record only the repository, its commit and that license. Nothing in this card comes from the repository's code or configs. Searched: the abstract, the project page (efficientgaussian.github.io), and GitHub repository search by method name.

Pitfalls

• Densification interval. §4.3 says the cloning and splitting happen every 100 iterations, while the implementation settings give 175 §4.3 §5.1 §5.3. The paper does not say whether the 100 describes 3D-GS only.
• SH degree. §4.1 uses degree 4 for the storage comparison §4.1. The settings fix the colour degree at 3 §5.1. The formula d = 3f² is written with f as the harmonic degree, and its value depends on whether f counts degrees or basis functions. The paper does not say which reading it uses.
• Deep Blending margin. The text gives a 0.56 dB gain over Mip-NeRF360 on Deep Blending §5.2. The table gives 29.86 against 29.40, a difference of 0.46 dB tab. 2. The text's 0.45 dB Mip-NeRF360 drop matches the table's 27.69 against 27.23 only to rounding.
• Speed claims across scenes. The text says training is faster and FPS higher "on all scenes" than 3D-GS §5.2. The per-scene table disagrees for some scenes tab. 7: Kitchen (25m05s against 24m57s), Room (21m38s against 21m37s) and Treehill (21m49s against 21m46s) train slower, and Bonsai (177 against 187 FPS), Counter (138 against 139) and Room (123 against 131) render slower.
• Per-scene averages. The nine per-scene PSNR values for EAGLES average about 27.19, while the table average is 27.23 tab. 7 tab. 1. For the authors' 3D-GS runs the nine per-scene values average about 27.51, while the table gives 27.45 tab. 7 tab. 1. The averages are not reproduced exactly.
• Speed over INGP. The text says rendering is more than 15 times faster than INGP on all datasets §5.2. On Mip-NeRF360, 131 FPS against 9 FPS is about 14.6 times tab. 1.
• Variant names. §5.2 describes three variants by iteration count and pruning strength, without names. The tables name them Ours, Ours-Small and Ours-Fast tab. 1. The mapping of names to settings is not stated. The Small setting's pruning schedule is also not given.
• Pruning details. The number of iterations N over which influence is summed is not stated §4.3. The first pruning iteration is not stated §5.1. Ties and the handling of Gaussians with zero influence are not described. The fraction of 15% is stated only in the settings §5.1.
• Coarse-to-fine details. The cosine form is named but not written out §5.1. The ablation variants are said to use the same schedule "gradually" §5.4, but the paper does not state whether the ground-truth images are downscaled too.
• Entropy coding. Storage figures depend on the entropy coder, which is not named §4.1. The paper says most of the remaining storage after quantization comes from attributes that were left unquantized §5.3. Those are position, scale and base colour, so a reimplementation's storage figures depend on how it encodes them.
• Latent learning rate. The latent rate is the attribute rate scaled and divided by the decoder norm [supp. Hyperparameters]. The paper does not say which norm, and it does not give the base learning rates.
• Decoder initialisation. Latents are initialised by inverting the decoder with least squares [supp. Hyperparameters]. For an MLP, the paper gives no method.
• Runtime memory gains are small. Latents are decoded to floats before each forward and backward pass, so per-Gaussian runtime memory does not shrink §5.5. The savings in tab. 5 come from fewer Gaussians, not from the quantization.
• FPS and decoding. FPS is measured after all latents are decoded once, so the cost of decoding is not in the FPS figure §5.1.
• Hardware not stated. Training and rendering times depend on the GPU, and no model is given §5.1. Times for 3D-GS come from the original paper for one row and from the authors' runs for the other tab. 1. Compare times within one row group only.
• Seeds. The resize-scale sweep uses three seeds on one scene §5.3 fig. 8. The filter comparison uses one scene tab. 4.
• Gradient threshold. The split and clone threshold τ is named but no value is given §3 §4.3.

Lineage

• Builds on: 3D Gaussian splatting (Kerbl et al.), whose rasteriser, loss, densification and training defaults are used §3 §5.1 [supp. Hyperparameters].
• Compared against: Plenoxels, Instant-NGP (INGP) and Mip-NeRF360 tab. 1 tab. 2; 3D-GS, both as reported and as rerun by the authors tab. 1.
• Superseded by: not recorded in the wiki.