Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

In plain words

Feature 3DGS makes a 3D scene that can answer "which part is the car?" as well as show what the scene looks like. It starts from 3D Gaussian Splatting, which draws a scene as many soft blobs. Each blob also gets a short list of numbers that describes what it is, and those numbers are trained to match what a large 2D image model says about each photo.

Once trained, you can ask the scene for an object by typing its name, or click a spot to get its mask, and the answer holds from any camera angle. The 2D models produce much longer lists than the blobs can afford to draw, so the method draws a shorter list and expands it with a small learned layer. That keeps training and rendering fast.

The authors use it for segmentation, for deleting or recolouring objects by text, and for prompted segmentation from new views.

TL;DR

Feature 3DGS adds a feature vector to every 3D Gaussian and trains it by distillation from a 2D foundation model (LSeg or SAM), using the same differentiable splatting that renders colour. Rendering the teacher's full channel count is slow, so an optional 1x1 convolution lets the renderer draw fewer channels. The trained feature field supports text-driven segmentation and editing, and point or box prompts from new views. All quantitative results are author-reported on Replica; LLFF and LERF scenes appear only in qualitative figures. The official code is research-only.

Problem

Feature-field distillation with NeRF is slow, and its colour and feature branches share network layers, so learning the feature field can interfere with the radiance field. The usual fix is a small feature-loss weight, and the authors say NeRF is very sensitive to that weight §3.1 §3.2.

Rasterising RGB and a feature map separately, in two stages, gives lower quality for both, because features and colour are correlated §3.1. The core mismatch is that the teacher's feature has M channels, for example 512 for LSeg and 256 for SAM, while colour has 3, and the feature map has a resolution that can differ from the RGB image §3.2 app. C.

The method assumes a per-view teacher feature map has been computed in advance, which the code loads from files dataset_readers.py.

Method

The overall pipeline is shown in fig. 2. The method has four parts: the rendering model, the joint rasterizer, the speed-up module, and the optimisation and query procedure.

1. Gaussian state. Gaussians are initialised from a Structure-from-Motion point cloud §3.1. Each Gaussian stores a position x, a rotation quaternion q, a scale s, an opacity α, spherical harmonics up to order 3 for colour c, and a feature f with N channels, where N can be any size §3.1.
2. Covariance. The covariance is Σ = R S Sᵀ Rᵀ, built from q and s, so it stays positive semi-definite eq. 2. It is projected to the image with Σ' = J W Σ Wᵀ Jᵀ, where W is the world-to-camera transform and J the Jacobian of the affine approximation eq. 1.
3. Compositing. The pixel colour and the feature pixel are both front-to-back alpha composites over the sorted Gaussians: C = Σ c_i α_i T_i and F_s = Σ f_i α_i T_i, with transmittance T_i = Π_{j<i} (1 − α_j) eq. 3. The subscript s marks the student feature. The teacher F_t is the encoder output of the 2D foundation model on the ground-truth image §3.1.
4. Joint rasterization. Colour and feature go through the same tile-based rasterizer. The screen is split into 16×16 tiles, and Gaussians are culled per frustum and per tile. Both maps are rasterized at the same resolution but with different channel counts, namely 3 and N §3.1 app. A.
5. Speed-up module. The renderer draws N channels with N smaller than M. The rendered map is resized to the teacher's grid, then a 1x1 convolution maps N channels up to M §3.2 app. A app. B. The paper calls the module optional and says it does not change downstream performance §3.2, although the dimension sweeps in the supplement show that N matters app. F.
6. Loss. L = L_rgb + γ L_f eq. 4. The colour term is L_rgb = (1 − λ) L1(I, Î) + λ D-SSIM(I, Î). The feature term is L_f = ‖F_t(I) − F_s(Î)‖_1, with the rendered feature resized bilinearly to the teacher's H×W. The weights are γ = 1.0 and λ = 0.2 §3.2.
7. Optimisation. The Gaussians are optimised with Adam following the 3DGS setup §4 app. B. A refinement step removes transparent or very large Gaussians and splits or clones high-gradient ones, as in alg. 1. The thresholds in alg. 1 are not stated in the paper.
8. Query. A prompt or label feature q is compared to each Gaussian's feature by cosine similarity s eq. 5. A softmax over the label set gives a probability p(τ | x) eq. 6. Selection uses a threshold on the probability, an argmax over labels, or both combined with OR for deletion §3.3 app. E. The selected Gaussians then get opacity 0 for extraction or deletion, or a new colour for appearance edits app. E.

For SAM, the rendered feature is decoded by the SAM decoder alone, so the SAM encoder is skipped for novel views §4.2.

Training step, in pseudocode after the paper's equations eq. 3 eq. 4:

# Feature 3DGS training step  [§3.1] [§3.2] [eq. 4]
# Input: camera, ground-truth image, teacher map (M, H, W)
# Output: scalar loss

def training_step(camera, gaussians, decoder, teacher_map):
    # Splat colour and the N-dim feature in one pass  [§3.1] [eq. 3]
    image, student_map = rasterize(gaussians, camera)
    # Bring the rendered feature onto the teacher grid  [§3.2]
    student_map = resize_bilinear(student_map, teacher_map.shape[1:])
    # Lift N channels to M with a 1x1 convolution  [§3.2] [app. A]
    if decoder is not None:
        student_map = decoder(student_map)            # (M, H, W)
    # Photometric loss: L1 mixed with D-SSIM, lambda = 0.2  [§3.2]
    loss_rgb = (1 - lambda_ssim) * l1(image, gt_image)
    loss_rgb = loss_rgb + lambda_ssim * (1 - ssim(image, gt_image))
    # Feature loss: L1 between teacher and student maps  [§3.2] [eq. 4]
    loss_feature = l1(teacher_map, student_map)
    # Total loss, feature weight gamma = 1.0  [§3.2] [eq. 4]
    return loss_rgb + gamma * loss_feature

Refinement, and the query that selects Gaussians for editing or segmentation:

# Refinement, run on some iterations  [alg. 1]
# eps, tau_p and tau_S are not stated in the paper

def refine(gaussians):
    # Drop Gaussians that are transparent or too large  [alg. 1]
    drop = (gaussians.opacity < eps) | too_large(gaussians)
    gaussians.remove(drop)
    for gaussian in gaussians:
        # Only Gaussians with a large view-space gradient change  [alg. 1]
        if gradient(gaussian) > tau_p:
            # Large ones are split, small ones are cloned  [alg. 1]
            if scale(gaussian) > tau_S:
                split(gaussian)
            else:
                clone(gaussian)

# Select Gaussians for a label query  [§3.3] [app. E]
# Input: features (G, N), label features (K, N), label index, threshold
# Output: boolean mask over the G Gaussians

def select_gaussians(features, label_features, label, threshold):
    # Cosine similarity of every Gaussian to every label  [eq. 5]
    similarity = cosine(features, label_features)       # (G, K)
    # Softmax over the label set  [eq. 6]
    probability = softmax(similarity, axis=1)           # (G, K)
    # Soft selection: queried label passes the threshold  [app. E]
    soft_mask = probability[:, label] >= threshold
    # Hard selection: queried label is the top label  [app. E]
    hard_mask = argmax(probability, axis=1) == label
    # Hybrid selection, used for deletion: OR of both  [app. E]
    return soft_mask | hard_mask

Key equations, as written in the paper:

\Sigma = R S S^{\top} R^{\top} eq. 2

\Sigma' = J W \Sigma W^{\top} J^{\top} eq. 1

C = \sum_{i \in \mathcal{N}} c_i \alpha_i T_i, \quad F_s = \sum_{i \in \mathcal{N}} f_i \alpha_i T_i eq. 3

\mathcal{L} = \mathcal{L}_{rgb} + \gamma \mathcal{L}_f eq. 4

s = \frac{f(x) \cdot q(\tau)}{\|f(x)\| \, \|q(\tau)\|} eq. 5

\mathbf{p}(\tau \mid x) = \frac{\exp(s)}{\sum_{s_j \in \mathcal{T}} \exp(s_j)} eq. 6

Equation numbers follow the order of the numbered equations in the main text. The supplement has no numbered equations.

Hyperparameters

Values are from the paper unless the row says "code". Code values are read from the official repository at code_commit, and the row shows where they differ or add detail.

Results

Author-reported. Setup: Replica, LSeg teacher (except SAM), 5,000 iterations, baseline 3DGS or NeRF-DFF §4 app. D. Hardware is not stated in the LaTeX source.

Table 1, novel-view RGB quality, averaged over Replica scenes tab. 1. The values are as printed, with their standard deviations.

Table 2, semantic segmentation against NeRF-DFF on Replica tab. 2. The paper's text says the speed-up variant runs at more than double the FPS of the full model, which the table bears out (14.55 vs 6.84).

Other headline numbers, author-reported:

• Inference with dim 128 is 1.66× faster than the baseline shown in fig. 3. The caption does not name the baseline precisely, and the text compares to NeRF-DFF §4.1.
• SAM decoding from a rendered feature is up to 1.7× faster in total than rendering a novel view and passing it through the SAM encoder and decoder, with nearly equivalent masks fig. 4 §4.2.
• Editing results compare to NeRF-DFF on extraction, deletion and appearance changes, with cleaner extractions and fewer floaters in the background fig. 6 §4.3.

Dimension sweeps, from the supplement. Time is printed as mm:ss and the figure caption calls it training time app. F fig. A.

Table A, LSeg teacher, Replica segmentation, by rendered dimension tab. A:

Table B, LSeg teacher, image quality by rendered dimension tab. B:

Table C, SAM teacher, image quality and FPS by rendered dimension tab. C:

The paper picks dimension 128 for Replica because it trains 2.4× faster than 256 with similar accuracy app. F.

Paper vs code

Code: official repository at 6c570d6e7e4375129e57a588a206e6f629be2ff1, linked from the authors' project page. Its LICENSE is the Gaussian-Splatting License, which permits research use only and bars commercial use without the licensors' consent LICENSE.md LICENSE.md. This card cites the repository by path and line and does not quote it.

Where paper and code differ, or where the paper is silent:

• SAM rendered dimension. The paper sets the rendered dimension to 128 for SAM, which is half the 256 latent §4.2 app. C. The code's speed-up sets the rendered dimension to one quarter of the teacher's, so SAM gets 64 train.py gaussian_model.py. I did not check which setting produced the paper's SAM numbers.
• LSeg text encoder. The paper says LSeg uses a CLIP ViT-L/16 text encoder §4 app. C. The code's clip_vitl16_384 backbone, which the README uses for feature extraction README.md, loads CLIP ViT-B/32 lseg_vit.py. The paper also uses ViT-B/32 for the editing text encoder §4.3 app. E. The paper is internally inconsistent here.
• Iteration count. The paper's Replica runs use 5,000 iterations app. D. The code default is 30,000 [code: arguments/init.py#L76], and the README example uses 7,000 README.md. The README example is not the paper's setting.
• Resize target. The paper says the rendered feature is resized to H×W so it matches the per-pixel loss §3.2. The code resizes it to the teacher's grid with align_corners=True train.py. The paper does not mention the corner convention.
• Loss reduction. The paper writes the feature term as an L1 norm §3.2. The code takes the mean over all elements loss_utils.py, so the effective weight depends on the teacher's channel count and resolution.
• Feature initialisation. The paper does not say how the feature is initialised. The code starts it at zero gaussian_model.py.
• Densification thresholds. alg. 1 uses ε, τ_p and τ_S without values. The code uses the values listed in Hyperparameters [code: arguments/init.py#L90-L94] train.py.
• Matches. The loss weights γ = 1.0 and λ = 0.2 eq. 4 §3.2, the feature learning rate 0.001, the decoder learning rate 1e-4, the SH schedule, and the optional speed-up all agree with the code.

Pitfalls

• The speed-up is not free. The paper says the module does not affect downstream performance §3.2. The sweeps show the rendered dimension matters at the low end: LSeg at dimension 8 drops mIoU to 0.354 tab. A, and SAM at dimension 8 has the lowest PSNR in its sweep tab. C. Training time grows steeply with dimension: 128 takes 19:55 and 512 takes 1:29:42 tab. A.
• Editing scores are unscaled. Eq. 5 is a cosine, which lies in [-1, 1]. Eq. 6 applies a softmax to those cosines with no temperature, so the probabilities stay close to uniform. The edit thresholds are therefore very sensitive. The code computes the scores in fp16 [code: gaussian_renderer/init.py#L24], which adds rounding on top.
• The edit threshold is not stated. The paper gives no value. The example config uses 0.24 for a five-object list edit_extraction.yaml. Treat it as tuned per scene.
• Resolution mismatch. The teacher map (for example 360×480 for LSeg app. C) can differ from the image resolution. The rendered feature must be resized to the teacher grid, not the image grid, and the paper's wording leaves this open §3.2.
• SAM padding. The SAM teacher is computed on a padded 1024×1024 input, and only part of the 64×64 map holds content. The cropped map is 64W/H × 64 for landscape images. At inference the rendered feature must be padded back to 64×64 before the SAM decoder app. C.
• Floaters and boundaries. The authors say the 3DGS base produces noise-like floaters that limit feature quality §5. Failure cases show coarse mask boundaries and holes from weak features app. G fig. C, and tiny objects in cluttered scenes survive deletion app. G.
• Replica labels are hand-edited. Some pixels were relabelled, such as "rugs" and "floor", following NeRF-DFF app. D. Segmentation numbers depend on that labelling.
• Hardware and timing. Timings are reported, but the LaTeX source does not state the GPU or hardware used app. A §4. The FPS comparison with NeRF-DFF is only as good as the setup behind it.
• Teacher features are precomputed. The pipeline needs an extra encoder pass per training view, and the teacher features are large, around 100 to 200 MB per file README.md. Plan storage accordingly.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., 2023), whose rasterizer and optimiser are the base §3.1 app. B. Its Structure-from-Motion initialisation is taken from a separate SfM reference §3.1. None of these have a card here.
• Teacher models: SAM (Kirillov et al., 2023) for promptable segmentation §3.3 §4.2 and LSeg (Li et al., 2022) for language-driven segmentation §3.3 §4. No cards here.
• Compared against: NeRF-DFF (Kobayashi et al., 2022), the feature-field baseline in tab. 2 and fig. 5, and base 3DGS in tab. 1.
• Superseded by: not stated in the content read here.