Gaussian Grouping: Segment and Edit Anything in 3D Scenes

In plain words

A Gaussian Splatting scene is built from many small coloured blobs (3D Gaussians) that can be drawn quickly from new viewpoints. The blobs capture how the scene looks, but not which blob belongs to which object. This paper adds a small label vector to every blob. The labels are learned from the object masks that the Segment Anything model finds in ordinary photos of the scene. The masks are first matched across photos, so that one object keeps the same ID in every view.

Because each blob carries its label, the scene splits into groups that can be treated as units. A group can be deleted or moved, and its colour can be changed, without retraining the whole scene. Filling the hole left by a deleted object needs a short extra fine-tuning step.

The authors report that the segmentation comes at little cost to rendering quality, and that it is faster than the NeRF-based methods they compare against. The method is limited to static scenes.

TL;DR

• Adds a 16-d identity encoding to every 3D Gaussian and learns it by differentiable rendering §3.2 (c) eq. 1.
• Supervision comes from SAM masks, tracked across views with DEVA, plus a 3D kNN KL-consistency loss §3.2 (a)–(d) eq. 3.
• Grouped Gaussians support removal, recolouring, recomposition, inpainting and style transfer §3.3.
• Relevant if you need object-level segmentation or editing on a reconstructed 3DGS scene; not for dynamic scenes §4.4.

Problem

Gaussian Splatting gives fast, high-quality novel-view synthesis but has no object or stuff-level understanding §1 §3.1. NeRF-based methods that lift 2D masks or distil CLIP/DINO features are slow to optimise, and their MLPs are hard to edit locally §2. Lifting SAM to 3D with NeRFs usually handles only one or a few objects §2 "SAM in 3D". Gaussian Grouping targets automatic "segment everything" mode for a full scene, without 3D labels §1 §3.

Method

Overview. Each Gaussian keeps its usual attributes (centre, size, rotation, opacity, SH colour). A new identity encoding e_i is added: a learnable vector of length 16, using SH degree 0 because the identity should not vary with view §3.2 (c). Training renders these encodings alongside colour, then supervises them with a 2D identity loss and a 3D regularisation loss §3.2 (d) eq. 4 fig. 2.

1. Input masks. Run SAM in everything mode on each training view independently §3.2 (a). Associate masks across views by treating the views as a video and using the DEVA tracker, which also gives the total number of identities K §3.2 (b). The authors prefer this to the cost-based linear assignment of Panoptic Lifting, reporting it is over 60× faster and avoids per-iteration matching §3.2 (b). In their ablation, 2K iterations take 1 hour with linear assignment and 1 minute with associated masks §4.2.

2. Identity rendering. The identity feature of a pixel is the 3DGS alpha-blend of the encodings of the Gaussians along the ray, in depth order §3.2 (c) eq. 1. The per-Gaussian 2D weight comes from the projected covariance eq. 2 §3.2 (c).

E_{\text{id}} = \sum_{i \in \mathcal{N}} e_i \, \alpha'_i \prod_{j=1}^{i-1} (1 - \alpha'_j) eq. 1

\Sigma^{\text{2D}} = J W \Sigma^{\text{3D}} W^{T} J^{T} eq. 2

Here \alpha'_i is the 2D Gaussian weight times the learned opacity, \Sigma^{\text{3D}} and \Sigma^{\text{2D}} are the 3D and projected covariances, J is the Jacobian of the affine projection approximation, and W is the world-to-camera transform eq. 1 eq. 2 §3.2 (c).

3. 2D identity loss. A linear layer f maps the 16-d rendered feature back to K logits. Softmax and cross-entropy against the associated mask IDs give \mathcal{L}_\text{2d} §3.2 (d). The released code uses a 1×1 convolution with 256 output channels train.py train.json.

4. 3D regularisation loss. For m sampled Gaussians P, take the k nearest Gaussians Q by 3D Euclidean distance. The loss is the KL divergence between the softmax distributions F(e) of each sample and its neighbours, averaged over the m \cdot k pairs §3.2 (d) eq. 3. The authors' motivation is that Gaussians inside an object, or hidden in nearly all views, get little 2D supervision §3.2 (d) §4.2 fig. 7.

\mathcal{L}_\text{3d} = \frac{1}{m} \sum_{j=1}^{m} D_{\text{kl}}(P \,\|\, Q) = \frac{1}{mk} \sum_{j=1}^{m} \sum_{i=1}^{k} F(e_j) \log\left(\frac{F(e_j)}{F(e'_i)}\right) eq. 3

Here P holds the encoding e_j of a sampled Gaussian and Q=\{e'_1,\dots,e'_k\} holds the encodings of its k nearest neighbours. F is the softmax after the shared linear layer f §3.2 (d) eq. 3.

5. Total loss. The reconstruction loss \mathcal{L}_\text{rec} is the standard 3DGS image loss. The total is \mathcal{L}_\text{rec} + \lambda_\text{2d} \mathcal{L}_\text{2d} + \lambda_\text{3d} \mathcal{L}_\text{3d} eq. 4 §3.2 (d). All parts are trained jointly end to end; unlike Panoptic Lifting, the segmentation gradients are not blocked from the reconstruction branch §4.2.

6. Density control. Algorithm 1 includes pruning and densification steps alg. 1. The text says the density-control details are inherited from 3DGS and omitted from Figure 2 fig. 2 caption.

# Gaussian Grouping training  [alg. 1]
# Input:  views with images and mask ids (SAM + DEVA), SfM points
# Output: Gaussians with 16-d identity encodings, linear layer f

def train(views, sfm_points, num_classes, reg_interval, num_iterations):
    # Init Gaussians from SfM points; identity encodings start random
    # [alg. 1] [code: scene/gaussian_model.py#L139]
    gaussians = init_gaussians(sfm_points, identity_dim=16)
    # Linear layer f maps identity features to K class logits
    # [§3.2 (d)] [§4.1.2]
    classifier = linear(in_dim=16, out_dim=num_classes)
    for iteration in range(num_iterations):  # 30K in the paper [§4.1.2]
        # Sample a training view with its image and mask  [alg. 1]
        view = sample_view(views)
        # Splat colours and identity encodings with the 3DGS rasteriser
        # [eq. 1] [eq. 2] [§3.2 (c)]
        image, identity_map = rasterize(gaussians, view)  # (3,H,W), (16,H,W)
        # Reconstruction loss from 3DGS; weights not stated in the paper
        # [eq. 4] [code: train.py#L102]
        loss = reconstruction_loss(image, view.image)
        # 2D identity loss: per-pixel cross-entropy over mask ids
        # [§3.2 (d)] [eq. 4]; lambda_2d = 1.0 [§4.1.2]
        logits = classifier(identity_map)  # (H, W, num_classes)
        loss = loss + 1.0 * cross_entropy(logits, view.mask_ids)
        # 3D regularisation on a schedule (interval not stated in paper)
        # [alg. 1] [code: train.py#L97]
        if iteration % reg_interval == 0:
            # Softmax over classes for every Gaussian  [§3.2 (d)] [eq. 3]
            probs = softmax(classifier(gaussians.identity))  # (N, C)
            # lambda_3d = 2.0  [app. More implementation details]
            loss = loss + 2.0 * knn_kl_loss(gaussians.positions, probs)
        # Adam step on Gaussian attributes and the linear layer  [§4.1.2]
        optimizer.step(loss)
        # Densify and prune as in 3DGS  [fig. 2 caption]
        densify_and_prune(gaussians, iteration)
    return gaussians, classifier


def knn_kl_loss(positions, probs, k=5, m=1000):
    # Sample m Gaussians (P) and find each one's k nearest neighbours (Q)
    # by 3D position  [eq. 3] [§4.1.2]
    sampled = random_indices(len(positions), count=m)
    neighbours = knn(positions[sampled], positions, k)  # (m, k)
    p = probs[sampled]  # (m, C)
    q = probs[neighbours]  # (m, k, C)
    # KL divergence per (sample, neighbour) pair, summed over classes  [eq. 3]
    kl = p[:, None] * (log(p[:, None]) - log(q))  # (m, k, C)
    # Average over the m*k pairs; the code also divides by C, eq. 3 does not
    # [code: utils/loss_utils.py#L109-L113]
    return kl.sum(dim=-1).mean() / probs.shape[1]

Scene editing (§3.3). Editing acts on groups and leaves the rest of the scene frozen §3.3.

• Removal: delete the Gaussians of the target. No parameter tuning is needed §3.3.
• Recomposition: swap the 3D centres of two groups. No tuning is needed §3.3.
• Colourisation: tune only the SH colour parameters of the group, which keeps the geometry §3.3.
• Inpainting: delete the target, find the hole as the region that no remaining Gaussian projects onto, and inpaint each view with LaMa. Clone 200K new Gaussians near the deletion region. Fine-tune only the new Gaussians, with L1 loss outside the object mask and LPIPS loss inside its bounding box §3.3 app. 3D Object Inpainting Pipeline.
• Style transfer: fine-tune the group's Gaussians except their positions, using InstructPix2Pix-edited target images. Losses are restricted to the object mask (L1 inside, LPIPS in its bounding box) §3.3 app. 3D Object Style Transfer Pipeline.

Hyperparameters

Results

Author-reported. Setups are given per row. No standard deviations or repeated runs are reported in any table.

The mBIoU column in tab. 3 is also highest for Gaussian Grouping in all three LERF-Mask scenes.

Paper vs code

Checked against the official repository at code_commit (Apache-2.0).

• Reg loss weight in the text. Paper §4.1.2 says \lambda_\text{2d} = 2.0 in the text; the code and the appendix use \lambda_\text{3d} = 2.0 [code: arguments/init.py#L94].
• Reg loss schedule. Algorithm 1 adds \mathcal{L}_\text{id} at every iteration alg. 1. The code applies the 3D term only every reg3d_interval iterations. The default is 2 [code: arguments/init.py#L92] and the released dataset config sets 5 train.json train.py.
• Point subsample. The paper says downsample to 300K app. More implementation details. The code default is 300K [code: arguments/init.py#L95], but the released dataset config sets 200K train.json.
• Normalisation. Code divides the 2D cross-entropy by log(num_classes) train.py. The 3D loss is also divided by num_classes loss_utils.py. Neither is in the paper's equations.
• Neighbours include the sample. The top-k search runs over all points, so each sampled Gaussian is its own nearest neighbour loss_utils.py. With k=5 that leaves four other neighbours. The paper does not mention this.
• Neighbour positions are detached. Neighbours are found from detached positions, so gradients flow only through the predicted distributions train.py.
• Densification length. Paper says "same as 3DGS" fig. 2 caption. The released dataset config stops densification at 10K iterations train.json; the code default is 15K [code: arguments/init.py#L89].
• Class count. The paper uses 256 outputs §4.1.2. The code default is 200 [code: arguments/init.py#L58] train.py; the released configs and scripts pass 256 train.json.
• Consistent parts. The identity dimension (16), SH degree 0, learning rates (2.5e-3 and 5e-4), k=5, m=1000 and 30K iterations match between paper and code §4.1.2 [code: arguments/init.py#L75] [code: arguments/init.py#L80] train.py.

Pitfalls

• Unclear 3D loss weight. The main text has a typo for \lambda_\text{2d} §4.1.2. Use \lambda_\text{3d} = 2.0 app. More implementation details [code: arguments/init.py#L94].
• Unstated regularisation schedule. The paper gives no interval. Applying it every iteration (as Algorithm 1 suggests) is more expensive than the released config.
• Memory cost of the 3D loss (inferred from shapes). Each call computes an m \times N distance matrix loss_utils.py. With m=1000 and N=300K in float32 that is about 1.2 GB per call, and it is recomputed from current positions each time.
• Fixed class width. Released configs fix the classifier at 256 classes train.json. The paper defines K as the number of masks from tracking §3.2 (b) but does not say what happens when K exceeds the class width.
• Mask quality matters most. The repo docs say the mask quality largely affects the results train.md. DEVA association errors also limit the open-world segmentation app. Limitation. Multi-view training can correct some input errors fig. 5 §4.2.
• Dimension claim has no matching figure. The text on the 16-d choice cites fig. 4, but fig. 4 shows the mask-association ablation §4.2. The 32-d comparison (1.3× slower, no better reconstruction) has no table or figure of its own.
• Regularisation k tuned on one scene. K=5 was selected on the Kitchen scene §4.2 tab. 2 and is used for all datasets §4.1.2.
• No semantic labels. Groups are anonymous IDs. Text queries need Grounding DINO to pick a mask §4.3 app. Limitation. Complex prompts can select the wrong mask, and the approach then gives no category name app. Limitation.
• Panoptic results depend on semantics. Both methods use the same semantic labels, generated per view by Mask-DINO, for the comparison §4.3, so the mIoU reflects semantic labels as much as grouping.
• Editing evaluation is thin. Editing is judged by CLIP direction similarity on one prompt pair per task app. Quantitative Evaluation of Editing tab. 5, plus visual comparison §4.4. No user study is reported.
• Inpainting pipeline is multi-stage. It needs Grounding DINO hole masks, DEVA association and LaMa per view app. 3D Object Inpainting Pipeline. Reported time is 1 h plus 20 min of fine-tuning on one A100 fig. 10.
• Static scenes only. The method has no time-dependent modelling §4.4 Limitation.
• Cost of grouping. Adding the 16-d encoding lowers Mip-NeRF 360 PSNR by 0.26 dB and FPS from ~200 to ~170 tab. 1.
• Timing comparisons. The paper says the SA3D timing was measured in the same running environment app. Segmentation Efficiency. No other timing setup is described.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., 2023) for the rasteriser and density control §3.1 fig. 2 caption; the DEVA tracker (Cheng et al.) for mask association §3.2 (b); SAM (Kirillov et al., 2023) for masks §1.
• Compared against: Panoptic Lifting §4.3 tab. 4; LERF, SA3D, LangSplat, DEVA on LERF-Mask tab. 3; Distilled Feature Fields (DFF) for removal §4.4 tab. 5; SPIn-NeRF and Instruct-NeRF2NeRF for inpainting and style transfer §4.4 tab. 5.
• Superseded by: not known from the content of this wiki.