SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM

In plain words

A SLAM system is the part of a robot or headset that works out where the camera is while it builds a map of the room. SGS-SLAM builds that map out of many small soft blobs in 3D, called Gaussians. Each blob stores how it looks, how far away it is, and a label-like colour that says what kind of object it belongs to.

Because every blob carries that object colour, the map can separate one object from another. The system uses this to pick better moments to refine the map and to move or remove a group of objects as a unit. The authors report that this gives sharper object edges than earlier neural-field SLAM systems, while still rendering fast.

TL;DR

SGS-SLAM is an RGB-D SLAM system built on 3D Gaussian Splatting, with an extra semantic colour per Gaussian. It tracks the camera and maps the scene with the same three-channel loss. Keyframes are chosen partly by a semantic check. On Replica the authors report the best mapping quality and a competitive tracking error, but a slow full-system frame rate. Check the Pitfalls before relying on the numbers.

Problem

Neural implicit SLAM systems (MLP-based) over-smooth object edges, forget earlier regions when the scene grows, and are slow to update §1. Gaussian-based SLAM is fast, but earlier Gaussian SLAM systems had no semantic channel §2.3. This paper assumes that depth and a 2D semantic image are available for every frame §5, Limitations.

Method

The map is a set of isotropic 3D Gaussians. Each has a position, a colour, an opacity, a radius and a semantic colour §3.1 eq. 1. Each channel is rendered by the same splatting and front-to-back compositing: colour eq. 3, depth eq. 4 and semantic colour eq. 5. A silhouette, the depth rendered with each Gaussian's depth set to 1, says whether a pixel is covered by the map §3.1.

The system runs in two steps per frame, tracking and mapping §3.2. Tracking fixes the Gaussians and optimises only the camera pose §3.2.1 eq. 6. Keyframes are stored every few frames and selected for the mapping window using overlap and semantic checks §3.2.2 eq. 7. Mapping adds Gaussians where the map does not yet explain the frame, then optimises the Gaussians with the camera pose fixed §3.2.3 eq. 8 eq. 9. Scene editing selects Gaussians by their semantic label and applies a visibility mask and a rigid transform §3.3 eq. 10.

# SGS-SLAM, one frame  [§3.2]  [code: scripts/slam.py#L766-L1040]
# Input:  rgb (3, H, W); depth (1, H, W); sem (3, H, W) colours
# Output: pose of this frame; updated Gaussians; keyframe list

def process_frame(rgb, depth, sem, params, keyframes, t):
    # Start from a constant-velocity guess (from the third frame on)
    # [§3.2.1]  [code: scripts/slam.py#L481-L500]
    pose = extrapolate(params.poses[t - 1], params.poses[t - 2])
    best_pose = pose
    best_loss = inf
    # Track: optimise the pose only, Gaussians fixed  [§3.2.1]  [eq. 6]
    for step in range(tracking_iters):
        loss = tracking_loss(params, pose, rgb, depth, sem)
        pose = update(pose, grad(loss, pose))
        # Keep the candidate pose with the lowest loss seen
        # [code: scripts/slam.py#L803-L806]
        if loss < best_loss:
            best_loss = loss
            best_pose = pose
    pose = best_pose
    # Store a keyframe every 5th frame  [§3.2.2]
    # [code: scripts/slam.py#L1023]
    if t % 5 == 0:
        keyframes.append(make_keyframe(t, pose, rgb, depth, sem))
    # Pick window-2 earlier keyframes that overlap this view  [§3.2.2]
    # [eq. 7]  [code: utils/keyframe_selection.py#L40-L110]
    chosen = select_overlapping(keyframes[:-1], depth, pose, window - 2)
    chosen.append(keyframes[-1])
    chosen.append(current_frame(t, pose, rgb, depth, sem))
    # Densify where the silhouette is low or the frame shows a new object
    # [§3.2.3]  [code: scripts/slam.py#L421-L478]
    params = densify(params, rgb, depth, sem, pose, sil_thres=0.5)
    # Map: optimise Gaussians on random chosen frames, pose fixed
    # [§3.2.3]  [eq. 8]  [eq. 9]
    for step in range(mapping_iters):
        frame = random_choice(chosen)
        loss = mapping_loss(params, frame)
        params = update(params, grad(loss, params))
    return pose, params, keyframes

Tracking loss, per frame eq. 6:

# Tracking loss: only pixels with a confident silhouette  [§3.2.1]
# [eq. 6]  [code: scripts/slam.py#L249-L332]
def tracking_loss(params, pose, rgb, depth, sem):
    rendered = render_all_channels(params, pose)
    # Silhouette above T_sil = 0.99 and a valid depth value  [§3.2.1]
    mask = (rendered.silhouette > 0.99) & (depth > 0)
    # Depth and colour: L1 summed over the mask  [code: L313, L321]
    # Weights 1.0 depth, 0.5 colour, 0.05 semantic  [§ Experiment Settings]
    loss_depth = abs(depth - rendered.depth)[mask].sum()
    loss_color = abs(rgb - rendered.rgb)[mask].sum()
    loss_sem = abs(sem - rendered.sem)[mask].sum()
    return 1.0 * loss_depth + 0.5 * loss_color + 0.05 * loss_sem

Mapping loss, per selected frame eq. 8, eq. 9:

# Mapping loss: depth, colour and semantic terms  [§3.2.3]
# [eq. 8]  [eq. 9]  [code: scripts/slam.py#L329-L332]
def mapping_loss(params, frame):
    rendered = render_all_channels(params, frame.pose)
    # Depth: L1 averaged over valid pixels  [code: scripts/slam.py#L315]
    loss_depth = mean_abs(frame.depth - rendered.depth)
    # Colour and semantic: 0.8 * L1 + 0.2 * (1 - SSIM)  [eq. 9]
    # The paper leaves alpha unstated; the code uses 0.8
    loss_color = 0.8 * l1(frame.rgb, rendered.rgb)
    loss_color += 0.2 * (1 - ssim(frame.rgb, rendered.rgb))
    loss_sem = 0.8 * l1(frame.sem, rendered.sem)
    loss_sem += 0.2 * (1 - ssim(frame.sem, rendered.sem))
    # Paper multiplies the whole sum by the uncertainty U_t  [§3.2.2]
    # The released loss does not (see Paper vs code).
    return 1.0 * loss_depth + 0.5 * loss_color + 0.1 * loss_sem

Gaussian state §3.1 eq. 1:

• mean (3), colour (3), unnormalised rotation quaternion (4), logit opacity (1), one log-scale (1, isotropic), semantic colour (3).
• Initial values in code: identity rotation, opacity logit 0 (opacity 0.5), log-scale from the projected mean squared distance, camera pose identity for frame 0 and zero translation slam.py.
• The paper does not state the initial opacity, the initial scale rule or the camera pose initialisation.

Hyperparameters

Values are shown for the dataset where the paper or the released config states them. "Not stated" means neither the paper nor its code gives the value for that setting.

Results

Author-reported. Setup: the Replica dataset for mapping and tracking, four scenes for semantic mIoU, ScanNet and ScanNet++ for tracking and novel views. The baselines' numbers are cited as reported in their papers or in tab. 2 and the supplement.

Replica, training-view rendering, average over eight scenes tab. 1:

Replica tracking and speed, average over eight scenes tab. 2:

Replica semantic mIoU, training view, four scenes tab. 3: NIDS-SLAM 82.37, DNS-SLAM 84.77, SNI-SLAM 87.41, SGS-SLAM 92.72 (Room0 92.95, Room1 92.91, Room2 92.10, Office0 92.90).

Replica ATE RMSE per scene (supplement, tab. 6): SGS-SLAM average 0.41 cm, with ESLAM 0.63, Point-SLAM 0.52, Co-SLAM 0.86, NICE-SLAM 2.50 and Vox-Fusion 3.09 as reported.

ScanNet ATE RMSE, six scenes (supplement, tab. 7): SGS-SLAM average 9.87 cm. ESLAM is listed at 7.88 and Co-SLAM at 9.73 in the same table. The paper's table marks ESLAM as best on average.

ScanNet++ ATE RMSE, two scenes (supplement, tab. 8): SGS-SLAM average 1.62 cm; ESLAM 170.06 cm, which the paper says failed to track.

Ablation on ScanNet scene0000_00 tab. 4: removing colour, depth, semantic or the silhouette threshold worsens the full system's mIoU (70.27%) to 68.19, 54.52, n/a and 28.07 respectively, and ATE RMSE (11.26 cm) to 24.59, 40.47, 13.81 and 357.48. Removing the semantic channel makes mIoU inapplicable.

Keyframe ablation on scene0000 tab. 5: without T_geo, ATE RMSE is 15.55 cm (full 11.26); without T_sem, 12.89 cm; without U, 11.43 cm.

Scene editing (removing or moving a group of Gaussians by label) is shown in fig. 4. The paper gives no quantitative editing metric.

Paper vs code

Repository: https://github.com/ShuhongLL/SGS-SLAM at commit e4183986204242a8bb422624618af07780a49d26 (main branch, shallow clone). The abstract links it, which is how this card treats it as the authors' repository; the owner handle was not checked against the author list. The supplement says "the code will be released soon", which is out of date. Code licence: BSD-3-Clause in the root LICENSE (copyright Nikhil Varma Keetha, 2023). Three utility files, utils/slam_external.py, utils/gs_external.py and utils/graphics_utils.py, carry a header restricting use to non-commercial research and evaluation, so this card cites the code and does not quote it.

• Tracking and mapping iterations differ. Paper: ScanNet 120/40, ScanNet++ 220/50 § Experiment Settings. Code: ScanNet 100/30 (slam.py), ScanNet++ 200/60 (slam.py). Replica matches (40/60).
• Keyframe selection is different. Paper: a geometric overlap η = 0.05 and a semantic mIoU filter T_sem = 0.7, with up to 25 keyframes §3.2.2 § Experiment Settings. Code: sample 1600 valid depth pixels of the current frame, back-project them with the estimated pose, count the share landing inside each keyframe with a 20-pixel margin, keep keyframes with any overlap, shuffle, and take window-2 of them (keyframe_selection.py). The semantic filter is commented out (keyframe_selection.py). The paper samples Gaussians, not depth pixels.
• Uncertainty weighting is absent. The paper weights the mapping loss by U_t = exp(−τt) §3.2.2 eq. 8. The code computes an uncertainty term only to mask NaNs (slam.py); no exp decay or τ appears in the loss.
• Depth loss normalisation differs. Tracking depth and colour are L1 sums over the mask (slam.py, slam.py). Mapping depth is a mean (slam.py). The paper gives no normalisation.
• Semantic weight in tracking is 0 on ScanNet. The paper gives λ_S = 0.05 for tracking § Experiment Settings. The ScanNet config sets seg = 0 and comments out 0.05 (slam.py).
• Camera learning rates are per dataset. The paper states 2e-3 for the transition. Code: Replica 2e-3 translation and 4e-4 rotation; ScanNet 5e-4 both; ScanNet++ 4e-3 translation and 1e-3 rotation (see Hyperparameters).
• Densification condition has an extra factor. The paper adds Gaussians where the silhouette is low or the GT depth is "much smaller" than the rendered depth §3.2.3. Code adds them where the silhouette is below 0.5, or where the rendered depth exceeds GT and the depth error is more than 50 times its median (slam.py).
• Pruning is in the code, not in the paper. The mapping loop removes Gaussians with opacity below 0.005 at mapping iterations 0 and 20 for Replica and ScanNet (slam.py, slam_external.py).
• Tracking keeps the best pose seen. Code returns the pose with the lowest loss across all tracking iterations (slam.py). The paper does not describe this selection.
• Constant-velocity start. Code uses the velocity model only from the third frame; earlier frames copy the previous pose (slam.py). The paper gives the rule without a start index.
• Mapping SSIM mix. The paper's α is unstated. Code uses 0.8 L1 + 0.2 (1 − SSIM) for colour and semantic terms (slam.py).
• ScanNet++ optional tracking extension. The ScanNet++ config doubles tracking iterations once if the depth loss stays above 20000 (slam.py, slam.py). The paper does not describe it.
• Rasteriser is a submodule. diff_gaussian_rasterization is imported from a git submodule that points to a third-party fork (slam.py, .gitmodules). It is not in this clone.
• Optimiser reset per stage. Code creates a fresh optimiser for the tracking step and again for the mapping step of each frame (slam.py, slam.py). The paper does not say whether optimiser state persists.

Pitfalls

• Semantic labels are assumed given. The method needs a 2D semantic image for every frame. The paper says these come with datasets or off-the-shelf models §3.1; the released loop reads them from the dataset loader (replica.py). The paper's limitations section says system quality drops when depth or semantics are scarce §5.
• Semantic colour is optimised like an RGB value. Each Gaussian's semantic colour is an RGB-style triple that is rendered and compared with the 2D semantic image §3.1 eq. 5. The paper does not say how a rendered colour is mapped back to a class label for the mIoU §4.3.
• Memory grows with the scene. The paper reports large memory use on large scenes §5; the typical use is under 12 GB on the reported scenes § Experiment Settings.
• Speed claims need care. The abstract says the system renders in real time. Table 2 gives a SLAM frame rate of 2.11 FPS and a tracking rate of 5.27 FPS, below ESLAM and Co-SLAM tab. 2.
• Headline claims are narrower than the tables. The text says "outperforming them by a margin of 10dB in PSNR" §4.2; that is against the NICE-SLAM average (24.42). Against SplaTAM, the next-best row, the gap is 0.68 dB tab. 1. The paper also says tracking is state of the art, but the ScanNet table lists ESLAM as better on average (7.88 vs 9.87 cm) tab. 7.
• Ground-truth novel views have holes. The ground truth comes from offline meshes with holes, so novel-view numbers include ground-truth defects § Experiment Settings fig. 3.
• Scene editing leaves holes. Removing or moving an object leaves unobserved regions empty, because the Gaussians were only fit to the trajectory views § Additional Experiment Results.
• Pixel-level numerical care. The depth loss uses mean in mapping and sum in tracking. Changing one without the other changes the effective step size, and the learning rates above were tuned for the sums.
• Overlap threshold is inactive. Because η and T_sem are not used in code, reproducing the paper's keyframe rule means implementing them yourself; the code's rule is different (see Paper vs code).
• Ablation scene only. The keyframe and channel ablations use one ScanNet scene tab. 4 tab. 5.

Lineage

• Builds on: SplaTAM, which the paper follows for isotropic Gaussians and silhouette-based visibility §3.1 eq. 1; the rendering method of Luiten et al. §3.1.
• Compared against: NICE-SLAM, Co-SLAM, ESLAM, Vox-Fusion, Point-SLAM (supplement) and SplaTAM for mapping and tracking; NIDS-SLAM, DNS-SLAM and SNI-SLAM for semantic mIoU §4.1.
• Superseded by: not known from content already in this wiki.