GS3LAM: Gaussian Semantic Splatting SLAM

In plain words

A robot or camera that maps a room has to answer three questions at once: where it is, what the geometry looks like, and what each surface is. GS3LAM keeps all three in one model. The model is a cloud of small soft 3D blobs. Each blob carries a position, a size, a colour, an opacity and a short learned code that stands for a semantic class. The blobs can be drawn onto a camera image quickly, so the system compares what it would see with what the camera actually sees and then adjusts the blobs and the camera pose together.

The paper makes three points. Storing the semantic code in the same blobs as colour and shape keeps the map dense and continuous. A regulariser keeps blob sizes in a plausible range so that surfaces stay sharp. And a change in how old frames are revisited during mapping keeps the whole map consistent: refining only the frames that overlap the current one leaves some regions under-trained and others over-trained.

TL;DR

• GS3LAM models the scene as a Semantic Gaussian Field (SG-Field): 3D Gaussians with colour, depth and a low-dimensional semantic feature that a learned layer decodes to class scores with softmax §3.2 eq. 8.
• Each frame is tracked by optimising only the camera pose against photometric, depth and semantic error on an observability mask §3.4 eq. 20 eq. 21. The field is then mapped on the current frame and randomly drawn keyframes (RSKM) §3.3.3.
• Author-reported on Replica: rendering PSNR 36.26 over eight scenes tab. 2, mIoU 96.63 over four scenes tab. 4, and tracking ATE 0.37 cm against SplaTAM's 0.36 cm tab. 3.
• Use it as a reference for 3DGS-based semantic RGB-D SLAM. The released Replica config turns off the scale regulariser that the paper names as a contribution, and several loss and sampling details differ from the text (see Paper vs code).

Problem

Dense semantic SLAM needs a map that is dense, consistent and cheap to render. Explicit maps (points, surfels, meshes, voxels) are limited in resolution and cannot predict unseen areas §1 §2. Implicit NeRF-based semantic SLAM can represent continuous scenes that predict unseen areas, but its ray-traced volume rendering cannot meet real-time requirements §1 §2. 3DGS-based SLAM renders fast, but its incremental optimisation forgets earlier frames, so map quality varies from view to view. Prior work reports mean PSNR and largely ignores the variance app. B.2. The paper also finds that adding semantic features makes forgetting worse §5.

Assumptions: RGB-D input with unknown camera poses (the first pose is the identity) and one semantic label map per frame, either from the dataset or from a segmentation model §3.1 §3.3.4 app. B.1. Tracking is frame-to-model: each new frame is aligned to the current field §3.4.

Method

Scene model. The scene is a set of N semantic Gaussians. Each one has a mean, a covariance, an opacity, an RGB colour and an N_{sem}-dimensional semantic feature §3.2 eq. 1. The covariance is written as a rotation times a diagonal scale, which keeps it positive semi-definite during gradient descent §3.2 eq. 2. The released configs use isotropic Gaussians instead; see Paper vs code.

Rendering. Gaussians are projected to the image with a Jacobian of the perspective map §3.2 eq. 3. Colour, depth and semantic feature are each alpha-composited front to back over the Gaussians that overlap a pixel §3.2 eq. 4 eq. 6 eq. 7, with the alpha from the opacity and a Gaussian falloff §3.2 eq. 5. The semantic feature goes through a learned layer \mathcal{F}_{cnn} that restores K_{sem} class channels, and a softmax gives the label §3.2.2 eq. 8. The paper calls this a CNN decoder; the released code uses a per-pixel 1×1 convolution (see Paper vs code).

Decoupled optimisation. Tracking optimises the pose parameters with the field frozen. Mapping optimises the field against the estimated poses §3.2.3 eq. 9. The paper does not write out the optimiser. The released code uses Adam (see Hyperparameters).

Adaptive expansion. After a frame is tracked, it is re-rendered and the accumulated opacity is computed per pixel §3.3.1 eq. 10. Pixels that are not yet covered (low accumulated opacity, or depth that lies well in front of the current geometry) give the mask M_{unobs} eq. 11. New Gaussians are added only where this mask is true. This also keeps the Gaussian count down where the field already explains the geometry §3.3.1. Each new Gaussian takes the pixel's colour, a random semantic feature (degree-N_{sem} coefficients), the unprojected depth as its centre, opacity 0.5, and a scale of d/f §3.3.2.

Depth-adaptive scale regularisation (DSR). A penalty pushes scales above the mean plus two standard deviations and below the mean minus two standard deviations §3.3.2 eq. 12. The name says depth-adaptive, but the equation uses only global statistics over all Gaussians. Depth enters the scale only through the initial d/f.

Random-sampling keyframe mapping (RSKM). During mapping of frame t, each iteration picks one frame f from the keyframe set \mathcal{K} with probability p(f) §3.3.3 eq. 13. The paper says this avoids the keyframe selection step used by SplaTAM and Point-SLAM §3.3.3. The mapping objective is summed over k_m iterations §3.3.4 eq. 14.

Mapping loss. The colour term is an L1 loss mixed with a D-SSIM term with weight \lambda = 0.2 §3.3.4 eq. 15. The semantic term is a binary cross-entropy on the decoded labels §3.3.4 eq. 16. The depth term is an L1 loss §3.3.4 eq. 17. The scale terms from eq. 12 are added to a weighted sum §3.3.4 eq. 18.

Tracking. The initial pose of frame t comes from a constant-velocity model §3.4 eq. 19. The pose is then optimised by minimising a loss over the observable region M_{obs}, which needs high accumulated opacity and a depth error below ten times the median §3.4 eq. 20 eq. 21. Every field attribute is frozen during tracking, which the paper says separates camera motion from changes to the field §3.4.

Pseudocode. The loop below follows the paper's equations. Constants such as tracking_iters and t_opt are the values in Hyperparameters.

# One frame of GS3LAM: track the pose, add Gaussians, then map  [§3.1]
# Input:  RGB image, depth map, semantic labels of frame t
# Output: updated SG-Field (field) and pose poses[t]

def process_frame(image, depth, labels, field, poses, keyframes):
    # Initial pose from the constant-velocity model  [§3.4] [eq. 19]
    poses[t] = poses[t - 1] @ inverse(poses[t - 2]) @ poses[t - 1]

    # Tracking: optimise the pose only, field attributes frozen  [§3.4]
    for step in range(tracking_iters):
        # Observable-region mask from opacity and depth error  [eq. 21]
        mask = observable_mask(field, poses[t], depth)
        # Photometric, depth and semantic error on that mask  [eq. 21]
        loss = tracking_loss(field, poses[t], image, depth, labels, mask)
        # Gradient step on the pose only, minimising the tracking loss  [eq. 20]
        poses[t] = adam_step(poses[t], loss)

    # Unobserved mask from accumulated opacity and depth  [§3.3.1] [eq. 11]
    unobserved = unobserved_mask(field, poses[t], depth)
    # New Gaussians on unobserved pixels: opacity 0.5, scale d / f  [§3.3.2]
    field = add_gaussians(field, image, depth, labels, unobserved)

    # Mapping: refine the field on the current frame and keyframes  [§3.3.3]
    for step in range(mapping_iters):
        # Sampling weights over the keyframe set  [§3.3.3] [eq. 13]
        weights = rskm_weights(keyframes, current=t, step=step)
        # Draw one keyframe with probability p(f)  [§3.3.3] [eq. 13]
        chosen = sample_frame(keyframes, weights)
        # Colour, depth, semantic and scale-regularisation loss  [eq. 18]
        loss = mapping_loss(field, poses, chosen)
        field = adam_step(field, loss)
    return field, poses[t]


def rskm_weights(keyframes, current, step):
    # Uniform weight 1 / |K| for every keyframe  [§3.3.3] [eq. 13]
    size = len(keyframes)
    weights = [1 / size for _ in keyframes]
    # Extra weight for the current frame on every t_opt-th step  [eq. 13]
    if step % t_opt == 0:
        weights[current] += 1 - 1 / size
    return weights

Key equations. Each equation is reproduced as written in the paper. The paper gives no separate algorithm box.

\mathcal{G} := \{(\boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i, o_i, \mathbf{c}_i, \mathbf{f}_i) \mid i = 1,\dots,N\} eq. 1

\boldsymbol{\Sigma}_i = \mathbf{R}_i\mathbf{S}_i\mathbf{S}_i^{T}\mathbf{R}_i^{T} eq. 2

\boldsymbol{\Sigma}^{2D}_i = \mathbf{J}_i\mathbf{R}_{CW}\boldsymbol{\Sigma}_i\mathbf{R}_{CW}^{T}\mathbf{J}_i^{T} eq. 3

\hat{\mathbf{c}}_{pix} = \sum_i^M \mathbf{c}_i\,\alpha_i\prod_j^{i-1}(1-\alpha_j) eq. 4

\alpha_i = o_i \exp\!\left(-\tfrac{1}{2}\boldsymbol{\sigma}_i^{T}(\boldsymbol{\Sigma}^{2D}_i)^{-1}\boldsymbol{\sigma}_i\right) eq. 5

\hat{d}_{pix} = \sum_i^M d_i\,\alpha_i\prod_j^{i-1}(1-\alpha_j) eq. 6

\hat{\mathbf{f}}_{pix} = \sum_i^M \mathbf{f}_i\,\alpha_i\prod_j^{i-1}(1-\alpha_j) eq. 7

\hat{s}_{pix} = \mathrm{softmax}\!\left(\mathcal{F}_{cnn}(\hat{\mathbf{f}}_{pix})\right) eq. 8

\boldsymbol{\Theta}_{\mathcal{T}} := \{(\mathbf{q}_i, \mathbf{t}_i)\}_{i=1}^{P}, \quad \boldsymbol{\Theta}_{\mathcal{G}} := \left\{\{(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i,o_i,\mathbf{c}_i,\mathbf{f}_i)\}_{i=1}^{N}, \mathcal{F}_{cnn}(\cdot)\right\} eq. 9

\hat{o}_{pix} = \sum_i^M \alpha_i\prod_j^{i-1}(1-\alpha_j) eq. 10

M_{unobs} = \mathbb{I}(\hat{o}_{pix} < \tau_{unobs}) \lor \mathbb{I}\!\left(\hat{d}_{pix} > d_{gt} \land \mathcal{L}_1(d_{gt},\hat{d}_{pix}) > 50\,\tilde{\mathcal{L}}_1(d_{gt},\hat{d}_{pix})\right) eq. 11

\mathcal{L}_{big} = \frac{\sum_i s_i\,\mathbb{I}(s_i > s_{big})}{\sum_i \mathbb{I}(s_i > s_{big})}, \quad \mathcal{L}_{small} = \frac{\sum_i -\log(s_i)\,\mathbb{I}(s_i < s_{small})}{\sum_i \mathbb{I}(s_i < s_{small})} eq. 12

with s_{big} = \mu_s + 2\sigma_s and s_{small} = \mu_s - 2\sigma_s §3.3.2.

p(f) = \frac{1}{|\mathcal{K}|} + \left(1 - \frac{1}{|\mathcal{K}|}\right)\delta_{f,f_{cur}}\,\delta_{\mathrm{mod}(k_m,t_{opt}),0} eq. 13

\mathcal{G}^{*}_{t} = \arg\min_{\Theta_{\mathcal{G}}} \sum_{i}^{k_m} \mathcal{L}_{mapping}\!\left(\mathcal{R}(\mathbf{T}_i\odot\mathcal{G}_t), \mathcal{O}_i\right) eq. 14

\mathcal{L}^{m}_{color} = (1-\lambda)\,\|\hat{\mathbf{c}}_{pix}-\mathbf{c}_{gt}\|_1 + \lambda\left(1-\text{D-SSIM}(\hat{\mathbf{c}}_{pix},\mathbf{c}_{gt})\right), \quad \lambda = 0.2 eq. 15

\mathcal{L}_{sem} = -\left(s_{gt}\log(\hat{s}_{pix}) + (1-s_{gt})\log(1-\hat{s}_{pix})\right) eq. 16

\mathcal{L}_{depth} = \|\hat{d}_{pix} - d_{gt}\|_1 eq. 17

\mathcal{L}_{mapping} = \lambda^m_c\mathcal{L}^m_{color} + \lambda^m_d\mathcal{L}_{depth} + \lambda^m_s\mathcal{L}_{sem} + \lambda^m_{big}\mathcal{L}_{big} + \lambda^m_{small}\mathcal{L}_{small} eq. 18

\mathbf{T}_t = \mathbf{T}_{t-1}\,\mathbf{T}_{t-2}^{-1}\,\mathbf{T}_{t-1} eq. 19

\mathbf{T}^{*}_t = \arg\min_{\Theta_{\mathcal{T}}} \mathcal{L}_{tracking}\!\left(\mathcal{R}(\mathbf{T}_t\odot\mathcal{G}_{t-1}), \mathcal{O}\right) eq. 20

\mathcal{L}_{tracking} = M_{obs}\left(\lambda^t_c\mathcal{L}^t_{color} + \lambda^t_d\mathcal{L}_{depth} + \lambda^t_s\mathcal{L}_{sem}\right), \quad M_{obs} = \mathbb{I}(\hat{o}_{pix} > \tau_{obs}) \land \mathbb{I}\!\left(\mathcal{L}_1(d_{gt},\hat{d}_{pix}) < 10\,\tilde{\mathcal{L}}_1(d_{gt},\hat{d}_{pix})\right) eq. 21

with \mathcal{L}^t_{color} = \|\hat{\mathbf{c}}_{pix} - \mathbf{c}_{gt}\|_1 §3.4. \tilde{\mathcal{L}}_1 is the median of the L1 error §3.3.1.

Jacobian of the camera pose (supplement). The supplement derives the gradient of the semantic and colour losses with respect to the pose, then notes that each part can be expanded directly; the expansion is not written out app. B.1. With view-independent features, the semantic loss gradient has this form, where \mathbf{g}_i = \mathbf{K}\mathbf{T}_{CW}\boldsymbol{\mu}_i / d_i app. B.1:

\frac{\partial \mathcal{L}_{sem}}{\partial \mathbf{T}_{CW}} = \frac{\partial \mathcal{L}_{sem}}{\partial \hat{s}_{pix}}\frac{\partial \hat{s}_{pix}}{\partial \hat{\mathbf{f}}_{pix}}\frac{\partial \hat{\mathbf{f}}_{pix}}{\partial \alpha_i}\left(\frac{\partial \alpha_i}{\partial \boldsymbol{\Sigma}^{2D}_i}\frac{\partial \boldsymbol{\Sigma}^{2D}_i}{\partial \mathbf{T}_{CW}} + \frac{\partial \alpha_i}{\partial \mathbf{g}_i}\frac{\partial \mathbf{g}_i}{\partial \mathbf{T}_{CW}}\right) app. B.1

The colour-loss gradient has the same form with \mathcal{L}_{color} and \hat{\mathbf{c}}_{pix} in place of \mathcal{L}_{sem} and \hat{s}_{pix} app. B.1.

Hyperparameters

Values marked "config" come from the released code, not the paper.

Results

Author-reported. Setup: Replica (eight scenes for rendering and tracking, four for semantics) and ScanNet (six sequences in the tables) §4.1 tab. 1 tab. 4. Baselines: NICE-SLAM, Vox-Fusion, ESLAM, Co-SLAM, Point-SLAM, SplaTAM and GS-SLAM for rendering and tracking; NIDS-SLAM, DNS SLAM and SNI-SLAM for semantics §4.1. Values are averages over scenes as printed.

Ablation on Replica Office 1 tab. 5. The compiled paper gives no error bars for these runs.

The paper's RSKM and DSR claims that are not in these tables come from figures only: a 5.49 dB PSNR gain and a 24.42× lower PSNR variance for RSKM on Replica Office 3 §4.5 fig. 7. The optimisation-bias comparison between LCKM and RSKM is shown in fig. 2, with a Replica version in the supplement app. B.2 app. C.6. Qualitative comparisons are in fig. 5.

Paper vs code

Official code: lif314/GS3LAM at commit 6695e2d8e8834aefcb81ba6c317f2e1ed09e9767 (shallow clone, HEAD). It is linked from the abstract, and the README links the first author's GitHub profile README.md. The repository has no LICENSE file. Its README points to SplaTAM's license for use README.md, and some source headers say the files are outside "the overall project's MIT license" metric_utils.py. No license text is in the repository, so code_license is recorded as not stated. The code is cited by path and line only, and nothing is quoted. The rasterizer submodule gaussian-semantic-rasterization carries the Gaussian-Splatting License, which grants research use and not commercial use LICENSE.md.

Discrepancies between the paper and the code at this commit:

1. Constant-velocity pose. Eq. 19 composes matrices. The code extrapolates the rotation quaternion and translation linearly from the last two parameter values, and only when at least two earlier poses exist (frame index 2 or later) Tracker.py Tracker.py. The two agree for pure translation but differ for rotation.
2. Pose selection. The paper's argmin eq. 20 is not turned into an explicit rule. The code keeps the lowest-loss iterate of the tracking run, not the last one GS3LAM.py GS3LAM.py.
3. Scale regularisation is off for Replica. The Replica config sets use_reg_loss=False replica.py; the ScanNet config sets it to True scannet.py. The paper presents DSR as a contribution and gives no dataset condition.
4. "Depth-adaptive" name. The regulariser in the code uses the global mean and standard deviation over all Gaussians, with no depth term Loss.py, as in eq. 12. Depth enters only through the initial scale and the pruning radius Mapper.py GaussianManager.py.
5. Semantic loss. Eq. 16 is a binary cross-entropy. The code uses softmax cross-entropy over K_{sem} classes, divided by \log K_{sem} Loss.py Loss.py Loss.py.
6. Semantic decoder. The paper calls it a CNN §3.2.2. The code uses one 1×1 convolution from N_{sem} to K_{sem} channels, which has no spatial context Decoder.py. A multi-layer decoder class exists in the same file but is not used Decoder.py.
7. Gaussian shape. The paper uses a diagonal scale matrix §3.2 eq. 2. The released Replica and ScanNet configs use isotropic Gaussians replica.py scannet.py. The rendering code supports both Render.py.
8. RSKM sampling. Eq. 13 gives the current frame an extra term only when \mathrm{mod}(k_m, t_{opt}) = 0. The code does something different. It uses the current frame on every opt_rskm_interval-th step and on the first frame, and otherwise draws a keyframe uniformly from a list that does not yet contain the current frame GS3LAM.py. The list gets a frame every fifth frame, plus frame 0 and frame N-2 GS3LAM.py.
9. Colour loss. The code uses 0.8\,\mathrm{L1} + 0.2\,(1 - \mathrm{SSIM}) Loss.py. The paper writes 1 - \text{D-SSIM} and does not define D-SSIM, so the two can differ by the convention used.
10. Reductions. The tracking depth and colour losses are sums over the masked pixels; the mapping depth loss is a mean Loss.py Loss.py Loss.py. The paper says nothing about reductions, and the loss weights are only meaningful with the code's reductions.
11. ScanNet iteration counts. The supplement gives 100 tracking and 30 mapping iterations app. C.1. The ScanNet configs use 200 and 60 scannet.py. The ScanNet run script uses the same config eval_full_scannet.bash.
12. Settings only in code. Densification gating, pruning, the semantic decoder learning rate, the ScanNet camera learning rate and the ScanNet densification threshold of 0.5 are not in the paper GS3LAM.py scannet.py.
13. Labels. The Replica loader reads the dataset's semantic-class images replica.py. The repository contains no segmentation model. The paper allows labels from either source §3.3.4.
14. Unused keyframe selection. The window-based keyframe selection (mapping_window_size) is commented out GS3LAM.py. The active rule is the fixed interval above.

Pitfalls

• "Real-time" means rendering here. The paper's 106.56 FPS (Office 0) and 109.12 FPS (average) are rendering speeds app. C.2. In the same table, mapping takes 4.29 s and tracking takes 3.01 s per frame on Office 0, a sum of 7.30 s. Tracking per frame is the largest in that table, above SplaTAM's 2.82 s. Do not read the "real-time" claim as a full SLAM loop rate.
• The Replica config turns off the scale regulariser. The paper names DSR as a contribution and its Replica numbers do not say whether DSR was on §4.5 replica.py. Decide which setting to reproduce and state it.
• Eq. 13 does not sum to one. When the current frame is one of the keyframes and the condition holds, its weight is 1 and every other keyframe keeps 1/|\mathcal{K}|, so the weights sum to more than one. Whether the current frame belongs to \mathcal{K} during its own mapping is not stated. The code avoids the question by not including it GS3LAM.py.
• Missing values in the compiled paper. N_{sem}, K_{sem}, \tau_{obs}, \tau_{unobs}, t_{opt}, the keyframe interval, k_{init} on ScanNet and the optimiser are not in the compiled paper. A commented-out line in the LaTeX source (main.tex line 518, not in the PDF) gives N_{sem}=16, K_{sem}=256, \tau_{unobs}=0.5, \tau_{obs}=0.99, t_{opt}=10, and k_{init} of 1000 for Replica and 500 for ScanNet. The released code uses t_{opt}=5 and \tau_{unobs}=0.1 on Replica, so the commented values are not the released ones.
• Gradients for the pose are left unexpanded. The supplement's Jacobian is written as chain-rule terms, and the paper says each can be expanded directly, without giving the expansion app. B.1. An implementer must derive the terms.
• Text and table mismatches. (a) The text says five ScanNet subsets §4.1; the tables show six sequences tab. 1 app. C.3 and the config lists six scenes. (b) The text gives a 0.039 LPIPS gap to the runner-up on Replica §4.2. The table gives 0.030 against GS-SLAM (0.082 − 0.052) and 0.072 against Point-SLAM tab. 2. (c) The text gives a 2.22 dB PSNR gain for M_{unobs} §4.5; the ablation table gives 2.11 dB (41.21 − 39.10) tab. 5. (d) The text gives a 3.36% mIoU gain for DSR; the table gives 3.39 points (97.35 − 93.96) §4.5 tab. 5. (e) The ESLAM LPIPS average is printed as 0.336 in tab. 2; its eight per-scene values average about 0.239.
• Tracking is not the best. The paper calls tracking "competitive". On Replica, SplaTAM's ATE average is 0.36 against 0.37 tab. 3. On ScanNet, NICE-SLAM's average is 10.70 against 11.24 app. C.3.
• Semantic results depend on the labels. The paper notes that ScanNet labels are imprecise app. C.5. The code's mIoU is computed against the dataset's masks Evaluater.py.
• Figures carry some claims. The RSKM gains of 5.49 dB and 24.42× variance appear only in fig. 7 (Office 3). The text gives no table for them.
• Rasterizer licence. The rasterizer submodule is under the Gaussian-Splatting License, which does not allow commercial use LICENSE.md. Reuse of the code for a commercial product needs a different rasterizer or permission from the licensors. This card quotes no code.
• Seeds. The repository's scripts loop over three seeds per scene eval_full_replica.sh. The paper does not say whether the tables average over seeds.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., TOG 2023) for the rendering and the covariance parameterisation §3.2 eq. 2. Not a card in this wiki.
• Builds on: SplaTAM (arXiv:2312.02126) for frame-to-model tracking with a frozen field, and its local co-visibility keyframe mapping (LCKM), which RSKM replaces §3.4 §3.3.3 app. B.2. Not a card in this wiki.
• Builds on: MonoGS (arXiv:2312.06741), which also uses LCKM §3.3.3. Not a card in this wiki.
• Builds on: random pixel and frame sampling from NeRF-based SLAM, adapted to 3DGS app. B.2. Not a card in this wiki.
• Compared against: NICE-SLAM, Vox-Fusion, ESLAM, Co-SLAM, Point-SLAM, SplaTAM and GS-SLAM (arXiv:2311.11700) for rendering and tracking §4.1; NIDS-SLAM (arXiv:2304.14560), DNS SLAM and SNI-SLAM for semantics §4.1; concurrent semantic SLAM work (SGS-SLAM, SemGauss-SLAM, NEDS-SLAM) app. C.6.
• Superseded by: not known from wiki content.