SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM

In plain words

A SLAM system has to answer two questions at once: where is the camera, and what does the room look like? SplaTAM answers both with one model of the scene made of many small soft blobs floating in space. Each blob has a colour, a position, a size and a transparency. To locate the camera for a new depth-and-colour frame, the system draws the blobs as the camera would see them and nudges the camera until the drawing matches the frame. The blobs are then held still while the next step runs. Where the frame shows something the map does not yet cover, new blobs are added. Keyframes that overlap the current view are then replayed to refine the blobs. Because drawing blobs is fast and the whole process is differentiable, the camera pose and the map can be improved by gradient descent. The method reports better tracking than earlier dense methods on several indoor datasets, and it can also render views the camera never took.

TL;DR

• Dense RGB-D SLAM whose map is a set of 3D Gaussians, optimised online through a differentiable splatting renderer §1 [§3 Gaussian Map Representation].
• Per frame: track the camera pose with the map fixed, densify the map where it is empty or incomplete, then refine the map on a window of keyframes with poses fixed §3 SLAM System.
• Author-reported: best average tracking error among the compared dense methods on Replica and TUM-RGBD, and about 1.2 cm on ScanNet++ where Point-SLAM and ORB-SLAM3 fail to track tab. 1.
• Its training-view rendering on Replica is not better than Point-SLAM, which uses ground-truth depth to sample its map tab. 2 §5.

Problem

Dense SLAM needs a map that can be tracked against, edited, and that explains unseen views. The paper argues that explicit point, surfel and signed-distance maps depend on rich geometric features and high frame rates for tracking, and explain only observed parts of the scene §1. Implicit neural-field SLAM methods can render new views, but the paper says they are slow to optimise, hard to edit, do not model geometry explicitly, and forget earlier content §1. The assumed input is a single unposed RGB-D camera §1. The method also needs known camera intrinsics and dense depth as input §5 Limitations.

Method

The method has three parts: a Gaussian map, a differentiable renderer that produces colour, depth and silhouette images, and a per-frame SLAM loop §3.

Gaussian map. Each Gaussian carries eight values: three for RGB colour, three for centre, one for radius and one for opacity. The paper simplifies the original 3D Gaussian Splatting representation by using view-independent colour and isotropic Gaussians §3 Gaussian Map Representation. The influence of a Gaussian on a 3D point x is given by eq. 1.

f(\mathbf{x}) = o \exp\left(-\frac{\|\mathbf{x} - \boldsymbol{\mu}\|^2}{2r^2}\right) eq. 1

Rendering. Gaussians are sorted front to back and alpha-composited in each pixel. Colour uses eq. 2, where the splatted footprint is obtained by projecting the centre and radius with the intrinsics K, the extrinsics E_t and the focal length f, using eq. 3 with depth d = (E_t μ)_z [§3 Differentiable Rendering]. Depth is composited the same way with each Gaussian's depth d_i in place of its colour eq. 4. The silhouette, which says whether a pixel has any map content, uses weight 1 per Gaussian eq. 5 slam_helpers.py.

C(\mathbf{p}) = \sum_{i=1}^{n} \mathbf{c}_i f_i(\mathbf{p}) \prod_{j=1}^{i-1} (1 - f_j(\mathbf{p})) eq. 2

D(\mathbf{p}) = \sum_{i=1}^{n} d_i f_i(\mathbf{p}) \prod_{j=1}^{i-1} (1 - f_j(\mathbf{p})) eq. 4

S(\mathbf{p}) = \sum_{i=1}^{n} f_i(\mathbf{p}) \prod_{j=1}^{i-1} (1 - f_j(\mathbf{p})) eq. 5

Initialisation. The first frame is not tracked; the pose is the identity. Every pixel becomes a Gaussian with its colour, centred at the unprojected depth, opacity 0.5, and radius r = D_GT / f §3 Initialization eq. 6.

Camera tracking. The pose for a new frame starts from a constant-velocity extrapolation of the two previous poses eq. 7 §3 Camera Tracking. The pose is then optimised by gradient descent while the Gaussians stay fixed. Only pixels whose silhouette is above 0.99 count, so that the loss uses only well-observed map regions §3 Camera Tracking eq. 8.

L_t = \sum_{\mathbf{p}} \Bigl(S(\mathbf{p}) > 0.99\Bigr) \Bigl(L_1\bigl(D(\mathbf{p})\bigr) + 0.5\, L_1\bigl(C(\mathbf{p})\bigr)\Bigr) eq. 8

Gaussian densification. A pixel is densified when the map is sparse (silhouette below 0.5) or when the ground-truth depth lies in front of the rendered depth and the depth error exceeds λ times the median depth error, with λ = 50 §3 Gaussian Densification eq. 9. Gaussians are then added with the same rule as the first frame, using the tracked pose.

M(\mathbf{p}) = \Bigl(S(\mathbf{p}) < 0.5\Bigr) + \Bigl(D_{\textrm{GT}}(\mathbf{p}) < D(\mathbf{p})\Bigr)\Bigl(L_1\bigl(D(\mathbf{p})\bigr) > \lambda\,\textrm{MDE}\Bigr) eq. 9

Map update. Poses are fixed and all Gaussians are optimised. The map is warm-started from its previous state. Every n-th frame becomes a keyframe. From the keyframes, the frames optimised are: the current frame, the most recent keyframe, and k − 2 earlier keyframes with the highest overlap with the current frame §3 Gaussian Map Updating. The loss has no silhouette mask, and the RGB term adds an SSIM term §3 Gaussian Map Updating. The paper also culls Gaussians with near-zero opacity or that are too large §3 Gaussian Map Updating.

# Per-frame SLAM step  [§3 SLAM System]
# Input:  map (Gaussians), keyframes, frame (RGB-D)
# Output: pose of this frame, updated map

def process_frame(map, keyframes, frame):
    # Constant-velocity guess for the pose  [§3 Camera Tracking] [eq. 7]
    pose = extrapolate_pose(previous_poses)
    # Fit the pose to the frame; the map is held fixed  [§3 Camera Tracking]
    pose = track(map, pose, frame)
    # Add Gaussians where the map is sparse or misses geometry  [§3 Densif.]
    map = densify(map, pose, frame)
    # Pick the frames that will be optimised  [§3 Gaussian Map Updating]
    window = select_window(keyframes, pose, frame)
    # Refine all Gaussians with the poses fixed  [§3 Gaussian Map Updating]
    map = map_update(map, window)
    # Every n-th frame is a keyframe; n = 5 in code  [§3 Gaussian Map Updating]
    # [code: scripts/splatam.py#L912-L913]
    if is_keyframe(frame):
        keyframes.append(Keyframe(frame, pose))
    return pose, map

# Tracking: fit the pose to the frame, map fixed  [§3 Camera Tracking]
# Input:  map, initial pose, frame. Output: pose with the lowest loss seen
# Code keeps the lowest-loss pose; not in the paper
# [code: scripts/splatam.py#L682-L711]

def track(map, pose, frame):
    best_pose = pose
    best_loss = infinity
    for iteration in range(tracking_iters):  # 40 on Replica  [§5 Runtime]
        # Loss over valid-depth pixels with silhouette > 0.99  [eq. 8]
        loss = tracking_loss(map, pose, frame)
        # Adam step on the pose only; the Gaussians get no gradient
        pose = adam_step(pose, loss)
        if loss < best_loss:
            best_loss = loss
            best_pose = pose
    return best_pose

# Tracking loss  [§3 Camera Tracking] [eq. 8]
# Code sums masked pixels, not a mean  [code: scripts/splatam.py#L278-L288]

def tracking_loss(colour, depth, silhouette, frame):
    mask = (frame.depth > 0) & (silhouette > 0.99)
    depth_term = sum(abs(frame.depth - depth)[mask])
    colour_term = sum(abs(frame.colour - colour)[mask])
    return 1.0 * depth_term + 0.5 * colour_term

# Map update: all Gaussians, poses fixed  [§3 Gaussian Map Updating]
# Code: loss has no silhouette mask  [code: scripts/splatam.py#L847-L849]

def map_update(map, window):
    for iteration in range(mapping_iters):  # 60 on Replica  [§5 Runtime]
        # Frame drawn at random from the window, current frame included
        # [code: scripts/splatam.py#L831]
        frame = random_choice(window)
        loss = mapping_loss(map, frame)
        map = adam_step(map, loss)
        # Prune at iterations 0 and 20 only
        # [code: utils/slam_external.py#L167-L180]
        if iteration in (0, 20):
            map = prune(map)
    return map

# Mapping loss  [§3 Gaussian Map Updating]
# Paper adds an SSIM term to the RGB loss, no weights  [§3 Gaussian Map Updating]
# Code: 0.8 * L1 + 0.2 * (1 - SSIM)  [code: scripts/splatam.py#L290]

def mapping_loss(map, frame):
    mask = frame.depth > 0
    depth_term = mean(abs(frame.depth - depth)[mask])
    colour_term = (
        0.8 * l1(frame.colour, colour)
        + 0.2 * (1 - ssim(frame.colour, colour))
    )
    return 1.0 * depth_term + 0.5 * colour_term

# Splatting: per-pixel alpha compositing  [§3 Differentiable Rendering] [eq. 2]
# Gaussians are sorted front to back; footprints[i] is f_i(p)

def render(values, footprints):
    transmittance = 1.0
    pixel = 0.0
    for i in range(num_gaussians):
        pixel = pixel + values[i] * footprints[i] * transmittance
        transmittance = transmittance * (1 - footprints[i])
    return pixel

Hyperparameters

Replica values are the ones the paper states, or the ones its code uses where the paper is silent. "not stated" means neither the paper nor its code gives the value in the files checked. The paper evaluates on Replica, TUM-RGBD, ScanNet and ScanNet++; its text gives iteration counts for Replica only.

Per-dataset settings in the code, not stated in the paper:

Results

Author-reported. Setup: four datasets (Replica, TUM-RGBD, ScanNet and ScanNet++), an NVIDIA RTX 3080 Ti for runtime §4 §5. Baseline numbers for Replica, TUM-RGBD and ScanNet are taken from Point-SLAM; the ScanNet++ baselines were run by the authors §4 Baselines.

Camera tracking, ATE RMSE [cm], average over sequences tab. 1

Rendering on training views, Replica, averaged over 8 scenes tab. 2

ScanNet++ (novel and training views, averaged over 2 scenes) tab. 3

Point-SLAM is reported at PSNR 11.91 (novel) and 14.46 (training) tab. 3.

Runtime on Replica R0, RTX 3080 Ti tab. 6

Ablations (Replica R0 unless stated):

• Colour or depth only. Depth-only tracking fails (ATE 86.03 cm). Colour-only tracking gives ATE 1.38 cm, with depth L1 12.58 cm. Full loss: 0.27 cm, 0.49 cm, PSNR 32.81 dB tab. 4.
• Tracking. Without forward velocity propagation the ATE is 2.95 cm. Without the silhouette mask it is 115.80 cm. A silhouette threshold of 0.5 instead of 0.99 gives 1.30 cm. Full: 0.27 cm tab. 5.
• Gaussian shape, ScanNet++ S1. Anisotropic: ATE 0.55 cm, train PSNR 28.11. Isotropic: 0.57 cm, 27.82 dB, at 83.3% of the time and 57.5% of the memory [supp. tab. S.1].
• 3DGS with ground-truth poses on ScanNet++ reaches novel-view PSNR 24.45 dB. SplaTAM reaches 24.41 dB. Re-fitting 3DGS on SplaTAM's poses and map gives 25.14 dB [supp. tab. S.2].

Paper vs code

Official code: spla-tam/SplaTAM at commit da6bbcd24c248dc884ac7f49d62e91b841b26ccc, linked in the supplementary material's overview [supp. Overview]. Checked against the scripts, configs and utilities in the repository. The rasterizer is not in the repository (see Pitfalls).

• Keyframe selection. Paper: the k − 2 earlier keyframes with the highest overlap §3 Gaussian Map Updating. Code sorts by overlap, then shuffles the keyframes that have any overlap and keeps the first k. The choice is random among overlapping keyframes, not the top-k by overlap [code: utils/keyframe_selection.py#L88-L94].
• Overlap test. Paper: points "inside the frustum" of each keyframe [§3 Gaussian Map Updating]. Code projects 1600 random valid-depth pixels and counts those that land inside the image with a 20-pixel margin and positive depth keyframe_selection.py [code: utils/keyframe_selection.py#L78-L84].
• Mapping colour loss. Paper: "we add an SSIM loss" with no weights [§3 Gaussian Map Updating]. Code: 0.8 × L1 + 0.2 × (1 − SSIM) [code: scripts/splatam.py#L290].
• Sum vs mean. Tracking sums the masked L1 terms. Mapping averages the depth term splatam.py [code: scripts/splatam.py#L280] splatam.py. Paper: mapping uses "a similar loss as during tracking" §3 Gaussian Map Updating.
• Best-pose selection. Not in the paper. The code keeps the pose with the lowest loss over the tracking iterations and writes it back at the end [code: scripts/splatam.py#L682-L711] splatam.py.
• Pose parameterisation. Paper: "camera center + quaternion" space [§3 Camera Tracking]. Code: a quaternion and the translation of the world-to-camera transform, relative to the first frame [code: scripts/splatam.py#L266-L274].
• Constant-velocity initialisation. Paper: eq. 7 for each new frame. Code applies it only from the third frame. The second frame copies the first frame's pose splatam.py [code: scripts/splatam.py#L439-L440].
• Pruning. Paper: cull Gaussians with near-zero opacity or that are too large §3 Gaussian Map Updating, with no thresholds. Code: opacity below 0.005, or scale above 0.1 × scene radius, applied at mapping iterations 0 and 20 only slam_external.py [code: configs/replica/splatam.py#L105-L107].
• Scene radius. Not in the paper. Code: max first-frame depth divided by 3 splatam.py splatam.py.
• Replica image size. Paper's runtime text: "full 1200x980 pixel image" §5 Runtime. Code's Replica config: 680 × 1200 [code: configs/replica/splatam.py#L52-L53].
• Gradient-based densification. Present in the code but off in every config splatam.py. The paper describes only the mask-based densification §3 Gaussian Densification.
• Keyframe rule. Code also makes frame 0 and the second-to-last frame keyframes, and requires a finite ground-truth pose [code: scripts/splatam.py#L912-L913]. The paper only says "every n-th frame".

Pitfalls

• Best-pose off-by-one. The loss is computed before the Adam step, but the pose stored with it is taken after the step [code: scripts/splatam.py#L693] splatam.py [code: scripts/splatam.py#L708-L711]. An implementation that copies this will return a pose whose loss does not match. Decide which to keep, and say so.
• Adam state is reset every frame. Both the tracking and the mapping optimisers are rebuilt on each frame splatam.py [code: scripts/splatam.py#L822]. The paper does not say this.
• Adam epsilon. The mapping optimiser uses epsilon 1e-15, the tracking one uses the default splatam.py. Not stated in the paper.
• Median includes zeros. The densification test takes the median over all pixels after multiplying the depth error by the valid-depth mask. Invalid pixels therefore pull the median towards zero, which changes the λ = 50 test when much of the depth is missing splatam.py.
• Sum scale matters less than weights. Adam normalises the overall gradient scale, so the sum-vs-mean difference matters less than the relative weight of the depth and colour terms. In tracking, the two terms are both sums over the same mask. In mapping, the depth term is a mean and the colour term is a mean-based SSIM mix splatam.py. Their relative weights are not directly comparable.
• Random overlap sampling. Keyframe selection draws its 1600 pixels with torch.randint and shuffles keyframes with np.random [code: utils/keyframe_selection.py#L58] keyframe_selection.py. Results depend on the seeds set by the script [code: scripts/splatam.py#L1004].
• Rasterizer not included. The renderer is an external git submodule, diff-gaussian-rasterization-w-depth, that is not in the repository [code: .gitmodules#L1-L2]. Its licence was not checked here. The code cannot run without it.
• W&B on by default. The Replica config sets use_wandb=True with an external entity splatam.py. Turn it off for offline runs.
• Rendering cost. Each iteration renders the full image, about 1.2 million pixels at the paper's resolution, not a sparse sample §5 Runtime. The paper credits the speed of rasterising 3D Gaussians for its runtime §5 Runtime.
• Training-view rendering is not a fair test. The paper says the Replica training-view numbers are an overfitting check, and that Point-SLAM uses ground-truth depth to sample its volume §5 Rendering Quality. Novel-view numbers come only from ScanNet++, aligned with the ground-truth pose of the novel view to the first SLAM frame §5 Rendering Quality.
• Limitations. Sensitive to motion blur, large depth noise and aggressive rotation. Needs known intrinsics and dense depth §5 Limitations.
• Dense methods on TUM and ScanNet. The paper says feature-based ORB-SLAM2 still beats the dense methods on TUM-RGBD, and that no dense volumetric method gets below 10 cm on the original ScanNet §5 Camera Pose Estimation.
• Figure and table numbers. Numbers follow source order in the LaTeX, and the compiled PDF was not checked. Cite by section if a number looks off.

Lineage

• Builds on: 3D Gaussian Splatting §3 Method, cited as 3dgs. The bibliography gives the title only, with no arXiv id, so no link is given here.
• Compared against: Point-SLAM (main dense baseline), NICE-SLAM, Vox-Fusion, ESLAM, ORB-SLAM2, ORB-SLAM3, DROID-SLAM, Kintinuous and ElasticFusion [§4 Baselines].
• Superseded by: not stated.