ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields

In plain words

ESLAM is a system that builds a 3D model of a room while it works out where the camera is. The input is a stream of colour images with a depth reading for each pixel, taken by a moving camera. Nothing has to be trained in advance: the system starts from empty and learns the scene as the frames arrive.

The scene is kept in a compact form. Instead of a dense 3D grid of values, ESLAM stores a few flat grids, one per pair of axes (like three perpendicular sheets of graph paper), at a coarse and a fine resolution. A small neural network reads the values at any point and says how far that point is from the nearest surface and what colour it has. Rendering those predictions along camera rays gives a picture and a depth that can be compared with the real frame, and the difference is used to correct both the scene and the camera pose.

The authors report that this is more accurate and much faster than two earlier neural SLAM methods on three standard test sets, and that it stays affordable in memory as the room gets larger. All numbers come from the authors' own runs.

TL;DR

• ESLAM is an RGB-D dense SLAM method. It jointly optimises the camera pose and the scene, and it needs no pretraining §1 §3.4.
• Geometry is a truncated signed distance field (TSDF), read from axis-aligned feature planes at two scales, with separate planes for geometry and colour §3.1.
• Author-reported on Replica: ATE RMSE 0.63 cm against 2.05 cm for NICE-SLAM tab. 1, and frame processing time 0.18 s against 2.10 s tab. 4.
• Use it as a reference for a feature-plane SLAM back end. Check the Paper vs code and Pitfalls sections before reimplementing: the code differs from the paper in several places.

Problem

Earlier NeRF-style dense SLAM methods are slow, and their detail is limited. iMAP fits one large MLP to the whole scene and is slow §1. NICE-SLAM stores features in voxel grids, whose memory grows with the cube of the scene side length, so resolution has to be low §1 §2. NICE-SLAM also relies on frozen pretrained MLPs, which limits how well it generalises to new scenes §1 §2. The authors add that these methods are too costly for real-time use §1.

Method

The scene is a pair of feature-plane sets with three planes each: one set for geometry (F_g) and one for appearance (F_a), each at a coarse and a fine scale §3.1. For a 3D point, each plane is read by bilinear interpolation from its four nearest cells. At each scale the three plane reads are summed, and the two scales are concatenated eq. 1 eq. 2. Two shallow two-layer MLPs decode these features: one gives a normalised TSDF, the other a raw colour eq. 3 §3.1.

Rendering follows the SDF-based density of StyleSDF §3.2. The TSDF is turned into a density with a learnable sharpness beta eq. 4. Standard volume-rendering weights then give the ray colour and the ray depth eq. 5. Samples are drawn stratified along each ray, plus extra samples near the surface chosen from the depth measurement or from the previous weights §3.2.

The loss has five terms eq. 6 to eq. 11 §3.3:

1. A free-space term pushes the TSDF to 1 between the camera and the truncation band eq. 6.
2. A signed-distance term in the truncation band fits the depth sensor. The band is split into a middle part (within 0.4 T of the measured depth) and a tail part, and the two get separate weights eq. 7 eq. 8.
3. A depth term matches rendered and measured depth on rays with valid depth eq. 9.
4. A colour term matches rendered and observed colour on all rays eq. 10.
5. The total is a weighted sum; mapping and tracking use different weights eq. 11 §4.

Mapping runs every 4 input frames §3.4 [supp. §1]. It updates a window of W = 20 keyframes: the current frame, the two previous keyframes and W − 3 other keyframes §3.4 [supp. §1]. It jointly optimises planes, decoders and the window's camera poses, with no staged schedule §3.4. Tracking optimises only the new frame's camera pose, using the same loss with the tracking weights, with Adam §3.4 §4. Rays whose depth error is more than 10 times the batch median are treated as outliers and dropped §3.4.

# Feature query at one 3D point  [§3.1] [eq. 1] [eq. 2]
def query_features(points, geo_planes, app_planes):
    # Read and sum the three planes per scale, then concatenate the scales
    geo_coarse = sum_plane_reads(geo_planes.coarse, points)
    geo_fine = sum_plane_reads(geo_planes.fine, points)
    feat_geo = concat(geo_coarse, geo_fine)     # (N, 64)
    app_coarse = sum_plane_reads(app_planes.coarse, points)
    app_fine = sum_plane_reads(app_planes.fine, points)
    feat_app = concat(app_coarse, app_fine)     # (N, 64)
    return feat_geo, feat_app

# Decode TSDF and colour  [§3.1] [supp. §1]
def decode(feat_geo, feat_app, h_geo, h_app):
    # Two-layer MLP with ReLU hidden layer and Tanh output
    tsdf = tanh(h_geo(feat_geo))                # (N,)
    # Two-layer MLP with ReLU hidden layer and Sigmoid output
    colour = sigmoid(h_app(feat_app))           # (N, 3)
    return tsdf, colour

# Density from TSDF with learnable sharpness beta  [§3.2] [eq. 4]
def density(tsdf, beta):
    return beta * sigmoid(-beta * tsdf)

# Rendered colour and depth along one ray  [§3.2] [eq. 5]
def render_ray(sigma, colour, z_vals):
    # Transmittance: product of (1 - alpha) over earlier samples
    transmit = exp(-cumulative_sum_before(sigma))
    weights = transmit * (1 - exp(-sigma))
    rendered_colour = sum(weights * colour)
    rendered_depth = sum(weights * z_vals)
    return rendered_colour, rendered_depth

# Free-space term: TSDF should be 1 before the truncation band  [§3.3] [eq. 6]
def free_space_loss(tsdf, free_mask):
    return mean((tsdf[free_mask] - 1) ** 2)

# Truncation-band terms: fit the sensor depth to the implied surface  [§3.3]
# Middle band (|z - D| < 0.4 T) and tail band use this residual  [eq. 7] [eq. 8]
def band_loss(z_vals, tsdf, depth, band_mask, T):
    surface = z_vals + tsdf * T
    return mean((surface[band_mask] - depth) ** 2)

# Total loss: weighted sum of the five terms  [§3.3] [eq. 11] [§4]
def total_loss(terms, weights):
    # terms: fs, middle, tail, depth, colour; weights differ per stage  [§4]
    return sum(w * t for w, t in zip(weights, terms))

# Tracking: optimise only the new frame's camera pose  [§3.4]
def track_frame(frame, planes, decoders, pose):
    for _ in range(tracking_iters):     # 8 Replica, 30 ScanNet, 200 TUM [supp. §1]
        rays = sample_rays(frame, tracking_pixels)   # 2000 rays [supp. §1]
        # Drop rays whose depth error exceeds 10 times the batch median  [§3.4]
        rays = drop_outliers(rays)
        loss = total_loss(terms_for(rays), tracking_weights)  # [§4]
        pose = adam_step(pose, loss)     # rotation as quaternion  [supp. §1]
    return pose

# Mapping: jointly refine planes, decoders and the window's poses  [§3.4]
def map_window(keyframes, planes, decoders, poses):
    # Current frame, two latest keyframes and W - 3 others  [§3.4]
    window = select_window(keyframes, window_size=20)   # W = 20 [supp. §1]
    for _ in range(mapping_iters):      # 15 Replica, 30 ScanNet, 60 TUM
        rays = sample_rays(window, mapping_pixels)   # 4000 rays [supp. §1]
        loss = total_loss(terms_for(rays), mapping_weights)  # [§4]
        update(planes, decoders, poses, loss)

Hyperparameters

Values from the paper are given with their citation. The code value is shown where the official code at code_commit sets one, so you can see any difference. Where the paper gives no value, the table says so.

Results

Author-reported. Setup: Replica (8 scenes, 5 runs each), ScanNet (6 scenes, 5 runs each, localization only because ground-truth meshes are incomplete), and TUM RGB-D (3 sequences) §4 [supp. §1]. Baselines: iMAP* (a reimplementation from NICE-SLAM, since the original iMAP is not open source) and NICE-SLAM §4. Runtime on one NVIDIA RTX 3090, on Replica room0 and ScanNet scene0000 tab. 4.

Replica, mean over eight scenes tab. 1:

ScanNet, localization, average over six scenes tab. 2:

TUM RGB-D, ATE RMSE (cm) tab. 3:

Runtime and size tab. 4:

Iteration-count study on Replica, supplementary [supp. §Effect of Frame Processing Time]: ESLAM with 2× iterations has ATE RMSE 0.42 cm and frame time 0.35 s. With 10× iterations it has 0.31 cm and 1.72 s. The default is 0.63 cm and 0.18 s.

Ablations, supplementary [supp. §2]: on ScanNet the full model gives ATE RMSE 7.38 cm; removing mapping pose optimisation raises it to 11.27 cm, and using shared planes for geometry and appearance gives 7.49 cm. On Replica the full model gives 0.63 cm ATE; the removal of mapping pose optimisation gives 4.85 cm. Depth-resolution robustness on room0, with depth downsampled to 1/8 of its resolution, gives ATE 0.72 cm for ESLAM against 2.01 cm for NICE-SLAM [supp. §2].

Qualitative results (meshes and trajectories) appear in fig. 1 (teaser), fig. 3 (Replica geometry), fig. 4 (ScanNet trajectories), fig. 5 (ScanNet geometry), fig. 6 (TUM) and [supp. §1] (textured and untextured Replica meshes). Figures are not reproduced on this card.

Paper vs code

Searched: the paper's abstract, body and bibliography have no code link. The project page idiap.ch/paper/eslam links no repository. A GitHub repository search for "eslam" returned idiap/ESLAM, owned by the Idiap organisation. The paper's author block lists the first author at Idiap Research Institute. The repository is Apache-2.0 and is the official code. Commit 654d5044f292914896e15e4760a46bf6b8f87b2e was cloned with --depth 1. The code differs from the paper in these places:

• Decoder width. The paper uses 32 hidden channels §4. The decoder class defaults to hidden_size=16 and get_model does not override it decoders.py config.py.
• Keyframe selection. The paper samples W − 3 keyframes at random §3.4. The default config uses overlap selection ESLAM.yaml and picks 19 keyframes by overlap, then adds the last two keyframes and the current frame, giving 22 frames in total, not W = 20 Mapper.py.
• Joint pose optimisation. The paper says the window's poses are optimised jointly, with no gating condition stated §3.4. The code enables it only when more than four keyframes exist Mapper.py, controlled by ESLAM.yaml.
• Replica tracking translation. The paper gives 0.001 for translation and rotation on Replica [supp. §1]. replica.yaml sets translation to 0.002 replica.yaml.
• Free-space and band terms. The paper averages each term per ray, then over rays eq. 6 eq. 7 eq. 8. The code takes one flat mean over all selected points Mapper.py. Rays with more samples in a band get more weight in the code.
• Colour loss in tracking. The paper defines the colour loss on all rays §3.3. In tracking the code applies the outlier mask to the colour term too Tracker.py.
• TUM mapping schedule. The general mapping rule is every 4 frames [supp. §1]. The TUM config maps every frame and makes each frame a keyframe tum.yaml.
• Extra settings. Feature planes are initialised with std 0.01 ESLAM.py. The first frame uses 1000 iterations and a learning-rate factor of 5 ESLAM.yaml ESLAM.yaml. Tracking uses Adam with betas (0.5, 0.999) Tracker.py. The initial beta is 10 decoders.py. None of these values is in the paper.
• Learnable beta on TUM. The paper says beta is learnable §3.2. The TUM config fixes it tum.yaml.

Pitfalls

• Decoder width. Using the paper's 32-wide decoders will not match the code's reported numbers. Pick one, and say which in your implementation.
• Keyframe window. The code's window is about 22 frames, not 20 as the paper says. The selection is driven by overlap, not random.
• Loss normalisation. Flat versus per-ray averaging changes how strongly rays with more band samples weigh. Pick the paper's per-ray form if you want the paper's objective.
• Gated joint optimisation. In the code, window poses are not jointly refined until more than four keyframes exist Mapper.py. The paper does not state this condition.
• Hidden parameters. Feature-plane initialisation, the first-frame iteration count, the Adam betas and beta's initial value are not in the paper. Any reimplementation will have to choose them.
• Hyperparameters found by grid search. The loss weights and learning rates come from grid search §4 [supp. §1]. The search space is not reported.
• Truncation is scale-sensitive. T = 6 cm is set against a 1 cm mesh resolution for evaluation §4. The band loss scales with T, and the decoder's TSDF output is a Tanh, so it lies in (-1, 1) [supp. §1] decoders.py. Change T together with the plane resolutions.
• Memory grows quadratically, not linearly. Plane storage grows as L² tab. 4. ESLAM can still be large: the ScanNet model has 17.63 M parameters tab. 4, and the code keeps keyframes on the GPU by default ESLAM.yaml.
• Forgetting. The conclusion says that planes shared across the scene can overwrite earlier geometry. The authors counter this by replaying older keyframes §5 Conclusion. The cost of the replay is not reported.
• Baseline is a reimplementation. iMAP* is the NICE-SLAM authors' reimplementation, not the original iMAP, which is not open source [§4 Baselines].
• Timing is on one GPU, one scene. Frame times come from an RTX 3090 on Replica room0 and ScanNet scene0000 only tab. 4. The "up to 10x" figure in the abstract is a best case; on ScanNet the caption gives 6x.
• ScanNet evaluation is localization only. The ground-truth meshes are incomplete, so no reconstruction metrics are reported there §4 [supp. §2].
• Mesh culling. Reconstruction metrics use culling of faces outside all camera frusta or occluded in all frames §4 Metrics [supp. §1]. Results without culling are not comparable.

Lineage

• Builds on: NICE-SLAM, the main baseline, which uses voxel features and frozen pretrained decoders §1 §2. ESLAM replaces the voxels with feature planes and does not use pretrained networks §1 §3.1.
• Builds on: iMAP, whose mapping and keyframe practice ESLAM follows §3.4.
• Builds on: the tri-plane design of Chan et al. (2022), cited as chan2022efficient §3.1; StyleSDF's SDF-based rendering §3.2; and the per-point TSDF losses of Azinovic et al. §3.3.
• Compared against: iMAP* and NICE-SLAM tab. 1 tab. 2 tab. 3 tab. 4.
• Concurrent work named but not compared numerically: iDF-SLAM, Orbeez-SLAM, MeSLAM, MonoNeuralFusion and NeRF-SLAM §1.
• Superseded by: not known from this wiki's content.