NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields

In plain words

NeRF-SLAM builds a photo-realistic 3D model of a room while the camera moves through it, using only an ordinary monocular camera and no depth sensor. Two jobs run side by side. One estimates where the camera is and how far away each pixel's surface is. The other fits a neural radiance field, a compact learned model of the scene's colour and density, to the frames.

The main idea is that the tracking side also reports how unsure it is about each depth value. The radiance field is taught from those depth values in proportion to their certainty, so guesses that are likely wrong count for less. Without that weighting, noisy depth pulls the reconstruction off course. The aim is a reconstruction that is both geometrically and visually accurate, and that runs in real time.

TL;DR

• Dense monocular SLAM (DROID-SLAM) supplies camera poses, dense depth maps and their marginal covariances; a hash-grid radiance field (Instant-NGP) is fitted to them in parallel §3.1 §3.3 §3.4.
• The depth term of the mapping loss is weighted by the depth covariance, which the paper argues is what makes noisy monocular depth usable §3.2 §4.4.
• Author-reported on Replica, it has the highest PSNR in every scene. On depth L1 it beats Nice-SLAM with GT depth in only two of eight scenes, and that row is lower on average tab. 1.
• Runs at about 10 frames per second on 640 × 480 images on a single RTX 2080 Ti (11 GB) §4.5 §3.4.
• Relevant if you need a dense, photometrically accurate map from a monocular video and can afford a GPU with about 11 GB of memory §5.

Problem

Monocular reconstruction lacks depth measurements, and NeRFs without depth supervision tend to form floating ghost geometry. Depth supervision helps, but dense monocular depth is noisy and contains outliers, including on textureless regions, so using it unweighted biases the result §1 §3.2 §4.4. Radiance fields normally need known camera poses, and fitting them in real time had been shown only with posed images §1.

Method

The system has two threads that run in parallel §3.3:

1. Tracking. DROID-SLAM computes dense optical flow between frame pairs with a RAFT-like network, with a convolutional GRU that outputs a flow and a per-measurement weight. A dense bundle adjustment then solves for keyframe poses and inverse-depth maps §3.1. Keyframes are added when the motion gate passes (see Hyperparameters), and the tracker sends each keyframe's pose, image, depth and covariances to the mapper §3.3.
2. Covariances. The Hessian of the bundle adjustment is written in block form over poses and depths eq. 1. The marginal covariances of depths and poses follow from its Schur complement, with the pose covariance taken as the inverse of the Cholesky-factored reduced camera matrix eq. 2 §3.1. The paper refers to earlier work for the real-time computation §3.1.
3. Mapping. An Instant-NGP radiance field is trained on the received keyframes. Its loss is the colour loss plus λ_D times a depth loss, where the depth loss is the squared depth residual weighted by the depth's marginal covariance eq. 3 eq. 4 §3.2. Depth is rendered as the expected termination distance along each ray by alpha-compositing densities eq. 5 eq. 6. The colour loss is the standard NeRF photometric loss eq. 7. The mapper optimizes all keyframes it has received and has no sliding window §3.3.

The poses are refined jointly with the radiance field: the loss is minimized with respect to both the poses T and the network parameters Θ eq. 3 §3.2. The printed paper has no pose-prior term. A pose-prior passage in the source is hidden by a macro, so this card leaves it out.

Pseudocode for the covariance step follows the paper's block form eq. 1 eq. 2. The Schur complement is the standard one for the block matrix in eq. 1; it is not written out in the paper.

# Marginal covariances of depths and poses from the dense BA  [§3.1]
# Input:  C, E, P  (blocks of the Hessian of [eq. 1])
# Output: Sigma_d (per-pixel depth covariance), Sigma_T (pose covariance)

def marginal_covariances(C, E, P):
    # Reduced camera matrix: Schur complement of P in H  [§3.1] [eq. 1]
    P_inv = inverse(P)
    H_T = C - E @ P_inv @ transpose(E)
    # Cholesky factor of the reduced camera matrix  [§3.1]
    L = cholesky(H_T)                        # H_T = L L^T
    # Pose marginal covariance  [§3.1] [eq. 2]
    Sigma_T = inverse(L @ transpose(L))
    # Depth marginal covariance  [§3.1] [eq. 2]
    coupling = transpose(P_inv) @ transpose(E) @ Sigma_T @ E @ P_inv
    Sigma_d = P_inv + coupling
    return Sigma_d, Sigma_T

Pseudocode for the mapping loss and the depth rendering follows eq. 3–eq. 7. The norm subscript on the depth term is not defined in the paper; the reading below (division by the variance) is ours.

# Mapping loss for one batch of keyframes  [§3.2] [eq. 3] [eq. 4] [eq. 7]
# Inputs:  I, I_star  (colour, target and rendered)  (B, H, W, 3)
#          D, D_star  (depth, target and rendered)   (B, H, W)
#          Sigma_D    (depth covariance, per pixel)  (B, H, W)
# Output:  scalar loss

def mapping_loss(I, I_star, D, D_star, Sigma_D, lambda_D):
    # Colour term: squared photometric error  [§3.2] [eq. 7]
    loss_rgb = sum((I - I_star) ** 2)
    # Depth term weighted by the depth covariance  [§3.2] [eq. 4]
    # Reading of the subscript norm: divide by Sigma_D (not defined in paper)
    loss_depth = sum((D - D_star) ** 2 / Sigma_D)
    # Total loss; the paper sets lambda_D = 1.0  [§3.2] [eq. 3]
    return loss_rgb + lambda_D * loss_depth

# Expected depth along one ray by volume rendering  [§3.2] [eq. 5] [eq. 6]
# Inputs:  d      (S,)  sample depths, sorted along the ray
#          sigma  (S,)  volume density at each sample
# Output:  scalar expected depth

def render_depth(d, sigma):
    # Spacing between consecutive samples  [§3.2] [eq. 5]
    delta = d[1:] - d[:-1]                   # (S-1,)
    # Optical depth of each segment  [§3.2] [eq. 5]
    tau = sigma[:-1] * delta                 # (S-1,)
    # Transmittance before each sample: sum over earlier samples  [eq. 6]
    trans = exp(-(cumsum(tau) - tau))        # (S-1,)
    # Weight of each sample on the ray  [§3.2] [eq. 5]
    weights = trans * (1 - exp(-tau))        # (S-1,)
    return sum(weights * d[:-1])

Hyperparameters

Values in the paper are listed with their source. Code values are from the official repository at the commit below and are not stated in the paper.

Results

Author-reported on the Replica dataset: 2000 frames rendered from a random trajectory per scene §4.1. Metrics: depth L1 in cm, lower is better, and PSNR in dB, higher is better §4.3. Ours uses the poses and depths from the dense SLAM tracker; the compared methods are described in §4.2. Table values are copied from tab. 1.

Source: tab. 1. In the source, depth L1 is bolded for Nice-SLAM (GT depth) in six scenes and for ours in two (room-1 and room-2). The paper does not say how the average is computed.

Headline claims in the text §4.3:

• "Up to 179% better PSNR in office-1" and "up to 86% better L1 in room-2" are relative to the Nice-SLAM (no depth) row. We checked this from tab. 1: (53.44 − 19.14) / 19.14 ≈ 179% and (19.03 − 2.58) / 19.03 ≈ 86%. The text does not name the variant. Against Nice-SLAM with GT depth the same figures are about 112% (PSNR, office-1) and about 12% (L1, room-2), by our calculation.
• Against Nice-SLAM with GT depth, ours has a lower depth L1 only in room-1 and room-2. Its average depth L1 (4.49 cm) is higher than that row's (4.08 cm). Its PSNR is higher in every scene against that row.

Ablation on the Cube-Diorama dataset (synthetic, with ground truth) §4.4:

• With GT poses and GT depth, the field converges fastest fig. 4. With GT poses and no depth it still converges, more slowly. With noisy poses and no depth it does not converge within 60 s fig. 4.
• Raw (unweighted) dense depth lowers the depth error but makes PSNR about 3 dB worse than using no depth. The paper attributes this to biased geometry §4.4 fig. 6.
• Weighting the depth by its marginal covariance keeps PSNR close to the no-depth case and reaches the depth error of the raw-depth case §4.4 fig. 6.
• Reported numbers: no depth reaches 7.8 cm depth L1 at 500 s; raw depth reaches 4.1 cm §4.4. For unweighted depth, the text says PSNR and L1 are worse by 4 dB and 7 cm after 120 s, without naming the baseline §4.4 fig. 5.
• The printed paper has no pose-prior experiment. The source holds a placeholder for one, hidden by a macro, so it is not reported here §4.4.

Qualitative results: fig. 3 shows the raw tracker point cloud, a σ-Fusion mesh, Nice-SLAM and ours on Replica office-0. fig. 1 shows the input image, the depth uncertainty, a thresholded point cloud and the rendered field.

Real-time performance §4.5: the paper reports about 12 fps in one sentence and 10 fps overall, with tracking at about 15 fps, about 10 keyframes per second (depending on motion) and mapping at about 10 fps. Parallelism and custom CUDA kernels are credited.

Paper vs code

Official code: ToniRV/NeRF-SLAM at commit 4e407e8cb1378c6ece18e621b19ccd5be982b7dd, BSD-2-Clause. The repository owner is the first author (display name Toni Rosinol, personal MIT page), and the README names all three authors and the paper. The README's project-page link is commented out. This is the only official code we found; see "Code and licence" below.

• Depth loss weight. λ_D = 1.0 in both §3.2 and the code nerf_fusion.py. The loss type is set to L2 nerf_fusion.py.
• Depth weighting. The code passes per-pixel depth covariances into the Instant-NGP training data nerf_fusion.py. The weighting itself lives in the instant-ngp submodule (thirdparty/instant-ngp, from ToniRV/instant-ngp, branch feature/nerf_slam) .gitmodules, which is not in our clone. We could not check the exact form of the weighted depth loss, so the pseudocode above is the paper's reading.
• Keyframe rule. The paper says a keyframe is added when the mean optical flow exceeds 2.5 px §3.3. The code first admits a frame when the mean flow from one GRU update exceeds 2.4 px visual_frontend.py visual_frontend.py, and then rejects the keyframe if the DROID frame distance to the previous keyframe is below 4.0 visual_frontend.py visual_frontend.py. The distance metric is not the mean optical flow.
• Motion filter module. networks/motion_filter.py defines a 2.5 px motion filter, but nothing in the repository imports it motion_filter.py. The active keyframe rule is the one in the visual frontend.
• Window size. The paper says at most 8 keyframes in the active tracking window §3.3. The code's frontend window is 25 keyframes visual_frontend.py. The code also ends the run when the buffer is full visual_frontend.py.
• Depth covariance. The code computes the inverse-depth covariance from the reduced camera matrix and then converts it to depth with the factor 1/d⁴ visual_frontend.py visual_frontend.py. The paper gives eq. 2 without this conversion.
• Pose covariance. The code takes the Cholesky factor of the assembled Hessian, inverts it, and stores the diagonal blocks as the body covariances visual_frontend.py visual_frontend.py. This matches the form of eq. 2.
• Covariance upsampling. The code upsamples the depth covariance to full resolution with the learned convex upsampling weights visual_frontend.py. The paper does not state this.
• Mask variants. The default is ours, which uses the covariance without thresholding slam_demo.py. The ours_w_thresh option drops depths above the median covariance square root nerf_fusion.py. The no_depth and raw options are for ablations nerf_fusion.py.
• Mapping schedule. The paper says the mapper continuously minimizes the loss §3.2. The code calls one training frame per received packet by default nerf_fusion.py nerf_fusion.py. Depth annealing is off by default nerf_fusion.py.
• Depth metric in evaluation. The L1 metric in the code scales the estimated depth by the ratio of the mean depths before comparing, and clips each per-pixel error at 2.0 m nerf_fusion.py. The comment next to the clip says 1 m. The paper does not describe a scale alignment or a clip.
• Evaluation frames. The evaluation loop renders every second training frame (stride 2) nerf_fusion.py. The paper describes 2000 frames per scene. The shipped scripts/replica_results.py sets final_k = 2000 replica_results.py and buffer = 100 replica_results.py. Its args_to_test dictionary lists only the raw mask variant replica_results.py.
• Ground-truth fitting path. In the ground-truth path, gt_depths is set from the input depths rather than from ground truth nerf_fusion.py. Use it with care if you build a ground-truth baseline from this code.
• Initial pose prior. When the first frame's pose is known, the frontend adds a prior with standard deviation 1e-4 on the first keyframe pose, taken from the initial pose it is given. The function is named for ground truth visual_frontend.py visual_frontend.py. The paper does not mention a prior on the first pose.
• Damping. The frontend adds 0.2 * damping + EP to the damping term, with a code comment asking what the damping is visual_frontend.py. The paper does not describe a damping term.

Pitfalls

• Weighted loss form. The paper writes the depth term as a squared norm with a covariance subscript, but does not define it. Dividing by the variance is the natural reading eq. 4. Check it against the instant-ngp submodule before you rely on it; that code is not in our clone.
• Two frame rates. The paper gives both 12 fps and 10 fps for the overall pipeline §4.5. Plan for the lower figure unless you measure your own setup.
• Depth covariance upsampling. The covariance is estimated at 1/8 resolution and upsampled with learned convex weights visual_frontend.py visual_frontend.py. Full-resolution covariances are not computed directly.
• Memory. The paper reports about 11 GB, and the limitations section names dense correlation volumes and hierarchical grids as the causes §5. Each keyframe buffer holds full-resolution images and depth; the run stops when the buffer is full visual_frontend.py. Set the buffer for your run length.
• Averages. The table's average column is given without a formula tab. 1. Compute your own average over the same scenes if you compare.
• Baseline fairness. The paper notes that iMAP and Nice-SLAM use ground-truth depth, which is better than a casually moving RGB-D camera would give §4.2. The no-depth Nice-SLAM row is the closest comparison for a monocular system.
• Keyframe gate. Two different thresholds gate keyframes in the code (2.4 px flow and a distance of 4.0). The paper gives only one (2.5 px). Tune both if you change the frame rate or resolution visual_frontend.py visual_frontend.py.
• Unstated optimization details. The paper does not give the learning rate, optimizer, batch size, number of mapping iterations, hash-grid size or MLP sizes §3.2 §3.4. The network configuration lives in the instant-ngp submodule, which is not in our clone.
• Global scale. The code's depth metric aligns scale per evaluated frame by a mean ratio nerf_fusion.py. The paper does not say how depth scale is resolved for its reported numbers.
• Bias from unweighted depth. Ablation numbers show that raw depth can make PSNR worse than no depth at all §4.4. Do not add a depth term without weights.

Lineage

• Builds on: DROID-SLAM for dense monocular tracking and its covariance computation, and the probabilistic volumetric fusion of Rosinol et al. (σ-Fusion) §2.1 §3.1. Instant-NGP for the hash-based radiance field §3.2 §3.4. The depth-supervision idea follows deng2022depth §1.
• Compared against: TSDF-Fusion, σ-Fusion, iMAP and Nice-SLAM §4.2 tab. 1.
• Future direction named by the paper: metric-semantic SLAM and 3D dynamic scene graphs §6.

Code and licence

• Repository: ToniRV/NeRF-SLAM, BSD-2-Clause (LICENSE.BSD, copyright 2022 Massachusetts Institute of Technology). The README says that changes to instant-NGP fall under an NVIDIA licence and that the tracker reuses parts of DROID-SLAM, which is also BSD-licensed. We cited only this repository. The submodules are not in our clone, and we did not quote code.
• Commit read: 4e407e8cb1378c6ece18e621b19ccd5be982b7dd (shallow clone).
• Files read: examples/slam_demo.py, fusion/nerf_fusion.py, slam/visual_frontends/visual_frontend.py, networks/motion_filter.py, scripts/replica_results.py, README.md, .gitmodules.