iSDF: Real-Time Neural Signed Distance Fields for Robot Perception

In plain words

A robot that moves through a room needs to know how far it is from the nearest wall or object, at every point, and it needs that answer while it is still exploring. iSDF builds this map on the fly. It starts with a small neural network that knows nothing about the room. As the camera streams depth images, the network is trained on points sampled along the camera rays, and it learns to say how far each point is from the nearest surface, with a sign for inside or outside.

The tricky part is that a depth camera only tells you where surfaces are, not how far away everything else is. Older methods first build a voxel grid of the surfaces and then spread distances outward through it, which is slow and coarse. iSDF skips that step. For each sampled point it uses the closest measured surface point in the current batch as a stand-in for the true distance. That gives a bound rather than an exact value, and the network learns to respect it. A small extra term keeps the field smooth away from surfaces.

Because the result is a neural network rather than a grid, it can also fill in parts of the room the camera has not seen, as the Results section describes. The paper's main claim is that this is more accurate than voxel-based methods and gives better collision costs and gradients for path planning.

TL;DR

• iSDF fits a randomly initialised 3D-coordinate MLP online to a stream of posed depth images, with no pretraining and no ground-truth SDF §1 §4.
• Supervision uses a free-space loss whose bound is the distance to the nearest surface point of the sampled batch, plus a direct L1 term near surfaces and an Eikonal regulariser §4.3 §4.4.
• Author-reported: lowest SDF error on the six main sequences, below 6 cm at the end of each, and about 33 ms per training iteration on an i7-8700K with an RTX 2080 §6.2 §6.3 §5.2.
• Use it if you need a compact, continuously queryable SDF with planner-friendly gradients and you can supply poses; the released configs and some stated hyperparameters differ from the paper (see Paper vs code).

Problem

Motion planners that optimise trajectories need many cheap collision-cost and gradient queries, so a precomputed signed distance field (SDF) is the common representation §1. Non-truncated SDFs at room scale have usually been assumed given or built offline. The fast real-time methods the paper compares against (Voxblox and a GPU baseline the authors wrote, KinectFusion+) first fuse depth into a voxel surface and then propagate distances through the grid. This limits the voxel size to about 5 cm to 7 cm, and the surface fusion leaves holes in regions that no ray reached §1 §2 §5.2 §6.1.

The paper assumes an external tracking module supplies camera poses; it does not do full SLAM §4. Results are reported on simulated ReplicaCAD and on handheld ScanNet sequences §5.1.

Method

Overview. The SDF is an MLP f(\mathbf{x};\theta) that maps a 3D point to a signed distance §4. Each iteration selects a few posed frames, samples points along rays from pixels of those frames, and takes one optimiser step on a loss built from a bound on each prediction §4 §4.2. The loss uses the sign convention that a point in front of the surface is positive §3.

Definitions. The surface is the zero level set of the field eq. 1. The true signed distance is the signed infimum distance to the surface eq. 2. Here D[u,v] is the measured depth of the ray through pixel (u,v) and d is the depth of the sample on that ray §4.3.

Network §4.1. A multilayer perceptron (MLP) receives an off-axis positional embedding of the coordinates, following NeRF but with directions taken from a twice-tessellated icosahedron, at frequencies 2^0 to 2^L with L=5. The embedding is also concatenated into a later layer. Hidden layers use softplus. The exact layer count is discussed under Paper vs code.

Frame and point sampling §4.2. Each iteration uses five frames: the two most recent, plus three keyframes drawn with probability proportional to their running losses. New frames are sampled for 10 iterations before the system checks for a more recent frame. From each frame, 200 pixels are drawn at random. Along each ray the paper takes N stratified depths in [d_{min}, D+\delta], M depths from a Gaussian around D, and the measured depth D itself. The batch is the union of these points over all selected frames.

Batch bound §4.3. The paper compares three ways to bound the prediction. The ray bound uses the signed depth difference along the ray [eq. 3 with \mathbf{p} = D\mathbf{r}]. The normal-corrected bound adjusts that for the angle between surface normal and ray. The batch bound uses the nearest measured surface point among all rays in the batch eq. 4. The batch bound is used in all experiments. Its approximate gradient eq. 5 comes from the same nearest point. \mathbf{r} and \hat{\mathbf{r}} denote the ray direction; \mathbf{n} the surface normal §4.3.

Loss §4.4. Points in free space (where |D-d| > t) get a one-sided loss that is zero for a prediction between 0 and the bound and grows linearly above it and exponentially below zero eq. 6. Points near the surface get an L1 term to the bound eq. 7 eq. 8. The gradient is supervised with a cosine distance to the approximate gradient, and at measured-depth samples to the surface normal eq. 9. Eikonal regularisation enforces unit gradient norm for points more than a from the measured surface eq. 10. The total loss combines these eq. 11.

Evaluation metrics §5.3. SDF error eq. 12, collision-cost error using the CHOMP cost with \epsilon=2 m eq. 13 eq. 14, and gradient cosine distance eq. 15. Each is evaluated on 200k points sampled from the frames.

# One online training iteration of iSDF  [§4]
# Input:  posed depth frames, network f with weights theta
# Output: updated weights theta

def isdf_iteration(frames, network):
    # Keep the 2 latest frames and draw 3 keyframes in proportion to
    # their running loss  [§4.2]
    selected = latest_two(frames) + sample_keyframes(frames, 3)

    # Draw 200 random pixels per selected frame and back-project them
    # to world-frame rays  [§4.2]
    rays = pixel_rays(selected, pixels_per_frame=200)

    # Per ray: 20 stratified depths in [d_min, D + delta], 8 Gaussian
    # depths around D (sigma 10 cm), and the measured depth D  [§4.2]
    depths = sample_depths(rays)
    points = rays.origin + depths * rays.direction    # (R, S, 3)

    # Predicted signed distances and their spatial gradients  [§4.1]
    predicted = network(points)
    gradients = autograd_gradient(predicted, points)

    # Bound each prediction by the signed distance to the nearest
    # measured surface point in the batch  [§4.3] [eq. 4]
    bounds = signed_batch_distance(points, measured_surface_points)

    # Free-space loss away from the surface, direct L1 near it,
    # with lambda_surf = 5  [§4.4] [eq. 6] [eq. 7] [eq. 8]
    sdf_loss = truncated_sdf_loss(predicted, bounds, t)

    # Cosine distance to the approximate gradient, with lambda_grad
    # = 0.02  [§4.3] [§4.4] [eq. 9]
    grad_loss = cosine_distance(gradients, approximate_gradients)

    # Eikonal term for points more than a = 10 cm from the surface,
    # with lambda_eik = 0.25  [§4.4] [eq. 10]
    eik_loss = eikonal_loss(gradients, measured_depth_gap)

    # Weighted total  [§4.4] [eq. 11]
    loss = sdf_loss + 0.02 * grad_loss + 0.25 * eik_loss

    # One Adam step, lr 0.0013, weight decay 0.012  [§4.4]
    theta = adam_step(theta, loss)
    return theta

Key equations. Sample counts and loss weights are given in the Hyperparameters table below.

\mathcal{S} = \{ \mathbf{x} \in \mathbb{R}^3 \mid f(\mathbf{x}) = 0 \} eq. 1

s(\mathbf{x}) = \mathrm{sgn}(D[u,v]-d)\,\inf_{\mathbf{y}\in\mathcal{S}} |\mathbf{x}-\mathbf{y}| eq. 2

b(\mathbf{x},\mathbf{p}) = \mathrm{sgn}(D[u,v]-d)\,|\mathbf{x}-\mathbf{p}| eq. 3

b(\mathbf{x},\mathcal{P}) = \mathrm{sgn}(D[u,v]-d)\,\min_{\mathbf{p}\in\mathcal{P}}|\mathbf{x}-\mathbf{p}| eq. 4

\mathbf{g}(\mathbf{x},\mathcal{P}) = \mathrm{sgn}(D[u,v]-d)\,\bigl(\mathbf{x} - \arg\min_{\mathbf{p}\in\mathcal{P}}|\mathbf{x}-\mathbf{p}|\bigr) eq. 5

\mathcal{L}_{\text{free}}(f,b) = \max\bigl(0,\; e^{-\beta f}-1,\; f-b\bigr),\quad \beta=5 eq. 6

\mathcal{L}_{\text{near}}(f,b) = |f-b| eq. 7

\mathcal{L}_{\text{sdf}} = \begin{cases} \lambda_{\text{surf}}\,\mathcal{L}_{\text{near}} & \text{if } |D-d| \le t \\ \mathcal{L}_{\text{free}} & \text{otherwise} \end{cases} eq. 8

\mathcal{L}_{\text{grad}} = 1 - \frac{\nabla f \cdot \mathbf{g}}{\|\nabla f\|\,\|\mathbf{g}\|} eq. 9

\mathcal{L}_{\text{eik}} = \begin{cases} \bigl|\,\|\nabla f\|-1\,\bigr| & \text{if } |D-d| \ge a \\ 0 & \text{otherwise} \end{cases} eq. 10

l(\theta) = \mathcal{L}_{\text{sdf}} + \lambda_{\text{grad}}\,\mathcal{L}_{\text{grad}} + \lambda_{\text{eik}}\,\mathcal{L}_{\text{eik}} eq. 11

e_{\text{sdf}}(\mathbf{x}) = |\hat{s}(\mathbf{x})-s(\mathbf{x})| eq. 12

c(s) = \begin{cases} -s + \tfrac{1}{2}\epsilon & \text{if } s<0 \\ \tfrac{1}{2\epsilon}(s-\epsilon)^2 & \text{if } 0<s\le\epsilon \\ 0 & \text{otherwise} \end{cases} eq. 13

e_{\text{collision}}(\mathbf{x}) = |c(\hat{s}(\mathbf{x})) - c(s(\mathbf{x}))| eq. 14

e_{\text{grad}}(\mathbf{x}) = 1 - \frac{\nabla\hat{s}(\mathbf{x})\cdot\nabla s(\mathbf{x})}{\|\nabla\hat{s}(\mathbf{x})\|\,\|\nabla s(\mathbf{x})\|} eq. 15

Notes on the equations. Eq. 3 is the bound used by the ray variant (substitute \mathbf{p} = D\mathbf{r}); the normal-corrected bound is written inline in §4.3. Eq. 8 uses |D-d| as the switch and Eq. 10 the same quantity against a §4.4. The per-sample loss is computed on sampled points and the paper does not state how the terms are averaged over the batch.

Hyperparameters

Results

Author-reported. Setup: ReplicaCAD (6 synthetic sequences) and ScanNet (6 real handheld sequences). The main text reports 6 of the 12 sequences: 3 synthetic and 3 real §5.1 §6. Baselines: Voxblox and KinectFusion+ §5.2. Numbers are read from the text; the figure values are not tabulated.

The paper reports that iSDF fills in unobserved regions and the back of objects qualitatively §6.1 fig. 5 fig. 7. These are visual results without a numerical measure in the text.

Paper vs code

Official code: facebookresearch/iSDF at commit 7c2e1d294599b47f228e1b27abba8e18bee12c42, MIT license LICENSE. The repository is linked from the authors' project page, which is named in the abstract. Configs used: isdf/train/configs/replicaCAD.json and scannet.json, which have identical sample, model and loss blocks.

Discrepancies found:

1. Sample counts. Paper: N=20 stratified plus M=8 Gaussian plus the surface sample §4.2. Code: n_strat_samples = 19 [code: isdf/train/configs/replicaCAD.json#L43]; n_surf_samples = 8, which includes the surface sample and 7 Gaussian offsets [code: isdf/train/configs/replicaCAD.json#L44] sample.py. The Gaussian std of 0.1 m matches \sigma_s sample.py.
2. Bound method. Paper: batch distance in all experiments §6. Both configs set bounds_method to "ray" replicaCAD.json. The batch method is implemented in bounds_pc loss.py.
3. Loss weights. Paper: \lambda_{surf}=5, \lambda_{grad}=0.02, \lambda_{eik}=0.25 §4.4. Configs: trunc_weight 5.38344020 (the \lambda_{surf} role), grad_weight 0.018, eik_weight 0.268 [code: isdf/train/configs/replicaCAD.json#L74] replicaCAD.json replicaCAD.json.
4. Truncation distance. Paper does not state t §4.4. Configs use trunc_distance 0.29365022 m replicaCAD.json.
5. Network depth. Paper: 4 hidden layers §4.1. Code: hidden_layers_block = 2 replicaCAD.json. The model has an input block, two blocks in mid1, a concatenation block, and two in mid2, which is six 256-wide blocks fc_map.py.
6. Activation. Paper: softplus §4.1. Code: Softplus(beta=100) [code: isdf/modules/fc_map.py#L51-L55].
7. Optimiser. Paper: Adam with weight decay 0.012 §4.4. Code: AdamW with the same lr and weight decay trainer.py.
8. Output scaling and noise (not in paper). The network output is multiplied by scale_output 0.14 replicaCAD.json [code: isdf/modules/fc_map.py#L109]. Training adds Gaussian noise to the raw output, std 0.25 for the default, 0.08 for keyframes and 0.04 for other frames [code: isdf/train/configs/replicaCAD.json#L53-L55] fc_map.py.
9. Input scaling (not in paper). Coordinates are multiplied by scale_input 0.05937489 before the embedding projection [code: isdf/train/configs/replicaCAD.json#L64] embedding.py. The paper gives no input scale.
10. Keyframe selection. Paper: iMAP information-gain metric with a frozen network §4.2. Code: renders depth from the frozen network on 400 rays with samples out to 0.8 m behind the surface. A frame is a keyframe when the fraction of pixels with relative depth error below kf_dist_th 0.1 is under kf_pixel_ratio 0.65 trainer.py replicaCAD.json. A frame is also forced to be a keyframe when more than 5 s have passed and the run is not live trainer.py.
11. Keyframe iterations (not in paper). Code uses 60 iterations per keyframe replicaCAD.json. The paper states only the 10 iterations per new frame §4.2 [code: isdf/train/configs/replicaCAD.json#L60].
12. Frame selection (consistent). Two latest frames plus three earlier frames drawn with probability proportional to their average loss [code: isdf/modules/trainer.py#L652-L670]. Window size is 5 [code: isdf/train/configs/replicaCAD.json#L56], matching §4.2.
13. Eikonal region. Paper: applied where |D-d| \ge a §4.4. Code: zeroed where the signed bound is below eik_apply_dist 0.1 [code: isdf/modules/loss.py#L197]. This also zeroes samples behind the surface, which the paper's absolute value would include. Since behind-surface samples lie within 10 cm, the two agree in practice.
14. Free-space switch. Paper: near-surface L1 where |D-d| \le t §4.4. Code: free-space branch where the signed bound exceeds t [code: isdf/modules/loss.py#L132]. Behind-surface samples always take the L1 branch. This has no effect while dist_behind_surf (0.1 m) is below t (0.29 m) replicaCAD.json.
15. Gradient fallback (not in paper). Where the approximate gradient is NaN, the surface normal is used instead trainer.py.
16. Embedding (consistent). Twenty-one icosahedron directions and six frequencies 2^0 to 2^5 (n_embed_funcs = 5) [code: isdf/modules/embedding.py#L40-L62] embedding.py trainer.py. The embedding also appends the raw coordinates embedding.py.
17. Gradient loss (consistent). Cosine distance to the batch-derived gradient, with the surface normal on surface samples [code: isdf/modules/trainer.py#L820-L825] eq. 9.
18. Collision cost (consistent). The CHOMP cost with \epsilon=2 m matches eq. 13 metrics.py.

Pitfalls

• Truncation distance t is missing from the paper. Implementers must pick it. The released configs use about 0.29 m replicaCAD.json. The paper does not report sensitivity to it.
• Batch bound memory. The batch distance builds a tensor of shape (rays, samples, rays, 3) loss.py. Its memory grows with the square of the number of rays in the batch. The paper's 2.3 ms timing [app. "SDF Supervision Bounds ablation"] reflects its hardware and batch size only.
• Weight decay under Adam versus AdamW. The paper says Adam with weight decay §4.4. The code uses AdamW, which decouples decay from the gradient trainer.py. These are different updates.
• Undocumented scaling. The output scale 0.14 and the input scale 0.0594 change the effective frequencies and gradient magnitudes [code: isdf/train/configs/replicaCAD.json#L52] replicaCAD.json. Their effect on the reported results is not stated in the paper.
• Training noise. The code injects noise into the output during training fc_map.py. The paper describes no output noise.
• Batch bound depends on coverage. The paper notes that the batch bound is tight only if the batch has good surface coverage §4.3. Sparse frames give weak bounds.
• Eikonal regularisation over-smooths singularities. The paper states that the regulariser does not handle singular points and smooths around them, and that fine object detail can be lost to the low-frequency embedding §7.
• Poses are assumed. iSDF assumes an external tracker §4. The released config keeps pose refinement off replicaCAD.json.
• ScanNet evaluation is partial. Ground truth exists only in visible regions for ScanNet, because its meshes are not watertight §5.1.
• Real-time claim. The 33 ms figure is per optimisation iteration on one hardware setup §5.2 §6.3. Frame rate and tracking are not evaluated.
• Results are figure-only. The main quantitative results are plotted in Figure 8, and the text gives only bounds such as "below 6 cm" §6.2.
• Stochastic results. iSDF is averaged over 10 network seeds; baselines are deterministic §5.3.
• Paper's simulated versus real data. Experiments use simulated ReplicaCAD and handheld ScanNet data §5.1. The paper describes no robot-platform experiment.

Lineage

• Builds on: iMAP (Sucar et al., ICCV 2021) for frame-level keyframe selection and active sampling, changed here to a depth-based test in the released code §2 §4.2. Not a card in this wiki.
• Builds on: NeRF positional embedding (Mildenhall et al., ECCV 2020) §4.1. Not a card in this wiki.
• Builds on: Eikonal regularisation (Gropp et al., ICML 2020; arXiv:2002.10099), with the distance threshold changed to a=10 cm §4.4.
• Compared against: Voxblox (Oleynikova et al., IROS 2017) §5.2. KinectFusion+, the authors' own GPU baseline built on KinectFusion (Newcombe et al., ISMAR 2011) §5.2 app. "KinectFusion+".
• Superseded by: not known from wiki content.