Paper · 2022-09-27 · cs.RO, cs.CV
Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping
Monocular visual SLAM without pre-training: ORB-feature visual odometry sets camera poses, and an instant-ngp NeRF trains online on the keyframes for a dense map. Author-reported: TUM ATE 1.9/0.3/1.0 cm (ORB-SLAM2 1.6/0.3/0.9), ScanNet average 8.66 cm vs NICE-SLAM 9.63, and 360–800x faster than NICE-SLAM.
Neural radiance field (NeRF) and variantsSimultaneous localization and mapping (SLAM)Bundle adjustmentEmpty space skippingvisual-slamneural-radiance-fieldmonocular-slamorb-featuresreal-time
arXivPDFCodeMarkdown11 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A robot that moves through a room needs to know where its camera is, and most classic systems also keep a sparse set of landmark points. That is enough to navigate, but it says little about what the room looks like. This paper's system uses one ordinary camera and combines two parts. A classic feature-based tracker follows the camera through the video. A neural scene model, which learns colour and density at every point in space, is trained while the camera moves.
Earlier neural approaches to this problem need depth sensors, or a model pre-trained on similar scenes, or both. This system does neither. The tracker gives reasonable camera positions from the first frames, so the scene model can learn from them directly, one new room at a time.
The scene model is kept fast by storing its work in a grid that skips empty space. The output is a dense picture of the scene that can be rendered from new viewpoints or used by other software, such as a robot planning a route. The authors report real-time speed on one desktop GPU and tracking accuracy close to the classic tracker it builds on.
TL;DR
Orbeez-SLAM runs an ORB-SLAM2-style visual odometry front end on a single RGB camera and trains an instant-ngp NeRF online on the keyframes it selects. It needs no depth input and no pre-training, and it produces a dense rendered map in real time. Results are author-reported on TUM RGB-D, ScanNet and Replica. Tracking is close to ORB-SLAM2 on TUM, and the paper reports a small average advantage over NICE-SLAM on ScanNet. The code is GPL-3.0, so only paraphrase it.
Problem
The goal is a SLAM that copes with unfamiliar rooms with no pre-training step and still produces dense maps in real time §I. Classic SLAM tracks well but gives sparse maps §I §II.B. Tandem builds dense maps with TSDF fusion but needs a pre-trained depth module §I §II.B. iMAP and NICE-SLAM use NeRF as the map, but they need RGB-D input because they optimise poses without visual odometry, and NeRF training is slow §I §II.B. The authors' aim is a monocular system that is pre-training-free and real-time, with a dense map §I.
Method
The system has a tracking process and a mapping process that run concurrently. Tracking extracts features and estimates the camera pose by visual odometry. Mapping triangulates map points, runs bundle adjustment on keyframes and map points, and feeds them to the NeRF §IV.A fig. 3. A frame becomes a keyframe only when enough frames have passed or tracking is weak, and it has fewer matches than the reference keyframe. If mapping is busy, the frame is dropped and bundle adjustment is interrupted §IV.A fig. 3 Tracking.cu.
- Visual odometry (tracking). Estimate the pose by minimising the reprojection error of map points §IV.B eq. 6 eq. 7. The authors state that this is the only pose objective in the final system §IV.B §V.C.
- Keyframe selection. Insert a keyframe when tracking is weak or enough frames have passed, and mapping accepts it §IV.A fig. 3 (exact rules are in the hyperparameter table).
- Local mapping. Triangulate new map points, then run bundle adjustment on the reprojection error of keyframe poses and observed points §IV.A §IV.B eq. 8. The keyframes and points then go to the NeRF §IV.A.
- NeRF training. Render rays through the current keyframe poses and train the network with the photometric loss on pixel colour §IV.B eq. 10. The rays are sampled near the surface using the skip-voxel density grid §III.B fig. 2 §IV.B.
- Dense points. Count how often each density-grid voxel is hit by rays, then keep voxels with high counts as dense points §IV.C fig. 4.
The NeRF is a radiance field with volume rendering §III.A eq. 1 eq. 3. The colour of a ray is the sum over samples of transmittance times opacity times colour eq. 3. The sample weight is eq. 5. The paper does not describe the network architecture beyond saying it is built on instant-ngp §I §V.A.
# Tracking, per frame [§IV.A] [fig. 3]
# Input: frame, previous pose; output: pose and maybe a new keyframe
def track_frame(frame, previous_pose):
# Estimate the pose by reprojection error [§IV.B] [eq. 7]
pose = minimise_reprojection(frame, previous_pose)
# Insert a keyframe only if mapping can accept it [§IV.A]
if needs_keyframe(frame, pose) and mapping_is_idle():
return make_keyframe(frame, pose)
return None# Mapping, per new keyframe [§IV.A] [§IV.B]
# Input: keyframe, map; output: updated map and NeRF weights
def map_keyframe(keyframe, map_points, nerf):
# Triangulate new points from matched features [§IV.A]
map_points = triangulate_new_points(keyframe, map_points)
# Bundle adjustment on reprojection error [§IV.B] [eq. 8]
keyframes, map_points = bundle_adjust(keyframe, map_points)
# Train the NeRF weights on photometric error, poses fixed [§IV.B] [eq. 10]
train_nerf(nerf, keyframes, sample_rays(keyframes))
return map_points# Ray-casting triangulation [§IV.C] [fig. 4]
# Input: rays, voxel counts, weight threshold; output: updated counts
def count_ray_hits(rays, voxel_counts, weight_threshold):
for ray in rays:
# A sample whose weight exceeds the threshold marks its voxel [fig. 4]
for sample in ray.samples:
if sample.weight > weight_threshold:
# Threshold value not stated [§IV.C]
voxel_counts[sample.voxel] += 1
return voxel_counts# Dense points from voxel counts [§IV.C]
# Input: voxel counts, min_count = 64 in the paper; output: point list
def dense_points(voxel_counts, min_count):
# The authors chose 64 from experience [§IV.C]; comparison not stated
return [voxel_centre(v) for v in voxel_counts
if voxel_counts[v] >= min_count]Equations as published. Eqs. (1) to (5) describe NeRF rendering §III.A:
C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,dt eq. 1
T(t) = \exp\!\Big(-\int_{t_n}^{t} \sigma(\mathbf{r}(s))\,ds\Big) eq. 2
\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i \big(1-\exp(-\sigma_i(t_{i+1}-t_i))\big)\,\mathbf{c}_i eq. 3
T_i = \exp\!\Big(-\sum_{j=1}^{i-1} \sigma_j (t_{j+1}-t_j)\Big) eq. 4
w_i = T_i \big(1-\exp(-\sigma_i(t_{i+1}-t_i))\big) eq. 5
Eqs. (6) to (10) are the SLAM and training objectives §IV.B §IV.C:
L_{rpj} = \sum_{ij} \big\lVert u_{ij} - \pi(\mathscr{C}_j, P_i) \big\rVert_2 eq. 6
\min_{\{[R|t]_j\}} L_{rpj} eq. 7
\min_{\{[R|t]_j\},\{P_i\}} L_{rpj} eq. 8
r(d) = -R^{T}t + d\,R^{T}K^{-1}[u,v,1]^{T}, \quad d \in \mathbb{R} eq. 9
L_{pht} = \sum_{ij} \big\lVert C(\mathbf{r}_{ij}) - \hat{C}(\mathbf{r}_{ij}) \big\rVert_2 eq. 10
Symbols in eq. 6: u_{ij} is the observed pixel of point i in camera j, \pi projects point P_i with intrinsics K_j and pose [R|t]_j §IV.B. Equation 9 is the ray through a pixel of a keyframe §IV.B.
Figure descriptions, no images in this card: fig. 1 shows the process over time, with a NeRF result after 27 seconds of training and the update after novel views are seen fig. 1. fig. 2 shows the skip-voxel strategy, where unoccupied voxels are skipped and voxels that intersect the surface are sampled §III.B. fig. 3 shows the two-process pipeline and the keyframe conditions §IV.A. fig. 4 shows the ray-casting triangulation counter §IV.C. fig. 5 compares rendered RGB and depth with NICE-SLAM §V.D. fig. 6 shows NeRF output over time on TUM, Replica and ScanNet, and the failure on the large ScanNet scene §V.D.
Hyperparameters
Values not in the paper are marked "not stated". Values from code are cited to
the repository at code_commit. The camera and feature values come from the
example config files, not from the paper.
| Name | Value | Where | Notes |
|---|---|---|---|
| Hardware | Intel i7-9700 CPU, NVIDIA RTX 3090 GPU | §V.A (Implementation Details) | Desktop PC |
| Base SLAM | ORB-SLAM2 code, instant-ngp code | §V.A (Implementation Details) | Loop closing inherited; keyframe culling off §V.A |
| Pose solver | Levenberg-Marquardt, via g2o | §IV.B Optimizer.cu | Paper follows ORB-SLAM2 §IV.B |
| Keyframe interval, max | mMaxFrames = camera fps (30 in shipped configs) | Tracking.cu freiburg1_desk.yaml | Paper: not stated |
| Keyframe interval, min | mMinFrames = 0 | Tracking.cu | Paper: not stated |
| Monocular reference ratio | 0.9 | Tracking.cu | Paper: not stated |
| Min inliers for keyframe | more than 15 | Tracking.cu | Paper: not stated |
| Keyframes before local BA | more than 2 | NerfMapping.cu | Paper: not stated |
| Huber delta, 2D | sqrt(5.99) | Optimizer.cu Optimizer.cu | Paper: not stated |
| Outlier chi-square, mono | 5.991 | Optimizer.cu | Paper: not stated |
| ORB features, TUM | 1000 | freiburg1_desk.yaml | Paper: not stated |
| ORB features, Replica | 2000 | office0.yaml | Paper: not stated |
| NeRF aabb scale, TUM | 4 | freiburg1_desk.yaml | Paper: not stated |
| NeRF scale, TUM | 0.33 | freiburg1_desk.yaml | Paper: not stated |
| NeRF offset, TUM | [0.7, 0.5, 0.1] | freiburg1_desk.yaml | Paper: not stated |
| NeRF aabb scale, Replica | 2 | office0.yaml | Paper: not stated |
| NeRF scale, Replica | 0.15 | office0.yaml | Paper: not stated |
| NeRF offset, Replica | [0.5, 0.5, 0.4] | office0.yaml | Paper: not stated |
| NeRF network config | instant-ngp base.json | freiburg1_desk.yaml | File is in a submodule not in the repo |
| Voxel hit threshold | 64 | §IV.C | Chosen by the authors from experience |
| Weight threshold for hits | not stated | §IV.C fig. 4 | Paper: not stated |
| Bonus for map-point voxels | "significant number", value not stated | §IV.C | Paper: not stated |
| Learning rate, batch, steps, samples per ray | not stated | — | Paper: not stated; code in submodule |
| Local-map inlier checks | 50 (after relocalisation) and 30 | Tracking.cu | Not described in the paper |
Results
Author-reported. Setups are as stated in the paper. Orbeez-SLAM rows do not say whether the monocular or RGB-D mode produced them §V.B. Baseline numbers are mostly copied from NICE-SLAM §V.B (Table I caption).
Tracking, TUM RGB-D, ATE RMSE in cm, best of 5 runs tab. I. Columns are the three TUM sequences in order of the table: desk (fr1), xyz (fr2), office (fr3).
| Method | desk | xyz | office |
|---|---|---|---|
| DI-Fusion | 4.4 | 2.3 | 15.6 |
| iMAP | 4.9 | 2.0 | 5.8 |
| iMAP* | 7.2 | 2.1 | 9.0 |
| NICE-SLAM | 2.7 | 1.8 | 3.0 |
| Orbeez-SLAM | 1.9 | 0.3 | 1.0 |
| BAD-SLAM | 1.7 | 1.1 | 1.7 |
| Kintinuous | 3.7 | 2.9 | 3.0 |
| ORB-SLAM2 | 1.6 | 0.3 | 0.9 |
Tracking, ScanNet, six scenes, ATE RMSE in cm tab. II
| Method | 0000 | 0059 | 0106 | 0169 | 0181 | 0207 | Avg. |
|---|---|---|---|---|---|---|---|
| DI-Fusion | 62.99 | 128.00 | 18.50 | 75.80 | 87.88 | 100.19 | 78.89 |
| iMAP* | 55.95 | 32.06 | 17.50 | 70.51 | 32.10 | 11.91 | 36.67 |
| NICE-SLAM | 8.64 | 12.25 | 8.09 | 10.28 | 12.93 | 5.59 | 9.63 |
| Orbeez-SLAM | 7.22 | 7.15 | 8.05 | 6.58 | 15.77 | 7.16 | 8.655 |
| ORB-SLAM2 | 7.57 | 6.92 | 8.30 | 6.90 | 16.42 | 8.78 | 9.15 |
Mapping, Replica, averaged over office 0–4 and room 0–2 tab. III
| Method | Depth L1 cm, no GT | Depth L1 cm, GT | PSNR dB, no GT | PSNR dB, GT |
|---|---|---|---|---|
| NICE-SLAM | 13.49 | 4.22 | 17.74 | 24.60 |
| Orbeez-SLAM | 11.88 | — | 29.25 | — |
NICE-SLAM uses ground-truth depth during rendering in the GT columns tab. III. Its source for these values is not stated.
Speed, frames per second, TUM RGB-D tab. IV §V.B
| Method | fr1/desk (613 images) | fr2/xyz (3669) | fr3/office (2585) |
|---|---|---|---|
| NICE-SLAM | 0.056 | 0.028 | 0.037 |
| Orbeez-SLAM | 19.210 | 22.725 | 21.542 |
Ratios computed from tab. IV, not stated in the paper: about 343x, 811x and 582x. The text reports 360 to 800 times §V.B and the abstract says up to 800x.
Ablation, Replica tab. V §V.C
| Pose loss | ATE cm | Depth L1 cm | PSNR dB |
|---|---|---|---|
| L_rpj + L_pht | 5.3 | 13.43 | 25.52 |
| L_rpj only (final) | 0.8 | 11.88 | 29.25 |
The "L_rpj only" row matches the Orbeez-SLAM row of tab. III. The authors say L_pht alone "produces horrible results" and do not report it §V.C.
Paper vs code
Code: https://github.com/MarvinChung/Orbeez-SLAM at 559323e81563c0dacf7175a1089b95f3d0c7d204.
The paper's abstract links this repo. The README says it is from the author's
master thesis, but it does not list the author names, so ownership is inferred
from that link and the README, not verified. The repo is GPL-3.0 and inherits ORB-SLAM2.
Its NeRF core is in a submodule (Thirdparty/instant-ngp-kf), which is not in
the repo and was not read.
- Photometric pose loss. The paper uses only the reprojection loss for poses
§IV.B §V.C. The three monocular examples start the system with
bTrainExtrinsicWithPhotometric = truemono_tum.cu mono_replica.cu mono_scannet.cu. That flag setsoptimize_extrinsicson the NeRF Map.cu. The TUM example prints "+ photometric error if optimize extrinsic == true" [code: Examples/Monocular/mono_tum.cu#L171]. Whether the published tables used this path is not stated. - Keyframe culling. The paper says keyframes are not culled §V.A. The
code defines
KeyFrameCullingNerfMapping.cu but the mapping loop does not call it NerfMapping.cu. Consistent with the paper. - Keyframe rule. Fig. 3 gives two conditions fig. 3. The code has three time/tracking triggers and one ratio test Tracking.cu. The weak-tracking trigger is off for monocular Tracking.cu. The ratio is 0.9 for monocular Tracking.cu, which the paper does not state.
- Loss form. Eq. (6) sums unsquared L2 norms eq. 6. The code minimises a squared error with Huber robust kernels and chi-square outlier checks [code: src/Optimizer.cu#L104-L152] Optimizer.cu. The paper does not mention these.
- Ray-casting triangulation. No implementation in this repo. It lives in the submodule, so the 64 threshold and the bonus value cannot be checked here.
- Feature count. Configs use 1000 ORB features for TUM and 2000 for Replica. The paper does not give a feature count.
No other discrepancies found within the parts of the code read.
Pitfalls
- Which mode produced Tables I–III? The paper says all methods were run in the depth version §V.A. Orbeez-SLAM's mode in those tables is not stated, and Fig. 5 shows both monocular and RGB-D fig. 5.
- Pose loss in shipped code. The monocular examples turn on photometric pose training, which the paper's ablation argues against §V.C. Start from the paper's setting (reprojection only) and do not inherit the example flag.
- Mapping metrics are not mesh metrics. Depth L1 and PSNR are computed along the GT trajectory using only keyframes for training §V.A. They are not comparable with mesh-based numbers in other NeRF-SLAM papers.
- Source of baselines. Table I and II captions say most baseline numbers come from NICE-SLAM tab. I tab. II. ORB-SLAM2's source and NICE-SLAM's Replica values are not stated. The TUM rows are "best out of 5 runs" tab. I. ScanNet run count is not stated.
- Small margins. On ScanNet the average is 8.655 against ORB-SLAM2 at 9.15 and NICE-SLAM at 9.63 tab. II. The authors attribute the gap to ORB-SLAM2 to "randomness" §V.B. No variance is reported.
- Speed claim. The text says 360 to 800 times faster §V.B. Tab. IV gives about 343x to 811x. The measurement method (what is timed, whether mapping counts, how fps is computed) is not stated beyond "whole tracking and mapping process" §V.B.
- Voxel hit definition. The text says the counter records how often rays sample each voxel §IV.C. Fig. 4 says the counter goes up when a voxel's weight exceeds a threshold fig. 4. The weight threshold and bonus values are not stated, so the two readings cannot be resolved from the paper.
- Skip-voxel occupancy. The occupancy rule and grid update schedule are not stated in the published text §III.B fig. 2.
- Large scenes. The NeRF fails to reconstruct one desk in ScanNet 0207 and ScanNet 0181 has compartments that the authors single out §V.B fig. 6. The paper names large scenes as future work.
- No conclusion in the source. The LaTeX conclusion section has no live text. Do not cite a conclusion.
- Licensing for implementers. The paper is CC BY 4.0. The code is GPL-3.0, so do not copy code; implement from the paraphrase. The NeRF submodule is a fork of NVIDIA instant-ngp under the NVIDIA Source Code License-NC, which bars commercial use. The README says so README.md. Check that before building the NeRF part.
- Hardware and time. Runtime and memory are measured on one RTX 3090 with an i7-9700 §V.A. Scaling to other hardware is not stated.
Lineage
- Builds on: ORB-SLAM2 (feature-based SLAM, reprojection and bundle adjustment, loop closing) §I §IV.B §V.A, and instant-ngp (multi-resolution hash encoding, density grid, skip voxels) §I §III.B. Neither has a card here.
- Compared against: NICE-SLAM, iMAP, iMAP*, DI-Fusion, BAD-SLAM, Kintinuous, ORB-SLAM2 tab. I tab. II tab. III. Tandem is discussed §I but not in the tables.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping” by Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, Wen-Chin Chen, Yi-Ting Chen, Winston H. Hsu (arXiv:2209.13274), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/MarvinChung/Orbeez-SLAM at commit 559323e81563 (GPL-3.0). Any quoted code is shown under that license.