BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

In plain words

BundleSDF follows an object through a short video recorded with a depth camera, and builds a 3D model of it at the same time. You only mark the object in the first frame. The system needs no 3D model of the object beforehand and no knowledge of what is moving it, so a hand, a person or a robot arm all work the same way.

Two parts run side by side. The first keeps a small set of stored past views, and it adjusts the estimated position of the object so that all of these views agree with each other. The second learns a neural model of the object's shape and colour from those same views. Each part helps the other: the shape model corrects old positions, and corrected positions give the shape model cleaner data.

The method is meant for hard cases: objects with few visual markings, objects that are partly hidden, and shiny objects. Those are the cases where simpler trackers drift.

TL;DR

BundleSDF estimates the 6-DoF pose of an unknown rigid object from a monocular RGB-D video, using a mask of the object in the first frame only §1 §3. It runs a pose-graph optimiser over a memory pool of keyframes in near real time, about 10 Hz on the authors' desktop app. B. A per-video neural signed distance field learns the object shape and feeds corrected poses back to the tracker §3.4. Author-reported results are best on HO3D and BEHAVE, and best on most metrics on YCBInEOAT, where ADD is close to BundleTrack tab. 1 tab. 2 tab. 3.

Problem

The task is to track the pose of an object that is unknown in advance, from one camera, while also reconstructing its shape §1. Instance-level trackers need a textured CAD model, and category-level methods struggle with unseen objects §1. The method assumes a rigid object, a mask in the first frame, and causal processing that never reads future frames §3. It must cope with an interacting agent (hand, body or robot arm) that causes severe occlusion, and with objects that have little texture, specular highlights or motion blur §1 app. E.

Fusing each frame into one global model accumulates pose errors that cannot be undone, and long complete occlusion makes re-registration to that model hard §3.2.

Method

The method has three parts that run in two threads fig. 2. A tracking thread estimates each frame's pose from a coarse guess that is refined by a pose graph over selected memory frames §3.1 §3.3. A keyframe memory pool stores posed frames §3.2. A field thread trains a neural object field on the memory pool and writes refined poses back §3.4. Everything is causal §3.

Coarse pose §3.1. The object region is segmented in each new frame by an object-agnostic video segmenter, chosen so that no knowledge of the object or agent is needed §3.1. RGB correspondences to the previous frame come from a transformer matcher pretrained on internet photos §3.1. Depth lifts the matches to 3D, a RANSAC estimator with least squares gives the hypotheses, and the hypothesis with the most inliers is kept §3.1.

Memory pool §3.2. The first frame is always stored and defines the canonical coordinate frame §3.2. A new frame is stored only when its viewpoint adds diversity. Rotation about the camera's optical axis is ignored in that comparison, because in-plane rotation adds no new view information §3.2.

Subset selection §3.3. For each stored frame, the method computes normals and tests visibility of the object in the new frame. A frame is a candidate only if the visible fraction of its points exceeds 0.1 §3.3. Among candidates, the K frames with the best viewing overlap are kept, with K = 10 in the experiments §3.3 app. A.

Pose graph §3.3. The graph has the new frame and the selected memory frames as nodes eq. 1. Its loss sums a feature term, a point-to-plane term and a unary term that compares the frame with the learned field eq. 1 eq. 2 eq. 3 eq. 4. Only memory frames that the field has not yet refined are moved by the pose graph, and the field term is switched on after the field's first training has converged §3.3.

Neural object field §3.4. The object is a signed distance function, with colour predicted by a second network that takes the geometry feature, the surface normal and the view direction §3.4. Normals come from the gradient of the distance function §3.4. The field trains per video in its own thread on all memory frames, with the memory-frame poses as trainable offsets anchored on the first frame §3.4. After each round it writes the poses back to the memory pool §3.4.

Rendering and sampling §3.4. A ray's colour integrates a bell-shaped weight around the surface over a window that starts one truncation distance in front of the observed depth and ends half a truncation distance behind it eq. 6 eq. 7. Points are drawn uniformly along each ray in occupied octree voxels, plus extra points drawn near the observed depth app. A.

Hybrid SDF losses §3.4. The space along each ray is split into uncertain free space, empty space and near-surface space §3.4. Each region has its own loss eq. 8 eq. 9 eq. 10. The total loss combines these with a colour loss and an Eikonal regulariser eq. 11 eq. 12 eq. 13.

# Per-frame tracking step  [§3, fig. 2]
# Input:  rgb (H, W, 3), depth (H, W), pool (memory frames), field (SDF)
# Output: object pose xi_t in the first frame's coordinate system

def track_frame(rgb, depth, pool, field):
    # Segment the object, match RGB features to the previous frame, and
    # lift matches to 3-D with depth, then estimate a coarse pose  [§3.1]
    mask = segment_object(rgb)
    matches = match_features(rgb, prev_rgb, mask)
    coarse_pose = ransac_least_squares(matches, depth, prev_depth)

    # Keep at most K memory frames that see the object well  [§3.3] [app. A]
    # Candidates need a visibility ratio above 0.1 [§3.3]
    selected = select_memory(pool, coarse_pose, K)

    # Refine the new pose and the selected memory poses jointly  [§3.3]
    # Memory frames already refined by the field stay fixed  [§3.3]
    pose, new_poses = pose_graph_optimise(coarse_pose, selected, field)

    # Store the frame only if its viewpoint adds diversity  [§3.2]
    maybe_add_to_pool(pool, rgb, depth, mask, pose)
    return pose

# Neural object field round, run in its own thread  [§3.4] [app. A]
# Input:  pool (posed memory frames), field (current network weights)
# Output: refined memory poses, written back to the pool

def field_round(pool, field):
    # Re-initialise the network weights for every round  [app. A]
    # Keep the per-frame pose offsets from the last round  [app. A]
    field.reset_network_weights()
    for step in range(training_steps):  # 300 stated  [app. A]
        # Draw 2048 rays; sample N = 128 points in occupied voxels and
        # N' = 64 points near the observed depth  [app. A]
        rays = sample_rays(pool, batch_size=2048)
        loss = hybrid_sdf_loss(rays, field)  # [eq. 8]-[eq. 13]
        adam_update(field, loss)  # Adam, learning rate 0.01  [app. A]
    write_back_poses(pool, field)  # [§3.4]

Hyperparameters

Values are the paper's unless marked code-only. "Not stated" means the paper gives no value and the code was not used to fill it in. Code values are cited to the official repository at the commit above.

Results

Author-reported. Setup: three real datasets, single monocular RGB-D view, first-frame mask only, no re-initialisation after tracking failure §4.1 §4.3. Every method gets the same segmentation masks where it accepts them §4.3. ADD and ADD-S are AUC percentages; the threshold is 0.1 m for HO3D and YCBInEOAT and 0.5 m for BEHAVE tab. 1 tab. 2 tab. 3. Chamfer distance (CD) is in cm; lower is better.

HO3D v3 evaluation set: 4 objects, 13 sequences, 20428 frames §4.1 tab. 1.

YCBInEOAT: 5 objects, 9 videos, 7449 frames, dual-arm robot, egocentric view §4.1 tab. 2. Starred rows are copied from the BundleTrack paper, not re-run §4.5.

BEHAVE: 16 objects, 70 videos, 107982 frames, single-view setting with one camera chosen by ground-truth visibility §4.1 app. D tab. 3. ADD thresholds are up to 0.5 m.

Ablation on HO3D, as reported in the paper's ablation table §4.7 tab. 4:

Speed app. B: tracking runs at about 10.2 Hz; segmentation takes 18 ms, coarse matching 24 ms and the pose graph 56 ms on average. The field thread takes 6.7 s per round. DROID-SLAM runs at 6.1 Hz and BundleTrack at 11.2 Hz on the same hardware. Hardware: Intel i9-10980XE CPU and one NVIDIA RTX 3090 GPU app. B.

Per-video results and recall curves are in the appendix app. D and are not reproduced here.

Paper vs code

Searched: the paper text names no code link. The project page links the repository https://github.com/NVlabs/BundleSDF [project page], which is under the NVIDIA organisation, and the authors are NVIDIA researchers. Cloned at commit ffa67d425240b5b76d2e387a7dd3d3735a7cf1a1.

Licence: NVIDIA Source Code License-NC (non-commercial). Its use clause limits use to research or evaluation LICENSE.txt. This card cites the code by path and line and quotes no code.

Weights in the field loss. The paper's stated weights app. A are not the effective weights in the code. The field-loss terms are multiplied by factors that the paper does not mention: the free-space and surface terms are scaled by 0.5 inside the mask helper nerf_helpers.py, and then by fs_weight 100 and trunc_weight 6000 at the call site nerf_runner.py. The empty-space term is added after the 0.5 factor, so it keeps an effective weight of 1 nerf_helpers.py. Effective values: free space 50 (paper 100), surface 3000 (paper 1000), colour 10 (paper 100, from rgb_weight 10 config.yml).

Eikonal term. The paper uses weight 0.1 app. A; the code sets eikonal_weight 0, which disables the term config.yml nerf_runner.py.

Colour loss form. The paper writes an unsquared L2 norm eq. 12. The code uses a squared error averaged over rays nerf_runner.py.

Training length. The paper says 300 steps per round app. A; the code's default is 500 config.yml.

Rendering weight. The paper's bell-shaped weight is a function of the learned signed distance eq. 7. The code computes it from the observed depth instead: a sigmoid of the depth distance over the truncation, multiplied by a window, then normalised to sum to one. The signed distance argument is not used nerf_runner.py. A code comment says this "does not work well" nerf_runner.py. The code keeps sdf_lambda 5 for this sharpness config.yml. The paper's alpha is "set to a constant" and its value is not stated.

Integration window. The paper integrates from one truncation distance in front of the depth to half a truncation distance behind it eq. 6. The code's window is symmetric, one truncation distance on each side, because neg_trunc_ratio is 1 config.yml nerf_runner.py. Sampling in occupied voxels also stops at one truncation distance behind the depth, not half nerf_runner.py.

Extra samples. The paper draws N' points from a normal distribution centred on the observed depth §3.4. The code draws them uniformly between the depth minus and plus one truncation distance nerf_runner.py.

Memory-pool selection. The paper ranks candidates by rotation distance that ignores in-plane rotation, after a visibility threshold of 0.1 §3.3. The configured subset method normal_orientation_nearest ranks keyframes by a covisibility score and keeps the top max_BA_frames config_ho3d.yml Bundler.cpp. The rotation-ignoring distance is used for the keyframe rule instead: a new keyframe must differ by at least 5 deg from every existing keyframe Bundler.cpp Bundler.cpp Utils.cpp.

Pose freezing. The paper freezes a memory frame's pose once the field has updated it §3.3. The code sets a nerfed flag on every keyframe after each field sync and then overwrites all keyframe poses bundlesdf.py. Whether the pose graph honours the flag was not traced.

Extra mechanisms not in the paper: a 100 px dilation of the first-frame mask and 60 px for later frames nerf_runner.py; removal of depth points more than 0.02 m from an octree cloud nerf_runner.py; bounds on pose correction at 0.02 m and 20 deg config.yml; and sample weight 10 on first-frame rays config.yml nerf_runner.py.

Segmenter. The paper uses an object-agnostic video segmenter §3.1. The repository does not include it, for licence reasons, and asks users to add their own wrapper readme.md.

Per-dataset settings. The paper does not state per-dataset parameters. The repository ships separate tracker configs for HO3D, YCBInEOAT and BEHAVE config_ho3d.yml, config_ycbineoat.yml, config_behave.yml, but one shared field config config.yml. Differences between the three tracker configs were not compared.

Pitfalls

• Units of the field. The code parametrises the signed distance in truncation units. The surface term multiplies it by the truncation distance nerf_helpers.py nerf_runner.py. So epsilon, the empty-space target and the sign of distances are all in truncation units. The paper gives epsilon = 0.001 without units app. A. Read as metres, 0.001 m is a tenth of the 1 cm truncation distance. The code's value is 0.001 truncation units, which is 1e-5 m.
• Equation numbers were counted from the LaTeX source, since no PDF was read. If they differ in the PDF, cite the section.
• The pose-graph sum runs over ordered pairs i and j with i not equal to j, so each pair appears twice eq. 1. The paper does not say whether this double count is intended.
• Visibility filter. The 0.1 threshold appears in the paper's candidate rule §3.3, and in the repository it is non_neighbor_min_visible for a different selection path config_ho3d.yml. Do not assume the two are the same test.
• Point rejection angle. The paper says 20 deg app. A. The config lists 30 deg for feature_edge_normal_thres config_ho3d.yml. Which value applies to L_p was not traced.
• Uncertain free space. The paper's term applies to background pixels in the mask §3.4. The code applies its free-space term where the observed depth exceeds the far bound nerf_helpers.py. The full mapping from background pixels to these terms was not traced.
• Table inconsistency. The full method's chamfer distance on HO3D is 0.57 cm in the main table tab. 1 and 0.61 cm in the ablation table tab. 4. Use the main-table value and note the difference.
• Thresholds differ. ADD and ADD-S use 0.1 m for HO3D and YCBInEOAT, but 0.5 m for BEHAVE tab. 3. Do not compare the BEHAVE numbers directly with the others.
• Baselines. YCBInEOAT rows marked with an asterisk are copied from the BundleTrack paper and were not re-run §4.5. BundleTrack was re-run with this paper's masks and TSDF fusion for reconstruction §4.5.
• Depth is required. The method does not work for transparent objects, where depth sensing fails, and it fails when occlusion, segmentation error and lack of texture happen together app. F.
• Debugger left in code. When a frame has too few matches with its reference, the code calls the Python debugger, which halts the run bundlesdf.py.
• Data directory. The code deletes and recreates the configured data directory on every field round bundlesdf.py. Point it at a dedicated directory.
• Hard-coded paths. The configs point to paths on the authors' machine config.yml config_ho3d.yml.
• Hidden costs. The pose graph and the matcher need a CUDA GPU §3.3 . The field needs a few seconds of GPU training per round app. B.
• Interaction agent. The method makes no assumption about the agent §1. The 10 Hz figure comes from the authors' desktop app. B.

Lineage

• Builds on: BundleTrack, whose pairwise feature and point-to-plane losses the pose graph reuses §3.3. The memory pool replaces the global fused model of earlier fusion methods §3.2. The bell-shaped weight is the one cited as [wang2021neus] §3.4. The hash encoding is the multi-resolution one cited as [mueller2022instant] §3.4. The object field is inspired by [yariv2020multiview] §3.4.
• Compared against: DROID-SLAM, NICE-SLAM, KinectFusion, SDF-2-SDF and BundleTrack §4.3.
• Superseded by: none known from content in this wiki.