Paper · 2021-08-01 · cs.CV, cs.AI, cs.GR, cs.RO
BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models
BundleTrack tracks the 6D pose of a rigid object in RGB-D video without any CAD model. Each frame it propagates the object mask, matches learned keypoints to the previous frame for a coarse pose, then refines a few memory keyframes jointly in a pose graph using feature and depth terms. Author-reported 87.4% under 5 deg / 5 cm on NOCS (6-PACK, category-level: 33.3%); about 10 Hz on one GPU.
Pose-graph optimizationKeyframe selectionLocal feature matchingOutlier rejectionpose-tracking6d-posemodel-freepose-graphrobot-manipulation
arXivPDFCodeMarkdown18 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A robot often needs to know where a held object is, and the object's position can change quickly while it is moved. Most tracking methods need a 3D model of the object, either the exact one or a model of its whole category. This paper asks what is possible without any model at all.
The idea is to follow the object from frame to frame, the way one follows a moving thing by remembering how it looked before. A segmentation step first says which pixels belong to the object. Points that look distinctive are then matched between the new frame and the previous one to make a rough guess of how the object moved. That guess is not trusted alone: a small set of earlier "keyframes" from different viewing angles is kept, and all of these views are adjusted together so that they agree with each other and with the depth images.
Because the earlier views are kept rather than merged into one 3D model, a mistake made in one frame can be corrected later instead of being baked into the model. The authors aim for tracking that stays accurate over long videos, even with heavy occlusion and moving objects.
TL;DR
BundleTrack estimates the 6D pose of a novel rigid object from RGB-D video without an instance or category CAD model §I. It has three parts: video object segmentation to get the object mask §IV.A, keypoint matching for a coarse frame-to-frame pose §IV.B, and an online pose-graph optimisation over the current frame and a bounded set of keyframes §IV.C §IV.D. Author-reported results are strong on NOCS and competitive on YCBInEOAT, and the code is Apache-2.0 for the repository itself (see the licence notes below).
Problem
Forward kinematics is often not enough to know an object's pose during manipulation, because of slippage, handoffs and suction-cup compliance §I. Single-image pose estimators are redundant when run on every frame and give less coherent estimates over consecutive frames §I. Trackers that do use temporal information usually rely on an instance or category CAD model §I. The paper targets rigid objects for which no 3D model exists, for both training and registration §III. Its stated goal is long-term, low-drift tracking under significant occlusion and object motion [abstract].
Inputs are an RGB-D sequence, a binary mask on frame 0 and an optional initial pose in the camera frame §III. The tracker computes the relative transform from the first frame to each timestamp, and this is mapped to a pose in the camera frame §III.
Method
The pipeline runs once per frame, in order, and is shown in fig. 2. The mask is propagated, a coarse relative pose is computed, keyframes are selected from a memory pool, a pose graph is optimised, and the new frame is added to the pool if it shows a new view §IV, overview.
A. Mask propagation §IV.A. A transductive video object segmentation network propagates the previous binary mask to the current frame. It was trained on DAVIS 2017 and YouTube-VOS §IV.A. Only a binary mask is needed, and the initial mask may come from any method, such as semantic segmentation, plane removal or clustering §IV.A §III.
B. Keypoints, matching and coarse pose §IV.B. Keypoints and descriptors come from LF-Net. Each frame gives n = 500 keypoints with 128-dimensional descriptors §IV.B. Matches are found between the two frames and pruned with RANSAC. A 3-pair sample is turned into a pose by least squares. An inlier has a point distance below 5 mm and a normal angle below 45 degrees §IV.B. The coarse pose is T_tilde_t = T_{t-1} T_t^{t-1}, where T_t^{t-1} is the best sampled hypothesis §IV.B.
C. Keyframe selection §IV.C. At most K = 15 keyframes take part in the optimisation §IV.C. Keyframe 0 is always included as the drift-free reference. The rest are chosen greedily: each step adds the candidate with the smallest sum of geodesic rotation distances to the current frame and to all already-chosen keyframes §IV.C. The paper notes that exact selection is combinatorial and that the greedy rule keeps the cost small §IV.C.
D. Online pose graph §IV.D. The graph has one node per selected timestamp plus the current one, so |V| = k + 1 with k at most K §IV.D. Edges carry two residual types. The feature term E_f uses matched 3-D points between two nodes eq. 2. The geometric term E_g uses dense pixel correspondences and a point-to-plane depth residual eq. 3. The total energy sums both terms over node pairs with weights lambda_1 = lambda_2 = 1 eq. 1 §IV.D. Both residuals use a Huber M-estimator. The pose of frame 0 is held fixed §IV.D. Poses are updated in se(3) by iteratively re-weighted Gauss-Newton, with the normal equations solved by preconditioned conjugate gradients (PCG) §IV.D. The output is causal: the corrected poses of past frames are not changed in the output, but they serve as better initial values for later graphs §IV.D.
E. Memory update §IV.E. Frame 0 is always kept. A later frame joins the pool only if its rotation differs from every existing keyframe by more than the threshold alpha, written as arccos(10 deg) in the paper §IV.E.
# Per-frame tracking loop [§IV] [fig. 2]
# Inputs: frames (RGB-D per timestamp), mask_0 (binary, frame 0)
# trials, delta, normal_limit: see Hyperparameters
# Output: object pose in the camera frame per timestamp (4x4, SE(3))
def track(frames, mask_0):
keyframes = [0] # frame 0 is always kept [§IV.E]
mask = mask_0
poses = {0: initial_pose} # identity unless T_0^C is given
for t in range(1, len(frames)):
# Propagate the object mask to frame t [§IV.A]
mask = propagate_mask(frames[t - 1], frames[t], mask)
# Keypoint matches to the previous frame [§IV.B]
matches = match_keypoints(frames[t - 1], frames[t], mask)
# Coarse pose from the matches [§IV.B]
coarse = coarse_pose(matches, poses[t - 1], trials, delta,
normal_limit)
# Choose the graph nodes, at most K keyframes [§IV.C]
rotations = [rotation(poses[k]) for k in keyframes]
nodes = select_keyframes(rotations, rotation(coarse), max_nodes=15)
# Jointly refine the current pose and the nodes [§IV.D]
poses[t] = pose_graph_refine(t, [keyframes[n] for n in nodes], coarse)
# Keep the frame if it shows a new viewing direction [§IV.E]
if is_novel_view(poses[t], [poses[k] for k in keyframes]):
keyframes.append(t)
return poses# Coarse relative pose from keypoint matches [§IV.B]
# Inputs: matches (3-D point pairs in the two camera frames)
# Output: pose of frame t, chained from the previous pose (4x4)
def coarse_pose(matches, previous_pose, trials, delta, normal_limit):
best_relative = None
best_count = 0
for _ in range(trials):
# Sample 3 pairs and fit a rigid transform by least squares [§IV.B]
sample = random.sample(matches, 3)
relative = rigid_fit(sample)
# Inlier: point distance below delta (5 mm) and normal angle
# below normal_limit (45 deg) [§IV.B]
inliers = [m for m in matches
if residual(relative, m) < delta
and normal_angle(relative, m) < normal_limit]
if len(inliers) > best_count:
best_count = len(inliers)
best_relative = relative
# Chain onto the previous pose: T_tilde = T_{t-1} T_t^{t-1} [§IV.B]
return previous_pose @ best_relative# Greedy keyframe subset for the graph [§IV.C]
# Inputs: rotations (one 3x3 per keyframe), current (3x3), max_nodes
# Output: indices of the selected keyframes
def select_keyframes(rotations, current, max_nodes):
# Keyframe 0 is the reference and is always chosen [§IV.C]
chosen = [0]
while len(chosen) < max_nodes:
unchosen = [k for k in range(len(rotations)) if k not in chosen]
if not unchosen:
break
# Smallest summed geodesic distance to the current frame and
# to every chosen keyframe [§IV.C]
best = min(unchosen, key=lambda k: sum(
geodesic(rotations[k], rotations[s]) for s in chosen)
+ geodesic(rotations[k], current))
chosen.append(best)
return chosen# Pose-graph energy over the node poses [§IV.D] [eq. 1]
# Nodes: current frame and selected keyframes; frame 0 pose is fixed
# Feature term per pair [eq. 2]; dense point-to-plane term per pair [eq. 3]
def graph_energy(poses, pairs, lambda_f, lambda_g):
total = 0.0
# Eq. (1) sums over ordered pairs; each unordered pair is used once
# here, which only rescales the energy and keeps the minimiser [§IV.D]
for (i, j) in pairs:
# Huber-robust distance between matched 3-D points [eq. 2]
total += lambda_f * feature_term(
poses[i], poses[j], matches[(i, j)])
# Huber-robust point-to-plane depth residual [eq. 3]
total += lambda_g * dense_term(
poses[i], poses[j], frames[i], frames[j])
return total# Robust Gauss-Newton with PCG inside [§IV.D]
# Unknowns: se(3) increments for every free node (6 per node)
def refine(poses, residuals, jacobian, outer, inner, delta):
for _ in range(outer):
# Huber weights from the current residuals [§IV.D]
weights = huber_weights(residuals(poses), delta)
# Normal equations, solved approximately by PCG with the
# diagonal of J^T W J as preconditioner [§IV.D]
lhs = jacobian.T @ weights @ jacobian
rhs = jacobian.T @ weights @ residuals(poses)
step = pcg(lhs, rhs, iterations=inner)
# Update in the tangent space and map back to SE(3) [§IV.D]
poses = retract(poses, step)
return posesHyperparameters
Values marked "not stated" are absent from the paper. Code values come from the
released configuration files for NOCS (config_nocs.yml) and YCB-Video-in-EOAT
(config_ycbineoat.yml) at the commit in the frontmatter. Code values are not
paper values and are cited separately.
| Name | Value | Where | Notes |
|---|---|---|---|
| keypoints per frame n | 500 | §IV.B | Server default top_k 500 run_server.py; not in the YAML |
| descriptor dimension | 128 | §IV.B | LF-Net output |
| LF-Net input size | 400 x 400 | not stated | Crop to a square around the mask, then resize FeatureManager.cpp |
| RANSAC sample size | 3 pairs | §IV.B | config_nocs.yml |
| RANSAC iterations | not stated | not stated | Code 2000 config_nocs.yml |
| inlier distance delta | 5 mm, all experiments | §IV.B | NOCS config 0.005 m config_nocs.yml; YCB config 0.01 m config_ycbineoat.yml |
| inlier normal angle | 45 deg | §IV.B | config_nocs.yml; not applied in RANSAC, see Paper vs code |
| keyframe limit K | 15 | §IV.C | Code caps graph nodes at 15 including the current frame config_nocs.yml |
| keyframe novelty threshold alpha | arccos(10 deg), read as 10 deg | §IV.E | Code: 10 deg geodesic config_nocs.yml |
| lambda_1, lambda_2 | 1 and 1 | §IV.D | Stated "unless otherwise specified" |
| Huber delta | not stated | not stated | Code 0.005 m config_nocs.yml and config_ycbineoat.yml |
| Gauss-Newton outer iterations | not stated | not stated | Code 7 config_nocs.yml; read at Bundler.cpp |
| PCG inner iterations | not stated | not stated | Code 5 config_nocs.yml; read at Bundler.cpp |
| minimum feature edges to the new frame before optimising | not stated | not stated | Code 10 NOCS config_nocs.yml, 5 YCB config_ycbineoat.yml; below it the frame gets status NO_BA Bundler.cpp |
| dense-term image downscale | not stated | not stated | Code 4 config_nocs.yml |
| dense point-to-plane gating | not stated | not stated | Code distance 0.02 m, normal 45 deg config_nocs.yml |
| feature gating, non-neighbour pairs | not stated | not stated | Code 0.02 m and 45 deg config_nocs.yml |
| feature gating, neighbour pairs | not stated | not stated | Code 10000 and 180 (NOCS) config_nocs.yml; 0.03 m and 45 deg (YCB) config_ycbineoat.yml |
| mutual nearest-neighbour matching | not stated | not stated | Code on config_nocs.yml |
| minimum matches per pair | not stated | not stated | Code 5 FeatureManager.cpp |
| depth erosion | not stated | not stated | Radius 1, diff 0.001, ratio 0.8 config_nocs.yml |
| depth bilateral filter | not stated | not stated | Radius 2, sigma_D 2, sigma_R 100000, applied twice config_nocs.yml Frame.cpp |
| mask dilation | not stated | not stated | 5 x 5 rectangle Frame.cpp |
| network training | none in this work | §V.D | Weights and hyperparameters fixed across all evaluations |
| hardware | Xeon E5-1660 v3 3.0 GHz, one RTX 2080 Ti | §V | |
| run time | about 10 Hz including video segmentation | §V.D | Author-reported |
Results
Author-reported. Setup: NOCS real test set (6 categories, 18 unseen instances, 3,200 frames) and YCBInEOAT (9 videos, 5 objects, static camera with moving objects) §V.A. On NOCS the initial pose is a perturbed ground-truth pose; on YCBInEOAT it is the ground-truth pose. Neither benchmark allows re-initialisation during tracking §V.B §V.C. The NOCS metric is 5 deg / 5 cm accuracy with IoU25 and mean rotation and translation errors §V.B. YCBInEOAT uses ADD and ADD-S area under the curve §V.C. Baselines that use category or instance CAD models are marked as such in the paper tab. I tab. II.
Table I. NOCS results tab. I, all values as printed in the paper. Columns are the six categories and the overall mean.
| Assumption | Method | Metric | bottle | bowl | camera | can | laptop | mug | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Category-level 3D model | NOCS | 5 deg 5 cm | 5.5 | 62.2 | 0.6 | 7.1 | 25.5 | 0.9 | 17.0 |
| IoU25 | 48.7 | 99.6 | 90.6 | 77.0 | 94.7 | 82.8 | 82.2 | ||
| R_err | 25.6 | 4.7 | 33.8 | 16.9 | 8.6 | 31.5 | 20.2 | ||
| T_err | 14.4 | 1.2 | 3.1 | 4.0 | 2.4 | 4.0 | 4.9 | ||
| KeypointNet | 5 deg 5 cm | 5.9 | 16.8 | 1.8 | 4.3 | 49.2 | 3.1 | 13.5 | |
| IoU25 | 23.1 | 74.7 | 30.9 | 42.6 | 94.6 | 52.0 | 53.0 | ||
| R_err | 28.5 | 9.8 | 45.2 | 28.8 | 6.5 | 61.2 | 30.0 | ||
| T_err | 9.5 | 8.2 | 8.5 | 13.1 | 4.4 | 6.7 | 8.4 | ||
| 6-PACK without temporal | 5 deg 5 cm | 23.7 | 53.0 | 8.4 | 25.0 | 62.4 | 22.4 | 32.5 | |
| IoU25 | 92.0 | 100.0 | 91.0 | 89.9 | 97.8 | 100.0 | 95.1 | ||
| R_err | 15.7 | 5.3 | 43.9 | 12.5 | 4.9 | 20.3 | 17.1 | ||
| T_err | 4.2 | 1.6 | 5.5 | 5.0 | 2.5 | 1.8 | 3.4 | ||
| 6-PACK | 5 deg 5 cm | 24.5 | 55.0 | 10.1 | 22.6 | 63.5 | 24.1 | 33.3 | |
| IoU25 | 91.1 | 100.0 | 87.6 | 92.6 | 98.1 | 95.2 | 94.2 | ||
| R_err | 15.6 | 5.2 | 35.7 | 13.9 | 4.7 | 21.3 | 16.0 | ||
| T_err | 4.0 | 1.7 | 5.6 | 4.8 | 2.5 | 2.3 | 3.5 | ||
| No model | ICP | 5 deg 5 cm | 10.1 | 40.3 | 12.6 | 17.2 | 14.8 | 6.2 | 16.9 |
| IoU25 | 29.9 | 79.7 | 53.1 | 40.5 | 50.9 | 27.7 | 47.0 | ||
| R_err | 48.0 | 19.0 | 80.5 | 47.1 | 37.7 | 56.3 | 48.1 | ||
| T_err | 15.7 | 4.7 | 12.2 | 9.4 | 9.2 | 9.2 | 10.5 | ||
| TEASER++ (on our segmented point cloud and correspondences) | 5 deg 5 cm | 13.9 | 35.5 | 10.7 | 11.7 | 40.9 | 7.5 | 20.0 | |
| IoU25 | 100.0 | 99.9 | 99.9 | 100.0 | 99.9 | 99.9 | 99.9 | ||
| R_err | 17.0 | 10.6 | 18.8 | 20.4 | 7.2 | 23.0 | 16.2 | ||
| T_err | 2.7 | 1.8 | 2.8 | 2.7 | 2.6 | 2.4 | 2.5 | ||
| MaskFusion | 5 deg 5 cm | 15.5 | 32.3 | 11.7 | 8.8 | 73.9 | 16.4 | 26.5 | |
| IoU25 | 51.4 | 71.4 | 60.8 | 49.7 | 99.9 | 56.2 | 64.9 | ||
| R_err | 36.7 | 12.3 | 43.0 | 34.9 | 3.4 | 40.6 | 28.5 | ||
| T_err | 11.3 | 5.3 | 11.1 | 9.3 | 3.5 | 9.2 | 8.3 | ||
| BundleTrack (this paper) | 5 deg 5 cm | 86.5 | 99.6 | 85.8 | 99.2 | 99.9 | 53.6 | 87.4 | |
| IoU25 | 100.0 | 99.9 | 99.9 | 100.0 | 99.9 | 99.9 | 99.9 | ||
| R_err | 1.6 | 1.7 | 3.0 | 1.5 | 1.5 | 5.2 | 2.4 | ||
| T_err | 2.3 | 2.1 | 2.1 | 2.1 | 2.2 | 2.2 | 2.1 |
R_err is mean rotation error in degrees and T_err is mean translation error in centimetres. Both are averaged only over estimates with IoU25 above 25 percent §V.B.
Table II. YCBInEOAT results tab. II, AUC of ADD and ADD-S, as printed. The 6-PACK row has values only for bleach cleanser, tomato soup can and mustard bottle, which the paper evaluated with released category networks. The paper states that cracker box and sugar box could not be evaluated for 6-PACK §V.C.
| Assumption | Method | Cracker box ADD | ADD-S | Bleach ADD | ADD-S | Sugar box ADD | ADD-S | Tomato ADD | ADD-S | Mustard ADD | ADD-S | All ADD | ADD-S |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Instance 3D model | RGF | 34.78 | 55.44 | 29.40 | 45.03 | 15.82 | 16.87 | 15.13 | 26.44 | 56.49 | 60.17 | 29.98 | 39.90 |
| dbot PF | 79.00 | 88.13 | 61.47 | 68.96 | 86.78 | 92.75 | 63.71 | 93.17 | 91.31 | 95.31 | 78.28 | 89.18 | |
| se(3)-TrackNet | 90.76 | 94.06 | 89.58 | 94.44 | 92.43 | 94.80 | 93.40 | 96.95 | 97.00 | 97.92 | 92.66 | 95.53 | |
| Category-level 3D model | 6-PACK | - | - | 4.18 | 18.00 | - | - | 12.82 | 60.32 | 34.49 | 80.76 | - | - |
| No model | MaskFusion | 79.74 | 88.28 | 29.83 | 43.31 | 36.18 | 45.62 | 5.65 | 6.45 | 11.55 | 13.11 | 35.07 | 41.88 |
| TEASER++ (on our segmented point cloud and correspondences) | 63.24 | 81.35 | 61.83 | 82.45 | 51.91 | 81.42 | 41.36 | 71.61 | 71.92 | 88.53 | 57.91 | 81.17 | |
| BundleTrack (this paper) | 85.07 | 89.41 | 89.34 | 94.72 | 85.56 | 90.22 | 86.00 | 95.13 | 92.26 | 95.35 | 87.34 | 92.53 |
Summary of the headline numbers.
- NOCS, overall 5 deg / 5 cm: BundleTrack 87.4 against 33.3 for category-level 6-PACK and 17.0 for category-level NOCS tab. I. The paper states the prior best rose from 33.3 to 87.4 §I.
- NOCS, 5 deg / 5 cm: BundleTrack is highest in every category tab. I. On IoU25 it ties TEASER++ at 99.9 overall tab. I.
- YCB: BundleTrack's All ADD 87.34 and ADD-S 92.53 are below se(3)-TrackNet (92.66 and 95.53), which uses instance CAD models. The paper describes the result as comparable to these methods §V.C tab. II.
- Run time: about 10 Hz for the whole framework including video segmentation. The paper gives 4 Hz for 6-PACK, 11 Hz for TEASER++ and 17 Hz for MaskFusion on the same machine §V.D.
- Drift: the paper says keeping keyframes as separate graph nodes, rather than fusing them into a global model, reduces long-term drift §I, contribution 2. Fig. 5(d) plots rotation and translation error against timestamp, compared with 6-PACK, MaskFusion and TEASER++ on NOCS fig. 5.
- Failure cases: the paper shows intermediate imprecise estimates during severe self-occlusion of the mug handle, and a segmentation that drops part of the bowl near the end of a video, which causes a small translation offset fig. 6.
Paper vs code
Official repository: https://github.com/wenbowen123/BundleTrack, linked from
the abstract, owned by the first author. Commit 3d0544a (shallow clone). The
code is C++ with CUDA plus Python scripts. LF-Net runs as a separate
TensorFlow server reached over ZeroMQ FeatureManager.cpp.
Video object segmentation is a separate Python project, and the tracker reads
precomputed masks.
- Coarse pose. Paper: the coarse pose is the best 3-pair sample, chained as T_{t-1} T_t^{t-1} §IV.B. Code: the coarse pose is a least-squares rigid fit over all RANSAC inliers, and the offset is applied on the left, offset times the previous pose FeatureManager.cpp Bundler.cpp. The product order differs from the paper, so check conventions before reusing either form.
- RANSAC inlier test. Paper: inliers need a point distance below delta and
a normal angle below alpha §IV.B. Code: the GPU kernel tests the point
distance only cuda_ransac.cu. The normal gate runs
earlier, when correspondences are pruned FeatureManager.cpp.
The config keys
inlier_normal_angle,max_trans_*andmax_rot_*are read but never used in the RANSAC path FeatureManager.cpp FeatureManager.cpp. - RANSAC sampling. A CPU sample list is built but not used; the GPU kernel samples its own triplets FeatureManager.cpp. The GPU random generator is seeded with a fixed value (0) for every thread, so the sampled triplets repeat from run to run cuda_ransac.cu.
- Node count. Paper: up to k selected keyframes plus the current frame, with k at most K = 15 §IV.C §IV.D. Code: at most 15 nodes in total, including the current frame Bundler.cpp config_nocs.yml.
- Which frames enter the optimisation. Code: only the current frame and the keyframes enter, and recent non-keyframes do not Bundler.cpp. The window size only controls how many frames stay in memory Bundler.cpp.
- Keyframe pool. Code never removes keyframes, so the pool grows with the video Bundler.cpp. The paper's bounded K applies only to the graph, not to the pool.
- Unused key.
min_intervalis read but never used Bundler.cpp. - Keyframe threshold. Paper writes alpha as arccos(10 deg) §IV.E. Code compares the geodesic angle with 10 degrees directly Bundler.cpp Bundler.cpp.
- Inlier distance on YCB. Paper: 5 mm in all experiments §IV.B. The YCB config uses 10 mm config_ycbineoat.yml.
- Dense term. Code downsamples by a factor of 4 for the dense term config_nocs.yml LossGPU.cu. The colour term weight is zero, so only depth is used SBA.cpp. This matches the paper's depth-only E_g.
- Gating of the optimisation. Code skips the optimisation when the new frame has too few feature edges, and marks it NO_BA Bundler.cpp. The paper does not describe this.
- Non-neighbour matching. Code adds a map-point propagation step for pairs that are not consecutive frames, before RANSAC FeatureManager.cpp FeatureManager.cpp. The paper describes one data association step for all pairs §IV.B.
- Failure handling. Code marks a frame FAIL when a consecutive pair has fewer than 5 matches, and then sets a re-initialisation flag FeatureManager.cpp Bundler.cpp. The paper's evaluation allows no re-initialisation §V.B, which the released code does not enforce on its own.
- Abort on large residual. For consecutive frames, if the mean residual of
the coarse fit exceeds 1 mm, the code prints an error and calls
std::abort()FeatureManager.cpp. - Possible compile error. In the same function, a bare
return;appears inside a function declared to return a 4x4 matrix FeatureManager.cpp. I did not build the repository, so I cannot say whether this compiles. - Masks. The paper runs the segmentation network in the loop and reports 10 Hz including it §V.D. The tracker reads PNG masks from disk Frame.cpp. Running time without the segmentation step is not reported.
- Mask post-processing. For NOCS, the code keeps only the largest connected
component, replaces it by its convex hull, and dilates by a 5 x 5 rectangle
Frame.cpp. The paper does not mention this. The config
key
seg_dilation_iteris never read config_nocs.yml. - Depth preprocessing. The code erodes the depth map and applies two
bilateral passes Frame.cpp. The config key
outlier_removalis not read in the source. - Rotated crop. The code always passes a rotation of zero, so the rotated-crop branch never runs in this release Bundler.cpp FeatureManager.cpp.
- Initial pose noise. Paper: a uniform translation within a 4 cm range §V.B. Code: each axis is sampled uniformly in plus or minus 2 cm, and the default call adds no rotation noise eval_nocs.py eval_nocs.py.
- Frame dropping. Paper: 450 of 3,200 frames are dropped §V.B. I searched the released scripts for frame dropping and found none.
- YCB AUC threshold. The paper does not state the integration limit. The evaluation code integrates up to 0.1 eval_ycbineoat.py eval_ycbineoat.py.
- Iteration counts. The paper does not state the Gauss-Newton and PCG counts. The released config uses 7 and 5 config_nocs.yml, which the code reads as outer and inner loop counts Bundler.cpp.
Licence notes for the released code. The repository root is Apache-2.0 LICENSE. Some folders carry stricter terms that this card does not copy from:
src/cuda/has a licence file that states CC BY-NC-SA 4.0 for files derived from BundleFusion LICENSE. The solver and RANSAC files cited above are in this folder. They are cited here for location only; no code is quoted.lf-net-release/is for research and evaluation only, and commercial use needs written approval from the LF-Net authors' institutions LICENSE. The keypoint network the pipeline depends on falls under this terms.
Anyone implementing this method for commercial use should not reuse those folders and should check which keypoint network they use.
Pitfalls
- Huber scale. The Huber delta is a length in metres in the released config, and the code applies it to the squared residual config_nocs.yml SolverBundlingEquationsLie.h. Pick the threshold in the same unit system as your point clouds.
- Iteration counts. The paper gives no counts. The released config uses 7 outer and 5 inner iterations config_nocs.yml. Check convergence on your own data before fixing these numbers.
- Keyframe pool grows. The code keeps every keyframe, so memory and the greedy selection cost grow with video length. The paper's complexity note refers to the pool size N §IV.C.
- Mask quality drives the result. The paper attributes its shown failure cases to self-occlusion and to noisy segmentation fig. 6.
- Causal output. Past poses are not revised in the output stream §IV.D. Anything downstream must accept a pose that later graphs would have corrected.
- Speed depends on GPU work. The reported 10 Hz relies on a CUDA implementation of matching and optimisation, and on a TensorFlow keypoint server reached over sockets §V.D FeatureManager.cpp.
- No re-initialisation in the experiments. The paper reports none in its benchmarks §V.B §V.C. The released code sets a re-initialisation flag on failure rather than recovering by itself Bundler.cpp, so plan a reset path for real deployments.
- Licence. See the licence notes in Paper vs code. The paper itself is CC BY 4.0, but parts of the official code are not free for commercial use.
Lineage
- Builds on, named but without a card here: LF-Net for keypoints and descriptors §IV.B; transductive video object segmentation §IV.A; BundleFusion-style pose-graph energies, to which the paper refers for the feature and geometric residuals §IV.D; RANSAC §IV.B.
- Compared against: NOCS, KeypointNet and 6-PACK (category-level) tab. I; ICP, MaskFusion and TEASER++ (no model) tab. I tab. II; RGF, dbot PF and se(3)-TrackNet (instance CAD models) tab. II.
- Superseded by: not stated.
Source and license
Adapted from “BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models” by Bowen Wen, Kostas Bekris (arXiv:2108.00516), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/wenbowen123/BundleTrack at commit 3d0544a9fa2b (Apache-2.0). Any quoted code is shown under that license.