NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping

In plain words

A robot that works in the same rooms for weeks has to know where it is while keeping a map that stays correct after people move things around. Most mapping systems describe the scene with low-level points and lines. Those carry no notion of "this is a mug", so a changed scene can produce false matches between old and new observations.

This paper gives each object a short learned code, built from a partial view of the object. The code is designed so that when the object is turned, moved or made bigger, the code changes in the same way. Because of this, the relative position of two views of the same object can be read directly from their codes, without repeatedly rendering the object and adjusting it until it fits.

The authors plug these codes into a SLAM pipeline, either on their own or alongside an existing system. They use them to match objects across frames, to add position constraints to the camera trajectory, and to notice when an object has been moved or removed. They test on simulated and real sequences in which objects are added, removed or relocated. The reported results are better camera localization and cleaner object maps than the baselines they compare against.

TL;DR

NeuSE is a latent code per object that is SE(3)-equivariant and scale-equivariant, so relative poses between two views are a closed-form (Horn) fit between codes, not an optimization over renders. The codes do data association, supply camera pose constraints for a pose graph, and drive a change-aware object map. Results are author-reported on simulated and real sequences; trained only in simulation on five object categories §1 §5.1.

Problem

Traditional SLAM describes the scene with unordered low-level primitives (points, lines, planes). Without semantics these are hard to match to object-level changes, so scene changes cause false correspondences §1.

Object-based SLAM supplies object-level constraints, but earlier approaches either need a model library, a fixed geometric primitive (ellipsoids, cuboids, dual quadrics) that loses detail, or neural shape priors that must be refined iteratively by rendering to obtain the camera-object constraint §1 §2. Neural scene or object representations are not directly interpretable, so obtaining a pose needs an expensive render-and-optimize step that is sensitive to initialization §1.

The paper targets long-term consistency: objects are added, removed or moved between visits, and the map and trajectory must stay consistent §1 §4. It assumes objects are segmented (Mask R-CNN in the experiments) and that depth is available §5.1 app. A2.

Method

Latent embedding and equivariance

Each partial object point cloud P (n × 3) is mapped by an encoder f_θ to a latent point cloud z (k × 3), and a decoder Φ maps a query point x and the latent z to an occupancy value in [0, 1] eq. 3 §3.1 §3.2. The paper uses k = 512 §5.1 app. A1. The full shape is recovered by sampling query points in a box around the partial cloud and keeping those whose occupancy exceeds v0 = 0.5 §3.2 §5.1.

The latent code is required to transform with the object:

\mathbf{T}\,\mathbf{z} = f(\mathbf{T}\,\mathbf{P}) eq. 1

s\,\mathbf{z} = f(s\,\mathbf{P}) eq. 2

Equation 1 holds for any SE(3) transform T, equation 2 for any uniform scale s §3.1 fig. 3.

Rotation equivariance comes from an encoder built on vector neurons (a rotation-equivariant encoder) §3.2 app. A1:

f_\theta(\mathbf{R}\,\mathbf{P}) = \mathbf{R}\,\mathbf{z}, \quad \mathbf{R} \in SO(3) eq. 4

Scale equivariance comes from normalizing the cross-product encoding by the point cloud's norm, so the following vector-neuron layers are scale-equivariant by construction §3.2. The exact normalization is not written out; see [Pitfalls].

Translation is handled by centering. The encoder runs on the zero-centred cloud P − P̄, where P̄ is the partial center, and the partial center is added back to the result. The decoder is then fed the query shifted by the latent center z̄ and the centred latent z′ §3.2:

\mathbf{z}_0 = f_\theta(\mathbf{P} - \bar{\mathbf{P}}), \quad \mathbf{z} = \bar{\mathbf{P}} + \mathbf{z}_0, \quad \mathbf{z}' = \mathbf{z} - \bar{\mathbf{z}} eq. 5

\Phi(\mathbf{x}, \mathbf{z}) = \Phi(\mathbf{x} - \bar{\mathbf{z}}, \mathbf{z}') eq. 5

Here z̄ is the center of z eq. 5 §3.2. The paper states that translation equivariance is learned, not built in: the center of z0 is trained to learn the offset between P̄ and the true object center §3.3.

Relative pose from latent codes

For two views of the same object, with latent clouds z1 and z2 that share correspondences, the relative SE(3) transform is the closed-form solution of Horn's method eq. 6 §3.3:

(\mathbf{R}, \mathbf{t}) = \Psi(\mathbf{z}_1, \mathbf{z}_2) eq. 6

The target is that this transform is close to T2⁻¹T1 §3.3. When several covisible objects are used together, their latent codes are concatenated in each frame and Horn's method runs on the stacked codes §3.4 §4.2.

Pose ambiguity: two training objectives

Symmetric objects (cylinders, boxes) admit several valid poses, so the training target depends on the shape class §3.4.

• Unambiguous objects (e.g. mugs with a handle). The predicted rotation and translation are pushed toward ground truth eq. 7:

L^{uab}_{transform} = \lVert \hat{\mathbf{R}}\mathbf{R}^{T} - \mathbf{I}_{3\times3} \rVert_F + \lVert \hat{\mathbf{t}} - \mathbf{t} \rVert_2 eq. 7

• Ambiguous objects (e.g. upright bottles). The model is not asked for a single correct transform. The Chamfer distance between the two full object clouds, after alignment with the predicted transform, must be small eq. 8:

L_{amb} = CD(\mathbf{T}_{1,2}\mathbf{P}_{o,1}, \mathbf{P}_{o,2}) eq. 8

CD(\mathbf{P}_1, \mathbf{P}_2) = \frac{1}{|\mathbf{P}_1|} \sum_{\mathbf{x} \in \mathbf{P}_1} \min_{\mathbf{y} \in \mathbf{P}_2} \lVert \mathbf{x} - \mathbf{y} \rVert_2 + \frac{1}{|\mathbf{P}_2|} \sum_{\mathbf{y} \in \mathbf{P}_2} \min_{\mathbf{x} \in \mathbf{P}_1} \lVert \mathbf{x} - \mathbf{y} \rVert_2 eq. 8

The full object clouds are available because training runs in simulation §3.4 §3.7. Extra samples are added by fixing the second view and generating N transforms T1i that rotate the first view around the object's symmetry axis (the observation looks the same) §3.4 fig. 4:

L^{amb}_{transform} = \sum_{i=0}^{N} CD(\mathbf{T}_{1,2}\mathbf{P}_{1i}, \mathbf{P}_{20}) eq. 9

For cylindrical bottles the paper uses N = 180 with angles over [0°, 360°]. For rectangular objects it applies the 180° transform about the symmetry axis §3.4.

Ambiguity can also be resolved across objects: intersecting the candidate pose sets of several covisible ambiguous objects gives an exact transform, and unambiguous objects refine it §3.4 fig. 4.

Shape descriptor and shape consistency

The rotation-invariant part of z0 is the shape descriptor: each of its k rows is taken as a norm, s_i = ‖(z0)_i‖₂, giving a k-vector s §3.5.

Across viewing angles, descriptors of the same object should match and those of different objects should not. The authors use a batch-hard triplet loss on cosine similarity D eq. 10 §3.5. Each batch holds N objects with M partial views each, and within it the most dissimilar same-object pair and the most similar other-object pair are used for each anchor §3.5 §5.1:

L_{b\_shape} = \frac{1}{|B|} \sum_{i=1}^{N} \sum_{j=1}^{M} \left( -\min_{k \in [1,M]} D(o_{ij}, o_{ik}) + \max_{m \neq i} D(o_{ij}, o_{mn}) \right) eq. 10

Here o_ij is the j-th observation of object i in the batch §3.5.

Scale

Since z0 is scale-equivariant, its full norm n = ‖z0‖₂ scales with the object §3.6. Norms of different shapes differ even at equal size, so the norm is trained to match the length of the diagonal d of the minimal bounding box around the partial cloud eq. 11 §3.6:

L_{scale} = \lVert n_i - d_i \rVert_2 eq. 11

Training objective and data

The total loss is the sum of occupancy cross-entropy, the transform loss, the shape loss and the scale loss, all with weight 1 eq. 12 app. A1. The occupancy term is a standard cross-entropy on query points with their true occupancy §3.7.1:

L = L_{occ} + L_{transform} + L_{b\_shape} + L_{scale} eq. 12

Unambiguous and ambiguous categories are trained separately with their respective transform loss, and share the other three terms §3.7.1.

Training data are rendered in PyBullet with ShapeNet models §5.1. Each scene places a randomly posed main object on a table with 2–4 unambiguous or 1–2 ambiguous objects of trained categories. Camera poses are sampled over a hollow cubical volume around the table §3.7.2 §5.1. Separate models are trained for mug, bowl, bottle, trash bin and can app. A1.

Object-based SLAM

The SLAM system is a pose graph whose vertices are keyframe camera poses. An edge exists between two vertices when an inter-frame transform is measured from NeuSE or another source. The residual is e_ij = log(Z_ij T̂_j⁻¹ T̂_i)^∨, where Z_ij is the measured transform §4.1 fig. 6. Per keyframe the system keeps the latest pose, the NeuSE codes of the objects it sees, and per object its visibility, change status, last partial cloud and latest shape descriptor §4.1. Localization uses codes in the local camera frame. Mapping uses world-frame codes, initialized on first sight and updated by averaging back-projected codes using keyframe poses §4.1.

Data association runs in this order §4.2 fig. 5:

1. Semantic label: only candidates of the same category are kept §4.2.
2. Scale: candidates within |δ_size| of the query's latent norm are kept §4.2.
3. Shape: candidates whose descriptor cosine similarity exceeds δ_shape are kept as candidates. If none pass, a new object is created §4.2.
4. Spatial proximity: the current partial center is projected into the world frame with a Horn transform from the concatenated codes. The nearest candidate below δ_prox is matched; otherwise the observation is unmatched and goes to change detection §4.2.

Unambiguous objects are associated first; ambiguous objects use the transform from associated unambiguous objects, or an exhaustive search over covisible combinations of candidates §4.2. Covisible objects therefore fall into three groups: matched (used for pose constraints), unmatched (checked for change), and new §4.2.

Pose constraints §4.3:

• Short range: each new keyframe is paired with its K−1 predecessors that share objects, and the Horn transforms become edges of a K-keyframe sliding window, with K = 10 §4.3 §5.1 fig. 6.
• Long range: when an object returns after a gap, the current frame is linked to the first frame of each earlier observation period. The global optimization starts from the sliding-window estimate §4.3.
• Keyframes are added when a new object appears or when the frame is at least 0.04 m from the previous keyframe along accumulated odometry, and after change detection §4.3.

Change detection §4.4 fig. 7 runs only on unmatched objects that match in shape but are spatially apart. It compares relative layout, not absolute position, against the matched objects used as anchors. Each object gets edges to anchors, E = {c_ui − c_mj}, with centers taken from the decoded full reconstruction and back-projected with the latest pose §4.4. Two edges are compared when they connect vertices of similar shape. An object is unchanged if at least one edge pair agrees within δ_e eq. 13 §4.4:

\sum_{j} \mathbb{1}\left( \lvert e_{ij} - e_{i'j'} \rvert \le \delta_e \right) = \begin{cases} 0 & \text{changed} \\ \ge 1 & \text{unchanged} \end{cases} eq. 13

Changed objects are updated in the library. An object is removed when it was never shape-matched during a period with global loop closure §4.4.

Map maintenance app. A2: a new object is instantiated only after it is recognized as new three consecutive times, and only when it is within 2.5 m of the camera or its segmentation mask exceeds 4,000 (unit not stated). If a small object's box overlaps a large object's box by at least 0.95, the small one is treated as a partial copy and removed. If a changed object's box intersects an older object by at least 20% of the changed object's box volume, the older one is removed app. A2.

Pipeline (block diagram)

Six modules work in sequence on partial point clouds built from depth and masks, updating a database of objects and frames. Outputs are camera poses and a full-shape object map §1 fig. 2.

Pseudocode

The blocks below paraphrase the steps above. They are not the authors' code (none is public; see [Paper vs code]).

# Encode one partial object point cloud  [§3.2] [eq. 3] [eq. 5]
# Input:  points (n, 3), partial cloud in the camera frame
# Output: latent (k, 3), shape descriptor (k,), scale scalar
def encode_partial_object(points):
    # Partial center is the initial translation estimate  [§3.2]
    partial_center = mean(points, axis=0)              # (3,)
    # Encoder runs on the zero-centred cloud  [§3.2] [eq. 5]
    latent_zero = encoder(points - partial_center)     # (k, 3)
    # Add the partial center back  [eq. 5]
    latent = partial_center + latent_zero              # (k, 3)
    # Centred latent used by the decoder  [eq. 5]
    latent_centred = latent - mean(latent, axis=0)     # (k, 3)
    # Shape descriptor: norms of the rotation-invariant part  [§3.5]
    shape = norm(latent_zero, axis=1)                  # (k,)
    # Scale: norm of the full latent code  [§3.6]
    scale = norm(latent_zero)                          # scalar
    return latent, latent_centred, shape, scale

# Relative SE(3) transform between matched objects  [§3.3] [eq. 6]
# Input:  latents_a, latents_b: stacked codes (m*k, 3), same order
# Output: rotation (3, 3), translation (3,)
def relative_pose(latents_a, latents_b):
    # Closed-form Horn solution; no rendering or iterative refinement  [§3.3]
    rotation, translation = horn(latents_a, latents_b)
    return rotation, translation

# Pose loss for one pair of views of one object  [§3.4] [eq. 7] [eq. 9]
# Input: predicted (R, t), ground truth (R_gt, t_gt) for unambiguous objects,
#        or sampled symmetric views for ambiguous ones
def pose_loss(rotation, translation, gt_rotation, gt_translation):
    # Unambiguous: distance to ground-truth transform  [eq. 7]
    frobenius = norm(rotation @ gt_rotation.T - identity(3), "fro")
    return frobenius + norm(translation - gt_translation)

# Batch-hard triplet loss on shape descriptors  [§3.5] [eq. 10]
# Input: descriptors (N, M, k): N objects, M views each (N=8, M=15 [§5.1])
def batch_hard_triplet(descriptors):
    loss = 0
    for object_index in range(N):
        for view_index in range(M):
            anchor = descriptors[object_index, view_index]
            # Least similar view of the same object  [§3.5]
            hardest_positive = min(cosine(anchor, descriptors[object_index]))
            # Most similar view of any other object  [§3.5]
            hardest_negative = max(cosine(anchor, other_objects(object_index)))
            loss += hardest_negative - hardest_positive
    return loss / (N * M)                              # 1/|B| in [eq. 10]

# Associate one observation with the object library  [§4.2] [fig. 5]
# Input:  observation (label, latent, shape, scale, center), library
# Output: matched library object, or None (new or unmatched)
def associate(observation, library):
    # Semantic label selects the category  [§4.2]
    candidates = [o for o in library if o.label == observation.label]
    # Scale: latent norms must be close  [§4.2] [§5.1]
    candidates = [o for o in candidates
                  if abs(o.scale - observation.scale) < delta_size]
    # Shape: descriptor cosine similarity above threshold  [§4.2]
    candidates = [o for o in candidates
                  if cosine(o.shape, observation.shape) > delta_shape]
    if not candidates:
        return None                                    # new object  [§4.2]
    # Spatial proximity: nearest candidate centre in the world frame  [§4.2]
    best = min(candidates, key=lambda o: proximity(observation, o))
    if proximity(observation, best) < delta_prox:
        return best                                    # matched  [§4.2]
    return None                                        # unmatched  [§4.2]

Hyperparameters

Encoder and decoder layers tab. A1 tab. A2:

The encoder input layer and the exact form of the scale normalization are not listed in tab. A1 app. A1.

Results

Author-reported throughout. Each experiment is listed with its setup. Baselines are named per table.

Equivariance under partial observations §6

Setup: 500 synthetic test samples per experiment (100 per category, five categories), PyBullet, ShapeNet instances, same camera view before and after each transform §6. Each equivariance error is measured between z and z′ after the transform §6.

• Rotation §6.1 fig. 8a: normalized RMSE with mean 0.83%, varying within (0.4%, 1.2%) over the rotation range.
• Translation §6.2 fig. 8b: RMSE mostly below 0.01 m across categories and translation magnitudes up to 0.4 m.
• Scale §6.3 fig. 8c: ratio error averages around 0.011, mostly between −0.01 and 0.03, for scales from about 1 to 6.75. For scales of 3 to 6.75, the error stays mostly below 0.05.

Synthetic localization §7.1

Setup: PyBullet tabletop scenes on 10 m × 15 m. Mug + Bottle set: 50 unseen instances on 10 tables. Multi set: about 75 unseen instances across five categories on 12 tables. Planar (upright) and non-planar (half the objects lying down) layouts, with 9 to 10 changes per sequence §7.1. Baselines: Mug-only, and the SE(3)-only variant from Fu et al. (2023), named in the paper as [fu2023neural] §7.1.

tab. 1 RMSE of ATE and translational RPE (m, m/f). Best per row in bold.

The paper reports lower error on the non-planar layout and credits lying-down mugs for resolving handle ambiguity §7.1.

tab. 2 Data association accuracy, point-cloud overlap (PC-based) vs NeuSE (Ours).

tab. 3 ATE and RPE with noisy external odometry; Raw Odometry, point-cloud association (PC-based), and NeuSE association (Ours) §7.1.

tab. 4 Depth-noise sensitivity on the mug-and-bottle occlusion sequence §7.1.

The paper reports that at 1 cm noise NeuSE fails to recover a valid trajectory §7.1 fig. 14. The noise model scales the standard deviation with depth relative to the frame's largest object depth eq. 14 §7.1:

d_{noisy}(u,v) = d(u,v) + \eta(u,v), \quad \eta(u,v) \sim N(0, \sigma(u,v)), \quad \sigma(u,v) = \sigma_{max} \frac{d(u,v)}{d_{max}} eq. 14

Real-world localization §7.2

Setup: RealSense D515 on a Clearpath Jackal with Vicon ground truth. Controlled environment: 49 instances, five tables, 7 m × 5 m; 4-Round and Triple-Infinity loops, with 14–16 objects added, removed or relocated to form two arrangements §7.2. Hand-held lab-office sequences (Indoor and Freeform) with 45 instances and 16 tables, and a corridor blackout where no objects are visible. Vicon covers only the indoor office part, 24% of the trajectory §7.2. Baselines: ORB-SLAM3 in non-stop (ORB3-NS) and piecewise (ORB3-PW) mode §7.2, CubeSLAM, EM-Fusion and VOOM §7.2. Raw Odometry comes from Open3D RGB-D odometry §7.2.

tab. 5 RMSE of ATE (m) on the controlled sequences. Best per trajectory in bold.

Notable author-reported comparisons: ORB3-PW + Ours reduces the Triple-Infinity RMSE from 0.275 m to 0.132 m, a 52% reduction §7.2. ORB3-NS + Ours reduces the 4-Round full RMSE from 0.128 m to 0.093 m, a 27.3% reduction §7.2.

tab. 6 RMSE of ATE (m) on hand-held sequences. Best per trajectory in bold.

CubeSLAM and EM-Fusion are reported to fail to complete both hand-held sequences, and VOOM completes only Indoor with heavy drift §7.2.

Change-aware mapping §8

Setup: PMT (panoptic multi-TSDF mapping) is the change-detection baseline, fed with the lowest-ATE trajectory estimates §8.2.

tab. 7 Change detection: true positives (TP), false positives (FP), false negatives (FN), precision (Pr), recall (Re) §8.2.

Speed §7

Author-reported on the evaluation laptop: NeuSE network inference takes 6 ms per object, the inter-frame constraint 1 ms, and one-time rendering for the map 30 ms per object with 20,000 query points. Data association with the front end runs about 20 fps. The full localization system runs about 4 fps in the crowded controlled setting, with sequential processing and no software optimization §7. For the point-cloud baseline, data association drops to 1–3 Hz when revisiting 75 objects §7.1.

Paper vs code

No official code.

No link to code in the paper or its arXiv metadata. The only link is a YouTube supplementary video §1 [raw metadata]. Searched: (1) the paper and arXiv comments for a repo link (none); (2) no project page is linked; (3) GitHub repository search for "neuse" (100 hits, none by the authors), "neuse slam" (one organization, neuse-slam, with two repositories and no public members, README, or author names, and not linked from the paper, so not used) and "neuse equivariant" (no hits). Nothing was cloned.

Pitfalls

• Equivariance is learned, not exact. Equations 1, 2 and 4 are design goals; the paper measures residual errors of about 0.8% for rotation and below 0.01 m for translation §6.1 §6.2 fig. 8.
• Norms are underspecified. The scale norm is written as ‖z0‖₂ of a k × 3 matrix, and the shape descriptor as per-row norms. The paper does not say whether the first is a Frobenius norm over all entries, and it gives no formula for the "cross-product encoding normalized by the point cloud norm" §3.2 §3.6 §3.5.
• Symmetry sampling is ambiguous. §3.4 says "N random transforms" but later "draw transforms from [0°, 360°]". It is not stated whether the draws are uniform, random or a fixed grid. It is also not stated whether the L_amb definition eq. 8 is a separate training term or only the one used inside eq. 9 eq. 12.
• Batch-hard loss has no margin and an index typo. Eq. 10 has no margin term, and its negative term uses an index (o_mn) that is not defined in the sum. The text is the reliable source: positives are the least similar same-object views, and negatives are the most similar other-object views §3.5. The 1/|B| normalization is not defined: it is not stated whether |B| counts objects, views or both eq. 10.
• Units missing on some thresholds. δ_size is a "scale difference" with no unit given, and the mask-size threshold of 4,000 has no unit §4.2 §5.1 app. A2.
• Unambiguous loss is unsquared and unweighted eq. 7 app. A1. All four loss terms have weight 1, so their relative magnitudes are whatever the raw norms give.
• Association thresholds are tuned per setting. Values differ between synthetic, controlled and hand-held data §5.1. The paper gives no rule for choosing them on a new dataset.
• Averaging of world-frame codes is underspecified. Codes are updated by averaging back-projected codes §4.1. The paper does not say whether the average is taken before or after aligning codes, which affects equivariance.
• Box-overlap thresholds are untested. The 0.95 and 20% rules app. A2 have no sensitivity study in the paper.
• Depth noise depends on the frame's farthest object. σ scales with depth relative to d_max, the maximum object depth in the current frame eq. 14. Noise on a near object therefore depends on whether a far object is also in view.
• Category-level only. NeuSE needs a separate model per category app. A1 and the paper names this as its main limitation for open-set use §9.
• Co-linear symmetry axes are not handled. The paper notes the rare case of co-linear symmetry axes across all visible objects is not covered §3.4.
• Hand-held sequences add untrained object sizes on purpose. Two small white jars were added to the trash-bin category to test size generalization §7.2. Their results are not comparable to the controlled sequences.
• Depends on segmentation and semantic labels. A wrong semantic label breaks association. The paper proposes confidence thresholding as a fix but does not implement it §9.
• Training data cost. Ambiguity sampling and training need full object clouds and occupancy from simulation §3.4 §3.7. Real data is not used for training.
• The 65% figure is not tied to a table. The text says NeuSE gives a 65% ATE improvement on the lower half of one trajectory §7.1. Table 3 gives the full-trajectory reduction against the point-cloud baseline as about 58% (planar) and 66% (non-planar), so use the table values, not the text figure.

Lineage

• Builds on: Vector Neurons for the rotation-equivariant encoder §3.2 app. A1; Neural Descriptor Fields for the encoder and decoder layout app. A1; the batch-hard shape loss follows the authors' earlier work §3.5; Horn's method for the closed-form transform §3.3; Mask R-CNN §5.1; GTSAM with iSAM2 §5.1.
• Earlier version: the SE(3)-only variant (NeuSE without scale equivariance) is reported from Fu et al. (2023) [fu2023neural] §7.1 §7.2.
• Compared against: ORB-SLAM3 §7.2, CubeSLAM, EM-Fusion and VOOM §7.2; panoptic multi-TSDF mapping (PMT) for change detection §8.2.
• Superseded by: not stated in the paper.