OnePose: One-Shot Object Pose Estimation without CAD Models

In plain words

Knowing where an object sits and how it is turned (its 6D pose) usually needs either its exact 3D model or a neural network trained on that object or on its category. OnePose skips both. You walk a camera around the object once and record a short video. The system builds a sparse 3D point cloud of the object from that video, much as visual localization maps a room from photos.

For a new image, the system finds visual features and matches them directly to the 3D points of that cloud, then solves for the pose with a standard geometric method. Matching directly avoids the slow step of first finding similar reference images.

The new part is the matcher. A graph attention network lets the image features that belong to the same 3D point share information, so each point gets a richer description built from several viewpoints. Attention layers then let the image features and the point features look at each other before they are compared.

TL;DR

OnePose estimates object pose from one RGB video scan, without a CAD model or training for each object §1. Its graph attention network matches 2D image features directly to the 3D points of a sparse SfM model, and PnP with RANSAC gives the pose §3.2. Author-reported: it is on par with HLoc (SuperPoint + SuperGlue) at about a tenth of the runtime, and it beats the instance-level PVNet and the category-level Objectron on the authors' dataset §4.3 tab. 1 tab. 2 tab. 3.

Problem

Instance-level methods need the object's CAD model, which is often unavailable for everyday objects §1. Category-level methods need many annotated samples per category, generalise poorly to new shapes and appearances, and need a network per category, which is costly on mobile AR §1. The localization-based setting removes both needs but is slow with the usual visual localization pipeline: it matches the query to several retrieved database images with a learned matcher, which often takes seconds per frame §1 §3.1. The paper assumes a static object on a flat surface during capture and a known 2D object box at test time §3.1 §4.2.

Method

The pipeline has an offline mapping phase and an online localization phase §3.1.

Mapping. The object is captured as a video with camera poses from ARKit or ARCore. An annotated 3D bounding box defines the object frame §3.1. Structure from Motion gives a sparse point cloud P_j in that frame. Each point keeps the set of matched 2D keypoints and descriptors that form its feature track §3.1. The mapping keeps the camera poses fixed and triangulates the points, without bundle adjustment on the poses §4.2. Only points inside the annotated 3D box are kept, and only 2D features inside the detected 2D box are used at localization §4.2.

Localization. For each query image, 2D keypoints and descriptors are extracted from the box region. The aggregation step below turns each 3D point's feature track into one 3D descriptor §3.2. A stack of attention groups then matches query features to these 3D descriptors. Pose comes from PnP with RANSAC on the matched 2D-3D pairs §3.2.

Aggregation-attention (per 3D point). Each 3D descriptor starts as the average of its 2D descriptors. A graph attention layer then re-weights those 2D descriptors according to their relevance to each query. The layer is defined as §3.2:

\mathrm{Aggr}(\{F_{k'}^{2D}\}, F_j^{3D}) = F_j^{3D} + \sum_{k' \in \mathcal{G}_j} \alpha_{k'} F_{k'}^{2D}

\alpha_{k'} = \mathrm{softmax}_{k' \in \mathcal{G}_j}\big(\mathrm{sim}(W F_{k'}^{2D}, W F_j^{3D})\big)

Here W is a d x d weight matrix and sim is an inner product, so the attention weights are normalised over the 2D descriptors of the feature track only §3.2.

Attention group. An attention group runs the aggregation layer, then a self attention on the query features, a self attention on the 3D descriptors, and a cross attention between the two sets §3.2 fig. 3:

\hat{F}_j^{3D} = \mathrm{Aggr}(\{F_k^{2D}\}, \{F_j^{3D}\}) \tilde{F}_q^{2D} = \mathrm{Self}(\{F_q^{2D}\}, \{F_q^{2D}\}) \tilde{F}_j^{3D} = \mathrm{Self}(\{\hat{F}_j^{3D}\}, \{\hat{F}_j^{3D}\}) \{F'^{2D}_q\}, \{F'^{3D}_j\} = \mathrm{Cross}(\{\tilde{F}_q^{2D}\}, \{\tilde{F}_j^{3D}\})

The network stacks N = 4 attention groups §4.2. Linear attention is used in all attention layers, following LoFTR §4.2. Other details are not in the paper (see Paper vs code).

Matching. A score matrix S(q, j) = <F'_q, F'_j> is computed between query features and 3D descriptors §3.2. Match confidence is the product of two softmaxes, one over the 3D points for each query and one over the queries for each 3D point, following the dual-softmax operator of LoFTR §3.2:

C_{3D}(q, j) = \mathrm{softmax}(S(q, \cdot))_j \cdot \mathrm{softmax}(S(\cdot, j))_q

Pairs above a confidence threshold theta become the 2D-3D matches M_3D §3.2. The threshold value is not stated.

Training. Ground-truth matches come from filtered 2D-3D correspondences in the SfM maps of the training set §3.2. The loss is a focal loss on the confidence values §3.2:

L = -(1 - C'_{3D}(q, j))^{\gamma} \log C'_{3D}(q, j)

with C' = C_3D where the ground truth is a match, and C' = 1 - C_3D otherwise §3.2. Gamma is not stated in the paper.

Pseudocode. The listing below follows the paper's equations. The official code differs in several details (see Paper vs code).

# Build one 3D descriptor per SfM point from its 2D feature track  [§3.2]
def aggregate_3d(desc_3d, desc_leaves, W):
    # desc_3d:     (M, d) starts as the mean of the leaves  [§3.2]
    # desc_leaves: (M, n_leaf, d) sampled or padded to 8  [§4.2]
    scores = inner(desc_leaves @ W, desc_3d @ W)  # (M, n_leaf)
    weights = softmax(scores, axis=leaves)        # over leaves only  [§3.2]
    return desc_3d + sum(weights * desc_leaves)   # residual sum  [§3.2]

# One attention group: aggregation, self, self, cross  [§3.2]
def attention_group(desc_query, desc_3d, desc_leaves):
    # Aggregate the feature tracks into the 3D descriptors  [§3.2]
    desc_3d = aggregate_3d(desc_3d, desc_leaves, W)
    # Self attention on the query features  [§3.2]
    desc_query = self_attention(desc_query, desc_query)
    # Self attention on the 3D descriptors  [§3.2]
    desc_3d = self_attention(desc_3d, desc_3d)
    # Cross attention between query and 3D features  [§3.2]
    desc_query, desc_3d = cross_attention(desc_query, desc_3d)
    return desc_query, desc_3d

# Full matcher and pose solver  [§3.2] [§4.2]
def localize(query_feats, object_model):
    # Stack N = 4 attention groups  [§4.2]
    desc_query, desc_3d = query_feats, object_model.desc_3d
    for group in range(4):
        desc_query, desc_3d = attention_group(
            desc_query, desc_3d, object_model.desc_leaves
        )
    # Dual softmax over the score matrix  [§3.2]
    scores = inner(desc_query, desc_3d)            # (Q, M)
    confidence = softmax(scores, axis=3D) * softmax(scores, axis=query)
    # Keep 2D-3D pairs above theta; theta is not stated  [§3.2]
    matches = pairs_above(confidence, theta)
    # Solve the pose with PnP inside RANSAC  [§3.2]
    return pnp_ransac(matches.pts_2d, matches.pts_3d, intrinsics)

Key equations are listed in the Method prose above with their section citations §3.2; the paper's equations are unnumbered in the LaTeX source.

Online tracking. A feature-based tracker runs on every frame of the test sequence and keeps its own online 3D map and key-frame pool. The pose estimator handles key frames and re-initialises the tracker when it fails §3.2. The paper refers to the supplementary material for tracker details, which is not in the source fetched for this card. The tracker is outside the scope of this card.

Dataset. The authors capture each object with an AR-based interface, draw an adjustable 3D box on the image, and refine the poses by bundle adjustment with COLMAP, using SuperGlue matches restricted to the object box §3.3 fig. 4. The dataset has 450 sequences of 150 objects, about 30 seconds per recording on average, randomly split into training and validation sets §3.3. The dataset is not used by this card.

Hyperparameters

Values marked "paper" come from the paper text. Values marked "code" are from the official repository at code_commit and are not stated in the paper. The paper's supplementary material may hold more, and it is not in the source fetched.

Results

Author-reported. Setup: the authors' OnePose dataset, with the validation split §4.1 §3.3. Runtime on an NVIDIA TITAN RTX GPU §4.3. Baselines are HLoc with SIFT or SuperPoint and nearest neighbour or SuperGlue §4.1 tab. 1, PVNet §4.1 tab. 2, and Objectron, using the released models §4.1 tab. 3.

Tab. 1 (visual localization). Pose accuracy per object size, and runtime in ms tab. 1.

L, M and S are large, medium and small objects tab. 1. The paper calls the method on par with or slightly better than HLoc (SPP + SPG), which takes about ten times as long §4.3.

Tab. 2 (instance-level baseline, PVNet, 5cm-5deg). Scores per object ID tab. 2.

Tab. 3 (category-level baseline, Objectron). Mean 2D projection error (lower is better) and average precision at 15° azimuth and 10° elevation error. Cup and Shoe columns hold Objectron's originally reported category numbers, not evaluated on this dataset tab. 3.

Objectron (S) and (S+C) apply a scale adjustment and a centre alignment to its predictions, which the paper uses to close a gap in the annotations §4.3.

Tab. 4 (ablation, 1cm-1deg). Replacing parts of the method tab. 4 §4.4.

SPP is SuperPoint and NN is nearest neighbour tab. 4. The paper reads the drop in (i) as showing that aggregation-attention matters §4.4.

Paper vs code

Repository: https://github.com/zju3dv/OnePose, commit d0c313a5c36994658e63eac7ca3a63d7e3573d7b, Apache-2.0 LICENSE. The repository is linked from the project page the abstract names [abstract]. The owner is the ZJU 3D vision group, to which the authors belong. Searched: project page, then GitHub owner search for "onepose" inside zju3dv.

The following differ between paper and code. Each is cited to both places.

• Aggregation weights. The paper's softmax runs over the feature track's 2D descriptors only, with a residual sum with the 3D descriptor §3.2. The code uses additive graph attention with LeakyReLU (slope 0.2) and includes the 3D node itself among the softmax entries, with no separate residual on that node GATs.py GATs.py GATs_SuperGlue.py.
• Similarity in aggregation. The paper uses an inner product of W-projected features §3.2. The code uses an additive score, a_3D . Wh_3D + a_2D . Wh_2D passed through LeakyReLU GATs.py.
• Linear attention. The paper says linear attention is used in all attention layers §4.2. In the code only the self and cross layers use it GATs_SuperGlue.py GATs_SuperGlue.py. The aggregation layers use softmax attention GATs.py.
• Dual-softmax temperature. The paper's score has no temperature §3.2. The code divides the scores by 0.07 before the softmaxes GATs_SuperGlue.py train_GATsSPG.yaml.
• Match selection. The paper keeps pairs above theta, giving a permutation matrix §3.2. The code keeps only mutual nearest neighbours whose score is above 0.2 GATs_SuperGlue.py train_GATsSPG.yaml.
• Focal loss. The paper's loss has no alpha or class weights §3.2. The code uses alpha 0.5 and weights of 0.5 on the positive and negative means focal_loss.py train_GATsSPG.yaml.
• Dropout. The code sets dropout 0.6 on the graph attention layers, but the forward pass never applies it GATs_SuperGlue.py GATs.py. The paper does not mention dropout.
• Sinkhorn. The training config sets sinkhorn_iterations: 100 train_GATsSPG.yaml. The GATs matcher never reads it; only the separate SuperGlue matcher uses it superglue.py.
• Keypoint encoding and normalisation. The paper does not describe how positions enter the network. The code encodes 2D keypoints (x, y, score) and 3D points (x, y, z, score) with an MLP into the descriptor dimension, and L2-normalises the final descriptors GATs_SuperGlue.py GATs_SuperGlue.py.
• Padding caps. The training code pads or samples to 1000 2D keypoints and 2000 3D points, and the test config caps 3D points at 2500 train_GATsSPG.yaml test_GATsSPG.yaml. The paper does not state caps.
• Mapping. The paper triangulates with fixed poses and no bundle adjustment §4.2. The repository's pipeline does the same run.py. A bundle-adjustment module exists in the repository global_ba.py, but the code we read never imports it.
• PnP. The paper says RANSAC with PnP §3.2. The code uses OpenCV's solvePnPRansac with EPnP, a 5 px reprojection threshold and 10000 iterations eval_utils.py.

No discrepancies were found for the number of attention groups (N = 4) and the eight features per 3D point train_GATsSPG.yaml GATs_SuperGlue.py.

Pitfalls

• Unnumbered equations. The LaTeX source has no numbered equations, so this card cites them by section §3.2. The table and figure numbers follow source order and should be checked against the PDF before quoting them as printed.
• Identical columns in Tab. 1. For large objects, the 3cm-3deg and 5cm-5deg columns hold identical values for every method tab. 1. This may be a typesetting error, and the card reports the values as printed. The PDF was not available to check it.
• Runtime is not end to end. The 58 ms and 618 ms times cover feature extraction and 2D-3D matching only, excluding 2D detection and PnP §4.3. The paper gives no end-to-end timing.
• Test-time box. Localization assumes the 2D object box is known §4.2. The table runtimes do not include the detector, and the paper names YOLOv5 as an option only in passing §4.2.
• Stated baseline changes. PVNet was trained with box corners as keypoints and auxiliary mask supervision, and its masks needed dense 3D reconstruction §4.1 §4.3. Objectron is evaluated with the authors' released models, trained on the original dataset §4.1.
• Scale and centre alignment. The paper does not say how the scale and centre alignment for Objectron (S) and (S+C) is computed §4.3 tab. 3.
• Training details are missing. The paper gives no learning rate, batch size, epoch count or loss reduction §4.2 §3.2. The code values in the Hyperparameters table are the only ones available. They come from the training config, not from the paper.
• Ablation wording. The paper calls the drop for variant (i) significant §4.4. The gap is 0.022 on large objects and 0.027 on medium objects tab. 4. The paper does not report variance across runs.
• Mapping cost. Each object needs a video scan, an annotated box and an SfM run §3.1 §3.3. The dataset uses bundle adjustment with COLMAP on SuperGlue matches §3.3.
• Failure modes. The paper names textureless objects and large scale changes between the scan and the test sequence as failures §5.
• Tracker not described here. Real-time claims depend on a tracker that the paper describes only in its supplementary material §3.2 §1.

Lineage

• Builds on: visual localization with HLoc, SuperPoint and SuperGlue §1 §3.1; graph attention networks §3.2; the dual-softmax operator and linear attention from LoFTR §3.2 §4.2; the focal loss §3.2.
• Compared against: HLoc with SIFT or SuperPoint, and nearest neighbour or SuperGlue tab. 1; PVNet tab. 2; Objectron tab. 3; traditional 2D-3D nearest neighbour variants tab. 4.
• Superseded by: none recorded in this wiki yet.