Pose (rigid position and orientation of an object)

Overview

A rigid object's pose has six degrees of freedom: three for orientation and three for position. Many pipelines estimate it in two steps. A network first finds 2-D image points that match fixed 3-D keypoints on the object model, and a geometric solver then fits the pose to those matches 1911.08324 §1. The first step is trained on a 2-D error, and the second is a random-sampling solver, so the whole chain is not trained end to end 1911.08324 §1.

The idea shared by the variant below is to keep the 2-D matching network and replace the solver with a small learned function. That function takes every candidate for every keypoint and outputs a rotation, as a quaternion, and a translation 1911.08324 §3.1. It has to respect two facts. Candidates inside one keypoint's group have no order, while the groups do follow the fixed keypoint order 1911.08324 §3.1.1.

The word "pose" also appears in other senses on other pages: the camera pose that a PnP solver recovers (Perspective-n-point), the camera parameters fed to a generator as positional codes (Camera pose encoding), the human skeleton in Multi-person pose estimation, and the graph of poses in Pose-graph optimization. This page covers only the rigid object pose sense.

Variants

Only one paper in the wiki covers this concept, so the variants below are the ones it compares against and the one it proposes.

Learned replacement for RANSAC-PnP

Each candidate is embedded by a small MLP whose weights are shared across all keypoints. For each keypoint, the features are max-pooled over its candidates. The keypoint results are concatenated in their fixed order, and three fully connected layers output a quaternion and a translation 1911.08324 §3.1.2 1911.08324 eq. 3. Training uses a loss on the 3-D keypoint distance under the predicted pose and under the true pose 1911.08324 §3.2 1911.08324 eq. 7. The network sits after a correspondence network, either the SegDriven or the PVNet encoder-decoder, with the 3-D keypoints taken as the eight corners of one shared cube and 200 candidates per keypoint 1911.08324 §3.2. Card: Single-Stage 6D Object Pose Estimation.

Reported gains on Occluded-LINEMOD, ADD-0.1d averages: SegDriven rises from 27.0 to 32.3 and PVNet from 40.8 to 43.3 1911.08324 tab. 1. On YCB-Video the same measure is 53.9 for this method against 39.0 for SegDriven 1911.08324 tab. 4. All these numbers are the authors' own.

Geometric PnP with RANSAC (EPnP, P3P)

The classical baseline. On synthetic data with growing 2-D noise, the RANSAC variants do about as well as each other and beat the learned method at very low noise 1911.08324 §4.1 1911.08324 fig. 6. The learned method becomes clearly more accurate as noise grows 1911.08324 fig. 6.

Voting-based PnP (PVNet)

PVNet's own solver. The paper reports that voting wins at low noise, and the learned replacement is more robust at high noise 1911.08324 fig. 8.

When to use which

Implementation sketch

The shared core of the learned variant, with notes on what the others change. Shapes are per batch: n keypoints, m candidates per keypoint.

# Input: candidate points, grouped by keypoint, shape (n, m, 2)
# Keypoint identity is the group index, not an input feature [arxiv:1911.08324 §3.2]

def pose_from_groups(groups):
    # Shared MLP on every candidate, weights tied across keypoints
    # [arxiv:1911.08324 §3.1.2]
    feats = shared_mlp(groups)                # (n, m, D)
    # Max over the candidates of each keypoint: order inside a group is ignored
    # [arxiv:1911.08324 eq. 3]
    per_kp = feats.max(axis=1)                # (n, D)
    # Concatenate in keypoint order, so position in the vector encodes identity
    # [arxiv:1911.08324 §3.1.2]
    vec = per_kp.reshape(n * D)
    # Three fully connected layers give 4 quaternion values and 3 translation values
    # [arxiv:1911.08324 §3.1.2]
    q, t = fc_head(vec)
    return q, t

# Training signal: mean distance of the 3-D keypoints under predicted and true pose
# [arxiv:1911.08324 eq. 7]

A single global max over all candidates would ignore keypoint order. The paper reports that this degrades accuracy markedly 1911.08324 fig. 7.

Contradictions

• Training schedule. The paper reports 300 synthetic epochs with Adam at a learning rate of 1e-3 1911.08324 §4.1. The public code, as recorded in the card's Paper vs code section, uses 200 epochs at 1e-4. Neither side is resolved here.
• Noise. The paper describes its 2-D noise by a variance 1911.08324 §4.1. The public code uses the same range as a standard deviation (see the card). The two readings differ by a square in the noise level.
• Synthetic grid sampling. The paper samples candidate cells inside the predicted mask 1911.08324 §4.1. The public code samples uniformly over the whole image (see the card).
• Loss. The paper's total loss has three terms 1911.08324 eq. 6. The public code computes only the pose term (see the card).
• Real-data results. The numbers in the tables are author-reported 1911.08324 tab. 1 1911.08324 tab. 4. No second source in the wiki reproduces them.

Papers

• Single-Stage 6D Object Pose Estimation: a learned, order-aware network that replaces RANSAC-PnP in two-stage 6-D pose pipelines.