wikillm

Paper · 2021-08-18 · cs.CV

SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose regresses a 6-DoF object pose from one RGB crop without PnP/RANSAC. Besides dense 2D-3D correspondences it predicts self-occlusion coordinates on three object planes, tied together by 3D and 2D cross-layer consistency losses. Author-reported: 45.9 ADD(-S) at 0.02d on LINEMOD, 62.3 mean ADD(-S) on Occluded-LINEMOD with real+pbr data.

Pose (rigid position and orientation of an object)Dense correspondencePerspective-n-point (camera pose from 2-D/3-D correspondences)6d-poseself-occlusiondirect-regressiontwo-layer-representation

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many recent pose estimators look only at the visible surface of an object. They match each visible pixel to a point on a 3D model of the object, then solve for the object's rotation and position from those matches. This paper adds a second description of the same object. For each pixel, the network also predicts where the line of sight from the camera crosses the object's coordinate planes, which carries information about the parts of the object that are hidden from view. Two consistency losses force the two descriptions to agree on the same line of sight. The result is one network that outputs the pose in a single pass, with no separate optimisation step afterwards. The authors report that it beats other single-pass methods on the benchmarks they test, and that it comes close to slower methods that refine the pose iteratively.

TL;DR

  • SO-Pose takes one zoomed-in 256 x 256 RGB crop and predicts two layers with a shared encoder and two decoders: 2D-3D correspondences, and self-occlusion coordinates on the three coordinate planes §4.1 §3.1.
  • It regresses rotation and translation directly, with no PnP/RANSAC step §3.3 §4.1.
  • Two cross-layer consistency losses (3D and 2D) tie the layers together. They are switched on after 20% of training §3.2 §4.2.
  • Author-reported: best single-pass results on LINEMOD, Occluded-LINEMOD and YCB-V, and about 30 ms per object on a TITAN X §4.3 §4.5.
  • Reimplementation is not straightforward: the code differs from the paper on symmetric objects, the loss norm, the normalisation and the loss weights (see Paper vs code).

Problem

Single-layer end-to-end methods predict the pose from visible 2D-3D correspondences only. In clutter the correspondence field is noisy, and many poses fit it almost equally well, so training settles at a weak optimum §1. Two-stage methods (correspondences, then PnP inside RANSAC) are accurate but cannot be trained end to end and need extra optimisation at test time §1. The method assumes a single RGB image and a 3D model of the object fig. 2, and a 2D detection that crops the object fig. 2 §4.2.

Method

The network maps an image to a rotation R and translation t of the object §3 eq. 1. A crop from an off-the-shelf 2D detector is enlarged to 256 x 256 fig. 2 §4.1. The overall structure is:

  1. Shared encoder. A ResNeSt50 backbone for LMO and YCB-V, and ResNet34 for LINEMOD (the paper's setting; no LINEMOD config is in the repository) §4.1 gdrn_selfocc_multistep_40E.py.
  2. Decoder 1: correspondences. Three 64 x 64 groups of maps: the visible-object mask, the 2D-3D correspondence field, and a surface-region attention map. The last two follow GDR-Net §4.1.
  3. Decoder 2: self-occlusion. Six 64 x 64 channels, read as the Q0 coordinates of the three planes (two per plane) §4.1 §3.1. The code layout is in the next section conv_selfocc_head.py.
  4. Pose regression. Both decoder outputs feed the GDR-Net pose regression network, which outputs the pose directly §4.1.
  5. Training. The total loss is the GDR-Net pose loss plus two cross-layer terms and one self-occlusion term §3.3 eq. 12 eq. 13 eq. 14.

Self-occlusion definition §3.1. A ray from the camera centre through a visible object point P crosses the object's three coordinate planes (through the object origin, with normals along x, y and z) at Q_x, Q_y and Q_z fig. 3. The crossings are computed analytically from P, R and t eq. 6, so no renderer is needed. Crossings outside the minimum bounding cuboid Omega of the object are dropped, because a ray parallel to a plane may never cross it §3.1 fig. 3. Each plane crossing lies on its plane, so it has two free coordinates §3.1. Points are expressed in the object frame (P0 = R^T P - R^T t, Q0 likewise) and normalised by the object diameter §3.1.

Equation 6, in the paper's notation with rho the homogeneous pixel eq. 3:

Q_x = \frac{(R n_x)^\top t}{(R n_x)^\top K^{-1}\rho}\, K^{-1}\rho eq. 6

Substituting n_y or n_z gives Q_y and Q_z §3.1 eq. 5.

Cross-layer consistency §3.2. Since P and Q share one line of sight, two identities hold, and each gives a loss. Expanding eq. 6 gives the 3D identity in eq. 7. The 3D loss is eq. 8 and the 2D loss is eqs. 9 to 11, where pi(X) denotes the pinhole projection of a camera-frame point X, equal to K X divided by its depth:

(R n)^\top t\,(RP_0 + t) = (R n)^\top (RP_0 + t)\,(RQ_0 + t) eq. 7

\mathcal{L}_{cl\text{-}3D} = \frac{1}{|Q_0|}\sum_{P \in V,\, Q_0 \in \Omega} \left\| (Rn)^\top t\,(RP_0 + t) - (Rn)^\top (RP_0 + t)\,(RQ_0 + t) \right\|_1 eq. 8

\mathcal{L}_{cl\text{-}2D} = \frac{1}{|Q_0|}\sum_{P \in V,\, Q_0 \in \Omega} \left( \|e_{PQ}\|_1 + \|e_{Q\rho}\|_1 \right) eq. 9

e_{PQ} = \pi(RQ_0 + t) - \pi(RP_0 + t) \qquad e_{Q\rho} = \pi(RQ_0 + t) - \rho eq. 10 eq. 11

The projection pi divides by depth, so the depths of P and Q both enter the residuals eq. 10 §3.2.

Objective §3.3. The full loss is L = L_pose + L_cl + L_occ eq. 12, with L_cl = lambda_1 L_cl-2D + lambda_2 L_cl-3D eq. 13 and L_occ = lambda_3 L_Q eq. 14. L_Q = L_Q1 + L_Q2 eq. 15. L_Q1 is the L1 error of the predicted Q0 against ground truth, averaged over Omega eq. 16. L_Q2 projects the predicted Q0 with the ground-truth pose and penalises the distance to rho eq. 17. L_pose is GDR-Net's pose loss, which the paper does not spell out §3.3.

Pseudocode
# SO-Pose inference  [§4.1] [fig. 2]
# Input:  crop (B, 3, 256, 256)
# Output: rotation (B, 3, 3), translation (B, 3)

def so_pose(crop):
    # Shared encoder (ResNeSt50 for LMO and YCB-V)  [§4.1]
    features = encoder(crop)                  # (B, C, 8, 8)
    # Decoder 1: visible mask, correspondence field, region  [§4.1]
    mask, coords, region = corr_decoder(features)
    # Decoder 2: six self-occlusion channels at 64 x 64  [§4.1]
    occlusion = occ_decoder(features)         # (B, 6, 64, 64)
    # Pose regression from both layers, as in GDR-Net  [§4.1]
    rotation, translation = pose_head(
        coords, region, occlusion)
    return rotation, translation
Pseudocode
# Ground-truth Q0 for one image  [§3.1] [eq. 6]
# Input: object mask, camera intrinsics K, pose (R, t), box of Omega
# Output: per pixel and plane, Q0 or "no crossing"

def gt_q0(mask, K, R, t, box_min, box_max):
    # Walk over the pixels of the object mask  [§3.1]
    for each pixel rho in mask:
        # Ray direction through the pixel  [eq. 6]
        ray = inverse(K) @ rho
        for each plane normal n in (n_x, n_y, n_z):
            # Intersect the ray with the plane  [eq. 6]
            crossing = dot(R @ n, t) / dot(R @ n, ray) * ray
            # Back to the object frame  [§3.1]
            q0 = transpose(R) @ (crossing - t)
            # Keep only crossings inside Omega  [§3.1]
            keep = all(box_min < q0) and all(q0 < box_max)
            store(q0, keep)
Pseudocode
# Cross-layer losses  [§3.2] [eq. 8] [eq. 9]
# P0, Q0: object-frame points (normalised), per plane
# valid: pixels where Q0 exists inside Omega

def loss_cl_3d(R, t, P0, Q0, valid):
    # Camera-frame points  [§3.2]
    P = R @ P0 + t
    total = 0
    for each plane normal n in (n_x, n_y, n_z):
        Q = R @ Q0[n] + t
        # Residual vanishes when P, Q and the camera share a ray
        residual = dot(R @ n, t) * P - dot(R @ n, P) * Q
        # L1 in paper, L2 in code  [code: losses/crosstask_loss.py#L100]
        total += norm(residual) * valid[n]
    return total / count(valid)
Pseudocode
# 2D cross-layer loss  [§3.2] [eq. 9] [eq. 10] [eq. 11]
# rho: observed pixel of each point, from the ground-truth mask

def loss_cl_2d(R, t, P0, Q0, rho, K, valid):
    # Project the visible and occluded points  [eq. 10]
    p_img = project(K, R @ P0 + t)
    q_img = project(K, R @ Q0 + t)
    # P and Q must hit the same pixel  [eq. 10]
    e_pq = q_img - p_img
    # Q must hit the observed pixel  [eq. 11]
    e_qrho = q_img - rho
    return sum(abs(e_pq) + abs(e_qrho), valid) / count(valid)
Pseudocode
# Self-occlusion losses  [§3.3] [eq. 16] [eq. 17]
# Q0_gt, R_gt, t_gt: ground truth; rho: observed pixel

def loss_occ(Q0_pred, Q0_gt, R_gt, t_gt, rho, K, valid):
    # L_Q1: L1 to ground-truth Q0 over Omega  [eq. 16]
    l_q1 = mean(abs(Q0_pred - Q0_gt), valid)
    # L_Q2: predicted Q0 must project onto rho with ground-truth pose  [eq. 17]
    q_img = project(K, R_gt @ Q0_pred + t_gt)
    l_q2 = mean(abs(q_img - rho), valid)
    return l_q1 + l_q2
Pseudocode
# Total training loss  [§3.3] [§4.2] [eq. 12] [eq. 13] [eq. 14]
# f: focal length in pixels;  epoch counts from 1

def total_loss(epoch, n_epochs, loss_pose, losses):
    loss = loss_pose + losses.occ          # lambda_3 = 1  [eq. 14]
    # Cross-layer terms start after 20% of training  [§4.2]
    if epoch > 0.2 * n_epochs:
        # lambda_1 = 1 / f, lambda_2 = 10  [§4.2] [eq. 13]
        loss += losses.cl_2d / f
        loss += 10.0 * losses.cl_3d
    return loss

Hyperparameters

Values are from the paper unless the "Where" column names code. Items the paper does not state are marked.

NameValueWhereNotes
Network input256 x 256 zoomed-in RGB crop§4.1
Decoder outputs6 self-occlusion channels; 64 x 64 maps§4.1 conv_selfocc_head.pyThree upsampling stages from the 8 x 8 encoder output GDRN.py
BackboneResNet34 (LINEMOD); ResNeSt50 (LMO, YCB-V)§4.1 gdrn_selfocc_multistep_40E.pyNo LINEMOD config in repo
OptimiserRanger§4.2 gdrn_selfocc_multistep_40E.pylr 1e-4, weight decay 0
Batch size24§4.2 gdrn_selfocc_multistep_40E.py
Base learning rate1e-4§4.2
LR scheduleCosine annealing from 72% of training§4.2 gdrn_selfocc_multistep_40E.py
Warm-upnot stated (paper)gdrn_selfocc_multistep_40E.pyCode: factor 0.001 for 1000 iterations
lambda_1 (2D cross-layer)1 / f§4.2 crosstask_projection_loss.pyCode divides by constant 572.3
lambda_2 (3D cross-layer)10§4.2 GDRN.py
lambda_3 (self-occlusion)1§4.2 GDRN.pyCode weights L_Q2 by 10 (LMO)
Cross-layer startafter 20% of total epochs§4.2 GDRN.py engine.pyEpoch counts from 1
Training lengthnot stated (paper)gdrn_selfocc_multistep_40E.pyCode: 40 epochs (LMO), 15 (YCB-V)
Augmentationcolour augmentation; mask erosion§4.2 gdrn_selfocc_multistep_40E.pyCode: coarse dropout, blur, additive shift, invert, multiply, contrast
Zoom-in paddingnot stated (paper)gdrn_selfocc_multistep_40E.pyCode: scale 1.5
2D detectorFaster R-CNN (LMO); FCOS (YCB-V)§4.2
Symmetric objectsnot specially handled (paper)§4.2 gdrn_selfocc_multistep_10e.pyYCB-V config sets HANDLE_SYM=True
Hardwaresingle TITAN X GPU§4.2
Minimum Q0 region20 valid pixels per planegdrn_selfocc_multistep_40E.py GDRN.pyCounted over the whole batch, not per sample; the symmetric path hard-codes 20 coor_sym_loss.py
Omegaminimum bounding cuboid of the object§3.1 generate_Q0.pyCode: min/max of model vertices per axis
Normalisationby object diameter§3.1 data_loader.pyCode: per-axis extent, not diameter

Results

Author-reported. All numbers are from the paper, with the setup it gives. The LINEMOD ablation tab. 1 uses the LINEMOD protocol §4.2.

Ablation on LINEMOD (ADD(-S) and 2 deg 2 cm / 5 deg 5 cm) tab. 1:

MethodADD(-S) 0.02dADD(-S) 0.05dADD(-S) 0.1d2 deg 2 cm5 deg 5 cm
CDPN tab. 1--89.9-94.3
GDR-Net tab. 135.376.393.762.195.6
SO-Pose without 3D and 2D terms (Ours(S)) tab. 136.676.894.059.197.0
SO-Pose without 3D term tab. 141.681.795.767.497.1
SO-Pose without 2D term tab. 144.781.395.573.198.0
SO-Pose (full) tab. 145.983.196.076.998.5

YCB-V tab. 2 (P.E.: 1 = one model for all objects, M = one model per object; Ref.: refinement):

MethodP.E.Ref.ADD(-S)AUC ADD-SAUC ADD(-S)
PoseCNN tab. 2121.375.961.3
SegDriven tab. 2139.0--
PVNet tab. 2M--73.4
S.Stage tab. 2M53.9--
GDR-Net tab. 2149.189.180.2
DeepIM tab. 21yes-88.181.9
CosyPose tab. 21yes-89.884.5
SO-Pose, ResNet34 tab. 2154.689.782.3
SO-Pose, ResNeSt50 tab. 2156.890.983.9

Occluded-LINEMOD, ADD(-S) average recall tab. 3. +syn means real plus synthetic images; +pbr means real plus physically based renders §4.2 §4.3. Mean over the eight objects:

MethodMean ADD(-S)
PoseCNN tab. 324.9
PVNet tab. 340.8
S.Stage tab. 343.3
HybridPose tab. 347.5
GDR-Net (+syn) tab. 347.4
GDR-Net (+pbr) tab. 356.1
SO-Pose (+syn) tab. 354.3
SO-Pose (+pbr) tab. 362.3
DPOD, with refinement tab. 347.3
DeepIM, with refinement tab. 355.5

Per object for SO-Pose (+pbr), in the order Ape, Can, Cat, Driller, Duck, Eggbox, Glue, Holep. tab. 3: 48.4, 85.8, 32.7, 77.4, 48.9, 52.4, 78.3, 75.3. Per object for SO-Pose (+syn): 46.3, 81.1, 18.7, 71.3, 43.9, 46.6, 63.3, 62.9. The paper states SO-Pose (+pbr) is best on five of eight objects §4.3. Averaging the per-object values in tab. 3 gives 62.4 for +pbr and 54.3 for +syn. The table reports 62.3 for +pbr, so the per-object values are rounded.

BOP metrics tab. 4 (mean AR over the three error types, per dataset):

MethodP.E.Ref.LMO VSDLMO MSSDLMO MSPDYCB-V VSDYCB-V MSSDYCB-V MSPDMean AR
CosyPose tab. 41yes0.4800.6060.8120.7720.8420.8500.727
EPOS tab. 410.3890.5010.7500.6260.6770.7830.621
PVNet tab. 4M0.4280.5430.754----
CDPN-v2 tab. 4M0.4450.6120.8150.3960.5700.6310.578
GDR-Net tab. 41---0.5840.6740.726-
SO-Pose tab. 410.4420.5810.8170.6520.7310.7630.664

The paper says SO-Pose is second to CosyPose, which uses refinement §4.3.

Ablation on top of CDPN tab. 5. Adding the self-occlusion branch to CDPN, with the consistency term L_cdpn eq. 18 (it swaps the predicted rotation for the ground-truth one in both terms), improves ADD(-S), 2 deg 2 cm and 5 deg 5 cm. The dataset and protocol are not named in the caption or text tab. 5 §4.4.

MethodADD(-S)2 deg 2 cm5 deg 5 cm
CDPN* (updated baseline) tab. 592.8268.5697.03
with self-occlusion, no consistency term tab. 593.0969.3797.06
with self-occlusion and consistency term tab. 594.7771.3397.07

Runtime §4.5 fig. 5. On a 640 x 480 YCB-V image: about 30 ms for one object and 50 ms for all objects, on an Intel 3.30 GHz CPU and a TITAN X (12 GB). The total includes about 15 ms for YOLOv3 2D localisation. The paper reports an AR of 0.715 for real-time speed, second best §4.5. This differs from the 0.664 in tab. 4; the text does not say which AR it uses.

Paper vs code

The official code is at github.com/shangbuhuan13/SO-Pose, commit ede1981036c4bd58ace249e5564ee0afac6b9517 (Apache-2.0). The paper footnote links this repository, and its owner matches the first author's handle, so the card treats it as official. Code paths below are shortened to the suffix under core/gdrn_selfocc_modeling/ or configs/gdrn_selfocc/.

Matched without change: the 6-channel self-occlusion head, the 64 x 64 output size, the 20% start of cross-layer training, and the Ranger, learning-rate, batch-size and cosine-schedule settings §4.2.

Pitfalls

  • Channel order. The head names channels "xy", "xz" and "yz", but the names do not match the planes. The order is: channels 0 and 1 hold the y and z coordinates of Q_x; channels 2 and 3 hold x and z of Q_y; channels 4 and 5 hold x and y of Q_z generate_Q0.py data_loader.py. Use this order, not the names.
  • Epoch gate. The cross-layer terms switch on when the 1-based epoch exceeds 0.2 times the total GDRN.py engine.py. The per-iteration gate in the training loop is commented out engine.py.
  • Batch-level skip. If a plane has fewer than three valid pixels per sample in the batch, that plane's 3D term is set to zero for the whole batch crosstask_loss.py. This is not per sample.
  • Degenerate crossings. A ray nearly parallel to a plane gives a huge crossing. The code drops points where |z| is below 1e-4 crosstask_projection_loss.py and tests Omega with strict inequalities generate_Q0.py. Points on the box boundary are dropped.
  • GPU only. Several loss helpers allocate tensors with .cuda() crosstask_loss.py. CPU runs need edits.
  • Unstated details. Warm-up, weight decay, zoom padding, augmentation strengths, the training length and the LINEMOD configuration are not in the paper. The code values are the best available reference; they were not checked against the reported numbers gdrn_selfocc_multistep_40E.py.
  • Ablation dependence. The results depend on detector quality. LMO uses Faster R-CNN and YCB-V uses FCOS §4.2. The paper's ablations do not vary the detector.
  • Internal inconsistency. The runtime section gives AR 0.715, the BOP table gives 0.664 §4.5 tab. 4. The text does not say which metric produces 0.715.

Lineage

  • Builds on: GDR-Net (CVPR 2021), cited as [GDRN] in the paper. The paper uses its pose loss, its pose regression network and its 2D-3D correspondence formulation §3.3 §4.1. The code is built on the GDR-Net repository README.md.
  • Related two-layer idea: the paper cites multi-layer models in 3D reconstruction, which store the coordinates of self-occluded points §3.1. SO-Pose stores plane crossings instead.
  • Compared against: CosyPose, EPOS, CDPN-v2, DeepIM, PVNet, HybridPose, DPOD, S.Stage and PoseCNN tab. 2 tab. 3 tab. 4.
  • Superseded by: not known from content already in this wiki.

Source and license

Adapted from “SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation” by Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, Federico Tombari (arXiv:2108.08367), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/shangbuhuan13/SO-Pose at commit ede1981036c4 (Apache-2.0). Any quoted code is shown under that license.