Self-supervised 6D Object Pose Estimation for Robot Manipulation

In plain words

Teaching a robot to pick up things needs software that can tell where each object is and how it is turned. Training that software on real photos means someone has to mark those positions by hand, which is slow. Training on computer-made images is cheap, but real cameras and lighting look different, so the software often fails on real objects.

This paper's system lets the robot label its own real images. It starts from software trained on computer-made images, looks at a scene from several angles with a camera on its wrist, and works out where each object is and how it is turned. It keeps only the estimates it can check, then moves an object by pushing or grasping it, and repeats. The labelled real images are used to retrain the software, which then does better on real objects and grasps them more reliably.

TL;DR

• Self-supervised 6D pose labelling for a manipulator with a wrist RGB-D camera; no human annotation of the real images §I §III-A.
• Pose initialisation and tracking use PoseCNN and PoseRBPF; an SDF fit refines poses; a render-and-compare check decides whether to save a frame §III-B §III-C.
• The robot pushes or grasps objects between tracking passes to create new layouts §III-D.
• Author-reported: segmentation F1 60.8 to 88.7 with all real data tab. 1; grasp success 46.70% to 86.70% over 30 trials tab. 2.

Problem

Real-world 6D pose labels are costly to make by hand, and networks trained only on synthetic images often fail on real objects with textureless surfaces, metal reflections or shine, or altered product textures §I §IV-D. Earlier self-supervised robot work covered grasping, segmentation and 2-D detection, but not 6D pose labels. The paper states that the hard parts are estimating 6D pose accurately and checking the estimate automatically §I.

The method assumes a known 3-D model for each object §II, a calibrated RGB-D camera mounted on the arm, and forward kinematics of the arm for the motion prior §III-C. The paper gives no statement on how the method behaves for unknown objects.

Method

Overview §III-A fig. 2: the system has four modules: pose initialisation, pose tracking, robot interaction and self-supervised training.

1. Pose initialisation §III-B fig. 3. Segment and detect objects with PoseCNN, which labels pixels and uses Hough voting for centres and boxes §III-B1. Sample particles around each detected centre using the depth image, and estimate rotation and translation with PoseRBPF's auto-encoder-based particle filter §III-B2. Refine the pose with the SDF fit of step 3 §III-B3. Evaluate the result and re-initialise any object that fails §III-B4. The paper says the camera is moved to a high-elevation view first; the angles are not stated in the text.
2. Particle sampling §III-B2. For each detected object, translation particles are drawn with (u, v) Gaussian around the detected centre and z uniform within ±d/2 of the depth at (u, v). Each particle gives a region of interest that is passed through the auto-encoder to score its likelihood. Then K further filtering steps are run on the same image; the expected rotation and translation are read from the particles.
3. SDF refinement §III-B3 eq. 1 eq. 2. Keep the object pixels where the PoseCNN mask, the rendered mask and the measured-versus-rendered depth agree within a margin m §III-B3. Back-project those pixels to a point cloud eq. 1. Minimise the absolute signed distance of the points to the object model, plus a translation prior weighted by λ eq. 2.
4. Pose evaluation §III-B4 fig. 4. Render each object at its estimate. Compute the cosine similarity s between auto-encoder codes of the rendered and measured regions of interest §III-B4. Compute the depth error e as a sum of absolute differences over the object mask §III-B4. Declare failure if s < s* or e > e* §III-B4.
5. Pose tracking §III-C. After initialisation, the camera moves through predefined waypoints and collects frames in real time. Particles are propagated with the forward-kinematics motion eq. 3 eq. 4. Rotation is shifted on the discrete grid by the pitch, yaw and roll change of the arm, and the grid values are bilinearly interpolated §III-C. One filtering step is then run on the current frame for every object §III-C. A frame is saved only if both evaluation metrics pass §III-C. Saved frames are refined with step 3 §III-C.
6. Robot interaction §III-D fig. 5. Each object is either pushed in a random direction and radius, toward the centre of the other objects or of the workspace, or grasped and placed elsewhere. The grasps are 100 parallel-jaw grasps per object, sampled from the robust grasps of Eppner et al. (ISRR 2019) §III-D; at run time they are transformed into the camera frame, checked for kinematic feasibility and collisions, and planned with RRT* in MoveIt! §III-D. The gripper then rotates the object and places it at a random location in the workspace §III-D. The paper states that one object is pushed or grasped after each tracking pass §I.
7. Self-supervised training §III-E. Start with single-object scenes; if segmentation fails, use global particle initialisation in PoseRBPF §III-E. Fine-tune the segmentation network on the single-object data, then move to cluttered scenes and fine-tune the segmentation network and the auto-encoders §III-E. Synthetic and real data are mixed during fine-tuning §III-E. Data collection and fine-tuning are alternated §III-E.

Pose-estimation metrics §IV-B (ADD, ADD-S from the cited references; the paper gives them inline, unnumbered):

\mathrm{ADD} = \frac{1}{m}\sum_{\mathbf{x} \in \mathcal{M}}\left\| (\mathbf{R}\mathbf{x} + \mathbf{t}) - (\tilde{\mathbf{R}}\mathbf{x} + \tilde{\mathbf{t}}) \right\| \qquad [§IV\text{-}B]

\mathrm{ADD\text{-}S} = \frac{1}{m}\sum_{\mathbf{x}_1 \in \mathcal{M}} \min_{\mathbf{x}_2 \in \mathcal{M}} \left\| (\mathbf{R}\mathbf{x}_1 + \mathbf{t}) - (\tilde{\mathbf{R}}\mathbf{x}_2 + \tilde{\mathbf{t}}) \right\| \qquad [§IV\text{-}B]

Key equations from the method:

\mathbf{P}_{\text{obj}} = \left\{ \mathbf{D}(u,v)\,\mathbf{K}^{-1}(u,v,1)^{T} : (u,v) \in \mathbf{\Omega}_{\text{obj}} \right\} \qquad [eq. 1]

(\hat{\mathbf{t}}, \hat{\mathbf{R}}) = \arg\min_{\mathbf{t},\mathbf{R}} \sum_{\mathbf{p}_i \in \mathbf{P}_{\text{obj}}} \left| \mathbf{SDF}_{\text{obj}}(\mathbf{p}_i, \mathbf{t}, \mathbf{R}) \right| + \lambda \tfrac{1}{2} \| \mathbf{t} - \bar{\mathbf{t}} \|^2 \qquad [eq. 2]

P(\mathbf{t}_k \mid \mathbf{t}_{k-1}, \Delta\mathbf{t}, \Delta\mathbf{R}) = \mathcal{N}(\Delta\mathbf{R}\,\mathbf{t}_{k-1} + \Delta\mathbf{t}, \Sigma_{\mathbf{t}}) \qquad [eq. 3]

P(\mathbf{R}_k \mid \mathbf{R}_{k-1}, \Delta\mathbf{t}, \Delta\mathbf{R}) = \mathcal{N}(\Delta\mathbf{R}\,\mathbf{R}_{k-1}, \Sigma_{\mathbf{R}}) \qquad [eq. 4]

Pose evaluation quantities §III-B4 (inline in the paper, unnumbered):

s = \frac{\hat{\mathbf{c}} \cdot \mathbf{c}}{\|\hat{\mathbf{c}}\| \, \|\mathbf{c}\|} \qquad e = \sum_{(u,v) \in \mathbf{\Omega}_{\text{obj}}} \left| \hat{\mathbf{D}}(u,v) - \mathbf{D}(u,v) \right|

Pseudocode of the outer loop §III-A §III-B §III-C §III-D §III-E. The paper has no algorithm box; this is a reading of the text:

# One collection-and-training round. No algorithm box in the paper.
# Input:  objects, networks (segmentation + auto-encoders), dataset
# Output: updated networks

def self_supervised_round(objects, networks, dataset):
    # Initialise poses from one RGB-D view; retry failed objects  [§III-B]
    poses = initialise_poses(objects, networks)
    # Track while the camera visits the predefined waypoints  [§III-C]
    for view in predefined_waypoints:
        poses = track_poses(poses, view.rgbd, view.kinematics_delta)
        # Keep the frame only if both metrics pass  [§III-C] [§III-B4]
        if evaluation_passes(poses, view.rgbd):
            poses = refine_sdf(poses, view.rgbd)
            dataset.append(view.rgbd, poses)
    # Push or grasp one object to create a new layout  [§I] [§III-D]
    push_or_grasp_one_object(objects)
    # Fine-tune on a mix of self-labelled real and synthetic data  [§III-E]
    networks = fine_tune(networks, dataset, synthetic_data)
    return networks

Pose evaluation, as a pass/fail check §III-B4:

# Render-and-compare check; failure if s < s_star or e > e_star  [§III-B4]
# Inputs: poses, rgbd. s_star and e_star from [§IV-C].

def evaluation_passes(poses, rgbd):
    # Render each object at its estimated pose  [§III-B4]
    rendered = render_rgbd(poses)
    # Cosine similarity of auto-encoder codes of the two regions  [§III-B4]
    s = cosine(encode(rendered.roi), encode(rgbd.roi))
    # Sum of absolute depth differences inside the object mask  [§III-B4]
    mask = object_mask(poses, rgbd)
    e = sum(abs(rendered.depth - rgbd.depth)[mask])
    # Pass only if neither failure condition holds  [§III-B4]
    return s >= s_star and e <= e_star

SDF refinement, as a function §III-B3 eq. 1 eq. 2:

# Refine one object's pose against its measured points  [§III-B3]
# Inputs: pose (t_bar, R_bar), rgbd, seg_mask (PoseCNN), margin m [§IV-C]

def refine_sdf(pose, rgbd, seg_mask):
    # Render depth and mask at the current pose  [§III-B3]
    rendered_depth, rendered_mask = render(pose)
    # Keep pixels in all three masks: segment, render, depth  [§III-B3]
    depth_close = abs(rgbd.depth - rendered_depth) < margin_m
    object_mask = seg_mask & rendered_mask & depth_close
    # Back-project the kept pixels to 3-D points  [§III-B3] [eq. 1]
    points = backproject(rgbd.depth, object_mask, camera_intrinsics)
    # Minimise SDF residuals plus the translation prior  [§III-B3] [eq. 2]
    t, R = minimise_sdf_objective(points, pose, lam)
    return t, R

Particle propagation under the motion prior §III-C eq. 3 eq. 4:

# Move every particle by the arm's camera motion  [§III-C]
# Inputs: particles (each: t, R on a discrete SO(3) grid), delta_t, delta_R

def propagate(particles, delta_t, delta_R):
    for particle in particles:
        # Translation: rotate, shift, add Gaussian noise  [§III-C] [eq. 3]
        particle.t = delta_R @ particle.t + delta_t
        particle.t = particle.t + noise(sigma_t)
    # Rotation: shift the grid, then bilinearly interpolate  [§III-C]
    rotation_grid = shift_grid(particles.R_grid, delta_angles)
    particles.R_grid = bilinear_interpolate(rotation_grid)
    return particles

Hyperparameters

All values from the paper's text §IV-C unless the row says otherwise. Hardware and split are included because they affect what can be reproduced.

Results

Author-reported. Setup: 20 YCB objects, 232 test scenes §IV-A for segmentation; 6D pose on the same test set with ADD and ADD-S §IV-B; grasping with 30 trials in six clutter sets, five objects each §IV-F against the same robot and a synthetic-only baseline.

Segmentation, mean F1 over 20 objects on the test set tab. 1:

Per-object values are in tab. 1. Objects below 60% on synthetic data are marked red in the table, and the text names textureless objects, metal-reflective objects and objects whose real textures differ from the models §IV-D.

Grasping, 30 trials in clutter tab. 2:

Four grasps failed: three because of pose-estimation errors and one because MoveIt! found no feasible trajectory §IV-F.

6D pose estimation with ADD and ADD-S on the test set fig. 6: the text says both methods (this one and PoseCNN) improve as real data grows; using depth and SDF refinement helps; more particles help, less so once refinement is applied; and the RGB-only particle filter with SDF refinement is best §IV-E. Numbers are only in the figure, so none are given here. The text attributes the RGB-only advantage to the auto-encoders being trained only on RGB images §IV-E.

Qualitative examples: fig. 7 (synthetic network fails, fine-tuned succeeds) and fig. 8 (grasps in clutter, 5 successes and 1 failure shown).

Paper vs code

No official code.

Searched for code in these places, none of which gave a repository by the authors:

• The paper's links: only the video link in the abstract (youtu.be/W1Y0Mmh1Gd8) [abstract]. The LaTeX has no code link.
• The arXiv abstract page: no code link, project page or comment giving one.
• GitHub API repository search, three queries: "self-supervised 6d pose" with org:NVlabs; "self-supervised pose estimation robot manipulation"; "self-supervised 6d object pose". The results were third-party work or unrelated.
• NVlabs repository list for "pose": no repository on self-supervised labelling or robot manipulation. The only PoseRBPF match is NVlabs/PoseRBPF, which is the upstream tracker §III-B2, not this paper's system; it carries a NOASSERTION license and is not used here.

Pitfalls

• Initialisation K is not stated. The text says K filtering steps are run §III-B2 but gives no value. A commented-out draft in the LaTeX source lists K = 2; that is not part of the paper and is not used here. Confirm it with the authors before reproducing.
• Depth error is called a mean but is a sum. The text calls e "the mean depth error" §III-B4, but the formula is a sum over the mask. Its scale grows with mask size, so e* = 0.03 m §IV-C cannot be read as a per-pixel mean. Implement the sum, as written, and check.
• Failure is stated with strict inequalities. Failure is s < s* or e > e* §III-B4. Whether a value exactly equal to a threshold passes is not stated; the card's pass rule uses ≥ and ≤ as the natural reading.
• Refinement regulariser depends on point count. Eq. 2 sums absolute SDF values over all points with no normalisation, while the translation prior is fixed eq. 2. So the relative weight of the translation prior, and hence the effect of λ = 0.001, changes with the number of points in P_obj. The paper does not say how many points occur, nor which optimiser is used.
• Mask intersection depends on the depth margin. Ω_obj is the intersection of the PoseCNN mask, the rendered mask and a depth-margin mask §III-B3. A bad render or a bad PoseCNN mask removes or keeps the wrong pixels; the paper does not discuss failure rates of this step.
• Particle sampling parameters are incomplete. d, the particle count and the rotation sampling are not stated §III-B2, so the initial distribution cannot be reproduced exactly from the text.
• Rotation shift conventions are not stated. The rotation change is written as pitch, yaw and roll §III-C, but the axis order, sign convention and the grid resolution are not given. Bilinear interpolation on the discrete grid is the only stated detail.
• Labels for segmentation are not spelled out. The paper describes self-annotated images and masks §III-B3 §III-E but does not state exactly how the training masks for the segmentation network are produced from Ω_obj or from the rendered masks.
• The text's relative improvement uses absolute points for segmentation. "The overall F1 score is increased by 25%" §IV-D: tab. 1 gives 60.8 to 85.9, which is +25.1 points (about +41% relative). Quote the points, not the percentage.
• Initialisation time reduction is rounded. The text says the time falls by 55.3% §IV-F. From tab. 2 (10.04 s to 4.48 s) the relative change is 55.4%. The success-rate gain (+85.7% in the text) is relative: 46.70 to 86.70 is +40.0 points.
• Trial counts per row are not explicit. tab. 2 caption says "30 grasping trials in clutter" but does not say whether both rows use 30 trials each. Read the rows as 30 trials each until the authors confirm.
• Grasp selection needs a pre-computed set. Grasps come from a precomputed set of 100 per object §III-D. Objects outside that set cannot be grasped by this method, and the set's generation is only referenced, not described.
• Real-world dependence on hardware. Motion priors depend on forward kinematics and hand-eye calibration; the paper names noise from encoders and calibration bias as sources of error §III-C. Reproducing the tracking step needs a calibrated arm.
• Timeouts shape the reported numbers. A 45 s per-object initialisation timeout counts as failure §IV-F. Whether the average initialisation times in tab. 2 include timed-out objects is not stated.
• Training recipe is not given. No optimiser, learning rate, schedule, epoch count or batch size is stated for the segmentation network or the auto-encoders §III-E. Use the cited prior work for these, and do not treat any value from it as this paper's.
• Version. The card follows the LaTeX source of the arXiv version fetched by wikillm fetch (metadata version v2, updated 2020-03-07).

Lineage

• Builds on: PoseRBPF, a Rao-Blackwellised particle filter for 6D tracking §III-B2; PoseCNN for segmentation and Hough-voting centres §III-B1; the implicit-orientation auto-encoder of Sundermeyer et al. (ECCV 2018) §III-B2; SDF-based refinement from DART §III-B3; curriculum learning §III-E; the grasp set from Eppner et al. (ISRR 2019) and MoveIt! §III-D. None of these has a card in this wiki yet; they are named, not linked.
• Compared against: PoseCNN's direct regression of distance and rotation, under the same segmentation network §IV-E; a synthetic-only baseline tab. 1 tab. 2.
• Superseded by: not known from this wiki's content.