High precision grasp pose detection in dense clutter

In plain words

A robot that wants to pick things out of a pile needs to know where to close its gripper. One approach first makes a long list of possible gripper poses from the depth scan of the pile, then decides for each pose whether it would actually hold something. This paper works on that second decision.

The authors make the decision with a small image classifier. Each candidate pose is turned into a few small pictures of the space between the fingers, seen from three directions. The pictures show where the surface is, where the camera could not see, and which way the surface faces. A network trained on many labelled examples then says "grasp" or "no grasp".

Three things help the classifier. Showing it the parts the camera missed helps a little. Pretraining on depth images rendered from computer models speeds up learning. Knowing what kind of object is being grasped helps too, and knowing the exact object helps most. On a real robot in a cluttered tray, the authors report that the system grasped successfully far more often than their earlier version.

TL;DR

GPD (grasp pose detection) draws several thousand 6-DoF parallel-jaw candidates from a point cloud §3.2 and classifies each one. Each candidate becomes a 15-channel 60 x 60 image set built from three projections of a voxelised box between the fingers §3.3. A four-layer LeNet-style CNN scores it §3.2. Use this if you need the classifier design or the training labels; the authors' 93% robot result is author-reported on one robot §5.4. The paper's labels assume frictionless antipodal grasps, but the released code uses a 20 degree friction cone, so check which you want §3.4 [code: cfg/generate_data.cfg#L54].

Problem

Grasp pose detection finds grasps from sensor data without a CAD fit, so it can handle objects it has never seen §1. Many such methods reach 75% to 95% grasp accuracy on isolated or lightly cluttered objects, and the authors argue this is too low for practical grasping §1. Classification accuracy also says little about success: a grasp system needs high precision, because a false positive costs a failed attempt §4.1. The paper targets recall at high precision, for example recall at 99% precision §4.1.

Method

Pipeline §3.2 alg. 1. GPD has two steps. Step 1 samples grasp candidates from the cloud. Step 2 classifies each candidate with a CNN §3.2.

# GPD  [alg. 1]
# Input:  point cloud with viewpoints CV; region to grasp C_G; hand geometry
# Output: set H of 6-DOF hand poses

def grasp_pose_detection(CV, C_G, theta):
    # 1. Sample several thousand grasp candidates  [§3.2]
    candidates = sample_grasp_candidates(CV, C_G, theta)
    # 2. Classify each candidate as grasp or not  [§3.2]
    hands = classify(candidates, CV)
    return hands

Candidate sampling §3.2. Sample points uniformly from C_G. At each point, estimate the surface normal and the axis of major principal curvature. Place the hand at regular orientations orthogonal to the curvature axis, push it forward until the fingers touch the cloud, and discard it if no cloud points lie between the fingers. Stop when enough candidates exist. The full method is in the authors' earlier paper §3.2.

# Step 1: sample candidates  [§3.2]
# Each candidate is a 6-DOF hand pose in SE(3).

def sample_grasp_candidates(CV, C_G, theta):
    candidates = []
    while len(candidates) < desired_count:
        # Draw a point and its local surface frame  [§3.2]
        point = random_point(C_G)
        normal, curvature_axis = local_frame(point, CV)
        for orientation in orientations_orthogonal_to(curvature_axis):
            # Push the hand in until the fingers contact the cloud  [§3.2]
            hand = push_until_contact(point, normal, orientation, theta)
            # Keep it only if cloud points lie between the fingers  [§3.2]
            if points_between_fingers(hand, CV):
                candidates.append(hand)
    return candidates

Network §3.2. The classifier is a four-layer CNN: two convolution and pooling stages, one fully connected layer with ReLU, and one output layer with softmax §3.2. The authors say the kernel sizes, output counts and pooling strides are identical to Caffe's LeNet solver §3.2. The paper does not restate those numbers; the released Caffe prototxt has them [code: models/caffe/15channels/lenet_15_channels.prototxt#L8-L86].

Grasp representation §3.3. The region R is the cuboid swept by the fingers as they close. Its rectangular fingers make it a cuboid §3.3. R is scaled to the unit cube and voxelised into a 60 x 60 x 60 grid §3.3. Each voxel has an occupied flag V and an observed flag U, and each occupied voxel has a unit outward normal §3.3. The paper ignores RGB §3.3.

Three projections are taken, along the hand approach axis (x), the curvature axis (y) and the side axis (z) §3.3 fig. 2. For each projection, the paper defines three images §3.3. The equations there are unnumbered in the source, so they are cited by section:

• I_o: averaged height of occupied voxels along the projection axis, a 60 x 60 image.
• I_u: averaged height of unobserved voxels along the projection axis, a 60 x 60 image.
• I_n: averaged surface normal of occupied voxels, a 60 x 60 x 3 image. The paper writes its value as the absolute value of the summed normal, divided by the occupied count §3.3.

Five channels per projection give 15 channels in total §3.3.

# Grasp image for one candidate  [§3.3]
# occupied V, observed U: (60, 60, 60) in the unit cube
# normals: (60, 60, 60, 3), unit outward normal per occupied voxel

def grasp_image(occupied, observed, normals):
    # Three projections: approach (x), curvature (y), side (z)  [§3.3]
    images = []
    for axis in [0, 1, 2]:
        # Averaged height of occupied voxels  [§3.3]
        height_occupied = mean_height(occupied, axis)      # (60, 60)
        # Averaged height of unobserved voxels  [§3.3]
        height_unobserved = mean_height(1 - observed, axis)  # (60, 60)
        # Averaged normal, magnitude of the summed normal over occupied
        # voxels, divided by their count  [§3.3]
        normal_image = mean_normal(normals, occupied, axis)  # (60, 60, 3)
        images += [height_occupied, height_unobserved, normal_image]
    # Stack to 15 channels  [§3.3]
    return stack_channels(images)                          # (60, 60, 15)

Training labels §3.4. Candidates are labelled from mesh models. A candidate is positive when a frictionless antipodal grasp exists, so each contacting surface normal is anti-parallel to the closing direction and the contacts are collinear §3.4 fig. 3. The authors allow each mesh vertex a 1 mm position error and accept a grasp if any perturbation of the vertices gives an antipodal pair §3.4. The paper describes two sources of mesh data: simulated depth images of 3DNET CAD models, or real depth scans with reconstructed meshes from BigBird §3.4 §4.3.

# Label one candidate  [§3.4]
# Positive only if a frictionless antipodal pair of contacts exists,
# allowing 1 mm vertex noise on the mesh.

def label_candidate(candidate, mesh):
    # Keep the mesh points between the closing fingers  [§3.4]
    points = points_between_fingers(candidate, mesh)
    # Accept if some perturbation (up to 1 mm) gives an antipodal pair [§3.4]
    return has_frictionless_antipodal_pair(points, tolerance_mm=1.0)

Training data for the experiments §4.2. The benchmark uses 55 BigBird objects. The authors' selection uses 26 box-type, 16 cylindrical and 13 other objects, from 125 objects with a complete mesh that a 10 cm gripper can grasp §4.2. The paper says 216k candidates are balanced between positives and negatives, split 185k train and 31k test by view, so test views never appear in training §4.2. Candidates come from two stereo sensors whose viewpoints are 53 degrees apart §4.2.

Hyperparameters

Results

Author-reported, all on the authors' own data and robot. Accuracy is the share of correct predictions. RAHP is recall at 99% precision §4.1. The RAHP values for the Figure 5 representations are read from that figure's legend, because the text does not state them fig. 5.

Failure causes, passive: 5 registration or kinematic, 13 GPD perception, 2 finger collision, 2 object drop §5.4. Active: 5 registration or kinematic, 9 GPD perception, 4 finger collision, 2 object drop §5.4.

Paper vs code

Compared with code_commit 6327f20 at https://github.com/atenpas/gpd, the repository of Andreas ten Pas, an author. Licence: BSD-2-Clause, copyright Andreas ten Pas [LICENSE.md].

• Learning rate. Paper: 0.00025 §3.2. Caffe solver: base learning rate 0.01 with "inv" decay lenet_solver_15_channels.prototxt. PyTorch trainer: SGD with lr 0.00001 and momentum 0.9 train_net.py. No file in the repo contains 0.00025.
• Friction and labels. Paper: frictionless antipodal §3.4. Code: cone of 20 degrees generate_data.cfg antipodal.cpp. Only FULL antipodal grasps are labelled positive; HALF grasps are not [code: src/gpd/candidate/hand_search.cpp#L122-L127].
• Mesh tolerance. Paper: 1 mm vertex error §3.4. Code: 3 mm extremal threshold hand_search.cpp. Code also requires 6 points per side generate_data.cfg; the paper states no minimum.
• Channel values. Paper: averages of heights and normals in grid coordinates §3.3. Code: depth channel is 1 minus the mean z, then min-max scaled over the image and stored as 8-bit [code: src/gpd/descriptor/image_strategy.cpp#L158-L191]. Normal channel uses per-component absolute values with a weighted running update, min-max scaling and 8-bit storage image_strategy.cpp. Occlusion channel: reversed mean depth of shadow points, min-max scaled and 8-bit image_strategy.cpp. The code's per-projection list is normals (3), depth, shadow [code: src/gpd/descriptor/image_15_channels_strategy.cpp#L86-L105]. The paper's I_n formula, a norm over the summed normal, is not the code's per-component rule.
• Robot representation. Paper uses the 3-channel representation on the robot §5. The code's 3-channel strategy builds one normals image [code: src/gpd/descriptor/image_3_channels_strategy.cpp#L25-L33]. The paper's "curvature axis" projection is consistent with this, but I did not check the projection axis in detail.
• Candidate counts. Paper: "several thousand" §3.2. Code defaults: 400 for data generation generate_data.cfg and 30 for detection caffe_params.cfg.
• Robot pipeline. The clutter-removal system (IKFast, OpenRAVE, TrajOpt, utility weights) is not in this repo. A search of the repo for IKFast, OpenRAVE and TrajOpt found nothing. The 3 cm to 7 cm filter and utility function cannot be checked against code.
• Released weights. The repo ships Caffe weights, including two_views_15_channels_53_deg.caffemodel [code: models/caffe/15channels/two_views_15_channels_53_deg.caffemodel]. The paper does not say these match its reported numbers, and the 3DNET-pretrained weights are not in the repo.
• Config paths. Some configs point to the author's home directory, for example data_root = /home/andreas/data/bigbird/ [code: cfg/bigbird_train_params.cfg#L1]. They must be changed before use.

Pitfalls

• Learning rate and optimizer are not reproducible from the paper alone. The paper gives 0.00025 and nothing else about the optimizer §3.2. The released Caffe solver uses 0.01 with "inv" decay, and the PyTorch trainer uses 0.00001. Choose one and record it.
• The labels are frictionless in the paper, a 20 degree cone in the code. The two produce different positive sets. The paper argues that frictionless antipodal grasps are in force closure for any positive coefficient of friction §3.4. The code's cone is looser.
• Mesh noise changes the labels. BigBird meshes are reconstructed from sensor data and are noisy §3.4. The 1 mm tolerance and the code's 3 mm threshold both assume the mesh is accurate to that level.
• Channel scaling is not in the paper. The code rescales depth and normal channels to 8-bit images per image. A reimplementation that uses raw averages will not match the released weights.
• Equations are unnumbered in the source. The paper's representation and labelling formulas have no equation numbers, so this card cites sections.
• Occlusion channels cost more. The authors say the occlusion information needs a TSDF or exact viewpoints and a large volume evaluation §4.2. The gain is about 2 points of accuracy, or 7 points of RAHP (89% versus 82%) §4.2.
• Split by view, not by object, in the main comparison. The 216k dataset is split by view, so the test objects are the training objects §4.2. The "novel object" evidence comes from the category experiment §4.4 and from the robot tests with objects not in the training set §5.2.
• RAHP is sensitive to the 99% threshold. The paper's single-instance example gives 27% recall at 99% precision for one shampoo bottle §4.1, which shows how a single operating point can vary between objects.
• Robot evidence is small. One robot, 45 rounds, 15 with passive sensors and 30 with the wrist sensor §5.2. The 93% and 84% figures come from different sensing setups and should not be read as a general success rate.
• Pretraining helps early, less later. The benefit of pretraining fades with more iterations §4.3.
• Point cloud, not TSDF. The method takes a point cloud. The authors say it would be easy to take a TSDF instead, but they do not test that §3.1.
• RGB is ignored. The representation uses depth and normals only §3.3.
• Hidden costs. Candidate sampling and the CNN run per candidate §3.2. The paper reports that the algorithm typically returns hundreds of grasp candidates in about 3 seconds §5.3. Filtering then needs IK checks and collision checks for each candidate §5.3.

Lineage

• Builds on: ten Pas and Platt's earlier geometry-based grasp sampling and representation, cited as [tenPas2015] and not linked here §2 §3.2. The representation here is compared with that paper's three-channel representation §4.2.
• Compared against: Herzog et al. 2012 and Kappler et al. 2015 (projection along the approach axis, RAHP 19%) §4.2; Lenz et al. 2015 and Pinto and Gupta 2016 (RGB-D grasp rectangles, cited as prior work) §2 §4.2.
• Uses: the BigBird dataset for real depth scans and meshes §3.4 §4.2; the 3DNET CAD models for simulated pretraining data §3.4 §4.3.