Grasp Pose Detection in Point Clouds

In plain words

Picking things up from a cluttered pile is hard for a robot. Older methods first work out what each object is and where it sits, then fit a known 3D model to it. This paper skips that step. It looks at a 3D point cloud from a depth sensor, proposes many ways to close a two-finger gripper on the visible surfaces, and scores each proposal with a small neural network. The robot does not need a model of the object or need to know what the object is.

Each proposal is turned into a small picture-like encoding of the surfaces the fingers would touch and of the space behind them, so the network can judge it. The network is trained on depth scans of real objects, labelled with whether each proposal would have held the object.

The authors report about 93% grasp success on a real robot when it clears piles of household objects it was not trained on. They also show which parts matter most: how the sensor view is built, how candidates are chosen and ranked, and how the network is pretrained. All numbers are the authors' own.

TL;DR

Grasp Pose Detection (GPD) samples many 6-DOF hand poses from a point cloud, encodes each one as a 15-channel 60x60 grasp image, and scores it with a four-layer CNN §3 §5.1 §5. The CNN is trained on about 1.5M labelled candidates from 55 BigBird objects, scanned from 20 views each §6. The paper ranks candidates with a cost function over height, approach and arm travel, and reports 93% grasp success over 288 attempts in dense clutter tab. 3. Without the selection heuristics the rate falls to 75%, and without classification to 53% tab. 3. All results are author-reported.

Problem

The task is grasp pose detection. Given a viewpoint cloud and a region of interest, find one or more 6-DOF hand poses that form a force-closure grasp of some object §2 [Problem 1]. The paper restricts itself to two-finger hands, where the hand pose alone determines the contact points §2 [Definition 1].

Grasp detection avoids two costly steps of the usual pipeline: object segmentation and registering a CAD model to a noisy cloud §1 §10. The paper argues that this makes the method usable where no CAD model exists §1. The cost is that the method does not know object identity, so object-specific use needs a separate detector §9.

Many earlier methods reach 75% to 95% grasp success on novel objects in isolation or light clutter, and the paper argues that is too low for practical use §1.

Method

The method has two parts: a sampler that proposes hand poses, and a classifier that scores them §3.

1. Pipeline alg. 1 §3.

1. Preprocess the cloud by voxelizing, removing outliers and similar steps. The paper gives no numbers for this step §3.
2. Define a region of interest. It can be the whole scene, several objects, or a rough object location §3.
3. Sample N candidate hand poses, several thousand in the paper §3 §4 alg. 2.
4. Encode each candidate as a multi-channel image §5.1.
5. Score each candidate with a four-layer CNN §5.
6. Select a grasp to execute from the scores and other considerations §3 §8.5 eq. 1.

2. Sampling candidates §4 alg. 2. A candidate is a 6-DOF hand pose. The sampler wants candidates on all visible graspable surfaces, not only on the centres of convex-hull faces §4.1.

1. Draw a point p from the cloud in the region of interest, uniformly at random §4.
2. Build a local frame at p from the surface normals in a small ball around p. The frame is a Darboux-style frame whose third axis is the surface normal, and it is flipped to point outward §4.
3. On a grid over an offset along one frame axis and a rotation about another, slide the hand along its approach direction until a finger or the hand base touches the cloud §4.
4. Keep the hand only if its body is collision-free (Condition 1) and its closing region holds at least one cloud point (Condition 2) §4.

The paper uses |Y| = 10 offsets and |Φ| = 8 rotations in its experiments §4. Figure 2 shows the frame with the gripper at its origin fig. 2. Figure 3 shows sampled candidates on a partial soup-can cloud fig. 3. Figure 4 compares this grid with a version that rotates about the other frame axis, and reports at least three times as many true positives for two-finger hands on two BigBird objects fig. 4 §4.1.

3. Grasp image §5.1. For each candidate, the points near the hand are expressed in the hand frame and cropped to a box around the closing region. Each point cloud point and its normal is binned into a 2-D grid for each of three orthogonal projections. The occluded space is also estimated, by sampling points behind observed points along the camera rays. This gives 5 channels per projection: the three components of the averaged normal, the averaged occupied depth I_o, and the occluded (unobserved) depth I_u. Together they form 15 channels §5.1 fig. 5.

4. Classifier §5. The CNN has the LeNet layout from Caffe: two convolution and max-pool stages, one inner-product layer with ReLU, and an output layer with two classes and a softmax §5. Training uses stochastic gradient descent with learning rate 0.00025 §5. The paper does not state a batch size for GPD training.

5. Training data §6. Each candidate is labelled by checking whether it is a frictionless antipodal grasp on a registered mesh of the object §6.1. The BigBird objects provide meshes paired with real depth scans, 20 registered clouds per object from two viewpoints 53 degrees apart §6.

6. Selection §8.5 eq. 1. The paper ranks candidates by a cost J(h). It encodes three preferences: grasps at the top of the pile, an approach direction tied to gravity, and short arm travel from a nominal configuration. The formula is:

J(h) = l\left(0.5(\hat{a}(h)^T \hat{g} + 1)\right)\, l\left(\frac{\nabla q(h)}{\nabla q_{m}}\right)\, l\left(\frac{z_{m} - z(h)}{10\, z_{m}}\right) eq. 1

Here l(x) = max(1 - x, 0), a(h) is the approach direction, g the gravity direction, z(h) the hand height, and ∇q(h) the configuration-space distance from a nominal start. The highest-scoring grasp is executed §8.5 eq. 1. Removing this selection drops grasp success from 93% to 75% tab. 3.

Pseudocode. The following is the paper's pipeline, written in our words alg. 1 §3.

# Grasp pose detection pipeline  [alg. 1] [§3]
# Input:  point cloud, region of interest, hand geometry, N samples
# Output: grasp candidates, ranked by score

def detect_grasps(cloud, region, hand, num_samples):
    # 1. Denoise and voxelize the cloud (no numbers given)  [§3]
    cloud = preprocess(cloud)
    # 2. Draw sample points in the region of interest  [§4] [alg. 2]
    samples = sample_points(cloud, region, num_samples)
    # 3. Place hand candidates around each sample  [§4]
    hands = []
    for sample in samples:
        hands += place_hands(cloud, sample, hand)
    # 4. Encode each candidate as a 15-channel image  [§5.1]
    images = [grasp_image(cloud, h) for h in hands]
    # 5. Score each candidate with the CNN  [§5]
    scores = [cnn_score(image) for image in images]
    # 6. Rank by score; the paper also applies the cost of eq. 1  [§8.5]
    return rank(hands, scores)

The following is the per-sample candidate search alg. 2 §4.

# Hand placements around one sample point  [alg. 2] [§4]
# Input:  cloud, sample point, hand geometry
# Output: candidate hands, at most one per orientation

def place_hands(cloud, sample, hand):
    # Local frame from the normals in a ball around the sample  [§4]
    frame = local_frame(cloud, sample, radius=r)
    hands = []
    # Rotation angles: eight values in [-pi/2, pi/2)  [§4]
    for angle in orientations:
        # Express the nearby points in the hand frame  [§4]
        points = to_hand_frame(cloud, sample, frame, angle)
        # Keep points inside the hand height band  [§4]
        points = crop_to_hand_height(points, hand.height)
        # Find feasible finger placements at the initial bite  [§4]
        placement = middle_feasible_placement(points, hand)
        # Push the hand deeper while the fingers stay clear  [§4]
        placement = deepen(points, placement, hand.depth)
        # Keep the hand only if the closing region has a point  [§4]
        if closing_region_has_points(points, placement, hand):
            hands.append(make_hand(sample, frame, angle, placement))
    return hands

Grasp image pseudocode §5.1.

# 15-channel grasp image: three projections, five channels each  [§5.1]
# Input:  points and normals in the hand frame; occluded points
# Output: (60, 60, 15) image, values scaled to 0-255

def grasp_image(points, normals, occluded_points):
    channels = []
    # Three orthogonal projections of the hand frame  [§5.1]
    for projection in three_projections:
        # Per-cell average of the three normal components  [§5.1]
        channels += normal_channels(points, normals, projection)
        # Per-cell average depth of observed points (I_o)  [§5.1]
        channels += occupied_depth(points, projection)
        # Per-cell depth of occluded points (I_u)  [§5.1]
        channels += occluded_depth(occluded_points, projection)
    return stack(channels)  # (60, 60, 15)

Note: this is a condensed form of the paper's description. The image size M and the rule that maps the closing region into the unit square are not numerically stated §5.1. Code values are in the Hyperparameters table.

Hyperparameters

Values marked "not stated" are missing from the paper. Code values come from the official repository at code_commit, and are shown for reference only.

Results

Author-reported. Setup: a Baxter Research Robot with a parallel-jaw gripper §8.2. Each round puts 10 household objects, chosen from 27 that differ from the 55 training objects, into a box, shakes it, and pours it into a tray §8.1 tab. 1 fig. 11. The round ends when the tray is empty, no grasp is found in three attempts, or the same failure repeats three times tab. 1. The CNN is trained on all 55 BigBird objects with the 3-channel representation §8.6.

Dense clutter grasp results tab. 3 §8.6.

The paper also reports 723 dense clutter grasp trials in total, with 288 in the best setting §10.

Classification and representations §7.2 fig. 8.

Pretraining on simulated data §7.3 fig. 9.

Prior object knowledge §7.4 fig. 10. Accuracy and recall at 99% precision (RA99P), by training set:

Runtime §8.4 tab. 2. Times for 1k candidates, averaged over ten cluttered scenes on an Intel Core i7-4770K with a GTX 970:

The paper says a full grasp takes one to eight seconds §8.4.

Combined grasp and object detection §9 fig. 14 tab. 4. A 99% test accuracy is reported for single-object detection with 11 objects §9. In nine cluttered scenes with 1197 grasp-derived proposals, 1023 were classified correctly, 85.4% §9 tab. 4.

Paper vs code

The official repository is atenpas/gpd, linked from the paper's bibliography, and is copyright Andreas ten Pas under BSD 2-Clause LICENSE.md. The checkout is 6327f20. The repository's changelog says version 2.0.0 removes the Caffe dependency CHANGELOG.md, so the C++ classifier re-implements the same layers and reads raw weight files eigen_classifier.cpp. The paper's Caffe files remain in models/caffe lenet_15_channels.prototxt. Differences below were checked by reading the code.

• Candidates per sample. Paper: one candidate per grid cell, so 10 x 8 = 80 per sample §4. Code: one candidate per orientation and per axis, with one finger placement chosen (the middle feasible one, deepened). With the default hand_axes = 2, that is 8 per sample hand_set.cpp hand_set.cpp eigen_params.cfg.
• Push to contact. Paper: push until a collision, x* = min x with no collision §4. Code: start at the initial bite of 0.01 m and deepen in 0.005 m steps while the middle finger pair stays clear, up to the hand depth finger_hand.cpp.
• Frame axes. Paper: v1 is the largest eigenvalue direction, v3 the smallest, and v3 is the normal §4. The scatter matrix of the normals has the normal at its largest eigenvalue, so the paper's text is internally inconsistent. Code takes the largest eigenvalue direction as the normal and the smallest as the curvature axis local_frame.cpp.
• Rotation axis. Paper: rotation about the z axis of F(p) §4. Code: rotation about the hand-frame axis selected by hand_axes, which is the curvature axis for the default value 2 hand_set.cpp hand_set.cpp eigen_params.cfg.
• Neighbourhood radii. Paper: the radius r is not stated §4. Code uses 0.01 m for frame estimation hand_search.cpp and a point-retrieval radius of max(outer diameter - finger width, depth, height/2), which is 0.11 m hand_search.cpp.
• Image scale. Paper: the closing region is scaled into a unit cube §5.1. Code: fixed physical bounds from the image geometry image_strategy.cpp. The window width is 0.10 m image_geometry_15channels.cfg, narrower than the 0.12 m hand outer diameter hand_geometry.cfg. The crop uses the image width image_strategy.cpp.
• Voxels. Paper: points are voxelized into an M x M x M grid §5.1. Code: no 3D voxel grid. Points are binned directly into 60 x 60 cells per projection image_strategy.cpp.
• Normal channels. Paper: averaged unit normals, scaled so [1, m] covers all directions §5.1. Code: absolute values of each normal component, a running update with step 1/‖v‖ (not a count-based mean), and min-max normalisation to 0 to 255 image_strategy.cpp.
• Depth channel. Code stores 1 minus the average coordinate per cell, then min-max normalises the image image_strategy.cpp. Occluded depth is the maximum minus the average, so closer points get larger values image_strategy.cpp.
• Channel order. Paper: I_o, I_u, I_n §5.1. Code: normals (3), then occupied depth, then occluded depth, per projection image_15_channels_strategy.cpp. Pretrained weights depend on this order.
• Score. Paper: softmax probability §5. Code: logit of class 1 minus logit of class 0, not a probability eigen_classifier.cpp. The pruning threshold min_score is read as an integer, so fractional values are truncated grasp_detector.cpp.
• Selection. Paper: cost J of eq. 1 §8.5. Code: no cost function is in this repository. It keeps the num_selected highest scores, default 100 grasp_detector.cpp grasp_detector.cpp.
• Aperture limit. Code's default max aperture is 0.085 m grasp_detector.cpp. The ROS config sets 0.85 ros_eigen_params.cfg, which removes the width filter.
• Labels. Paper: frictionless antipodal, with 1 mm vertex error on the mesh §6.1. Code: contact band of 3 mm hand_search.cpp, friction cone of 20 degrees antipodal.cpp eigen_params.cfg, and at least 6 contact points on each side antipodal.cpp. Ground-truth labels come from the mesh with its normals reversed by default data_generator.cpp data_generator.cpp grasp_detector.cpp. I did not find the 1 mm vertex perturbation in the code paths I read.
• Dataset config. Paper: 20 clouds per object, split by view, with no split count given for the 1.5M set §6. Code: 20 views per object, with 5 held out for test generate_data.cfg generate_data.cfg. The data-generation config sets image_num_channels = 12 generate_data.cfg, not the 15 channels that the paper uses for its main results §7.2. The 400 samples per view generate_data.cfg and the 100 to 500 grasps per view generate_data.cfg are not in the paper.
• Training. Paper: learning rate 0.00025 §5. Code: Caffe solver at 0.01 with the inv policy lenet_solver_15_channels.prototxt and the solver's training path points to the author's machine. The PyTorch example uses 1e-5 for one epoch train_net.py train_net.py train_net.py.
• ReLU after conv. The PyTorch network applies ReLU after both convolutions network.py. The Caffe prototxt and the C++ forward pass do not eigen_classifier.cpp.
• Object detector. The 120 x 120 proposal step (§9) is not in the source tree. A recursive search of src/ and include/ for proposal and edge-box code found nothing.
• Shipped configs. Two configs choose different sample counts (30 and 500) and different selection counts (5 and 100) eigen_params.cfg ros_eigen_params.cfg. Set the value you need explicitly.

Pitfalls

• Reproduce the clutter numbers with the 3-channel model. Section 8 says the robot experiments used the 3-channel representation §8 §8.6. The repository's default config is 15-channel eigen_params.cfg. A 3-channel model directory exists params, but I did not check whether its weights are the paper's clutter weights.
• Eight candidates per sample, not 80. The paper's grid suggests 80 per sample. The code keeps one placement per orientation. Copying the paper's grid without the placement rule gives a different candidate set §4 hand_set.cpp.
• Frame normal. Follow the code's choice of the largest eigenvalue for the normal. Taking the smallest, as the paper's text says, gives a surface-tangent axis, not a normal §4 local_frame.cpp.
• Hand geometry is not in the paper. The paper gives a 3 cm to 7 cm width range and a Baxter gripper, but no hand dimensions §8.2 §8.5. The repository's values are for a generic hand, so they will not match the Baxter. The image window is narrower than the hand image_geometry_15channels.cfg hand_geometry.cfg.
• Logit score. The score is a logit difference, so the usual threshold of 0.5 does not apply, and min_score must be an integer eigen_classifier.cpp grasp_detector.cpp.
• Cost function is incomplete. Eq. 1 is the only numbered equation, but ∇q_m is described only as "the diameter" and the nominal configuration is not given eq. 1 §8.5. The text does not say whether â and ĝ point the same way, so the sign of the approach term is unclear. The paper says J is maximised §8.5. The repository has no implementation to check against.
• Two different gains. The paper says no selection gives "18% more grasp failures" §8.5, while Table 3 gives 93% against 75%, a gap of 18 points in success tab. 3. Use the table.
• Ablation samples differ. The active run has 30 rounds, and the other three have 15 each, so the totals in Table 3 are not directly comparable tab. 3 §8.6.
• Manual intervention. Two objects that land upside down are reset by hand during pouring §8.6. This is part of the protocol the paper reports.
• Label tolerance. Labels depend on the 1 mm vertex error §6.1. The code uses 3 mm and a friction cone, not a frictionless test hand_search.cpp eigen_params.cfg. Expect a different label distribution if you relabel.
• Occlusion costs time. The occluded channels roughly double the encoding time, for about 2% accuracy §7.2. Use the 12-channel or 3-channel representation if latency matters §7.2 §8.4.
• Runtime scales linearly. Runtime grows roughly linearly with candidate count, so more grasps means more time §8.4.
• Generalisation limits. Results are for 27 household objects on one robot in one lab. Two were excluded from object detection §9. Object-specific accuracy in clutter is 85% §9 §10.1.
• Double grasps. The method can detect a grasp across two touching objects as one object, and the paper suggests segmentation as a fix §10.1 fig. 16.

Lineage

• Builds on: the sampling strategy and 3-channel representation of ten Pas and Platt (ISRR 2015), and the descriptor, category-knowledge and RAHP metric of Gualtieri et al. (IROS 2016) §1.1.
• Compared against: template methods of Herzog et al. and Kappler et al., which use one-dimensional rotation searches and a convex hull §4.1 §7.2; learned 3-DOF detectors such as Lenz et al., Redmon and Angelova, and Pinto and Gupta §1.2; kernel-density methods of Detry et al., Kroemer et al. and Kopicki et al. §1.2.
• Superseded by: none in this wiki.