Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

In plain words

To pick up an object, a robot needs to know where to put its gripper and how wide to open it. In a messy scene with many objects, that is hard, because the gripper can hit neighbours, and the possible gripper poses form a huge space. Older pipelines first found the object, then guessed grasps on that object alone, then threw out the ones that would collide. Each step could fail.

This paper starts from the camera's point cloud of the whole scene. Every point on a visible surface is a candidate place to close the fingers. The network asks two questions for each point: "would a grasp touching me succeed?" and "if so, which way should the gripper approach, and how wide should it open?" Because the gripper is anchored to a point the camera actually sees, the network only has to learn a direction and a width, not a free-floating pose. Training uses many simulated grasps, and the authors report that it transfers to real depth sensors and to objects it has never seen.

TL;DR

Contact-GraspNet is a point-cloud network that predicts a grasp for every point in a scene. Each grasp is a contact point, a rotation and a width, so the grasp space shrinks from full 6-DoF to a per-point 4-DoF problem §3.1. It is trained on about 17.7 million simulated grasps from the ACRONYM dataset §3.2. On a Franka robot the authors report 90.20% success on 51 unseen objects in cluttered scenes, against 80.39% for their closest baseline tab. 1. Use it if you need collision-aware 6-DoF grasps from a single depth view. Expect to tune its confidence thresholds yourself.

Problem

Grasping unknown objects in clutter needs 6-DoF gripper poses and collision avoidance. Multi-stage methods depend on accurate instance masks, reason about the target object alone, and sample grasps in the unconstrained 6-DoF space, which is hard to learn and slow to search §1. Planar grasp methods avoid that space but assume a near-perpendicular camera §1, §2. Predicting approach directions per point is ambiguous at curved surfaces such as mug rims and handles, and multiple approach directions can be valid for one contact §2. The paper also states that a grasp with no visible contact often fails to keep the object's pose, so it targets grasps with surface contact on the visible side §3.1.

Method

Grasp representation §3.1. Each candidate grasp is anchored at a contact point c on the observed cloud. The grasp is defined by an approach vector a, a baseline vector b (both unit length), a width w and a constant distance d from the gripper baseline to the gripper base. The translation and rotation are given by eq. 1 and eq. 2. Because the contact lies on the observed surface, the network predicts only the rotation and the width, which is 3 + 1 degrees of freedom instead of 6 §3.1. The paper does not give a numerical value for d §3.1; the code uses 0.1034 m [code: contact_graspnet/contact_graspnet.py#L193].

Training data §3.2. The authors use the ACRONYM dataset, which has dense simulated grasps on ShapeNet meshes §3.2. Meshes are placed at random stable poses in table-top scenes §3.5. Grasps that collide with the gripper model are dropped, and the rest are mapped to their contact points on the mesh fig. 2. During training a virtual camera renders a point cloud of the scene. A recorded point is positive if a valid mesh contact lies within 5 mm of it eq. 3. Each positive point is assigned the grasp of its nearest mesh contact eq. 4, eq. 5.

Network §3.3. The input is 20,000 points. A PointNet++-style U-shaped network with set abstraction and feature propagation layers §3.3 predicts on 2048 points. Four heads, each with two 1D convolutions, output per point: a contact score s, two 3-vectors z1 and z2, and 10 width-bin logits o §3.3. Two Gram-Schmidt steps make the outputs orthonormal. b is z1 normalised, and a is z2 with its b-component removed, then normalised eq. 6. The predicted width is the centre of the bin with the highest confidence, and the 10 bins are meant to counter data imbalance §3.3.

# Build one 6-DoF grasp from a contact point  [§3.1] [eq. 1] [eq. 2]
# contact (3,) on the observed cloud; base, approach: unit (3,)
# width: scalar in metres; output: 4x4 pose in camera frame
def build_grasp(contact, base, approach, width):
    # Translation: half the width along the baseline, then d along
    # the approach direction  [eq. 1]
    translation = contact + (width / 2) * base + d * approach
    # Rotation columns are base, approach x base, approach  [eq. 2]
    second_column = cross(approach, base)
    rotation = column_stack(base, second_column, approach)
    return pose(rotation, translation)

Losses §3.4. Only the top 512 per-point contact-score errors are trained with binary cross-entropy §3.4. The pose and width terms are only applied at positive points. The main pose term is ADD-S, a symmetric point-distance loss over five points on the gripper model fig. 3 eq. 8. The predicted pose at each positive point is compared with the nearest ground-truth pose, weighted by the predicted contact score. The width loss is a weighted multi-label binary cross-entropy over the 10 bins, with weights inversely related to bin size §3.4. The total loss is the weighted sum with α = 1, β = 10, γ = 1 §3.4. The paper's total-loss formula does not list the direction and approach cosine terms that the code also uses; see Paper vs code.

# Training loss for one scene  [§3.4] [eq. 8]
# conf (N,): predicted contact scores; is_contact (N,): 0 or 1
# pred_poses, gt_poses: per-point grasp poses (N, 4, 4)
def scene_loss(conf, is_contact, pred_poses, gt_poses, width_logits):
    # Contact score: BCE, keeping only the 512 largest errors  [§3.4]
    errors = binary_cross_entropy(conf, is_contact)
    contact_loss = mean(top_k(errors, k=512))
    # Pose and width terms only at positive contacts  [§3.4]
    positives = indices where is_contact == 1
    n_positive = number of positives
    # Symmetric ADD-S over five gripper points, weighted by confidence
    # and normalised by the number of positives  [§3.4] [eq. 8]
    pose_loss = sum over i in positives of
        conf[i] * min over ground-truth u of
        distance(gripper_points(pred_poses[i]),
                 gripper_points(gt_poses[u]))
    pose_loss = pose_loss / n_positive
    # Width: weighted multi-label BCE over the 10 bins  [§3.4]
    width_loss = weighted_binary_cross_entropy(width_logits, width_bins)
    # Weights alpha = 1, beta = 10, gamma = 1  [§3.4]
    return 1.0 * contact_loss + 10.0 * pose_loss + 1.0 * width_loss

Inference and selection §4.1. A full scene, or a local cube around a target object, is passed through the network. Grasps are kept if their contact score exceeds 0.23. Farthest point sampling over those contacts spreads the choice across the surface. If too few grasps remain for an object, the paper lowers the threshold to 0.19 §4.1. The executed grasp is the most confident one that is kinematically reachable and does not collide with the scene §4.1.

Hyperparameters

Results

Author-reported. Setup: a Franka Panda robot with a parallel-jaw gripper and an Intel RealSense L515 depth camera. Nine cluttered scenes with 51 unseen objects, replicated from an earlier paper §4.3. Baselines are 6-DoF GraspNet and its combination with CollisionNet §4.3 tab. 1. Each object gets at most two attempts without rearrangement §4.2.

The paper does not define the #Attempts column beyond its header. The text discusses trials per object §4.2, so read the column as a count of trials. The success figures are consistent with 51 objects: 90.20% is 46 of 51, and 84.31% is 43 of 51 (derived).

Simulation (FleX) ablations plot success rate against coverage §4.2 §4.4. Their values appear only in plots, not in the text: removing the weighted width bins hurts success and coverage fig. 5. The pose term helps at high confidence fig. 5. Local regions improve coverage at similar success, and training on the smaller 110-object dataset does not generalise to new categories fig. 6. Numeric values: not stated.

Run time §4.1: 0.28 s for a full scene, about 0.19 s for a local region. The hardware for this measurement is not stated.

Paper vs code

Code: in the official repository NVlabs/contact_graspnet at commit 721706067cc7f7889c32afe7b4f7955eb90bd866. The repository README links this paper, which is how we know it is the authors' code.

• Translation sign: the paper writes + d · a eq. 1 §3.1; the code uses - gripper_depth · approach contact_graspnet.py and the same in the NumPy path contact_graspnet.py. Whether the code's approach vector points the other way is not checked here.
• Total loss: the paper lists the contact, pose and width terms only §3.4. The code also adds a direction cosine term and an approach cosine term, each with weight 1 tf_train_ops.py.
• Gripper-to-grasp ADD-S term: the code has a second pose term (ground-truth to prediction) that the paper does not describe. It is disabled in the shipped config config.yaml tf_train_ops.py.
• Pose-term normalisation: the paper divides by the number of positive contacts n+ §3.4 eq. 8. The code takes the mean over all points in the batch, multiplies by a factor that is 1 when a scene has any positive, and does not use n+ contact_graspnet.py.
• Distance norm: the code computes the sum of squared differences over the five points and takes the square root once, i.e. a norm over all 15 coordinates contact_graspnet.py. The paper's ||·||₂ is not specified further eq. 8.
• Learning rate: the paper decays from 0.001 to 0.0001 §3.5. The code uses exponential decay with factor 0.7 every 200,000 samples and a floor of 0.00001 tf_train_ops.py config.yaml.
• Training length: the paper gives 144,000 iterations at batch size 3 §3.5. The code gives max_epoch 16 config.yaml. The two are not converted in either source.
• Width prediction: the paper uses the centre of the most confident bin §3.3. The code takes the single argmax bin, uses bin midpoints, sets the last bin to its upper edge, and caps values at 0.075 m (0.08 minus the 5 mm extra opening) contact_graspnet.py config.yaml. The inference code also computes an opening with the extra 5 mm added contact_grasp_estimator.py; I did not trace where that value is used.
• Second threshold: the paper says "reduce the confidence threshold to 0.19" when there are too few grasps §4.1. The code keeps the 0.19 threshold as a fill-up threshold inside the same selection call contact_grasp_estimator.py contact_grasp_estimator.py.
• Width-bin weights: the paper says weights are anti-proportional to bin size §3.4, but gives no values. The code uses fixed constants config.yaml.
• Bin edges: the paper says the 10 bins are equidistant §3.3. The code's edges step by about 0.0079 and end at 0.08 config.yaml.
• Dropout and batch-norm schedule: not stated in the paper. The code sets them contact_graspnet.py tf_train_ops.py.
• Depth noise: the paper reports that training with Gaussian noise helps generalisation fig. 6 but gives no sigma. The code uses sigma 0.001 with clip 0.005 config.yaml.

Pitfalls

• The distance d is not given in the paper. It fixes where the gripper base sits relative to the contact. The code uses 0.1034 m for the Panda gripper contact_graspnet.py. Using another value shifts every grasp along its approach axis.
• Sign convention of the approach term. Eq. 1 says +d·a, the code says -d·a. If the sign is not resolved, the two translations differ by 2d along the approach axis, which is about 0.21 m with the code's d (derived). Check the approach-vector convention in the code before using either form.
• Confidence thresholds are model-specific. 0.23 and 0.19 come from this network's calibration §4.1. The pose term is weighted by confidence §3.4, which ties the score to pose quality. Do not reuse the thresholds with another model without re-calibrating.
• Numerically sensitive pose term. The pose term takes a square root of a sum of squared distances eq. 8. The derivative of a square root is unbounded at zero, and the code adds no epsilon before it contact_graspnet.py. A reimplementation that uses a different order of operations may behave differently near zero distance (derived from the code; the paper does not discuss it).
• Only one table, no algorithm box. The pseudocode above is ours, written from the equations and text. The paper does not give a step-by-step algorithm.
• Equation numbers are counted, not read from the PDF. We numbered the equations by counting display environments in the source (eq. 1 to 8). The PDF numbering should match, but if a citation disagrees, trust the PDF.
• Figure numbers follow source order. Fig. 1 to Fig. 7 are counted in the source; float placement in the PDF could change the order.
• The paper's grasp-filtering subsection is commented out in the LaTeX source §3. It is not in the compiled paper, so this card does not describe it.
• Real-world evidence is narrow. One robot, one depth camera, nine scenes and 51 objects §4.3. The 90% figure is an author-reported success rate on that setup, not a general benchmark result.
• Simulation metric is conservative. A grasp in simulation counts only if the gripper does not collide and the object stays in the hand after a shake §4.2. Real grasps that touch slightly and survive would count as failures.
• Cost. Training took about 40 h on one V100 §3.5. Inference runs in about 0.2 to 0.3 s per scene §4.1, with the hardware not stated.
• The paper's "failure rate halved" claim is in the abstract [abstract]. By the table, failures fall from 19.6% to 9.8% (derived).

Lineage

• Builds on: PointNet++ set abstraction and feature propagation layers §3.3, the ACRONYM grasp dataset §3.2, and Gram-Schmidt orthonormalisation for rotation outputs, following Zhou et al. §3.3.
• Uses for segmentation: the pipeline segments unknown objects with Xiang et al. (2020) fig. 4. The method does not depend on accurate masks §1 fig. 7.
• Compared against: 6-DoF GraspNet (Mousavian et al.) and the CollisionNet pipeline of Murali et al. §1 tab. 1.
• Superseded by: not stated in this wiki.