Robotic grasp generation

Overview

A grasp is a gripper pose plus an opening width. The pose has six degrees of freedom, and most of those poses collide with something in a cluttered scene. The problem is to find poses that close on the object, stay clear of its neighbours, and can be reached. The wiki has nine papers on it. Most of them work from a depth point cloud. Grasp Pose Detection and Contact-GraspNet both avoid fitting a model to a known object 1706.09911 §1 2103.14127 §1. The others change the input or the output. GraspLDM samples grasps from a learned latent space 2312.11243 §3. SparseGrasp works from a few RGB views through a Gaussian scene 2412.02140 §3. A wearable prototype uses fixed geometric rules and no learning 2411.08169 §II.C.

The papers differ in where candidates come from. Grasp Pose Detection draws hand poses on the visible surface, then scores each pose with a classifier 1706.09911 §4 §5. Its 2016 conference precursor, which shares the sample-then-classify design, is covered under the same variant below 1603.01564 §3.2. Contact-GraspNet asks, for every point in the scene, whether a grasp touching it would succeed, and predicts the grasp along with the answer 2103.14127 §3.1. In both, the candidate is anchored at a point the depth sensor already sees, which keeps the search away from unobserved space.

Variants

Seven method families are covered, plus one survey. The sources also name other families, such as template methods, learned three-degree-of-freedom detectors and kernel-density methods 1706.09911 §1.2, but no card exists for them yet.

Sample, then classify each candidate

Grasp Pose Detection samples a point from the cloud, builds a local frame from the surface normals around it, and places a two-finger hand over a grid of offsets and rotations. It slides each hand along its approach direction until it touches the cloud 1706.09911 §4. A candidate is kept only when the hand body is collision-free and the closing region holds at least one point 1706.09911 §4. Each survivor becomes a 15-channel 60 x 60 image of the points in the closing region, including estimated occluded space, and a four-layer CNN scores it 1706.09911 §5.1 §5. The authors train it on candidates labelled by a frictionless antipodal test against meshes of 55 BigBird objects, with 20 scanned views each 1706.09911 §6. The executed grasp is chosen by a cost over height, approach direction and arm travel 1706.09911 §8.5 eq. 1. Card: Grasp Pose Detection in Point Clouds.

The 2016 precursor, High precision grasp pose detection in dense clutter, uses the same two steps with a shorter description. It classifies each candidate with a four-layer CNN over 15 channels from three projections of the space between the fingers 1603.01564 §3.2 §3.3. Its labels are frictionless antipodal grasps, and it reports that occlusion channels and CAD pretraining raise recall at 99% precision 1603.01564 §4.2 §4.3. Its real-robot clutter result is 93% (20 failures of 288) with an active wrist sensor, against 73% for the authors' prior work 1603.01564 tab. 2 §6.

Per-point contact prediction on a scene point cloud

A network reads a point cloud of the scene and gives each point three outputs: a contact score, an approach direction and a width 2103.14127 §3.3. The approach and baseline vectors are made orthonormal with two Gram-Schmidt steps 2103.14127 eq. 6. A grasp is kept when its score passes a threshold, and farthest-point sampling spreads the kept contacts over the surface 2103.14127 §4.1. Training uses simulated grasps from ACRONYM, placed on ShapeNet meshes and rendered as point clouds 2103.14127 §3.2. Card: Contact-GraspNet.

Generative sampling in a learned latent space

GraspLDM trains a conditional VAE. A grasp encoder, conditioned on a point-cloud encoding, maps each grasp to a short latent, and a decoder maps it back to a pose 2312.11243 §3.1. A DDPM is then trained on the latents of the frozen VAE, which the authors describe as a prior that repairs the gap between the VAE prior and its posterior 2312.11243 §3.2. Sampling starts from noise in the latent space, denoises it, and decodes the result. The paper also reports DDIM with 100 steps as a drop-in sampler with no retraining 2312.11243 §4.3. For a new downstream task, only a small latent denoiser is trained, with a task label as an extra input 2312.11243 §4.2. Card: GraspLDM.

Language-queried grasps on a Gaussian scene

SparseGrasp fits a 3D Gaussian Splatting scene to three RGB views, initialised from dense DUSt3R points 2412.02140 §3.1 §4. Each Gaussian carries a semantic feature distilled from MaskCLIP region features after PCA compression to 16 dimensions 2412.02140 §3.2. When an object moves, the Gaussians that match the moved pixels are re-posed by render-and-compare, which the paper reports at about 200 ms 2412.02140 §3.3 tab. 2. A retrained GraspNet reads each Gaussian's centre, scale and rotation to propose grasps 2412.02140 §3.4. Card: SparseGrasp.

Geometric grasp points from a wearable depth camera

This variant has no learned model. A wrist-mounted depth camera and an IMU classify the scene as tabletop, doorknob or pot handle. The classification uses RANSAC plane detection, a plane-hull extrusion and clustering 2411.08169 §II.C. For each object, the grasp points are the two points nearest the minor principal axis on opposite sides 2411.08169 §II.C. The pot-handle mode uses a height test, and the authors list the RANSAC threshold and the cluster size as values the implementer must choose 2411.08169 §II.C. Card: Point Cloud Context Analysis for Rehabilitation Grasping Assistance.

Dexterous grasp as the start state for reinforcement learning

For a 16-DoF Allegro hand, this method first segments the functional part of the object, runs AnyGrasp on it, and maps each two-finger candidate to dexterous hand configurations 2412.15587 §3.1. Collision-free candidates are kept, and the one closest to the camera becomes the start state. A PPO policy then refines the arm and hand from there 2412.15587 §3.1 §3.4. The paper compares this with a fixed, human-designed start, and reports higher success in simulation on four articulated-object tasks 2412.15587 tab. 1. Card: Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge.

Human-hand grasp synthesis (not a gripper)

FastGrasp produces a MANO human hand mesh for an object point cloud. It uses a latent diffusion model over a hand-pose autoencoder, then an MLP that adds an offset trained with contact and penetration losses 2411.14786 §3.1 §3.3. It answers a different question from the gripper variants. Its output is a hand, so it fits hand-object animation or avatars, and it does not select a parallel-jaw pose. Card: FastGrasp.

Survey

A survey of grasping in virtual reality, set against robotic grasping, groups methods into physics-based, visual and data-driven families and reports no experiments of its own 2411.06244 §2.2 §4. Card: Grasping Object: Challenges and Innovations in Robotics and Virtual Reality.

When to use which

The rows below rest on the constraints each source reports. No paper compares all of them directly, so the table says what each one supports.

Two caveats on the figures. The success rates are author-reported and come from different robots, grippers, object sets, simulators and protocols. The Contact-GraspNet numbers are against 6-DoF GraspNet and CollisionNet baselines 2103.14127 tab. 1. The sample-then-classify numbers come from its own ablations 1706.09911 tab. 3. The paper's simulation ablations report their values only in plots, so this page does not quote them 2103.14127 fig. 5.

Implementation sketch

Language-agnostic pseudocode of the per-point variant's common core. The other variants have different steps. Their cards carry their own sketches, for example GraspLDM for latent sampling and SparseGrasp for the update step.

# Build one 6-DoF grasp from a contact point  [arxiv:2103.14127 §3.1]
# contact (3,): point on the observed cloud
# base, approach (3,): unit vectors from the network
# width: metres; d: gripper base distance (paper gives no value)
def build_grasp(contact, base, approach, width, d):
    # Translation: half the width along the baseline, plus d along
    # the approach  [arxiv:2103.14127 eq. 1]
    translation = contact + (width / 2) * base + d * approach
    # Rotation columns: base, approach x base, approach  [eq. 2]
    rotation = column_stack(base, cross(approach, base), approach)
    return pose(rotation, translation)

# Select grasps for execution  [arxiv:2103.14127 §4.1]
def select_grasps(points, scores, threshold=0.23):
    # Keep contacts whose score passes the threshold
    kept = points[scores > threshold]
    # Spread the choice over the surface with farthest point sampling
    chosen = farthest_point_sample(kept, n=150)
    # Execute the most confident grasp that is reachable and collision-free
    return sort_by_score(chosen)

The sign of the approach term differs between the paper and its code. See the Contradictions section.

Contradictions

No two papers report the same quantity, so none of their numbers conflict directly. The success rates and timings above are not comparable: the robots, grippers, object sets and trial protocols differ 1706.09911 §8.1 1706.09911 §8.6 2103.14127 §4.2 2103.14127 §4.3 2412.02140 §4 2411.14786 §4.2.

Within each source, paper and code disagree on points that matter for a reimplementation:

• Contact-GraspNet, the sign of the approach offset: the paper writes plus, and the code subtracts 2103.14127 eq. 1.
• Contact-GraspNet, the pose-loss normalisation: the paper divides by the number of positive contacts, and the code does not 2103.14127 eq. 8 contact_graspnet.py.
• Sample-then-classify, the grasp labels in the 2016 precursor: the paper labels frictionless antipodal grasps as positive, and the released configuration uses a 20-degree friction cone 1603.01564 §3.4 [code: cfg/generate_data.cfg#L54]. The two give different positive sets.
• Sample-then-classify, the normal of the local frame: the paper's text takes the smallest-eigenvalue axis as the normal, and the code takes the largest 1706.09911 §4 local_frame.cpp.
• Sample-then-classify, the score: the paper says softmax probability, and the code outputs a logit difference 1706.09911 §5 eigen_classifier.cpp.

Within a single paper, text and table disagree on some headline numbers:

• Geometric grasp points: the abstract says grasp points fall within 1 cm on average, while the body reports 5 to 6 mm for one object with a caliper ground truth 2411.08169 §III.B.
• GraspLDM: the text puts the partial-point-cloud gain over its VAE baseline at about 4.75%, and the table gives a difference of 3.75 points 2312.11243 §4.5 tab. 2.
• RL grasp prior: the text gives 92%, 87% and 87% for bucket, laptop and toilet, and the table gives 94.17, 90 and 87.88 for the same method 2412.15587 §4.1.3 tab. 1. The paper does not say which checkpoint each figure comes from.
• SparseGrasp: the paper says it is strong on novel views, but its table has the 17-view F3RM baseline higher on novel views, 0.75 against 0.71 2412.02140 §4.1 tab. 3. It names FastSAM for masks in one section and SAM ViT-H in another 2412.02140 §3.1 §4.

Papers

• High precision grasp pose detection in dense clutter: the 2016 precursor of grasp pose detection. It samples 6-DoF candidates from a point cloud and scores each with a 15-channel CNN. Author-reported 93% active-sensor success on a Baxter.
• Grasp Pose Detection in Point Clouds: samples two-finger 6-DoF hand candidates from a depth point cloud, encodes each as a 15-channel image, and scores it with a CNN. Author-reported 93% in dense clutter on a Baxter.
• Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes: a per-point contact network that predicts a collision-aware 6-DoF grasp from one depth point cloud, trained on simulated ACRONYM grasps.
• GraspLDM: Generative 6-DoF Grasp Synthesis using Latent Diffusion Models: a conditional VAE with a DDPM prior in its latent space, so a new task only trains a small denoiser. Author-reported 80% and 78.75% real-robot success on two arms.
• FastGrasp: Efficient Grasp Synthesis with Diffusion: a latent diffusion model and an adaptation MLP that produce a human hand mesh for an object point cloud in one pass. Human hand, not a gripper.
• SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images: a 3DGS scene from three RGB views with distilled semantic features, updated by render-and-compare after objects move. Author-reported 55 of 70 grasps.
• Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge: an AnyGrasp-based dexterous grasp as the start state for PPO on an Allegro hand. Author-reported gains over DexArt in simulation.
• Point Cloud Context Analysis for Rehabilitation Grasping Assistance: a wrist-mounted depth camera and IMU classify the scene and pick two opposing grasp points with geometric rules. No learning and no baseline.
• Grasping Object: Challenges and Innovations in Robotics and Virtual Reality: a survey of grasping in VR and robotics. No method and no experiments.