FastGrasp: Efficient Grasp Synthesis with Diffusion

In plain words

Making a robot or an avatar hold an object with a realistic human hand is hard. Earlier methods first guess helper maps, such as where the hand should touch, and then run a slow search that bends a hand model until it fits. That search is what makes them slow.

FastGrasp skips both steps. A diffusion model, the same family of model that generates images from noise, produces a compact code for a hand pose. It is guided by the shape of the object, given as a cloud of points. A second small network then adjusts that code so the hand presses on the object without sinking into it. One pass through the network gives a finished hand.

Because the diffusion model starts from random noise, repeated runs give different grasps for the same object. That is the diversity the method aims for, and the speed comes from avoiding the search.

TL;DR

FastGrasp turns an object point cloud into a human hand mesh in a single forward pass after training. It uses a latent diffusion model over a learned hand latent, conditioned on the object, plus an MLP adaptation module that adds an offset trained with contact, penetration and consistency losses. Author-reported: on out-of-domain HO-3D (trained on GRAB) it has the lowest penetration volume in its table (5.23; the next lowest is GraspTTA at 7.37) and 0.14 s inference on an NVIDIA A40.

Problem

Synthesising a grasp needs a physically plausible contact (no interpenetration, real touch) and a natural hand pose. Two-stage methods first generate intermediate maps such as contact maps, then run an iterative optimisation to fit a hand mesh. The paper argues that this optimisation is computationally expensive and that the final pose depends on the accuracy of the first stage §1. Training a diffusion model to learn the physical constraints directly was observed to oscillate and not converge, which motivates the two-step training §3.3.

Method

Overview: a one-stage pipeline. A hand is encoded to a 768-dimensional latent by an autoencoder. A latent diffusion model, conditioned on the object point cloud, learns the distribution of these latents. An adaptation MLP, trained after the diffusion model is frozen, adds an offset to the sampled latent. A decoder maps the sum to 61 MANO parameters, and the differentiable MANO layer produces 778 hand vertices §3.1 §3.2 eq. 10 eq. 11.

Training has two steps §3.1. Step one trains the latent diffusion model, which needs a trained autoencoder first §3.2. Step two freezes the autoencoder and the diffusion model and trains only the adaptation module §3.3. The pseudocode below labels the autoencoder stage 1a and the diffusion stage 1b; the paper does not name them separately. The released code runs the diffusion model in its own script train_diffusion.py.

Autoencoder. The hand input is 778 vertices, passed through a PointNet encoder and an MLP to a 768-dimensional latent. The decoder outputs 61 MANO parameters rather than vertices, which the paper says regularises the decoder. The reconstruction loss is a weighted sum of MSE on the parameters and Chamfer distance on the vertices §3.2 eq. 1 eq. 2 eq. 3. Three physical losses from GraspTTA are added: a consistency loss on the contact region, a contact loss, and a penetration loss §3.2 eq. 4 eq. 5 eq. 6. The paper defers their details to its supplementary material, which does not define them [supp.]. The total loss is given as one weighted sum §3.2 eq. 7.

Latent diffusion. The latent is viewed as 256 tokens of three values each. The denoiser predicts the noise added to the latent, conditioned on a PointNet embedding of the object's 3000 sampled surface points §3.2 eq. 8. The paper calls the denoiser a U-Net; the code uses a PointNet++-style set network diffusion.json.

Adaptation module. After the diffusion model is trained, an MLP f_adapt takes the diffusion output z1 and returns an offset z2 §3.3 eq. 9. The decoder receives z1 plus z2. The adaptation module is trained with the same loss as the autoencoder, so the decoded hand is judged physically §3.3 eq. 10 eq. 11.

Inference. Gaussian noise and the object points go through an N-step denoising process, which the paper says uses DDIM §3.4. The result is z1. The adaptation module gives z2, the decoder maps z1 + z2 to MANO parameters, and the MANO layer gives the mesh §3.4 eq. 9 eq. 10 eq. 11.

Pseudocode, inference:

# FastGrasp inference  [§3.4] [eq. 9] [eq. 10] [eq. 11]
# Input:  object points (B, 3000, 4)
# Output: hand vertices (B, 778, 3)

def generate_grasp(object_points, ddim_steps):
    # 1. Sample a latent prior, conditioned on the object  [§3.4]
    latent_prior = sample_latent(object_points, ddim_steps)  # (B, 768)

    # 2. Add the adaptation offset; the MLP also sees the object  [eq. 9]
    offset = adaptation_mlp(latent_prior, object_points)     # (B, 768)

    # 3. Decode the sum to MANO parameters  [eq. 10]
    mano_params = decoder(latent_prior + offset)             # (B, 61)

    # 4. Run the MANO layer to get the hand mesh  [eq. 11]
    hand_vertices = mano_layer(mano_params)                  # (B, 778, 3)
    return hand_vertices

Pseudocode, stage 1a, autoencoder loss:

# Stage 1a: hand autoencoder  [§3.2] [eq. 1]-[eq. 7]
# Input:  hand vertices (B, 778, 3), object points (B, 3000, 4), GT params
# Output: scalar loss

def autoencoder_loss(hand_vertices, hand_params, object_points):
    # 1. Encode the hand: PointNet, then an MLP to a 768-d latent  [§3.2]
    latent = encoder(hand_pointnet(hand_vertices))           # (B, 768)

    # 2. Decode to 61 MANO parameters  [§3.2]
    pred_params = decoder(latent)                            # (B, 61)

    # 3. Run the MANO layer to get predicted vertices  [§3.2]
    pred_vertices = mano_layer(pred_params)                  # (B, 778, 3)

    # 4. Reconstruction: MSE on parameters, Chamfer on vertices  [eq. 1]-[eq. 3]
    param_loss = mse(pred_params, hand_params)
    mesh_loss = chamfer(pred_vertices, hand_vertices)

    # 5. Physical terms: contact, penetration, consistency  [eq. 4]-[eq. 6]
    contact_loss = contact(object_points, pred_vertices)
    penetr_loss = penetration(pred_vertices, object_points)
    consist_loss = consistency(object_points, pred_vertices, hand_vertices)

    # 6. Weighted total; the code applies the contact term from epoch 5  [eq. 7]
    return (
        lambda_1 * param_loss
        + lambda_2 * mesh_loss
        + lambda_3 * contact_loss
        + lambda_4 * penetr_loss
        + lambda_5 * consist_loss
    )

Pseudocode, stage 1b, latent diffusion (autoencoder frozen):

# Stage 1b: latent diffusion  [§3.2] [eq. 8]
# Input:  hand vertices (B, 778, 3), object points (B, 3000, 4)
# Output: scalar noise-prediction loss

def diffusion_loss(hand_vertices, object_points):
    # 1. Encode, then view the 768-d latent as 256 tokens of 3 values  [§3.2]
    latent = encoder(hand_pointnet(hand_vertices)).view(-1, 256, 3)

    # 2. Sample a timestep t and Gaussian noise  [eq. 8]
    t = sample_timestep(1, T)
    noise = sample_gaussian_like(latent)

    # 3. Form the noisy latent from the clean latent at t  [§3.2]
    noisy_latent = forward_diffusion(latent, noise, t)

    # 4. Predict the noise, conditioned on object features  [eq. 8]
    pred_noise = denoiser(noisy_latent, t / T, object_features(object_points))

    # 5. Mean squared error between true and predicted noise  [eq. 8]
    return mse(noise, pred_noise)

Pseudocode, stage 2, adaptation module (autoencoder and diffusion frozen):

# Stage 2: adaptation module only  [§3.3] [eq. 9]
# Input:  hand vertices, GT params, object points
# Output: scalar loss; gradients reach the adaptation module only

def adaptation_loss(hand_vertices, hand_params, object_points):
    # 1. Latent of the hand, from the frozen encoder  [§3.2]
    latent = encoder(hand_pointnet(hand_vertices))           # (B, 768)

    # 2. Diffusion-side input: the code uses a one-step estimate of
    #    the clean latent at a random t  [code: train_adapt.py#L62]
    prior = clean_estimate(latent, object_points)            # (B, 768)

    # 3. Adaptation offset, conditioned on the object  [eq. 9]
    offset = adaptation_mlp(prior, object_points)            # (B, 768)

    # 4. Decode and build the mesh  [eq. 10] [eq. 11]
    pred_params = decoder(prior + offset)                    # (B, 61)
    pred_vertices = mano_layer(pred_params)                  # (B, 778, 3)

    # 5. Same weighted physical and reconstruction loss as stage 1a  [§3.3]
    return (
        lambda_1 * mse(pred_params, hand_params)
        + lambda_2 * chamfer(pred_vertices, hand_vertices)
        + lambda_3 * contact(object_points, pred_vertices)
        + lambda_4 * penetration(pred_vertices, object_points)
        + lambda_5 * consistency(object_points, pred_vertices, hand_vertices)
    )

Hyperparameters

Values in the paper are in the first table rows. The code values are from the released configs (GRAB and OakInk), listed where they differ. "not stated" means the paper and the released code do not give the value for the stage or dataset shown.

Results

Author-reported. Dataset, model and baseline are in each table. No numbers were re-run. Column units are as printed in the paper's tables; the inference times are as printed.

Out-of-domain (HO-3D) tab. 2, §4.5. Ours¹ is trained on GRAB; ContactGen² and Ours² are trained on OakInk. Baseline rows follow tab. 2. A dash means the paper gives no value.

In-domain (OakInk and GRAB) tab. 3, §4.5. The caption marks ContactGen* as trained on OakInk with the authors' released code. The text of §4.5 attributes the GRAB rows to a separate comparison, but the caption covers both datasets.

Ablation tab. 1, §4.4. Rows compare a CVAE baseline, a diffusion model without the adaptation module, a diffusion model trained with physical losses directly, and the full pipeline. The HO-3D rows are out-of-domain with GRAB training.

Qualitative results are in fig. 4 and fig. 5. The user study §4.6 fig. 6 reports, as a chart only, the percentage of ten participants choosing each method's grasp. Those percentages are not in the text and are not transcribed here.

Paper vs code

Code: repository at commit 14b6c57a94a14a4cc998fadc557c34cb57c09b05. The paper links it in the abstract. The owner name matches the first author, and the README links the author's site (wuxiaofei01.github.io). The repository has no LICENSE file, so the code is cited by path and line only and no code text is quoted.

• Autoencoder weights. The paper's values 0.1, 1, 1000, 10, 10 §4.3 match the OakInk config ae.json. The GRAB config uses 10, 10, 100, 10, 10 ae.json. The code's weight order is mesh, parameter, contact, penetration, consistency, KL train_ae.py train_ae.py. In the paper λ1 weights the parameter MSE and λ2 the mesh Chamfer eq. 1; OakInk's 0.1 (parameter) and 1.0 (mesh) match that assignment.
• KL term. The paper's total loss has no KL term eq. 7. The code adds KL on the latent with weight 1e-4 (GRAB) or 0 (OakInk) train_ae.py ae.json.
• Contact term schedule. In autoencoder training the contact term is applied only from epoch 5 train_ae.py. In adaptation training there is no epoch check train_adapt.py.
• Adaptation weights. The paper gives 100, 0.1, 1000, 20, 0.1 §4.3. Neither released config matches: GRAB adapt.json gives 1, 0, 100, 100, 1 and OakInk adapt.json gives 1, 10, 1000, 5, 10.
• Adaptation input. The paper writes z2 = f_adapt(z1) with no object input eq. 9. The code concatenates a 1024-d PointNet feature of the object to the latent adapt_layer.py.
• Adaptation training input. The paper says the diffusion output z1 is the adaptation input §3.3. Training uses a one-step clean-latent estimate at a random timestep train_adapt.py ddim.py. Evaluation uses the full sampled latent eval_adapt.py. The paper does not discuss this mismatch.
• Sampler. The paper says inference uses DDIM and that this gives speed §3.4. The GRAB and OakInk eval configs set the sampler to DDPM, which runs the full 1000-step ancestral loop and ignores the step count eval_adapt.py eval_adapt.json eval_adapt.json. HO-3D evaluation uses DDIM with 20 steps eval_ho3d.py ho3d.json.
• Classifier-free guidance. The paper does not mention guidance. Sampling computes (1 + w) ε_cond − w ε_uncond with the unconditional branch set to zeros ddim.py ddim.py ddim.py. Training never drops the condition: the drop probability is stored ddim.py and the masking line is commented out ddim.py.
• Diffusion network. The paper says "denoising U-Net" §3.2. The code is a PointNet++-style set network with a 128-d time embedding and a 1024-d class condition diffusion.json pointnet2_ssg_sem.py.
• Object input. The paper uses N_o points with 3 coordinates §3.2. The code uses 4 channels: xyz plus object scale grab_dataset.py oakink_dataset.py.
• Batch sizes and learning rates. The paper gives 256 and 1e-4 for all training §4.3. The configs differ by stage and dataset diffusion.json adapt.json. The GRAB adaptation learning rate is 1e-5 adapt.json.
• Hand model per dataset. The GRAB adaptation and evaluation configs set mano to oakink adapt.json eval_adapt.json, so the adaptation and evaluation hand reconstruction uses the OakInk MANO path train_adapt.py. The GRAB autoencoder and diffusion training use the GRAB MANO layer train_ae.py. The paper describes one MANO model §4.1.
• Contact thresholds. The paper defers loss details to its supplementary material §3.2, which does not define the thresholds [supp.]. The code uses a squared-distance threshold of 0.01² for contact, which is 1 cm train_ae.py, and a threshold of 0.005 on the square-rooted distance for consistency loss.py.
• Samples per object. The GRAB evaluation draws 20 samples per object eval_adapt.py; HO-3D draws 10 eval_ho3d.py. The paper does not say how many samples the reported metrics average over §4.2.
• Augmentation. Random object orientations are not mentioned in the paper. The GRAB diffusion and adaptation configs use aug_ratio 10 and 20 diffusion.json adapt.json; the orientation draw is at grab_dataset.py.
• Latent non-negativity. The paper does not say what the latent looks like. The encoder adds a ReLU after every linear layer, including the last vae.py, so latents are non-negative.

Pitfalls

• DDIM time input. In sample_ddim the network is given t = i/T for i from 1 to ddim_step ddim.py. The alpha schedule uses the stepped timesteps ts ddim.py ddim.py. Training gives the network t/T with t in 1..T ddim.py ddim.py. As written, the time input during DDIM sampling is at most ddim_step/1000, for example 0.02 at 20 steps. The DDPM sampler uses i/T with i up to T ddim.py, which is consistent. Check this before relying on the DDIM path.
• Speed depends on the sampler. The reported 0.14 s is per the author's timing setup §4.2 and does not say which sampler was used. The GRAB and OakInk eval configs run the 1000-step DDPM eval_adapt.json. The README gives evaluation times of about 5 minutes for HO-3D, 6 for GRAB and 5 hours for OakInk README.md. Training of the full pipeline takes about one week README.md.
• Adaptation schedule never decays. The milestones are round(epochs × x) for x in [10, 50, 100], which are multiples of the epoch count, not fractions train_adapt.py. With 500 epochs no milestone is reached, so the learning rate stays constant. The autoencoder and diffusion schedules use fractions train_ae.py train_diffusion.py.
• Stochastic sampling. With simple variance on, η is forced to 1 and fresh noise is added at every step ddim.py ddim.py. Repeated runs differ even with a fixed model. The eval configs set a seed eval_adapt.json; a seed alone does not make the GPU stochastic path fully reproducible.
• Loss scale. The contact weight is 1000 for OakInk and 100 for GRAB ae.json ae.json. Combined with the sum-reduced Chamfer term, the relative size of each term depends on the dataset; the paper does not discuss this.
• Chamfer reduction. Chamfer distance sums over points and averages over the batch train_ae.py. The parameter MSE is summed over all elements and divided by the batch size train_ae.py. Both are scaled differently from a per-point mean, which matters once they are weighted against the physical terms.
• Hard-coded paths. The autoencoder module appends an absolute path to sys.path autoencoder.py. The MANO assets path in adaptation training is absolute train_adapt.py. The GRAB diffusion config points at an absolute checkpoint path diffusion.json. These must be changed before running.
• Stage checkpoints. The README asks for the checkpoint from the previous training stage to be used at each step README.md. Mixing checkpoints from different runs is therefore not what the code is set up for.
• Table reference. §4.5 cites the in-domain table for both the OakInk and GRAB comparisons; its caption covers both datasets tab. 3. This card reproduces it as captioned.

Lineage

• Builds on: latent diffusion models and DDIM sampling, both cited in §3.2 and §3.4 without arXiv ids here. Physical losses (consistency, contact, penetration) are taken from GraspTTA §3.2.
• Compared against: GrabNet, GraspTTA, HALO, GF and ContactGen tab. 2 tab. 3.
• Superseded by: not stated in this wiki.