Hand-Object Contact Consistency Reasoning for Human Grasps Generation

In plain words

Given a 3D object, the method produces a human hand that could hold it. A human hand has far more freedom of movement than a gripper, so there are many possible grasps, and most are not natural or stable. The authors observe that a good grasp needs two things to agree: the parts of the hand that touch the object should lie on the spots that people usually touch, and those spots should be the ones the hand actually touches.

The method uses two networks that check each other. One proposes a hand grasp from the object's shape. The other looks at a hand and an object and predicts where the object is usually touched. During training, both are pushed to agree. When a new object arrives, the grasp is adjusted on that one object, using the agreement between the two networks as the main signal and no extra labels. The authors report that this adjustment helps most on objects that differ from the training set.

TL;DR

• Generates a hand mesh (MANO parameters) from the object's point cloud with a conditional VAE, trained with two hand-object contact losses §3.1.
• A second network, ContactNet, predicts the object's contact map from a hand and an object §3.2.
• At test time, the generated grasp is fine-tuned on each object so that its contact region matches ContactNet's prediction. No outside supervision is used §1 §3.3.
• Author-reported: on Obman, penetration volume 5.12 cm³ and perceptual score 3.54, against 6.05 cm³ and 3.02 for GF tab. 1.

Problem

Predicting parallel-jaw robot grasps is well studied. Human grasps with a multi-finger hand are harder: the hand has many more degrees of freedom, and the grasp must be both physically plausible and natural §1. Earlier generative approaches model the hand and its contacts with the object. They ignore that the object has its own likely contact regions, or use those regions only as a constraint §1 §2. The paper's goal includes generalising to unseen and out-of-domain objects §1 §4.4.

Method

The paper trains two networks separately on ground-truth data, then combines them at test time §3 fig. 2.

1. GraspCVAE §3.1 fig. 3 app. A.1. Two PointNet encoders read the hand (778 vertices) and the object (N points). Their 1024-d features are concatenated and fed to a fully connected encoder that outputs the mean and variance of a 64-d latent Gaussian. A decoder takes the sampled latent and the object feature and outputs 61 MANO parameters: 10 shape values and 51 pose values, which include the root rotation and translation §3.1. The hand mesh is produced by the MANO layer §3.1.
   • Training uses the ground-truth hand, so the encoder is used. The baseline loss has a reconstruction term on hand vertices and MANO parameters, a KL term on the latent, and a penetration term §3.1 eq. 1.
   • Two contact losses are added. The object-centric loss compares the object contact map of the generated hand with the ground-truth map §3.1 eq. 2. The hand-centric loss pulls the hand's six prior contact regions fig. 5 towards nearby object points §3.1 eq. 3. The total training objective is eq. 4.
   • The object contact map is built from the distance of each object point to the nearest prior hand vertex. A sigmoid-based function maps that distance, in cm, to a score in [0, 1] §3.1 fig. 4.
2. ContactNet §3.2 fig. 6 app. A.2. Two PointNet encoders read the hand and the object. The output is one value per object point, from four 1-D convolutions with a sigmoid. It is trained with an L2 loss to the ground-truth contact map, using ground-truth hands as inputs §3.2.
3. Test-time adaptation §3.3 fig. 2. The GraspCVAE decoder produces an initial grasp from a latent sampled from a standard Gaussian. No encoder and no hand are needed for this step §3.1 app. A.1. The grasp's contact map is compared with ContactNet's prediction on the same object, and the decoder is updated with the loss in eq. 5. The hand-centric and penetration terms keep the grasp physically plausible. Other parts of the networks are frozen §3.3.
4. Adaptation schedule app. B.3.2. Each test sample gets 10 iterations. Each iteration makes 32 copies of the object, each randomly shifted by up to 5 cm in each axis. Scaling and rotation are reported as harmful app. B.3.2. The paper compares four variants: offline (network reset per sample), optimisation-only (MANO parameters updated directly), noisy-ContactNet and online (network reset once per sequence) §4.5.3.

Loss definitions

As written in the paper:

• Baseline: L_base = L_R + λ_KLD · L_KLD + λ_p · L_penetr eq. 1. L_R is a weighted sum of L2 terms on vertices, pose and shape. L_KLD and L_penetr are defined inline §3.1.
• Object-centric: L_O = ||Ω̂ − Ω||², where Ω is the contact map of the ground-truth hand and Ω̂ that of the generated hand eq. 2 §3.1.
• Hand-centric: L_H sums the squared distances D of object points whose D is at most T = 1 cm eq. 3 §3.1.
• Training total: L_grasp = L_base + λ_H · L_H + λ_O · L_O eq. 4.
• Test-time total: L_TTA = L_refine + λ_H · L_H + λ_p · L_penetr eq. 5, where L_refine = ||Ω_M̂ − Ω^c||² compares the generated hand's contact map with ContactNet's §3.3.
• Penetration: L_penetr averages, over object points inside the hand, the squared distance to the nearest hand vertex §3.1.

# GraspCVAE training objective  [§3.1] [eq. 1] [eq. 2] [eq. 3] [eq. 4]
# Input:  hand vertices (B, 778, 3), object points (B, 3000, 3)
# Output: scalar training loss

def grasp_cvae_loss(hand_gt, obj_points):
    # Encode the hand and the object with two PointNets  [§3.1]
    feat_hand = pointnet_hand(hand_gt)                 # (B, 1024)
    feat_obj = pointnet_obj(obj_points)                # (B, 1024)
    # Posterior mean and log-variance from the concatenated features  [§3.1]
    mu, log_var = cvae_encoder(concat(feat_hand, feat_obj))  # (B, 64)
    # Reparameterised latent sample  [§3.1]
    eps = random_normal(B, 64)
    z = mu + exp(0.5 * log_var) * eps
    # Decode 61 MANO parameters from latent and object feature  [app. A.1]
    params = cvae_decoder(concat(z, feat_obj))         # (B, 61)
    hand_pred = mano_layer(params)                     # (B, 778, 3)
    # Reconstruction: vertex L2 plus pose and shape L2  [§3.1] [§4.1]
    # lambda_v is not stated in the paper
    loss_recon = lambda_v * vertex_l2(hand_pred, hand_gt) \
        + 0.1 * pose_l2(params, gt_params) \
        + 0.1 * shape_l2(params, gt_params)
    # KL term; the paper writes L_KLD = -KL(Q || N(0, I))  [§3.1]
    # lambda_kld is not stated; the sign is discussed in Pitfalls
    loss_kld = lambda_kld * kl_term(mu, log_var)
    # Penetration: mean squared distance of inside object points  [§3.1]
    loss_penetr = penetration_mean(obj_points, hand_pred)
    # Object-centric term: match contact maps  [§3.1] [eq. 2]
    cmap_gt = contact_map(obj_points, hand_gt)
    cmap_pred = contact_map(obj_points, hand_pred)
    loss_obj = sum((cmap_pred - cmap_gt) ** 2)
    # Hand-centric term: prior hand vertices near the object  [§3.1] [eq. 3]
    loss_hand = sum_below_threshold(obj_to_prior_sq_dist(obj_points, hand_pred))
    # Baseline total with lambda_p = 5  [eq. 1] [§4.1]
    loss_base = loss_recon + loss_kld + 5.0 * loss_penetr
    # Full objective with lambda_H = 1500 and lambda_O = 100  [eq. 4] [§4.1]
    return loss_base + 1500.0 * loss_hand + 100.0 * loss_obj

# Test-time adaptation for one object  [§3.3] [eq. 5] [app. B.3.2]
# Input:  object points (3000, 3), trained decoder and ContactNet
# Output: adapted decoder

def adapt_one_object(obj_points, decoder, contact_net):
    # Sample the latent from N(0, I)  [§3.3] [app. A.1]
    # Whether it is resampled per iteration is not stated
    z = random_normal(64)
    # 10 self-supervised iterations per test sample  [app. B.3.2]
    for step in range(10):
        # Make 32 copies of the object, shifted by up to 5 cm  [app. B.3.2]
        batch = random_translate(obj_points, copies=32, max_cm=5.0)
        # Hand from the current decoder output  [§3.3]
        hand = mano_layer(decoder(concat(z, object_feature(batch))))
        # Contact map of the generated hand  [§3.3]
        cmap_hand = contact_map(batch, hand)
        # Target from ContactNet, treated as fixed (detached in release)
        # [code: gen_diverse_grasp_ho3d.py#L86-L87]
        cmap_target = contact_net(batch, hand).detach()
        # Self-supervised consistency between the two maps  [§3.3] [eq. 5]
        loss_refine = sum((cmap_hand - cmap_target) ** 2)
        # Hand-centric and penetration terms with lambda_H = 1, lambda_p = 5
        # [eq. 5] [§4.1]
        loss = loss_refine + 1.0 * loss_hand(batch, hand) \
            + 5.0 * loss_penetr(batch, hand)
        # SGD, momentum 0.8, learning rate 6.25e-6; decoder only  [§3.3] [§4.1]
        decoder = sgd_step(decoder, loss, lr=6.25e-6, momentum=0.8)
    return decoder

The pseudocode is our rendering of the paper's equations. The paper does not publish this loop, so details such as whether the loss is averaged over the 32 copies are not stated app. B.3.2.

Hyperparameters

Training schedule and loss weights are from §4.1. The learning-rate check holds: 1e-4 halved four times is 6.25e-6, as the paper states §4.1.

Results

Author-reported. GraspCVAE and ContactNet are trained on the Obman training set, and Obman is evaluated on its test set. HO-3D and FPHA are never seen in training and are evaluated as out-of-domain objects §4.2 §4.4. Baselines are the ground truth (GT) and Grasping Field, cited as GF tab. 1. All results in Table 1 are evaluated after test-time adaptation §4.4. Hardware, runtime and sample counts are not stated.

Table 1 tab. 1. Grasp quality on three datasets. Arrows show the direction of better values.

The perceptual score is from Amazon Mechanical Turk raters: three raters per sample, on a 1-to-5 scale §4.3 app. B.2. The paper states that its grasps improve on GF in penetration, grasp stability and perceptual score on all three datasets, and that its perceptual scores on out-of-domain objects are close to the in-domain score §4.4.

Table 2 tab. 2. GraspCVAE training losses on Obman. Two rows are grey in the paper, marked as modified variants used for comparison. The paper does not say whether this table is before or after test-time adaptation.

The paper states that the hand-centric loss lowers displacement and raises contact, and that the object-centric loss mainly improves object-side contact and stability §4.5.1. It also reports that fixed ground-truth contact regions (L_H(gt)) and raw distances instead of the contact map (L_O(dist)) are worse §4.5.1.

Table 3 tab. 3. ContactNet design, Obman. Error is the average absolute error of the predicted contact map over all object points (units as stated in the caption: contact-map score).

Table 4 tab. 4. Test-time adaptation on out-of-domain HO-3D and FPHA. The caption gives no units. Column headings are as in the paper. The FPHA depth and volume values do not match tab. 1 (see Pitfalls).

The paper reports that every adaptation method improves on HO-3D and FPHA, that online adaptation is strongest on HO-3D, and that its gain on FPHA is smaller, which the paper links to shorter video sequences §4.5.3.

Appendix comparison app. B.3.1. Losses trained with the paper's objective against losses from GRAB, on Obman: penetration volume 5.12 against 8.41, stability 1.52 against 1.66, perceptual score 3.54 against 2.97, and contact 99.97% against 98.25%. The authors measured these on the Obman test set.

Penetration and stability §4.6 fig. 11. The paper plots penetration volume against displacement for its grasps, GT and GF on the three datasets, and reports that its grasps lie near the origin. It notes that larger penetration can buy stability, which is why both metrics are read together.

Paper vs code

Official code: https://github.com/hwjiang1510/GraspTTA, linked from the paper's project page, and owned by the account hwjiang1510, which matches the first author. The commit is ef7fcfab1477a5f3b8a8e2057fb8614087361028 on master. The repository has no LICENSE file, and the GitHub API reports no license, so code_license is recorded as none. Code is cited by path and line only. Nothing is quoted.

The release contains the diverse-grasp generator for HO-3D and the network definitions. It does not contain training code: the README says training code is available only on request README.md. The training-stage settings in the paper therefore cannot be checked against code.

Discrepancies, each with citations:

• Test-time mechanism. The paper updates only the GraspCVAE decoder §3.3 eq. 5. The release updates no network. It creates one 61-d MANO vector per sample and optimises that vector with SGD. ContactNet is in evaluation mode and is never optimised gen_diverse_grasp_ho3d.py gen_diverse_grasp_ho3d.py. This is closer to the paper's optimisation-only variant, which the appendix says optimises only the 45-d hand pose app. B.3.2. Nothing shows that Table 1 comes from this code.
• Iterations and augmentation. The paper uses 10 iterations with 32 translated copies per iteration app. B.3.2. The release runs 300 SGD steps on a batch of one gen_diverse_grasp_ho3d.py. Its translation augmentation functions are defined but never called utils_TTT.py.
• Loss weights. The paper's test-time weights are λ_H = 1 and λ_p = 5 §4.1. The release uses 1 × contact + 1 × consistency + 7 × penetration gen_diverse_grasp_ho3d.py. Each term is also scaled internally: the contact term by 2.5 × 3000 loss.py loss.py, the consistency term by 0.0001 loss.py, and the penetration term by 120 with no averaging over penetrating points loss.py. The paper's penetration loss averages over inside points §3.1.
• Hand-centric threshold. The paper uses T = 1 cm §3.1. The release compares squared nearest-neighbour distances with 0.02² loss.py. Squared distances are the convention here: the contact-map code takes a square root and converts to cm utils.py. The effective threshold is therefore 2 cm, not 1 cm.
• Object-centric term at test time. The paper lists λ_O = 5 for test time §4.1. The test-time loss in eq. 5 has no object-centric term, and the release has none either gen_diverse_grasp_ho3d.py.
• ContactNet target. The release divides the ContactNet output by its per-sample maximum, and detaches it, before the loss gen_diverse_grasp_ho3d.py. The paper does not state the normalisation.
• ContactNet fusion. The paper adds the hand and object global features, repeats the sum 3000 times and concatenates it with the object's local features §3.2 app. A.2. The code concatenates object and hand per-point features along the point axis (3000 + 778 entries) and mixes them with a learned 1-D convolution from 3778 to 3000 entries cmapnet_objhand.py.
• CVAE encoder. The appendix lists encoder layers 2048, 1024, 512, 256, 64 app. A.1. The code has hidden layers 1024, 512, 256 on a 2048-d input, then two linear 64-d heads for mean and log-variance affordanceNet_obman_mano_vertex.py CVAE.py. The paper's final 64-unit layer may be these heads; this is not stated.
• Object input. The paper's object input is N × 3 §3.1. The release adds a fourth channel: the object's bounding diameter, repeated for every point, with default obj_inchannel=4 HO3D_diversity_generation.py utils_HO3D_FPHA.py gen_diverse_grasp_ho3d.py.
• Object sampling in the release. Each sample applies a random rotation and a translation drawn from a fixed offset plus up to 0.2 per axis gen_diverse_grasp_ho3d.py. Saved grasps must have penetration volume under 4 cm³ and simulated displacement under 3 cm, with up to 100 grasps per object gen_diverse_grasp_ho3d.py gen_diverse_grasp_ho3d.py. The paper does not describe this selection. The release has 10 HO-3D object IDs gen_diverse_grasp_ho3d.py and no FPHA loader.
• Metrics in the release. The release computes penetration volume (voxel pitch 0.005 m, matching 0.5 cm), sample-level contact and displacement gen_diverse_grasp_ho3d.py. It does not compute penetration depth. The perceptual score is an AMT study and is not in the release. Other metrics are in metric/, which was not checked.
• Simulation fallback. If the simulation throws, the release sets displacement to 0.10, which is above its 0.03 threshold, so the grasp is rejected gen_diverse_grasp_ho3d.py. The V-HACD binary path is hard-coded gen_diverse_grasp_ho3d.py.
• MANO pose. The release passes the 45-d pose slice to a MANO layer loaded with use_pca=True and 45 components gen_diverse_grasp_ho3d.py gen_diverse_grasp_ho3d.py. Whether these are PCA coefficients or axis-angle depends on the MANO package, which is not in this repository. The paper describes axis-angle rotations app. B.3.2.
• Device. TTT_loss moves tensors to CUDA unconditionally loss.py.
• Paper identifier. The README links the report as arXiv 2108.05877 README.md. This card covers arXiv 2104.03304. The two were not checked to be the same paper.

The contact-map formula matches the paper: distance in cm, passed through 1 − 2·(sigmoid(2d) − 0.5) utils.py §3.1. The TTA learning rate and momentum match §4.1 gen_diverse_grasp_ho3d.py. The penetration voxel size and contact threshold match §4.3 gen_diverse_grasp_ho3d.py gen_diverse_grasp_ho3d.py. The decoder widths 1088 → 1024 → 256 → 61 match app. A.1 CVAE.py affordanceNet_obman_mano_vertex.py. The 61-d parameter split is 10 + 3 + 45 + 3 gen_diverse_grasp_ho3d.py.

Pitfalls

• KL sign. The paper writes L_KLD = −KL(Q || N(0, I)) and says the loss keeps the latent close to N(0, I) §3.1. Minimising that term as written favours a larger KL. Training code is not released, so the intended sign cannot be checked. Do not infer it.
• Unstated weights and normalisation. λ_V and λ_KLD are not given §3.1 §4.1. The ContactNet loss is stated as an L2 distance, with no mean or sum normalisation §3.2.
• Threshold and units. T = 1 cm is compared with D, which the paper defines as a squared norm §3.1. The release uses 2 cm (see Paper vs code). Choose one convention and state it.
• Which points form Ω̂. The paper does not say whether the generated hand's contact map uses all 778 vertices or only the prior regions. The release uses all vertices gen_diverse_grasp_ho3d.py. The ground-truth Ω uses nearest prior vertices §3.1.
• Inside test for penetration. The paper uses "object points inside the hand" but does not say how inside is decided. The release uses the sign of the dot product between the nearest hand vertex normal and the offset vector utils_loss.py.
• Test-time λ_O. The paper lists λ_O = 5 §4.1, but eq. 5 has no L_O term. Do not add an object-centric term at test time without a source.
• Table 4 FPHA values differ from Table 1. Table 1 gives FPHA "Ours" depth 1.58 and volume 6.37 tab. 1. Table 4 gives FPHA TTA depth 6.37 and volume 1.58 tab. 4. Both tables use the same column names, so either one has the two columns swapped or the definitions differ. The paper does not say which.
• Which TTA variant Table 1 uses. Table 1's HO-3D "Ours" (depth 1.05, volume 4.58, displacement 3.21 ± 3.79, contact 99.61) matches the TTA-online row of Table 4. FPHA "Ours" matches the plain TTA row, with depth and volume swapped tab. 1 tab. 4. The paper does not say which variant is used per dataset. The TeX source has a commented-out row with the Obman "Ours" numbers, labelled TTA tab. 2 source comment. The compiled paper does not show that row.
• Table 2 is not labelled as pre- or post-TTA. Its last row (0.48, 4.92, 1.63, 99.94) differs from Table 1's Obman "Ours" (0.46, 5.12, 1.52, 99.97).
• Online adaptation is per sequence. The network is reset only once per video sequence §4.5.3 app. B.3.2. The FPHA video sequences are, on average, 1/20 the length of HO-3D sequences §4.5.3, which the paper gives as the reason for its smaller FPHA gain.
• Sample counts. The paper does not state how many test samples produce Table 1, nor how the displacement mean and variance are aggregated beyond "over all test samples" §4.3.
• Ground-truth inputs at train time. ContactNet is trained with ground-truth hands and then receives generated hands at test time §3.2 §3.3. The paper argues the target map carries the ideal contact pattern §3.3 but does not measure the effect of this shift.
• FPHA filtering. The paper excludes huge objects such as the milk bottle and fits ground-truth FPHA hands to joints app. B.1. The release has no FPHA data.
• Reconstruction error is not a metric. The paper does not use hand reconstruction error to judge generated grasps, because each sample has only one ground-truth grasp app. B.2. Do not add it as a metric.
• Hardware and runtime are not stated. Test-time cost is 10 iterations of 32 copies per sample app. B.3.2 in the paper, and 300 steps in the release gen_diverse_grasp_ho3d.py.

Lineage

• Builds on: the conditional VAE used for GraspCVAE §3.1 (cited as Sohn2015LearningSO), the VAE training rule (cited as kingma2013auto) §3.1, PointNet §3.1 and the MANO hand model §3.1. Test-time self-supervision is inspired by Sun et al. (2019), but the paper says its objective optimises the grasp goal directly rather than an auxiliary task §2.
• Contact-map supervision is motivated by Brahmbhatt et al. (2019, 2020), and the hand penetration constraints follow Hasson et al. (2019) §2 §3.1.
• Compared against: GraspingField (GF), Karunratanakul et al. (2020), on all three datasets tab. 1 §4.2. GRAB losses, Taheri et al. (2020), in the appendix app. B.3.1.
• Superseded by: not stated.