Reconstructing Hand-Object Interactions in the Wild

In plain words

Photos of people holding things show hands and objects that overlap, so a camera sees only part of each. Most systems that estimate 3D hand and object shapes were trained on labelled lab recordings, where the scene is simple and the object list is short. This paper asks how to get a 3D picture of a hand holding an everyday object from a single ordinary photo, without 3D labels.

The approach is a fitting process rather than a single network. It starts from a hand estimate and an object estimate, then nudges both until they agree with the 2D picture (outline of the object, hand keypoints, a depth guess). It then pushes the hand and object to touch without passing through each other, and finally adjusts the grip so the contact looks like a real one. The authors also used this tool, together with human checks, to label 500 in-the-wild images from the MOW dataset.

TL;DR

• Reconstructs a hand (MANO mesh) and an object (a 3D model supplied per image) from one RGB image, without 3D supervision §1 §3.
• Four stages: hand fit to 2D keypoints, object pose by differentiable mask and depth rendering, joint hand-object optimisation with distance and collision terms, and contact-prior refinement §3.
• Lab results (author-reported) beat the cited baseline on hand error on HO3D and FPHA; on FPHA the object error is worse §4.1 tab. 2.
• Each image needs a 3D object model chosen by a human annotator, and the loss weights are tuned per image during annotation §5.1 §5.2.
• Introduces the MOW dataset: 500 annotated in-the-wild images, 121 object categories §1 §5 tab. 1.

Problem

Monocular 3D hand-object reconstruction in lab settings relies on 3D annotations that are hard to collect outside the lab, and the existing datasets have few participants and objects §1 tab. 1. Feed-forward models trained on those datasets generalise poorly to unseen objects in the wild §4.2. A single image is also under-constrained in depth and scale, and mutual occlusion between hand and object makes it worse §1 §3.3. The paper assumes a single hand-object pair per image §3.

Method

The method has four steps, shown in fig. 3 §2. Each step is a fitting problem; the paper gives no training procedure for the hand and object steps.

1. Hand pose §3.1. A learned regressor (FrankMocap) gives an initial MANO pose, shape and weak-perspective camera. The pose and shape are then fitted to 2D hand keypoints eq. 1. The camera is turned into a perspective one with a fixed focal length f, using depth f / s_h eq. 2.
2. Object pose §3.2. An object mask is taken from a class-agnostic instance model, keeping the instance that overlaps the detected hand box most (highest IoU). The object's scale, rotation and translation are fitted by differentiable rendering against the mask eq. 3 and a monocular depth map eq. 4. The fit is run from many random starting rotations and the lowest-loss result is kept app. Implementation Details.
3. Joint optimisation §3.3. Hand and object are optimised together with the object-pose terms plus an interaction term (symmetric chamfer distance between the two meshes) eq. 6 and a collision term that penalises object vertices inside the hand, read from a voxel signed-distance grid eq. 7 eq. 8. The total is a weighted sum eq. 9.
4. Contact refinement §3.4. A small network takes the hand parameters and a distance field from hand vertices to their nearest object vertices eq. 10, updates the hand, and recomputes the field. The paper runs three iterations app. Implementation Details. It is trained on ground-truth hand poses from the GRAB MoCap data §3.4, with noise added to simulate imperfect input.

The core fitting loop, in our own notation:

# Reconstruct one hand-object pair from one image  [§3]
# Input:  image (H, W, 3); object_model (mesh, about 800 faces) [app. Impl.]
# Output: hand_vertices (V_h, 3); object_vertices (V_o, 3)

def reconstruct(image, object_model):
    # 1. Fit the MANO hand to 2D keypoints  [§3.1] [eq. 1]
    hand = fit_hand_to_keypoints(image)
    # 2. Object mask: the instance that best overlaps the hand box  [§3.2]
    object_mask = pick_instance_by_hand_iou(image, hand)
    # 3. Monocular depth as the target for object depth  [§3.2]
    object_depth = monocular_depth(image)
    # 4. Fit object scale, rotation, translation from many starts  [eq. 5]
    object_pose = best_of_starts(object_model, object_mask, object_depth)
    # 5. Jointly optimise hand and object with all four terms  [§3.3] [eq. 9]
    hand, object_pose = optimise_jointly(hand, object_pose)
    # 6. Refine the contact with a small network, three iterations  [§3.4]
    for iteration in range(3):
        distance_field = hand_to_object_distances(hand, object_pose)
        hand = refine_network(hand, distance_field)
    return hand, object_pose

Key equations, numbered as in the paper's main text, in the paper's notation (see Pitfalls for the unclear symbols):

Hand fit eq. 1: \boldsymbol{\theta}^{*}, \boldsymbol{\beta}^{*} = \arg\min_{\boldsymbol{\theta}, \boldsymbol{\beta}}\, L_{joints} + L_{reg}

Hand vertices in the camera frame eq. 2: \boldsymbol{V}^{*}_h = H(\boldsymbol{\theta}^{*}, \boldsymbol{\beta}^{*}) + [t_x, t_y, f/s_h]

Mask loss eq. 3 and depth loss eq. 4, both rendered from object scale s, rotation \boldsymbol{R} and translation \boldsymbol{T}: L_{mask} = \lVert \textrm{NR}_m (s, \boldsymbol{R}, \boldsymbol{T}) - \boldsymbol{M} \rVert L_{depth} = \lVert \textrm{NR}_d (s, \boldsymbol{R}, \boldsymbol{T}) - \boldsymbol{D} \rVert

Object objective eq. 5: s^*, \boldsymbol{R}^*, \boldsymbol{T}^* = \argmin_{s,\boldsymbol{R}, \boldsymbol{T}}\, L_{mask} + L_{depth}

Interaction loss, symmetric chamfer distance between object vertices \boldsymbol{V}_o and hand vertices \boldsymbol{V}_h eq. 6: L_{dist} = \frac{1}{|\boldsymbol{V}_o|} \sum_{x \in \boldsymbol{V}_o} \min_{y \in \boldsymbol{V}_h} \|x - y\|_2 + \frac{1}{|\boldsymbol{V}_h|}\sum_{y \in \boldsymbol{V}_h} \min_{x \in \boldsymbol{V}_o} \|x - y\|_2

Penetration value of a voxel cell, from the hand SDF eq. 7: \phi(\boldsymbol{c}) = -\min(\textrm{SDF}(c_x, c_y, c_z), 0)

Collision loss over object vertices eq. 8: L_{collision} = \sum_{\boldsymbol{v} \in \boldsymbol{V}_o^*} \phi(\boldsymbol{v})

Joint objective eq. 9: L = \lambda_1 L_{mask} + \lambda_2 L_{depth} + \lambda_3 L_{dist} + \lambda_4 L_{collision}

Refinement distance field, squared distance from each hand vertex to its nearest object vertex eq. 10: \boldsymbol{F}(\boldsymbol{v}_h) = \min_{\boldsymbol{v}_o \in \boldsymbol{V}_o^*} \lVert \boldsymbol{v}_h - \boldsymbol{v}_o \rVert^2_2

Hyperparameters

Results

Author-reported. Setup: lab evaluation on HO3D and FPHA with the baseline of Hasson et al. (CVPR 2020), which takes the same RGB image and known object model and is trained with full 3D supervision §4.1. The method is not trained on these datasets' 3D labels, and FPHA uses a different split from the baseline's §4.1. The in-the-wild comparison is qualitative §4.2.

Ablation of the joint terms (cumulative; dataset not stated) tab. 3:

Annotation quality on 100 images tab. 4 §5.2:

Human quality rating on a 1–5 scale: 4.16 on average; 3D object model matches the true object in 92% of cases tab. 4. The number of raters and the size thresholds are not stated.

Dataset comparison tab. 1: HO3D (lab, video, 10 participants, 10 objects), CP (lab, video, 50, 25), GRAB (lab, MoCap, 10, 51), this work (in the wild, images, 450 participants, 121 objects).

Paper vs code

No official code for the method. The only repository the paper's project page links is ZheC/MOW (https://github.com/ZheC/MOW), which is the dataset release: setup script, vis_anno.py visualisation, poses.json, and example data. It has no method code and no license (checked 2026-10-19).

Searches run for the method: the project page (https://zhec.github.io/rhoi; the old Berkeley URL redirects there), the GitHub user ZheC's public repositories (no hand-object method repository in the first 20 results, the list was cut), a repository search for "RHO hand object reconstruction" (0 results), and a search for the paper title (returned EasyHOI by lym29 and HOTrack by PKU-EPIC; neither is by the authors, so not used).

Discrepancy inside the dataset release: the abstract and text say 500 images §1 §5; the ZheC/MOW README states 512 images.

Pitfalls

• A 3D model per object is required. Each image needs a known 3D object model. In the annotation pipeline a human picks it from a collection or finds it online §5.1. The abstract's "without 3D supervision" does not mean the method runs without 3D input.
• Loss weights are tuned per image. The annotator chooses the loss weights, and the default weights are only a starting point §5.2. The λ values in app. Implementation Details are defaults, not settings the paper tests across data. The λ symbols are defined only in the appendix: the sentence defining them in the main text is commented out in the source §3.3.
• Human verification is part of the pipeline. Reconstructions go back and forth between reconstruction and verification until the annotator accepts them §5.2. The dataset is therefore not the output of a fully automatic method.
• Scale and depth ambiguity. Hand depth is f / s_h, so the focal length f directly changes hand depth, yet f is not stated §3.1 eq. 2. The paper itself names scale ambiguity as the reason for joint optimisation §3.3.
• Monocular depth alignment is unstated. The depth map from Ranftl et al. is compared directly with the rendered depth eq. 4. The paper does not say how the depth map's scale and shift are aligned before this comparison.
• Loss scales differ widely. λ3 = 1000 weights the distance term and λ4 = 5 weights collision, a 200× spread. The units of the mask, depth and distance terms are not stated, so the weights cannot be checked for balance from the paper alone.
• Squared vs. unsquared distances. The interaction term eq. 6 uses unsquared norms, while the refinement distance field eq. 10 uses squared norms. Use the form written for each term.
• Notation in refinement. The refinement input is written (θ, γ) but the update is written for (θ, β) §3.4. γ is not defined; the text suggests it is the translation, but this is not stated.
• Hand fit terms undefined. L_joints and L_reg are named but not written out eq. 1. Their exact form, and their weights, are not stated.
• Symmetric chamfer is over whole meshes. The interaction term uses all object and hand vertices, not only contact regions eq. 6. The contact behaviour comes from the collision and refinement terms.
• Collision grid resolution. The 32³ hand grid over a tight box trades accuracy against cost. The box margin and cell size are not stated app. Implementation Details.
• Local minima. The joint optimisation can converge to undesirable local minima, and the object fit depends on 1000 random starts app. Failure Cases app. Implementation Details.
• Cost. About one minute per image on a GTX 2080 Ti [app. Implementation Details]; the 1000 object starts are the main contributor to the object step.
• Failure modes. Hand estimation fails under heavy occlusion or cropping, object pose is wrong under mask ambiguity or a poor model, and joint optimisation can settle in local minima app. Failure Cases fig. 11.
• Baseline comparison caveats. The baseline is trained with full 3D supervision and the FPHA split differs §4.1. The FPHA object error is worse than the baseline tab. 2.
• Dataset eval is small. The annotation quality numbers come from 100 images §5.2 tab. 4.
• Ablation dataset. The dataset for the ablation table is not stated §4.3 tab. 3.
• Object-size thresholds in the dataset evaluation are not stated tab. 4.

Lineage

• Builds on: FrankMocap for the initial hand estimate §3.1 (arXiv:2008.08324); MANO hand model §3.1; PointRend for the instance mask §3.2; the neural 3D mesh renderer for differentiable rendering §3.2; monocular depth from Ranftl et al. 2020 §3.2; the GRAB MoCap data for contact priors §3.4.
• Compared against: Hasson et al., CVPR 2020, "Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction" §4.1.
• Superseded by: none recorded.