Human mesh recovery

Overview

The problem is that a single image gives a small, often blurred face and hands, plus a body that may be self-occluded, and whole-body 3D ground truth is scarce 2108.06428 §1. Earlier single-image whole-body methods relied either on slow fitting or on one network trained for every part 2108.06428 §1.

The shared idea in the one paper so far is modular: run a separate estimator for the face, each hand and the body, all in the SMPL-X parameterisation, then merge the outputs into one whole-body mesh 2108.06428 §3 2108.06428 §3.1. Because the parts share a parameterisation, each can use a strong single-part method without whole-body training data 2108.06428 §1 2108.06428 §5. The invariant is the output: one SMPL-X pose, shape and camera per person.

Variants

Only one paper has a card. The variants below are the three merge options that paper proposes on top of the same per-part estimators.

Copy-paste merge

Take the body module's shape and camera, its body joints except the wrists, and the finger poses from the two hand modules. Each wrist is then set from the hand module's global wrist orientation, converted to a local rotation through the body skeleton 2108.06428 §3.5 2108.06428 eq. 8. The authors say the extra compute is almost nothing 2108.06428 §3.5. Reported whole-body All V2V on EHF is 69.9 2108.06428 tab. 4. Card: FrankMocap.

Wrist-correction network

After copy-paste, a six-layer MLP adjusts the arm so the wrist lands where the hand module saw it, with no gradient steps at run time 2108.06428 §3.5 2108.06428 app. A2.2. Its input is the arm pose plus a 2D direction between the body wrist and the hand wrist, scaled by arm length 2108.06428 §3.5. It is trained on pseudo-ground-truth from the optimiser, with noise added to the shape and to the hand's 2D position 2108.06428 app. A2.3. The noise amounts are not stated. Reported All V2V is 69.3 at the same runtime as copy-paste 2108.06428 tab. 4.

Per-image optimisation

Fit all SMPL-X parameters to the image in two stages. The cost combines a 2D keypoint reprojection term, a 3D term that ties the wrists to the hand meshes, and a shape prior 2108.06428 §3.5 2108.06428 eq. 9. Stage one drops the hand-mesh term; stage two adds it, plus a joint-stability term that the text describes but equation 9 omits 2108.06428 §3.5. Reported All V2V is 63.5, the lowest of the three merges, at about 2 s per frame 2108.06428 tab. 4.

Reference points from the same table: SMPLify-X reports All V2V 146.2 at 60 s per frame and ExPose 81.9 at 0.15 s per frame 2108.06428 tab. 4. Units are not stated in the paper.

Parts on their own

The body module starts from SPIN and is fine-tuned for SMPL-X on Human3.6M and EFT-style pseudo-labels, using pose and 2D keypoints only 2108.06428 §3.3. Its 3DPW MPJPE is 94.3, against 92.3 for EFT and 93.4 for ExPose 2108.06428 tab. 3. The hand module is a ResNet-50 regressor on 224 × 224 crops and is trained on right hands, with left hands flipped in and out 2108.06428 §3.2 2108.06428 app. A1.1. Its hand-only scores are close to the best hand-only methods on STB, RHD and MPII+NZSL 2108.06428 tab. 1.

Motion blur is the largest single augmentation gain in the hand ablation: 2D AUC on MPII+NZSL rises from 0.622 to 0.655 2108.06428 tab. 2. See Data augmentation for the general pattern.

When to use which

The card records the paper-vs-code gaps in full. Copy-paste is the only merge in the released repository, and the face module is not released. The repository is non-commercial, so use it as a reference and do not copy its code.

Implementation sketch

Our own sketch of the modular pipeline, following the steps described in the paper 2108.06428 §3 2108.06428 eq. 8. Lines are short; the merge is the only step that changes between variants.

# Input: one RGB image; Output: one whole-body SMPL-X mesh [arxiv:2108.06428 §3]

def whole_body_mesh(image):
    # 1. Find boxes for the body, both hands and the face [arxiv:2108.06428 app. A3]
    body_crop, left_crop, right_crop, face_crop = crop_boxes(image)
    # 2. Body module: SMPL-X pose, shape and camera [arxiv:2108.06428 §3.3] [eq. 6]
    body = body_module(body_crop)
    # 3. Hand modules; the left hand is flipped to match the right-hand model
    #    [arxiv:2108.06428 §3.2]
    left = hand_module(flip(left_crop))
    right = hand_module(right_crop)
    # 4. Face module; keep the first 10 expression values [arxiv:2108.06428 §3.4]
    face = face_module(face_crop)
    # 5. Merge with one of the three variants [arxiv:2108.06428 §3.5]
    return merge(body, left, right, face)

Contradictions

Only one paper has a card, so no disagreement between papers is known. Within the paper, the text says the optimisation merge is best in general and that the wrist network beats copy-paste 2108.06428 §4.3. Table 4 does not support both claims on every row: copy-paste has the lowest body V2V, and ExPose has the lowest face V2V 2108.06428 tab. 4. Compare per row, not on the headline.

Papers

• FrankMocap (arXiv:2108.06428): per-part SMPL-X regressors for face, hands and body, merged by copy-paste, a wrist network or optimisation.