---
kind: concept
slug: human-mesh-recovery
title: Human mesh recovery
summary: >-
  Recovering a 3D human body mesh, with face and hands, from one RGB image. The variant so far estimates each part with its own SMPL-X regressor and merges them into one mesh by copy-paste, a wrist-correction network or per-image optimisation. Trade-off: per-frame cost against whole-body error. Only one card so far; author-reported.
tags: [3d-human-body, whole-body-pose, smpl-x, hand-pose, computer-vision]
papers: ["2108.06428"]
related: [parametric-body-model, data-augmentation]
status: draft
updated_at: 2026-11-06
---

## Overview

The problem is that a single image gives a small, often blurred face and hands,
plus a body that may be self-occluded, and whole-body 3D ground truth is scarce
[arxiv:2108.06428 §1]. Earlier single-image whole-body methods relied either on
slow fitting or on one network trained for every part [arxiv:2108.06428 §1].

The shared idea in the one paper so far is modular: run a separate estimator
for the face, each hand and the body, all in the SMPL-X parameterisation, then
merge the outputs into one whole-body mesh [arxiv:2108.06428 §3] [arxiv:2108.06428 §3.1].
Because the parts share a parameterisation, each can use a strong single-part
method without whole-body training data [arxiv:2108.06428 §1] [arxiv:2108.06428 §5].
The invariant is the output: one SMPL-X pose, shape and camera per person.

## Variants

Only one paper has a card. The variants below are the three merge options that
paper proposes on top of the same per-part estimators.

### Copy-paste merge

Take the body module's shape and camera, its body joints except the wrists, and
the finger poses from the two hand modules. Each wrist is then set from the hand
module's global wrist orientation, converted to a local rotation through the body
skeleton [arxiv:2108.06428 §3.5] [arxiv:2108.06428 eq. 8]. The authors say the
extra compute is almost nothing [arxiv:2108.06428 §3.5]. Reported whole-body
All V2V on EHF is 69.9 [arxiv:2108.06428 tab. 4]. Card:
[FrankMocap](/papers/2108.06428).

### Wrist-correction network

After copy-paste, a six-layer MLP adjusts the arm so the wrist lands where the
hand module saw it, with no gradient steps at run time [arxiv:2108.06428 §3.5]
[arxiv:2108.06428 app. A2.2]. Its input is the arm pose plus a 2D direction
between the body wrist and the hand wrist, scaled by arm length [arxiv:2108.06428 §3.5].
It is trained on pseudo-ground-truth from the optimiser, with noise added to the
shape and to the hand's 2D position [arxiv:2108.06428 app. A2.3]. The noise
amounts are not stated. Reported All V2V is 69.3 at the same runtime as
copy-paste [arxiv:2108.06428 tab. 4].

### Per-image optimisation

Fit all SMPL-X parameters to the image in two stages. The cost combines a 2D
keypoint reprojection term, a 3D term that ties the wrists to the hand meshes,
and a shape prior [arxiv:2108.06428 §3.5] [arxiv:2108.06428 eq. 9]. Stage one
drops the hand-mesh term; stage two adds it, plus a joint-stability term that the
text describes but equation 9 omits [arxiv:2108.06428 §3.5]. Reported All V2V is
63.5, the lowest of the three merges, at about 2 s per frame [arxiv:2108.06428 tab. 4].

Reference points from the same table: SMPLify-X reports All V2V 146.2 at 60 s
per frame and ExPose 81.9 at 0.15 s per frame [arxiv:2108.06428 tab. 4]. Units
are not stated in the paper.

### Parts on their own

The body module starts from SPIN and is fine-tuned for SMPL-X on Human3.6M and
EFT-style pseudo-labels, using pose and 2D keypoints only [arxiv:2108.06428 §3.3].
Its 3DPW MPJPE is 94.3, against 92.3 for EFT and 93.4 for ExPose [arxiv:2108.06428 tab. 3].
The hand module is a ResNet-50 regressor on 224 × 224 crops and is trained on
right hands, with left hands flipped in and out [arxiv:2108.06428 §3.2]
[arxiv:2108.06428 app. A1.1]. Its hand-only scores are close to the best hand-only
methods on STB, RHD and MPII+NZSL [arxiv:2108.06428 tab. 1].

Motion blur is the largest single augmentation gain in the hand ablation: 2D AUC
on MPII+NZSL rises from 0.622 to 0.655 [arxiv:2108.06428 tab. 2]. See
[Data augmentation](/concepts/data-augmentation) for the general pattern.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Live video, low latency needed | Copy-paste merge | Reported 0.15 s per frame, with All V2V 69.9 [arxiv:2108.06428 tab. 4] |
| Offline, lowest whole-body error matters and about 2 s per frame is affordable | Per-image optimisation | Lowest All V2V, 63.5 [arxiv:2108.06428 tab. 4] |
| Want near-optimisation whole-body error without per-frame fitting | Wrist-correction network | All V2V 69.3 at 0.15 s per frame [arxiv:2108.06428 tab. 4]. You must produce training data with the optimiser first [arxiv:2108.06428 app. A2.3] |
| Face detail is the main target | Look beyond the merges | Face V2V is 30.7 for copy-paste and wrist network, 20.4 for optimisation, and 19.5 for ExPose [arxiv:2108.06428 tab. 4] |
| Only hands are needed | The hand module alone | Scores close to the best hand-only methods [arxiv:2108.06428 tab. 1] |
| No GPU available | Check first | The paper does not state its hardware [arxiv:2108.06428 §4]. The released code requires CUDA (see the card) |

The card records the paper-vs-code gaps in full. Copy-paste is the only merge
in the released repository, and the face module is not released. The repository
is non-commercial, so use it as a reference and do not copy its code.

## Implementation sketch

Our own sketch of the modular pipeline, following the steps described in the
paper [arxiv:2108.06428 §3] [arxiv:2108.06428 eq. 8]. Lines are short; the
merge is the only step that changes between variants.

```pseudocode
# Input: one RGB image; Output: one whole-body SMPL-X mesh [arxiv:2108.06428 §3]

def whole_body_mesh(image):
    # 1. Find boxes for the body, both hands and the face [arxiv:2108.06428 app. A3]
    body_crop, left_crop, right_crop, face_crop = crop_boxes(image)
    # 2. Body module: SMPL-X pose, shape and camera [arxiv:2108.06428 §3.3] [eq. 6]
    body = body_module(body_crop)
    # 3. Hand modules; the left hand is flipped to match the right-hand model
    #    [arxiv:2108.06428 §3.2]
    left = hand_module(flip(left_crop))
    right = hand_module(right_crop)
    # 4. Face module; keep the first 10 expression values [arxiv:2108.06428 §3.4]
    face = face_module(face_crop)
    # 5. Merge with one of the three variants [arxiv:2108.06428 §3.5]
    return merge(body, left, right, face)
```

## Contradictions

Only one paper has a card, so no disagreement between papers is known. Within
the paper, the text says the optimisation merge is best in general and that the
wrist network beats copy-paste [arxiv:2108.06428 §4.3]. Table 4 does not support
both claims on every row: copy-paste has the lowest body V2V, and ExPose has the
lowest face V2V [arxiv:2108.06428 tab. 4]. Compare per row, not on the headline.

## Papers

- [FrankMocap (arXiv:2108.06428)](/papers/2108.06428): per-part SMPL-X regressors for face, hands and body, merged by copy-paste, a wrist network or optimisation.