FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

In plain words

Most tools that estimate a 3D human pose from one photo look at only the body, only the hands, or only the face. Full-body capture needs all three at once, and training one network for all of them needs whole-body 3D data, which is hard to collect.

FrankMocap takes a practical route. It runs a separate, specialised estimator for the face, for each hand and for the body, then stitches the results into one body model. The stitching comes in three versions. The fastest copies the parts together and corrects the wrists using the hand's orientation. A slower version adjusts the whole model to fit the image so the parts agree. A middle version uses a small network to nudge the arms. The fast versions suit live video, and the slow one suits offline work.

The point is that each part can use the best existing single-part method, and the parts can be swapped as better ones appear.

TL;DR

• Face, each hand and the body are estimated separately from one image with SMPL-X-based regressors, then merged into one whole-body mesh §3.
• Three merge options trade speed for accuracy: copy-paste, a wrist-correction network and per-image optimisation §3.5.
• Author-reported: the optimisation merge gives the lowest whole-body vertex error on EHF (V2V 63.5, ExPose 81.9) tab. 4.
• The released code has the hand and body estimators and only the copy-paste merge. It has no face module, no wrist network and no optimisation, so those results cannot be reproduced from the repository (see Paper vs code).

Problem

Whole-body 3D capture is hard because faces and hands are small in the image, hands blur and self-occlude during motion, and whole-body 3D ground truth is scarce even in a lab §1. Earlier single-image whole-body methods depended on slow optimisation or on one end-to-end model trained for all parts §1. The authors argue that a modular design can use per-part state of the art without needing whole-body training data §1, §5.

Method

FrankMocap runs four estimators on one image and merges their outputs into one SMPL-X mesh §3 fig. 2. The face, hand and body models all share the SMPL-X parameterisation §3.1, so their outputs can be combined directly.

SMPL-X parameters §3.1 eq. 1. The whole-body model takes a global orientation φ_w, body, hand and jaw pose parameters θ_w, a shape vector β_w and ten expression parameters ψ_f. The body has 21 joints with 3 axis-angle values each, each hand 15 joints, and the face one jaw joint §3.1. The text says the mesh has 10,745 vertices, while the matrix dimension is 10475 §3.1; see Pitfalls.

V_w = W(\phi_w, \theta_w, \beta_w, \psi_f) eq. 1

Stand-alone hand model §3.1 eq. 2 fig. 3. The hand model is the hand part of SMPL-X: a 15-joint pose θ_h, a global orientation φ_h and a shape β_h that shares SMPL-X's shape space. It has 778 vertices and 21 joints (wrist, 15 finger joints and 5 fingertips) §3.1 app. A1.1.

V_h = H(\phi_h, \theta_h, \beta_h) eq. 2

Hand module §3.2 eq. 3 eq. 4 fig. 4. A ResNet-50 encoder with two extra fully connected layers regresses the hand parameters from a crop around one hand §3.2 app. A1.1. The camera is weak perspective: a 2D translation t_h and a scalar scale s_h §3.2.

[\phi_h, \theta_h, \beta_h, c_h] = M_H(I_H), \quad c_h = (t_h, s_h) eq. 3

J^{2D}_{h,i} = s_h \, \Pi(J^{3D}_{h,i}) + t_h eq. 4

Here Π is orthographic projection §3.2. The authors train the hand module on right hands only. Left-hand images and labels are flipped, and at inference a left hand is flipped, processed as a right hand, and flipped back §3.2. The paper says the flip is vertical §3.2; the code mirrors the image left to right (see Pitfalls).

Hand training loss §3.2 eq. 5. Three losses use the three kinds of label: a squared error on axis-angle pose (L_θ), a squared error on 3D joints (L_3D), an L1 error on 2D joints (L_2D), and a shape regulariser on β_h (L_reg). The weights are λ1 = 10, λ2 = 100, λ3 = 10 and λ4 = 0.1 §3.2. The FreiHAND shape labels are not used because they belong to the MANO model §3.2.

L = \lambda_1 L_\theta + \lambda_2 L_{3D} + \lambda_3 L_{2D} + \lambda_4 L_{reg} eq. 5

Motion blur augmentation §3.2 fig. 4. Besides scale, translation, colour jitter and rotation, training adds synthetic motion blur. Blur kernels come from the two methods the paper cites for this, and are applied by 2D filtering §3.2. The kernel size and length distribution are not stated.

Body module §3.3 eq. 6. The body module takes a crop around one person and predicts the body pose, shape and camera. It starts from SPIN, which was built for SMPL, and fine-tunes it for SMPL-X §3.3. Fine-tuning uses Human3.6M and pseudo-ground-truth from EFT. Only pose and 2D keypoints are used, because the SMPL shape labels do not match SMPL-X §3.3.

[\phi_b, \theta_b, \beta_b, c_b] = M_B(I_b) eq. 6

Face module §3.4 eq. 7. The face module is the off-the-shelf RingNet model, which is built on FLAME §3.4. It returns face pose and expression. The expression vector has 50 values, of which only the first 10 are kept, because SMPL-X uses 10 principal components of the same expression space §3.4.

[\theta_f, \psi_f] = M_F(I_f) eq. 7

Copy-paste integration §3.5 eq. 8 fig. 5. The fastest merge takes the body's shape, camera and global orientation; its joints except the wrists; and the hand modules' finger poses. Each wrist is set from the hand module's global wrist orientation, converted to a local rotation relative to its parent in the body skeleton §3.5. The conversion uses the body's forward kinematics §3.5. The authors say this costs almost no extra compute §3.5.

(\phi_w, \beta_w, c_w, \hat\theta^b_w, \theta^{lh}_w, \theta^{rh}_w) = (\phi_b, \beta_b, c_b, \hat\theta_b, \theta_{lh}, \theta_{rh})

(\theta^{lwrist}_w, \theta^{rwrist}_w) = (\Gamma_l(\theta_b, \phi_{lh}), \Gamma_r(\theta_b, \phi_{rh})) eq. 8

Here θ̂_b are the body pose parameters without wrists, and Γ_l and Γ_r are the global-to-local wrist conversions §3.5.

# Copy-paste integration  [§3.5] [eq. 8]
# [code: integration/copy_and_paste.py#L94-L126]
# Input:  body (rotations, betas, cam); left and right hand outputs
# Output: whole-body SMPL-X pose, shape and camera

def integrate_copy_paste(body, left, right):
    # Shape and camera come only from the body module  [§3.5] [eq. 8]
    betas = body.betas
    cam = body.cam
    # Keep the body joints, except the two wrists  [§3.5]
    pose = body.rotations_without_wrists
    # Copy the finger poses from the two hand modules  [§3.5] [eq. 8]
    left_fingers = left.pose[3:]                  # (45,)
    right_fingers = right.pose[3:]                # (45,)
    # Set each wrist from the hand's global orientation  [§3.5] [eq. 8]
    pose.right_wrist = to_local(body, right.pose[:3], joint=21)
    pose.left_wrist = to_local(body, left.pose[:3], joint=20)
    return smplx_forward(betas, pose, left_fingers, right_fingers, cam)

# Global wrist orientation to local rotation  [§3.5] (Γ)
# [code: integration/copy_and_paste.py#L11-L38]
# Input:  body rotations (24, 3, 3); global rotation (3, 3); joint index
# Output: local rotation (3, 3), relative to the wrist's parent

def to_local(body, global_rotation, joint):
    # Multiply the body's local rotations from the root down to the
    # wrist's parent (the elbow for a wrist)  [code: copy_and_paste.py#L24-L30]
    chain = identity()
    for ancestor in path_from_root_to_parent(joint):
        chain = chain @ body.rotation[ancestor]
    # Local rotation is the transpose of the chain times the global one
    # [code: integration/copy_and_paste.py#L32-L33]
    return transpose(chain) @ global_rotation

Optimisation integration §3.5 eq. 9 fig. 5. The slower merge fits all SMPL-X parameters to the image. Its cost has three parts: F_2d, a reprojection error between 2D keypoints from OpenPose and the projected hand, body and face joints; F_mesh, a 3D distance between the whole-body hand mesh and the hand module's mesh vertices placed on the body's wrist joints; and F_pri, a shape prior §3.5. The authors replace explicit pose priors with EFT-style network fine-tuning, so F_pri is only ‖β_w‖² §3.5. The fit runs in two stages. Stage one leaves out F_mesh. Stage two adds F_mesh and an extra 3D term F_3D that keeps the joints near their stage-one locations §3.5. F_3D is not in equation 9.

\mathcal{F}(\phi_w, \theta_w, \beta_w, c_w) = \mathcal{F}^{2d} + \mathcal{F}^{mesh} + \mathcal{F}^{pri} eq. 9

# Two-stage optimisation of the whole-body model  [§3.5] [eq. 9]
# Input:  initial params (phi, theta, beta, cam); OpenPose 2D keypoints;
#         hand meshes from the hand modules
# Output: fitted whole-body params

def fit_whole_body(params, keypoints_2d, hand_meshes):
    # Stage 1: 2D reprojection plus shape prior, no mesh term  [§3.5]
    params = minimise(
        f_2d(params, keypoints_2d) + f_pri(params), params)
    # Stage 2: add the wrist-mesh term and hold joints near stage 1  [§3.5]
    fixed_joints = joints(params)
    cost = f_2d + f_mesh(params, hand_meshes) + f_pri
    cost = cost + f_3d(params, fixed_joints)
    return minimise(cost, params)

Wrist integration network §3.5 eq. 10 app. A2. The middle merge uses a six-layer MLP that adjusts the arm after copy-paste so that the wrist lands where the hand module saw it, without gradient steps §3.5. Its input is the arm pose and a 2D direction d. The direction is the difference between the wrist from the body and the wrist from the hand, divided by arm length §3.5. The network works on the shoulder as a global orientation and the elbow as a local rotation app. A2.1. The paper calls the network an approximation of the optimiser's gradient §3.5.

\tilde\theta^{arm}_w = \mathcal{H}(\theta^{arm}_w, d) eq. 10

# Wrist integration network  [§3.5] [eq. 10] [app. A2.1] [app. A2.2]
# Input:  arm pose after copy-paste (2, 3); direction d (2,)
# Output: updated arm pose (2, 3)

def refine_arm(arm_pose, direction):
    # Convert the shoulder to a global orientation; elbow stays local
    # [app. A2.1] [eq. 11]
    arm_global = to_global_shoulder(arm_pose)
    # Stack the 2 x 3 arm pose and the 2-d direction  [app. A2.2]
    inputs = concat(arm_global, direction)        # (8,)
    # Six-layer MLP, hidden widths 128, 256, 512, 256, 128  [app. A2.2]
    outputs = mlp(inputs)                         # (6,)
    # Convert the shoulder back to a local rotation  [app. A2.1] [eq. 13]
    return to_local_shoulder(outputs)

The appendix writes the network's direct output as a different quantity from the final arm pose app. A2.1 eq. 12. Equations 11 to 13 are the appendix forms app. A2.1:

\theta^{arm}_w = [\Upsilon_s(\theta^s_w, \theta_w), \theta^e_w] eq. 11

\ddot\theta^{arm}_w = \mathcal{H}(\theta^{arm}_w, d) eq. 12

\tilde\theta^{arm}_w = [\Gamma_s(\phi_w^s, \theta_w), \tilde\theta^e] eq. 13

Here Υ_s converts the shoulder's local rotation to a global orientation by running the body's kinematics down to the shoulder, and Γ_s converts back [app. A2.1]. The network is trained on right arms only, and left arms are flipped vertically at inference app. A2.2.

Training data for the wrist network app. A2.3 fig. 13. The authors captured videos of varied arm motion and ran the optimiser to get pseudo-ground-truth poses. They then added random noise to the shape β_w, shifted the projected right-hand joints by a random 2D offset d̂, and re-optimised only the right shoulder and elbow to get the target arm pose app. A2.3. The noise and offset ranges are not stated.

Bounding boxes and inputs app. A3. The body box comes from a lightweight OpenPose implementation app. A3. Face and hand boxes come from OpenPose, or for the live demo from projecting the body estimate's face and hand parts into the image app. A3. The hand module reads crops resized to 224 × 224 [app. A1.1].

Hyperparameters

Training data for the hand module tab. 5 app. A1.2: the authors use five datasets for training and one for validation. FreiHAND and HO-3D are split 80/20 into training and validation. MTC is filtered to remove samples where the hands are hidden or too small. STB and RHD use their training splits. MPII+NZSL is used only for validation app. A1.2.

tab. 5 is an appendix table. Sample counts are as printed.

Results

Author-reported throughout. Hand benchmarks: STB and RHD with 3D AUC over 20 to 50 mm; MPII+NZSL with 2D AUC over 0 to 30 px §4.1. Body: 3DPW test set, with the body module compared only to HMR-based methods §4.2; training used COCO and Human3.6M §4.2. Whole-body: EHF with V2V and PA-V2V; SMPLify-X and ExPose were run from their official code §4.3. The hardware used is not stated. Lower is better for the EHF and 3DPW error rows, and higher is better for AUC rows.

Hand module against hand-only methods tab. 1. Values are as printed; bold marks the best in each row.

The paper says the hand module is comparable to hand-only methods that use MANO, even though its hand model has restricted shape variation §4.1. Kulon's code is not public, and the authors could not compare on in-the-wild scenes against it or against CMR, which needs known camera intrinsics §4.1.

Hand augmentation ablation tab. 2 fig. 7. 2D AUC on MPII+NZSL.

Motion blur adds 0.033 (0.622 to 0.655) in this table, the largest step tab. 2. The qualitative figure shows gains on blur, pose variation and occlusion §4.1 fig. 7.

Hand dataset ablation tab. 6 app. A1.3. 2D AUC on MPII+NZSL.

Body module against body-only methods tab. 3. Values are as printed; bold marks the best in each row.

The body module is described as comparable, including to ExPose §4.2. The authors suggest the small gap may come from training on fewer datasets §4.2.

Whole-body integration against SMPL-X methods tab. 4 fig. 8 fig. 9. EHF, errors as printed; units are not stated in the paper. Bold marks the best in each row.

FM-CP, FM-WI and FM-OPT are the copy-paste, wrist-network and optimisation merges. The text says the optimisation variant is best in general, and that the wrist network beats copy-paste §4.3. Both claims hold for the "All" rows but not for every part: copy-paste has the lowest body V2V, and ExPose has the lowest face V2V tab. 4. The paper describes its face output as comparable to other methods §4.3.

Paper vs code

The repository is facebookresearch/frankmocap, the official repository named in the abstract (abstract). The copy checked out at code_commit is the one used for every citation below. Its top-level LICENSE is Attribution-NonCommercial 4.0 International, so the code is non-commercial. The same file also carries a BSD-3-Clause notice for the SPIN-derived body-module parts. No code is quoted in this card. Code is cited only by path and line, and the mapping is described in our own words.

• Left-hand flip axis. Paper: left hands are flipped vertically for training and inference §3.2 app. A2.2. Code: the image is mirrored left to right hand_mocap_api.py. Paper gives no rule for returning the pose; the code negates the y and z components of each axis-angle triple, the x component of the vertices and joints, and the x translation hand_mocap_api.py.
• Hand regression head. Paper: an encoder-decoder where the decoder regresses the parameters from features in one pass, with two extra FC layers §3.2 app. A1.1. Code: after ResNet-50 features and one hidden layer of width 1024, three residual refinement steps start from a mean hand and each step adds a correction computed from the features and the current estimate h3dw_networks.py. The mean hand has camera scale 5.0, mean pose from a pickle, and zero global rotation and shape h3dw_model.py.
• Hand boxes at test time. Paper: boxes come from OpenPose 2D hand keypoints app. A1.1. Code: the default demo runs a hand detector demo_frankmocap.py. The fast mode, which the paper's live demo describes, takes boxes from the body estimate's hand joints with a 20% margin body_mocap_api.py demo_frankmocap.py.
• Merge frame. Paper: the wrist conversion compares the hand orientation with the body's forward kinematics §3.5 but does not give the frame. Code: the chain starts at the body's root rotation and the hand's global orientation is applied in that frame, for both wrists copy_and_paste.py copy_and_paste.py.
• Shape in the merge. Paper: the whole-body shape comes only from the body module eq. 8. Code: consistent; the hand shape values are stored but not used for the output mesh copy_and_paste.py.
• Missing merges. Paper: copy-paste, optimisation and wrist network §3.5. Code: the demo calls only copy-paste demo_frankmocap.py. There is no optimisation code and no wrist-network code or weights.
• Missing face module. Paper: RingNet with 10 expression values §3.4. Code: no face estimator and no RingNet or FLAME code. The demo and merge handle body and hands only demo_frankmocap.py. Face outputs in tab. 4 cannot be reproduced from the repository.
• Training code absent. Paper: loss weights, optimiser and training data for the hand module §3.2 app. A1.1; body fine-tuning §3.3. Code: inference only. The option file has training defaults, such as batch size 128 base_options.py, but no training loop, so the loss weights and schedules cannot be checked against code.
• Body checkpoint loading. Code loads the body checkpoint with strict matching turned off body_mocap_api.py. Paper: not stated.
• GPU only. Code asserts CUDA for the demo and creates CUDA tensors in the hand model demo_frankmocap.py h3dw_model.py. Paper: no hardware given for the runtimes in tab. 4.

Checked and consistent: the hand pose layout (3 global + 45 finger values, 21 hand joints) or in the 224 × 224 hand crop h3dw_model.py h3dw_model.py.

Pitfalls

• Left-hand flip. The paper says vertical, the code mirrors horizontally, and the paper gives no rule for undoing the flip. Take the code's rule when reproducing the hand outputs for left hands, and test it on a known left hand.
• Pose vector length. The paper treats the hand global orientation φ_h (3 values) and finger pose θ_h (45 values) as separate §3.1. The code concatenates them into 48 values base_options.py hand_mocap_api.py. Slice them as the code does, or the pose will be shifted by three values.
• Head design. The paper's hand head is one pass to parameters. The code runs three residual steps from a mean hand. A literal one-pass implementation will not match the code's outputs.
• Vertex count. The text says 10,745 vertices and the matrix dimension says 10475 §3.1. Check the count against the SMPL-X model file you load.
• Copy-paste shape. Only the body's shape is used eq. 8. Do not average or copy the hand shapes into the merged mesh.
• Frame of the hand orientation. The paper does not say which frame the hand global orientation is in. The code applies it in the body's root frame. An implementation that picks a different frame will not match the code's wrist angles, and the paper gives no way to check which frame is right.
• Optimisation details. Iteration counts, optimiser, step sizes and convergence test are not stated, and no code is released. F_3D appears only in the text, not in equation 9 §3.5. Include it in stage two if you implement the optimisation.
• Wrist network details. Training noise, shift ranges and the number of captured sequences are not stated app. A2.3. The paper also says that left arms are flipped vertically, which the code cannot confirm.
• Face module. Only the first 10 of 50 RingNet expression values are kept §3.4. There is no face code, so this step cannot be checked against code.
• Body checkpoint. Loading with strict matching off can silently skip mismatched weights body_mocap_api.py. Log the missing and unexpected keys before trusting body outputs.
• Loss mix. Two of the three hand losses are squared L2 and one is L1 §3.2. The weights are given, but without training code the mix cannot be verified.
• Blur is the largest single gain. The 0.033 2D AUC gain from blur in tab. 2 comes from one validation set tab. 2. Blur kernel parameters are not stated, so results may not transfer to other blur models.
• Ambiguous "best in general". The optimisation merge is not best on every row tab. 4. Compare per row, not on the headline.
• Units and baselines. EHF units and 3DPW units are not stated tab. 3 tab. 4. The source of each baseline number is not stated, except that Kulon's code is not public §4.1.
• Cost. Copy-paste is cheap, the wrist network is cheap §3.5, and the optimisation takes 2 s per frame in the authors' table tab. 4. The 60 s runtime for SMPLify-X is as printed; the hardware is not stated.

Lineage

• Builds on: SMPL-X (Pavlakos et al., 2019), the parameterisation for all parts §3.1. The body module builds on SPIN (Kolotouros et al., 2019), with EFT (Joo et al., arXiv:2004.03686) as the fine-tuning source §3.3. The hand module follows HMR-style body methods §3.2. The face module is RingNet (Sanyal et al., 2019) §3.4.
• Compared against: SMPLify-X (Pavlakos et al., 2019), ExPose (Choutas et al., 2020), MTC (Xiang et al., 2019) and hand-only methods such as Bouk, Baek, Ge, Zhang, Kulon and CMR §4.1 §4.3.
• Superseded by: not stated in this wiki.