wikillm

Paper · 2023-03-28 · cs.CV

One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer

OSX recovers SMPL-X body, hands and face from one image in a single network. A ViT encoder regresses the body and yields a feature map; a decoder upsamples that map, crops hand and face with RoIAlign and refines keypoint-guided queries with deformable attention. Author-reported: EHF all-part MPVPE 70.8 vs 76.8 for Hand4Whole trained with extra hand/face data.

Human mesh recoveryParametric body modelVision Transformer (ViT) backbonesObject queries (learned query slots in transformer decoders)human-mesh-recoverywhole-bodytransformerone-stage

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Most systems that estimate a whole person from one photo run several separate networks: one for the body, one for each hand, one for the face. Each network gets a cropped, enlarged view of its part, and the results are pasted together afterwards. The parts do not know about each other, so a wrist can end up at an unnatural angle.

OSX does the job with one network. A large image encoder looks at the whole picture and estimates the body. Hands and face are small in such photos, so the network enlarges the relevant region inside its own feature maps instead of running separate networks on cropped images. Each small part is then decoded with rough keypoints as starting points, and the body and hand estimates share one model.

The authors also release UBody, a dataset of upper-body video scenes with whole-body labels.

TL;DR

A one-stage whole-body mesh recovery network (OSX) with a Component Aware Transformer: a ViT encoder predicts body parameters and a feature map, and a decoder upsamples the map at x1, x2 and x4, crops the hands and face with RoIAlign, and refines keypoint-guided component tokens with deformable attention. Author-reported gains over Hand4Whole come from the paper's tables. Read the Paper vs code section before implementing, because several settings differ.

Problem

Whole-body mesh recovery must estimate body, hand and face parameters from a single image §1. Hands and face cover very few pixels, so a single network at body resolution cannot resolve them. Existing methods therefore detect each part, enlarge it and run a separate expert network, then fuse the results. The authors argue that late fusion cannot make the parts consistent, which gives implausible wrist rotations and unnatural poses §1.

The authors also tested the naive one-stage route on Hand4Whole. Replacing its separate backbones with one shared backbone raised the all-part MPVPE on AGORA-val from 183.8 to 202.3 mm and on EHF from 77.5 to 84.7 mm tab. 2. The main model is trained on COCO-WholeBody, MPII and Human3.6M, without hand-only or face-only datasets §5.1. The paper also introduces UBody, an upper-body dataset of real-life video scenes with 2D and 3D whole-body annotations §4 app. D.

Method

The method is a single encoder-decoder network (called the Component Aware Transformer, CAT). The encoder handles the whole image; the decoder handles the hands and face §3.2 fig. 3.

Overview, in order:

  1. The image is resized to a 256 x 192 network input app. A [code: main/OSX.py#L193].
  2. The encoder is a ViT. Patch tokens are prepended with a set of learnable task tokens and passed through standard transformer blocks §3.3. In the released code the task tokens are shape, camera, expression, jaw, two hand tokens and body-pose tokens OSX.py [code: main/transformer_utils/mmpose/models/backbones/vit.py#L183].
  3. Body regression uses fully connected layers on the body tokens. It outputs the root and body joint rotations (22 joints x 3 values), shape (10) and global translation (3) §3.2 §3.3.
  4. Hand regression outputs left and right hand rotations (15 joints x 3 each); face regression outputs jaw rotation (3) and expression (10) §3.2.
  5. Hand and face bounding boxes are predicted from the feature map. The paper says FFNs on the encoder's feature tokens; the code uses a deconvolution head with soft-argmax centres and small MLPs for sizes (see Paper vs code) §3.4 module.py [code: main/OSX.py#L208].
  6. Feature-level upsampling: the encoder map is reshaped and deconvolved into copies at x1, x2 and x4, three scales by default §3.4 §5.3. RoIAlign then crops each part from every scale using the predicted boxes §3.4 module.py.
  7. Keypoint-guided queries: a 2D keypoint regressor on the lowest-resolution crop gives one reference point per keypoint (the code uses the x2 crop; see Paper vs code). Each component token is the sum of the reference keypoint's feature, a pose positional embedding and learnable embeddings §3.4 OSX.py.
  8. The component tokens go through N deformable attention blocks. Each block has self-attention, multi-scale deformable cross-attention with a few sampling points around the reference keypoint (four in the paper's example), and an FFN §3.4 eq. 1. The code uses six decoder blocks [code: main/transformer_utils/configs/osx/decoder/hand_decoder.py#L120].
  9. The decoded hand and face tokens go to their regression heads. The body, hand and face parameters are then passed to an SMPL-X layer to produce the mesh §3.2.
  10. Training minimises the sum of four L1 terms: SMPL-X parameters, 3D keypoints, projected 2D keypoints and hand/face boxes eq. 2. The released code uses more terms; see Paper vs code.

Pseudocode of the forward pass (the paper gives no algorithm box; this is a summary of the sections above and of the released code):

Pseudocode
# Forward pass for one image  [§3.2] [fig. 3]
# Input:  image (B, 3, 256, 192)  [app. A]
# Output: SMPL-X parameters, 2D and 3D joints

def forward(image):
    # Global encoder: patch tokens and body tokens through a ViT  [§3.3]
    feat_map, body_tokens = encoder(image)  # (B, C, 16, 12), (B, T_b, C)
    # Body parameters from the body tokens  [§3.3]
    body_params = regress_body(body_tokens)  # root, pose, shape, camera
    # Hand and face boxes from the feature map  [§3.4]
    hand_boxes, face_boxes = predict_boxes(feat_map)
    # Upsample to x1, x2, x4 and crop each part with RoIAlign  [§3.4] [§5.3]
    hand_maps = crop_upsampled(feat_map, hand_boxes)  # left hand flipped
    face_maps = crop_upsampled(feat_map, face_boxes)
    # Reference points from the x2 crop  [code: main/OSX.py#L221]
    hand_refs = regress_keypoints(hand_maps[2])
    face_refs = regress_keypoints(face_maps[2])
    # Keypoint-guided deformable decoder per part  [§3.4] [eq. 1]
    hand_tokens = hand_decoder(hand_maps, hand_refs)
    face_tokens = face_decoder(face_maps, face_refs)
    # Hand and face parameters from the decoded tokens  [§3.2]
    hand_params = regress_hand(hand_tokens)
    face_params = regress_face(face_tokens)
    return body_params, hand_params, face_params

Key equations:

Deformable cross-attention for one keypoint query Q at reference point p_q eq. 1. l indexes feature levels, k sampling points; A is the attention weight, W a learnable projection, phi a position rescaling and Delta p a learned offset:

\mathrm{CA}(\mathbf{Q}, \mathbf{V}, p_q) = \sum_{l=1}^{L} \sum_{k=1}^{K} A_{lqk}\, W\, \mathbf{V}_l\big(\phi_l(p_q) + \Delta p_{lqk}\big) eq. 1

Training objective eq. 2:

L = L_{smplx} + L_{kpt3D} + L_{kpt2D} + L_{bbox2D} eq. 2

The paper states that each item is an L1 distance, that L_smplx supervises the SMPL-X parameters, and that the other items supervise 3D whole-body keypoints, projected 2D keypoints, and hand and face 2D boxes §3.4. Loss weights are not given in the paper; the appendix is referred to for "more details" §3.4 but contains no weights app. A.

Training labels are pseudo ground truth: SMPL/SMPL-X fits come from EFT and NeuralAnnot, and the UBody labels from an annotation pipeline that uses this network as the 3D initialiser §5.1 app. D.

Hyperparameters

NameValueWhereNotes
OptimizerAdam§5.1 app. Acode: torch.optim.Adam with its default betas base.py
Adam beta1, beta2beta1 = 0.1, beta2 = 0.999app. Acode does not set them, so defaults (0.9, 0.999) apply base.py. Conflict, see Paper vs code
Learning rate1e-4§5.1 app. Acode default --lr 1e-4 train.py
LR schedulecosine annealingapp. Acode: CosineAnnealingLR over all iterations, eta_min 1e-6 base.py; warm-up not stated
Epochs14§5.1 app. Acode default config.py
Batch size192app. AREADME training command uses 16 README.md; config default 48 config.py. Conflict, see Paper vs code
Network input256 x 192 (H x W)app. Acode input_body_shape config.py
Hand crop inputnot stated-code 256 x 256 config.py
Face crop inputnot stated-code 192 x 192 config.py
Body tokens T_b27§5.1code uses 25 body-pose tokens inside 31 task tokens vit.py. Conflict
Component tokens92§5.1equals 20 hand + 72 face keypoints config.py; see Paper vs code
Upsampling scalesx1, x2, x4 (default); x8 tested§5.3code upscale=4 gives three levels config.py
Sampling points per queryfour§3.4code num_points=4 hand_decoder.py
Decoder blocksN (not stated)§3.4code 6 hand_decoder.py
Attention headsnot stated-code 8 hand_decoder.py
Decoder widthC' = C/2§3.4code uses 256 for the decoder hand_decoder.py. Conflict
Encoder backbone sizenot stated-code default ViT-L: 1024 wide, 24 blocks, 16 heads body_encoder_large.py config.py
Pretrained encodernot stated-code loads osx_vit_l.pth from a download link; its training data is not documented config.py OSX.py README.md
Hand box extensionnot stated-code passes 2.0 for hands and 1.5 for the face OSX.py
Loss weights1 for all termseq. 2code sums about twelve terms, each weight 1; SMPL-X shape term x2 for the AGORA benchmark OSX.py config.py
Data augmentationscaling, rotation, horizontal flip, colour jitter§5.1magnitudes not stated in paper
Training dataCOCO-WholeBody, MPII, Human3.6M§5.1no hand-only or face-only data
Training timenot stated in paper-README: about 20 hours on one NVIDIA A100 for setting 1 README.md

Results

Author-reported. Setup: trained on COCO-WholeBody, MPII and Human3.6M without hand-only or face-only data §5.1; evaluated on AGORA-test, EHF and 3DPW §5.2. Baseline Hand4Whole x is the Hand4Whole variant fine-tuned with extra hand-only and face-only data tab. 3 caption. Metrics are in millimetres.

SettingMetricBaselineThis paperSource
AGORA-test, all partsMPVPE (lower is better)Hand4Whole x: 135.5122.8tab. 3
AGORA-test, all partsN-MPVPE (with misses)Hand4Whole x: 144.1130.6tab. 3
EHF, all partsMPVPEHand4Whole x: 76.870.8tab. 3
EHF, all partsPA-MPVPEHand4Whole x: 50.348.7tab. 3
3DPW, body onlyMPJPEHand4Whole x: 86.674.7tab. 3
3DPW, body onlyPA-MPJPEHand4Whole x: 54.445.1tab. 3
UBody, intra-scene, all partsMPVPEHand4Whole: 104.192.4 (81.9 after fine-tuning on UBody)tab. 5
UBody, inter-scene, all partsMPVPEHand4Whole: 109.4100.7 (82.0 after fine-tuning)tab. S-4

Ablations on EHF tab. 4 (hand MPVPE, lower is better): full model 53.7; without the hand decoder 55.3; without keypoint guidance 55.1; without both 56.4. Upsampling scale: x1 54.9, x2 54.3, x4 53.7 (chosen), x8 54.1. The face decoder ablation gives 26.4 for the full model and 27.2 without it.

Efficiency tab. S-1 (single-person, one NVIDIA A100, same input size): the paper's model takes 54.6 ms per inference, 102.9 M parameters and 25.3 GFLOPs. Hand4Whole takes 73.3 ms, 77.9 M parameters and 16.7 GFLOPs. Its NMJE-All on AGORA-test is 127.6 against 141.1 for Hand4Whole tab. S-1.

Paper vs code

Official code: IDEA-Research/OSX (MIT), commit 14b1cb41a3d2b14c9303290f45fa8189f6f685b3, linked from the authors' project page README.md. The repository was cloned and read at that commit. The paper and code differ in the following places; each item gives both citations.

  • Adam betas. Paper: beta1 = 0.1 app. A. Code: default Adam betas (0.9, 0.999) base.py.
  • Batch size. Paper: 192 app. A. README command: 16 README.md. Config default: 48 config.py. Training script default: 32 train.py.
  • Keypoint level. Paper: keypoints are regressed from the lowest-resolution crop F_lr §3.4. Code: keypoints come from the x2 crop, index [-2] of the reversed scale list OSX.py [code: main/OSX.py#L241]. The lowest-resolution crop is used only for the face global feature OSX.py.
  • Decoder width. Paper: C' = C/2 §3.4. For the default ViT-L (C = 1024) that is 512. Code: the hand and face projections map to 256 and the decoder uses embedding size 256 module.py [code: main/transformer_utils/configs/osx/decoder/hand_decoder.py#L48].
  • Box regression. Paper: boxes are regressed from the encoder tokens with FFNs §3.4. Code: a deconvolution head on the encoder map plus the body joint heatmaps, soft-argmax centres, and MLPs for sizes [code: common/nets/module.py#L116-L148] OSX.py.
  • Body token count. Paper: 27 body tokens §5.1. Code: the ViT defaults to 31 task tokens (1 + 1 + 2 + 2 + 25), of which 25 are body tokens and match the 25 body joints of the SMPL-X joint set [code: main/transformer_utils/mmpose/models/backbones/vit.py#L183] [code: common/utils/human_models.py#L99-L106]. No override appears in the encoder config body_encoder_large.py.
  • Component tokens. Paper: 92 §5.1. Code: 20 hand keypoints and 72 face keypoints config.py. 92 = 20 + 72 suggests the hand decoder runs once for both hands and is shared, which the code does by stacking left and right hands in one batch OSX.py [code: main/OSX.py#L224]. This is an inference from the code; the paper does not say it.
  • Hand flip. The code horizontally flips left-hand features so both hands use one decoder, then flips the left-hand outputs back [code: common/nets/module.py#L199] OSX.py. The paper does not describe this.
  • Loss. Paper: four L1 terms eq. 2. Code: about twelve terms in a dict (SMPL-X pose, shape, expression, 3D joints, SMPL-X joints, three box centre and size pairs, projected joints, image joints, face image joints, SMPL-X image joints), summed with weight 1 OSX.py [code: main/train.py#L70-L73]. Shape term is multiplied by smplx_loss_weight, which is 1 by default and 2 for the AGORA benchmark [code: main/config.py#L37] config.py.
  • Projected joints. The code shifts projected hand and face keypoints by the mean offset to the ground truth over visible joints during training OSX.py. The paper does not mention this step.
  • Gradient stops. Body joint coordinates, box predictions and hand joint coordinates are detached before they feed the next stage [code: main/OSX.py#L202] OSX.py OSX.py [code: main/OSX.py#L226]. The paper does not say which gradients are cut.
  • Pretrained encoder. Paper: not stated. Code loads a ViT-L checkpoint named osx_vit_l.pth OSX.py [code: main/config.py#L62]. Its origin is not documented in the paper or the README.
  • Cosine schedule. Paper: cosine annealing app. A. Code: minimum learning rate 1e-6, no warm-up base.py.
  • Input size. Paper: 256 x 192 app. A. Code matches [code: main/config.py#L19] OSX.py.
  • Decoder blocks. Paper: N, with four sampling points in its example §3.4. Code: six decoder blocks, eight heads, four points [code: main/transformer_utils/configs/osx/decoder/hand_decoder.py#L120] [code: main/transformer_utils/configs/osx/decoder/hand_decoder.py#L131] [code: main/transformer_utils/configs/osx/decoder/hand_decoder.py#L136].
  • Model variants. The repository also provides an encoder-only model and a no-face-decoder model OSX_WoDecoder.py [code: main/OSX_WoFaceDecoder.py]. The README recommends the no-face-decoder model for speed, saying it costs little accuracy README.md. The paper uses the decoder-free model for AGORA-test app. A.

Pitfalls

  • Batch size and optimizer are not reliable from the paper. Batch 192 in the appendix, 16 in the README, 48 in the config. Adam beta1 is 0.1 in the appendix and 0.9 in the code. Use the code's settings unless you have a reason to match the paper; record which one you used.
  • Decoder width. The paper's C' = C/2 gives 512 for ViT-L. The code uses 256. Implementers following the paper will double the decoder cost and mismatch the released weights.
  • Keypoint level. The paper names the lowest-resolution crop; the code uses the x2 crop. Results may differ if you follow the text.
  • Box regression. The paper's FFN-based box head is not what the code does. The code's box head is a small deconvolution network with a soft-argmax centre. Both are plausible; the code is what was evaluated.
  • Left and right hands share one decoder. The left-hand features are flipped before decoding and the outputs are flipped back. Forgetting either step produces mirrored hands.
  • Stopped gradients. Box predictions and body joints are detached, so the hand and face decoders do not train the body or box predictors through those inputs. Changing this changes the training dynamics.
  • Loss terms and weights. The paper's four-term loss is not the training loss. The released loss has about twelve terms with weight 1, and the shape term doubles for the AGORA benchmark.
  • AGORA-test uses the encoder only. The appendix says the decoder is removed for AGORA-test because it does not help there app. A. The README uses the no-decoder setting for that benchmark README.md. Comparing the paper's AGORA-test row with a full model is not like-for-like.
  • Percentages in the text do not all match the tables. AGORA-test: 135.5 to 122.8 is a 9.4 % drop, the text says 9.5 %. 3DPW: 86.6 to 74.7 is a 13.7 % drop, the text says 13.4 % §5.2 tab. 3. UBody: the text claims a 16.1 % gain from training on UBody §5.4; the table gives 92.4 to 81.9, which is 11.4 % tab. 5.
  • Pseudo ground truth. The SMPL/SMPL-X training targets are pseudo ground truth from EFT and NeuralAnnot, not manual labels §5.1. The UBody labels come from the authors' own annotation pipeline app. D.
  • Speed is not the same as size. The paper's model is faster than Hand4Whole but has more parameters (102.9 M vs 77.9 M) and more FLOPs (25.3 G vs 16.7 G) tab. S-1.
  • Encoder size. The paper does not state the encoder size. The released model uses ViT-L, so reproducing the released weights needs that configuration body_encoder_large.py.
  • Hand4Whole x is not a like-for-like baseline. It uses hand-only and face-only data that OSX does not use tab. 3 §5.2.
  • Fine-tuning changes the story. Hand4Whole fine-tuned on AGORA got worse on UBody; the authors attribute this to a data distribution gap §5.4. Results depend on the training mix.

Lineage

  • Builds on: the Vision Transformer encoder §3.3, the feature upsampling of ViTDet §3.4, deformable attention from Deformable DETR §3.4, RoIAlign from Mask R-CNN §3.4 and the SMPL-X model §3.2. None of these are compiled here.
  • Compared against: Hand4Whole, ExPose, PIXIE, FrankMocap and PyMAF-X tab. 3 tab. 5 tab. S-1. Multi-stage methods such as Hand4Whole use separate networks per part §1.
  • Superseded by: not stated in this wiki.

Source and license

Adapted from “One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer” by Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, Yu Li (arXiv:2303.16160), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/IDEA-Research/OSX at commit 14b1cb41a3d2 (MIT). Any quoted code is shown under that license.