Parametric body model

Overview

An image-based model sees appearance, not body structure. A silhouette or a frame does not state the body's shape, the pose over time or the viewing angle 2204.02569 §1. A compact 3D body description per frame adds that structure. The three papers so far use it in different ways. In SMPLGait it is an input that conditions a recognition network 2204.02569 §3.2. In FrankMocap it is the output that the estimator must produce [arxiv:2108.06428 §3.1]. In NARF it is an input in a different form: a joint tree gives each bone a rigid frame, and a radiance field is evaluated in those frames 2104.03110 §3.2 §3.3. In the two estimation papers the parameters are per frame and come from images or video, not from a 3D scan 2204.02569 §1 2108.06428 §1. NARF takes its bone poses from SMPL parameters in a dataset, and joint pose estimation is listed as future work 2104.03110 §4 §6.

Variants

Three variants have cards. They differ in role: one feeds a parameter vector into a downstream network, one produces parameters, and one turns bone poses into a renderable model.

SMPL vector driving a spatial transformer

Each frame gives 85 values: 24 joints times 3 pose values, 10 shape values and 3 camera values 2204.02569 §3.2. A three-layer network maps the vector to a transform, which is reshaped into a square matrix and applied to the silhouette feature map as (I + G) 2204.02569 §3.2. The mechanism is a learned viewpoint and shape alignment in feature space, not a rotation of the mesh 2204.02569 §3.2. The authors report 53.20% Rank-1 on Gait3D at 88x128, against 42.60% for GaitSet 2204.02569 tab. 2. Card: SMPLGait paper card.

SMPL-X parameters from separate part regressors

FrankMocap estimates the face, each hand and the body with separate SMPL-X-based regressors, all from one RGB image 2108.06428 §3. The SMPL-X vector has 21 body joints with 3 axis-angle values each, 15 joints per hand, one jaw joint, a shape vector and ten expression values 2108.06428 §3.1. The outputs are then merged into one whole-body mesh in one of three ways 2108.06428 §3.5:

• Copy-paste. Body shape and camera come from the body module, finger poses from the hand modules, and each wrist is set from the hand's global orientation converted through the body's kinematic chain 2108.06428 eq. 8.
• Wrist-correction network. A six-layer MLP adjusts the arm after copy-paste, so that the wrist lands where the hand module saw it, without gradient steps 2108.06428 §3.5 2108.06428 app. A2.2.
• Per-image optimisation. All SMPL-X parameters are fitted to 2D keypoints, the hand meshes and a shape prior, in two stages 2108.06428 eq. 9.

Author-reported EHF All V2V errors are 69.9 for copy-paste, 69.3 for the wrist network and 63.5 for optimisation, against 81.9 for ExPose and 146.2 for SMPLify-X 2108.06428 tab. 4. The table gives 0.15 s per frame for the first two merges and 2 s for optimisation; the hardware is not stated 2108.06428 tab. 4. Card: FrankMocap paper card.

Bone frames inside a neural radiance field

NARF has no parameter vector. The kinematic model is a tree of joints joined by bones of fixed length. Forward kinematics gives each bone a rigid pose (R, t) 2104.03110 §3.2. A global point x is mapped into a bone's frame as R^-1 (x - t), and the view direction as R^-1 d 2104.03110 §3.3. The recommended variant, NARF_D, has one small occupancy network per bone working in that bone's frame. A softmax over bones gives each point a probability per bone. Each bone's encoded inputs are scaled by that probability and fed to one NeRF 2104.03110 §3.4. The other two variants are NARF_P, one NeRF per part, and NARF_H, one NeRF on all parts concatenated 2104.03110 §3.4.

On the authors' synthetic renders of human bodies, NARF_D has the best mask, PSNR and SSIM in all four test settings 2104.03110 §4. Its novel pose, novel view PSNR is 27.24 and its mask error 123.8, against 26.50 and 125.8 for NARF_P and 23.98 and 374.6 for NARF_H 2104.03110 §4. The cost table gives NARF_D 205M FLOPs and 382K memory per ray, close to P-NeRF at 156M and 356K, and far below NARF_P at 2140M and 6544K 2104.03110 tab. 1. Card: NARF paper card.

When to use which

Implementation sketch

The sketch follows the paper's description of the training step for SMPLGait 2204.02569 §3.2 2204.02569 §3.3. Dimensions are per batch of identities; the paper uses 30 frames per sequence 2204.02569 §5.1.3.

# Input: silhouettes (B, S, H, W); smpl vectors (B, S, 85); labels (B,)

def training_step(silhouettes, smpl, labels):
    # 1. Per-frame appearance feature maps from a six-layer CNN [§3.2] [eq. 1]
    maps = silhouette_cnn(silhouettes)            # (B, S, C, h, w)

    # 2. Per-frame transform vector from the 85 SMPL values [§3.2] [eq. 2]
    vectors = smpl_transform_net(smpl)            # (B, S, n*n)

    # 3. Reshape to a square matrix G and add identity [§3.2]
    G = reshape_square(vectors) + identity        # (B, S, n, n)

    # 4. Zero-pad each map to square, then multiply by (I + G) [eq. 3]
    aligned = pad_to_square(maps) @ G[:, :, None]

    # 5. Pool over frames, then horizontal pyramid pooling [§3.2]
    seq = horizontal_pyramid(max_over_frames(aligned))

    # 6. Triplet loss and cross-entropy, weighted 1.0 and 0.1 [§3.3] [eq. 4]
    return 1.0 * triplet(seq, labels) + 0.1 * cross_entropy(seq, labels)

The paper does not give the pooling operator for frames beyond the name "Set Pooling" 2204.02569 §3.2. The sketch uses a max, which is what the released code does; see Contradictions.

The FrankMocap variant reverses the interface: the estimator outputs the vector, and the merge assembles it from parts. The wrist step needs the body's forward kinematics to convert the hand orientation to a local rotation, and the paper does not state the frame in which that orientation is given 2108.06428 §3.5.

The NARF variant has a different core. There is no parameter vector; the per-bone frames are the input. The snippet covers only the step that maps points and directions into each bone's frame 2104.03110 §3.3.

# Map points and directions into each bone's frame  [§3.2] [§3.3]
# bones: list of (R, t) from forward kinematics over the joint tree

def to_bone_frames(x, d, bones):
    # R is a rotation, so its inverse is its transpose  [§3.3]
    local_x = [R.T @ (x - t) for R, t in bones]   # each (3,)
    local_d = [R.T @ d for R, _ in bones]         # each (3,)
    return local_x, local_d

Contradictions

• Input resolution. Headline results are at 88x128 2204.02569 tab. 2. The released configs for this method use 64-pixel silhouettes, which gives 44x64 input, and no 128-pixel config is in the repository smplgait_gait3d.yaml.
• Training length. The paper trains for 1,200 epochs with learning-rate steps at epochs 200 and 600 2204.02569 §5.1.1. The released config uses 180,000 iterations with steps at 30,000 and 90,000 smplgait_gait3d.yaml. The number of iterations per epoch is not stated, so the two cannot be converted.
• Feature grid. The paper sets the grid size to max(h, w) 2204.02569 app. A.1. The released model hard-codes a 16 x 16 grid smplgait.py.
• Input order. The paper lists pose, shape and camera 2204.02569 §3.2. The preprocessing code concatenates camera, pose and shape, which keeps the same length of 85 pretreatment_smpl.py.
• Released configs. Two configs in the same repository differ: one uses cosine similarity with Adam, the other Euclidean distance with SGD smplgait.yaml. The paper states cosine and Adam 2204.02569 §3.3 2204.02569 §5.1.1. Which one produced the reported numbers is not stated.
• Merge ranking. The paper says the optimisation merge is best in general, and that the wrist network beats copy-paste 2108.06428 §4.3. Both hold for the All rows of tab. 4. They do not hold for every part: copy-paste has the lowest Body V2V (83.7, against 83.9 for the wrist network and 93.1 for optimisation), and ExPose has the lowest Face V2V (19.5) 2108.06428 tab. 4.
• Face output. The paper describes its face output as comparable to other methods 2108.06428 §4.3. In tab. 4 its Face V2V is 30.7 for copy-paste and the wrist network, 20.4 for optimisation and 19.5 for ExPose 2108.06428 tab. 4.
• Vertex count. The FrankMocap text gives 10,745 SMPL-X vertices, while its matrix dimension gives 10475 2108.06428 §3.1. The paper does not say which is right.
• NARF autoencoder text versus table. The appendix says NARF_D outperforms the other methods on all metrics and settings. In its own autoencoder table the CNN baseline is best in 8 of 12 metric cells [arxiv:2104.03110 app. "Ablation Studies for AutoEncoder"]. The paper does not say which statement the authors intended, so the table's numbers are the ones to rely on.

Interpretation of the transform as viewpoint normalisation is the authors' own reading 2204.02569 §1 2204.02569 §3.2.

Papers

• SMPLGait paper card: SMPLGait, a silhouette CNN with a 3D spatial transformer driven by SMPL parameters, and the Gait3D benchmark.
• FrankMocap paper card: FrankMocap, monocular whole-body capture from separate SMPL-X regressors and three merge options.
• NARF paper card: NARF, a neural radiance field over bone frames of an articulated body, with a soft part assignment (NARF_D).