MotionBERT: A Unified Perspective on Learning Human Motion Representations

In plain words

Reading a person's movement from a video is much easier when you already know how human bodies move. This paper trains one model on that general knowledge first, then reuses it for several jobs: estimating 3D poses, recognising actions, and building 3D body meshes.

The pretraining game is a fill-in-the-blanks puzzle. The model sees a stick-figure version of motion, with some joints and frames blanked out and some jitter added, and must recover the full 3D motion. Motion capture data gives the true 3D answers. Ordinary videos, which have only 2D joint positions, are used by checking the guess against those 2D positions. Because the model has to fill gaps, it learns how joints are linked and how movement changes over time.

Its architecture runs two kinds of attention in parallel, one across the body at each moment and one across time for each joint, then blends the two with learned weights. After pretraining, each task only needs a small head added on top, and the whole network is trained a little more on that task.

TL;DR

• Pretrains a 5-block DSTformer (spatial and temporal attention in two parallel streams with learned fusion) to recover 3D motion from corrupted 2D skeleton sequences §3.1 §3.2 §3.3.
• Uses mocap (AMASS, Human3.6M) with 3D targets and PoseTrack and InstaVariety videos with 2D-only targets §4.2 app. A.1.
• Finetunes with a linear layer or one-hidden-layer MLP on 3D pose, NTU-RGB+D action recognition, and SMPL mesh recovery §3.4.
• Author-reported: scratch 3D pose MPJPE 39.2 mm and finetuned 37.5 mm on Human3.6M with detected 2D poses tab. 1; 93.0% and 97.2% X-Sub and X-View on NTU-RGB+D tab. 2.
• The official code differs from the paper in several training details (see Paper vs code).

Problem

Human-centric video tasks (3D pose, action recognition, mesh recovery) are usually trained separately, each from its own data. Mocap has accurate 3D labels but only indoor scenes. Action datasets have semantic labels but little pose data. In-the-wild video is diverse but has no 3D labels and costly 2D labels §1 fig. 1. The paper argues that one shared motion representation learned from all these sources can serve every task §1.

Method

The pipeline has two stages: unified pretraining, then task-specific finetuning §3.1 fig. 1. Both stages take 2D skeleton sequences as input, because they can be extracted from many sources §3.1.

Motion encoder (DSTformer) §3.2. Input is a sequence of T frames with J joints and C_in channels (x, y and confidence). The model projects each joint to a feature of width C_f, adds a learnable spatial and a learnable temporal position embedding, runs N dual-stream blocks, and applies a linear layer with tanh to get the motion representation E §3.2. A linear layer on E then gives 3D motion §3.2.

• Spatial block S: multi-head self-attention across joints within one frame, then residual, LayerNorm, MLP and residual §3.2 eq. 1 eq. 2.
• Temporal block T: the same operation across frames for each joint §3.2 eq. 3.
• Dual stream §3.2 eq. 4: the first stream applies S then T, the second applies T then S. Their outputs are blended per frame and per joint with weights from a learned linear map over the concatenated outputs, normalised with softmax over the two streams eq. 5. Blocks do not share weights.

Unified pretraining §3.3. The pretext task is 2D-to-3D lifting.

• For mocap data, the 2D input is the orthographic projection of the 3D motion §3.3. The 2D input is then corrupted by random joint-level and frame-level masks and by added noise, to imitate detector failures §3.3 app. A.1.
• The 3D loss is a joint position error L_3D plus a velocity error L_O on frame-to-frame differences, weighted by λ_O eq. 6.
• For in-the-wild video, there is no 3D target. The model projects its 3D estimate back to 2D and compares it with the detected 2D joints, weighted by visibility or detection confidence eq. 7.
• Total loss: 3D term plus λ_O times velocity term for 3D data, and 2D term for video data eq. 8.

Task-specific finetuning §3.4. The authors keep the heads small. For 3D pose, the whole pretrained network is reused, with no extra masks or noise at test time §3.4. For action recognition, features are average-pooled over persons and time and fed to an MLP with one hidden layer §3.4; one-shot recognition adds a linear layer and uses supervised contrastive learning §3.4 app. A.1. For mesh recovery, an MLP regresses per-frame SMPL pose and a second MLP regresses one shape vector per clip from temporally pooled features §3.4 eq. 9 eq. 10.

Pseudocode: pretraining step (paper version)

# Pretraining step: 2D-to-3D lifting on corrupted 2D skeletons  [§3.3]
# Input:  motion_2d (B, T, J, 3) with confidence; motion_3d or None
# Output: scalar loss

def pretrain_step(motion_2d, motion_3d, has_3d, has_detection_noise, conf):
    # Mask joints (5%) and frames (10%) of the 2D input  [app. A.1]
    corrupted = mask_joints_and_frames(motion_2d)
    # Add noise unless the 2D input already carries detector noise  [§3.3]
    if not has_detection_noise:
        corrupted = add_noise(corrupted)
    # Estimate 3D motion with the encoder  [§3.2] [eq. 4]
    pred_3d = encoder(corrupted)                  # (B, T, J, 3)
    if has_3d:
        # Joint position loss plus lambda_O times velocity loss  [eq. 6]
        loss = joint_error(pred_3d, motion_3d)
        loss = loss + lambda_o * velocity_error(pred_3d, motion_3d)
    else:
        # Reproject to 2D; weight by visibility or confidence  [eq. 7]
        loss = weighted_reprojection_error(pred_3d, motion_2d, conf)
    return loss

Pseudocode: one dual-stream block (paper version)

# One dual-stream block: two stream orders and adaptive fusion  [§3.2]
# Input:  features (B, T, J, C)
# Output: fused features (B, T, J, C)

def dual_stream_block(features):
    # Spatial attention first, then temporal attention  [eq. 4]
    st_stream = temporal_block(spatial_block(features))
    # Temporal attention first, then spatial attention  [eq. 4]
    ts_stream = spatial_block(temporal_block(features))
    # Learned per-(frame, joint) weights over the two streams  [eq. 5]
    weights = softmax(linear(concat(st_stream, ts_stream)))
    return weights[..., 0] * st_stream + weights[..., 1] * ts_stream

Hyperparameters

Values in this table come from the paper unless the row says "code". Where the paper says nothing, the cell says "not stated".

Results

Author-reported. Setup: Human3.6M (subjects 1, 5, 6, 7, 8 train; 9, 11 test), MPJPE in mm after root alignment §4.3. Action results use NTU-RGB+D and NTU-RGB+D-120 with HRNet 2D skeletons §4.4. Mesh results use Human3.6M and 3DPW, adding COCO during training §4.5. Table numbers follow source order in the LaTeX and have not been checked against the compiled PDF.

3D pose, Human3.6M tab. 1

The paper uses Stacked Hourglass 2D skeletons for the detected-2D rows §4.3. Baseline rows are the values listed in the table, which the authors say are the best reported by each work tab. 1 (caption).

Skeleton-based action recognition tab. 2

Human mesh recovery tab. 3 (Human3.6M MPVE / MPJPE / PA-MPJPE, then 3DPW)

The "+ SPIN" and "+ HybrIK" rows use the paper's refiner module on top of those image-based predictors §4.5; the paper does not give the refiner's hyperparameters beyond the loss weights in the table above. Among rows compared in the same table, the HybrIK-refined row is the lowest on every column.

Ablations (author-reported, Human3.6M unless stated)

• Pretraining steps, cumulative tab. 5: from scratch 39.2 mm MPJPE and 87.7% X-Sub; plain 2D-to-3D pretraining 38.8 mm and 89.4%; with noise 38.1 mm and 90.7%; with masks 37.4 mm and 91.9%; with 2D video 37.5 mm, 65.5 mm MPVE and 93.0%.
• Frozen encoder tab. 4: random encoder 404.4 mm MPJPE, 47.6% X-View, 46.8% one-shot; pretrained encoder 40.3 mm, 87.3%, 60.7%.
• Backbones tab. 6: TCN 50.1 mm scratch and 47.9 mm finetune; PoseFormer 44.8 mm scratch and 41.5 mm finetune.
• Architecture variants, MPJPE mean ± std over 5 runs tab. 7: S-T only 40.58 ± 0.31; T-S only 41.05 ± 0.24; sum of S and T 41.76 ± 0.22; ST-MHSA without MLP 41.54 ± 0.35; averaged two-stream fusion 39.87 ± 0.32; adaptive two-stream fusion (DSTformer) 39.25 ± 0.27.
• Depth, heads and channels tab. 8: best 39.2 mm at N = 5, h = 8, C = 512.
• Mask ratio fig. 7: pretraining helps finetuning only while the mask ratio stays below about 45%.

Paper vs code

Compared at commit 705d3a95354db8bdb696b3492e47a3b5537174ff of Walter0807/MotionBERT, the repository linked from the paper's abstract and project page. Its LICENSE file is Apache-2.0.

• Pretraining loss has an extra scale term. Paper's total loss has only the 3D and velocity terms eq. 8. Code adds lambda_scale * n_mpjpe with lambda_scale: 0.5 train.py MB_pretrain.yaml. The paper does not mention this term.
• Optimizer. Paper says Adam app. A.1. Code uses AdamW with weight decay 0.01 train.py MB_pretrain.yaml.
• Learning rate decay. Paper does not state a decay schedule. Code multiplies the learning rate by lr_decay: 0.99 after every epoch train.py MB_pretrain.yaml.
• 3D pose epochs. Paper: 60 epochs from scratch and 30 for finetuning app. A.1. Code configs: 120 and 60 epochs MB_train_h36m.yaml MB_ft_h36m.yaml. Learning rate and batch size match (2e-4, 32).
• Backbone learning rate in 3D pose. Paper says finetuning uses 0.1 × the head rate §4.1. The 3D pose finetune code builds one AdamW optimizer over all parameters with one learning rate train.py, so this ratio is not in that code path. The action and mesh configs do use separate lr_backbone and lr_head values with a 0.1 ratio MB_ft_NTU60_xsub.yaml MB_ft_pw3d.yaml.
• Action recognition, from scratch. Paper: 200 epochs at lr 1e-3 app. A.1. Code MB_train_NTU60_xsub has 300 epochs and lr 1e-4 for backbone and head MB_train_NTU60_xsub.yaml MB_train_NTU60_xsub.yaml.
• One-shot batch size. Paper: batch 16, formed as 8 action pairs app. A.1. Code config: batch_size: 32, 50 epochs MB_train_NTU120_oneshot.yaml. The code's pair-batching logic was not checked.
• Velocity and 3D losses use mean, not sum. Paper writes both as sums over frames and joints eq. 6. Code uses means for both loss_mpjpe and loss_velocity loss.py loss.py. The two terms are normalised the same way, so the λ_O ratio holds, but the absolute scale differs.
• Noise only on data with a 3D target. Code adds noise only when the batch has ground-truth 2D (noise=(args.noise and has_gt)) train.py. Masking applies to all data. PoseTrack batches are marked has_gt=True, and InstaVariety batches are has_gt=False train.py. The paper says extra noise is added unless the input already has detection noise §3.3, which is consistent with this.
• Final LayerNorm before the representation. Code applies a LayerNorm after the last block and before the tanh projection DSTformer.py. The paper describes the tanh layer but not this norm §3.2.
• Fusion weight initialisation. Code initialises the per-block fusion layer to zero weights and 0.5 bias, so the initial blend is 50/50 DSTformer.py. The paper does not state an initialisation.
• Mask and noise ratios match. Joint mask 5% and frame mask 10% in code MB_pretrain.yaml match the appendix. Noise keyframes (27) and std (0.002) match augmentation.py.
• Not verified. Mesh refiner module, PoseTrack and InstaVariety data loaders, and the 2D ablation numbers in tab. 5 were not run or traced in code.

Pitfalls

• Clip lengths differ by data source. Mocap clips are 243 frames; PoseTrack is 30 and InstaVariety 81 app. A.1. A single fixed-length batch will not reproduce the pretraining setup.
• Learning schedule is not fully specified. The paper states the pretraining learning rate and batch size but not decay; the code decays by 0.99 per epoch. Implementations that copy the paper's constants will train differently from the released weights.
• Masked entries are multiplied to zero. The code multiplies the input by a random joint mask and a frame mask, so masked joints and frames become zero in every channel, with no learned mask token augmentation.py. Pipelines that expect a learned mask embedding will behave differently.
• Masking is more sensitive than the paper suggests. Above about 45% masking, pretraining stops helping finetuning fig. 7. The MAE result of 75% does not transfer here, which the authors attribute to skeletons being abstract.
• Depth is missing from the pretext input. The paper says the depth channel, one third of the original data, is intrinsically masked when the input is 2D app. A.2.
• Mesh results rely on refinement for the best numbers. The lowest mesh errors in tab. 3 come from the refiner added to image-based methods, not from the motion encoder alone §4.5. The 2D-only rows are the ones that use only motion input.
• Single-person limitation. The authors state that the representation covers single-person skeletons and is insensitive to appearance, scene and interaction app. A.3.
• Table and figure numbers here follow source order in the LaTeX, not the compiled PDF. Check before citing them externally.
• Duplicate LaTeX label. tab:ablation_arch is defined twice in the source (appendix depth and heads table, and the main-text module-variant table). Cite by content, not by label.

Lineage

• Builds on: masked pretraining and cloze objectives from BERT, MAE and BEiT §3.3; the velocity loss follows earlier 3D pose work cited there, including VideoPose3D §3.3.
• Architecture is related to PoseFormer, MHFormer and MixSTE, which use spatio-temporal Transformers for 3D pose §3.2 app. A.2.
• Compared against on 3D pose: VideoPose3D, MixSTE, PoseFormer, MHFormer, P-STMO tab. 1. On action recognition: CrosSCLR, MCC, Skel-Con, PoseConv3D tab. 2. On mesh recovery: SPIN, MAED, HybrIK, SmoothNet, PoseBERT tab. 3.
• Superseded by: not stated in the paper; no later version is in this wiki.