Learning from Massive Human Videos for Universal Humanoid Pose Control

In plain words

Teaching a humanoid robot a new movement usually means either training it in a simulator with reinforcement learning, which covers only a narrow set of skills, or recording a person controlling the robot, which costs human effort. Videos of people doing things are everywhere, but a video has no robot joint angles in it.

The paper's answer has two parts. First, it builds a large dataset by pulling human videos, describing each clip in a sentence, estimating the person's 3D body, mapping that body onto the robot's shape, and training a controller that can physically follow the result. Second, it trains a language model-style Transformer on that dataset. The Transformer reads a text command and writes out a sequence of short motion "words" that a decoder turns into robot movements. The result is meant to make the robot respond to plain-text instructions such as "play guitar" or "wave to a friend".

TL;DR

• UH-1 is a Transformer that generates humanoid motion tokens from a text command; a VQ-VAE-style tokenizer turns those tokens into joint trajectories §4.
• Humanoid-X is built from 163,800 single-person clips, each captioned, pose-estimated, retargeted to the robot and tracked by an RL policy §3.1 §3.2.
• Author-reported: on a HumanoidML3D variant, UH-1 reaches FID 0.445 against 0.582 for MDM and 0.667 for T2M-GPT tab. 1; on a real Unitree H1-2 it succeeds in 70-100% of trials per command tab. 3.
• The released code only covers text-to-keypoint inference, the goal-conditioned RL policy and the retargeting scripts; the captioning and pose-estimation stages are "work in progress" in the repository.

Problem

Humanoid robots have many more degrees of freedom than arms, so learning control for them is harder. Reinforcement learning in simulation gives robust locomotion but tends to cover only a few skills such as walking or jumping; teleoperation covers many skills but needs a human operator for every demonstration §1. Human videos are abundant but do not contain robot actions, so a method is needed to convert them into actions a physical humanoid can execute §1 §2.

The paper assumes that human and humanoid skeletons are similar enough that human motion can be mapped to the robot after a shape fit, and that a tracking policy can make the mapped motion physically feasible §3.2 app. B.5 app. B.6.

Method

The work has two halves: a data pipeline (Humanoid-X) and a text-conditioned model (UH-1). Figure 2 summarises the pipeline fig. 2.

Humanoid-X pipeline §3.2 app. B.

1. Mining. Clips come from academic motion datasets, action-recognition datasets and YouTube. For YouTube, more than 400 hand-written search terms are used, with the top 20 results per term §3.2 app. B.2. The source mix is in the table below.
2. Clip extraction. Videos are resampled to 20 FPS. A YOLOv8 detector keeps frames with exactly one person. Motion is measured with frame-to-frame grayscale differences, on two thresholds (a small one per frame and a larger one per batch of frames). Runs of at least 64 consecutive qualifying frames become clips §3.2 app. B.2. The threshold values and the minimum chunk duration are not stated.
3. Captioning. Eight frames per clip go to a Video-LLaMA-style captioner, which is asked for one sentence describing the action and not the person's appearance app. B.3.
4. 3D pose. A video-based SMPL estimator (VIBE) yields shape, joint rotations and a root translation per frame eq. 2 app. B.4. Depth is derived from a weak-perspective camera; the focal length is not stated.
5. Retargeting. The SMPL shape parameters are optimised so the human T-pose matches the robot's T-pose, with each shape coefficient bounded to |β_i| < 5 eq. 4 eq. 5 app. B.5. The human joints are then mapped to 12 robot joints (hips, knees, ankles, shoulders, elbows, wrists) through forward kinematics §3.2 eq. 3 eq. 6. Robot joint angles come from inverse kinematics with a smoothness term weighted by λ = 0.05 eq. 7 app. B.5.
6. Tracking policy. A goal-conditioned PPO policy in Isaac Gym learns to track the retargeted keypoints, joint angles and root motion, trained on a portion of the CMU MoCap data from AMASS with some motions excluded §3.2 app. B.6. Its output is target DoF positions, which a PD controller turns into torques app. E.1.

The resulting motion samples have five modalities: video, text, SMPL pose, robot keypoints and robot actions §3.1. Video is kept at 20 FPS; actions are stored at 50 FPS with 27 DoF per frame app. C.1.

UH-1 model §4.

• Action tokenizer. A 1-D convolutional encoder-decoder with a vector-quantized bottleneck maps a clip of actions to a sequence of discrete tokens. Each token covers k = 4 frames, so the token sequence is about a quarter of the frame count eq. 10 app. D.1 app. D.3. The tokens form a shared vocabulary of motion primitives §4.
• Transformer. A causal Transformer receives a CLIP embedding of the command and predicts the next token given the previous ones, with an [End] token to stop §4 app. D.2. Training minimises the negative log-likelihood of the token sequences eq. 13.
• Two control modes §4 §5.4 fig. 5. In text-to-keypoint mode, the model emits keypoints and a goal-conditioned policy tracks them in closed loop. In text-to-action mode, it emits actions directly for open-loop control; the lower body then needs a separate locomotion policy §5.3. Figure 4 shows the architecture fig. 4.

Pseudocode for generation, following eq. 9 and eq. 11:

# UH-1 text-to-action generation  [§4] [eq. 9]
# Input:  command (str)
# Output: actions (T, d), decoded from action tokens  [app. C.1]

def generate_actions(command, text_encoder, transformer, decoder):
    # Embed the command with a CLIP text encoder  [§4]
    text_feature = text_encoder(command)              # (clip_dim,)
    # Predict action tokens one by one until the [End] token  [§4] [app. D.2]
    token_ids = transformer.sample(text_feature)      # (L,) codebook indices
    # Decode the tokens back into a motion clip  [eq. 11]
    actions = decoder(token_ids)                      # (T, d)
    return actions

Pseudocode for one tokenizer training step, following eq. 10 and eq. 12:

# UH-1 action tokenizer training step  [§4] [eq. 10] [eq. 12] [app. D.1]
# Input:  actions (B, T, d), with T = 64 during training  [app. D.3]
# Output: scalar loss

def tokenizer_step(actions, encoder, quantizer, decoder):
    # Encode every k = 4 frames into one latent vector  [§4] [app. D.1]
    latents = encoder(actions)                         # (B, T/4, code_dim)
    # Snap each latent to its nearest codebook entry  [eq. 10]
    tokens, quantized = quantizer(latents)             # (B, T/4)
    # Reconstruct the clip from the quantized latents  [eq. 11]
    recon = decoder(quantized)                         # (B, T, d)
    # L1 reconstruction plus an L1 first-difference term  [eq. 12]
    # Root and commitment weights: not stated  [app. D.1]
    loss = l1(actions, recon) + l1(diff(actions), diff(recon))
    return loss

Key equations, as numbered in the source (see Pitfalls):

\mathcal{L}_{recon} = \sum^T_{i} (|a^{\prime}_{i} - a_{i}| + |(a^{\prime}_{i+1} - a^{\prime}_{i}) - (a_{i+1} - a_{i})|) eq. 12

\mathcal{L}_{learn} = - \sum_{\mathcal{Z} \in \mathcal{D}} \log \prod_{i=1}^{|\mathcal{Z}|} p(z_i \mid z_{1:i-1}, l) eq. 13

\min_{\beta} \, \lVert \mathcal{P}^T_{joints} - \mathcal{P}^{T}_{robot} \rVert_2 eq. 4 eq. 5

q_{robot} = F_{ik}(\mathcal{P}_{robot}) eq. 7; the IK loss is an L1 keypoint error plus λ times a smoothness term app. B.5.

Goal-conditioned tracking policy §3.2 app. B.6 app. E.1. The observation contains root angular velocity, roll and pitch, yaw error, joint positions and velocities, and the previous action. The action space is 27 target joint positions, and the reward combines imitation terms with regularisation terms (table below). The paper does not describe the policy network size or training length.

Hyperparameters

Reward terms of the tracking policy. Paper weights app. B.6; code weights from the mimic environment config h1_2_mimic_config.py.

Results

Author-reported. Setups: HumanoidML3D (a retargeted version of HumanML3D) for Table 1 and 2; Humanoid-X for the scaling and ablation studies; a Unitree H1-2 for real-robot trials §5.1 §5.3. Confidence intervals are 95% over 20 repeated evaluations tab. 1. Metrics follow the HumanML3D protocol; the values below are on the retargeted humanoid motions.

Training data comparison, evaluated on the HumanoidML3D test set tab. 2:

Diffusion versus Transformer backbone tab. 4:

Real-robot success per command, 10 trials each, Unitree H1-2 tab. 3:

Other reported results:

• Scaling with data §5.2 fig. 7: FID falls from 0.689 at 1% of Humanoid-X to 0.463 at 100%; diversity rises from 5.900 to 6.149. These values are on a different scale from Tables 1 to 4 (see Pitfalls).
• Tokenizer vocabulary §5.4 fig. 9: FID falls from 0.539 (512 codes) to 0.463 (2048 codes); diversity rises from 6.050 to 6.149. The 1024-code point is not quoted in the text; it is only in the figure.
• Control modes in simulation §5.4 fig. 8: both modes average above 89% success over 100 sampled sequences per instruction. Text-to-keypoint is reported as slightly more robust.
• Diffusion backbone tab. 4: lower FID and MM Dist for the Transformer on the Humanoid-X training set.
• Relative gain claimed in §5.1: an FID reduction of over 23% against MDM; the table values give about 23.5%.

Paper vs code

Code: official repository https://github.com/sihengz02/UH-1 at commit ded9da2b2e5b0c8ae5194825c40c4d2b69b0e8a4 (shallow clone, HEAD). The repository is linked from the paper's project page, and its owner is an author (Siheng Zhao). The repository has no LICENSE file at its root. Subfolders legged_gym/ and rsl_rl/ carry their own LICENSE files with ETH Zurich and NVIDIA copyright notices. The code is cited by path and line only.

What the repository contains: text-to-keypoint inference with the released UH-1 checkpoints inference.py; the action tokenizer and Transformer definitions ; the retargeting scripts ; and the goal-conditioned PPO tracking policy with a checkpoint at iteration 49,500 model_49500.pt. The README points to a sample motion file that must be downloaded separately README.md. It does not include the tokenizer or Transformer training code, the captioning code, the pose-estimation code or the text-to-action mode.

Discrepancies between the paper and the code:

1. Codebook and depth. The paper gives a 2048-entry codebook and 18 Transformer layers app. D.3. The inference code builds both networks from defaults of 512 codes and 9 layers option.py and loads the released checkpoints with strict=True inference.py inference.py. The released model is therefore smaller than the paper's Appendix D.3 description.
2. Quantizer. The paper gives an embedding loss plus a commitment loss app. D.1. The code uses an exponential-moving-average codebook with decay 0.99 and dead-code reset, and a commitment MSE with no separate embedding loss quantize_cnn.py quantize_cnn.py quantize_cnn.py.
3. Decoding. The paper does not state a decoding strategy. The inference path takes the top-1 token at every step (greedy), so one command gives one motion transformer.py inference.py.
4. Normalisation. Inference un-normalises the decoded output with stored mean and std arrays inference.py inference.py. The paper does not mention this step.
5. Keypoint definition. Section 3.2 defines robot keypoints as 12 joint positions, but Appendix C.1 defines them as 27 DoF plus a 7-value root state. The released output is 34 values: 27 DoF, 3 root position and 4 root orientation README.md. Appendix C.1 matches the code.
6. Smoothness term. Appendix B.5 writes a signed sum of second differences. The code uses the absolute value of the difference between each frame and the mean of its neighbours retarget_motion_smplx.py.
7. Retarget loss. The paper's keypoint error is an L1 norm eq. 7 app. B.5; the code uses the L1 norm of each 3-D keypoint error, summed retarget_motion_smplx.py. Consistent.
8. Shape loss. The paper's L2 norm for the T-pose fit matches the code optim_betas_smpl.py.
9. Tracking policy rollout and entropy. Paper: 21 steps per rollout, entropy 0.01 app. B.6. Code: 24 steps h1_2_mimic_config.py, entropy 0.005 h1_2_mimic_config.py.
10. Learning rate. Paper: 1e-3 app. B.6. Code base: 2e-4 with an adaptive KL schedule h1_2_legged_robot_config.py. The mimic config does not override it.
11. Environment count. Paper: 6192 app. B.6. Code: 6144 h1_2_mimic_config.py.
12. Ankle and wrist handling. The paper says the two ankle roll joints are set to zero every step app. B.6. The code clamps ankle pitch to ±0.5 and ankle roll to ±0.2, and zeroes eight DoF (5, 11, 17, 18, 19, 24, 25, 26) h1_2_mimic.py.
13. Reward weights. Two terms are swapped in magnitude: DoF limit violation is -0.1 in the paper and -10.0 in code; DoF deviation is -10.0 in the paper and -0.1 in code (table above). The height term uses a clamp with max=0 on the norm of foot heights, which differs from the paper's max(·, 0) form h1_2_mimic.py.
14. Terms not in the paper (code only): a constant alive reward of 1 h1_2_mimic_config.py; future demonstration frames in the observation, with 2 steps at 0.1 s intervals h1_2_mimic_config.py; a motion curriculum h1_2_mimic_config.py; an in-place probability of 0.05 h1_2_mimic_config.py; gravity randomisation of ±0.1 h1_2_mimic_config.py; observation noise h1_2_mimic_config.py; gradient clipping at 1.0 h1_2_legged_robot_config.py; actor and critic widths 512, 256, 128 h1_2_legged_robot_config.py.
15. PD gains. The paper uses the robot documentation's gains, which it does not list app. E.1. The simulation gains are in the mimic config (for example knee 300, hip yaw 200) h1_2_mimic_config.py. Policy outputs are multiplied by an action scale of 0.25 h1_2_mimic_config.py h1_2_legged_robot.py.
16. Captioning and pose stages. The paper uses Video-LLaMA-style captions and VIBE. The repository labels these stages "work in progress" and only links to upstream projects README-Humanoid-X.md.
17. Checkpoint version. The README cites the Humanoids 2025 version under a different title README.md README.md. The arXiv v1 numbers may not match the code release.

No further discrepancies were checked: the tokenizer and Transformer training code is absent.

Pitfalls

• Keypoints have three definitions. The main text uses 12 joint positions, Appendix C.1 uses 27 DoF plus a root state, and the released code outputs 34 values. An implementer should follow the code's 34-value layout, and check the policy's input order before use.
• The dataset is described inconsistently. The introduction says "over 20 million humanoid robot poses" and "160,000 human-centric videos" §1. Section 3.1 says 163,800 motion samples and 20 million frames §3.1. Appendix B.2's table gives 163.8K clips and 20.7M frames. Appendix B.2's text gives a vocabulary of 3206 words, while its table gives 11,897 in total and 3,206 for verbs only.
• Numbers on different scales. Diversity is about 10 in Tables 1 to 4 but 5.9 to 6.1 in the scaling and vocabulary studies §5.2 §5.4. FID for the full Humanoid-X model is 0.379 in Table 2 but 0.463 at 100% data in the scaling study §5.2. The paper does not say whether these use different test sets.
• Table 1 and Table 2 disagree. The UH-1 row in Table 1 has R-precision 0.761, while the same HumanoidML3D-trained setting in Table 2 has 0.760.
• Real-robot claim. The text says "nearly 100%" across all instructions §5.3, but Table 3 ranges from 70% to 100%. Table 3 also states its values are for the real robot, while the text reports simulation results separately §5.4.
• Captioner identity. The appendix names Video-LLaMA, while the bibliography entry cites VideoLLaMA 2 (arXiv:2406.07476). The LaTeX also contains a commented-out line naming VideoLLaMA2-7B. Captions were generated with one of these, and the exact version is not confirmed.
• Mining and filtering thresholds are missing. The motion thresholds, batch thresholds, minimum chunk duration, and the full list of more than 400 search terms are not given app. B.2.
• Camera depth. The root depth formula uses a focal length f that is never stated app. B.4.
• Shape bound. |β_i| < 5 app. B.5 is the only constraint on the human shape. The code applies it as a clamp, so values past the bound get zero gradient rather than a penalty optim_betas_smpl.py.
• Sequence length cap. The code's default block size of 51 tokens limits generated sequences (roughly 204 frames at k = 4) option.py. The paper does not state a maximum length.
• Decoding strategy. The released inference takes the top-1 token at every step, and no sampling option is exposed transformer.py. The paper does not say which decoding produced the reported metrics, so diversity numbers may not be reproducible with the released code.
• Prompt phrasing. Captions are short, action-only and generated automatically, so the command vocabulary is shaped by the captioner. Robustness to other phrasings is not tested in the paper.
• Real-robot scope. In text-to-action mode the paper says open-loop control works only for the upper body, so the lower body comes from a pre-trained locomotion policy §5.3. That policy is not described and is not in the repository. Five human evaluators judged success by majority app. E.1.
• Scope. The paper studies pose control only; humanoid manipulation and loco-manipulation are left for future work §6.
• Compute. Full Humanoid-X training takes about 40 h for the tokenizer and about 400 h for the Transformer on one RTX 6000 Ada app. D.3. The RL tracking policy's training length is not stated.
• Data terms. The ethics statement says the Internet videos are for academic research and are "not intended for commercial use", and that the original videos will not be released app. A. The repository points to a Hugging Face dataset, but its license and terms were not checked here. Treat the dataset as unavailable for commercial reuse until its terms are verified.
• Figure and equation numbers in this card were counted from the LaTeX source, in source order. The compiled PDF was not checked; if numbers look off, the section citation is the reliable one.
• Retargeted-only evaluation. Table 1 evaluates motions retargeted onto the robot. The MDM and T2M-GPT baselines were retargeted with the same procedure §5.1, so the comparison measures generation, not the retargeting step.

Lineage

• Builds on: the VQ-VAE tokenizer approach (van den Oord et al., cited in the paper with no arXiv id in its bibliography) and T2M-GPT's architecture app. D.3; the retargeting step follows H2O (arXiv:2403.04436) §3.2; the tracking policy uses PPO (arXiv:1707.06347) §3.2; pose estimation uses VIBE and SMPL §3.2.
• Compared against: MDM and T2M-GPT, each with a human-motion-then-retarget pipeline §5.1.
• Related work cited for humanoid control: OmniH2O (arXiv:2406.08858) and Expressive Whole-Body Control (arXiv:2402.16796), the latter used as a retargeting comparison in the appendix §2 app. E.2.
• Superseded by: not known.