MotionGPT: Human Motion as a Foreign Language

In plain words

Think of human motion as a sentence. The paper first builds a small dictionary of motion pieces: a motion clip is turned into a short string of codes, and each code stands for a recognisable chunk of movement. A pretrained language model of the T5 family then treats these codes as extra words next to ordinary text.

Training happens in stages. The model first learns from text and motion together, then from many instruction-style requests such as "generate a motion for this sentence" or "describe this motion". One model covers text-to-motion, motion-to-text, predicting how a motion continues, and filling in missing frames.

The appeal is that one language-model-style system can cover several motion tasks instead of one specialised model per task. The reported numbers come from the authors, and they do not show that the model beats every specialised baseline.

TL;DR

• Motion is encoded by a VQ-VAE into discrete codes (codebook of 512) and these codes become words in a Flan-T5 vocabulary §3.1 §4.1.
• Training has three stages: tokenizer, text-motion pretraining with span corruption, and instruction tuning on prompt templates §3.3.
• On HumanML3D, the author-reported text-to-motion FID is 0.232 for the task-tuned model, while T2M-GPT reports 0.116 tab. 3.
• Use it if you want one motion-language model with prompt-based tasks. Do not assume it beats the strongest baseline on every metric.

Problem

Earlier text-to-motion and motion-to-text methods use separate modules for text and motion. They usually need strictly paired motion and text data, and their supervision is task-specific, so they generalise poorly to unseen tasks §1. The paper aims at one pretrained motion-language model that can be prompted for several motion tasks §1 §3.

Method

Overview. Three parts: a motion tokenizer that maps M frames to L = M / l tokens, a motion-aware language model, and a three-stage training scheme §3 fig. 3.

3.1 Motion tokenizer

A VQ-VAE with motion encoder E and decoder D. E applies 1D convolutions along time and produces latents. Each latent is replaced by its nearest codebook entry, with K entries of dimension d §3.1 eq. 1.

• The downsampling rate is l = 4 §4.1. The code uses two stride-2 stages default.yaml.
• Codebook: K = 512, d = 512 §4.1 app. H default.yaml.
• Loss: the paper sums a reconstruction term, an embedding term and a commitment term, without stated weights §3.1. The text also adds a smooth L1 loss and a velocity regulariser to the reconstruction term, and uses EMA and codebook reset for codebook utilisation §3.1.

3.2 Motion-aware language model

• Motion codes are mapped to indices s^1:L, and combined with the text vocabulary V_t into a unified vocabulary V = {V_t, V_m}. V_m includes boundary tokens for the start and end of a motion §3.2.
• Backbone: a T5-style encoder-decoder, Flan-T5-Base (220M) in the main results §3.2 §4.1.
• Objective: autoregressive log-likelihood of the target sequence given the source eq. 2.
• Inference: tokens are sampled from the predicted distribution until the end token §3.2.

3.3 Training strategy

1. Tokenizer. Train E and D with the objective in §3.1. The tokenizer is then frozen §3.3.
2. Motion-language pretraining. Continue from T5 on (a) span corruption over text and motion, with 15% of input tokens replaced by sentinels §3.3, and (b) supervised motion-text translation on HumanML3D and KIT §3.3.
3. Instruction tuning. Multi-task prompts built from HumanML3D and KIT, with 15 core tasks in §3.3 (14 in app. G) and dozens of templates per task, giving more than a thousand instruction prompts §3.3 app. G tab. 13.

# Stage 1: motion tokenizer (VQ-VAE)  [§3.1] [eq. 1]
# Input: motion (B, M, D); output: codes (B, L) and quantized latents
def tokenize_motion(motion):
    # 1D convolutions along time; L = M / 4  [§3.1] [§4.1]
    latents = motion_encoder(motion)               # (B, L, d)
    # Replace each latent by its nearest codebook entry, K = 512  [eq. 1]
    codes = nearest_code_index(latents, codebook)  # (B, L), values in [0, K)
    quantized = codebook[codes]                    # (B, L, d)
    return codes, quantized

# Stage 1 loss: reconstruction, velocity, commitment  [§3.1]
# [code: configs/config_h3d_stage1.yaml#L40-L45]
# Paper gives no weights; code uses 1.0, 0.5 and 0.02
def tokenizer_loss(motion, reconstruction, commit_term):
    # Smooth L1 on reconstructed features  [§3.1]
    feature_loss = smooth_l1(reconstruction, motion)
    # Smooth L1 on a slice of velocity features, not finite differences
    # [code: mGPT/losses/mgpt.py#L70-L81]
    velocity_loss = smooth_l1(
        velocity_slice(reconstruction),
        velocity_slice(motion),
    )
    # Commitment term; the EMA codebook has no separate embedding term
    return feature_loss + 0.5 * velocity_loss + 0.02 * commit_term

# Motion vocabulary: codes become words  [§3.2]
# Ids K and K+1 bracket a motion; K+2 is the in-between mask
# [code: mGPT/archs/mgpt_lm.py#L67-L68] [code: mGPT/archs/mgpt_lm.py#L384-L394]
def motion_to_words(codes, codebook_size):
    # Wrap the codes in start and end tokens
    words = ["<motion_id_%d>" % codebook_size]
    for code in codes:
        words.append("<motion_id_%d>" % code)
    words.append("<motion_id_%d>" % (codebook_size + 1))
    return words

# Stage 2: span corruption on a mixed token sequence  [§3.3]
# [code: mGPT/archs/mgpt_lm.py#L143-L173]
# About 15% of tokens are noise, grouped into spans
def corrupt_spans(tokens):
    # Choose noise spans so that about 15% of tokens are noise  [§3.3]
    noise_mask = random_span_mask(length(tokens), density=0.15, mean_span=3)
    # Each noise span becomes one sentinel in the source  [§3.3]
    source = replace_spans_with_sentinels(tokens, noise_mask)
    # Target: each dropped span after its sentinel, then a final sentinel
    target = spans_with_sentinels(tokens, noise_mask)
    return source, target

# Inference: sample target tokens until the end token  [§3.2]
# Decode the motion codes back to frames with the tokenizer decoder  [§3.1]
def generate(instruction, max_len=256):
    # Encode the instruction, which may contain motion words  [§3.2]
    encoder_states = language_model.encode(instruction)
    output = []
    while len(output) < max_len:
        # Sample from the next-token distribution  [§3.2]
        # [code: mGPT/archs/mgpt_lm.py#L231-L233]
        probs = language_model.next_token_probs(encoder_states, output)
        token = sample(probs)
        if token == end_token:
            break
        output.append(token)
    return output

Loss for the language model, from eq. 2:

# Cross-entropy over target tokens, given source and previous targets  [eq. 2]
# The paper writes a sum; the code's loss is averaged
# [code: mGPT/archs/mgpt_lm.py#L173-L179]
def lm_loss(source, target):
    # Decoder reads the target shifted right, predicts each next token
    logits = language_model(source, shift_right(target))  # (B, T, V)
    return cross_entropy(logits, target)                  # mean over tokens

Hyperparameters

Values for the HumanML3D base model. "Not stated" means the paper does not give the value. Code values come from the repo's HumanML3D stage configs.

Results

Author-reported. Setup: HumanML3D unless noted, Flan-T5 Base (220M), evaluation with the metric encoder of Guo et al. §4.1 §4.2. Table numbers follow the LaTeX source order.

Text-to-motion, HumanML3D tab. 3. Fine-tuned means task-tuned after instruction tuning. Pre-trained means the instruction-tuned model before task tuning.

Motion-to-text, HumanML3D tab. 4. Ground-truth text is used without pre-processing for the linguistic metrics (see Pitfalls).

Motion prediction and in-between, AMASS subset tab. 5. Predict from the first 20% of the motion, or fill a masked span. FID and DIV on the completed motion, ADE and FDE are displacement errors.

Ablation, model size and instruction tuning tab. 6. Text-to-motion (T2M), prediction and in-between, as printed in the source. Columns are FID and DIV unless noted; "-" means the task was not evaluated for that row.

Tokenizer codebook size tab. 8, evaluated on the motion part of HumanML3D with the MLD protocol. K=512 gives MPJPE 55.8 mm, FID 0.067 and DIV 9.675. K=256 gives 76.4 and 0.187; K=1024 gives 60.3 and 0.086; K=2048 gives 78.9 and 0.141.

Speed tab. 11. Frames per second on one V100 at batch size 1, averaged over generated frames: Small 421.31, Base 222.69, Large 119.75.

Headline claim. The abstract says MotionGPT "achieves state-of-the-art performances" on text-to-motion, captioning, prediction and in-between §1. The text of §4.2 is more careful: it says MotionGPT is "competitive" on most metrics §4.2.

Paper vs code

Searched: the paper's title page links github.com/OpenMotionLab/MotionGPT, which the repo's MIT license and the authors' lab identify as official. That is the repo cloned at the commit above.

• Tokenizer learning rate. Paper: 1e-4 §4.1. Code: 2e-4 in the tokenizer stage config_h3d_stage1.yaml. The LM learning rates match.
• Tokenizer loss. Paper: reconstruction, embedding and commitment terms §3.1 with no weights. Code: feature 1.0, velocity 0.5, commit 0.02 [code: configs/config_h3d_stage1.yaml#L40-L45], and no separate embedding term under the EMA quantiser quantize_cnn.py.
• Codebook. The paper names EMA and codebook reset §3.1. The code uses ema_reset with decay 0.99 mgpt_vq.py.
• In-between mask. Paper: "mask about 50% motion randomly" §4.2. Code: one contiguous block between 25% and 75% of tokens mgpt_lm.py.
• Motion vocabulary. Paper: boundary tokens §3.2. Code adds three tokens: start, end, and an in-between mask at K+2 mgpt_lm.py mgpt_lm.py.
• Loss scale. Paper writes the LM objective as a sum over positions eq. 2. Code uses the Transformers cross-entropy with labels, which averages over non-padding tokens mgpt_lm.py.
• Caption quotation. The code wraps the caption in quotation marks with probability 0.5 during templating mgpt_lm.py. The paper does not mention this.
• Prediction split. Paper: first 20% of motion §4.2. Code's pred branch cuts at length // 5 mgpt_lm.py, where length is the frame count passed in mgpt.py. With four frames per token, that keeps about 80% of tokens, not 20%. The training template path uses length / down_t and does give 20% mgpt_lm.py.
• Backbone and sizes. Paper: Flan-T5 Base §4.1. Code default is flan-t5-base default.yaml; the Large config is flan-t5-large t5_large.yaml.
• Iteration counts. The 150K tokenizer and 300K LM iterations §4.1 are not set in the configs (END_EPOCH: 999999). They cannot be checked from the repo.
• Feature size. The code's HumanML3D representation has 263 features mgpt.py. The paper names the components but not the size §4.2.

Pitfalls

• Not the best on every metric. T2M-GPT's text-to-motion FID (0.116) is lower than MotionGPT's fine-tuned value (0.232) tab. 3. The abstract's "state-of-the-art" wording should be read per metric.
• Which checkpoint is "MotionGPT". The "pre-trained" row of tab. 3 equals the base model after instruction tuning tab. 6. The base model with pretraining only is the other row tab. 6 tab. 9. The fine-tuned row is task-tuned tab. 10. In tab. 5 the completion numbers (in-between FID 0.214, ADE 3.762) match the uniform model, not the task-tuned in-between row (FID 0.209, ADE 3.281 tab. 10), although §4.2 says the model was fine-tuned for completion.
• Masking is contiguous in code. The paper says random §4.2. Reimplementations that follow the text will mask scattered tokens.
• Tokenizer loss. The paper's three-term loss has no weights and the code has no embedding term under EMA. Copying the paper's objective gives a different training signal than the released code.
• Learning rate conflict. The tokenizer's learning rate is 1e-4 in the paper and 2e-4 in the code. Pick one and note it.
• Hardware conflict. Section 4.1 says 8 V100s. Appendix H says 64 V100s §4.1 app. H. Do not infer wall-clock or cost from either without the other.
• Task count conflict. §3.3 says 15 core tasks; app. G says 14 §3.3 app. G.
• Noise span length is not stated. The code uses 3 mgpt_lm.py.
• Generation. Sampling is the default. The paper's own ablation says beam search and greedy decoding can slightly improve FID but reduce diversity sharply app. E tab. 12. Max output length is a code default of 256 tokens mgpt_lm.py, not a paper value.
• Costs the abstract hides. The authors report that the larger model gains little and is sometimes worse, and attribute this to the small motion dataset §4.3 app. H tab. 6. Inference is slower at larger sizes tab. 11.
• Undocumented augmentation. The caption-quotation step and the code's placeholder templates mgpt_lm.py are part of training but not in the paper.
• Instruction data is not released in this repo. The templates are in prepare/instructions/ template_instructions.json, but the paper says the processed dataset will be released app. G.

Lineage

• Builds on: VQ-VAE motion tokenisation used in TM2T, T2M-GPT and related work §3.1; T5 and Flan-T5 backbones §3.2 §4.1; span corruption from T5 §3.3; instruction tuning from InstructGPT and Flan §1 §3.3.
• Compared against: TM2T, T2M-GPT, MDM, MLD and MotionDiffuse on HumanML3D tab. 2 tab. 3 tab. 4 tab. 5.
• Superseded by: not known from this wiki.