Human Motion Diffusion Model

In plain words

Making a computer-animated character move in a natural way is hard. Human motion is varied, people notice small errors quickly, and it is hard to describe in words what a motion should look like. Earlier generators tend to map each prompt to a single motion or to a simple distribution, and many are built for one kind of input.

Diffusion models are a recent family of generators. They learn to turn noise into a sample by removing a little noise at a time. This paper adapts that idea to human motion. The network is a small transformer. Instead of learning to guess the noise at each step, it guesses the clean motion. That choice makes it possible to add checks on the guessed motion, such as keeping the feet from sliding.

The same model can be steered by a text prompt, by an action label, or by nothing at all. It can also fill in a missing part of a motion, such as the frames between two poses, or redo only the upper body. The authors report that it reaches leading results on the standard text-to-motion and action-to-motion benchmarks and that it trains on a single mid-range GPU in about three days.

TL;DR

MDM is a diffusion model with a transformer-encoder backbone that predicts the clean motion at each step §1 §3. It is trained with classifier-free guidance, so one model handles text, action and unconditional generation §3 §1. Author-reported: best FID on HumanML3D (0.544 against 1.067 for T2M) and KIT (0.497) tab. 1 tab. 2. The paper trains in about three days on one RTX 2080 Ti §4. The released code is MIT-licensed and differs from the paper in several places (see Paper vs code).

Problem

Motion generation from text or action labels has three difficulties. Motion is varied, text and labels describe it only loosely, and viewers notice artifacts such as foot sliding §1 §3, Geometric losses. Earlier text-to-motion methods learn an auto-encoder or a VAE, which assume a one-to-one mapping or a simple latent distribution, while many valid motions match one prompt §1. Diffusion models fit this many-to-many setting, but they are costly to train and hard to control, and their standard U-Net backbone is not well suited to motion §1. The paper argues that the required training cost can be kept low with a transformer and that geometric losses, which are non-trivial in diffusion, can be added §1 §3.

Method

Overview. The denoiser G takes a noisy motion x_t of N frames, a diffusion step t and a condition code c. It returns an estimate of the clean motion x_hat_0 §3, Framework fig. 2. The condition is CLIP text for text-to-motion §3, Model §4.1, a learned embedding per action class for action-to-motion §3, Model §4.2, or absent for unconditional generation §5.2.

Forward process. Each step adds Gaussian noise with a per-step coefficient alpha_t in (0, 1) eq. 1 §3, Framework. The paper states that alpha_t is a constant hyperparameter and gives no schedule formula there; Section 4 names a cosine schedule with T = 1000 steps §4.

Objective. G predicts x_0 directly, not the noise. The loss is the squared error between the predicted and the clean motion eq. 2 §3, Framework. The paper adds three geometric terms, one for joint positions, one for foot contact and one for velocity eq. 3 eq. 4 eq. 5 §3, Geometric losses. The total is the simple loss plus lambda-weighted geometric terms eq. 6 §3. The lambda values are not stated.

Foot contact. The mask f_i marks, for each frame and foot joint, whether the foot touches the ground. The paper takes it from binary ground-truth labels and uses it to zero the velocity of planted feet §3, Geometric losses.

Architecture. One transformer encoder with no decoder §3, Model. The time step and the condition are each projected by their own feed-forward network and summed into one token §3, Model. Each frame is projected linearly and given a positional embedding. The token and the frames form the encoder input. The output for the token is dropped, and the other outputs are projected back to the motion dimensions to give x_hat_0 §3, Model.

Classifier-free training. For 10% of the training samples the condition is replaced by the null condition §3, Sampling. At sampling time the two predictions are combined with scale s eq. 7 §3, Sampling. The paper uses s = 2.5 for text-to-motion §4.1.

Sampling. Start from Gaussian noise with the shape of the motion. For each step from T down to 1, predict x_hat_0, then draw x_{t-1} from the diffusion posterior, following the standard ancestral sampler §3, Sampling.

Editing (no retraining). At every sampling step the known entries of x_hat_0 are overwritten with the input motion §3, Editing. In-betweening fixes the first and last 25% of frames and generates the middle 50% §5.1. Body-part editing fixes the joints that should stay and generates the rest, for example the upper body §5.1 fig. 3.

# Training step: noise a clean motion, predict the clean motion  [§3]
def training_step(clean_motion, condition):
    # Draw one diffusion step per sample, T = 1000  [§4]
    step = sample_uniform_integer(1, T)
    noise = sample_gaussian_like(clean_motion)
    # Forward noising to step t  [§3, Framework] [eq. 1]
    noisy_motion = q_sample(clean_motion, step, noise)
    # Replace the condition with the null condition for 10% of samples  [§3]
    condition = replace_with_null(condition, probability=0.1)
    # The network predicts the clean motion, not the noise  [§3, Framework]
    predicted_motion = model(noisy_motion, step, condition)
    # Simple reconstruction loss  [eq. 2]
    loss = mean_squared_error(predicted_motion, clean_motion)
    # Optional geometric terms, each with its own lambda  [eq. 3-6]
    loss = loss + lambda_pos * loss_pos(predicted_motion, clean_motion)
    loss = loss + lambda_foot * loss_foot(predicted_motion, contact_mask)
    loss = loss + lambda_vel * loss_vel(predicted_motion, clean_motion)
    return loss

# Denoiser G: transformer encoder over time  [§3, Model]
def model(noisy_motion, step, condition):
    # Embed the step and the condition, then sum them into one token  [§3]
    time_token = time_embedding(step)                   # (B, d)
    condition_token = condition_embedding(condition)    # (B, d)
    token = time_token + condition_token                # (B, d)
    # Project each frame to d and add a positional encoding  [§3, Model]
    frames = linear_in(flatten_joints(noisy_motion))    # (B, N, d)
    frames = add_positional_encoding(frames)
    sequence = concatenate(token, frames)               # (B, N+1, d)
    encoded = transformer_encoder(sequence)             # (B, N+1, d)
    # Drop the token output and project the rest back to motion  [§3, Model]
    return linear_out(encoded[:, 1:])                   # (B, N, J*D)

# Classifier-free guided sampling with optional inpainting  [§3] [eq. 7]
def sample(condition, shape, scale, known_motion=None, known_mask=None):
    # Start from standard Gaussian noise in the motion's shape  [§3]
    motion = sample_gaussian(shape)
    for step in range(T, 0, -1):                   # T = 1000  [§4]
        # Two network passes per step: conditional and null  [§3, Sampling]
        predicted_conditional = model(motion, step, condition)
        predicted_null = model(motion, step, null_condition)
        # Extrapolate between the two predictions with scale s  [eq. 7]
        predicted = predicted_null + scale * (
            predicted_conditional - predicted_null
        )
        # Inpainting: overwrite the known entries of the x_0 prediction  [§3]
        if known_mask is not None:
            predicted = where(known_mask, known_motion, predicted)
        # Draw x_{t-1} from the posterior given the x_0 prediction  [§3]
        motion = posterior_sample(predicted, motion, step)
    return motion

Hyperparameters

Values stated by the paper are cited to the paper. Values marked "code" appear only in the official repository at code_commit, and the paper does not state them.

Results

Author-reported. All numbers come from the paper's own evaluation, with the metrics of Guo et al. for text and Guo et al. 2020 for action §4.1 §4.2. Values are mean plus or minus the 95% confidence interval, from 20 runs (5 for MultiModality) tab. 1. Arrows follow the paper's convention: "down" means lower is better, "up" means higher is better, and "->" means closer to the real motion is better.

Text-to-motion, HumanML3D test set tab. 1. Setup: real motion lengths, baselines JL2P, Text2Gesture, T2M §4.1.

Text-to-motion, KIT test set tab. 2. Setup: the same metrics, with baselines JL2P, Text2Gesture, T2M.

User study (text-to-motion, KIT) §4.1 fig. 4. 31 participants compared MDM with each baseline on side-by-side pairs; MDM was preferred most of the time, and was preferred over ground-truth samples in 42.3% of cases.

Action-to-motion, HumanAct12 tab. 3. Setup: FID, action-recognition accuracy, diversity, multimodality; baselines Action2Motion, ACTOR, INR; the table gives ground-truth rows for comparison §4.2.

Action-to-motion, UESTC tab. 4. Setup: cross-subject protocol, FID on train and on test split, baselines ACTOR and INR §4.2.

Unconstrained synthesis, HumanAct12 tab. 5. Setup: labels ignored, metrics of MoDi §5.2. No confidence intervals are reported.

Qualitative editing results (in-betweening, body-part editing) are shown only as figures fig. 3 and are not measured.

Paper vs code

Compared with the official repository at ef8edce6a53c6ab19e53b4d4dcf15bc0bc60a778 (MIT; owned by the first author, and linked from a footnote in the abstract).

• Foot-contact mask. The paper defines f_i from binary ground-truth contact labels §3, Geometric losses. The code instead marks a foot as in contact when its ground-truth speed is at most 0.01 gaussian_diffusion.py. The term is only applied to rot6d data for HumanAct12 and UESTC gaussian_diffusion.py.
• Velocity loss. The paper's velocity term runs over all joints eq. 5 §3, Geometric losses. The code drops the last joint index with the comment that it is the root location gaussian_diffusion.py.
• Geometric weights. The paper gives no values for lambda_pos, lambda_vel or lambda_foot §3. The code defaults all three to 0 parser_util.py. The released action and unconstrained training commands set them to 1 README.md README.md.
• Classifier-free dropout for action models. The paper trains the denoiser with classifier-free guidance, dropping the condition for 10% of samples §3, Sampling. The released action and unconstrained commands set the dropout to 0 README.md README.md. The code then forces the guidance scale to 1 parser_util.py, and the guided sampler refuses a model trained without dropout cfg_sampler.py. Action models as released therefore cannot use guidance.
• Training length. The paper states 500K steps for text-to-motion §4.1, and 750K and 2M for HumanAct12 and UESTC §4.2. The code default is 600K parser_util.py. The README's "reproduce the paper model" command does not set a step count README.md.
• Recommended flags. The README's "original paper model" command does not add --mask_frames or --use_ema, although it recommends both as improvements README.md README.md. The --mask_frames flag is described as a fix for a masking bug parser_util.py. The paper mentions neither flag.
• Noise schedule. The paper gives a per-step coefficient alpha_t eq. 1 and names a cosine schedule §4. The code builds betas from the cosine function with offset 0.008 and caps them at 0.999 gaussian_diffusion.py, then forms cumulative products for the forward process gaussian_diffusion.py gaussian_diffusion.py.
• Prediction target. Paper and code agree that the network predicts x_0 §3, Framework. The code asserts this for the inpainting path gaussian_diffusion.py.
• Variance. The paper does not state a variance choice. The code uses the small posterior variance by default parser_util.py model_util.py.
• Output clipping. The paper does not mention clipping. The sampling loop clips the x_0 prediction to [-1, 1] by default gaussian_diffusion.py, but the generation and editing scripts pass clip_denoised=False generate.py edit.py.
• Architecture defaults. Layers (8), latent dimension (512) and batch size (64) match the paper parser_util.py parser_util.py parser_util.py. Feed-forward size, heads, dropout and activation are hard-coded and absent from the paper model_util.py.
• Time embedding. The paper says the time step goes through a feed-forward network §3, Model. The code embeds the step with a sinusoidal table followed by a two-layer MLP mdm.py.
• Condition only. Without a condition, the code uses the time embedding alone as the token mdm.py, which matches the paper's unconditional setting §5.2.

No further discrepancies were found in the MSE objective, the classifier-free combination, or the frame masking for the text model.

Pitfalls

• Coefficient ambiguity. The forward process is written with a per-step alpha_t eq. 1, but sampling and noising in the code use cumulative products of the schedule. Implement with the cumulative form, as the code does gaussian_diffusion.py.
• Missing loss weights. Any geometric term needs a lambda the paper does not give. Start from the action-model values in the README (all 1) README.md, but treat them as untested for text-to-motion. The paper trains the text model without geometric losses §4.1.
• Foot-contact definition. Implementing the paper's contact labels is not the same as the code's velocity threshold. Use the code's threshold if you want to match the released models gaussian_diffusion.py.
• Dropout for action models. Guidance is disabled when the dropout is 0 parser_util.py. Keep dropout above 0 if you want guided action sampling.
• Sampling cost. The paper's inference cost counts about 1000 forward passes §6. Each guided step runs the network twice, once with and once without the condition cfg_sampler.py, so guided sampling needs about 2000 network calls per result.
• Clipping at sampling. The library default clamps predictions to [-1, 1] gaussian_diffusion.py. The released scripts turn it off generate.py. Pick one and document it, since the paper does not say.
• Text length cut-off. Text prompts are truncated to 20 CLIP tokens for HumanML3D and KIT mdm.py. Longer descriptions lose their tail silently.
• Inpainting needs x_0 prediction. The code's inpainting branch asserts that the network predicts x_0 gaussian_diffusion.py. A port that predicts noise would need its own editing rule; the paper does not give one.
• Unstated optimizer and widths. Heads, feed-forward size, dropout, optimizer and learning rate are all unstated in the paper. Use the code's values if you need to reproduce the released models model_util.py training_loop.py.
• Frame masking and averaging. The paper does not mention the masking fix or weight averaging. The README says both improve performance, so a reproduction should state which one it uses README.md parser_util.py.
• Claims versus tables. The text says MDM reaches state of the art on FID, diversity and MultiModality for HumanML3D §4.1. The bold MultiModality entry in the table is the decoder variant (2.927), not the main model (2.799) tab. 1. On HumanAct12 the caption says the model leads in three of four metrics tab. 3, but the best FID in that table is the variant without foot contact (0.080), and INR has a lower FID than the main model (0.088 against 0.100).
• Results for several variants. The paper's headline numbers mix the main model and its variants. Quote the main-model row unless you need a variant on purpose.
• Action-to-motion sequence length. The paper does not state the length of generated motions for HumanAct12 and UESTC. The code's CLI default is 60 frames parser_util.py.
• Training cost. The paper says every model took about three days on one RTX 2080 Ti §4, although the step counts differ (500K to 2M) §4.1 §4.2. Treat the three-day figure as the authors' setup, not as a cost per step.

Lineage

• Builds on: diffusion models for images and the classifier-free guidance approach, which the paper cites and adapts §2. Neither paper has a card here, so no content is taken from them.
• Compared against: JL2P, Text2Gesture, T2M, TEMOS, MotionCLIP, Action2Motion, ACTOR, INR and MoDi §2 §4. Only the tables above use their numbers.
• Concurrent work: Zhang et al. 2022 and Kim et al. 2022, which the paper names as contemporaneous diffusion approaches to motion §2.
• Superseded by: not stated on this card.