Human Motion Diffusion as a Generative Prior

In plain words

Teaching a computer to produce human motion from a sentence works best when the model has seen a lot of motion. Motion data is expensive to record, so most models have only seen short clips of one person. This paper asks what a pretrained motion model can do beyond its training data, without collecting much more.

The answer is to combine copies of the model at generation time. One method chains several text prompts into one long motion, and each piece is shaped so that the joins between neighbours look natural. A second method runs two copies of the model side by side and adds a small helper that lets the two people react to each other. A third method takes several models that were each taught to follow a different body-control signal, such as a root path or one hand, and blends their outputs so one motion follows several signals at once.

The helper pieces are small and the big model stays fixed. This is how the paper gets two-person results from only a few example sequences.

TL;DR

• The paper reuses one pretrained text-to-motion diffusion model (MDM) §1 §3.
• DoubleTake generates long, prompted motions by denoising overlapping intervals together, then refining each join §3.1.
• ComMDM is a small trained block that coordinates two frozen MDM instances for two-person motion; it is trained on 10 two-person sequences in the paper's setting §3.2 §4.2.
• DiffusionBlending blends outputs of several control-tuned models; it is the authors' generalisation of classifier-free guidance §3.3 eq. 3.

Problem

Motion data is scarce, expensive to capture and mostly short single-person clips, so models trained on it struggle with long sequences, with two people, and with fine control of a specific joint or trajectory §1. Previous long-sequence methods chain clips autoregressively, each conditioned on the suffix of the previous one; some learn the transitions from annotated data §2. For control, the inpainting scheme inherited from MDM does not follow trajectory or end-effector constraints well, which the authors report in their own experiments §3.3 §4.3.

Method

The base model is MDM, a denoising diffusion model that predicts the clean motion, conditioned on a CLIP text embedding. The forward noising process is written as a Gaussian step eq. 1; training uses a simple denoising loss plus geometric losses on joint positions, velocities and foot contact §3. The authors use two motion representations: SMPL joint rotations plus global positions for BABEL, and the HumanML3D feature vector elsewhere §3.

1. DoubleTake: sequential composition §3.1

DoubleTake produces a long motion from several text-prompted intervals, using a fixed prior trained only on short clips. The idea is to generate all intervals in one batch, each with its own prompt, and to tie neighbours together through a "handshake": a short overlap of about one second §3.1 fig. 3.

• Handshake: at the end of every denoising step, the overlapping frames of each pair of neighbouring intervals are replaced by one shared value, a frame-wise linear blend of the suffix of the earlier interval and the prefix of the later one eq. 2 §3.1.
• First take: intervals are denoised as one batch with the handshake constraint applied at each step §3.1 fig. 3.
• Second take: for each join, the sandwich (suffix, handshake, prefix) is partially re-noised and denoised back, with a soft mask that makes frames near the handshake refine more than frames far away §3.1 fig. 2 fig. 3.
• Unfolding: each refined sandwich is written back into the long timeline §3.1.

The second take exists because the first take alone shows artifacts at transitions between semantically different actions, such as running then crawling §3.1 fig. 5.

# DoubleTake: one long motion from n prompted intervals  [§3.1]
# Input:  prompts (n,), lengths (n,) in frames, handshake h, blend b
# Output: one motion of sum(lengths) - (n - 1) * h frames

def double_take(prompts, lengths, h, b):
    # First take: denoise all intervals as one batch, one prompt each  [§3.1]
    # After every denoising step, average each neighbouring pair's
    # h overlapping frames (handshake)  [eq. 2]  [§3.1]
    # [code: diffusion/gaussian_diffusion.py#L696-L706]
    intervals = first_take(prompts, lengths, h)   # list of (L_i, D)

    # Second take: one sandwich per join: suffix, handshake, prefix  [§3.1]
    # [code: utils/sampling_utils.py#L80-L99]
    sandwiches = make_sandwiches(intervals, h)

    # Re-noise each sandwich partially, then denoise it back  [§3.1]
    # Soft mask: frames next to the handshake refine more  [§3.1]
    # [code: utils/sampling_utils.py#L101-L129]
    refined = second_take(sandwiches, b)

    # Write the refined transition frames back into the timeline  [§3.1]
    # [code: utils/sampling_utils.py#L132-L152]
    return unfold(intervals, refined, h)

2. ComMDM: two-person generation §3.2

Two-person motion is handled by keeping one fixed MDM per person and training only a slim coordination block, ComMDM. It reads the activations of one transformer layer from both people and outputs a correction that is added to the current person's activations §3.2 fig. 4. To save parameters, the block outputs one correction term and is applied with the two people's roles swapped, so the same block serves both people §3.2.

The root positions of HumanML3D-style data all start at the origin and face the same way. To avoid two people starting at the same point, ComMDM also learns each person's initial pose as part of the diffusion process §3.2.

Two training tasks are used: prefix completion, with the text condition fully masked, and text-to-motion, with the text masked with probability 10% so that classifier-free guidance works §3.2.

# ComMDM correction at transformer layer n  [§3.2] [fig. 4]
# Two frozen MDMs, one per person; only the correction is trained  [§3.2]
# Input:  acts_cur, acts_other: (L, B, d) activations of layer n
# Output: corrected activations for the current person, (L, B, d)

def comm_correction(acts_cur, acts_other):
    # Combine both people's activations into one sequence  [§3.2]
    # [code: model/comMDM.py#L150-L156]
    joint = aggregate(concat(acts_cur, acts_other))
    # One small transformer predicts the correction term  [§3.2]
    delta = comm_transformer(joint)
    # Add the correction to the current person only  [§3.2]
    # [code: model/comMDM.py#L91-L92]
    return acts_cur + delta

3. Fine-tuned control and DiffusionBlending §3.3

Control signals are root trajectory (horizontal rotation angle and planar linear velocity, without height) or a single joint, and combinations of these §3.3 §4.3.

• Single control fine-tuning: the MDM is fine-tuned so the noise on the controlled features is set to zero during the forward diffusion. The model then learns to rely on those features to reconstruct the rest §3.3 alg. 1. Fine-tuning is initialised from the same MDM §4.3.
• Sampling with control: after each prediction the given control values are written back, and the noise on those features is again zeroed §3.3 alg. 2.
• DiffusionBlending: for two aligned models conditioned on different signals, the sample is the first model's output plus a scalar s times the difference to the second model's output eq. 3 §3.3. Experiments on composite controls use equal weights, s = 0.5 §4.3.

# Fine-tuning for a control signal  [alg. 1]
# Input:  motion x0 from the training set
# Output: loss for one gradient step

def finetune_loss(x0):
    # Sample a random diffusion step  [alg. 1]
    t = uniform_int(1, T)
    # Draw Gaussian noise  [alg. 1]
    eps = normal(0, I)
    # Zero the noise on the controlled features  [alg. 1]
    # [code: diffusion/inpainting_gaussian_diffusion.py#L15-L17]
    eps[controlled] = 0
    # Noise the motion, then predict the clean motion  [alg. 1] [§3.3]
    x_t = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps
    return norm(x0 - model(x_t, t))

# Sampling with a fixed control signal  [alg. 2]
# Input:  given control values for the controlled features
# Output: motion that follows the control values

def sample_with_control(given, T):
    # Start from zeros at the top step  [alg. 2]
    x0 = zeros_like(given)
    for t in range(T, 0, -1):
        # Write the given control values into the prediction  [alg. 2]
        x0[controlled] = given
        # Draw noise and zero it on the controlled features  [alg. 2]
        eps = normal(0, I)
        eps[controlled] = 0
        # Take one denoising step  [alg. 2]
        x0 = denoise_step(x0, eps, t)
    return x0

Blending pseudocode for DiffusionBlending is the formula itself:

G_s^{a,b}(X_t, t, c_a, c_b) = G^a(X_t, t, c_a) + s \cdot (G^b(X_t, t, c_b) - G^a(X_t, t, c_a)) eq. 3

Hyperparameters

Values stated by the paper are cited to the paper. Values marked "code" appear only in the official code; the paper does not state them.

Results

Author-reported. Setup: HumanML3D test set (single person, text), BABEL test set (transitions), 3DPW (two-person, few-shot), and a user study §4.

Long sequences compares DoubleTake with TEACH on BABEL and runs ablations on HumanML3D. Metrics follow the HumanML3D evaluators, applied to the intervals and separately to transitions; "Trans." columns measure transitions with a 70-frame or 30-frame margin §4.1 tab. 1.

Source: tab. 1 (BABEL). Table reproduced in full; the paper's bold and underline marks are omitted. The best motion FID is the "+ Trans. Emb" row (0.79). The best transition FIDs are Double Take without extra terms, 1.88 at 70 frames and 3.45 at 30 frames (TEACH: 3.86 and 7.93). The "+ Trans. Emb + geo losses" row is second best on transitions, at 2.39 and 6.05 tab. 1.

Source: tab. 2. Full table, marks omitted. The paper's headline: with soft masking and a one-second handshake, DoubleTake gives the best FID (0.60) among these variants §4.1. Ground-truth transition FID is 0.05.

Two-person prefix completion reports root and joint L2 error (metres) for a 1-second prefix and a 3-second completion, on 3DPW §4.2 tab. 3. Lower is better. The paper reports that MRT has lower L2 error than ComMDM but generates static motion; a user study (30 users) prefers ComMDM over MDM and MRT §4.2 fig. 6 fig. 8 app. A.

Source: tab. 3. Full table, marks omitted. The "ours" row is the one the caption identifies as the single-layer block at layer 8 tab. 3 §4.2.

Text-to-two-person motion has no quantitative table. The paper shows qualitative samples only, and says generalisation is limited to interaction types seen in training §4.2 fig. 7.

Fine-tuned control compares fine-tuned models and DiffusionBlending with MDM inpainting, on HumanML3D test samples with text and guidance 2.5 §4.3 tab. 4.

Source: tab. 4. Full table, marks omitted. Qualitative examples are in fig. 9. The paper states that fine-tuning the MDM is crucial for control §4.3.

Comparison with TEACH on BABEL also uses the paper's own processing; the paper notes that TEACH, which aligns intervals and adds interpolation frames, was evaluated without that post-processing to keep the comparison fair §4.1.

Paper vs code

Code: https://github.com/priorMDM/priorMDM, cloned at commit 176fbee62a4d62ae007399138bfc3b1722b58715, MIT licence. The paper links the repository, and the repository is by the authors.

1. ComMDM depth. Paper: a single-layer transformer, and the ablation keeps "ours" at one layer §3.2 tab. 3. Code: default multi_num_layers is 2 parser_util.py. The loader restores the saved training arguments from the checkpoint's args.json parser_util.py, and that file is not in the repository, so the depth of the released checkpoint cannot be confirmed from the code.
2. Second-take start. Paper: T' = 700 §4.1. Code: the second take starts after skipping skip_steps_double_take = 100 of 1000 steps, so it starts at step 900 double_take.py parser_util.py gaussian_diffusion.py. Starting values differ by 200 steps; the paper does not state how T' maps to a skip count.
3. Blend length. Paper: b = 10 §4.1. Code default: 20 parser_util.py, but the README evaluation commands pass 10 README.md.
4. Soft mask values. Paper: M_hard = 0.85 and M_soft = 0.1 §4.1. Code: the mask is 1.0 by default, and the linear ramp next to each handshake rises from 0.0 to a peak of 0.85 over blend_len frames sampling_utils.py. No 0.1 appears in the code. The paper's ablation names M_soft = 0.0 and 0.2 tab. 2.
5. Text in the second take. The paper's Figure 3 says the second take refines each join "conditioned on the neighboring intervals" fig. 3. The code turns off text guidance for the second take (scale 0 and an unconditional flag) sampling_utils.py sampling_utils.py.
6. Control fine-tuning length. Paper: "under 20K steps" §3.3 and 80K steps §4.3. Code: default num_steps is 600,000 parser_util.py; the README's control examples use checkpoint 280,000 README.md.
7. BABEL training length. Paper: 1.25M steps §4.1. README's "best model" training command: 10,000,000 steps README.md. Evaluation uses checkpoint 1,250,000 README.md.
8. DiffusionBlending. Paper: a weighted difference with a free scale s eq. 3. Code: a plain average of the models' outputs with equal weights 1/n model_util.py model_blending.py. With two models and s = 0.5 these are the same, but no code path sets s to any other value.
9. Prediction target. Algorithm 1 writes ε_θ inside a loss on x_0 alg. 1. The code selects a predict-x_0 mean type when predict_xstart is set model_util.py, and the paper says MDM predicts the clean motion §3. Implement the loss on x_0.
10. Dead branch. In p_sample_loop, the non-arbitrary-length handshake branch uses alpha without defining it gaussian_diffusion.py. The DoubleTake path never reaches it, so it does not affect results.

Pitfalls

• Frame arithmetic. The unfolded length is the sum of interval lengths minus h per join double_take.py. Intervals are clamped to a minimum of 2h + 2b + 10 frames double_take.py and a maximum equal to the longest interval double_take.py. The paper does not state these limits.
• Handshake blend is applied at every step. Both sides of a handshake get the same blended value after each denoising step, not once at the end gaussian_diffusion.py §3.1.
• Mask semantics. The mask scales the injected noise: noise is multiplied by (1 − mask) at both the forward and the reverse step inpainting_gaussian_diffusion.py inpainting_gaussian_diffusion.py. A mask of 1 injects no fresh noise; a mask of 0 injects full noise. Reading the mask the other way round changes the refinement strength.
• Cost. DoubleTake denoises a batch of n intervals in the first take, then n − 1 sandwiches in the second take; the paper's runtime is not stated.
• Interval length versus training length. The HumanML3D prior was trained on clips up to 10 s §4.1. The paper does not say how well intervals longer than that are handled.
• Guidance. The paper gives guidance 2.5 only for control experiments §4.3; DoubleTake's first-take scale is only in the code (default 2.5).
• Two-person data is tiny. ComMDM is trained on 10 sequences and validated on 4, after the test split is dropped as noisy §4.2. The paper says it does not generalise to interactions unseen in training §5.
• Metric choice. The paper argues that L2 prefix error favours dull, low-frequency motion §4.2. Read the L2 table together with the user study.
• Root drift. The 3DPW root position drifts. The authors partly correct this by removing the camera's drift from the root's drift §4.2. The procedure is not detailed, so it cannot be reproduced from the paper alone.
• Figure numbering. Figures are counted in the order of their captions in the source; check against the compiled PDF when citing a figure number.

Lineage

• Builds on: Motion Diffusion Model (MDM), Tevet et al., 2023, cited as the base prior §3 §4.1. No card yet.
• Compared against: TEACH §4.1 tab. 1; MRT §4.2 tab. 3; the MDM inpainting method §4.3 tab. 4.
• Superseded by: not known from content in this wiki.