Executing your Commands via Motion Diffusion in Latent Space

In plain words

Making a computer character move from a sentence such as "a person waves with both hands" is hard, because words and motion capture data look nothing alike. The usual tool, a diffusion model, works on the raw motion, which is long, repetitive and full of capture noise. That makes it slow to train and run, and the noise can leak into the output as artifacts.

This paper works in a smaller space. First, an autoencoder squeezes a whole motion clip into a short code that can be turned back into motion. Second, a diffusion model learns to produce such codes from the condition, starting from random noise and removing it step by step. To generate a motion, the diffusion model makes a code and the decoder turns it into a clip. The condition can be a text description or an action label, and the same code space serves both.

TL;DR

• MLD is a latent diffusion model for conditional human motion: a transformer VAE encodes a motion into a latent, and a transformer denoiser runs diffusion in that latent. §3
• The best diffusion setting uses a single 256-d latent token. §5 tab. 5
• Text conditions use a frozen CLIP encoder; action conditions use a learned embedding per class. Both use classifier-free guidance. §3.3
• Author-reported: best FID on HumanML3D and KIT among the compared methods, and much faster sampling than MDM. The text's claims of best R-Precision and MM Dist do not match the tables. See Pitfalls.

Problem

Text and motion come from very different distributions, so a direct mapping from text to raw motion is hard to learn. Cross-modal latent alignment methods (TEMOS-style) can lose diversity because two simple Gaussians are forced into a shared space. Diffusion on raw motion sequences is costly, and raw capture data is redundant and noisy. §1 §3.2

Method

MLD has two stages. Stage 1 trains a motion VAE, \mathcal{V} = \{\mathcal{E}, \mathcal{D}\}, to reconstruct motion through a latent z §3.1. Stage 2 freezes the encoder and trains a conditional denoiser \epsilon_\theta in latent space §3.2 §3.3.

Motion VAE §3.1. The encoder is a transformer with learnable distribution tokens, and it outputs the parameters of a Gaussian over a latent of size n \times d. The decoder is a transformer whose queries are L zero motion tokens and whose memory is the latent. Both use long skip connections, and the paper removes the action biases used in its ACTOR base. The VAE is trained on reconstruction and KL only. The appendix gives the objective as a reconstruction term plus \lambda_{reg} times a KL term app. H.1. The per-frame motion representation is described in app. G.

Latent diffusion §3.2. The forward process on the latent is a Markov noising process with per-step coefficient \alpha_t \in (0, 1) eq. 1:

q\left(z_t \mid z_{t-1}\right)=\mathcal{N}\left(\sqrt{\alpha_t}\, z_{t-1}, \left(1-\alpha_t\right) I\right) eq. 1

The training objective is the noise-prediction MSE, with z_0 = \mathcal{E}(x) and \mathcal{E} frozen eq. 2:

L_{\text{MLD}} := \mathbb{E}_{\epsilon, t}\left[\left\|\epsilon - \epsilon_\theta(z_t, t)\right\|_2^2\right] eq. 2

The denoiser is a transformer with long skip connections on the latent §3.2. At inference, the denoiser predicts \hat z_0 in T steps and the decoder runs once §3.2.

Conditioning §3.3. The condition enters as an embedding \tau_\theta(c) and is trained with the conditional objective eq. 3:

L_{\text{MLD}} := \mathbb{E}_{\epsilon, t, c}\left[\left\|\epsilon - \epsilon_\theta\left(z_t, t, \tau_\theta(c)\right)\right\|_2^2\right] eq. 3

Text uses a CLIP text encoder, which is frozen §3.3 §4.2. Action uses a learnable embedding per category §3.3. The paper compares concatenation with cross-attention and reports that concatenation works better §3.3 tab. 5.

Classifier-free guidance §3.3. During training, the condition is dropped for 10% of samples. At inference the two noise predictions are combined with scale s eq. 4:

\epsilon^s_\theta(z_t, t, c) = s\, \epsilon_\theta(z_t, t, c) + (1 - s)\, \epsilon_\theta(z_t, t, \varnothing) eq. 4

The paper states s > 1 strengthens the guidance eq. 4 §3.3. The appendix sweeps dropout p and scale s and recommends (p, s) = (0.1, 7.5) app. E tab. 11.

# Stage 1: motion VAE training  [§3.1] [app. H.1]
# Input: motions (B, L, F); per-frame features F as in [app. G]
# Output: scalar loss

def vae_training_step(motions, lengths):
    # Encode each motion into n latent tokens of width d = 256 [§3.1]
    mu, logvar = encode(motions, lengths)           # (B, n, d), n = 1 for MLD-1
    # Reparameterized sample; this sample is what the decoder receives [§3.1]
    z = mu + exp(logvar / 2) * random_normal_like(mu)  # (B, n, d)
    # Decode with L query tokens and cross-attention to z [§3.1]
    recon = decode(z, lengths)                      # (B, L, F)
    # Reconstruction term; the paper says MSE, the code uses Smooth L1 [§3.1]
    # [code: mld/models/losses/mld.py#L79-L82]
    loss_recon = smooth_l1(recon, motions)
    # KL between the posterior and N(0, I), averaged [§3.1]
    # [code: mld/models/losses/mld.py#L155-L162]
    loss_kl = kl_to_standard_normal(mu, logvar)
    return loss_recon + LAMBDA_KL * loss_kl

# Stage 2: latent diffusion training  [§3.2] [eq. 2] [eq. 3]
# Input: motions (B, L, F) and conditions c (text strings or action labels)
# Output: scalar loss

def diffusion_training_step(motions, lengths, conditions):
    # The encoder is frozen; take one latent per motion [§3.2]
    z0 = encode(motions, lengths)                   # (B, n, d)
    # Drop the condition for 10% of samples (empty text or zero action) [§3.3]
    conditions = drop_with_probability(conditions, 0.1)
    # Sample a timestep in [0, 1000) and Gaussian noise per sample [§4.2]
    t = random_int(0, 1000, size=B)
    eps = random_normal_like(z0)
    z_t = add_noise(z0, eps, t)                     # (B, n, d)
    # Predict the noise under the condition embedding [eq. 3]
    eps_hat = denoiser(z_t, t, embed(conditions))
    # Plain MSE between noise and prediction [eq. 2] [eq. 3]
    return mean((eps - eps_hat) ** 2)

# Sampling: DDIM with classifier-free guidance  [§3.3] [eq. 4] [app. C]
# Input: condition c, lengths, guidance scale s = 7.5 [app. E]
# Output: motion (B, L, F)

def sample_motion(c, lengths, s=7.5, steps=50):
    # Start from Gaussian noise in latent space [code: mld/models/modeltype/mld.py#L303-L310]
    z_t = random_normal((B, n, d))
    # Empty condition for the unconditional branch [eq. 4]
    empty = embed("")                              # zero vector for actions
    for t in ddim_timesteps(steps):                # 50 steps, eta = 0 [§4.2]
        # Conditional and unconditional noise predictions [eq. 4]
        eps_c = denoiser(z_t, t, embed(c))
        eps_u = denoiser(z_t, t, empty)
        # Guided prediction: s * eps_c + (1 - s) * eps_u [eq. 4]
        eps = s * eps_c + (1 - s) * eps_u
        z_t = ddim_step(z_t, eps, t)
    # Decode the final latent once with the frozen decoder [§3.2]
    return decode(z_t, lengths)

Hyperparameters

Values are as stated in the paper unless marked "code". Where the paper is silent, the table says so.

Results

Author-reported. Setup: 95% confidence interval from 20 runs of the evaluation §4.3 §4.4. Baseline numbers are mostly copied from their papers or from a benchmark, except TEMOS, which the authors trained §4.3. Metric definitions are in §4.1 and app. I.

Text-to-motion, HumanML3D tab. 1

Text-to-motion, KIT tab. 2

Action-to-motion tab. 3

HumanAct12 MModality: MLD 2.824, MDM 2.520, INR 2.569, ACTOR 2.530 tab. 3. UESTC MModality: MLD 13.57, MDM 14.26, INR 14.68, ACTOR 14.52 tab. 3.

Unconditional generation, HumanML3D tab. 6. MLD-1 with the skip denoiser: FID 1.055, diversity 8.577. MDM: FID 8.848, diversity 6.429. ACTOR: FID 14.14, diversity 5.123. VPoser-t: FID 36.65, diversity 3.259. tab. 6 also lists a row with the same numbers for z with 1 token (see Pitfalls).

VAE reconstruction, HumanML3D motion part tab. 4. MPJPE and PAMPJPE in mm; ACCL is acceleration error.

Diffusion ablation, text-to-motion, HumanML3D tab. 5

Inference cost, text-to-motion tab. 10 app. C. Total time in seconds to generate 2048 clips on one V100, by number of DDIM steps (50, 100, 200) and DDPM 1000 steps.

Main-text inference statement: MDM needs 24.74 s average per sentence and up to a minute at worst, on one V100 §5. The per-sentence AITS plot is Figure 6, which this card does not quote.

Classifier-free guidance sweep, HumanML3D tab. 11 app. E. At s = 7.5 the dropout sweep is:

At p = 0.10, the scale sweep gives FID 0.401 at s = 1.5, 0.350 at s = 2.5, 0.358 at s = 3.5, 0.375 at s = 4.5, 0.396 at s = 5.5, 0.424 at s = 6.5, 0.473 at s = 7.5, 0.504 at s = 8.5 and 0.555 at s = 9.5 tab. 11. R-Prec Top 3 rises from 0.648 (s = 1.5) to 0.772 (s = 7.5) tab. 11.

Figures. Figure 3 shows qualitative text-to-motion comparisons fig. 3. Figure 4 shows unconditional qualitative results fig. 4. Figure 5 plots unconditional FID and diversity against baselines fig. 5. The user study is described in words only; it reports that MLD is preferred over MDM and T2M app. F. No preference rates are given in the text.

Paper vs code

Compared against the official repository at commit 081ce3152e5a95b14a4495fb2d32939c257c6216 (MIT license, copyright Chen Xin). The repository is linked on the paper's title page and in the appendix introduction [title page] app. intro, so it is treated as the official code.

• Reconstruction loss. The paper uses MSE for the VAE §3.1. The code uses Smooth L1 for the feature and joint reconstruction terms mld.py and the update sums those terms mld.py.
• Joint loss. The paper's VAE objective has no joint-position term §3.1 app. H.1. The code adds one, weighted by LAMBDA_JOINT = 1.0 mld.py base.yaml.
• Mesh loss. The appendix gives an SMPL vertex loss L_{mesh} app. H.1. The code defines recons_verts but the VAE update does not add it mld.py mld.py. The SMPL path is not active in the HumanML3D config.
• KL term. The KL weight is not given in the paper. The code uses 10^{-4} config_mld_humanml3d.yaml with .mean() reduction mld.py.
• Diffusion loss. Both use MSE on the noise eq. 2 [code: mld/models/losses/mld.py#L66-L68] mld.py. The code takes the mean over all elements, so the loss scale differs from a per-sample squared norm by a constant factor.
• Latent sampling. The paper writes z_0 = \mathcal{E}(x) eq. 2. The code draws a reparameterized sample from the posterior every time it encodes mld_vae.py. Diffusion training therefore sees a random latent for each motion, not the posterior mean.
• Denoiser depth for actions. The appendix table lists 9 layers for the denoiser tab. 13. The action config uses 15 [code: configs/modules_humanact12/denoiser.yaml#L6]. The text config uses 9 denoiser.yaml.
• Diffusion epochs. The text says 3K diffusion epochs §4.2. The text config uses 2000 config_mld_humanml3d.yaml, and the appendix table gives 2000 for MLD-1 tab. 13.
• Noise schedule. The text says \beta_t is "scaled linearly" §4.2. The code uses diffusers' scaled_linear scheduler.yaml.
• Null text condition. The paper's \varnothing is not defined eq. 4. The code encodes the empty string with the CLIP encoder during training and sampling mld.py mld.py. For actions, the null embedding is a zero vector mld_denoiser.py.
• Latent scale. The paper does not mention a latent scale factor. The code applies none to z before diffusion mld.py and none after sampling mld.py.
• Text width. The paper says \tau_\theta(c) and w_{clip} are \mathbb{R}^{1 \times 256} §4.2 §4.3. The denoiser expects 768-d text features and projects them to 256 with ReLU and Linear denoiser.yaml mld_denoiser.py. The paper does not describe that projection.

Pitfalls

• Claims that the tables do not support. Section 4.3 credits MLD with the top score on FID, R-Precision and MM Dist, on both datasets §4.3. In the tables, MotionDiffuse has the best R-Precision and MM Dist on both datasets, and MLD is second on those metrics tab. 1 tab. 2. Only FID is clearly best for MLD.
• Speed claim. The abstract and §5 claim two orders of magnitude over diffusion on raw motion. The appendix table gives about 22x (DDIM-50) and about 30x (DDPM-1000) in total time tab. 10. The per-sentence AITS plot (Figure 6) is the only other evidence and is not quoted here.
• Layer count is not FID-optimal. Five denoiser layers give the lowest FID (0.314) in the ablation, against 0.473 for the chosen nine tab. 5. The text calls the layer counts "similar" §5.
• Stochastic latents. Because the encoder samples, the same motion gives different latents on each call. Diffusion targets and test-time decoding therefore differ from the posterior-mean picture in eq. 2.
• Length at inference. The paper does not say how a motion length is chosen when generating from text alone. The code takes lengths from the test batch mld.py.
• Duplicate-looking numbers. In the unconditional table, z with one token and the skip denoiser both show FID 1.055 and diversity 8.577 tab. 6. Check against the code before trusting either row.
• Model size. The text says 25M parameters for MLD-1 §4.3. The appendix table says 26.9M without CLIP tab. 13.
• Baselines copied, not re-run. Most baseline numbers come from their own papers or a benchmark §4.3. Evaluation settings may differ.
• Guidance cost. Guidance doubles the batch at inference mld.py mld.py. Benchmarks in app. C use the same two-pass cost.
• Scheduler offset. The code uses DDIM with steps_offset=1 scheduler.yaml. The paper does not state it.

Lineage

• Builds on: latent diffusion in image generation, which the paper cites as the inspiration for working in latent space §1. The motion VAE is a transformer autoencoder following ACTOR, with long skip connections §3.1.
• Compared against: TEMOS, T2M, MDM, MotionDiffuse, Seq2Seq, LJ2P, T2G, Hier (text-to-motion) §4.3 tab. 1 tab. 2; ACTOR, INR, MDM (action-to-motion) §4.4 tab. 3; VPoser-t and ACTOR (unconditional) §4.5 tab. 6.
• Superseded by: not known from content already in this wiki.