MotionCLIP: Exposing Human Motion Generation to CLIP Space

In plain words

Human motion data is scarce, and its text labels are sparse. A model trained only on motion does not learn the meaning of words well. MotionCLIP starts from CLIP, a model that already knows a lot about how words and pictures relate, and borrows that knowledge for motion.

Each human motion is squeezed into one vector, and the network is trained so that this vector points the same way as CLIP's vector for the motion's text label and for a picture of the pose. A second network turns the vector back into a motion. CLIP's vectors already place similar meanings close together, so the motion vectors take on that same layout.

The payoff is that you can type a phrase, get CLIP's vector for it, and decode a motion from that vector. This works even for phrases that never appear in the motion data, such as "couch" or "Spiderman". Because the vectors can be added and subtracted, the same space also supports blending and editing motions.

TL;DR

• MotionCLIP is a transformer auto-encoder for SMPL body motion, trained on 60-frame sequences at 30 fps §4.1.
• Its latent vector is trained to reconstruct the motion and to match CLIP's embedding of the motion's text labels (cosine distance) and of a rendered frame §3.
• CLIP stays frozen. A text prompt is encoded by CLIP and decoded directly into motion, with no text-motion training §4.2.
• Author-reported: in user studies, text-to-motion is preferred over JL2P (76.7% in-domain, 75.3% out-of-domain) tab. 1. On the BABEL-60 action-recognition benchmark, top-1 is 40.9% against 41.14% for 2s-AGCN tab. 3.
• The official code differs from the paper in loss weights, latent size and encoder token count. Read Paper vs code before implementing.

Problem

Labelled motion capture is scarce and expensive to acquire and label. Neural models trained only on labelled motion data do not generalize well to the full motion manifold or to natural language that describes it §1. The paper's goal is therefore a motion latent space that carries semantic structure from a knowledge-rich model §1.

Method

The paper trains a transformer motion auto-encoder whose latent space is aligned to CLIP's joint text-image space §1, §3. Training uses three losses: a reconstruction loss, a text loss that connects each motion to the CLIP embedding of its text label, and an image loss that connects each motion to the CLIP embedding of a rendered frame showing the pose §3. The figure of the architecture (labelled fig:architecture, §3) shows a transformer encoder that maps a motion to one latent vector and a transformer decoder that recovers the motion by attending to that vector.

Motion representation. A sequence has length T and is written p_{1:T}, where each frame p_i \in \mathbb{R}^{24 \times 6} holds 6D orientations for the global body orientation and 23 SMPL joints §3. Vertex positions v_{1:T} are computed from SMPL with \beta = 0 and a neutral-gender body model, following Petrovich et al. §3.

Encoder. Each frame is linearly projected to the model width and given a standard positional embedding. A learned prefix token z_{tk} is prepended. The first output is the latent z_p = E(z_{tk}, p_{1:T}), and the other outputs are dropped §3. The paper describes one prefix token. The code uses two learned tokens and reads the first output transformer.py. See Paper vs code.

Decoder. The latent is the key and value of a transformer decoder. The queries are the positional encoding of frames 1..T. Each output frame goes through a linear projection to pose space, giving \hat{p}_{1:T} = D(z_p) §3. The code uses zero-valued time queries plus the same sinusoidal positional encoding, which gives the same queries transformer.py.

Reconstruction loss. The loss compares orientations, vertex positions and frame-to-frame orientation velocities eq. 1:

\mathcal{L}_{recon} = \frac{1}{|p|T} \sum_{i=1}^{T} \lVert p_i - \hat{p}_i \rVert^2 + \frac{1}{|v|T} \sum_{i=1}^{T} \lVert v_i - \hat{v}_i \rVert^2 + \frac{1}{|p|(T-1)} \sum_{i=1}^{T-1} \lVert (p_{i+1} - p_i) - (\hat{p}_{i+1} - \hat{p}_i) \rVert^2 eq. 1

The paper does not define the normalisers |p| and |v| beyond the notation. The code uses element-wise mean squared error (see Paper vs code).

Text and image losses. Each loss is the cosine distance between the latent and a frozen CLIP embedding. The text loss uses the motion's text label eq. 2. The image loss uses the rendered frame eq. 3:

\mathcal{L}_{text} = 1 - \cos\big(\mathrm{CLIP}_{text}(t),\, z_p\big) \qquad \mathcal{L}_{image} = 1 - \cos\big(\mathrm{CLIP}_{image}(s),\, z_p\big) eq. 2 eq. 3

The rendered frame s is a single pose taken from the sequence and rendered to a synthetic image in an unsupervised way §3, §4.1. The total objective is eq. 4:

\mathcal{L} = \mathcal{L}_{recon} + \lambda_{text}\,\mathcal{L}_{text} + \lambda_{image}\,\mathcal{L}_{image} eq. 4

The equation numbers above follow the order of equations in the LaTeX source. The paper's values for \lambda are set in §4.1 and differ from the code (see Paper vs code).

Training step. The steps below combine the paper's equations with the code's training loop trainer.py.

# One training step, following [§3] and [eq. 1] to [eq. 4]
# Input:  motions (B, T, 24, 6) in 6D rotation form; texts (B); frames
# Output: scalar loss
def training_step(motions, texts, frames):
    # Encode each sequence into one latent vector  [§3]
    latents = encoder(motions)                    # (B, latent_dim)
    # Decode the latent into T frames of pose  [§3]
    recons = decoder(latents)                     # (B, T, 24, 6)
    # Reconstruction over orientations, vertices and velocities  [eq. 1]
    loss_recon = reconstruction_loss(motions, recons)
    # Frozen CLIP embedding of each text label  [§3]
    text_features = clip_text_encoder(texts)      # (B, d_clip)
    # Cosine distance between the latent and the text embedding  [eq. 2]
    loss_text = mean(1 - cosine(text_features, latents))
    # Frozen CLIP embedding of one rendered frame per sequence  [§3]
    image_features = clip_image_encoder(frames)   # (B, d_clip)
    # Cosine distance between the latent and the image embedding  [eq. 3]
    loss_image = mean(1 - cosine(image_features, latents))
    # Weighted sum of the three terms  [eq. 4]
    return loss_recon + lambda_text * loss_text + lambda_image * loss_image

Text-to-motion (inference only). The prompt is encoded by the frozen CLIP text encoder, and the output vector is decoded directly into a motion. No further training is done §4.2. The code encodes the prompt with CLIP and calls the decoder with a fixed duration of 60 frames by default visualize.py, generate.py.

# Text-to-motion at inference, with no training  [§4.2]
# Input:  prompt (str); num_frames (int)
# Output: motion (num_frames, 24, 6)
def text_to_motion(prompt, num_frames):
    # Encode the prompt with frozen CLIP; the raw vector is used  [§4.2]
    text_latent = clip_text_encoder(prompt)       # (d_clip,)
    # Decode the vector directly into a motion  [§4.2]
    return decoder(text_latent, num_frames)

Latent operations. The paper uses three operations on the latent space §4.3:

• Interpolation. Linear interpolation between two latent codes gives intermediate motions. The code uses weights that run linearly from 0 to 1 visualize.py.
• Editing by arithmetic. Adding a style vector (for example, the CLIP embedding of a style word) to a content latent changes the style. Upper-body and lower-body actions can also be composed §3, §4.3.
• Action recognition. A motion latent is compared with the CLIP embeddings of 60 class names, and softmax is applied to the similarities §4.3.

# Action recognition with CLIP class-name embeddings  [§4.3]
# Input:  latent (d_latent,); class_names (60 strings)
# Output: probabilities over the classes
def classify(latent, class_names):
    # Frozen CLIP embedding of each class name  [§4.3]
    class_features = clip_text_encoder(class_names)
    # Cosine similarity, scaled by 100 in the code, then softmax  [§4.3]
    similarity = 100.0 * cosine(latent, class_features)
    return softmax(similarity)

Scaling in the code. The classifier multiplies the cosine similarity by 100 before softmax action_classifier.py. The paper does not state this scale.

Hyperparameters

Results

Author-reported. Setup: BABEL §4.1; text-to-motion compared with JL2P on a user study, with a new common evaluation set §4.2; action recognition on the BABEL-60 benchmark, with the 2s-AGCN numbers "as reported by Punnakkal et al." §4.3. Each row cites its source table. The number of participants and of samples in the user studies is not stated.

Action recognition tab. 3

Text-to-motion user study tab. 1. Preference is side-by-side, against JL2P.

The out-of-domain set is built from Olympic sports that appear in neither dataset §4.2. Other text-to-motion evidence is qualitative: sports prompts §4.2, free-text styles §4.2, and abstract or cultural prompts such as "couch" and "Spiderman" §4.2. The paper reports no automated motion-quality metric for text-to-motion.

Style from text, against exemplar-based transfer tab. 2. Preference is side-by-side, against Aberman et al., which receives motion inputs.

The paper reads these results as comparable, with an expected edge for the method that was trained for style §4.2.

Latent-space results are qualitative: interpolation, editing and style arithmetic, with figures and no tables §4.3.

Paper vs code

Checked against the official repository at code_commit (MIT). The README's "paper-model" command is taken as the reference for the code's settings README.md.

• CLIP loss weight. The paper sets both \lambda values to 0.01 "throughout" §4.1. The code default for the cosine weight is 1.0 model.py, and the README run does not change it README.md. The README run uses reconstruction weights of 100 README.md, so the ratio of CLIP to reconstruction weight is 0.01. That matches the paper's ratio, but the absolute values differ. This ratio is inferred from the arguments, not stated by the code.
• Reconstruction weights. Eq. 1 has no weights eq. 1. The README run uses 100 for each of rc, rcxyz and vel README.md.
• Normalisation. Eq. 1 divides by |p|T, |v|T and |p|(T-1) eq. 1. The code uses F.mse_loss with mean reduction for orientations, vertices and velocities losses.py, losses.py, losses.py. The two differ by a constant factor that depends on the unstated dimension counts.
• Velocity term. Eq. 1 defines the velocity term on the orientation representation eq. 1. The code computes it as the difference between consecutive frames on the 6D output, masking the padded frames losses.py. This matches the equation. The vertex-velocity variant velxyz exists but is not in the default model name get_model.py, model.py.
• Latent dimension. The paper does not state it. The code default is 256 model.py, and the README run uses 512 README.md.
• Prefix tokens. The paper describes one learned token §3. The code creates two (muQuery and sigmaQuery), concatenates both before the frames, and returns only the first output as \mu transformer.py, transformer.py, transformer.py. The second output (logvar) is unused. The latent is therefore deterministic, with no KL term or sampling.
• Training data. §4.1 says the model is trained on BABEL §4.1. The README "paper-model" command uses an AMASS-derived database README.md, README.md. The README's classes-model command uses the BABEL database README.md. The code's BABEL labels are attached to AMASS sequences in the parser amass_parser.py. Which sequences the paper model actually used is not verified beyond the README.
• Sequence chunking. §4.1 says the data is cut into sequences of length 60 §4.1. The dataset class splits stored sequences into chunks of 100 frames amass.py. How 60-frame training windows are drawn from these chunks was not traced.
• Text label. §4.1 says all actions in a sequence are concatenated into one string, without a separator §4.1. The code joins them with "and " when the BABEL annotation has no per-frame labels amass_parser.py.
• Rendered frames. §4.1 renders one random frame per sequence with Blender and the SMPL-X add-on §4.1. The README downloads pre-rendered images into ./data/render README.md. The repository has no rendering script. The parser loads the PNG files for each sequence amass_parser.py.
• Softmax scale. §4.3 says softmax over cosine similarities §4.3. The code multiplies by 100 first action_classifier.py.
• Class-name training. §4.3 says the recognition model is trained on class names rather than raw text §4.3. The README run for this model is at README.md, and it does not show how class-name text is selected. That selection was not traced.
• Reproduction history. Early releases did not reproduce the paper's model. The README marks this resolved on 31 August 2022 README.md, README.md.

No further discrepancies were found in the loss structure, the CLIP freezing, or the text-to-motion inference path.

Pitfalls

• Latent norm is not controlled. The cosine alignment terms used in the README run ignore the latent's norm, so the norm is never constrained motionclip.py, motionclip.py, motionclip.py. At inference, the text-to-motion path passes raw CLIP text features to the decoder without normalisation visualize.py. Interpolation also works on raw vectors visualize.py. Implement the same convention, or check that the decoder behaves the same on normalised vectors. The paper does not discuss this.
• Use 512 latent dimensions. The cosine terms compare the latent with a CLIP embedding, so the sizes must match. The code default of 256 would not give a valid comparison with a CLIP ViT-B/32 embedding. The README run uses 512 README.md.
• Tokenisation length. The code calls clip.tokenize without a truncation option motionclip.py. The upstream CLIP library rejects prompts longer than its context length by default. Concatenated multi-action labels can be long. This is library behaviour, not verified in this repository. Check label lengths before training.
• Under-specified training details. Optimizer, learning rate, batch size, epochs, heads, feed-forward width and dropout are not in the paper. The implementation uses AdamW with learning rate 1e-4 and batch size 20 (README run) train.py, README.md. Heads, feed-forward width and dropout use constructor defaults transformer.py. The code does not set norm_first, so PyTorch's default layer order applies. The paper does not specify the layer order.
• Data licences. The repository says the datasets (AMASS, BABEL, rendered images) and the SMPL/SMPL-H models carry their own licences that must also be followed README.md. This card does not verify those licences. For a commercial site, check them before using the data, the pre-rendered images, or the body models.
• Evaluation is subjective. Text-to-motion results come from user studies only tab. 1, and the numbers of participants and samples are not stated. Treat the preference numbers as the authors' evidence, not as a benchmark score.
• Action recognition is close to, not better than, 2s-AGCN. MotionCLIP is 40.9% top-1 against 41.14% for 2s-AGCN, but 57.71% top-5 against 73.18% tab. 3. Removing the text loss drops top-1 to 4.54% tab. 3, which shows the text term is essential to this result in the paper's ablation.
• Figure numbers are not confirmed. The figures are cited by their label, not by number. The equation numbers follow source order eq. 1-eq. 4. Tables are numbered in source order.

Lineage

• Builds on: CLIP (cited in the paper as radford2021learning) as the frozen text and image encoder §1, §3. The transformer motion auto-encoder is adapted from the ACTOR setup (Petrovich et al.) §3 README.md.
• Compared against: JL2P (Ahuja et al., 2019) for text-to-motion §4.2 tab. 1; Aberman et al. (2020) for style transfer §4.2 tab. 2; 2s-AGCN (Shi et al., 2019) for action recognition §4.3 tab. 3.
• Superseded by: not stated.