DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer

In plain words

Most earlier systems that make dances from music output a long list of body positions, one set per frame. DanceFormer works more like a human animator. It first picks the key poses, placed on the beats of the music, and then fills in the movement between them with smooth curves. Two transformer networks do the two jobs. Both are trained with a second network that tries to tell generated dances from real ones. The authors also made a dataset of dances produced by animators rather than rebuilt from video or motion capture.

TL;DR

DanceFormer splits music-to-dance generation into key poses on beats §4.1 and spline curves between them §4.2. Each stage is a transformer-style network with a kinematic propagation module (KPM) and per-joint attention (SMAM) §4.3, trained adversarially with an L2 term §4.4. On AIST++ the authors report better NPSS, FD, PVar and beat consistency than AI Choreographer tab. 4. No model code is released; only a dataset repository and a README exist (see Paper vs code).

Problem

Music-conditioned 3D dance generation is usually cast as generating a long sequence of joint rotations and translations, frame by frame. Rendering a long sequence with high kinematic complexity and coherent movement is described as an open question §1 §2. Existing 3D dance datasets come from 2D video reconstruction, which the authors say can be inaccurate, or from motion capture, which they say can suffer from noise and music misalignment §1 §3.

Method

Representation. A dance is a set of key poses plus parametric motion curves. Key poses are placed on detected music beats. Each pose holds position and quaternion rotation for each of the 24 SMPL-style joints, giving (3 + 4) \times 24 = 168 values §4.1. Between adjacent key poses, every motion curve is a 4-knot Kochanek-Bartels (TCB) spline §4.2 fig. 2. The PhantomDance format stores 3 + 4 \times 24 = 99 curves per motion file §3.2.

Stage 1: key pose generation §4.1. Beats are found with a dynamic-programming beat tracker §4.1. For each beat, the network takes a 0.8 s Hamming window centred on the beat. Each window yields 40 audio values: 13 MFCC coefficients, their first and second differences, and an energy term. A 12-dimensional chroma vector is appended §4.1. A DanTrans encoder-decoder generates the key-pose sequence, with the decoder fed its own right-shifted outputs.

Stage 2: motion curve prediction §4.2. For each pair of adjacent key poses, the input is the music features between the two beats. The decoder is fed the right-shifted curve parameters, with the two bounding key poses concatenated at the start and end §4.2. It outputs seven control parameters per curve per step, as stated in §4.2 (see Pitfalls).

DanTrans network §4.3 fig. 1. The wave encoder is a stack of transformer encoders over the music features. The motion decoder has six layers (N = 6). Each layer has the two modules below, KPM and SMAM. An extra KPM at the start of the decoder embeds the output.

• KPM. The 24 joints form a tree. A block runs a forward pass from root to leaves, then a backward pass from leaves to root. In each pass a node adds the message from its neighbour and applies an FC layer (BN, linear, ReLU) §4.3 fig. 3. The output embedding uses one such block, and each intermediate KPM uses two. All KPM outputs are 64-dimensional §4.3.
• SMAM. Each joint node is one attention head. Two sets of 24 parallel FC layers, each of width 64, produce per-joint values and keys from the music encoder output. The per-head results are not concatenated, so the joint structure is kept for the next KPM §4.3 fig. 3.

Training §4.4. Each DanTrans is treated as a generator. The paper learns two discriminators; whether they are one per stage or shared is not stated. Each discriminator is built from three-layer MLPs. It takes the music features and the sequence (key poses or curve parameters), combines them, and outputs a real-or-fake score conditioned on the music. Training adds an L2 reconstruction loss to the generator objective §4.4.

# DanceFormer inference, two stages  [§4]
# Input:  music waveform (about 30 s in the paper's example)  [§1]
# Output: key poses on beats, then spline parameters per segment

def generate_dance(music):
    # 1. Track beat times with a dynamic-programming beat tracker  [§4.1]
    beats = track_beats(music)                       # (K,) seconds
    # 2. Stage 1 input: 0.8 s Hamming window centred on each beat  [§4.1]
    beat_feats = window_features(music, beats, width=0.8)  # (K, 52)
    # 3. Stage 1 emits one key pose per beat, auto-regressively  [§4.1]
    key_poses = stage1_dantrans(beat_feats)          # (K, 168)
    # 4. Stage 2 input: features of the music between adjacent beats  [§4.2]
    segment_feats = segment_features(music, beats)   # (K-1, T, 40), T varies
    # 5. Stage 2 emits TCB control parameters per curve  [§4.2]
    curve_params = stage2_dantrans(segment_feats, key_poses)
    # 6. Sample each TCB spline between its key poses  [§3.2]
    return interpolate(key_poses, curve_params)

# FK-IK block over the 24-joint tree  [§4.3]
# Input:  node_feats (B, 24, 64); the tree runs from root to leaves
def fk_ik_block(node_feats):
    # Forward pass: root to leaves; each child adds its parent's message,
    # then applies an FC layer  [§4.3]
    node_feats = forward_pass(node_feats, parent_to_child)
    # Backward pass: leaves to root, same add-then-embed step  [§4.3]
    node_feats = backward_pass(node_feats, child_to_parent)
    return node_feats                                # (B, 24, 64)

# Structured multi-head attention: each joint node is one head  [§4.3]
# Input:  node_queries (B, 24, 64) from the decoder; music_feats (B, T, d)
def structured_attention(node_queries, music_feats):
    # 24 parallel FC layers per role give per-joint keys and values  [§4.3]
    keys = [fc_key[j](music_feats) for j in range(24)]
    values = [fc_value[j](music_feats) for j in range(24)]
    # Each joint attends on its own; no concatenation afterwards  [§4.3]
    outputs = [
        attend(node_queries[:, j], keys[j], values[j])
        for j in range(24)
    ]
    return stack(outputs, dim=1)                     # (B, 24, 64)

# Generator objective with adversarial and L2 terms  [§4.4]
# Input:  music, real (ground truth), fake (generator output)
def generator_loss(music, real, fake):
    # Adversarial term: the discriminator should score fake as real  [§4.4]
    adversarial = adversarial_term(discriminator(music, fake))
    # L2 reconstruction keeps outputs near the ground truth  [§4.4]
    reconstruction = l2_distance(fake, real)
    # The weight on the L2 term is not stated in the paper  [§4.4]
    return adversarial + l2_weight * reconstruction

Hyperparameters

Results

Author-reported. Setup: AIST++ (official split) and PhantomDance (100 test sequences); baselines are Li et al. (2020), Music2Dance (2020) and AI Choreographer (2021) §5.4 tab. 4. Metrics: NPSS, PFD, VFD, PVar and BC (defined in §5.2 and eq. 1).

Ablation on PhantomDance tab. 2:

Attention variants on PhantomDance tab. 3:

The user study fig. 5 is pairwise, with 100 participants and 400 pairs over 100 music clips, scored on performance quality and music match. Its percentages appear only in the figure, not in the text. Qualitative frames are in fig. 4. Project-page videos are cited in §5.4; they are not part of the paper.

Paper vs code

No official model code. Searched: the abstract and paper link libuyu/PhantomDanceDataset; the project page footnoted in §3.3 lists only that dataset repository; a GitHub search for "DanceFormer" returned huiye-tech/DanceFormer, whose owner is the authors' affiliation, Huiye Technology. That repository's README only links the dataset repository and has no LICENSE README.md. The dataset repository is MIT-licensed. It is a Unity3D toolkit for viewing the dataset and contains no model or training code (a grep for transformer, kinematic, MFCC, Adam and learning-rate terms found none) LICENSE. Neither repository is recorded as code.

Dataset and paper disagree in these places:

• Length: the paper gives 9.6 hours §3.1 tab. 1. The dataset README states 9.5 hours for v1.2 README.md.
• Frame rate: the paper samples 60 FPS poses for the curve-to-frames baseline §5.3. The dataset JSON is 30 FPS README.md.
• Per-joint positions: the paper's key poses carry position for each of 24 joints §4.1. The dataset stores root position plus 24 joint rotations README.md README.md and the paper's own format has 99 curves §3.2. How non-root positions are obtained is not stated.

Pitfalls

• Control-parameter count. §4.2 says seven parameters: t, c, b and the positions of two intermediate knots. Listed that way they are five scalars. What the other two are (for example, knot times) is not stated.
• Per-joint positions. The 168-dimensional key pose §4.1 and the 7-dimensional NPSS space §5.2 assume a position per joint. The data format has root position only §3.2. Derivation of joint positions is not stated.
• BC formula. eq. 1 sums over i = 1..|\mathcal{B}^x| (kinematic beats) but indexes music beats t^y_i, while the text says the average is over every music beat §5.2. The expression is an exponential of squared distance, a similarity, although the text calls it an average distance. The unit of \sigma = 3 and of the times is not stated.
• VFD arrow. The VFD column is marked ↑ in tab. 2, tab. 3 and tab. 4, but bold best values and the text ("27% gain") treat lower as better. Read it as lower is better.
• Percentage gains in the text do not match the table. §5.4 claims 30% NPSS, 28% PFD, 27% VFD, 46% PVar and 93% BC gains over AI Choreographer. From tab. 4 on PhantomDance, computed by hand: NPSS about 24% lower, PFD about 31% lower, VFD about 25% lower, PVar about 46% higher, BC about 102% higher. Use the table values.
• Split naming. §5.1 has 100 test sequences. §5.4 says the user study used "the validation set" of 100 sequences. Which split tab. 2–tab. 4 report on is not stated.
• Cross-dataset claim. The abstract says the method works "even trained by existing datasets". The body describes only per-dataset training with official AIST++ splits §5.1. No cross-dataset experiment is described.
• Loss details. The L2 weight, the discriminator loss form, the discriminator widths and the discriminators' optimizer are not stated §4.4. Adam settings are given once §5.1.
• Teacher forcing. The decoders are described as taking right-shifted outputs §4.1 §4.2. Whether training feeds ground truth or the model's own outputs is not stated.
• Stage 1 start. The paper does not give the first decoder input or start pose for stage 1.
• Variable-length segments. Stage 2 segments are the gaps between beats, so lengths vary with tempo §4.2. Padding, batching and masking are not stated.
• Local attention. The "Gaussian mask on attention results before softmax" is ambiguous: additive or multiplicative is not stated §5.3. The learned mask's size is not stated; the final model uses it.
• Encoder and feature sizes. The encoder depth and head count are not stated §4.3. The 256-dimension default §5.1 is not shown to apply to the 64-dimensional KPM and SMAM widths §4.3.
• Commented-out drafts. The LaTeX source contains commented-out text about an earlier learned local attention (kernel size and standard deviation) and a BCE discriminator loss. These are not in the compiled paper, and this card does not use them. Source: raw/src/2103.10206/danceformer.tex.
• Compute. Training uses four GPUs for 300k to 400k steps §5.1. Inference cost and run time are not reported.

Lineage

• Builds on: the Transformer encoder-decoder §4.3; Kochanek-Bartels (TCB) splines as an animation primitive §4.2; the dynamic-programming beat tracker §4.1; the AIST++ dataset and its FD and beat metrics §5.2; local attention §5.3; graph neural networks as a KPM baseline §5.3.
• Compared against: Li et al. (2020) two-stream motion transformer; Music2Dance (2020); AI Choreographer (2021), the closest state of the art per §5.4 tab. 4.
• Superseded by: not stated in this wiki.