Robust Motion In-betweening

In plain words

Animators usually draw a few key poses and then have to fill in the motion between them, frame by frame. This paper trains a neural network to do that filling. Given the last few frames before a gap and the pose the character must reach, it produces the frames that lead smoothly into that pose, arriving on time.

A plain network of this kind does not know how many frames are left before it has to arrive, so transitions can stall or leave gaps. The authors add two small signals to its internal state. One is a countdown code that tells the network how far away the target is. The other is a noise signal aimed at the target: strong when the target is far away, fading to nothing as it gets close. Together they let the same two keyframes produce different, still plausible motions. Two discriminator networks push the output toward realistic movement. The method runs as a plugin inside an animation tool.

TL;DR

Robust Motion In-betweening trains an LSTM generator, with two adversarial critics, to produce the frames between seed frames and a target keyframe. Two additive embeddings make it robust: a sinusoidal time-to-arrival code, and a scheduled target noise that fades near the target. On the Human3.6M walking benchmark, the complete model's average L2Q is 0.82 against 1.02 for linear and spherical interpolation. It beats interpolation on all three metrics at all four lengths of the new LaFAN1 benchmark (author-reported). The official repository holds the dataset and the baseline evaluation, not the generator.

Problem

Hand-authored keyframes and motion-capture pipelines are slow, and curve-fitting between keyframes is limited in how much motion it can express §1. A state-of-the-art motion predictor can be turned into a transition generator by adding the target keyframe as conditioning, but this fails in two ways: a single model does not handle transitions of different lengths robustly, and the architecture is deterministic, so it cannot produce variations for the same keyframes §1 §3.3.

The task is defined as transition generation under sparse constraints: large gaps must be filled without dense control signals such as trajectory or contact information during the gap §2.3. The model takes 10 frames of past context and one target keyframe as inputs §1. According to the paper, gap length is not hard-capped, but performance is only expected up to a soft limit on length §1.

Method

The model is a recurrent generator in a GAN framework. It is built on the input separation of the Recurrent Transition Network (RTN) §3.2 fig. 2, and it adds two embedding modifiers, a time-to-arrival code and a scheduled target noise, plus two adversarial critics and reconstruction losses §1 §3.6.

1. Data. Poses are local quaternions for every joint (4 values per joint) plus a 3-D global root velocity §3.1. LaFAN1 also gives 4 binary foot-contact labels per frame §3.1. The offsets to the target are the root position offset and element-wise quaternion differences, not relative rotations §3.1. Positions come from forward kinematics with constant bone offsets §3.1. Each sequence is rotated around the up axis so that the root faces the +X axis on the last seed frame, and sequences are mirrored over the +X axis with probability 0.5 during training §3.1.
2. Generator. Three fully connected encoders (character state, offset, target) each have a 512-unit hidden layer and a 256-unit output, with PLU activations §3.2. The time-to-arrival code is added to the encoder outputs §1 §3.2. The offset and target embeddings are concatenated, and the scaled target noise is added to them §3.4. The three augmented embeddings feed an LSTM §3.2. A decoder with hidden layers of 512 and 256 units and a linear output predicts quaternion and root velocities §3.2. The predicted quaternions are normalized to unit length, then forward kinematics gives the global positions §3.2.
3. Time-to-arrival code. A sinusoidal positional encoding of the number of frames until the target, with basis 10000 as in transformer implementations §3.3. It is capped: the value is held constant past a maximum duration §3.3.
4. Scheduled target noise. A 512-dimensional vector z_target is sampled once per sequence from a zero-centered Gaussian with standard deviation 0.5 §3.4 §3.7.3. A multiplier λ scales it down as the target nears: zero within 5 frames of the target, a linear ramp over the next 25 frames, and full strength beyond 30 frames §3.4. The authors argue that additive noise is used because concatenated noise is ignored by the generator §3.4.
5. Critics. Two feed-forward critics, C1 on 10-frame windows (long-term) and C2 on 2-frame windows (short-term), each with 3 fully connected layers of 512 and 256 ReLU units and a linear output §3.5. Their scores are averaged over time §3.5 §3.6.2. They are trained as a Least Squares GAN §3.6.2. In practice they are 1-D temporal convolutions with stride 1, no padding, and receptive field 1 in the last two layers §3.7.2 app. A.1.
6. Reconstruction losses. L1 norms of quaternions, root, positions, and (LaFAN1 only) contacts, averaged over frames §3.6.1. The paper says global positions help implicitly weight the bone hierarchy. In theory they are not needed given the quaternion and root losses §3.6.1.
7. Training. Minibatches of 32 sequences, AMSgrad with learning rate 0.001 §3.7.3. The transition length is sampled uniformly per minibatch between a minimum of 5 frames and a current maximum that grows over epochs toward the true maximum §3.7.1.

# Time-to-arrival embedding  [§3.3]
# tta: frames until the target keyframe; dim: embedding size (256)  [§3.2]
# Cap t_max_tta = T_max(trans) + T_past - 5  [§3.3]
def time_to_arrival_embedding(tta, dim=256, basis=10000.0):
    # Hold the code constant beyond the cap  [§3.3]
    tta = min(tta, t_max_tta)
    embedding = zeros(dim)
    # Pair i shares one frequency; i runs over 0 .. dim/2 - 1  [§3.3]
    for i in range(dim // 2):
        angle = tta / basis ** (2 * i / dim)
        embedding[2 * i] = sin(angle)
        embedding[2 * i + 1] = cos(angle)
    return embedding

# Target-noise multiplier  [§3.4]
def target_noise_multiplier(tta):
    # No target noise within the last 5 frames before the keyframe  [§3.4]
    if tta < 5:
        return 0.0
    # Linear ramp over the 25 frames before that  [§3.4]
    if tta < 30:
        return (tta - 5) / 25
    # Full target noise further away than 30 frames  [§3.4]
    return 1.0

# One generator step  [§3.2] [fig. 2]
# state: (B, 4j + 3) quaternions and root velocity  [§3.1]
# offset, target: conditioning vectors for each sample  [§3.2]
def generator_step(state, offset, target, tta, target_noise):
    # Three separate encoders, each with 256-dim output  [§3.2]
    state_hidden = state_encoder(state)           # (B, 256)
    offset_hidden = offset_encoder(offset)        # (B, 256)
    target_hidden = target_encoder(target)        # (B, 256)
    # Add the time-to-arrival code to each encoder output  [§1] [§3.2]
    tta_code = time_to_arrival_embedding(tta)     # (B, 256)
    state_hidden = state_hidden + tta_code
    offset_hidden = offset_hidden + tta_code
    target_hidden = target_hidden + tta_code
    # Concatenate offset and target, then add scaled target noise  [§3.4]
    scale = target_noise_multiplier(tta)          # (B,)
    target_pair = concatenate(offset_hidden, target_hidden)   # (B, 512)
    target_pair = target_pair + scale * target_noise          # (B, 512)
    # Feed the three augmented embeddings to the LSTM  [§3.2]
    recurrent_input = concatenate(state_hidden, target_pair)  # (B, 768)
    hidden, cell = lstm_cell(recurrent_input, hidden, cell)   # size not stated
    # Decode the recurrent output into velocities  [§3.2]
    velocities = decoder(hidden)                  # (B, 4j + 3)
    quat_velocity, root_velocity = split_velocities(velocities)
    # Integration rule into the next state is not stated  [§3.2]
    next_quats = integrate(quats, quat_velocity)
    next_root = integrate(root, root_velocity)
    # Normalize quaternions to unit length  [§3.2]
    next_quats = normalize_unit(next_quats)
    # Forward kinematics with the constant bone offsets  [§3.1] [§3.2]
    next_positions = forward_kinematics(next_quats, next_root, bone_offsets)
    return next_quats, next_root, next_positions

# Generator loss  [§3.6] [§3.7.3]
# Reconstruction terms are L1 norms averaged over frames  [§3.6.1]
def generator_loss(generated, truth, critics):
    # Weights: quaternions 1.0, root 1.0, positions 0.5  [§3.7.3]
    loss = 1.0 * reconstruction_loss(generated.quats, truth.quats)
    loss = loss + 1.0 * reconstruction_loss(generated.root, truth.root)
    loss = loss + 0.5 * reconstruction_loss(generated.pos, truth.pos)
    # LaFAN1 only, weight 0.1  [§3.7.3]
    loss = loss + 0.1 * reconstruction_loss(generated.contacts, truth.contacts)
    # Least-squares adversarial term, weight 0.1  [§3.6.2] [§3.7.3]
    # Combination of the two critics is not stated  [§3.6.2]
    adversarial_terms = [
        generator_adversarial(critic, generated) for critic in critics
    ]
    loss = loss + 0.1 * combine(adversarial_terms)
    return loss

# Least-squares critic objective  [§3.6.2]
def critic_loss(past, real, future, generated, critic):
    # Real transitions should score 1  [§3.6.2]
    real_score = critic(past, real, future)
    # Generated transitions should score 0  [§3.6.2]
    fake_score = critic(past, generated, future)
    # Scores are averaged over the sliding windows in time  [§3.5]
    real_term = 0.5 * mean((real_score - 1) ** 2)
    fake_term = 0.5 * mean(fake_score ** 2)
    return real_term + fake_term

Key equations, as printed in the paper. The paper's equations are not numbered in the LaTeX source, so they are cited by section.

z_{tta,2i} = \sin\left(\frac{tta}{basis^{2i/d}}\right), \quad z_{tta,2i+1} = \cos\left(\frac{tta}{basis^{2i/d}}\right) §3.3

\lambda_{target} = \begin{cases} 1 & \text{if } tta \geq 30 \\ \frac{tta - 5}{25} & \text{if } 5 \leq tta < 30 \\ 0 & \text{if } tta < 5 \end{cases} §3.4

L_{quat} = \frac{1}{T}\sum_{t=0}^{T-1}\|\hat{q}_t - q_t\|_1, \quad L_{root} = \frac{1}{T}\sum_{t=0}^{T-1}\|\hat{r}_t - r_t\|_1 §3.6.1

L_{pos} = \frac{1}{T}\sum_{t=0}^{T-1}\|\hat{p}_t - p_t\|_1, \quad L_{contacts} = \frac{1}{T}\sum_{t=0}^{T-1}\|\hat{c}_t - c_t\|_1 §3.6.1

L_{gen} = \tfrac{1}{2}\,\mathbb{E}\left[(D(X_p, G(X_p, X_f), X_f) - 1)^2\right] §3.6.2

L_{disc} = \tfrac{1}{2}\,\mathbb{E}\left[(D(X_p, X_{trans}, X_f) - 1)^2\right] + \tfrac{1}{2}\,\mathbb{E}\left[D(X_p, G(X_p, X_f), X_f)^2\right] §3.6.2

Evaluation metrics on global quantities, with \mathcal{D} the test set and T the transition length:

L2Q = \frac{1}{|\mathcal{D}|}\frac{1}{T}\sum_{s \in \mathcal{D}}\sum_{t=0}^{T-1}\left\|\hat{g}^s_t - g^s_t\right\|_2 §4.2

L2P = \frac{1}{|\mathcal{D}|}\frac{1}{T}\sum_{s \in \mathcal{D}}\sum_{t=0}^{T-1}\left\|\hat{p}^s_t - p^s_t\right\|_2 §4.2

The paper also reports NPSS on the global angular frequency content, following Gopalakrishnan et al. §4.2.

Hyperparameters

Results

Author-reported. Setup: Human3.6M walking subset (train on 5–50 frames, evaluate on 5–100 frames, test on subject 5) §4.2; LaFAN1 (train on subjects 1–4 with lengths up to 30 frames, test on subject 5 with lengths 5–45) §4.3; baseline is linear interpolation of the root and spherical interpolation of the quaternions §4.2 §4.3. Hardware for speed: Intel Xeon E5-1650 at 3.20 GHz tab. 4.

Motion prediction on Human3.6M (no transition conditioning). MAE per horizon in ms, NPSS over 0–1000 ms tab. 1. The paper's bold best-value marks are not reproduced.

The paper says ERD-QV is competitive on angular error and better than the cited VGRU-d on NPSS for all actions tab. 1. VGRU and TP-RNN values are taken from prior papers tab. 1. The paper's stated reason for using velocity inputs in this predictor is empirical §4.1.

Transition generation on Human3.6M walking tab. 2. Rows are cumulative: each row adds one component to the row above it.

The AVG column is the mean over the six lengths. Lengths 75 and 100 lie beyond the 50-frame training maximum tab. 2. The paper reports that the target-noise component is off for these quantitative results §4.2. TG-QV is the variant that takes quaternion velocities as input §4.2.

LaFAN1 benchmark (trained with transitions up to 30 frames; evaluated at 5, 15, 30, 45 frames) tab. 3. TG_rec is the reconstruction-only generator without the embedding modifiers or adversarial loss.

Against interpolation, the complete model is lower on every metric at every length tab. 3. TG_rec, by contrast, is worse than interpolation on L2P at 30 and 45 frames and on NPSS at 45 frames tab. 3. The paper says its contributions improve all measurements §4.3.

Speed and memory in the animation plugin tab. 4. Times in seconds. The last column is 10 transitions of 1 second each.

The plugin with the loaded network uses 170 MB of memory §4.4.

Other reported results. Keeping only dance sequences from LaFAN1 gave results similar to a larger ablated set, but the full dataset was needed for dance-style transitions §5.2. Removing the triangular-prism RNN gave no gain in the predictor §5.3.

Paper vs code

The paper's footnote links an official repository, ubisoftinc/Ubisoft-LaForge-Animation-Dataset, at commit 94084601bacdf9cc3764b5c73daaeccae6035fac. Its license is CC BY-NC-ND 4.0 license.txt. Its contents are the LaFAN1 dataset archive, the extraction code, and baseline evaluation with the zero-velocity and interpolation models README.md evaluate.py . The repository contains no transition generator, training code, configuration file, or weights. Every model detail in this card therefore comes from the paper only. No code is quoted here.

Values that the repository and paper both state, and that agree:

• Training on subjects 1 to 4 and testing on subject 5 evaluate.py §4.3.
• Normalization statistics from 50-frame windows with offset 20 extract.py §4.3.
• Test windows every 40 frames evaluate.py §4.3.
• Baseline transition lengths 5, 15, 30 and 45 frames, with 10 frames of past and 10 of future context benchmarks.py benchmarks.py §4.3.
• 22 joints for LaFAN1 benchmarks.py §3.1.

Values the paper does not state, which the repository supplies:

• Test windows are 65 frames: 10 past, 45 transition and 10 future, the largest evaluated case. Shorter cases slice from the same windows evaluate.py benchmarks.py benchmarks.py. The paper gives the 40-frame stride, but not the window length.
• Foot-contact labels use a velocity threshold of 0.02 by default utils.py extract.py. The paper does not give a threshold.

Disagreements found: none in the values above. The repository does not implement the generator, so disagreements on model architecture and losses cannot be checked.

Pitfalls

• The LSTM is not specified. The paper gives no hidden size, layer count, or initialization §3.2. The recurrent input is the concatenation of the 256-dim state embedding and the 512-dim target pair, so its width is 768 by arithmetic on the stated sizes, not as stated §3.2 §3.4.
• Velocities to state. The decoder outputs velocities, and the paper says these give the next character state, but it does not say how. Adding the velocity to the previous quaternion and renormalizing is a plausible reading, but not stated. Root integration is likewise not stated §3.2.
• Time-to-arrival index range. The paper writes i in [0, ..., d/2] §3.3. Taken literally, index d/2 writes past the end of a d-dim vector. Use pairs 0 to d/2 − 1.
• Where the time-to-arrival code goes. The introduction says it is applied to the hidden representation of all inputs §1. The method says it is added to the 256-dim latent input representations §3.2. Adding it to all three encoder outputs before concatenation fits both statements, but the text does not say so explicitly.
• Time-to-arrival cap. The cap is T_max(trans) + T_past − 5 §3.3. The paper does not give the resulting value, so the 55 and 35 frames in the table above are derived by arithmetic on the stated lengths.
• Source comment disagrees with the rendered paper. The LaTeX source has a commented-out draft that says the time-to-arrival dimension is 512 §3.3. The rendered paper says 256 §3.2. This card follows the rendered paper.
• Noise notation. The text writes N(0, I·σ) §3.4 but calls σ a standard deviation §3.7.3. Use a standard deviation of 0.5.
• Target noise is zero when evaluating. The paper turns it off for the quantitative tables §4.2. The MotionBuilder variation parameter at 0 is described as deterministic §4.4. A zero noise vector and λ = 0 give the same encoder input. Implement evaluation with zero noise.
• Reconstruction loss is a sum over components. Each L1 term is the norm over all components of a frame, averaged over frames §3.6.1. This is not the mean absolute error, which divides by the number of components. The relative weights only transfer if this norm is used.
• Loss scale. The weights are relative to losses that were scaled to be roughly equal on LaFAN1 for an untrained network §3.7.3. The absolute scales are not stated, so the weights cannot be transferred to other loss definitions.
• Combining the two critics. The paper uses both critics with the same loss §3.6.2 but does not say how their terms are combined. Whether the generator loss sums or averages them is not stated.
• Discriminator updates. The ratio of critic updates to generator updates is not stated. Whether the critics use the same optimizer settings is implied by "all of our experiments", not stated.
• Critic kernel size. The critics are 1-D convolutions with stride 1 and no padding, and the receptive field is 1 in the last two layers §3.7.2. The first-layer kernel is implied by the window lengths of 10 and 2 frames §3.5, not stated.
• Curriculum. The paper says the maximum grows over n_ep-max epochs, but not the shape of the growth. It also gives training length in iterations and the curriculum in epochs, and does not convert between them §3.7.1 §3.7.3.
• Training-time normalization for positions. The paper normalizes positions with dataset statistics for the L2P metric §4.2. It does not say whether the positional loss in training is normalized the same way §3.6.1.
• Quaternion sign. The paper does not discuss the sign ambiguity of quaternions (q and −q are the same rotation) in its losses. Whether the training data or the loss handles it is not stated.
• Contact threshold. The paper states no threshold for the contact labels §3.1. The repository's default is 0.02 utils.py. Its use is a parameter choice, not a paper detail.
• Equation numbers. The paper's equations are unnumbered in the LaTeX source, so this card cites them by section.
• Figures. This card cites the figures by their captions and text and does not use values read from figure images. Figure 1 is a teaser. Figures 3 to 6 show the time-to-arrival effect, the noise comparison, the plugin scene, and the variations fig. 1 fig. 3 fig. 4 fig. 5 fig. 6.
• Costs the abstract hides. Model inference takes 0.30 to 0.31 s per transition on the reported hardware. Keyframe application dominates the total, at 6.79 s for ten transitions tab. 4. A single 1-second transition takes 1.37 s in total tab. 4.

Lineage

• Builds on: Recurrent Transition Networks (Harvey et al., 2018), whose separate input encoders and target conditioning the generator reuses §2.3 §3.2.
• Builds on: a state-of-the-art RNN motion predictor by Chiu et al. (2019) for the predictor and its velocity inputs §2.2 §4.1.
• Builds on: positional encodings from transformer networks, here adapted to time-to-arrival instead of token position §3.3.
• Builds on: Least Squares GAN (Mao et al., 2017) for the adversarial losses §3.6.2.
• Compared against: linear and spherical interpolation, zero-velocity, and the VGRU and TP-RNN predictors on prediction benchmarks tab. 1 tab. 2 tab. 3.
• Introduces: the LaFAN1 motion-capture dataset and the transition benchmarks on Human3.6M walking and LaFAN1 §1 §4.2 §4.3.
• Superseded by: no card in this wiki.