A Neural Temporal Model for Human Motion Prediction

In plain words

Imagine teaching a computer to continue a recording of a person walking, eating or talking, using motion-capture data (sensors that record the angles of each joint). A common approach is a recurrent network that reads the pose one frame at a time and predicts the next. That works for a moment. But if its own guesses are fed back in, small errors add up and the motion drifts into something no person would make.

This paper proposes a network with two parts. One part runs backwards from a random starting point and sketches a rough plan for the whole future. The other runs forwards, reads the real recent poses, and uses the plan as a hint for each next pose. Training mixes two lessons: first predict the next frame from real data, then gradually also learn to predict well when fed its own output.

The paper also argues that the usual error score, which compares poses frame by frame, punishes motion that looks right but is a little faster, slower or shifted in time. It proposes a score that compares how much of the motion happens at each rhythm instead, and it checks that score against a small user study.

The numbers quoted below are the authors' own results, with their setup.

TL;DR

• Predicts future 3D human motion on Human3.6M. The main target is long-term synthesis; short-term prediction is treated as a secondary goal §1.
• Core model (VTLN-RNN): a backward plan GRU and a forward body GRU §3.2, trained with a loss that mixes open-loop and closed-loop error, where the closed-loop weight λ ramps up from zero §3.4.
• Proposes NPSS, a power-spectrum metric. In the authors' user study it agreed with human judgements more often than MSE did (author-reported) §4.2.
• Implementers: the released code has no finite-difference derivative features and differs from the paper in the loss, the plan noise scale, the GRU gating and the NPSS formula. See Paper vs code before implementing.

Problem

Recurrent models are trained with ground-truth inputs at every step (open loop). At test time they must feed back their own outputs (closed loop). The errors then accumulate and the synthesized motion drifts away from plausible trajectories §1 §3.4. Earlier fixes were noise injection during training (used by ERD, LSTM-3LR and SRNN) §2, a sampling loss, an adversarial approach called Professor Forcing that the authors judge expensive and hard to tune, and auto-conditioning §3.4.

Evaluation is also hard. Mean squared error (MSE) over joint angles penalises a motion that is only slower (a frequency shift) or only shifted by a few frames (a phase shift), even when it looks right. Prior work already noted that MSE is not appropriate for long-term synthesis §3.5.

Method

Data. Human3.6M, with six subjects for training and subject 5 for test. Test is 8 sequences per action for four actions, 32 in total. The remaining subject-5 sequences are described as a validation subset for tuning §3.1. Poses are exponential maps of joints. Errors are reported as Euclidean distance in Euler-angle space §3.1.

Architecture (VTLN-RNN). A top-level plan RNN runs backwards in time over the whole horizon. Its initial state is sampled from a learned diagonal Gaussian and conditioned on the one-hot action label. It emits one guide vector per future step. A lower body RNN runs forwards. At each step it takes the joint-angle vector and the guide vector and predicts the next pose. Both RNNs are GRUs §3.2 fig. 1. The initial plan state is z = μ + Σ ⊗ ε with ε drawn from N(0, I), where μ and Σ are learned parameters §3.2. The authors chose the backward direction for gradient flow: the top-level parameters then depend more on the future. They describe the plan as "hints" that the body RNN refines §3.2.

The body GRU is the standard GRU with the guide vector p_t added to the gate and candidate inputs. The paper writes the update as follows; the candidate activation Φ is not named §3.2.

# Body-GRU step with a guide vector p_t  [§3.2]
# Input:  x (B, 54) joint vector, h_prev (B, H) state, p (B, H) guide
# Output: h (B, H) new state

def body_gru_step(x, h_prev, p):
    # Reset gate reads the input, the state and the guide  [eq. 1]
    reset = sigmoid(W_r @ x + U_r @ h_prev + V_r @ p)
    # Update gate reads the same three signals  [eq. 2]
    update = sigmoid(W_z @ x + U_z @ h_prev + V_z @ p)
    # Candidate uses the reset-scaled state  [eq. 3]
    # Φ is not named in the paper
    candidate = phi(W @ x + U @ (reset * h_prev) + V @ p)
    # Blend the old state with the candidate  [eq. 4]
    return update * h_prev + (1 - update) * candidate

Derivative features. Finite-difference derivatives of the joint angles, orders n = 1, 2, 3 with spacing h = 1, are appended to the joint-angle vector. They are cheap and need no parameters. In closed-loop generation they are recomputed at each step §3.3. The paper gives the formula as

\nabla_{h}^{n} [f] (x) = \sum_{i=0}^{n} (-1)^{n} { n \choose i } f(x - ih) eq. 5

Multi-objective loss. The open-loop term averages the error over the future frames with the ground truth as input. The closed-loop term averages the error over the same frames when the model's own output is fed back. The two passes are run for every sequence, which doubles the forward cost of training §3.4.

L(\widehat{y}, y) = \frac{1}{T} \sum_{t=0}^{T} (\widehat{y_{o}}^{t} - y^t)^2 + \frac{\lambda}{T'} \sum_{t_1 = 0}^{T'} (\widehat{y_{c}}^{t_1} - y^{t_1})^2 eq. 6

λ starts at zero and increases in steps over training, so the model first learns one-step prediction and then learns to predict in closed loop §3.4. The paper does not give the step values.

# Training step with the multi-objective loss  [§3.2] [§3.4] [eq. 6]
# Input:  seed (B, 50, 54) observed frames, future (B, 100, 54) targets
#         action (B, 150, A) one-hot label, repeated over all steps
#         noise (B, H) draws from N(0, I)
# Output: scalar loss

def training_step(seed, future, action, noise, weight):
    # Initial plan state from the learned Gaussian  [§3.2]
    plan_init = mu + sigma * noise
    # Backward plan RNN over all 150 steps, then reverse in time  [§3.2]
    plan = reverse_in_time(plan_rnn(action, plan_init))
    seed_plan, future_plan = split(plan, [50, 100])
    # Body RNN reads the observed frames with ground truth  [§3.2]
    seed_pred, state = body_rnn(seed, seed_plan)
    # Open loop: ground-truth frames are fed at every future step  [eq. 6]
    future_open = body_rnn_open(future, future_plan, state)
    # Closed loop: the model's own output is fed back  [§3.4] [eq. 6]
    future_closed = body_rnn_closed(seed_pred[-1], state, future_plan)
    open_error = mean_squared(future_open, future)
    closed_error = mean_squared(future_closed, future)
    # The closed-loop weight ramps up from 0 over training  [§3.4] [eq. 6]
    return open_error + weight * closed_error

At test time the model runs in closed loop, with no weight term §1 §3.4.

NPSS. For each action class, the paper compares the power spectrum of each true joint trajectory with that of the predicted trajectory. It normalises each spectrum, takes an L1 distance between the normalised spectra, and averages these distances weighted by power §3.5 eq. 7 eq. 8 eq. 9. Lower is better. The paper defines it as follows:

X^{\mathrm{norm}}_{ij}[f] = \frac{X_{ij}[f]}{\sum_f X_{ij}[f]}, \quad Y^{\mathrm{norm}}_{ij}[f] = \frac{Y_{ij}[f]}{\sum_f Y_{ij}[f]} eq. 7

\mathrm{emd}_{ij} = \lVert X^{\mathrm{norm}}_{ij} - Y^{\mathrm{norm}}_{ij} \rVert_1 eq. 8

\mathrm{NPSS} = \frac{\sum_i \sum_j p_{ij}\,\mathrm{emd}_{ij}}{\sum_i \sum_j p_{ij}}, \quad p_{ij} = \sum_f X^{\mathrm{norm}}_{ij}[f] eq. 9

# NPSS for one action class, as printed in the paper  [§3.5] [eq. 7-9]
# Input:  truth, pred: lists of k sequences, each (T, D) joint angles
# Output: scalar NPSS (lower is better)

def npss(truth, pred):
    numerator = 0
    denominator = 0
    for each sequence i and joint dimension j:
        # Squared magnitude of the DFT of each trajectory  [§3.5]
        truth_power = abs(dft(truth[i][:, j])) ** 2
        pred_power = abs(dft(pred[i][:, j])) ** 2
        # Normalise each spectrum to sum to one  [eq. 7]
        truth_norm = truth_power / sum(truth_power)
        pred_norm = pred_power / sum(pred_power)
        # L1 distance between the normalised spectra  [eq. 8]
        emd = sum(abs(truth_norm - pred_norm))
        # Weight p as printed: the sum of the normalised spectrum  [eq. 9]
        weight = sum(truth_norm)
        numerator = numerator + weight * emd
        denominator = denominator + weight
    return numerator / denominator

The paper's user study compares six models pairwise on videos of Human3.6M test sequences, with the majority vote as ground truth, and checks which score (MSE or NPSS) agrees with the human choice §4.2 tab. 3.

Hyperparameters

Values are as stated in the paper unless marked "code". "Not stated" means neither the paper nor the code gives the value.

Results

Author-reported. Setup: Human3.6M test subject 5, four actions (walking, eating, smoking, discussion) §3.1 §5.

Long-term NPSS (lower is better) tab. 4. Table 4 lists the test-set NPSS scores. The paper does not state the exact time window for this table in the text.

NPSS at three time scales, long-term row tab. 5. Long-term window is 2 to 4 s.

Short-term MSE in Euler angles at 400 ms tab. 6. The table covers multi-action training; the authors describe the short-term model as competitive with the MBR baseline and the zero-velocity baseline.

User study agreement tab. 3 §4.2. The study has 80 questions (20 per action), each comparing two models on videos.

A one-sided binomial test of proportions reports p = 1.7 × 10⁻⁵ for NPSS agreeing better than MSE §4.2.

Parameter counts tab. 1. ERD 14,842,054; LSTM-3LR 20,282,054; SRNN 18,368,534; MBR-long 3,425,334; GRU-d 2,735,670; VGRU-d 3,413,047. The authors state their model has significantly fewer parameters than ERD, LSTM-3LR and SRNN §3.2. Against MBR-long, VGRU-d is only about 0.4% smaller (3,413,047 against 3,425,334) tab. 1.

Long-term MSE tab. 7. The short-term model has the lowest MSE in most cells for walking and eating, but the authors argue MSE is not the right measure for long-term motion and point to the user study §5.

Paper vs code

Repository cr7anand/neural_temporal_models, commit 0b72be14e1fba0ed681322994e09116496ff19e8, MIT license, linked from the abstract. The repo has no config files and a one-line README, so its flag defaults are the only record of training settings README.md.

• Derivative features are not in the code. The paper appends finite differences of orders 1 to 3 §3.3 eq. 5. No derivative or difference computation exists in the released Python files, and the model input is the normalised joint vector alone motion_rnn_lm_v2.py motion_rnn_lm_v2.py. The directory name simple_gru_no_plan_no_deriv in translate_simple_lm.py suggests a no-derivatives variant was trained translate_simple_lm.py.
• Loss includes past frames. The paper's open-loop term averages over the future frames only §3.4 eq. 6. The code averages the squared error over the observed frames and the future frames together [code: motion_rnn_lm_v2.py#L266-L268] motion_rnn_lm_v2.py.
• Plan noise scale. The paper writes μ + Σ ⊗ ε with Σ as a diagonal covariance §3.2. The code uses the square root of softplus(σ) as the scale motion_rnn_lm_v2.py.
• GRU gating. The paper's candidate uses the guide vector directly §3.2 eq. 3. The code adds a third sigmoid gate on the guide vector and feeds the gated vector into the candidate rnn_cell_implement.py rnn_cell_implement.py.
• Clipping. The paper clips at norm 5 for the short-term model §4.1. The long-term script's default is 1 translate_lm_v2.py.
• Training horizon. The paper trains the long-term model to predict 100 frames §4.1. The training default is 25 output frames; 100 is used only in the sampling path translate_lm_v2.py translate_lm_v2.py.
• Short-term schedule. The paper's short-term schedule is ×0.8 every 5000 iterations up to 60,000 §4.1. The long-term script's decay is ×0.6 every 2000 iterations, and its 60,000-iteration cap is commented out [code: translate_lm_v2.py#L201]. The short-term model's script is not identified, and its defaults (×0.6 every 1500 iterations) differ [code: translate_simple_lm.py#L28-L29].
• NPSS formula. The paper's distance is an L1 norm of normalised spectra §3.5 eq. 8. The code takes the cumulative sums over frequency first and uses the L1 norm of the CDF difference, which is the 1-D earth mover's distance forward_kinematics_v2.py. The code weights each joint dimension by its total unnormalised power [code: forward_kinematics_v2.py#L206] forward_kinematics_v2.py, where the paper's p = Σ X^norm equals 1 by construction eq. 9.
• NPSS input. The code uses 100-frame windows of Euler angles and zeroes the first six dimensions (the root) before the transform [code: forward_kinematics_v2.py#L160] forward_kinematics_v2.py. The paper does not state these details.
• NPSS aggregation. The code computes one value per pair of sequences. The calls that would aggregate over an action class are commented out [code: forward_kinematics_v2.py#L274-L278], so the aggregation used for Table 4 is not in the repo.
• Test subject. The loader reads subject 5 as the test set [code: translate_lm_v2.py#L726]. No separate validation split appears in that script, although the paper describes one §3.1.
• Evaluation. Test-time closed-loop prediction is run with dropout 0 [code: translate_lm_v2.py#L210-L211].

Pitfalls

• Eq. 5 is not a finite difference as printed. Its binomial terms have no (−1)^i sign, so for n = 1 it gives −(f(x) + f(x−h)), which is a sum, not a difference §3.3 eq. 5. The standard backward difference puts (−1)^i inside the sum. The code computes no derivatives, so there is no reference.
• Paper text and its own table disagree. Section 5 says GRU-d and VGRU-d outperform VGRU-ac on all four actions §5. In Table 4, VGRU-ac has the lowest NPSS for discussion (1.651, against 1.688 and 1.777) tab. 4. The 2 to 4 s window shows GRU-d ahead for discussion (0.785 against 0.811) tab. 5. The claim holds for that window only.
• Unstated values. Φ, the activation of the candidate state, is not named §3.2. The λ step values are not stated §3.4. The clipping value for long-term training is not stated §4.1. The text does not say whether T in eq. 6 counts seed frames. How training windows are drawn is not stated; the code draws a start index uniformly from frame 16 to the end minus the window length motion_rnn_lm_v2.py.
• Cost. Each training batch needs two forward passes, open and closed loop §3.4. The paper does not report wall-clock or memory numbers.
• Stochastic results. The plan's initial state is sampled, so results are averaged over trials tab. 6 tab. 7. A single sample is not a reliable comparison.
• User study statistics. The binomial test treats the 80 questions as independent samples, and the ground truth is a majority vote tab. 3. The paper does not discuss either point. Treat p as author-reported.
• Supplementary material is not in the source. The user-study screenshot and the ablation details are in the supplementary material §4.2 §5. The ablation has no numbers in the main text, so its ranking of components is unverified here.
• Units. The paper does not state units for the Euler-angle errors §3.1.
• Figures. The only qualitative evidence is a snapshot figure of walking sequences fig. 2, with no quantitative content.

Lineage

• Builds on: the GRU cell §3.2; the re-parameterisation trick §3.2; a hierarchical dialogue model whose upper level runs forwards, changed here to run backwards §3.2; auto-conditioning, the VGRU-ac baseline §3.4 tab. 4.
• Compared against: ERD and LSTM-3LR §2 tab. 7; Structural-RNN (SRNN) §2 tab. 7; the MBR baselines, a 1-layer sequence-to-sequence GRU with a sampling loss §2 tab. 4; the zero-velocity baseline §5 tab. 6.
• Related methods discussed but not compared numerically: Professor Forcing and noise scheduling §3.4.
• Superseded by: not known from the content of this wiki.