Motion prediction

Overview

Given the recent past of one body or of several agents, predict where each one goes next. For a single skeleton the input is joint angles per frame; for several agents it is positions and, often, velocities 2303.10876 §3.1; for a face it is expression coefficients from a 3D face model 2308.10897 §3.1. The hard part is the long horizon. A model trained only to predict the next frame from real data drifts once it must feed on its own outputs, because errors add up 1809.03036 §1 [arxiv:1809.03036 §3.4]. The traffic paper meets the same compounding error by unrolling its model through a differentiable simulator during training and comparing the rollout with logged data 2101.06557 §2 §3.4. A second family changes the training target instead of the feeding scheme: the model outputs a block of upcoming frames at once, which the dance paper reports as avoiding both freezing and drift 2101.08779 §4.1. For several interacting agents, a further problem appears: a predictor that does not respect rotation and translation must learn the same pattern again for every orientation of the scene 2303.10876 §1.

The listener paper is the exception to "predict from past motion". It generates a listener's face from a speaker's words, with no audio or video 2308.10897 §6.4. It is kept here because its card lists this concept. Its metrics are not comparable with the forecasting metrics below.

The social-navigation paper uses forecasts rather than producing a new forecasting model for its results. A Social GAN predicts each tracked person's next 12 positions from their last 8 2411.01814 §2.1. A local planner then adds those predicted positions to its cost 2411.01814 §2.2. Its reported numbers measure navigation in a simulator, not forecast error.

Variants

Recurrent plan-and-body model (VTLN-RNN)

Two GRUs. A backward-running plan RNN, started from a sample of a learned Gaussian and conditioned on the action label, emits one guide vector per future step. A forward body RNN reads the joint vector and the guide vector at each step and predicts the next pose 1809.03036 §3.2. Training mixes an open-loop term, with ground-truth inputs, and a closed-loop term, with the model's own output fed back. The closed-loop weight starts at zero and rises in steps 1809.03036 §3.4 eq. 6. The paper's motivation for the backward plan is gradient flow: the top-level parameters then depend on the future 1809.03036 §3.2. Card: A Neural Temporal Model for Human Motion Prediction.

Reported results are on Human3.6M, test subject 5, four actions, using the authors' NPSS metric. In the 2 to 4 s window the single-layer model has the lowest NPSS for walking, eating and smoking, and the two-layer model has the lowest for discussion 1809.03036 tab. 5. Parameter count is about 3.4 M for the single-layer model, against 14.8 M for ERD [arxiv:1809.03036 tab. 1]. Each training batch needs two forward passes 1809.03036 §3.4. The derivative features the paper adds are not in the released code; see the card before implementing 1809.03036 §3.3.

Transformer with future-N supervision (FACT)

Motion tokens and music tokens each pass through a small transformer. Their outputs are concatenated and read by a 12-layer cross-modal transformer with no attention mask, so every token attends to every other [arxiv:2101.08779 §4.1] §5.2.1. The model reads a 120-frame seed (2 s at 60 FPS) and is trained to output the next N = 20 frames under a mean squared error 2101.08779 §4.1 §5.2.1. Generation is auto-regressive, one frame per step, so the model runs once per generated frame 2101.08779 §5.2.1. The authors report that a causal transformer trained to predict only the next frame freezes after several seconds, and that a full-attention model trained on one future frame drifts 2101.08779 §4.1. Their ablation has FID_k 111.69 for causal attention with shift-by-one supervision, 207.74 for full attention with one future frame, and 35.10 and 35.35 for full attention with 10 and 20 future frames 2101.08779 tab. 3. The paper chose N = 20 on FID_m and BeatAlign, since future-10 has the lower FID_k 2101.08779 tab. 3. Music and motion are fused early, which the paper reports as better on FID_k than late fusion (35.35 against 45.76) 2101.08779 tab. 4. The task is dance, with music as a second input, and the reported results are on AIST++ only 2101.08779 tab. 2. Card: AI Choreographer: Music Conditioned 3D Dance Generation with AIST++.

Equivariant interaction model (EqMotion)

Each agent carries two features. A geometric feature is a set of coordinates in 3D that rotates and shifts with the scene. A pattern feature is a vector built from invariant quantities, namely speed and turning angle 2303.10876 §4.1. A reasoning step predicts, for each ordered pair of agents, a categorical vector from invariant inputs only [arxiv:2303.10876 §4.2]. Each layer moves the coordinates by a category-weighted sum of differences to the other agents and then applies an equivariant non-linearity 2303.10876 §4.3. The paper proves that the layers as written are equivariant, with the pattern features and categories invariant 2303.10876 §4.6. Card: EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning.

Reported results cover particles, molecules, the H3.6M skeleton and ETH-UCY pedestrians. On particle interaction recognition, equivariance gives 100% consistency over 20 random Euclidean transforms, against 93.7% for NRI 2303.10876 tab. 1 §5.1. On the H3.6M skeleton, average MPJPE is 55.0 mm at 400 ms and 106.9 mm at 1000 ms 2303.10876 tab. 2 tab. 3. On ETH-UCY, the deterministic average is 0.49 m ADE and 1.03 m FDE 2303.10876 tab. 5. The paper also reports a best-of-20 setting for pedestrians; that row is an oracle selection and must not be compared with single predictions 2303.10876 §5.4. All of these are author-reported. The per-layer cost is quadratic in the number of agents, because the graph is fully connected in the experiments 2303.10876 §4.3. The paper mentions local neighbourhoods for large point sets but does not test them 2303.10876 §4.3.

Discrete-token language model (listener gestures)

A VQ-VAE maps windows of listener face motion to one of 256 codes, one per 8 frames 2308.10897 §3.2 app. B. GPT-2 Medium, started from text pretraining, then predicts those codes interleaved with the speaker's timed words 2308.10897 §3.3. The only external input is the speaker's transcript; the paper uses no audio or video 2308.10897 §6.4. Inference is greedy, not sampled 2308.10897 §3.3. During training, motion tokens in the input are replaced at random with probability 0.5 2308.10897 app. B. Card: Can Language Models Learn to Listen?.

On the Trevor Noah listener, the full model has the lowest FD (18.22), P-FD (19.63) and L2 Affect (6.36) of the table's methods. The unconditional model has lower L2 (0.39 against 0.43), but is worse on the other metrics 2308.10897 tab. 1 §5.1. Removing GPT-2 pretraining raises FD to 22.81. The variant without interleaving has FD 19.03, close to the full model's 18.22, and the paper calls that gap not significant 2308.10897 tab. 2 §6.1. In a human A/B test, listeners preferred this model over nearest-neighbour retrieval 70.1% of the time and over the unconditional model 92.8% 2308.10897 §5.2. The preference over the original L2L baseline was 55.7%, a small margin 2308.10897 §5.2.

Joint latent model of traffic (TrafficSim)

One latent variable is drawn for the whole scene. A decoder turns that sample, together with each actor's context, into a plan for every actor at once 2101.06557 §3.2 eq. 1. Plans drawn from one sample therefore fit together, and repeated samples from the same start give different scenes 2101.06557 §3.2. The map is encoded once per map. Each actor contributes three seconds of boxes, and a 10-step plan is predicted at each tick, of which only the first κ steps are applied 2101.06557 §3.3 app. A tab. 2. The paper compares simulation rates of 2, 1 and 0.5 Hz 2101.06557 tab. 4. Training unrolls the policy for 12 s in closed loop. The first 8 s also carry an imitation loss against logged driving, and prior samples carry a collision penalty 2101.06557 §3.4. Card: TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors.

Reported results are on ATG4D, with 15 samples per scenario. The collision rate is 0.50%, against 1.19% for IDM and 2.90% for the ILVM forecaster 2101.06557 tab. 1. Its minimum average displacement error is 0.57 m, against 1.33 m for ILVM tab. 1. Its traffic rule violation rate is 2.77%, against 0.25% for IDM tab. 1. In the ablation, removing the closed-loop training raises the collision rate from 0.50% to 5.92% tab. 2. A planner trained on 15k of its simulated scenes, compared with the same number of real examples, scores close to real data on collision rate (10.73% against 10.56%) and planning L2 (4.52 m against 4.85 m) tab. 3. The paper also includes an interactive tool that edits actors and traffic lights, shown only in a video with no numbers 2101.06557 §4.2. The optimiser, learning rate, batch size and epoch count are not stated, and the paper does not give the Huber δ, the KL weight β or the latent size 2101.06557 §3.4 app. A. No official code was found; see the card before reimplementing.

Forecasts as planner costs (MP-TEB)

A Social GAN forecaster, with a pooling module that models interactions between people, is run on tracked people. It reads the last 8 positions, sampled every 0.5 s, and outputs the next 12 positions, covering 6 s 2411.01814 §2.1. The positions come from RGB-D detections, turned into planar positions with the box centre, depth and camera intrinsics 2411.01814 eq. 1. The predicted paths enter a Timed Elastic Band local planner as three extra cost terms: a term that follows a predicted human path eq. 5, a term that tracks a predicted dynamic-obstacle position eq. 6, and a priority-side term that is subtracted from the cost eq. 7. They are added to the base TEB cost with a normalisation factor δ_mp eq. 8 2411.01814 §2.2. Card: the social-navigation planner paper.

Reported results are from Gazebo, four scenarios, 10 runs each averaged, against DWA and TEB 2411.01814 §3.1 §3.2. MP-TEB has the shortest total time in three of four scenarios and the largest minimum human-robot distance in three of four 2411.01814 tab. 1. The authors do not report variance §3.2 tab. 1. The planner itself is not in the released repository, so the cost has to be built from the equations the card's Paper vs code section gives the details.

When to use which

Implementation sketch

Five sketches, one for each of the recurrent, transformer, equivariant, listener and planner variants, plus one tick of the traffic model. They are not one shared implementation; the recurrent objective, the transformer's block loss, the equivariant layer, the token generation loop and the traffic tick are independent.

# Training objective of the recurrent variant  [arxiv:1809.03036 §3.4] [eq. 6]
# Input:  seed frames, future (T_f, D) ground truth, weight lam in [0, 1]
# Output: scalar loss

def recurrent_loss(seed, future, lam):
    # Open loop: ground-truth frames are fed in at every step  [§3.4]
    open_pred = rollout(seed, future, feed="truth")
    # Closed loop: the model's own output is fed back  [§3.4]
    closed_pred = rollout(seed, future, feed="own")
    # lam starts at zero and is raised in steps over training  [§3.4]
    return mse(open_pred, future) + lam * mse(closed_pred, future)

# Block loss of the transformer variant  [arxiv:2101.08779 §4.1]
# Input:  seed (120, D) observed motion; music (240, A) aligned with it
#         target (20, D) the next 20 frames
# Output: scalar loss

def future_n_loss(model, seed, music, target):
    # One forward pass emits a block of frames, not one  [§4.1]
    predicted = model(seed, music)              # (T_out, D)
    # Compare the first N = 20 predicted positions with the target  [§4.1]
    difference = target - predicted[: length(target)]
    return mean(square(difference))

# One update of the equivariant variant  [arxiv:2303.10876 §4.2] [§4.3]
# Input:  pos (N, C, 3) geometric features, h (N, D) pattern features
#         cat (N, N, K) interaction categories, from invariant inputs
# Output: pos (N, C, 3), h (N, D)

def equivariant_layer(pos, h, cat):
    # Pairwise distances are invariant, so they can weight messages  [§4.2]
    dist = pairwise_norm(pos)
    # Each agent moves by a category-weighted sum of its differences  [§4.3]
    pos = pos + aggregate(pos, h, dist, cat)
    # Pattern features update from invariant inputs only  [§4.4]
    h = message_pass(h, dist)
    return pos, h

# Greedy generation of listener codes  [arxiv:2308.10897 §3.3]
# Input:  speaker words with end frames; n_slots motion slots of 8 frames
# Output: one motion code (0 .. 255) per slot, decoded by the VQ-VAE

def generate_listener(words, word_ends, n_slots, lm):
    sequence = []
    next_word = 0
    for slot in range(n_slots):
        # Speaker words that end before this slot go in first  [§3.3]
        while next_word < len(words) and word_ends[next_word] < 8 * slot:
            sequence.append(words[next_word])
            next_word += 1
        # Greedy: the most likely code, no sampling  [§3.3]
        logits = lm(sequence)[-1]               # (256,) over motion codes
        sequence.append(motion_token(argmax(logits)))
    return sequence

# Social terms added to a TEB band cost  [arxiv:2411.01814 eq. 5-8]
# Input:  band (N, 2) planned robot positions; human_paths (P, 12, 2)
#         forecast per person; obstacles: list of (position, velocity)
# Output: scalar social term, added to the base TEB cost

def social_cost(band, human_paths, obstacles, w1, w2, delta_mp):
    total = 0.0
    for t in range(len(band)):
        # Human-path term, one person at a time  [eq. 5] [§2.2]
        # How several people combine is not stated; this sums them
        for path in human_paths:
            total += w1 * distance(band[t], path[t])
        # Obstacle term, moved by its velocity over time  [eq. 6]
        # Time unit of t is not stated; the sign of w1 and w2 is not stated
        for position, velocity in obstacles:
            total += w2 * distance(band[t], position + velocity * t)
        # Priority-side term, subtracted  [eq. 7] [eq. 8]
        total -= priority_side(band[t], obstacles)
    # Eq. 8 has no sum over the band; summing it here is an assumption
    return delta_mp * total

# One tick of the joint latent traffic model  [arxiv:2101.06557 §3.3] [§3.2]
# Input:  actor contexts (N, 192), cached map already encoded
# Output: the next kappa steps of a plan for each of the N actors

def joint_tick(context, kappa):
    # One latent sample for the whole scene, not one per actor  [§3.2] [eq. 1]
    z = sample_gaussian(*prior_network(context))
    # Decode a plan for all actors together, so the plans fit  [§3.2]
    plan = decoder(context, z)              # (N, T_plan, ...) per actor
    # Apply only the first kappa steps, then build the next context  [§3.3]
    return plan[:, :kappa]

Contradictions

• Within the recurrent paper. Section 5 says both GRU models beat the auto-conditioning baseline on all four actions 1809.03036 §5. In Table 4, that baseline has the lowest NPSS for discussion (1.651, against 1.688 and 1.777) 1809.03036 tab. 4. In the 2 to 4 s window, the two-layer model is ahead for discussion (0.785 against 0.811) 1809.03036 tab. 5. The claim holds for that window only.
• Between the papers, failure modes. The recurrent paper describes drift when a next-frame model feeds on its own output [arxiv:1809.03036 §3.4]. The dance paper describes freezing for a causal next-frame model and drift for a full-attention one 2101.08779 §4.1. These are different models and tasks, so the fixes are not interchangeable.
• Between the papers, metrics. The recurrent paper argues that per-frame error penalises plausible motion that is only a little faster, slower or shifted, and proposes NPSS 1809.03036 §3.5. The equivariant paper reports MPJPE and ADE/FDE 2303.10876 §5.3 §5.4. The dance paper reports FID_k, FID_m, diversity and BeatAlign 2101.08779 §5.2.2. The listener paper reports L2, FD, P-FD and L2 Affect on face coefficients 2308.10897 tab. 1. The traffic paper reports collision rate, rule violation and minimum or mean SFDE and SADE on ATG4D 2101.06557 tab. 1. The planner paper reports path length, total time and minimum human-robot distance in a simulator 2411.01814 tab. 1. No two of the six report the same metric set, so the numbers cannot be set side by side. The traffic and equivariant papers both report best-of-many displacement errors, and their definitions differ 2101.06557 §4.1 2303.10876 §5.4.
• Between the papers, protocol. The recurrent and equivariant papers both use Human3.6M with six training subjects and subject 5 for test 1809.03036 §3.1 2303.10876 app. D.1.3. The horizons, joint sets and metrics differ, so the recurrent paper's short-term MSE table is not comparable with the equivariant paper's MPJPE table 1809.03036 tab. 6 2303.10876 tab. 2. The dance paper uses its own AIST++ split 2101.08779 §5.2.1. The traffic paper uses ATG4D lidar tracks, a 12 s horizon with 8 s of labels, and 15 samples per scenario 2101.06557 §4. The listener paper uses the Learning to Listen data with its own segmentation 2308.10897 §4.1.
• Within the traffic paper, metrics. The full model M* has the lowest collision rate and traffic rule violation in its ablation, but the unrolled model with one-step plans, M2, has lower minimum SFDE and SADE (1.02 m and 0.54 m, against 1.13 m and 0.57 m) and a higher collision rate (1.28% against 0.50%) 2101.06557 tab. 2. The paper reports M*.
• Within the traffic paper, text and table. The text describes TrafficSim's rule violation as similar to IDM's 2101.06557 §4. Table 1 gives 2.77% against 0.25% tab. 1. Quote the table.
• Within the traffic paper, post-processing. Rejection sampling and gradient optimisation lower the collision rate from 0.50% to 0.33% and 0.12%, but raise rule violation from 2.77% to 3.01% and 3.00%. The text does not discuss this trade-off 2101.06557 tab. 5.
• Within the traffic paper, loss formulas. The text says the imitation weight falls to zero at 8 s, but the printed weight is zero up to 8 s and negative after it 2101.06557 app. A. The printed collision loss, max(1, ·), is at least 1 and does not depend on positions, while the description implies a value between 0 and 1 §3.4. Implement the stated intent and record the choice.
• Within the dance paper. FACT's FID_m is 12.40 in the main table and 12.39 in the ablations 2101.08779 tab. 2 tab. 3. Quote the table you use.
• Within the listener paper, history length. Section 6.3 calls 8 s of history the sweet spot. The configuration labelled "Ours" in the appendix uses 3 s, as does the README 2308.10897 §6.3 app. C. Pick one and cite the table you use.
• Within the listener paper, across listeners. On the Trevor Noah listener, the full model is best on FD. On Stephen, the Unaligned variant has the lowest FD (33.47) and P-FD 2308.10897 app. C. The headline is for one listener.
• Headline figures in the equivariant paper. The abstract gives four percentage gains without naming their metrics. The body gives 8.6% for H3.6M short-term and 3.8% for long-term 2303.10876 §5.3, 34.2% and 30.1% for molecules §5.2, and 10.4% and 7.9% for pedestrians §5.4. The body does not state the averaging baseline behind the abstract's 24.0% and 9.2% 2303.10876 §1. Treat these as unverified.
• Within the social-navigation paper, text and table. The text says MP-TEB beats DWA and TEB on both efficiency and safety in every scenario 2411.01814 §3.2. Table 1 does not show this. In the corridor scenario DWA has the shorter total time (23.12 s against 24.46 s), and in the multi-person scenario TEB has the larger minimum distance (0.46 m against 0.36 m) 2411.01814 tab. 1. Quote the table, not the claim.
• Between the planner paper and the forecasting papers, sign and scope. The planner paper's cost terms are added to the cost with a positive factor, which read literally pulls the robot toward the predicted human and obstacle positions 2411.01814 eq. 5-8. The text describes avoidance 2411.01814 §2.2. This is unresolved on the card and is not settled here. The forecasting papers do not address planners.

Papers

• A Neural Temporal Model for Human Motion Prediction: a backward plan GRU guides a forward body GRU, trained with a closed-loop weight to resist drift; proposes NPSS.
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++: a full-attention transformer that predicts 20 future frames from music and a 2-second seed; also introduces the AIST++ dataset.
• EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning: an equivariant graph-style predictor for interacting agents, with invariant interaction categories; author-reported results on four kinds of data.
• Can Language Models Learn to Listen?: a VQ-VAE turns listener face motion into 256 tokens, which a text-pretrained GPT-2 Medium predicts from the speaker's timed words; author-reported results on one listener.
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors: a latent-variable model that plans all vehicles jointly and is trained by unrolling through a differentiable simulator; author-reported results on ATG4D.
• Social navigation with forecasts inside a local planner: a Social GAN forecast of each person's next 12 positions adds three cost terms to a TEB local planner; author-reported results in Gazebo.