Back to MLP: A Simple Baseline for Human Motion Prediction

In plain words

The job is to watch a person's skeleton for about two seconds and guess where the joints will be over the next second. Recent systems do this with large networks built from recurrent layers, graph layers or attention, often with millions of weights and several training stages. The authors asked whether most of that machinery is needed, and found that a very small network built from plain fully connected layers does almost as well, and better on the standard tests.

Three habits do most of the work. The input is first converted into smooth frequency components over time, which is easier to model than raw positions. The network predicts how far each joint will move from its last observed position, instead of predicting absolute positions. And it is also trained to get the speed of the motion right, not only the positions. The network itself is a stack of identical blocks that mix information across time and then normalise it, with no other nonlinearity.

The result is a model with about a tenth of a million weights that the authors report as state of the art on three benchmarks. The authors offer it as a strong baseline for the community.

TL;DR

• Predicts future 3D joint positions with a stack of temporal fully connected layers, layer normalization and transposes, applied to DCT coefficients §3 eq. 3.
• Predicts residuals from the last observed pose, and adds a velocity loss to the position loss §3.3 eq. 7.
• Human3.6M, 1000 ms MPJPE 109.4 mm with 0.14 M parameters, against 109.7 mm for ST-DGCN-10-25 with 3.80 M tab. 1.
• AMASS and 3DPW are better at 320 ms and longer, but LTD-10-10 is better at 80 and 160 ms tab. 3. The "20× to 60× fewer parameters" claim does not hold for every baseline (see Pitfalls).
• Official MIT-licensed code is available; its training and evaluation differ from the paper in several details (see Paper vs code).

Problem

Human motion prediction takes a sequence of 3D poses and forecasts the next frames. Strong recent methods use RNNs, GCNs or transformers, usually with more than 2 M parameters, several training stages, or extra priors, which makes them hard to analyse or change §1 §2. The authors ask whether a simple network can match them. They report that even a single fully connected layer gives reasonable results, and that repeating the last pose is already a reasonable baseline §1 fig. 1.

The evaluation protocol matters for comparisons. Earlier papers used 8, 256 or all test samples per action. The authors use 256 samples per action and evaluate 22 joints on Human3.6M, following the protocol of the Hisrep paper §4.1. Averaged-error protocols from other work are reported separately tab. 4.

Method

Overview. The model takes T = 50 observed poses, each a vector of C = 3 K coordinates for K joints §3. It transforms the sequence along time with a DCT, applies one spatial FC layer, runs m MLP blocks along time, applies a second spatial FC layer, transforms back with the inverse DCT, and adds the last observed pose §3.1 §3.2 §3.3 fig. 2. For Human3.6M the best depth is m = 48 blocks §4.4 tab. 5.

Step 1. DCT over time §3.1 eq. 1 eq. 2. The DCT matrix D is T × T. Its row i is the i-th frequency and its column j is the frame index. The network sees D applied to the sequence, and the output is mapped back with the inverse D, which the code computes as a matrix inverse train.py.

\mathbf{D}_{i,j} = \sqrt{\frac{2}{T}} \frac{1}{\sqrt{1+\delta_{i,0}}}\cos\left(\frac{\pi}{2T}(2j+1)i \right) eq. 1

Step 2. Spatial FC on the input §3.2 eq. 4. One fully connected layer mixes the C channels of every frame, with C × C weights and a bias. The paper writes it as an FC layer applied to the DCT output, with a transpose so the next step can act on time.

Step 3. MLP blocks along time §1 §3.2 eq. 5. Each block applies a fully connected layer of size T × T across the frames of every channel, followed by layer normalization over the channels, and adds the result to its input. The paper states that there is no other activation. The skip connection appears only in the equation, not in the prose.

\mathbf{z}^i = \mathbf{z}^{i-1} + \text{LN}(\mathbf{W}_i \mathbf{z}^{i-1} + \mathbf{b}_i) eq. 5

Step 4. Spatial FC and inverse DCT §3.2 eq. 6. A second spatial FC layer maps back to C channels, and the inverse DCT returns the sequence to pose space. The network emits T frames, not N (see Pitfalls).

Step 5. Residual displacement §3.3 tab. 9 train.py. The prediction is the IDCT output plus the last observed pose x_T, applied after the inverse transform. Predicting absolute poses instead is clearly worse tab. 9.

Step 6. Loss §3.3 eq. 7 eq. 8 eq. 9 train.py. The total loss is the position term plus the velocity term, with weight 1 on each eq. 7. The position term is the L2 norm of the error for each joint, averaged over the batch, frames and joints eq. 8. The velocity term applies the same norm to frame-to-frame differences of predicted and true poses eq. 9.

# siMLPe forward pass  [§3.1] [§3.2] [eq. 3] [eq. 4] [eq. 5] [eq. 6]
# Input:  past poses (B, T, C), T = 50, C = 66 for Human3.6M
# Output: pose predictions (B, T, C); the first N frames are used

def forward(past, dct, idct, fc_in, blocks, fc_out):
    # Frequency coefficients along time  [§3.1] [eq. 1]
    coeffs = dct @ past                       # (B, T, C)
    # Spatial FC over channels, per frame  [eq. 4]
    feats = fc_in(coeffs)                     # (B, T, C)
    # Swap axes so the MLP blocks act on time  [§3.2]
    feats = feats.transpose(1, 2)             # (B, C, T)
    for block in blocks:
        # Temporal FC across frames, then layer norm over channels  [eq. 5]
        mixed = layer_norm(block.temporal_fc(feats))
        # Residual connection around the block  [eq. 5]
        feats = feats + mixed
    feats = feats.transpose(1, 2)             # (B, T, C)
    # Spatial FC back to pose space  [eq. 6]
    coeffs = fc_out(feats)                    # (B, T, C)
    # Inverse DCT to time-domain residuals  [§3.1] [eq. 6]
    residuals = idct @ coeffs                 # (B, T, C)
    # Residual displacement from the last observed pose  [§3.3]
    return residuals + past[:, -1:, :]

# Training loss  [§3.3] [eq. 7] [eq. 8] [eq. 9]
# Input:  predicted and true future poses (B, N, J, 3), in metres, J = 22
# Output: scalar loss

def training_loss(predicted, target):
    # Mean per-joint Euclidean error over batch, frames and joints  [eq. 8]
    position = mean(norm(predicted - target, axis=-1))
    # Frame-to-frame differences inside the N predicted frames  [eq. 9]
    vel_pred = predicted[:, 1:] - predicted[:, :-1]
    vel_true = target[:, 1:] - target[:, :-1]
    # Same norm and mean for velocities  [eq. 9]
    # [code: exps/baseline_h36m/train.py#L74-L76]
    velocity = mean(norm(vel_pred - vel_true, axis=-1))
    # Equal weights  [eq. 7] [code: exps/baseline_h36m/train.py#L107]
    return position + velocity

Step 7. Long-horizon inference by auto-regression §4.2 test.py. The model is trained to predict N = 10 frames on Human3.6M. For 25 frames the code calls it three times, each time appending the new 10 frames to the input window and dropping the oldest 10, then keeps the first 25.

# Auto-regressive rollout for Human3.6M evaluation  [§4.2]
# [code: exps/baseline_h36m/test.py#L43-L64]
# Input:  observed poses (B, 50, C); step = 10; horizon = 25
# Output: predicted poses (B, 25, C)

def rollout(model, observed, step=10, horizon=25):
    window = observed
    chunks = []
    # ceil(25 / 10) = 3 model calls  [code: exps/baseline_h36m/test.py#L45-L47]
    for _ in range(ceil(horizon / step)):
        # Model predicts a full window; keep the first `step` frames
        chunk = model(window)[:, :step]
        chunks.append(chunk)
        # Slide the window forward by `step` frames
        # [code: exps/baseline_h36m/test.py#L63]
        window = concat([window[:, step:], chunk], axis=1)
    return concat(chunks, axis=1)[:, :horizon]

Key equations are reproduced above eq. 1 eq. 5. The remaining equations are described in words in Steps 4 and 6.

Hyperparameters

Values as stated in the paper; "not stated" means neither the paper nor the code gives a value. "Code" means the official repository at code_commit.

Results

Author-reported. Setup: Human3.6M, 256 test samples per action, 22 joints, input 50 frames, error in mm at 80 to 1000 ms §4.1 §4.3. AMASS is trained on AMASS and tested on AMASS-BMLrub; 3DPW is tested with the AMASS-trained model, 18 joints §4.1. Rows marked as taken from other papers are copied from those papers, as the captions state tab. 1 tab. 3.

Human3.6M, MPJPE in mm tab. 1. Lower is better.

The paper's note: ST-DGCN is shown with its single-model result, which is the better one for long-term prediction tab. 1. Repeating the last frame and One FC are the paper's own baselines §4.3.

Human3.6M, per action tab. 2. Sixteen actions at 80, 400, 560 and 1000 ms for eight rows (seven baselines and siMLPe). The average row for siMLPe equals the 80, 400, 560 and 1000 ms columns of tab. 1. The method is not best in every action at long horizons. For example, at 1000 ms in "sittingdown" it reports 142.4 mm against 139.0 mm for ST-DGCN-10-25, and at 1000 ms in "posing" 168.7 mm against 167.0 mm.

AMASS (AMASS-BMLrub) and 3DPW, MPJPE in mm tab. 3. Model trained on AMASS.

Average error over horizons tab. 4. This uses the protocol of STS-GCN and STG-GCN, which the paper says differs from the standard protocol. Values are mm, averaged from the first frame to the given horizon.

Ablation: number of MLP blocks, Human3.6M tab. 5. Columns are mm at 80 to 1000 ms.

Ablation: network components, Human3.6M tab. 6. "Spa. only" keeps all FC layers spatial (no transposes around the blocks). "Temp. only" keeps all FC layers temporal (no transposes at all).

Ablation: data augmentation, Human3.6M tab. 7.

Ablation: losses, Human3.6M tab. 8.

Ablation: residual type, Human3.6M tab. 9.

The figures show the pose-error versus parameter plot fig. 1, the network pipeline fig. 2 and qualitative predictions on Human3.6M fig. 3. The paper says predictions match ground truth closely at short horizons, and errors grow with longer horizons because of auto-regression and inherent uncertainty §4.3.

Paper vs code

Code: https://github.com/dulucas/siMLPe at commit c92c537e833443aa55554e4f7956838746550187, MIT licence (LICENSE.md). The paper links the repository in its abstract. The owner handle dulucas is taken to match author Yuming Du; this is inferred from the handle and is not confirmed by the paper.

Checked: the Human3.6M model, MLP, configuration, training, dataset and evaluation code; the AMASS configuration and training script; the AMASS evaluation script. Not read: the AMASS dataset loader, the AMASS window sampler, and the 3DPW evaluation script.

Differences and gaps, each with both citations:

• Training length. Paper: 35k iterations on Human3.6M §4.2. Code default: 40k, the loop stops at cos_lr_total_iters = 40000 config.py train.py. The repo ships checkpoints/h36m_model_35000.pth and checkpoints/h36m_model_40000.pth. Which one the table uses is not stated.
• Initial learning rate. Paper: 3e-4 from the start, dropping to 1e-5 after 30k §4.2. Code: the optimiser is built with cos_lr_max = 1e-5 config.py train.py, and the 3e-4 value is set only after the first step train.py. Only the first update uses 1e-5.
• Weight decay. Not stated in the paper. Code: Adam with weight_decay = 1e-4 config.py train.py.
• Velocity term. Paper: L2 norm of the velocity error over N frames, with v_t = x_{t+1} − x_t §3.3 eq. 9, which needs a frame beyond the window. Code: N − 1 differences inside the predicted window; the step from the last observed pose to the first predicted frame is not included train.py train.py.
• Position term. Paper: an L2 norm §3.3 eq. 8. Code: the mean of per-joint Euclidean norms, not squared and not summed train.py.
• Layer norm. Paper: LN without axis, scale or shift §3.2 eq. 5. Code: normalised over the channel axis per frame, with learned per-channel scale and shift mlp.py config.py.
• Initialisation. Not stated in the paper. Code: temporal FC weights and output FC weights use Xavier with gain 1e-8 and zero bias, so the network starts close to predicting the last observed pose mlp.py model.py.
• Output length. Paper: the network maps T frames to N frames §3.2 eq. 3. Code: the network always emits T = 50 frames, and the first 10 (training) or 25 (evaluation) are used train.py.
• Joints in the metric. Paper: 22 joints §4.1. Code: the error is averaged over all 32 joint slots. Six joints outside the 22 are set equal to a parent joint before the error is computed test.py. This is the usual convention for this protocol, but the paper does not describe it.
• Flip augmentation. Paper: "randomly inverts the motion sequence" §4.2. Code: reverses the time order of the whole 60-frame window (input and target) with probability 0.5 h36m.py.
• Validation. Paper: S11 is the validation subject §4.1. Code: S11 is absent from the training list, and no validation or model selection is run h36m_train.txt train.py.
• Command-line defaults. The paper's 48 blocks are set in run.sh, but the argparse default for --num is 64, and --with-normalization is off unless passed train.py run.sh.
• AMASS. Learning rate 3e-4 dropping at 100k, 115k iterations, N = 25, K = 18: consistent with the paper config.py train.py. The AMASS evaluation script uses 18 joints and 25 frames with auto-regressive rollout, as the paper describes test.py.

Pitfalls

• The network width is T, not N. The temporal FC layers are T × T (50 × 50) and the DCT is built for T. Output length N is applied by slicing the first N frames train.py. Build the network from T, and treat N as a loss and inference setting. The paper's eq. 3 makes this look as if N were a model dimension.
• Padding for short inputs is not implemented. The paper says the input can be padded with its last frame when T < N §3.2. None of the released configurations use T < N, and the code has no padding path.
• Layer norm details. The axis (channels), the epsilon (1e-5 in the code) and the per-channel scale and shift are missing from the paper mlp.py.
• Zero initialisation. The output FC and the temporal FCs start at gain 1e-8. A reimplementation with default initialisation will train differently, especially at the start mlp.py model.py.
• Weight decay of 1e-4 in Adam. This is L2 regularisation through Adam's weight_decay, not decoupled weight decay train.py. Not in the paper.
• Learning rate at the start. The first step uses 1e-5, not 3e-4 config.py train.py.
• Iteration count. The paper's 35k and the code's 40k differ, and which checkpoint the table uses is not stated §4.2 config.py.
• Velocity loss on the window only. The paper's definition needs a frame past the N-frame window. The code uses N − 1 differences inside the window, so the first step is not penalised train.py.
• Training units. Training uses metres, and reported errors are in mm h36m.py test.py. Keep the same units for both loss and evaluation when comparing numbers.
• Auto-regression for long horizons. Evaluation calls the N = 10 model three times, feeding its own predictions back test.py. Errors accumulate, as the paper notes §4.3. Results at 1000 ms are therefore not a single-shot prediction.
• The 256-sample test set is fixed. For each action, 128 windows are drawn from each of the two sequence files, with a fixed seed of 1234567890 misc.py. Compare only with the same windows.
• The metric averages over 32 joint slots. Six slots are copies of parent joints test.py. Reported numbers are not per-joint numbers for the 22 modelled joints.
• Parameter claim. The introduction says "20× to 60×" fewer parameters §1. Dividing the Table 1 parameter counts by 0.14 M gives about 18× for LTD-10-10 and LTD-50-25, about 23× for Hisrep, about 27× for ST-DGCN and about 111× for convSeq2Seq (these ratios are derived, not stated in the paper).
• Short horizons on AMASS and 3DPW. siMLPe is not best at 80 and 160 ms on AMASS and 3DPW. LTD-10-10 is better there tab. 3. The paper's text says it is better on long-term prediction §4.3, which is true for 320 ms and longer.
• Per-action results are not uniformly best. At long horizons several actions favour ST-DGCN-10-25 or MSR-GCN tab. 2. The average row carries the claim tab. 1.
• The "48 blocks is best" statement. The 48-block model is best or tied at every horizon tab. 5. The 64-block model is within 0.3 mm and uses 30% more parameters.
• Cost figures are the authors' own. Training time, memory and the single-GPU setup §4.2 are not verified here.
• Notation. Eq. 1 uses δ_{i,0} while the text defines a general Kronecker delta eq. 2. Read it as 1 when i = 0 and 0 otherwise.
• Table and figure numbers here follow the order of the LaTeX source. The compiled PDF was not read, so check float placement before citing a number elsewhere tab. 1 to tab. 9 fig. 1 to fig. 3.
• Concurrent work. The paper says MotionMixer, which also uses MLP blocks, has squeeze-and-excitation and activations that siMLPe does not §2. The two should not be confused.

Lineage

• Builds on: DCT over time, used by Mao et al. for motion prediction §3.1; residual connections and residual prediction from Martinez et al. §2.1 tab. 9; layer normalization §3.2. The paper's own change is to keep only FC layers, layer norm and transposes, with no other activation §1.
• Compared against: Repeating last frame and One FC (the paper's own baselines) tab. 1; Res-RNN, convSeq2Seq, LTD, Hisrep, MSR-GCN and ST-DGCN tab. 1; STS-GCN and STG-GCN on the averaged protocol tab. 4; convSeq2Seq, LTD and Hisrep on AMASS and 3DPW tab. 3.
• Concurrent: MotionMixer, which uses MLP blocks with additional squeeze-and-excitation and activations §2.
• Superseded by: not stated in this wiki.