MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

In plain words

Predicting the next frames of a video is hard when things move in changing ways. A person's legs swing back and forth while the body keeps walking forward, and rain clouds grow, twist and spread. Older recurrent video predictors mostly track how the picture changes from one moment to the next, and they tend to lose these details, blurring limbs or shifting objects to the wrong place.

The authors split motion into two parts. One is the quick, local wobble from moment to moment. The other is the steady direction the motion has been heading in over the past frames. MotionRNN adds a small module, called MotionGRU, between the layers of an existing recurrent predictor. It estimates, for each small patch, how the content should be shifted, then moves the internal features along those shifts. A second path, the Motion Highway, carries content forward so the motion does not fade out as it passes through the stack.

The module can be bolted onto several existing recurrent predictors. The authors report better sharpness and accuracy on human-motion, radar-echo and synthetic moving-digit benchmarks.

TL;DR

MotionRNN is a plug-in for RNN video predictors (PredRNN, ConvLSTM, MIM, E3D-LSTM). Its MotionGRU unit predicts a per-pixel motion field as the sum of a transient variation and an exponentially averaged motion trend, then warps hidden states with it. A Motion Highway keeps motion from vanishing across layers. The authors report gains on Human3.6M, Shanghai radar and a new varied Moving MNIST dataset, with about 9% more parameters on PredRNN. Only the PredRNN backbone is in the released code.

Problem

Deterministic video predictors built from stacked recurrent blocks capture state transitions over time but not the changing motion inside those transitions. Optical-flow based predictors capture short-term local shifts but not long-term trends §1 §2. The authors argue that motion which keeps changing in both space and time needs both a short-term component and a long-term trend §1 fig. 1.

Method

Overview

MotionRNN keeps the original recurrent block (for example ConvLSTM or PredRNN) and inserts a MotionGRU between consecutive layers, so the state transition flow of the base model is not changed §3.1 fig. 2. Each layer receives its input through MotionGRU, which warps the previous hidden state by a learned motion filter §3.1 §3.2.3. The output of the warp is gated against the input hidden state eq. 4. A Motion Highway adds the unchanged part of the content back to the block output eq. 1.

Steps for layer l at time t:

1. Encode the hidden state with a stride-2 convolution to a quarter of the channels and half the resolution app. A.
2. Compute the transient variation F' with a ConvGRU-style update over the offset space, using the previous motion filter F_{t-1} and the encoded state eq. 2 §3.2.1.
3. Update the trending momentum D as a step toward the previous motion filter, D_t = D_{t-1} + alpha (F_{t-1} - D_{t-1}) eq. 3 §3.2.2.
4. Form the motion filter F_t = F' + D_t eq. 4 §3.2.3.
5. Form a mask over each k x k neighbourhood from the encoded state eq. 4 app. A.
6. Warp the encoded state by F_t with bilinear interpolation, multiply by the mask, and decode back to full resolution app. A §3.2.3.
7. Compute an output gate g from the decoded state and the layer input and mix the result with the previous hidden state eq. 4 app. A.
8. Pass the transformed state to the block (for example ConvLSTM or ST-LSTM) and add the Motion Highway term, (1 - o_t) times the layer's input hidden state eq. 1 §3.1.

The motion filter has 2k^2 channels per pixel, one offset for each of the k^2 neighbours in both axes, with k = 3 app. A §4 implementation.

# MotionGRU step for one layer at time t  [eq. 2-4] [app. A]
# Shapes: hidden (B, C, H, W); motion filter (B, 2*k*k, H/2, W/2)
def motion_gru_step(hidden, prev_filter, prev_trend, prev_hidden):
    # Encode the hidden state to a quarter of its channels  [app. A]
    encoded = encoder(hidden)                   # (B, C/4, H/2, W/2)
    # Transient variation: gated update over the offset space  [eq. 2]
    transient = transient_gru(encoded, prev_filter)
    # Trending momentum: moving average of past motion filters  [eq. 3]
    trend = prev_trend + alpha * (prev_filter - prev_trend)   # alpha = 0.5
    # Motion filter is the sum of the two components  [eq. 4]
    motion = transient + trend
    # Per-neighbour mask from the encoded state  [app. A]
    mask = broadcast(sigmoid(conv_mask(encoded)))
    # Bilinear warp of the encoded state by the motion filter  [app. A]
    warped = mask * warp(encoded, motion)
    # Decode back to the hidden-state resolution  [app. A]
    decoded = decoder(warped)
    # Output gate blends the decoded state with the previous hidden state  [eq. 4]
    gate = sigmoid(conv_gate(concat(decoded, hidden)))
    output = gate * prev_hidden + (1 - gate) * decoded
    return output, motion, trend

# One MotionRNN layer at time t, with a Motion Highway  [eq. 1]
def motionrnn_layer(layer_input, hidden_prev, cell_prev, highway_prev,
                    prev_filter, prev_trend):
    # Motion-based transition of the incoming hidden state  [eq. 4]
    moved, filt, trend = motion_gru_step(layer_input, prev_filter,
                                         prev_trend, hidden_prev)
    # The base predictive block (e.g. ConvLSTM, PredRNN) runs on the result  [eq. 1]
    block_out, cell, out_gate = block(moved, hidden_prev, cell_prev)
    # Motion Highway: add back the unchanged content weighted by (1 - o_t)  [eq. 1]
    hidden = block_out + (1 - out_gate) * layer_input
    return hidden, cell, filt, trend

The pseudocode above follows the equations as written. The official code differs in several places, listed under "Paper vs code".

Training objective

The paper states the training loss as L1 + L2 §4 implementation details. The loss is then applied to the predicted future frames. The loss is not written as an equation.

Backbones

MotionRNN is applied to ConvLSTM, PredRNN, MIM and E3D-LSTM. For E3D-LSTM, the encoder and decoder inside MotionGRU are replaced with 3D convolutions that downsample the 3D feature map to 2D §4 backbone models. Each backbone has four stacked blocks with 64-channel hidden states §4 backbone models.

Hyperparameters

Results

Author-reported. Setups: Human3.6M with 4 input and 4 output frames at 128 x 128 §4.1; Shanghai radar with 5 input and 10 output frames at 64 x 64 §4.2; varied Moving MNIST with 10 input and 10 output frames at 64 x 64, 100,000 training and 10,000 test sequences generated on the fly §4.3. Baselines are the backbone without MotionRNN.

Human3.6M tab. 1

Lower MSE, MAE and FVD are better; higher SSIM is better tab. 1. The text reports 29% lower MSE and 22% lower MAE for PredRNN with MotionRNN §4.1. The paper calls this state of the art on Human3.6M §4.1.

Parameters and compute on Human3.6M tab. 2

FLOPs are counted for predicting 4 frames from 4 frames tab. 2.

Ablation on Human3.6M tab. 3

The paper says the trend and transient components add about 9% and 13% on their own §4.1 ablation tab. 3. These two percentages are stated in the text and are not an exact row in the table.

Shanghai radar tab. 4

Lower GDL is better; higher CSI is better tab. 4. The paper reports a 24% GDL gain for E3D-LSTM §4.2.

Varied Moving MNIST tab. 5

Higher PSNR and SSIM are better; lower MSE and GDL are better tab. 5.

Guangzhou radar (appendix) tab. 6

Higher CSI is better tab. 6. The Guangzhou set uses 10 input and 10 output frames with frames every 6 minutes app. D.

Paper vs code

Official code: thuml/MotionRNN at commit 002415f4e7384f14fd20501b76febe8c5caaca7e, owned by the authors' lab (the README cites the paper and names Mingsheng Long). The repository has no LICENSE file and the README states no license, so code_license is recorded as not stated. The code is cited, not quoted.

The release contains only the MotionRNN-on-PredRNN model MotionRNN_PredRNN.py. There is no ConvLSTM, MIM or E3D-LSTM variant, and no data preparation for the radar datasets directory.

Differences that matter for an implementation:

• Loss. The paper uses L1 + L2 §4 implementation details. The code uses MSE only MotionRNN_PredRNN.py MotionRNN_PredRNN.py.
• Iteration count. The paper trains for 100,000 iterations §4 implementation details. The human script defaults to 80,000 MotionRNN_PredRNN_human_train.sh.
• Kernel sizes in MotionGRU. The paper states the update, reset and candidate convolutions are 1 x 1 §3.2.1. The code uses 3 x 3 kernels MotionGRU.py.
• Output gate. The paper's output is g * H_{t-1}^l + (1 - g) * Dec(H') eq. 4. The code computes o * decoded + (1 - o) * the current hidden state, so the gate weights are swapped and the hidden state is the current step, not the previous one MotionRNN_PredRNN.py MotionRNN_PredRNN.py.
• Gate convolution size. The paper does not state the gate kernel size. The code uses the filter size 5 MotionRNN_PredRNN.py.
• Motion Highway. The paper adds (1 - o_t) times the layer's input hidden state to the block output eq. 1. The code blends the cell's output with a highway state that is passed from the previous layer and reset to a projection of the input frame at each time step SpatioTemporalLSTMCell_Motion_Highway.py MotionRNN_PredRNN.py.
• Warp sign convention. The appendix writes the sampling position as m + p - F app. A. The code computes p_0 + p_n + offset MotionGRU.py. The two forms may be equivalent up to the sign of the learned offsets, but the code is not checked for equivalence.
• Scheduled sampling and time reversal. Neither is described in the paper. The script enables scheduled sampling run.py and each batch is also trained on its time reverse trainer.py.
• Two optimiser steps per iteration. With reverse_input=1, the trainer calls the model twice per iteration, and each call runs an Adam step trainer.py model_factory.py. One iteration is therefore two Adam updates.
• Evaluation data. The human script passes the same path to training and validation MotionRNN_PredRNN_human_train.sh. Any reported test number from this script is not held out.
• CSI conversion. The paper says pixel values are converted to dBZ but gives no formula §4.2. The code maps a pixel value v to dBZ = (v + 30) / 2 and thresholds that metrics.py.
• Learning-rate scaling hooks. The code registers backward hooks meant to scale gradients by 0.1 MotionGRU.py MotionGRU.py. Each hook builds generators and returns nothing, so the gradients are not modified. The paper does not mention a learning-rate scale.
• Shared memory. The code carries one spatiotemporal memory through all layers MotionRNN_PredRNN.py MotionRNN_PredRNN.py. The paper does not describe this.
• Last layer. The code has no MotionGRU after the last block; only the first L-1 layers get one MotionRNN_PredRNN.py MotionRNN_PredRNN.py. The paper says MotionGRU sits between stacked layers §3.1 and does not state the last layer.

No other paper-to-code gaps were checked. Algorithm-level details, equation forms and the motion-filter size match the paper eq. 2-4 app. A MotionGRU.py.

Pitfalls

• Momentum is a moving average with alpha = 0.5. Its effect is to make the trend weight the most recent filters more heavily eq. 3. The paper's sensitivity range is 0.5 to 0.7 fig. 7. The code does not expose alpha as a parameter.
• Gate weights are easy to get backwards. Following eq. 4 literally gives the previous hidden state the weight g. The code does the opposite, see "Paper vs code". Pick one and test it.
• Initial states are random. The paper does not state how states or offsets are initialised. The code draws recurrent states from xavier normal, so results may vary across seeds MotionRNN_PredRNN.py.
• Warped sampling is clamped. Offsets are clipped to the image bounds before bilinear sampling, so out-of-range samples pick up border values MotionGRU.py.
• Gradients reach the offsets. Integer corner indices are detached, but the bilinear weights are computed from the offsets, so gradients flow back through them MotionGRU.py MotionGRU.py.
• Cost is not zero. The paper reports 9.3% more parameters and 7.6% more FLOPs on PredRNN tab. 2. On E3D-LSTM the figures are 4.4% and 3.8% tab. 2. The FLOPs figure counts multiplications only tab. 2.
• The state-of-the-art claim is per metric. On Human3.6M, E3D-LSTM with MotionRNN has the highest SSIM (0.881), while PredRNN with MotionRNN has the best MSE, MAE and FVD tab. 1. The text claims state of the art for PredRNN with MotionRNN §4.1.
• Unstated values. The paper gives no LR schedule, gradient clipping, patch size, recurrent kernel size, betas or initialisation. The code fills some of these in (see Hyperparameters). An implementation based only on the paper will differ from the code in these details.
• Radar CSI definitions in the text are swapped. The paper says misses are false positives and false alarms are false negatives §4.2. Standard CSI uses misses as false negatives and false alarms as false positives. Use the standard definition.
• Backbone generality. The paper reports results on four backbones §4 backbone models, but only the PredRNN variant is in the released code. The other backbone numbers cannot be reproduced from the repository.

Lineage

• Builds on: PredRNN (ST-LSTM zigzag memory) §2, ConvLSTM §2, MIM and E3D-LSTM as cited backbones §4 backbone models, ConvGRU for the transient-variation update §3.2.1, the residual shortcut idea for the Motion Highway §1, and temporal-difference learning for the momentum update §3.2.2. These are named only; no cards exist for them in this wiki.
• Compared against: TrajGRU, Conv-TT-LSTM, ConvLSTM, MIM, PredRNN and E3D-LSTM tab. 1 tab. 4 tab. 5.
• Superseded by: not stated.