Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks

In plain words

Recognising what a person is doing from a stream of body-joint positions (a "skeleton", as produced by a depth camera) is easier than from raw video, but earlier neural methods looked at each frame's joints as one long list and mostly tracked how the whole pose changes over time. They largely ignored how the joints relate to each other in space, such as a hand being near the head.

This paper uses two networks side by side. One reads the skeleton as a movie: it follows how the body moves over time, either by reading all joints together or by reading each body part and then the whole body. The other reads the skeleton as a sequence of joints: it turns the body's joint layout into one ordered list and reads that list, so it learns how the joints relate to each other in space. The two networks each give their class scores, and the scores are combined. To make the model less fragile, the training skeletons are randomly rotated, scaled and sheared, which imitates different camera angles and different body sizes.

The approach matters because it improves on earlier skeleton methods on three benchmarks, covering generic actions, two-person interactions and gestures. The gains are largest for cross-view evaluation.

TL;DR

• Two RNN streams: a temporal stream (per-frame joint coordinates) and a spatial stream (joints as a sequence, one time window per joint) §4.
• Temporal stream is either stacked LSTM or a hierarchical LSTM over body parts §4.1.
• Spatial stream orders joints as a chain or as a graph traversal §4.2.
• 3D rotation, scaling and shear augmentation is used during training only §4.3.
• Author-reported: 71.3 / 79.5 % (NTU cross-subject / cross-view), 94.8 % (SBU), 91.7 F1 (ChaLearn). No official code found.

Problem

Skeleton-based action recognition takes 3D joint coordinates over time as input §1. The authors argue that earlier RNN methods model only temporal context, by concatenating all joints of each frame, and ignore the spatial configuration of articulated joints §1 §2. Handcrafted features and simple time-series models are cited as limited in the same way §1.

Data is limited in size and variety, so the authors add augmentation to reduce overfitting §4.3.

Method

Overview. Skeleton sequences are modelled by two streams that are trained end to end and combined at the output. The fusion is a combination of the softmax class posteriors of the two networks §4 fig. 1. The paper does not write the fusion formula out; see Pitfalls.

Temporal stream §4.1. Each time step concatenates the 3D coordinates of all joints (or of a body part). Both variants use LSTM units in every layer §5.2.

1. Stacked RNN §4.1: two LSTM layers over per-frame concatenated joints. The authors report that a third layer does not considerably help §4.1.
2. Hierarchical RNN §4.1 fig. 3: the five body parts (two arms, two legs, one trunk) each get an LSTM over their own joints. Their outputs are concatenated per time step and fed to a second LSTM for the whole body. Two layers vertically, as in the stacked version. No extra fully connected layers are used before the softmax classifier §4.1.

Spatial stream §4.2. Each joint is one input step. A joint has only three coordinates, so the input for a joint is the coordinates of that joint over a temporal window centred on the current frame §4.2. The step sequence is the joints in an order chosen by one of two methods.

1. Chain sequence §4.2 fig. 4(b): joints ordered arms, then trunk, then legs. The trunk sits in the middle. The drawback named by the authors: some neighbours in the list (joint 13 and joint 20 in the 20-joint MSR Action3D example) are not physically connected §4.2.
2. Traversal sequence §4.2 fig. 4(c): a graph walk that starts at the central spine joint, goes down the left arm and back, then the right arm, then the upper trunk, and so on, ending at the start. Most joints appear twice, once in each direction. The traversal idea is inspired by a tree traversal in an earlier spatio-temporal LSTM §4.2.

The spatial stream has the same structure as the stacked temporal RNN §5.2.

Augmentation §4.3. Applied only during training.

• Rotation: composed from three axis rotations (Euler angles) as R = R_z(γ) R_y(β) R_x(α) eq. 5. Axis matrices R_x, R_y, R_z are given as eq. 2, eq. 3 and eq. 4. The text says the input is rotated randomly "within a certain range" for the x and y axes, with the camera plane perpendicular to z. The authors say this simulates viewpoint change §4.3.
• Scaling: diagonal matrix S with scale factors s_x, s_y, s_z eq. 6, random per sample, meant to cover different body sizes §4.3.
• Shear: a 3×3 matrix with six off-diagonal shear factors eq. 7, described in words only §4.3.

Pre-processing §5.2. Each skeleton is centred by subtracting the mean of the hip centre, left hip and right hip. Each sequence is resampled or zero-padded to a fixed length T.

Training §5.2. Stochastic gradient descent, learning rate 0.02, multiplied by 0.7 every 60 epochs. Implemented in Theano with Lasagne, one NVIDIA TITAN X GPU.

Pseudocode of the two-stream forward pass. This is our summary of the method; the fusion weight and window size are described in §5.2 and §5.5.

# Two-stream RNN forward pass  [§4] [fig. 1]
# Input:  skeletons (B, T, J, 3), J joints per frame
# Output: class posteriors (B, C)

def two_stream_forward(skeletons):
    # 1. Centre each skeleton on the hip centre  [§5.2]
    skeletons = skeletons - hip_centre(skeletons)   # (B, T, J, 3)

    # 2. Temporal stream: one vector per frame with all joints  [§4.1]
    frames = reshape(skeletons, (B, T, 3 * J))
    temporal_posteriors = temporal_rnn(frames)      # (B, C)

    # 3. Spatial stream: order joints as a sequence  [§4.2]
    ordered = to_sequence(skeletons)  # chain or traversal order
    # 4. Each joint becomes its coordinates over a time window  [§4.2]
    windows = joint_windows(ordered, window)        # (B, n_steps, 3 * window)
    spatial_posteriors = spatial_rnn(windows)       # (B, C)

    # 5. Fuse the two posteriors  [§4] [§5.2]
    return fuse(temporal_posteriors, spatial_posteriors)

LSTM unit used in all RNN layers eq. 1 fig. 2(b). The paper uses peephole connections from the cell state to the gates. The gate equations are:

\begin{aligned} i_t &= \sigma(W_{xi} x_t + W_{hi} h_{t-1} + W_{ci} c_{t-1} + b_i) \\ f_t &= \sigma(W_{xf} x_t + W_{hf} h_{t-1} + W_{cf} c_{t-1} + b_f) \\ c_t &= f_t c_{t-1} + i_t \tanh(W_{xc} x_t + W_{hc} h_{t-1} + b_c) \\ o_t &= \sigma(W_{xo} x_t + W_{ho} h_{t-1} + W_{co} c_t + b_o) \\ h_t &= o_t \tanh(c_t) \end{aligned} eq. 1

Hyperparameters

Values are as stated in the paper unless marked.

Results

Author-reported. Setup: NTU RGB+D (cross-subject and cross-view protocol §5.1), SBU Interaction (5-fold CV, accuracy) and ChaLearn Gesture (only skeletons, precision, recall and F1 on the validation set) §5.1 tab. 1. Baselines are the authors' own ablations and the published numbers of earlier skeleton methods tab. 3–5.

Ablation tab. 1:

Structure study, NTU cross-view tab. 2: stacked R512-512 68.9, R512-512-512 69.2, R512 68.6, R1024-1024 68.9, R256-256 68.2. Hierarchical P128, B512 70.5, P128-128 B512 71.4, P256 B512 71.4, P128 B1024 70.6, P256 B1024 72.2.

Comparison with state of the art (cited values):

The text reports gains of 2.1 % and 1.8 % over trust-gate ST-LSTM on NTU §5.6, 7.9 % over the best handcrafted SBU method and 1.5 % over the best prior RNN method on SBU §5.6, and more than 16 % over VideoDarwin on ChaLearn §5.6.

Other findings (text and figures):

• Two-stream beats each single stream on all three datasets §5.3.
• 3D transform adds 2.7 % cross-subject and 7.8 % cross-view on NTU §5.3.
• Hierarchical beats stacked on NTU by about 1.6 % on average §5.3.
• Traversal beats chain for the spatial stream on every dataset §5.3 tab. 1.
• On SBU the two-stream model confuses fewer pairs than the temporal stream alone, and the push class drops from 22 % misclassified to none §5.4 fig. 6.
• Per-action accuracy on NTU is higher for two-stream on most actions; for brushing teeth, shaking head and walking towards the gain is more than 8 % §5.4 fig. 5.
• On SBU, accuracy versus the spatial window τ is best for 5 ≤ τ ≤ 17 and drops outside that range §5.5 fig. 7.

Paper vs code

No official code.

Searched: (1) the paper body and the bibliography contain no repository link. The only code link is a footnote to the Lasagne framework §5.2, which is not the authors' code. (2) The raw arXiv metadata does not include the comments field, so no link there was checked. (3) GitHub repository search for "two-stream RNN skeleton", "skeleton two-stream recurrent" and "skeleton action RNN stacked hierarchical" returned nothing. GitHub user search for "Hongsong Wang CASIA" returned nothing. A repository search for "hongsong wang" found two personal homepage repositories under the account "hongsong-wang". They contain no code for this paper and no licence file, so they were not used.

Pitfalls

• Symbols swapped in §5.5. The text says λ is the window size and τ the temporal weight. The axis labels and the figure caption (Figure 7) say λ is the temporal weight and τ is the window size. Follow the figure. We cannot tell from the paper alone which reading the authors intended for the numeric values.
• Fusion weight vs. best value. §5.2 fixes the temporal weight at 0.9. The §5.5 sweep on SBU shows the best accuracy at 0.8 or 0.9. Other datasets are not shown, only said to be similar §5.5.
• Window size. §5.2 sets the spatial window to T/4 (about 8.75 steps for T = 35). §5.5 reports a good range of 5 ≤ τ ≤ 17 on SBU. Integer rounding is not stated.
• Fusion formula not written. The paper says the fusion combines the softmax posteriors §4 and gives a weight for the temporal stream §5.2. The form (a weighted sum, a product, or a log-sum) and whether the streams are trained jointly with the fused loss or separately are not stated. The text says the network is end-to-end trainable §1.
• Augmentation is underspecified. No ranges or probabilities are given for rotation, scaling or shear §4.3. The "3D transform" row of Table 1 does not say which of the three transforms it includes. The text says rotation is applied to x and y, yet the composite rotation eq. 5 has three axes. Whether z-rotation is used is not stated.
• Centre joint. The centre is the mean of hip centre, left hip and right hip §5.2. The joint indices used for this in the code are not given in the paper.
• Variable T. Sequences are "sampled and zero padded" to T §5.2. The sampling scheme (uniform, random, first T frames) is not stated.
• Hierarchical default is not the best setting. The default P128, B512 gives 70.5 on NTU cross-view. P256, B1024 gives 72.2 but is larger tab. 2. The authors chose the smaller model "to make a fair comparison" §5.5.
• Hidden row in the LaTeX source. The source of Table 5 contains a commented-out HBRNN row at 92.0 F1 (ChaLearn) that is not in the published table. The published table lists Two-stream RNN at 91.7 F1 as the best. The comment is not part of the paper, but any reader of the source will see it.
• Peephole LSTM. Eq. 1 includes peephole terms from c to the gates, and the output gate uses the new cell state c_t. Standard LSTM implementations differ; check which one you use.
• Two-person handling. NTU samples are split into single persons and the scores averaged §5.2. SBU concatenates both persons into 6D per joint §5.2. Mixing the two conventions changes the input size.
• Cost. Two RNN streams, each with two layers of 512 (or 128 + 512) units, and a spatial stream that runs one sequence per window. No FLOPs, parameter counts or inference timings are reported.
• Traversal length. The traversal sequence visits most joints twice and returns to the start §4.2. Its exact length is not stated, so input shapes must be derived from the traversal definition in fig. 4(c).
• Figure-only details. Figure 4(c) defines the traversal order. The complete joint list for NTU (25 joints) and for ChaLearn is not in the text.

Lineage

• Builds on: LSTM §3 fig. 2. Hierarchical RNN for skeletons (Du et al., HBRNN) §2 §4.1, whose structure this paper simplifies. Part-aware LSTM (Shahroudy et al.) §2. Trust-gate spatio-temporal LSTM (Liu et al.), whose tree traversal inspired the traversal sequence §4.2.
• Compared against: HBRNN, part-aware LSTM, trust-gate ST-LSTM, Lie group, skeletal quads, FTP dynamic, and for SBU and ChaLearn, the handcrafted and VideoDarwin methods listed in tab. 3–tab. 5.
• Superseded by: not stated in this card.