TDN: Temporal Difference Networks for Efficient Action Recognition

In plain words

Recognising an action in a video needs motion, not only appearance. A single frame shows a person and a cup, but not whether the cup is being lifted or pushed. Most fast video models get motion by stacking frames or by running a separate optical-flow network, and both can be slow.

TDN takes a cheaper route. It looks at the difference between neighbouring frames, which is closely related to what optical flow measures and is used as a cheaper stand-in for it. Where the pixels barely change, the difference is near zero, so the difference marks the parts of the picture that move. The network adds that motion signal to an ordinary 2D image network. It does this at two scales: a short-term signal from frames that sit next to each other, and a long-term signal that compares chunks of the whole video and uses the result to highlight the parts that matter for the action.

The design keeps the cost of a standard 2D convolutional network and adds a small overhead. The authors report that it beats other efficient video models on motion-heavy datasets. The paper is about the design and its ablations; the code is public.

TL;DR

• TDN is a video classifier built on a 2D ResNet with two temporal difference modules (TDMs), one short-term and one long-term §3.1.
• The short-term TDM (S-TDM) adds RGB differences of a small frame window to early stages. The long-term TDM (L-TDM) compares features of neighbouring sampled segments and uses the result as an attention gate in later stages §3.2 §3.3.
• Author-reported: 52.3% top-1 on Something-Something V1 with 8 frames of ResNet-50 at 36 GFLOPs, and state of the art among RGB-only models on SSv1 and SSv2 with similar backbones §4.3 tab. 2.
• Relevant if you need a 2D-cost action recognizer that is sensitive to motion. Check the paper-vs-code section first: the released code differs from the paper in several places.

Problem

Action recognition needs temporal modelling. Two common routes are costly. Two-stream networks need optical flow, which is expensive to compute §1. 3D convolutions add cost in the time dimension and often model time weakly §1. Plain 2D networks with late fusion over sampled frames (TSN-style) fail to capture fine temporal structure §2. The paper targets a video-level model that captures short-term and long-term motion at low extra FLOPs §1 §3.1.

Assumptions: the frames of a clip are sampled sparsely and uniformly across the video, as in TSN §3.1. The paper argues that Something-Something labels depend on motion rather than on objects or scene §4.1.

Method

TDN samples T segments from each video and reads one centre frame per segment plus its temporal neighbours §3.1. Each segment is processed by the same 2D backbone with shared weights fig. 2. Two modules add motion:

1. S-TDM (early stages, Conv1 and Res2): RGB differences in a local window around the centre frame are downsampled, passed through a small CNN and upsampled, then added to the centre-frame feature §3.2 eq. 1 eq. 3.
2. L-TDM (Res3 to Res5): for each pair of neighbouring segments, an aligned difference is computed, passed through a multi-scale branch and a sigmoid, and used as a spatiotemporal gate on the feature of the earlier segment §3.3 eq. 2 eq. 4 eq. 5 eq. 6.

The S-TDM and the L-TDM are shown in fig. 3. The overall architecture is in fig. 2. The exemplar is TDN-ResNet, which takes the TSN ResNet and inserts the two modules, placing S-TDM in the first two stages and L-TDM in the last three §3.4. Stage naming follows the paper: Conv1, Res2 to Res5 §3.4 tab. 1(d).

Overview

The T sampled frames form a tensor I of shape [T, C, H, W] §3.1. The short-term module adds motion to the frame feature F_i:

# Short-term TDM as a residual add  [§3.1] [eq. 1]
# F_i: frame feature at an early stage; I_i: centre RGB frame
def short_term_fusion(frame_feature, centre_frame):
    motion = short_term_tdm(centre_frame)
    return frame_feature + motion

The long-term module is a multiplicative gate on the frame feature:

# Long-term TDM as a gated residual  [§3.3] [eq. 2]
# F_i: frame feature of segment i; F_{i+1}: next segment
def long_term_fusion(frame_feature, next_feature):
    gate = long_term_tdm(frame_feature, next_feature)
    return frame_feature + frame_feature * gate

Short-term TDM

The motion input is a stack of temporal RGB differences in a local window centred on I_i. The window contains four differences, D_-2, D_-1, D_1 and D_2, stacked along the channel axis §3.2. The module then works at low resolution, because the differences are small and sparse §3.2:

# Short-term TDM H(I_i)  [§3.2] [eq. 3]
# Input:  centre frame and its window of RGB differences
# Output: motion feature at the same size as the frame feature

def short_term_tdm(window):
    # Four differences D_-2, D_-1, D_1, D_2 stacked on channels  [§3.2]
    diffs = stack_channels(window_differences(window))   # (12, H, W)
    # Downsample by half with average pooling  [§3.2]
    low = avg_pool(diffs, factor=2)                      # (12, H/2, W/2)
    # Light-weight 2D CNN on the low-resolution differences  [§3.2]
    motion = small_cnn(low)                              # (C, H/2, W/2)
    # Upsample back to the frame feature size  [§3.2] [eq. 3]
    return upsample(motion)                              # (C, H, W)

window_differences and stack_channels are described in words here: the differences are consecutive differences of the window frames, flattened into a 12-channel tensor for five RGB frames. The paper's own wording for the differences and their order is loose, so see the pitfalls.

Most of the RGB difference values are small, and large values appear only in moving regions, so a low-resolution network is enough §3.2. The S-TDM is fused by addition at Conv1 and at Res2 §3.2 tab. 1(b).

Long-term TDM

The L-TDM works on the features of all T segments of one video, so it needs the batch of segments at once §3.3.

# Long-term TDM, one direction  [§3.3] [eq. 4] [eq. 5]
# feats: (T, C', H, W) features after channel reduction by ratio r  [§3.3]
# Output: attention map, one per segment, in (0, 1)

def aligned_difference(feats, neighbour):
    # Smooth the neighbour spatially with a channel-wise conv  [§3.3] [eq. 4]
    return feats - channelwise_conv(neighbour)           # (C', H, W)

def long_term_attention(feats, neighbours):
    # Aligned temporal difference for each pair of segments  [§3.3]
    diff = aligned_difference(feats, neighbours)         # (T, C', H, W)
    # Three spatial scales, summed  [§3.3] [eq. 5]
    scales = scale_branch_1(diff) + scale_branch_2(diff) + scale_branch_3(diff)
    # One more convolution and a sigmoid  [§3.3] [eq. 5]
    return sigmoid(conv(scales))                         # (T, C, H, W)

The gate is built in both directions and averaged, so the module sees both the past and the future neighbour §3.3 eq. 6:

# Bidirectional gate and residual output  [§3.3] [eq. 6] [eq. 2]
# frames: (T, C, H, W) frame features of one stage

def long_term_tdm(frames):
    forward = long_term_attention(frames, next_segment(frames))
    backward = long_term_attention(frames, previous_segment(frames))
    # Average the two directions  [§3.3] [eq. 6]
    gate = 0.5 * (forward + backward)                    # (T, C, H, W)
    # Residual gating of the frame features  [§3.3] [eq. 2]
    return frames + frames * gate

The paper does not say how the first and last segments, which have no neighbour on one side, are handled. The released code pads with zeros (see Paper vs code and Pitfalls).

TDN-ResNet

The exemplar follows TSN and V4D §3.4. The first two stages carry S-TDMs for local motion. The last three stages carry L-TDMs and a temporal convolution in each residual block for longer-range structure §3.4. The paper states that this adds about 9% FLOPs over the 2D TSN-ResNet §3.4. That figure is not verified here.

Hyperparameters

Values are from the paper where stated, and from the released training commands and defaults where the paper is silent. Entries marked as code-only are not in the paper.

Results

Author-reported. Setup: 1-clip centre-crop test on Something-Something unless noted; 10-clip, 3-crop on Kinetics §4.1 §4.3. Backbones are ResNet-50 or ResNet-101 with ImageNet pre-training. Numbers are from the paper's tables, as printed there. Something-Something V1 and V2 comparison, RGB only, similar backbones §4.3 tab. 2:

Notes on the table: "-" means the value was not available to the authors tab. 2. The paper's footnote marks CorrNet with Sports-1M pre-training tab. 2. The Something-Something V2 rows for TDN are the paper's values. Other baselines are listed at 8 and 16 frames exactly as in the source.

Kinetics-400, RGB, similar backbones, 10 clips and 3 crops §4.3 tab. 3:

The paper states that the best Kinetics Top-1 is the SlowFast plus non-local combination, which is slightly better than TDN on Top-1, while its Top-5 is lower §4.3.

Transfer, 16-frame TDN-ResNet-50 pre-trained on Kinetics-400, mean over three splits app. A: UCF101 97.4% and HMDB51 76.3%. The paper says the gain is more visible on HMDB51, by around 2.5 points, than on UCF101 app. A.

Ablations on Something-Something V1, 8-frame TDN-ResNet-50, 1-clip centre crop tab. 1:

Table 1(e) in the paper's text is the same S-TDM versus L-TDM comparison that the appendix cites. The paper reports the baseline for Table 1(f) at 8 frames with ResNet-50 and at 40 frames for the "++" rows §4.2. The table rows above use the paper's own labels. Top-5 for the module comparison is 80.6% for TDM against 76.2% to 79.1% for the baselines tab. 1(f).

Running time on a Tesla V100, batch 16, including data loading app. B: TSN with 8 frames 7.9 ms per video, TSM with 16 frames 16.7 ms, STM with 8 frames 11.1 ms, I3D with 32x3x2 2095 ms, and TDN with 8 frames 22.1 ms. The paper states that TDN is slower than the previous methods but still real-time at 25 FPS or more app. B.

Visualisation with Grad-CAM shows that the TDN models attend to the moving hand and the object it interacts with, where the temporal-convolution baseline focuses more on background §4.4 fig. 4 app. C. This is qualitative.

Paper vs code

Code: https://github.com/MCG-NJU/TDN, commit a7c7e9bcd109d8256cd27a11505bd6e00de39ce0, Apache-2.0. The repository belongs to the authors' lab and is linked from the abstract. The code differs from the paper in these places. Each item is a discrepancy, not a bug report; the README results match the paper's headline numbers, so the released checkpoints presumably follow the code, not the text. That is an inference.

1. S-TDM fusion. The paper adds the motion feature to the frame feature, F + H §3.2 eq. 1 tab. 1(b). The code blends them with fixed weights: 0.5 and 0.5 for 8 segments, 0.75 and 0.25 for 16 tdn_net.py. The fusion is applied after the stem and after Res2 tdn_net.py.
2. S-TDM motion branch. The paper calls the motion CNN light-weight §3.2. The code uses a 7x7 stride-2 convolution with 12 input channels, initialised from the mean of the RGB stem kernel and repeated four times tdn_net.py, then a max-pool and a full ImageNet-initialised ResNet layer 1 tdn_net.py tdn_net.py. That is not light-weight.
3. Window and differences. The paper gives the four differences D_-2, D_-1, D_1, D_2 §3.2. The code takes five consecutive frames per segment and forms four consecutive differences tdn_net.py dataset.py.
4. L-TDM difference sign. The paper writes C(F_i, F_{i+1}) = F_i - Conv(F_{i+1}) eq. 4. The code computes Conv(F_{i+1}) - F_i for the forward direction base_module.py. The sign is reversed.
5. L-TDM convolution. The paper calls the smoothing convolution channel-wise §3.3. The code uses a depthwise 3x3 convolution with groups equal to C/r base_module.py.
6. L-TDM gate. The paper's gate is 0.5 times the sum of the two directional attentions, in (0, 1), used as F + F * G eq. 2 eq. 6. The code subtracts 0.5 from each sigmoid, so the gate lies in (-0.5, 0.5), and the output is x + x * gate base_module.py.
7. L-TDM scale branches. The paper lists a short connection, a 3x3 convolution, and average pooling, a 3x3 convolution and bilinear upsampling, and sums them §3.3 eq. 5. The code uses three branches: the difference itself, a 3x3 convolution on the full-resolution difference, and a 2x2 average pool, a 3x3 convolution and upsampling with the default nearest mode. It averages the three branches with weight 1/3 each base_module.py base_module.py. The paper's upsampling is bilinear.
8. L-TDM normalisation. The paper's attention is a convolution followed by a sigmoid eq. 5. The code adds BatchNorm after the 1x1 convolution base_module.py base_module.py. The two directions share the BatchNorm modules base_module.py.
9. L-TDM input. The paper applies the module to the frame feature F_i eq. 2. The code applies it to the ReLU output of the block's first 1x1 convolution, inside the residual branch, before the 3x3 convolution base_module.py. The outer residual is unchanged.
10. Temporal convolution. The paper mentions a temporal convolution in each residual block of Res3 to Res5 without specifying it §3.4. The code uses a learnable depthwise 3-tap convolution along the segment axis, initialised as a shift: one eighth of the channels take the previous frame, one eighth the next, and the rest are unchanged base_module.py base_module.py. It is inserted after the L-TDM and before the 3x3 convolution base_module.py.
11. Layer placement. The paper puts L-TDMs in Res3 to Res5 §3.4. In the code, the BottleneckShift block, which carries the L-TDM and the temporal convolution, is used for Res3 to Res5. Res2 uses the plain bottleneck base_module.py.
12. Batch and learning rate for Something-Something V1. The paper uses batch 128 and learning rate 0.02 §4.1. The README command uses learning rate 0.01 and batch 8 per process on 8 GPUs README.md.
13. LR schedule. The paper decays the learning rate when validation saturates §4.1. The code uses fixed milestones README.md lr_scheduler.py.
14. Augmentation. The paper resizes the shorter side to 256 to 320 and takes a 224 crop §4.1. The code uses multi-scale cropping with scales 1, .875, .75 and .66 models.py and horizontal flipping, which the paper does not mention. For Something-Something, flipping swaps the left and right classes main.py models.py.
15. Partial BN. Not in the paper. The flag is defined with default=True, action="store_true", so it is always True, and partial BN is therefore disabled opts.py main.py.
16. Results. The paper gives 89.5% Top-5 for TDN-ResNet-50 with 16 frames on Something-Something V2 tab. 2. The README gives 89.7% README.md. The README also lists Something-Something V2 entries with multi-crop testing (8x3x1, 16x3x1 for ResNet-101, and 8+16 with 3 crops) that the paper does not report README.md README.md.

Pitfalls

• Five frames per segment, not one. The dataset loader reads five consecutive frames for each segment, and the network reshapes its input to five frames dataset.py models.py. Decoding and I/O cost is five times a TSN loader that reads one frame per segment. The paper never states this; the window size is only implied by the four differences §3.2.
• Fusion weights change the pretrained features. Adding the motion feature (paper) and blending it with 0.5 weights (code) are different operations for the pretrained layers. Before reproducing the numbers, check which one the training code uses. The code selects weights by segment count, using num_segments is 8, an identity comparison that works for small integers but is fragile tdn_net.py.
• Boundary segments. For the first and last segment, the L-TDM has no neighbour on one side. The paper does not say what it does. The code zero-pads the difference base_module.py.
• Sign and offset of the gate. The reversed difference sign (item 4) and the -0.5 offset (item 6) change the gate's range and direction. A reimplementation that follows Eq. 2 and Eq. 6 literally will not match the released code.
• Shared BatchNorm across directions. The forward and backward branches reuse the same BatchNorm and convolution modules base_module.py. The running statistics are computed on both directions combined. The paper does not mention this.
• "8+16" is not defined in the text. The tables use it, but the paper never says what it means. The FLOPs equal the sum of the 8- and 16-frame costs, which suggests an ensemble of two models. This is an inference. Compare "8+16" rows with care.
• Test-time protocol. The 1-clip rows use centre cropping with a shorter side of 256 §4.1. The 10-clip, 3-crop rows need 30 forward passes per video README.md. Reported numbers depend on this.
• Kinetics videos. The authors note that downloaded Kinetics videos may differ from the originals, so results may vary README.md. The paper uses about 240k training videos §4.1.
• Variance and seeds. Seeds, number of runs and variance are not stated for any table.
• Speed. The paper reports 22.1 ms per 8-frame video on a V100 with batch 16, including data loading app. B. That is slower than the baselines it names in the same table. The paper calls this real-time (25 FPS or more) app. B. The timing includes data loading, so it is not a pure model cost, and a five-frame loader affects it (see the first pitfall).
• Equation numbering. Equation numbers here (eq. 1 to eq. 6) follow the order of the equations in the LaTeX source. Verify against the PDF before citing a number in another document.
• Table numbering. Table 1 is the ablation with sub-tables (a) to (f) tab. 1. Table 2 and Table 3 are the Something-Something and Kinetics comparisons tab. 2 tab. 3, numbered by source order. Appendix tables and figures are cited by section.
• Efficiency claims. The paper's FLOP counts are per 224x224 1-clip inputs tab. 1 caption. Wall-clock time is a separate measure app. B and was not compared with the FLOPs.

Lineage

• Builds on: TSN sparse segment sampling and late fusion §3.1 §2; ResNet backbone §3.4; the stage split of V4D §3.4; feature-difference ideas from TEINet, TEA and STM §2; the released code is built on the TSN and TSM code bases README.md.
• Compared against: TSM, TEINet, TANet, TEA, TAM, GST, STM, SmallBigNet, CorrNet, GSM, I3D, Non-local I3D, SlowFast, X3D and others, as listed in tab. 2 and tab. 3.
• Superseded by: not stated in this wiki.