Frame difference

Overview

A frame difference is the subtraction of two frames, or of features computed from two frames, so that static pixels cancel and moving ones remain. The invariant that the variants share is that the difference is small where nothing moves and large mostly in moving regions 2012.10071 §3.2. It is a cheap way to add a motion cue to a 2D network, where optical flow would need a separate, costly network 2012.10071 §1.

Only one card in this wiki uses the idea so far. It is TDN, which applies the difference in two ways, at two depths of the network.

Variants

Short-term RGB differences

Differences between RGB values of consecutive frames in a small window around the centre frame. The window gives four differences, D_-2, D_-1, D_1 and D_2, stacked as channels. They are downsampled by half, passed through a small convolutional network, upsampled, and added to the centre-frame feature in the first two stages 2012.10071 §3.2 2012.10071 tab. 1(b). Card: TDN: Temporal Difference Networks for Efficient Action Recognition.

Reported effect on Something-Something V1, 8 frames, ResNet-50, one centre crop: 46.6% Top-1 without the modules, 51.3% with the short-term module at Conv1 only, and 51.5% at Conv1 and Res2 2012.10071 tab. 1(e).

Segment feature differences as a gate

Differences between features of neighbouring sampled segments. Each difference is aligned by a smoothing convolution, passed through three convolution branches at different scales, and a sigmoid turns the result into a gate in (0, 1). The gate is computed in both temporal directions and averaged, and it scales the features of the segment 2012.10071 §3.3 2012.10071 eq. 2 2012.10071 eq. 5 2012.10071 eq. 6. This is applied in the last three stages. Card: TDN.

Reported effect on the same ablation: 49.9% Top-1 with only this module, against 46.6% without modules and 52.3% with both modules [arxiv:2012.10071 tab. 1(e)]. The two numbers come from different placements, so they are not a direct comparison.

When to use which

The paper's numbers are for Something-Something V1 and V2 and Kinetics-400, with ResNet-50 and ResNet-101. Other motion cues are not compared on the same settings in this wiki, so the table does not rank frame differences against optical flow or 3D convolutions.

Implementation sketch

Two steps, one per variant. The short-term step works on pixels. The segment step works on features and gates them.

# Short-term motion cue from raw RGB frames  [arxiv:2012.10071 §3.2]
# frames: (K, C, H, W) consecutive RGB frames around one centre frame

def rgb_frame_difference(frames):
    # Difference of each adjacent pair; near zero where nothing moves
    return frames[1:] - frames[:-1]              # (K-1, C, H, W)

# Long-term motion gate from segment features  [arxiv:2012.10071 §3.3]
# feats: (T, C, H, W) features of T sampled segments in one stage

def segment_gate(feats):
    # Difference to the next segment, through scale branches  [eq. 4] [eq. 5]
    fwd = gate_branch(feats - shift(feats, -1))
    # Difference to the previous segment, averaged with the forward gate
    bwd = gate_branch(feats - shift(feats, +1))  # [eq. 6]
    gate = 0.5 * (fwd + bwd)                     # (T, C, H, W) in (0, 1)
    # Residual gating of the features  [eq. 2]
    return feats + feats * gate

gate_branch stands for the multi-scale convolutions and sigmoid eq. 5. shift stands for the neighbouring segment, with the boundary handled as the paper does not state 2012.10071 §3.3.

Contradictions

The paper's equations and its released code differ in the short-term fusion and in the gate. The paper adds the motion feature to the frame feature 2012.10071 eq. 1. The code blends the two with fixed weights that depend on the number of segments. The paper's sign for the segment difference also differs from the code's. Both differences are listed with citations on the TDN card under "Paper vs code". Use the code's behaviour if you need to match released checkpoints, and the paper's equations if you need to match the text.

Papers

• TDN: Temporal Difference Networks for Efficient Action Recognition: short-term RGB differences in early stages and segment feature differences as a gate in later stages, added to a 2D ResNet.