---
kind: concept
slug: frame-difference
title: Frame difference
summary: >-
  Subtracting neighbouring frames, or features of neighbouring segments, to get a cheap motion signal for a 2D video model. Only one card so far: TDN uses pixel differences in early stages for short-term motion and feature differences as a gate in later stages for long-term motion. Author-reported gains are from ablations on Something-Something V1.
tags: [video, action-recognition, temporal-modeling, efficient-inference]
papers: ["2012.10071"]
related: [optical-flow, convolutional-neural-network, efficient-vision-backbone, temporal-convolutional-network]
status: draft
updated_at: 2026-10-24
---

## Overview

A frame difference is the subtraction of two frames, or of features computed
from two frames, so that static pixels cancel and moving ones remain. The
invariant that the variants share is that the difference is small where
nothing moves and large mostly in moving regions [arxiv:2012.10071 §3.2]. It
is a cheap way to add a motion cue to a 2D network, where optical flow would
need a separate, costly network [arxiv:2012.10071 §1].

Only one card in this wiki uses the idea so far. It is TDN, which applies
the difference in two ways, at two depths of the network.

## Variants

### Short-term RGB differences

Differences between RGB values of consecutive frames in a small window around
the centre frame. The window gives four differences, D_-2, D_-1, D_1 and D_2,
stacked as channels. They are downsampled by half, passed through a small
convolutional network, upsampled, and added to the centre-frame feature in
the first two stages [arxiv:2012.10071 §3.2] [arxiv:2012.10071 tab. 1(b)].
Card: [TDN: Temporal Difference Networks for Efficient Action Recognition](/papers/2012.10071).

Reported effect on Something-Something V1, 8 frames, ResNet-50, one centre
crop: 46.6% Top-1 without the modules, 51.3% with the short-term module at
Conv1 only, and 51.5% at Conv1 and Res2 [arxiv:2012.10071 tab. 1(e)].

### Segment feature differences as a gate

Differences between features of neighbouring sampled segments. Each
difference is aligned by a smoothing convolution, passed through three
convolution branches at different scales, and a sigmoid turns the result
into a gate in (0, 1). The gate is computed in both temporal directions and
averaged, and it scales the features of the segment [arxiv:2012.10071 §3.3]
[arxiv:2012.10071 eq. 2] [arxiv:2012.10071 eq. 5] [arxiv:2012.10071 eq. 6].
This is applied in the last three stages. Card: [TDN](/papers/2012.10071).

Reported effect on the same ablation: 49.9% Top-1 with only this module,
against 46.6% without modules and 52.3% with both modules [arxiv:2012.10071
tab. 1(e)]. The two numbers come from different placements, so they are not a
direct comparison.

## When to use which

| Situation | Use | Why |
|---|---|---|
| A 2D backbone must see motion, and the cost budget is close to a plain 2D model | Short-term RGB differences in early stages | Operates at half resolution on a small network [arxiv:2012.10071 §3.2]; the paper reports 36 GFLOPs for 8 frames, against 33 GFLOPs without the modules [arxiv:2012.10071 tab. 1(d)] |
| Motion has to be compared across the whole sampled clip | Segment feature differences as a gate | It compares segments that are far apart in the clip [arxiv:2012.10071 §3.3] |
| Both short-range and long-range motion matter | Both, as in TDN | Best ablation result: 52.3% against 51.5% for the short-term module alone [arxiv:2012.10071 tab. 1(e)] |
| Latency is a hard limit, including data loading | Measure first | TDN takes 22.1 ms per 8-frame video on a V100 at batch 16, against 7.9 ms for TSN, including data loading [arxiv:2012.10071 app. B] |
| The data loader cannot read several consecutive frames per segment | Not this approach as written | The four differences need a window of consecutive frames [arxiv:2012.10071 §3.2]; the card's pitfalls list the window size used in the released code |

The paper's numbers are for Something-Something V1 and V2 and Kinetics-400,
with ResNet-50 and ResNet-101. Other motion cues are not compared on the same
settings in this wiki, so the table does not rank frame differences against
optical flow or 3D convolutions.

## Implementation sketch

Two steps, one per variant. The short-term step works on pixels. The
segment step works on features and gates them.

```pseudocode
# Short-term motion cue from raw RGB frames  [arxiv:2012.10071 §3.2]
# frames: (K, C, H, W) consecutive RGB frames around one centre frame

def rgb_frame_difference(frames):
    # Difference of each adjacent pair; near zero where nothing moves
    return frames[1:] - frames[:-1]              # (K-1, C, H, W)
```

```pseudocode
# Long-term motion gate from segment features  [arxiv:2012.10071 §3.3]
# feats: (T, C, H, W) features of T sampled segments in one stage

def segment_gate(feats):
    # Difference to the next segment, through scale branches  [eq. 4] [eq. 5]
    fwd = gate_branch(feats - shift(feats, -1))
    # Difference to the previous segment, averaged with the forward gate
    bwd = gate_branch(feats - shift(feats, +1))  # [eq. 6]
    gate = 0.5 * (fwd + bwd)                     # (T, C, H, W) in (0, 1)
    # Residual gating of the features  [eq. 2]
    return feats + feats * gate
```

`gate_branch` stands for the multi-scale convolutions and sigmoid [eq. 5].
`shift` stands for the neighbouring segment, with the boundary handled as
the paper does not state [arxiv:2012.10071 §3.3].

## Contradictions

The paper's equations and its released code differ in the short-term fusion
and in the gate. The paper adds the motion feature to the frame feature
[arxiv:2012.10071 eq. 1]. The code blends the two with fixed weights that
depend on the number of segments. The paper's sign for the segment difference
also differs from the code's. Both differences are listed with citations on
the [TDN card](/papers/2012.10071) under "Paper vs code". Use the code's
behaviour if you need to match released checkpoints, and the paper's
equations if you need to match the text.

## Papers

- [TDN: Temporal Difference Networks for Efficient Action Recognition](/papers/2012.10071): short-term RGB differences in early stages and segment feature differences as a gate in later stages, added to a 2D ResNet.