Video Swin Transformer

In plain words

Recognising what happens in a video means looking at many frames at once. A plain transformer lets every small patch of every frame look at every other patch, which is accurate but expensive, because the cost grows quickly with the number of frames and the frame size.

This paper keeps the attention local instead. The video is cut into small blocks that cover both space and time, and each block only attends to its neighbours inside a fixed box. Every other layer slides the boxes by half their size, so information can still flow across the edges of the boxes. The design is a video version of an image transformer that already worked this way, so it can start from weights trained on still images. The authors report that this gives top accuracy on standard action-recognition benchmarks with a smaller model and less pre-training data than the competing transformers.

TL;DR

Video Swin Transformer is a hierarchical video backbone whose self-attention is restricted to 3D windows that shift between consecutive blocks. It inherits the Swin image architecture, including its ImageNet pre-trained weights, and is fine-tuned on video. The authors report state-of-the-art top-1 accuracy on Kinetics-400 (84.9), Kinetics-600 (86.1) and Something-Something v2 (69.6), using multi-view inference that is costly. Read the Pitfalls section before reproducing the numbers.

Problem

Full spatio-temporal self-attention over all video tokens is very expensive in computation and memory. The earlier video transformers the paper reviews all use global self-attention, some of them factorised over space and time §2. The authors argue that videos have a locality the models do not exploit: pixels that are close in space and time are more likely to be correlated §1. A T-frame video gives T/2 × H/4 × W/4 tokens §3.1, and the paper notes that videos need far more tokens than images, so global attention is too costly §3.2.

Method

The backbone follows the four-stage hierarchy of Swin Transformer. It does not downsample time, only space §3.1. The steps below are the pieces an implementer needs.

1. 3D patch partition. Each 2×4×4×3 patch (two frames, 4×4 pixels, RGB) becomes one token of 96 values. A linear layer projects the token to the width C of stage 1 §3.1.
2. Four stages. Each stage runs a number of Video Swin blocks, with channel width doubling per stage. A patch merging layer between stages concatenates each 2×2 group of spatial neighbours (4C values) and projects it to 2C through a linear layer. No temporal merging is done §3.1.
3. Block structure. Each block is a pre-norm transformer layer. Layer norm, then 3D window multi-head self-attention, then a residual add. Then layer norm, a two-layer MLP with GELU, and another residual add §3.1. Blocks come in pairs. The first uses regular windows and the second uses shifted windows, as given in Eq. 1 §3.2.

# Video Swin block pair  [§3.1] [eq. 1]
# Input:  tokens (B, T', H', W', C), with T' = T/2, H' = H/4, W' = W/4
# Window: (P, M, M) = (8, 7, 7) by default  [§3.3]

def video_swin_block_pair(tokens):
    # Regular 3D windows: attention, then the MLP  [§3.2] [eq. 1]
    tokens = tokens + window_msa(layer_norm(tokens), (0, 0, 0))
    tokens = tokens + mlp(layer_norm(tokens))
    # Shifted 3D windows: the same pair of sub-layers  [§3.2] [eq. 1]
    half = (P // 2, M // 2, M // 2)  # shift in tokens, see Pitfalls
    tokens = tokens + window_msa(layer_norm(tokens), half)
    tokens = tokens + mlp(layer_norm(tokens))
    return tokens

4. Non-overlapping 3D windows. The token grid is split into windows of P×M×M tokens, with P=8 along time and M=7 along height and width by default. Attention is computed inside each window only §3.2.
5. Shifted windows. In the second block of a pair, the window grid is offset by (P/2, M/2, M/2) tokens. The paper reuses the batched shifted-window computation of Swin, so the number of windows stays the same §3.2, fig. 3. The code does this with a cyclic roll and an additive mask that blocks attention between tokens that were not neighbours before the roll swin_transformer.py.
6. Attention with 3D relative position bias. Each head computes softmax(QK^T/√d + B)V, where B holds one learned bias per head for each pair of relative offsets. The bias table has (2P−1)×(2M−1)×(2M−1) entries per head, and the values of B are looked up from it §3.2, eq. 2. The query and key width is d = 32 per head §3.3.

# Window multi-head self-attention  [§3.2] [eq. 2]
# Input x: (B, T', H', W', C); shift in tokens; d = 32 per head
def window_msa(x, shift):
    # Roll the grid so shifted windows line up with regular ones  [§3.2]
    # [code: mmaction/models/backbones/swin_transformer.py#L228-L229]
    if any(s > 0 for s in shift):
        x = roll(x, by=negate(shift), axes=(T, H, W))
    # Pad T', H', W' up to multiples of the window size (paper is silent)
    # [code: mmaction/models/backbones/swin_transformer.py#L222-L224]
    x = pad_to_multiple(x, (P, M, M))
    # Non-overlapping windows of N = P*M*M tokens each  [§3.2]
    windows = partition(x, (P, M, M))          # (B*nW, N, C)
    queries, keys, values = project(windows)   # d = 32 per head  [§3.3]
    # One learned bias per head and token pair, read from the table  [§3.2]
    # [eq. 2]
    bias = gather(relative_bias_table, relative_index)  # (heads, N, N)
    scores = queries @ transpose(keys) / sqrt(d) + bias
    # Mask pairs that came from different regions before the roll  [§3.2]
    # Masked entries get -100 in the code (paper is silent)
    if any(s > 0 for s in shift):
        scores = scores + window_mask
    windows = softmax(scores) @ values
    x = reverse_partition(windows)
    # Undo the roll
    # [code: mmaction/models/backbones/swin_transformer.py#L242-L243]
    if any(s > 0 for s in shift):
        x = roll(x, by=shift, axes=(T, H, W))
    return unpad(x)

7. Architecture variants. Four sizes, named after Swin's: Swin-T with C=96 and layers {2, 2, 6, 2}; Swin-S with C=96 and {2, 2, 18, 2}; Swin-B with C=128 and {2, 2, 18, 2}; Swin-L with C=192 and {2, 2, 18, 2}. In the paper's ratios, the four sizes span roughly a quarter to twice the compute of the reference size, in that order §3.3.
8. Initialisation from an image model. The 2D checkpoint is inflated to 3D in two places. The patch-embedding kernel is repeated over the two temporal slots and multiplied by 0.5, which keeps the output mean and variance. The relative bias table is repeated over the 2P−1 temporal offsets, so the bias is the same in every frame §3.4.

# Initialise a 3D Video Swin model from a 2D Swin checkpoint  [§3.4]
def inflate_checkpoint(state):
    # Repeat the patch kernel over the 2 temporal slots, then halve it  [§3.4]
    state.patch_kernel = repeat(state.patch_kernel, 2, axis=T) * 0.5
    # Repeat the (2M-1)^2 bias table over the (2P-1) temporal offsets  [§3.4]
    state.bias_table = repeat(state.bias_table, 2 * P - 1, axis=0)
    # The 3D relative index and attention masks are rebuilt, not loaded
    # [code: mmaction/models/backbones/swin_transformer.py#L578-L586]
    return state

Key equation for the block pair eq. 1:

\hat z^{l} = \text{3DW-MSA}(\text{LN}(z^{l-1})) + z^{l-1}, \quad z^{l} = \text{FFN}(\text{LN}(\hat z^{l})) + \hat z^{l}

\hat z^{l+1} = \text{3DSW-MSA}(\text{LN}(z^{l})) + z^{l}, \quad z^{l+1} = \text{FFN}(\text{LN}(\hat z^{l+1})) + \hat z^{l+1}

\text{Attention}(Q, K, V) = \text{SoftMax}(QK^T/\sqrt{d} + B)V \quad \text{[eq. 2]}

Hyperparameters

Values are as stated in the paper. Where the code differs, see Paper vs code.

Results

Author-reported. Setup: Kinetics-400 and Kinetics-600 with ImageNet pre-trained initialisation, Something-Something v2 initialised from the K400 model §4.1. Views are temporal clips × spatial crops tab. 1. FLOPs are in giga-units and parameters in mega-units, as captioned in tab. 1. The paper does not say whether FLOPs are per view.

Kinetics-400, top-1 and top-5 accuracy tab. 1:

Kinetics-600 tab. 2:

Something-Something v2 tab. 3:

Ablations, all on Kinetics-400 with Swin-T unless stated tab. 4 to tab. 9:

• Spatio-temporal attention variants tab. 4: joint 3D windows 78.8 / 93.6 (88 G, 28.2 M); split temporal layers 76.4 / 92.1 (83 G, 42.0 M); factorised temporal layers 78.5 / 93.5 (95 G, 36.5 M). The joint form is the default.
• Temporal tokens and window tab. 5: with 16 temporal tokens, a temporal window of 16 gives 79.1 / 93.8 at 106 G, 8 gives 78.8 / 93.6 at 88 G, and 4 gives 78.6 / 93.4 at 79 G. With 8 temporal tokens and window 8, the result is 78.5 / 93.2 at 44 G. With 4 tokens and window 4, it is 76.7 / 92.5 at 20 G.
• Shifted windows tab. 6: full 3D shifting 78.8 / 93.6; no temporal shift 78.5 / 93.5; no shifting 78.1 / 93.3.
• Backbone-to-head learning-rate ratio tab. 7, Swin-B: ImageNet-1K pre-training, 0.1× gives 80.6 / 94.6 and 1.0× gives 80.2 / 94.2; ImageNet-21K pre-training, 0.1× gives 82.6 / 95.7 and 1.0× gives 82.0 / 95.3.
• Initialisation tab. 8, tab. 9: patch-embedding inflate 78.8 / 93.6 vs centre 78.8 / 93.7; relative bias duplicate 78.8 / 93.6 vs centre 78.8 / 93.6.

Paper vs code

The official repository is at commit db018fb8896251711791386bbd2127562fd8d6a6. It is linked from the paper's abstract. The license is Apache-2.0 (LICENSE). The backbone is built on the mmaction2 framework, and the source headers carry an Open-MMLab copyright notice LICENSE.

• Relative bias table size. The text writes B ∈ R^{P²×M²×M²} §3.2, but the same paragraph and the code use one entry per relative offset, so the table is (2P−1)(2M−1)(2M−1) per head [code: mmaction/models/backbones/swin_transformer.py#L109-L111]. Treat the B dimensions in the text as a typo.
• Learning rate. The text gives 3e-5 for the backbone and 3e-4 for the head §4.1. The Swin-B 22K K400 config matches this: lr 3e-4, backbone ×0.1 swin_base_patch244_window877_kinetics400_22k.py. The Swin-T and Swin-S 1K K400 configs use lr 1e-3 with backbone ×0.1, so the backbone rate is 1e-4 swin_tiny_patch244_window877_kinetics400_1k.py swin_small_patch244_window877_kinetics400_1k.py. The text does not say which variants use which rate.
• Stochastic depth. The text gives 0.1 (T), 0.2 (S), 0.3 (B) §4.1. The code uses 0.1 for Swin-T swin_tiny_patch244_window877_kinetics400_1k.py, 0.1 for Swin-S swin_small_patch244_window877_kinetics400_1k.py and 0.2 for the Swin-B 22K config swin_base_patch244_window877_kinetics400_22k.py. The Swin-S and Swin-B values differ from the text.
• Weight decay matches the text: 0.02 for T and S, 0.05 for B swin_tiny_patch244_window877_kinetics400_1k.py, swin_base_patch244_window877_kinetics400_22k.py.
• Batch size 64. The code divides the per-GPU batch by update_interval and accumulates gradients over that many steps train.py optimizer.py. Base configs use 8 videos per GPU per step and 8 accumulation steps, so a 64-video update needs 8 GPUs. The Swin-T config uses update_interval=4 swin_tiny_patch244_window877_kinetics400_1k.py, which gives 2 videos per step and 8 per GPU after accumulation. The paper does not state the GPU count.
• K400/K600 augmentation. The text does not describe training augmentation for K400 or K600. The code uses RandomResizedCrop and a horizontal flip with probability 0.5 swin_base_patch244_window877_kinetics400_22k.py.
• SSv2 augmentation. The text names label smoothing, RandAugment and random erasing §4.1. The code uses RandAugment with n=4, m=7, random erasing with probability 0.25, label smoothing 0.1, and no horizontal flip swin_base_patch244_window1677_sthv2.py swin_base_patch244_window1677_sthv2.py. The code also samples frames uniformly for SSv2 training swin_base_patch244_window1677_sthv2.py. The text mentions none of these settings.
• SSv2 initialisation. The SSv2 config has no pretrained key. The README's SSv2 example passes the checkpoint with load_from on the command line README.md. The K400 starting point is therefore set outside the config.
• Missing configs. The repository has no config for Swin-B 1K on K400, for Swin-L (only its base model file exists), or for 384 input. The README links a Swin-B 1K K400 config that is not in this commit README.md. Table 1's Swin-B 1K and 384↑ rows cannot be reproduced from these configs.
• Initialisation. The code divides the patch kernel by patch_size[0] swin_transformer.py. With temporal patch size 2 this equals the text's duplicate-and-halve. The centre initialisation and the −4.6 cross-frame masking described for tables 8 and 9 §4.3 are not in the backbone file. The backbone implements only the duplicate-and-tile path swin_transformer.py.
• Heads per stage. The text does not give them. The code uses [3, 6, 12, 24] for Swin-T [code: configs/base/models/swin/swin_tiny.py#L8-L9] and [4, 8, 16, 32] for Swin-B [code: configs/base/models/swin/swin_base.py#L3-L4]. Both equal stage width divided by 32, which matches d = 32 §3.3.
• Mask value and padding. The code uses −100 for masked pairs swin_transformer.py, pads inputs to multiples of the window swin_transformer.py, and sets a window dimension to the input size when the input is smaller, with shift 0 swin_transformer.py. None of this is in the paper.
• Views. Test uses 4 clips with a three-crop, which matches the text swin_base_patch244_window877_kinetics400_22k.py. The base config sets max_testing_views=2. That batches views to save memory and is not a limit on the views used recognizer3d.py.

Pitfalls

• Half-window shift with odd M. The paper writes the shift as (P/2, M/2, M/2) §3.2. With M=7 this is 3.5, which is not an integer. The code uses integer halving, so 3 swin_transformer.py. Implement integer halving, and check the code's convention before changing it.
• Temporal window and token count. The window cannot exceed the number of temporal tokens. With 32 frames and stride 2 there are 16 tokens, so P=8 gives two temporal windows per clip and P=16 gives one §4.1, tab. 5. The code clamps window and shift when the input is smaller than the window swin_transformer.py.
• Padding for non-divisible sizes is not described. The code pads each axis to a multiple of its window size before partitioning swin_transformer.py.
• Inflation factor. The text's "multiply by 0.5" is the code's division by the temporal patch size. The two agree only for temporal patch size 2, the setting used here swin_transformer.py.
• Reproducibility. The paper gives no seeds, run-to-run variance or wall-clock time. Most ablation gaps are 0.1 to 0.7 points, each from a single run tab. 4 to tab. 9, so treat them as indicative.
• Multi-view cost. The best K400 and K600 numbers use 10 × 5 views at 2107 GFLOPs for Swin-L (384↑) tab. 1, tab. 2. The 4 × 3 setting, which Table 1 also reports, is the fairer comparison with other rows.
• Pre-training data ratio. The abstract claims about 20× less pre-training data and about 3× smaller models. The body gives only ImageNet-21K vs JFT-300M §1 and 200.0 M vs 647.5 M parameters §1. It never states the ratio.
• Input resolution. Table 1 and Table 2 report 384↑ results. The paper does not describe the training recipe for that resolution beyond the 224 settings in §4.1, and the code has no 384 config README.md.
• SSv2 learning rate. The text says the SSv2 batch size, learning rate and weight decay match K400 §4.1. The SSv2 config sets lr 3e-4 and weight decay 0.05, the same as the Swin-B K400 config swin_base_patch244_window1677_sthv2.py.
• Kinetics-600 config inherits K400. The K600 config extends the K400 22K config and changes the data paths and annotation files swin_base_patch244_window877_kinetics600_22k.py.

Lineage

• Builds on: Swin Transformer, whose shifted-window attention, hierarchy and patch merging this paper extends to space-time §1, §3.1.
• Compared against (not in the wiki, not used for content): ViViT, TimeSformer, MViT, VTN and the convolutional baselines I3D, SlowFast and X3D §2, tab. 1 to tab. 3.
• Superseded by: not stated.