Multiview Transformers for Video Recognition

In plain words

Recognising an action in a video needs two kinds of information at once: quick, small movements, such as fingers moving, and the slower context of the scene, such as a kitchen. A transformer that cuts the video into one fixed chunk size sees only one of these well.

The idea here is to look at the same video several times, each time with chunks of a different length. Short chunks keep fine motion, and long chunks keep the overall scene. Each look gets its own transformer. The coarse looks get a smaller transformer, and the detailed looks get a bigger one. At a few points the looks pass information to each other, and at the end a final transformer combines them into one answer.

The authors report that this beats a single-look transformer at a similar or lower cost, and that it reaches state-of-the-art results on six standard video benchmarks (author-reported). The top scores also depend on large pretraining datasets, and the repository does not ship the models trained on them.

TL;DR

MTV cuts each video into tubelets of several sizes, one size per "view". Each view has its own ViT encoder, and the encoders exchange information through cross-view attention in a few layers. A global encoder combines the per-view class tokens into the label. The authors report 81.8% top-1 on Kinetics-400 for MTV-B at 4x3 views, and state-of-the-art results on six datasets (author-reported), which improve further with large-scale pretraining. Read "Paper vs code" before you reproduce the global encoder, because the released code differs from the paper's description there.

Problem

Video understanding needs reasoning at several temporal resolutions, from short fine motions to events that last longer §1. The introduction notes that transformer video models do not model these resolutions explicitly §1. Pyramid designs and two-stream designs such as SlowFast lose spatio-temporal detail, through pooling or by sub-sampling frames §1. Joint self-attention over the tokens of all views is quadratic in the token count, which the authors judge infeasible for video §3.3. The conclusion also flags reliance on large image and video pretraining as a limitation §5.

Method

The model builds several views of one input video, encodes each view with its own transformer, and fuses them fig. 1. Input video: V in R^{T x H x W x C} §3.1.

1. Views and tokenisation §3.2. A view is a set of fixed-size tubelets. Larger tubelets give fewer tokens. Each view uses a 3D convolution as its tubelet embedding, which the authors chose because ViViT reported it as the best option §3.2. Views can have different hidden sizes d^(i) §3.2. In the naming convention, B/2+S/4+Ti/8 means three views: Base with 16x16x2 tubelets, Small with 16x16x4, and Tiny with 16x16x8 §4.1.
2. Class token and positions per view §3.1 eq. 1. Each view's tokens get a learnable class token and a positional embedding.
3. Attention within a temporal index §3.3.1. Self-attention inside each view only mixes tokens from the same temporal index, following ViViT's factorised encoder §3.3.1.
4. Per-view layers §3.1 eq. 2 eq. 3. Each layer is multi-head self-attention and an MLP with GELU, each with layer normalisation before it and a residual connection around it. Each view uses the ViT size assigned to it tab. 5.
5. Cross-view fusion at selected layers §3.3.2 fig. 2. The paper describes three ways to fuse adjacent views, ordered by token count. (a) Cross-view attention (CVA): the coarser view's tokens form the queries, and the finer view's tokens form the keys and values, through a projection; a residual connection is added, and the output projection is zero-initialised eq. 4 eq. 5. (b) Bottleneck tokens: a few learned tokens per view are concatenated to the view's tokens, and then passed on to the next view. (c) MLP fusion: the other view's tokens are concatenated along the hidden dimension inside the MLP block. The authors pick CVA for their experiments because it performed best §4.2. Which layers fuse is a design choice §3.3.2.
6. Global encoder §3.3.3 §3.1. Each view's class token goes into another transformer encoder that aggregates across views. A linear layer on the resulting class token gives the C class scores.

Equations as written in the paper:

\mathbf{z}^{0} = [\mathbf{z}_{cls}, \mathbf{E}x_1, \ldots, \mathbf{E}x_N] + \mathbf{p} eq. 1

\mathbf{y}^{\ell} = \text{MSA}\left(\text{LN}\left(\mathbf{z}^{\ell - 1}\right)\right) + \mathbf{z}^{\ell - 1} eq. 2

\mathbf{z}^{\ell} = \text{MLP}\left(\text{LN}\left(\mathbf{y}^{\ell}\right)\right) + \mathbf{y}^{\ell} eq. 3

\mathbf{z}^{(i)} = \text{CVA}\left(\mathbf{z}^{(i)}, \mathbf{W}^{\text{proj}} \mathbf{z}^{(i+1)}\right) eq. 4

\text{CVA}(\mathbf{x}, \mathbf{y}) = \text{Softmax}\left(\frac{\mathbf{W}^{Q}\mathbf{x}\,\mathbf{W}^{K}\mathbf{y}^{\top}}{\sqrt{d_k}}\right) \mathbf{W}^{V}\mathbf{y} eq. 5

Forward pass as the paper describes it. The code differs in its global encoder and in one detail of the fusion step; see "Paper vs code".

# Multiview forward pass, as the paper describes it  [§3.2] [§3.3]
# Input:  video (B, T, H, W, C); one tubelet length per view
# Output: logits (B, num_classes)

def mtv_forward(video, tubelet_lengths, fusion_layers):
    # One 3D-conv tubelet embedding per view  [§3.2]
    tokens = [tubelet_embed(video, t_len) for t_len in tubelet_lengths]
    # Each view gets a class token and position embedding  [§3.1] [eq. 1]
    tokens = [add_class_and_position(view) for view in tokens]
    for layer in range(num_layers):
        # Fusion layers exchange information between views first  [§3.3.2]
        if layer in fusion_layers:
            tokens = cross_view_attention(tokens)
        # Each view runs its own attention and MLP with residuals  [§3.1]
        for view in range(len(tokens)):
            tokens[view] = transformer_block(tokens[view], layer)
    # Keep each view's class token for the global encoder  [§3.3.3]
    class_tokens = [view[:, 0] for view in tokens]
    # Merging views with different widths is not described  [§3.3.3]
    merged = merge_views(class_tokens)                   # (B, V, d_g)
    # Global encoder with its own class token, then a linear head  [§3.3.3]
    out = global_encoder(prepend_class(merged))
    return linear_head(out[:, 0])                        # (B, num_classes)

# Cross-view attention between adjacent views  [§3.3.2] [eq. 4] [eq. 5]
# Input:  tokens[v] of shape (B, N_v, d_v), views ordered by token count
# Output: tokens[v] with information from the next finer view added

def cross_view_attention(tokens):
    # Walk adjacent pairs from the finest view towards the coarsest  [§3.3.2]
    # (the order is not stated in the paper; the code uses this order)
    for coarse in reversed(range(len(tokens) - 1)):
        fine = coarse + 1
        # Queries from the coarse view, keys and values from the fine view
        # with a projection to a common width  [§3.3.2] [eq. 4]
        keys_values = project(tokens[fine])               # W^proj
        # Attention output, added back with a residual  [§3.3.2] [eq. 5]
        update = attention(tokens[coarse], keys_values)
        # The output projection starts at zero, so this adds nothing at init
        # [§3.3.2]
        tokens[coarse] = tokens[coarse] + update
    return tokens

Hyperparameters

Training settings from the paper, as stated. The paper gives no weight decay, no dropout value and no gradient clipping. These gaps are discussed in "Paper vs code".

• Optimiser: synchronous SGD with momentum 0.9; cosine learning-rate schedule with linear warm-up of 2.5 epochs tab. 4 §4.1.
• Input: 32 frames with temporal stride 2, 224x224 pixels, for training and inference §4.1.
• Crops: test-time averaging over spatial and temporal crops; the number of crops is listed with each result §4.1. Kinetics uses 4 temporal and 3 spatial crops, which the paper writes as 4x3 §4.3. Epic-Kitchens uses 4x1 §4.3.
• Augmentation: random crop probability 1.0, scale jitter probability 1.0 with scale between 0.9 and 1.33 tab. 4. The rest varies by dataset, as in the table below.
• Initialisation: each view starts from a ViT of the matching size pretrained on a large image dataset. The tubelet embedding and the positional embedding are adapted to the view's tubelet shape, using the ViViT technique. The global encoder is randomly initialised §4.1.

Per-dataset settings, reproduced from the paper tab. 4. "--" means the method was not used.

Model sizes, reproduced from the paper tab. 5:

Other settings that the text states:

• The global encoder is Base-sized with 8 attention heads instead of 12. The reason given is divisibility of the hidden size by the head count §4.1 tab. 5.
• Ablations §4.2: Kinetics-400, largest backbone Base, learning rate 0.1, 30 epochs, no extra regularisation, 4x3 crops.
• Fusion layers: the paper sets {11, 23} for the L-based model and {11, 23, 31} for the H-based model §4.2. The paper does not state the fusion layers for MTV-B.
• Epic-Kitchens uses a single network with two heads, for noun and verb §4.1.
• Something-Something v2: no horizontal or vertical flips §4.3.
• Not stated: weight decay, dropout, gradient clipping, warm-up in steps, number of random seeds, and the bottleneck count B.

Results

Author-reported. Setup: six benchmarks (Kinetics-400, -600 and -700, Moments in Time, Epic-Kitchens-100, Something-Something v2) §4.1 §4.3. MTV-B means B/2+S/4+Ti/8, MTV-L means L/2+B/4+S/8+Ti/16, and MTV-H means H/2+B/4+S/8+Ti/16 §4.3. Apart from Kinetics, models start from a Kinetics-400 checkpoint and are fine-tuned §4.3. Pretraining is ImageNet-21K unless stated in parentheses tab. 2 caption. WTS is the Weak Textual Supervision dataset of Stroud et al. §4.3; JFT is JFT-300M §4.1.

Ablations (Kinetics-400, 4x3 crops) tab. 1

Table 1 is one table with six panels, (a) to (f). Model names as in the paper. GFLOPs and MParams are as reported.

Panel (f) is for B+S+Ti in the paper's text; the layer indices are as written in the table, and the paper does not say whether they start at 0 or 1 §4.2.

Headline comparison tab. 2

Accuracy in percent. Views and TFLOPs are given where the paper's table gives them. The table's TFLOPs are summed over all views tab. 2 caption.

Epic-Kitchens-100 action, verb and noun top-1 tab. 2. Results are averaged over 4x1 crops §4.3.

Trade-offs §4.3 fig. 3. On Kinetics-400, with four-frame tubelets for the smallest view, MTV is more accurate than ViViT-FE at every complexity level while using fewer FLOPs. The authors also report that B/2 (12 layers) beats ViViT-L/2 (24 layers) at lower FLOPs. Throughput was measured on Cloud TPU-v4 fig. 3 §A.1. In the appendix, the unfactorised encoder variant gains 0.4 points over ViViT-H at a reported 172% higher throughput §A.1.

Spatial resolution of tubelets, Kinetics-400, B/4+Ti/16 tab. 3, §A.2:

Paper vs code

Code: google-research/scenic, path scenic/projects/mtv, at commit 77d19427a71d9fb8318b656d9710d4dfd7af0ba7, Apache-2.0. The official abstract links this repository. The paths in citations are relative to the repository root.

1. Global encoder input (main difference). Paper: the global encoder reads each view's class token and aggregates across views §3.3.3. Code: every shipped config sets merge_axis: 'channel' [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L97]. Each view's class token is tiled along time to the longest temporal length, and the views are concatenated along the channel axis [code: scenic/projects/mtv/model.py#L527-L541]. A class token is prepended, and the encoder runs over temporal positions, not over views [code: scenic/projects/mtv/model.py#L543-L584]. The encoder width is the sum of the view widths, which is 1344 for Ti+S+B [code: scenic/projects/mtv/config_utils.py#L22]. The configured hidden_size of 768 is read and discarded in this path model.py. For K400 MTV-B that is 17 tokens: 16 time positions plus one class token. The paper's "Base-sized, 8 heads" description holds for the head count, MLP size and depth §4.1 tab. 5, but not for the width.
2. Fusion layer indices. Paper: fusion "at the 5th and 11th layers" §4.2. Code: fusion_layers: (5, 11) [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L90], and the loop counts layers from 0 model.py. So the code's layers are the 6th and 12th.
3. Normalisation inside one fusion block. Paper: adjacent pairs are fused "sequentially" §3.3.2. Code: all views are layer-normalised once, before the pair loop model.py. Keys and values for each pair come from those pre-block tensors, not from the residual stream that the previous pair just updated [code: scenic/projects/mtv/model.py#L155-L179]. Consequence: with three or more views, the finest view does not reach the coarsest one within one block.
4. Projection and heads in CVA. Paper: W^proj projects keys and values to a common width eq. 4; the head count is not stated. Code: there is no separate projection; the attention's width is the query view's width, and heads come from the query view when use_query_config is set, as in every shipped config model.py [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L89].
5. Fusion variants. Paper: CVA, bottleneck and MLP fusion §3.3.2. Code: CVA and bottleneck tokens are implemented [code: scenic/projects/mtv/model.py#L268-L308], but MLP fusion is not. Only the cross_view_attention and bottleneck types are accepted [code: scenic/projects/mtv/model.py#L380-L388]. No shipped config sets a bottleneck count, so the bottleneck variant has no config in this repository.
6. Temporal grouping. Paper: self-attention among tokens of the same temporal index §3.3.1. Code: every view is reshaped to the smallest temporal count (4 for K400 MTV-B), so the finer views' tokens are grouped by coarse index model.py [code: scenic/projects/mtv/model.py#L656-L666]. Each group therefore holds several time slices of the finer views.
7. Schedule and pretraining values that differ from Table 4. The code's values, with the paper's value in brackets tab. 4:
   • Kinetics-600 MTV-B base learning rate 0.2 [0.1] [code: scenic/projects/mtv/configs/kinetics/k600_mtv_b2_cva.py#L154].
   • Kinetics-700 MTV-B stochastic depth 0.0 [0.1] [code: scenic/projects/mtv/configs/kinetics/k700_mtv_b2_cva.py#L88].
   • SSv2 batch size 256 [512], epochs 60 [100], base learning rate 0.2 [0.5], stochastic depth 0.1 [0.3], colour jitter 0.8 ["--"] [code: scenic/projects/mtv/configs/ssv2/ssv2_mtv_b2_cva.py#L117] [#L116] [#L135] [#L89] [#L71].
   • MiT epochs 20 [30], base learning rate 0.2 [0.1], mixup 0.2 ["--"] [code: scenic/projects/mtv/configs/mit/mit_mtv_l2_cva.py#L115] [#L134] [#L78-L79].
   • Epic-Kitchens: no mixup key [0.1]; colour jitter 0.8 ["--"] [code: scenic/projects/mtv/configs/epic_kitchens/epic_mtv_b2_cva.py#L74].
8. Frames and stride. Paper: all models use 32 frames with stride 2 §4.1. Code: Kinetics configs use 32 frames, stride 2 [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L51-L52]. Epic-Kitchens uses 64 frames, stride 1 [code: scenic/projects/mtv/configs/epic_kitchens/epic_mtv_b2_cva.py#L47-L48]. SSv2 uses 48 frames, stride 1 [code: scenic/projects/mtv/configs/ssv2/ssv2_mtv_b2_cva.py#L48-L49]. MiT uses 32 frames, stride 3 [code: scenic/projects/mtv/configs/mit/mit_mtv_l2_cva.py#L47-L48].
9. Values the paper does not state. Code sets these, and the paper is silent: weight decay 0 k400_mtv_b2_cva.py; gradient-norm clipping at 1 [#L109]; dropout 0 [#L82-L83]; and a stochastic-depth rate that grows linearly with the layer index, divided by the layer count minus one model.py. The momentum default is 0.9 optax.py. The 3D-conv kernel uses a central-frame initialiser, and positions are tiled when resized [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L100] [#L143].
10. Pretrained weights. Paper: image pretraining, and JFT-300M or WTS for the top results §4.1 §4.3. Code: the K400 config's ViT checkpoint paths are placeholders [code: scenic/projects/mtv/configs/kinetics/k400_mtv_b2_cva.py#L131-L135]. The README's model zoo lists checkpoints for K400, K600, K700, Epic-Kitchens and MiT. It lists none for JFT or WTS.
11. Numbers in the README. MTV-B on K600 is 83.8 in the README against 83.6 in the paper. MTV-L on MiT is 42.5 at 1x3 in the README against 41.7 in the paper. The README's Epic-Kitchens row fine-tuned from K700 (48.0) does not appear in the paper.
12. Huge model. Paper: MTV-H is reported, with fusion {11, 23, 31} §4.2. Code: no H config ships. The config directory contains B and L variants only.

Pitfalls

• Global encoder. Follow the code if you want to reproduce the released checkpoints; follow the paper if you want the described design. These are different networks (see item 1 above). The paper does not say how views of different widths should be merged, so the paper alone does not determine the global encoder.
• Index base. Fusion layers are 0-based in the code. A reader who takes the paper's wording literally, 1-based, will place fusion one layer later.
• Layer choice for large models. The indices {11, 23} and {11, 23, 31} for the L and H models are extrapolated from the 12-layer ablation. They were not ablated for those sizes §4.2. For MTV-B the paper gives no fusion layers, and the code uses (5, 11), which is the best setting in panel (f).
• Fusion costs more than it looks. A fusion layer runs cross-attention, then the view's usual self-attention, then the MLP, all at the same index [code: scenic/projects/mtv/model.py#L321-L329]. So a fusion layer adds one attention sub-block on top of the view's usual layer. The FLOPs in table 1 include this.
• Zero init at start. The CVA output projection starts at zero §3.3.2 [code: scenic/projects/mtv/model.py#L173], so at initialisation the cross-view term adds nothing to the residual stream.
• Stochastic depth. The paper gives only the top rate tab. 4. The code ramps the rate linearly with the layer index for each view, so the top layer gets the full rate model.py.
• Compute for the reported numbers. Accuracy uses 4x3 clips per video, so 12 forward passes. The TFLOPs in the table are summed over views [tab. 2 caption], and the comparators in the same table use other view counts tab. 2.
• Throughput is hardware-specific. It was measured on Cloud TPU-v4 fig. 3 §A.1.
• Large-data results are not reproducible here. The JFT and WTS results need pretraining weights the repository does not provide (see item 10).
• No variance reported. The paper gives single accuracy values in its tables, with no seed count or spread §4.2 tab. 2. Several ablation gaps are 0.1 to 0.3 points, for example 81.00 against 81.08 in panel (f) tab. 1(f). Without variance, these gaps cannot be separated from run-to-run noise.
• Limitations stated by the authors. Large gains beyond Kinetics are still open, and the models rely on pretrained image or video weights §5.

Lineage

• Builds on: Vision Transformer (Dosovitskiy et al., ICLR 2021), for patch tokens and initialisation §3.1 §4.1. The arXiv id is not in this wiki.
• Builds on: ViViT (Arnab et al., ICCV 2021), for the 3D-conv tubelet embedding, the factorised encoder, and the training recipe §3.2 §3.3.1 §4.1. The arXiv id is not in this wiki.
• Related designs: SlowFast (Feichtenhofer et al., ICCV 2019), a two-stream CNN whose transformer version the authors implement as a baseline §4.2. CrossViT (Chen et al., ICCV 2021) fuses streams with cross-attention on images §3.3.2. Bottleneck tokens also appear in Perceiver and in Nagrani et al. §3.3.2.
• Compared against: ViViT-FE and ViViT-L, MoViNet-A6, MFormer, TimeSformer, MViT, VATT, CoVeR and Florence tab. 2.
• Named as future work: Multiscale ViT and Swin, for combining with spatial-pyramid designs §5.