UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer

In plain words

Recognising an action in a video means understanding what is in each frame and how things move between frames. Transformers handle long-range links well, but on video they spend a lot of effort comparing neighbouring pixels that carry almost the same information, and that makes them slow.

The authors do not train a video model from nothing. They start from an image transformer that has already learned from a large pile of pictures, and they add small video modules next to its existing blocks. One module looks at short stretches of time around each spot in the image. Another uses a single learned "query" that gathers a summary of the whole clip through cross-attention. The summaries from several depths are then combined into one answer.

The point is to get the strength of the image model, which is cheap to obtain because the weights already exist, together with video modules that are inexpensive compared with full space-time attention. The best reported numbers need many frames and a lot of compute, so the trade-off matters in practice.

TL;DR

• Keeps each pretrained ViT block and puts a temporal module in front of it (the local UniBlock), plus a query-based cross-attention module on selected deep layers (the global UniBlock) §3.1 §3.2.
• Chains the global query across those layers and mixes the last query with the final local class token §3.3.
• Author-reported: 90.0% top-1 on Kinetics-400 with ViT-L/14@336, 64 frames, 3 crops, 2 clips, after post-pretraining on the merged Kinetics-710 set tab. 1.
• The official code (Apache-2.0) differs from the paper in several places (activation, local module form, fusion details, learning rates, batch sizes). Read "Paper vs code" before implementing.

Problem

Global space-time attention over all tokens of a clip is expensive, and it does not exploit the fact that neighbouring frames are highly redundant §1. Models that apply global attention in shallow layers spend most of their compute on local detail §1.

A second obstacle is training. The UniFormer predecessor works well, but it needs a long image pretraining stage before video finetuning §1. Open image ViTs are already pretrained with rich supervision (supervised, contrastive or masked-image objectives) §1, so the paper asks whether their weights can carry the video model instead §1.

Assumptions (our reading of the code): the method needs an image ViT with a patch embedding, a position embedding and a class token. It is evaluated on sparse sampling of frames at 224 or 336 resolution §4.1 tab. 1.

Method

Overall framework. A 3D convolution with kernel 3×16×16 projects the clip to spatiotemporal tokens. Temporal stride is 2 and spatial stride is 16 §3. The ViT blocks are kept. In each block a local UniBlock adds a temporal relation module in front of the pretrained spatial attention and feed-forward layers §3.1 eq. 1–eq. 3. On the last layers a global UniBlock §3.2 reads the patch tokens through a depthwise spatiotemporal convolution, then lets one learnable query token attend to all tokens by cross-attention eq. 8–eq. 13. The queries are chained from one global block to the next, which is the sequential multi-stage fusion §3.3. The output mixes the last query with the final local class token §3.3.

Local UniBlock §3.1. The temporal module (LT_MHRA) relates each token only to tokens at the same spatial position in a short temporal window; the spatial module (GS_MHRA) is ordinary self-attention inside one frame. Both are pre-normalised residual branches, followed by the FFN eq. 1–eq. 3. The paper uses batch norm before the temporal module and layer norm before the spatial attention and the FFN §3.1. The spatial attention and FFN come from the pretrained ViT §3.1. The temporal affinity is a learnable weight per temporal offset (window size t) eq. 6; the spatial affinity is the usual softmax attention eq. 7. The authors drop the dynamic position encoding of UniFormer in this block, because the ViT position embedding already encodes location §3.1.

Global UniBlock §3.2. A depthwise 3D convolution (DPE) is added to the tokens, then a cross-attention module summarises all tokens into the query eq. 8 eq. 9. The query is a learnable vector, zero-initialised §3.2. The attention weights are a softmax between the query and every token eq. 13, and the aggregated values are concatenated over heads and projected with a learnable matrix U eq. 11 eq. 12. A feed-forward block follows eq. 10. The authors note that this costs O(L) in the number of tokens, against O(L²) for global self-attention §3.2.

Multi-stage fusion §3.3. The paper compares four ways to combine the per-stage queries: sequential, parallel concatenation, hierarchical key/value and hierarchical query. Sequential is chosen for its efficacy and efficiency §3.3; it ties with the best variant on the ablation tab. 9.

Final output §3.3. The last query token F and the final class token F^C of the local stream are blended as Z = αF + (1−α)F^C, with α = sigmoid of a learnable parameter §3.3.

The classification head is not described in the paper. The code uses layer norm, dropout 0.5 and a linear layer. See "Paper vs code".

Equations

Equation numbers follow the compiled paper, counted from the LaTeX source. Check them against the PDF if a citation looks off.

Local UniBlock §3.1:

X^T = \mathrm{LT\_MHRA}(\mathrm{Norm}(X^{in})) + X^{in} eq. 1

X^S = \mathrm{GS\_MHRA}(\mathrm{Norm}(X^T)) + X^T eq. 2

X^L = \mathrm{FFN}(\mathrm{Norm}(X^S)) + X^S eq. 3

Multi-head relation aggregator §3.1:

R_n(X) = A_n V_n(X) eq. 4

\mathrm{MHRA}(X) = \mathrm{Concat}(R_1(X); \dots; R_N(X))\, U eq. 5

Local temporal affinity §3.1:

A^{LT}_n(X_i, X_j) = a_n^{\,i-j}, \quad j \in \Omega^{t\times1\times1}_i eq. 6

Global spatial affinity inside one frame §3.1:

A^{GS}_n(X_i, X_j) = \frac{\exp\{Q_n(X_i)^{T} K_n(X_j)\}} {\sum_{j'\in\Omega_{1\times H\times W}} \exp\{Q_n(X_i)^{T} K_n(X_{j'})\}} eq. 7

Global UniBlock §3.2:

X^C = \mathrm{DPE}(X^L) + X^L eq. 8

X^{ST} = \mathrm{C\_MHRA}(\mathrm{Norm}(q), \mathrm{Norm}(X^C)) eq. 9

X^G = \mathrm{FFN}(\mathrm{Norm}(X^{ST})) + X^{ST} eq. 10

R^C_n(q, X) = A^C_n(q, X)\, V_n(X) eq. 11

\mathrm{C\_MHRA}(q, X) = \mathrm{Concat}(R^C_1(q, X); \dots; R^C_N(q, X))\, U eq. 12

A^C_n(q, X_j) = \frac{\exp\{Q_n(q)^{T} K_n(X_j)\}} {\sum_{j'\in\Omega_{T\times H\times W}} \exp\{Q_n(q)^{T} K_n(X_{j'})\}} eq. 13

Training-cost saving quoted for K710 post-pretraining app. A:

1 - \frac{0.66\times55 + 1.14\times5}{1.14\times55} \approx 0.33 eq. 14

Pseudocode

The paper has no algorithm box. This is our reconstruction of the forward pass from §3 and the equations above. Names are ours.

# UniFormerV2 forward pass, reconstructed from [§3] and [eq. 1]-[eq. 13]
# Input:  video (B, 3, T, H, W)
# Output: logits (B, num_classes)

def uniformerv2(video):
    # Project to tokens: 3x16x16 kernel, temporal stride 2  [§3]
    tokens = patch_embed(video)               # (B, T/2, H/16, W/16, d)
    # Add one [CLS] per frame and the pretrained position embedding  [§3]
    tokens = add_cls_and_position(tokens)     # (B*T/2, 1 + HW, d)
    # Learnable video query, zero-initialised  [§3.2]
    query = zeros(1, d)
    for layer, vit_block in enumerate(vit_blocks):
        # Local UniBlock: temporal module before the spatial block  [eq. 1]
        tokens = tokens + local_temporal(batch_norm(tokens))
        # Pretrained spatial self-attention  [eq. 2]
        tokens = tokens + spatial_attention(layer_norm(tokens))
        # Pretrained feed-forward  [eq. 3]
        tokens = tokens + ffn(layer_norm(tokens))
        if layer in global_layers:
            # Depthwise spatiotemporal conv, residual  [eq. 8]
            context = tokens + dpe[layer](tokens)
            # Query attends to every token; code keys include [CLS]  [eq. 9]
            query = cross_attention(layer_norm(query), layer_norm(context))
            # Feed-forward on the query, residual  [eq. 10]
            query = query + ffn(layer_norm(query))
    # Final local class token F^C  [§3.3]
    # (code averages it over frames; see Paper vs code)
    cls_final = tokens[:, 0]                  # (B, d)
    # Blend F and F^C with a learnable sigmoid weight  [§3.3]
    alpha = sigmoid(learnable_weight)         # scalar in text, (d,) in code
    fused = alpha * query + (1 - alpha) * cls_final
    return linear_head(dropout(layer_norm(fused)))

Hyperparameters

The table below is the paper's hyperparameter table tab. 11, copied in full. Columns are the training groups. "Base" and "Large" are read as ViT-B/16 and ViT-L/14 (our reading of the row labels). Several values disagree with the official scripts. See "Paper vs code".

tab. 11. Constant columns are listed once in the source table. Repeated sampling and "-" are as printed.

Values from the text and appendix that are not in tab. 11:

Architecture numbers that the code sets but the paper does not state:

Results

Author-reported, on the benchmarks the paper uses. Headline comparison, one row per dataset. The full tables are in the paper (tab. 1–tab. 7). Metric is top-1 for every row, as labelled in the tables.

Ablation: pretrained image ViTs tab. 8 (model size not stated in the table; §4.2):

Ablation: components and training scripts tab. 9 (fine-tuned from CLIP-ViT-B/16 by default; testing 8×4×3 for Kinetics and 16×1×3 for SSV2 §4.2):

The paper reports about a 33% training-cost saving from K710 pretraining, with consistent gains tab. 9 app. A eq. 14.

Further ablations (appendix) tab. 12–tab. 15:

Parameters and FLOPs for tab. 15: 86/114/86/86/86/128/105 M and 422/555/539/422/422/423/511 G, in the same order.

Dataset sizes tab. 10. Kinetics-710 is the merge of K400/600/700 training sets with duplicates removed by YouTube id and test leakage removed. It has 658,340 training and 66,803 validation videos, 710 classes, average length 10 s §4.1 tab. 10 app. F. The other sizes are listed in tab. 10.

Paper vs code

Code: official repository, found through the abstract's link (github.com/OpenGVLab/UniFormerV2) at commit 722a43440fc5b9662cc2a8f23b86caa205e45ebc, Apache-2.0 LICENSE. Main files: slowfast/models/uniformerv2_model.py and slowfast/models/uniformerv2.py. Configs and run scripts are in exp/.

1. Patch size of L/14 backbones. The text fixes the spatial stride at 16 with 3×16×16 patches §3. The L/14 constructors use patch_size=14 uniformerv2_model.py uniformerv2_model.py. The L/14 results in the tables use the 14-pixel model.

2. Activation. The text says the FFN is two linear layers separated by GeLU §3.1. The code uses QuickGELU, x·sigmoid(1.702x), in the ViT FFN and in the global FFN uniformerv2_model.py uniformerv2_model.py uniformerv2_model.py.

3. Local temporal module. The paper defines a learnable affinity per head and per temporal offset, with a fusion matrix U §3.1 eq. 6. The code instead uses batch norm, a 1×1×1 reduction by 1.5, a depthwise temporal convolution with kernel 3, and a 1×1×1 expansion that is zero-initialised uniformerv2_model.py. There are no explicit heads.

4. Second local module. Equations 1–3 contain one temporal module per block eq. 1. The ablation text adds a second one after the spatial attention §4.2. The code does this when double_lmhra is set uniformerv2_model.py config.yaml.

5. Query residual. Equation 9 has no residual connection on the query eq. 9. The code adds one uniformerv2_model.py.

6. Keys and values in the global block. The paper says the keys are "all spatiotemporal tokens" §3.2. The code flattens the per-frame sets, so each frame's [CLS] token is included as a key uniformerv2_model.py. The paper does not say whether the [CLS] tokens are meant to be included.

7. Fusion weight. The paper uses a scalar α that multiplies F §3.3. The code uses a per-channel vector, and the sigmoid of it multiplies F^C, the other term uniformerv2_model.py uniformerv2_model.py. Because the code's sigmoid weight sits on F^C, its value equals 1 minus the paper's α. The two forms are the same function.

8. Final class token. The paper uses the final local class token F^C §3.3. The code averages it over the frames of each clip uniformerv2_model.py.

9. Classifier head. The paper does not describe the head. The code uses layer norm, dropout 0.5 and a linear layer uniformerv2_model.py.

10. Frozen mode. The tables have "(frozen)" rows tab. 1. The code freezes the patch embedding, position embedding, pre-norm and the ViT blocks, and it skips the fusion weight uniformerv2.py uniformerv2_model.py. The paper does not define frozen mode.

11. Frozen learning rate. The appendix says 4e-4, no warmup app. A. The frozen Kinetics-400 run from CLIP uses 8e-4 run.sh. The run after K710 uses 5e-5 run.sh.

12. K710 base learning rate. tab. 11 gives 2e-5 for the Base run on K710. The B/16 run script uses 1e-5 run.sh. The Large run uses 1e-5, which matches the table run.sh.

13. Something-Something Large epochs. tab. 11 gives 15 Large epochs, and tab. 4 reports 15 finetune epochs for the L/14 row. The SSV2 L/14 script runs 22 run.sh, and so does the SSV1 L/14 script run.sh.

14. Batch sizes. The table gives 512 for K710 and MiT, 128 for SSV1/SSV2 and 64 for ANet/HACS tab. 11. The scripts pass other values: K710 B/16 uses 256 run.sh, MiT B/16 uses 256 run.sh, SSV2 L/14 uses 32 run.sh, and SSV2 B/16 uses 64 run.sh. The scripts pass different numbers of shards and GPUs to the same batch variable, so the global batch is not clear from the repository.

15. Input resolution. The text says 224 for all datasets §4.1. Several headline rows use 336 tab. 1, and the code has 336 constructors uniformerv2_model.py.

16. Pretrained weights. The code loads ViT weights from a hard-coded internal path uniformerv2_model.py. That path is not in the repository. Post-pretrained K710 checkpoints are linked in the model zoo MODEL_ZOO.md. We did not check the license of those checkpoints and did not use them.

17. Inflating 2D weights. When a 2D weight is loaded into a 3D layer, the code puts the 2D kernel at the centre temporal index and fills the rest with zeros uniformerv2_model.py uniformerv2_model.py. The paper does not say how image weights are mapped to 3D.

18. Label mapping. The code expects a label map at ./data_list/k710/label_mixto{N}.json uniformerv2.py. The data_list directory is not in the repository.

The dataset loaders and augmentation code were not reviewed.

Pitfalls

• Cost of the headline number. The 90.0 K400 result uses 64 frames, 3 crops and 2 clips, at 75.3 TFLOPs tab. 1. The 8-frame L/14 model with the 8×3×4 setting reaches 88.8 at 8.0 TFLOPs tab. 1. Choose the setting before comparing numbers.
• Settings in the MiT table. tab. 3 labels the 47.8 row as CLIP-400M only. The additional-results table gives the same numbers for CLIP-400M+K710+K400 tab. 17. Check which pretraining you use before comparing.
• Two readings of the Kinetics "5 epochs". tab. 11 lists 5 epochs for K400/600/700. That applies to finetuning after K710. The individual Kinetics recipe is 55 epochs app. A. The two are easy to mix up.
• Evaluation protocol. Top-1 depends on the number of temporal and spatial views §4.2 tab. 1. Use the same protocol as the table row you copy.
• Class merge for K710. When you load a K710 classifier into a K400/600/700 model, you must map the rows by label. The merged label list has 8 unique K400 labels and 2 unique K600 labels, and the classifier weights have to be re-indexed app. F.
• Zero initialisation. Several layers are zero-initialised, including the query, the output projections and the last layers of several branches app. A. Training is said to be unstable without this. Keep it.
• Weight decay. The official config switches weight decay off for 1-D parameters config.yaml. The paper does not say so.
• Patch projection. The 3D patch projection is initialised by inflating the 2D ViT weights, and the position embedding is reused as is (see Paper vs code, item 17). If you adapt an L/14 ViT, check the patch size against the paper's text and the code (item 1).
• The 33% saving. eq. 14 uses 55 and 5 epochs. It is a ratio of training data passes, not measured time.
• Equation numbers. They were counted from the LaTeX source, not from the rendered PDF. Verify against the PDF.
• Unverified claims. The "first model above 90% on K400" claim is the authors' own, "to our best knowledge" §1. We did not check it.
• Not state-of-the-art on SSV2. The table shows a higher SSV2 number for VideoMAE-L, which is trained without image pretraining tab. 4. The paper claims state-of-the-art on SSV1 §4.1, not on SSV2.

Lineage

• Builds on: UniFormer, the predecessor whose relation aggregator and local/global design it reuses. Cited in the paper as [uniformer]. Not compiled here.
• Uses: a pretrained ViT, CLIP-ViT by default §4.1. Other image pretrainings are compared in tab. 8.
• Compared against: TimeSformer, ViViT, MTV, EVL, X-CLIP, CoCa, VideoMAE, Swin-based video models and the MTV/ViViT family tab. 1–tab. 7.
• Superseded by: not known from this wiki.