Frozen CLIP Models are Efficient Video Learners

In plain words

Most video models are built by taking a picture model, then retraining the whole thing on videos. That is slow and needs a lot of memory, and it can wipe out what the picture model already knew. This paper takes a picture model trained on image-text pairs (CLIP) and leaves it untouched. It runs that model on each frame separately and reads the results with a small add-on network.

The add-on has one "question" slot that it fills by looking at the frame features of every frame, in several layers of the picture model. Extra small modules let it notice how things change from one frame to the next, so it can tell motion apart from a still image. Only the add-on is trained, so each training run is much cheaper. The result is competitive accuracy on standard video benchmarks.

TL;DR

EVL (Efficient Video Learning) trains a lightweight Transformer decoder on top of a frozen CLIP ViT, reading features from the last few blocks §3.1. A learned query token gathers frame features, and three local temporal modules add motion cues §3.2. The backbone never receives gradients §3.3. Author-reported: 82.9% top-1 on Kinetics-400 with ViT-B/16 and 8 frames, and 66.7% on Something-Something-v2 with ViT-L/14 and 32 frames tab. 1 tab. 9.

Problem

End-to-end finetuning of an image backbone on video needs a lot of computation and memory, because a video model processes several frames at once and is several times larger than its image counterpart §1. Finetuning can also destroy the features learned during image pretraining when the video data is small §1. Using the frozen image features directly, without any temporal model, gives subpar results §1. The paper asks whether a cheap trainable head on frozen CLIP features can close that gap §1.

Method

Overall structure §3.1

The CLIP backbone runs on every frame independently. The per-frame token features of several backbone blocks are stacked into a spatiotemporal volume, modulated with temporal information, and passed to a Transformer decoder. The decoder has M layers. Its query starts as a learned [CLS] token, and each layer takes the volume of one backbone block as keys and values. A linear layer on the final query gives the class scores §3.1 eq. 1-4.

Layer i of the decoder reads the features of backbone block N-M+i. Each decoder layer first applies the temporal modules to that block's features (eq. 1), then does cross-attention from the query and a residual MLP (eq. 2-3). The final prediction is a linear map of the last query (eq. 4) §3.1. Training is plain cross-entropy on the video label, and gradients stop at the backbone features, so no backbone weight is updated §3.1.

# EVL forward pass  [§3.1] [eq. 1-4]
# Input:  video (B, 3, T, H, W)
# Output: class logits (B, num_classes)

def evl_forward(video):
    # Run the frozen backbone on each frame; keep the outputs of the last
    # M blocks and each block's query and key projections  [§3.1]
    block_outs, block_qs, block_ks = backbone_last_blocks(video)
    # block_outs[i]: (B, T, 1 + HW, C); token 0 of each frame is [CLS]

    # Start from one learned query token per video  [§3.1] [eq. 4]
    query = learned_query.repeat(B, 1, 1)           # (B, 1, C)

    for i in range(M):
        # Temporally modulated volume for decoder layer i  [§3.2] [eq. 1]
        volume = temporal_modulate(
            block_outs[i], block_qs[i], block_ks[i], layer=i
        )                                           # (B, T, 1 + HW, C)
        volume = volume.flatten(1, 2)               # (B, T * (1 + HW), C)

        # Cross-attention from the query, then a residual MLP  [eq. 2-3]
        query = decoder_layer(query, volume, layer=i)

    # Linear head on the final query  [eq. 4]
    return classifier(query[:, 0])

Temporal modules §3.2

Three additive terms are computed for each decoder layer and added residually to the block features, Y = X + Y_conv + Y_pos + Y_attn §3.2.

Temporal convolution: a depthwise 1D convolution over time with kernel size 3 (offsets -1, 0, 1), one filter and one bias per channel §3.2 eq. 5.

Temporal position embedding: a learned table of T vectors of width C, one vector added to all tokens of frame t §3.2 eq. 6. Each decoder layer has its own table, since the module is indexed by layer i §3.2 eq. 1.

Temporal cross-attention: for each frame t, the attention between its patch tokens and those of frames t-1 and t+1 is computed with CLIP's own query and key projections §3.2 eq. 7. The attention maps are averaged over heads, then mapped to C channels with two learned spatial kernels, one for the previous frame and one for the next, indexed by the relative spatial offset §3.2 eq. 8.

# Temporal modulation of one block's features  [§3.2] [eq. 5-8]
# feats: (B, T, 1 + HW, C); queries, keys: CLIP projections of the same block

def temporal_modulate(feats, queries, keys, layer):
    # Local depthwise convolution over time, per channel  [eq. 5]
    conv_term = depthwise_conv1d_time(feats, kernel=3)
    # Learned temporal position vector of each frame, added to all tokens
    # of that frame  [eq. 6]
    pos_term = temporal_pos[layer]                # (T, C), broadcast
    # Attention maps between frame t and frames t-1 and t+1  [eq. 7]
    # Cross-frame attention, mapped through learned relative-offset kernels
    # [eq. 8]
    attn_term = cross_frame_attention(queries, keys, layer)
    return feats + conv_term + pos_term + attn_term

Cross-frame attention is computed only on patch tokens. The [CLS] row receives zero from this term model.py model.py.

Complexity §3.3

Per decoder block, the FLOPs of one query token attending to k key tokens with C channels and MLP factor alpha are written as a sum of four terms (eq. 9). For a ViT-B/16 backbone with alpha = 4, h = w = 14 and C = 768, the paper's ratio of decoder to encoder block FLOPs is about 1/6 (eq. 10). A full configuration (one decoder block per encoder output, all temporal modules on) is said to add within 20% of the backbone's FLOPs §3.3.

Because the backbone is not trained, back-propagation through it is skipped, which reduces memory and time per iteration §3.3 tab. 5.

Hyperparameters

Values for the ImageNet/CLIP setting are stated in the appendix app. A unless noted. Scripts in the repository are for Kinetics-400 only.

Results

Author-reported. Setup: Kinetics-400 (K400) and Something-Something-v2 (SSv2), plus UCF-101 and HMDB-51 in the appendix §4 app. B. Frame counts are given as frames per view x views.

Kinetics-400 against recent video models tab. 1

The paper's claim is competitive accuracy among regular video models. It says the models in the table with similar accuracy to EVL need substantially more computation §4.1.

CLIP-based methods on K400 tab. 2

The paper does not count ActionCLIP's text branch in its parameter numbers. The Efficient-Prompting parameter count is the authors' own estimate, because that paper's details are vague tab. 2.

Training and inference cost tab. 3 tab. 4 tab. 5

Inference, V100-32G, mixed precision tab. 3:

Training GPU-hours, as cited by the paper tab. 4: Uniformer-B 5,000 V100 hours (ImageNet-1k); TimeSformer 100 V100 hours (CLIP); ActionCLIP 480 RTX3090 hours (CLIP); EVL ViT-B/16 (8 frames) 60 V100 hours (CLIP). Some of these numbers are estimates or the authors' own reproduction (see Pitfalls).

Idealised step time on one V100-16G, 64 samples, 4 decoder layers tab. 5:

Something-Something-v2 tab. 9

The paper claims to be the first CLIP-based method to report SSv2 results. It also says a linear probe on CLIP ViT-B/16 reaches only about 20% there §4.1.

Pretraining quality and frozen vs finetuned §4.1 tab. 8 fig. 2

On CLIP backbones, the frozen EVL model with 8 frames (82.9%) beats TimeSformer-DividedST finetuned on CLIP (82.0%). On ImageNet-21k backbones the frozen EVL model (75.4%) trails finetuned TimeSformer (78.0%) tab. 7. The paper also reports that a finetuned backbone with the same decoder is a strong baseline, and that the training-time advantage of a frozen backbone grows with pretraining quality §4.1 fig. 2.

Ablations §4.2 §4.3

Decoder depth and feature layers on K400, ViT-B/16, 8 frames tab. 6a tab. 6b: depth 1/2/3/4/5 gives 81.1 / 82.1 / 82.6 / 82.9 / 83.0. Using the last four blocks gives 82.9; using the last block only gives 82.1; using blocks -7, -5, -3, -1 gives 82.0.

Feature shape tab. 6c: spatiotemporal features 82.9; spatial-only average 80.1; temporal-only with token 79.8; temporal-only average 75.8.

Temporal modules tab. 8a (K400 / SSv2): none 82.5 / 47.2; temporal conv 82.9 / 57.1; temporal position 82.5 / 58.5; cross-attention 82.6 / 59.5; all three 82.9 / 61.0. The paper says the modules add less than 0.5 points on K400 but +13.8 on SSv2 §4.3.

SSv2 decoder depth tab. 8b: 4 / 6 / 8 / 10 / 12 layers give 58.6 / 60.1 / 60.2 / 60.5 / 61.0.

Ensembles tab. 10 tab. 11 app. C

Fusing video-level scores with a weighted average (weight searched in steps of 0.1 on the validation set) §4.4. On K400, EVL ViT-B/16 (8 frames, 82.9%) with Uniformer-B (32 frames, 82.9%) gives 85.2%. The table's other pair, Uniformer-B with Swin-B, gives 84.7% tab. 10. On SSv2, EVL ViT-B (62.4%) with Uniformer-B (71.2%) gives 72.1%. The table's other pair, Uniformer-B with TimeSformer-L (62.4%), gives 71.4% tab. 11. The full K400 ensemble table is in app. C.

UCF-101 and HMDB-51 app. B

EVL ViT-L/14 reaches 98.5% (UCF-101) and 83.6% (HMDB-51); the 336px model reaches 98.6% and 83.2%, averaged over three splits. Among RGB-only methods listed, these are the highest UCF-101 and HMDB-51 numbers in the table. Methods that use optical flow or pose are shown greyed out and reach up to 98.6% and 84.3%.

Paper vs code

Official code: https://github.com/OpenGVLab/efficient-video-recognition (linked in the abstract). Commit b282e8b49e4ecd24f7357361719cf4fa5ab9c40d. The repository has no LICENSE file and the README states no licence, so code is cited by path and line only.

• Cross-attention scale. Eq. 7 has no scaling factor. The code divides the queries by sqrt(head dim) before the dot product model.py.
• Head handling. The paper says heads are omitted and then averaged. The code applies the softmax per head over keys, then averages the heads model.py. The paper's text does not fix the order.
• Normalisation. Eq. 2-3 have no LayerNorm. The code applies LayerNorm to the query, to the keys and values, and before the MLP vision_transformer.py vision_transformer.py. The MLP uses QuickGELU vision_transformer.py. The paper does not name the activation.
• Output head. Eq. 4 is a single linear map. The code applies LayerNorm, Dropout(0.5) and then the linear layer model.py model.py.
• Query init. Eq. 1-4 say the query is learned. The code uses a normal initialisation with std 0.02 model.py model.py.
• CLS token. Eq. 5 and 6 are written over all (t, h, w). The code applies the temporal convolution and position embedding to every token of every frame, including the [CLS] token model.py. The cross-attention term is zero for [CLS] model.py.
• Training clip. The paper says frames are evenly strided. Training in the code takes a random contiguous window with the given stride [code: video_dataset/dataset.py#L167-L168]. Testing uses evenly spaced temporal crops dataset.py, as the paper says.
• Scope of the repo. Only Kinetics-400 scripts are included (ViT-B/16 and ViT-L/14 at 8, 16 and 32 frames, decoder depth 4) . The SSv2, UCF-101, HMDB-51, ensemble and 336px training recipes are not in the repository. The README lists the 336px run with no script.
• Re-run accuracy. The README reports K400 top-1 of 82.8 (B/16, 8 frames), 83.7 (16), 84.3 (32), 86.3 (L/14, 8), 86.9 (L/14, 16) and 87.7 (L/14, 32). The paper reports 82.9, 83.6, 84.2, 86.3, 87.0 and 87.3. The README says these are re-runs and may differ by about 0.1%, and the 32-frame L/14 number differs by 0.4 points.

Pitfalls

• Internal GFLOPS mismatch. EVL ViT-B/16 at 8 frames is listed at 444 GFLOPS in tab. 1 and at 454 in tab. 3. Use one figure and state which table it came from.
• Frame counts differ in the speed comparison. The latency and throughput comparison tab. 3 pits EVL with 8 frames against Uniformer-B with 32 frames. Same-frame comparisons are not in that table.
• Training-time numbers are not like for like. Uniformer-B's time is the Kinetics-600 value from the Uniformer repository, halved. TimeSformer's time is the authors' own reproduction, which they say is a few times smaller than the figure in the TimeSformer paper. ActionCLIP's time is its 8-frame figure, doubled. Hardware differs (V100 vs RTX3090) tab. 4.
• Step-time numbers exclude data loading. The step times use fake data on one GPU tab. 5 §4.1.
• Full-finetune baselines are not the originals. TimeSformer runs use 25 epochs and a 100x lower backbone LR app. A. The paper says these settings gave better accuracy in its runs. Compare with the originals only with that in mind.
• Spatial and temporal split. Spatial information comes almost entirely from the frozen backbone; temporal information comes only from the decoder head §4.3. On motion-heavy SSv2, the temporal modules matter most (Ablations above), so a frozen-CLIP setup depends on them there.
• Cross-attention memory. The code builds a full token-by-token attention tensor for each frame pair and each layer model.py. This is an implementation cost, and the paper does not report it.
• Device is fixed to CUDA. The cross-attention module allocates its output on cuda model.py; CPU inference will need a change.
• Appendix experiments are not in the code. The UCF-101, HMDB-51 and ensemble results app. B app. C have no scripts in the repository.
• Ensemble weight is tuned on validation. The fusion weight is searched on the validation set §4.4; the paper does not say whether that set is held out from model selection.

Lineage

• Builds on: CLIP's visual encoder, used frozen §3.1; the Transformer decoder (Vaswani et al., cited in §1); depthwise temporal convolutions §3.2; the two-stream idea, as the motivation for the ensemble experiments §4.4.
• Compared against: CLIP-based ActionCLIP and Efficient-Prompting tab. 2; TimeSformer tab. 4 tab. 7; Uniformer-B tab. 1 tab. 10; and the other video models in tab. 1.
• Superseded by: not stated in this card.