Expanding Language-Image Pretrained Models for General Video Recognition

In plain words

Video models usually learn a fixed list of action names from labelled clips. A new action name then has nowhere to go. Image models trained on web pairs of pictures and captions can name things they were never trained on, but nobody has an equally large pile of video with captions, and training one from scratch is very expensive.

This paper takes the image models as they are. It adds a small module that lets the frames of a clip talk to each other, so the model sees motion and change over time. It also lets the text side adjust its description of each action based on what the video actually shows, rather than using one fixed wording for every clip. The authors say the result is competitive on standard video benchmarks, and the gains they report are largest when labelled examples are very few.

TL;DR

X-CLIP reuses a pretrained CLIP image encoder for video. Each frame gets a message token that is exchanged across frames inside every transformer block, and a small transformer merges the frame features into one video vector. Class names are turned into video-specific text prompts before matching by cosine similarity. Author-reported: 87.1% top-1 on Kinetics-400 with ViT-L/14 at 8 frames, with gains in zero-shot and few-shot settings.

Problem

Closed-set video recognition covers only the classes seen in training, and video-text pretraining at web scale is costly §1. The paper asks how to move language-image pretraining to video without that cost §1. It names two issues: how to use temporal information in a frame-based encoder, and how to get a discriminative text representation for each video §1.

Method

The model has a video encoder and a text encoder. The video encoder is a cross-frame communication transformer (CCT) applied to each frame, followed by a multi-frame integration transformer (MIT) over the frame features §3.2. The text encoder is the pretrained CLIP text encoder, expanded with a video-specific prompt generator §3.3. A class-name text and the video are embedded separately; the prompt generator then produces an instance-level text embedding for each video, and the score is the cosine similarity between the video vector and that embedding §3.1 eq. 1 eq. 2 eq. 3.

Video encoder §3.2:

1. Frames are split into patches and encoded by the CLIP image transformer. The [class] token of each frame is its frame representation §3.2.
2. In each CCT block, every frame contributes a message token, a linear transform of its [class] token §3.2.1.
3. Cross-frame fusion attention (CFA) runs over all message tokens of the clip, with a residual connection §3.2.1.
4. Intra-frame diffusion attention (IFA) runs over each frame's patch and [class] tokens together with its message token, with a residual connection. The message token is then discarded, and a feed-forward layer updates the frame tokens §3.2.1.
5. The MIT, a standard transformer with learnable temporal position embeddings, processes the frame vectors. The video vector is the average over frames §3.2.

Text side §3.3, §3.3.1: the class name alone is the text description §3.3. Its embedding is the query of a cross-attention over the video's patch tokens, averaged over time. This is followed by a feed-forward layer, and the result is scaled by a learnable parameter (initial value 0.1) and added to the class embedding §3.3.1 eq. 2. Two such prompting blocks are used in all variants §4.1.

The paper ablates which pretrained branches to fine-tune; the randomly initialised parts are always trained §4.5 tab. 7. Frames are sampled sparsely across the clip §4.2 app. C. Which branch the main tables used is not stated; see Paper vs code.

# X-CLIP scoring of one video against K class names  [§3.1] [eq. 3]
# Input:  video (B, T, 3, H, W); class names (K)
# Output: logits (B, K)

def score_video(video, class_names):
    # Frame encoding with message tokens exchanged across frames  [§3.2]
    frame_cls, frame_patches = cross_frame_encoder(video)
    # (B, T, D), (B, T, N, D)

    # Temporal integration over frames, then average  [§3.2]
    frame_vectors = mit(frame_cls)                      # (B, T, D)
    video_vector = mean(frame_vectors, over=T)          # (B, D)

    # Class names embedded once  [§3.3]
    class_text = text_encoder(class_names)              # (K, D)
    prompts = repeat(class_text, times=B)               # (B, K, D)

    # Patch tokens of the video, averaged over time  [§3.3.1]
    content = mean(frame_patches, over=T)               # (B, N, D)

    # Attend from class text to video content, then feed-forward  [§3.3.1]
    for block in prompt_blocks:
        # Text is the query; video content is key and value  [§3.3.1]
        prompts = prompts + attention(prompts, content, content)
        # Feed-forward on the attended text  [§3.3.1]
        prompts = prompts + feed_forward(prompts)

    # Scaled video-specific prompts added to the class text  [§3.3.1]
    video_text = class_text + alpha * prompts           # alpha: init 0.1

    # Cosine similarity per video and class  [eq. 3]
    logits = cosine(video_vector, video_text)           # (B, K)
    return logits

Cross-frame attention inside one CCT block §3.2.1:

# One CCT block  [§3.2.1]
# Input:  frame_tokens (B, T, N+1, D), where index 0 is the [class] token

def cct_block(frame_tokens):
    # Message token per frame, from its [class] token  [§3.2.1]
    messages = linear(frame_tokens[:, :, 0])            # (B, T, D)
    # Cross-frame fusion attention over the messages, with residual  [§3.2.1]
    messages = messages + cfa(layer_norm(messages))     # (B, T, D)
    # Intra-frame diffusion: tokens and message attend jointly  [§3.2.1]
    messages = messages[:, :, None]                     # (B, T, 1, D)
    joined = concat([frame_tokens, messages], axis=2)   # (B, T, N+2, D)
    joined = joined + ifa(layer_norm(joined))
    # Drop the message token before the feed-forward  [§3.2.1]
    frame_tokens = joined[:, :, :-1]                    # (B, T, N+1, D)
    frame_tokens = frame_tokens + feed_forward(layer_norm(frame_tokens))
    return frame_tokens

Hyperparameters

Training settings are given per setting in the supplementary table app. B. Values are listed as fully-supervised / few-shot / zero-shot where they differ. Architecture and input values are from app. A and app. C.

Results

Author-reported. Benchmarks: Kinetics-400 and Kinetics-600 (fully-supervised), HMDB-51 and UCF-101 (zero-shot and few-shot), and Kinetics-600 unseen classes (zero-shot) §4 app. C. Baselines differ in pretraining data, so compare within a row group. FLOPs are per view as listed in the table.

Kinetics-400, selected rows tab. 1:

The paper's text quotes X-CLIP-L/14 at 8 frames as 2.3 points above ViViT-H with 12× fewer FLOPs §4.2. The table lists MTV-H at 89.1 with a 70M video-text dataset and about 17B images tab. 1 §4.2, and CoVeR at 87.2 tab. 1.

Kinetics-600, selected rows tab. 2:

Zero-shot, trained on Kinetics-400 with 32 frames, single view tab. 3 tab. 4:

The X-Florence rows in tab. 3 and tab. 4 use a model this repository does not contain; see Paper vs code.

Few-shot, top-1 with 32 frames tab. 5:

Ablations, Kinetics-400, X-CLIP-B/16, 8 frames, single view unless noted §4.5 tab. 6 tab. 7 tab. 8 tab. 9 tab. 10 tab. 11:

Other numbers from the appendix, author-reported app. D. For video encoders adapted from ViT-B/16, the paper compares its cross-frame design with joint space-time attention: fully-supervised on Kinetics-400, 82.3 against 82.1; zero-shot on UCF-101, 70.0 against 69.3; few-shot on HMDB-51 at 2 shots, 50.8 against 41.3. Its FLOPs are 145 against 184 app. D.

Paper vs code

Official code: microsoft/VideoX, directory X-CLIP, at code_commit. It contains CLIP-backbone variants (B/32, B/16, L/14, L/14@336) and one training and evaluation script with configs for Kinetics-400, Kinetics-600, HMDB-51 and UCF-101. It does not contain X-Florence. I found no K-shot sampler and no zero-shot Kinetics-600 split code.

• Similarity scale. The paper scores videos by plain cosine similarity eq. 3. The code multiplies the cosine by exp(logit_scale), a parameter that starts at log(1/0.07) and is then overwritten by the pretrained state dict xclip.py xclip.py xclip.py. The default training setup freezes it optimizer.py.
• Which branch is trained. The paper does not state which encoders are trained for the main tables. The code default freezes the text encoder and caches its output once config.py optimizer.py xclip.py. The paper's ablation shows fine-tuning both branches gives 72.9 zero-shot, against 70.0 with the text branch frozen tab. 7.
• Temporal residual. The paper's MIT output is the average of MIT(H + e_temp) §3.2. The code adds the unmodified frame features H back after the transformer, before averaging mit.py.
• Prompt scale. The paper's α is a scalar §3.3.1. The code uses a vector with one value per channel, initialised to the same value prompt.py.
• Prompt input. The paper uses the average of the final frame tokens as content §3.3.1. The code excludes the [class] token, passes the patch tokens through a layer norm and a linear projection to the embedding size, averages over frames, and applies another layer norm inside the prompt generator cct.py xclip.py prompt.py. The paper mentions none of this.
• Frame output. The paper's frame representation is the [class] token output §3.2. The code applies a layer norm and a linear projection to it cct.py.
• Prompting blocks. The code gives each block a layer norm on the query and a feed-forward layer with its own layer norm, with heads set to embedding size divided by 64, and chains the blocks, each taking the previous output as its query prompt.py prompt.py prompt.py. The paper does not describe chaining or heads §3.3.1.
• Zero-shot epochs. The Kinetics-400 config for 32 frames and ViT-B/16 (configs/k400/16_32.yaml) sets no epoch count, so the code default of 30 applies 16_32.yaml config.py. The paper's supplementary table gives 10 epochs for zero-shot app. B. The repository does not say which config produced the zero-shot checkpoint.
• Batch size. The paper's table gives 256 app. B and says fully-supervised training used 32 V100 GPUs §4.2. The README's recommended total is 8 GPUs × 8 per GPU × 4 accumulation steps = 256 README.md. Most Kinetics-400 configs give 32 per GPU (4 × 8) 14_8.yaml; the 336px config gives 32 as well (1 × 32) 14_16_336.yaml. The total therefore depends on the GPU count.
• Three-crop testing. The paper's protocol resizes the shorter side to 256 and takes 224 crops app. C. With three crops the code resizes the shorter side to 224 and takes three 224 crops build.py.
• Multi-view aggregation. The paper does not say how views are combined. The code sums the softmax probabilities over views and takes the argmax main.py.
• Initialisation. The paper says the cross-frame fusion attention is randomly initialised §3.2.1. The message modules have no explicit initialisation in the code, and the init_weights method in cct.py is not called anywhere in X-CLIP cct.py cct.py.
• Constructor defaults differ. XCLIP defaults to α = 1e-4 and one MIT layer xclip.py. build_model defaults to four MIT layers xclip.py. load, which the training script calls, uses α = 0.1, two prompt blocks and one MIT layer xclip.py main.py. Building the class directly gives different values from the script.
• Not in the repository. X-Florence and its FLD-900M pretraining. The few-shot configs read a fixed train file 16_32.yaml, so the K-shot sampling is outside the code. The zero-shot Kinetics-600 protocol (220 unseen classes, 160 drawn per run) app. C is also outside it. The paper's zero-shot and few-shot numbers cannot be reproduced from this repository alone.

Pitfalls

• Frame-token count for the prompt content. The paper writes the content as N × d §3.3.1 but does not say whether the [class] token is included. The code uses patch tokens only cct.py.
• Text varies per video. Because each video has its own prompts, a class text embedding cannot be cached across videos. The code caches only the raw text output, and the prompt generator runs once per video and class xclip.py.
• Message tokens are rebuilt in every block. They are created from the current [class] token, used once, and dropped before the feed-forward layer §3.2.1. They are not carried to the next block.
• Testing cost scales with views. Multi-view inference multiplies compute by the number of views, and the paper's FLOPs column is per view tab. 1. The paper reports that single-view with sparse training is close to multi-view: 82.3 against 83.4 at 8 frames tab. 11.
• Memory depends on which branch is trained. The paper's memory figures are for 2 inputs of 8 frames. Freezing the text encoder cuts memory from 22 GB to 6 GB. It costs 0.1 points fully-supervised, but 2.9 zero-shot and 3.8 few-shot tab. 7.
• Few-shot comparisons. The paper says the CLIP-pretrained comparisons are not fair and adds an ImageNet-pretrained ablation to separate the effect of text §4.4 §4.5. Read the few-shot rows as a comparison of pretraining sources as well as methods.
• Benchmark sets differ. Kinetics-600 zero-shot numbers average three random draws of 160 of the 220 unseen classes app. C. HMDB-51 and UCF-101 average three splits, with standard deviations tab. 3.
• Supplement numbering. The supplementary sections are not lettered in the source; this card cites them by their order: A architecture, B hyperparameters, C datasets and protocols, D additional experiments.
• Reproducibility. The authors' own README says results vary by about ±0.2% top-1 across machines README.md.
• Not stated. The paper gives no seed, no mixed-precision setting, no initialisation for the new modules, no weight-decay exclusions, and no switch probability for mixup and CutMix.

Lineage

• Builds on: CLIP, for the contrastive image-text pretraining and the image and text encoders §1 §3.1; the vision transformer §3.2; the transformer attention and layer normalisation used in the MIT and prompt generator §3.2 §3.3.1.
• Compared against: ActionCLIP, a concurrent CLIP-based video method §2; ViViT, TimeSformer, Video Swin, MViT and MTV among video transformers §2 tab. 1; ER-ZSAR, DEVISE, ALE and other zero-shot baselines tab. 3 tab. 4; TSM, TimeSformer and Video Swin for few-shot §4.4 tab. 5.
• Related prompting methods named in the ablation: CoOp-style learned vectors and CLIP's prompt ensemble tab. 9.
• Superseded by: not stated in the paper.