InternVideo: General Video Foundation Models via Generative and Discriminative Learning

In plain words

Video is much harder to learn from than images: a clip is many frames, so it costs far more to process, and many existing video models are good at only one kind of task. Some are trained to fill in hidden parts of a clip and become good at recognising actions. Others are trained to match a video with its text description and become good at search and question answering.

InternVideo trains both kinds of model separately, one on hiding and filling in video, the other on matching video with text. It then learns how to combine what the two models know, while keeping both of them mostly frozen. The goal is one video model that works across many tasks: recognising actions, localising them in time, and linking video with language.

The authors report strong results on action-recognition benchmarks such as Kinetics-400 and Something-Something V2, and on video-to-text retrieval. These are their own numbers, and the full training recipe needs a large GPU cluster.

TL;DR

• InternVideo is a video foundation model trained in two self-supervised ways: masked video modeling (a ViT-H) and video-language contrastive learning with a caption decoder (a CLIP-initialised UniFormerV2-L/14) §3.1.
• Both encoders get a supervised post-pretraining step on Kinetics-710 §3.2.
• A cross-model attention module then fuses the two frozen encoders with a learned linear score combination §3.3.
• Author-reported: 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2 tab. 2 tab. 3; the paper claims results on 39 datasets across 10 tasks §1.
• Relevant if you need one video backbone for action recognition, temporal localisation, or video-text retrieval. Only parts of the method are released in code (see Paper vs code).

Problem

Image foundation models are cheap to adapt, but video needs temporal modeling, which costs at least an order of magnitude more compute than images at similar resolution with a usual temporal sampling ratio of 16 §1. Many video benchmarks can be solved from appearance alone, so the authors argue that video-specific foundation models get less attention §1. Existing video foundation models tend to specialise: they target action understanding or video-language alignment, but not both §1. The paper's assumption is that a model trained on both kinds of self-supervision and combined afterwards can cover both §1 §3.

Method

InternVideo uses a ViT and its variant UniFormerV2, with extra local spatiotemporal modules for multi-level interaction §3. Training has four parts. The first two are self-supervised pretraining of two separate encoders. The third is supervised post-pretraining of each encoder on Kinetics-710. The fourth fuses the two frozen encoders §3 fig. 2.

1. Masked video encoder §3.1 fig. 3a. The clip is downsampled in time, cut into non-overlapping 3D patches, and linearly embedded. Tube masking hides a high share of the tokens (the paper's example is 90%). An asymmetric encoder-decoder ViT reconstructs the hidden content. The decoder has half the encoder's channel width and 4 blocks by default §3.1. Joint space-time attention lets all visible tokens interact §3.1.

2. Multimodal video encoder §3.1 §3.2 fig. 3b. The starting point is CLIP ViT, with UniFormerV2 as the video encoder for better and cheaper temporal modeling §3.1. The paper uses an align-before-fuse design §3.1. Video and text are encoded separately and a contrastive loss aligns them in one embedding space. A caption decoder with cross-attention then fuses the modalities through a captioning objective §3.1. The additional parameters are initialised so the output equals the original CLIP model at the start, which the authors call essential for zero-shot performance §3.2. The captioning module is a 6-layer transformer decoder of width 768 followed by a two-layer MLP §3.2. Global UniBlocks are inserted in the last 4 layers of the ViT-B/L backbone §3.2.

3. Supervised post-pretraining §3.2. Action classification is used as a meta task. Both encoders are fine-tuned on Kinetics-710 (K710), a combined label set built from Kinetics-400, -600 and -700 §3.2 §4.1. The masked encoder uses a larger learning rate and layer-wise decay. The multimodal encoder follows the UniFormer recipe §3.2.

4. Cross-model interaction §3.3 fig. 4. Both backbones are frozen except for the classification layers and the query tokens of the multimodal encoder §3.3 §4.2. New cross-model attention (CMA) blocks, each a multi-head cross-attention plus an FFN, are inserted. In each stage the masked encoder's tokens are queries, and the multimodal encoder's tokens are keys and values §3.3. The last CMA reverses the direction: the multimodal class token is the query and the masked encoder's tokens are keys and values §3.3. The two prediction scores are then combined with a learnable linear combination §3.3. The paper also uses tanh gating in the extra attention and FFN, as in Flamingo, and sets the dynamic-sum weights to zero at initialisation §4.2.

Pretraining data (UnlabeledHybrid, for the masked encoder) is Kinetics-710, Something-Something V2, a chunked AVA set, a random sample of WebVid2M and an authors' self-collected set §4.1. Table 1 lists the sources: Kinetics-400, WebVid2M, WebVid10M, HowTo100M, AVA, Something-Something V2, self-collected videos and K710, with clip counts and frames × sample rate per source tab. 1.

The pseudocode below summarises the stages as stated in the text. The paper has no algorithm box, so this is not a verbatim algorithm. Numbers are in the Hyperparameters table.

# Recipe as stated in the paper (no algorithm box)  [§3] [§4.2]
# Input:  video clips; video-text pairs; image-text pairs; labelled K710
# Output: fused video classifier scores

def pretrain_masked_encoder(clips):
    # Tube-mask 90% of the tokens, then reconstruct them  [§3.1] [§4.2]
    visible = tube_mask(patchify_3d(clips), ratio=0.9)
    return reconstruction_pretrain(vit_huge, visible)

def pretrain_multimodal_encoder(video_text, image_text):
    # Start from CLIP; contrastive loss plus a caption decoder  [§3.1]
    video_feat, text_feat = encode_separately(video_text)
    loss = contrastive(video_feat, text_feat)
    # Captioning uses cross-attention over the video tokens  [§3.1]
    loss = loss + caption_loss(cross_attn_decoder(video_feat, text_feat))
    return loss

def finetune_on_k710(encoder, k710, base_lr, batch_size):
    # Supervised action classification on K710  [§3.2]
    # Learning rate scales with batch size  [§3.2]
    lr = base_lr * batch_size / 256
    return train_classifier(encoder, k710, lr=lr)

def cross_model_fusion(masked_enc, multi_enc, k710):
    # Freeze both backbones; train heads, query tokens and CMA  [§3.3] [§4.2]
    for block in cma_blocks:
        # Masked-encoder tokens query multimodal-encoder tokens  [§3.3]
        masked_tokens = gated_cross_attention(masked_tokens, multi_tokens)
    # Last block: multimodal class token queries masked-encoder tokens  [§3.3]
    class_token = last_cma(class_token, masked_tokens)
    # Learnable linear combination of the two prediction scores  [§3.3]
    scores = w_masked * scores_masked + w_multi * scores_multi
    return scores

Key equations: the paper has no numbered equations. The one formula in the text is the learning-rate scaling rule, written in prose as learning rate = base learning rate × batch size / 256 §3.2.

Hyperparameters

Values are from the paper unless the "Where" column names code. "not stated" means the paper and the released code do not give the value. Learning rates and other settings are given as written.

Results

Author-reported. Setup: InternVideo variants are fused models; the -T variant (1.3B parameters) is the headline model, and the -D variant (1.0B parameters) is the masked encoder plus the CLIP-based UniFormerV2-L fused tab. 2. Baselines are the paper's comparison models, with the best listed baseline shown per row. Downstream evaluation protocols and hardware were not extracted for this card.

Action recognition (top-1 accuracy, %):

Temporal and spatiotemporal localisation (mAP, %):

Video-text retrieval, fine-tuned (R@1, %; MSR-VTT, text-to-video / video-to-text):

The fine-tuned table also covers MSVD, LSMDC, ActivityNet, DiDeMo and VATEX tab. 6.

Zero-shot video retrieval (R@1, %):

The paper also reports video question answering, video-language navigation, zero-shot multiple choice and open-set recognition §4.3 tab. 7 tab. 8 tab. 10 tab. 11. These are not tabulated here. The introduction also claims wins in five tracks of recent Ego4D competitions §1.

Paper vs code

Repository: https://github.com/OpenGVLab/InternVideo, commit 3965eef16e2dadd0ea6c8d0cc29c8a3039df52e3, root LICENSE Apache-2.0. The paper's code lives under InternVideo1/ in this monorepo. The repo also holds later generations (InternVideo2, 2.5, 3, Next), which this card does not use. Paths below are relative to the repository root.

• Masked pretraining schedule. The paper says 1200 epochs at base learning rate 2.5e-4 §4.2. The released hybrid pretraining script uses --epochs 301, --lr 1e-3, --min_lr 5e-5, --warmup_epochs 15, --mask_ratio 0.9 and --decoder_depth 8 slurm_train_vit_h_hybrid_pt.sh. The checkpoint name used by the fine-tune script is vit_h_hybridv2_pt_1200e.pth slurm_train_vit_h_mixk_ft.sh, so the 1200-epoch run seems to exist but the script in the repo was not updated. The paper's values are not reproduced by the script as released.
• Decoder size. The paper says 4 decoder blocks and half the encoder's channel width by default §3.1. The ViT-H pretraining model uses encoder width 1280 and decoder width 512, which is not half modeling_pretrain.py, and sets decoder_depth=8 for the huge model modeling_pretrain.py. The argparse default is 4 run_mae_pretraining.py.
• Tubelet size. The paper does not state the temporal patch size. The code uses tubelet size 2 modeling_pretrain.py.
• Masked K710 fine-tune. The released script matches the text on base learning rate 1e-3, drop path 0.2, head dropout 0.5, layer decay 0.8, repeated sampling 2 and 40 epochs slurm_train_vit_h_mixk_ft.sh. It uses batch 4 per GPU and the GPU count defaults to 1 slurm_train_vit_h_mixk_ft.sh slurm_train_vit_h_mixk_ft.sh. The paper's 32-GPU run is not in the script. The learning-rate scaling rule is implemented as the paper states run_class_finetuning.py.
• Multimodal K710 fine-tune. The released run script uses base learning rate 1e-5 and 40 epochs, which match §3.2 run.sh run.sh. It sets the training batch size to 128, but the paper says 512 §3.2 run.sh. The run script also sets 5 warmup epochs, which the paper does not state run.sh. Neither the paper nor the script says whether 512 is a global or per-device batch; this card does not resolve it.
• Local modules. The paper describes local spatiotemporal modules in the video encoder §3. The released K710 L/14 config sets NO_LMHRA: True config.yaml, and N_LAYERS: 4 config.yaml. Which local-module setting the paper's fused model uses is not stated, and I did not resolve it.
• Cross-model attention is not released. I searched the repository for the cross-model attention, the gated MHCA, and the EMA settings from §4.2. I found no implementation. InternVideo1/Pretrain/VideoMAE/ensemble.py only averages per-clip softmax scores over two sets of saved predictions ensemble.py. It does not implement the learned fusion described in §3.3. The cross-model stage (§3.3, §4.2) cannot be checked against code.
• Multimodal contrastive and caption pretraining is not released at the paper's settings. The paper's recipe (batch 14,336 and 86,016, 400k steps, learning rate 8e-5) is not in the repository. The CoTrain configuration uses a batch of 4096 and learning rate 1e-4 config.py config.py. The ViCLIP configuration uses learning rate 4e-4 and batch 512 config.py config.py. These are other recipes, and I do not use them as evidence for this paper.
• Appendix. content/appendix.tex in the source is not included by main.tex. It has placeholders such as "xx" and unfinished notes, so the card does not use it.

Pitfalls

• Learning rate written as 5×10^5. The cross-model learning rate is written as "5\times10^{5}" §4.2. This is almost certainly 5×10^-5, but the text does not say so, and no code for this stage is released.
• Dropout 0.9 for cross-model training. The paper gives dropout 0.9 §4.2. This is unusually high and is not explained.
• Kinetics-710 size. The text says 650K videos §4.1. Table 1 gives 680k tab. 1. The paper does not say which is used.
• Leakage filtering. The paper says training clips that also appear in the Kinetics test sets were removed, but gives no details of the filter §4.1.
• Effective batch size. The learning rate scales with total batch size §3.2. The paper does not give the total batch for the masked fine-tune, so the effective learning rate cannot be reproduced exactly.
• Which model gives the headline numbers. The 77.2% on Something-Something V2 in the abstract comes from tab. 3, which does not name the -D or -T variant. The -D and -T rows that include Something-Something V2 are in a block of tables/kinetics_ensemble.tex that is disabled with \iffalse in the source, so the compiled paper does not show them.
• Benchmark counts disagree. The abstract and introduction say 39 datasets across 10 tasks §1. Section 3 says 34 benchmarks from 10 tasks §3. Both counts appear in the paper.

Lineage

• Builds on: VideoMAE (named in §3.1; no arXiv id is given in the paper's bibliography), CLIP §3.1, UniFormerV2 (arXiv:2211.09552 §3.2), and the Flamingo tanh-gating idea §4.2.
• Compared against: specialised action models and video foundation models in tab. 2, tab. 3 and tab. 6 (MTV, CoCa, MaskFeat, MerlotReserve, X-CLIP, TS2Net, CLIP4Clip).
• Superseded by: not stated in this card's sources.