Unmasked Teacher: Towards Training-Efficient Video Foundation Models

In plain words

Training a model to understand video from scratch takes a lot of data and compute. A common shortcut is to start from a model that already understands single pictures, but its knowledge does not transfer cleanly to motion. This paper trains a video model from scratch instead, and uses a picture model as a coach.

The student video model is shown only a small part of each frame, and it must produce the same internal descriptions that the coach produces for those parts. The parts are chosen to be the ones the coach considers important, so the student learns from useful content rather than empty background. There is no step that redraws the missing pixels, which saves memory and avoids a clash between low-level detail and high-level meaning.

Once this video model is trained, a second stage connects it to text, so it can match clips to captions and answer questions about them. The authors argue that this recipe learns motion well, needs less data than earlier video pre-training, and is cheaper than some large alternatives.

TL;DR

• A vanilla ViT video encoder is trained from scratch with masked video modeling, but with no pixel decoder.
• Unmasked tokens (20% at the default ratio) are chosen with a frozen CLIP teacher's attention and are regressed onto the teacher's features from its last 6 layers.
• Stage 2 adds text encoders and co-trains with contrastive, matching, masked-language and the same alignment loss on image-text and video-text data.
• Author-reported: strong Kinetics, Something-Something V2, AVA and video-text results with public data, on 32 A100 GPUs in about 6 days §1.

Problem

Video foundation models are hard to build for three reasons §1. Image foundation models adapted to video can lose generality, and their strong spatial features let a model solve clips from single frames, which hides motion such as "opening" versus "closing". Masked video pre-training such as VideoMAE is data-efficient but needs very long schedules (2400 epochs on 160k videos is quoted) §1. Its pixel-reconstruction target also clashes with the cross-modal alignment needed for video-language tasks, and its extra decoder costs memory §1 §4.2.

Method

The method has two parts: the Unmasked Teacher pre-training target §3.1, and a two-stage schedule §3.2.

Unmasked Teacher (Stage 1 target) §3.1

• Teacher. A CLIP ViT is used as a frozen teacher. Its spatial architecture is kept, and each frame is processed on its own, with all tokens, class token included §3.1. For the base model the teacher is CLIP ViT-B/16; for the large model it is CLIP ViT-L/14, run at 196 px to match the token count §4.1.
• Student. A plain ViT with no class token, using spatiotemporal attention across all frames. It has no temporal downsampling, so tokens can be aligned frame by frame §3.1. The patch is 1 × 16 × 16 (time × height × width), and the input is 8 sparsely sampled frames tab. 18 §4.1.
• Semantic masking. The class token's attention to the patch tokens in the teacher's last self-attention layer is averaged over heads, giving a score per patch eq. 1 eq. 2. Per frame, a set of tokens is drawn with a multinomial distribution from these scores, and the drawn tokens are kept visible §3.1. The masking ratio is 80% §3.1 §4.1. Frames are sampled sparsely, with no temporal downsampling for the student §3.1 §4.2.
• Visible tokens. The student receives only the visible tokens, about L(1 − r)T in number, where L is tokens per frame, r the masking ratio, and T the number of frames §3.1. At 224 px with 8 frames and r = 0.8, this gives roughly 320 visible tokens.
• Target. The teacher is run on the full, unmasked clip. Its outputs at the visible positions are the targets. They pass through the teacher's pre-trained visual projection §3.1. The student output for the same positions passes through a small linear projection to the teacher's dimension §3.1 app..
• Loss. A squared error between the normalised student and teacher vectors, averaged over the visible tokens and over the last 6 teacher layers, which are aligned with separate student heads §3.1 §4.2. No pixel term is used.

# One Stage-1 training step  [§3.1] [§4.2]
# Input:  clip (B, 3, T, 224, 224); frozen CLIP teacher; student ViT
# Output: scalar loss

def stage1_loss(clip, student, teacher, mask_ratio=0.8):
    # Teacher pass on the full clip, frame by frame. Returns the projected,
    # L2-normalised features of the last 6 blocks and the class token's
    # attention to each patch in the last block  [§3.1]
    teacher_feats, cls_attn = teacher(clip)     # (6, B*T, N, C), (B*T, N)
    # Semantic masking: per frame, keep a subset of patches drawn without
    # replacement with probability proportional to cls_attn  [§3.1] [eq. 1-2]
    num_visible = N - int(N * mask_ratio)      # N = 196 -> 40 per frame
    visible = sample_without_replacement(cls_attn, num_visible)
    # Student sees only the visible tokens of all frames; there is no decoder
    # and no pixel target  [§3.1]
    student_feats = student(clip, visible)     # (6, B, T*num_visible, C)
    # Targets are the teacher features at the same visible positions  [§3.1]
    targets = gather(teacher_feats, visible)   # (6, B, T*num_visible, C)
    # Squared error between unit vectors, averaged over tokens and layers.
    # The released code uses its "l2" form; see Paper vs code.
    # [§3.1] [code: single_modality/engines/engine_for_pretraining_umt.py#L92-L93]
    return mean(squared_error(student_feats, targets))

Progressive pre-training (Stage 2) §3.2

• Stage 1 data and goal. The encoder is pre-trained on Kinetics-710 with unmasked token alignment only, which gives a video-only model §3.2 §4.1.
• Stage 2 model. The encoder is paired with a text encoder and a cross-modal decoder. Both are initialised from BERT: for the base model, the first 9 layers of BERT-base form the text encoder and the last 3 form the decoder; for the large model, the first 19 layers of BERT-large and the last 5 §3.2 app.. The released config uses the same fusion points [code: multi_modality/configs/model.py#L7-L14].
• Stage 2 objectives. All four losses have weight 1 app.:
  • video-text contrastive learning (VTC), a symmetric contrastive loss on pooled embeddings §3.2;
  • video-text matching (VTM), a binary cross-entropy with hard-negative mining over fused tokens §3.2;
  • masked language modeling (MLM), BERT-style but masking 50% of text tokens, predicted from the other text and the visible video tokens §3.2;
  • unmasked token alignment (UTA), the Stage-1 loss applied on the Stage-2 data §3.2.
• Stage 2 masking. Images are treated as single-frame videos. 50% of image tokens and 80% of video tokens are masked §4.1 tab. 19 [code: multi_modality/exp/pretraining/b16_25m.py#L62-L66]. Text tokens are masked at 50%.

# Stage-2 objective on one batch of paired data  [§3.2] [app.]
# Input:  visual (B, 3, T, H, W) with T=4 frames (images: T=1), text tokens

def stage2_loss(visual, text, encoder, text_enc, cross_dec):
    # Visible video or image tokens from the Stage-1 encoder, masked at the
    # ratios given in the setup  [§4.1]
    visible = encoder(visual)
    # Contrastive (VTC) on pooled embeddings  [§3.2]
    vtc = symmetric_contrastive(pool(visible), pool(text_enc(text)))
    # Matching (VTM) with hard negatives from the batch  [§3.2]
    vtm = matching_bce_with_hard_negatives(cross_dec, visible, text)
    # Masked language modeling, 50% of text tokens  [§3.2]
    mlm = masked_word_loss(cross_dec, visible, mask_text(text, p=0.5))
    # Alignment to the frozen CLIP teacher, same as Stage 1  [§3.2]
    uta = unmasked_token_alignment(encoder, visual)
    return vtc + vtm + mlm + uta      # all weights are 1  [app.]

Hyperparameters

Stage 1 and fine-tuning settings are in the paper's appendix tables. The table numbers below count captions in LaTeX order, so they are a reading of the source and not checked against the PDF. Items marked "not stated" are absent from the paper and not set elsewhere in the card.

Stage 1: pre-training on Kinetics-710 app. tab. 18

Stage 2: multi-modality co-training §4.1 app. tab. 19

Fine-tuning settings are in tab. 20 (action recognition), tab. 21 (AVA), tab. 22 (video QA), tab. 23 (multiple-choice QA) and tab. 24 (video-text retrieval) app..

Results

Author-reported. The setup differs by row: pre-training data, teacher, model size, input size, and whether intermediate fine-tuning was used. The daggers in the source tables mark intermediate fine-tuning, and the rows should not be compared across dagger marks.

Kinetics-400, top-1 tab. 8 (spatial pre-training on K710 unless noted)

Kinetics-600 / 700, top-1 tab. 9 (K710 pre-training and intermediate FT)

Something-Something V2, top-1 tab. 11

Moments in Time V1, top-1 tab. 10: UMT-L at 8 × 224² gets 48.0; at 8 × 384² it gets 48.7. UniFormerV2-L at 8 × 224² gets 47.0 tab. 10.

AVA v2.2, mAP tab. 12: UMT-L with K710 pre-training gets 39.8 and with K400 pre-training 39.0. UMT-B gets 33.5 with K710 and 32.7 with K400. VideoMAE-L with K700 gets 39.3 tab. 12.

Text-to-video retrieval, R@1, zero-shot tab. 13: UMT-L with 17M pairs gets MSRVTT 42.6, DiDeMo 46.4, ActivityNet 42.8, LSMDC 25.2 and MSVD 49.9. With 25M pairs it gets 40.7, 48.6, 41.9, 24.9 and 49.0.

Text-to-video retrieval, R@1, fine-tuned tab. 14: UMT-L with 25M pairs gets MSRVTT 58.8, DiDeMo 70.4, ActivityNet 66.8, LSMDC 43.0 and MSVD 58.2 (see Pitfalls). On SSv2-label and SSv2-template the text reports 73.3% and 90.8% §4.4.

Video question answering, accuracy tab. 16: UMT-L with 25M pairs gets ActivityNet-QA 47.9, MSRVTT-QA 47.1, MSRVTT-MC 97.3 and MSVD-QA 55.2.

Ablations §4.2 tab. 1 tab. 2 tab. 3 tab. 4 tab. 5 tab. 6 tab. 7 (all K400 or SSv2 pre-training, ViT-B/16, unless noted):

• Target: unmasked alignment alone reaches 70.2 SSv2 and 84.9 K400 at 16.0 GB, against 44.0 GB for pixel reconstruction (67.1 SSv2, 78.8 K400) tab. 1. Adding masked-token recovery to alignment costs accuracy on K400 and MSRVTT tab. 1.
• Masking: semantic masking gives the best K400 score (84.9 vs 84.6 random, 84.3 tube) and ties on SSv2 (70.2). Dense sampling or temporal downsampling is worse tab. 2.
• Teacher, attention, and masking interact: spatiotemporal student attention and masking both matter tab. 3.
• Teacher choice: DINO, CLIP and BEiTv2 teachers all give students that beat their own fine-tuned teachers tab. 4.
• Multi-modality objectives: all three of VTC, VTM and MLM plus UTA give the best MSRVTT and SSv2 scores (67.0, 80.8), against 60.3 and 74.3 for VTC with UTA alone tab. 6.
• Masking ratios for stage 2: 50% image and 80% video is best, at 18.6 GB tab. 5.
• Training schedule: longer schedules help, and this method converges faster than VideoMAE §4.2 fig. 5.

Paper vs code

Official code: OpenGVLab/unmasked_teacher at commit 4fb4049f5a87919882e68ccc427615ae7dab1c33, MIT licence §1 [code: raw/code/2303.16058/LICENSE]. The paper links it from its author block §1, and the repository owner is the OpenGVLab organisation. The stage-1 and stage-2 code paths were read; the fine-tuning code was not compared with tab. 20–tab. 24.

• Loss form. The paper calls the target "mean squared error" between normalised vectors §3.1. The released scripts pass --clip_loss_type 'l2' l16_ptk710_f8_res224.sh, which computes 2 - 2·dot, a squared distance summed over channels and averaged over tokens engine_for_pretraining_umt.py. A per-element MSE would divide that by the channel count (768 for L, 512 for B), so the loss scale differs by that factor. A mse option exists in the code engine_for_pretraining_umt.py but the scripts do not use it.
• Projection order. The appendix describes a layer norm followed by a linear layer app. tab. 17. The code applies the linear layer first, then a layer norm, then L2 normalisation modeling_pretrain_umt.py.
• Positional embedding not in the paper. The code adds a fixed sinusoidal table to the visible student tokens before the per-layer heads [code: single_modality/models/modeling_pretrain_umt.py#L260-L262]. The paper does not mention this.
• Epochs. The paper says 200 §4.1. The L/16 Kinetics-710 script runs 201 epochs l16_ptk710_f8_res224.sh.
• Stage-2 batch size. The paper says 4096 on 32 A100 for both sizes §4.1. The base stage-2 scripts use 2 nodes × 8 GPUs = 16 GPUs [code: multi_modality/exp/pretraining/b16_25m.sh#L14-L15] with 128 per GPU [code: multi_modality/exp/pretraining/b16_25m.py#L16], so 2048 in total. The large scripts use 4 nodes, or 32 GPUs l16_25m.sh, which gives 4096.
• Learning-rate scaling. The Stage-1 code scales the learning rate, minimum and warmup rates by total batch / 256 run_umt_pretraining.py. The stated 1.2e-3 follows only for batch 2048. The Stage-2 config sets 1e-4 directly, with no scaling found in the task code [code: multi_modality/tasks/pretrain.py]. The paper does not state the scaling rule.
• Teacher features. The teacher returns its projected, L2-normalised output from the last 6 blocks, and the class-token attention, averaged over heads clip.py [code: single_modality/models/clip.py#L155-L170]. This matches the method's description.
• Teacher input size. When the frame is not already at the teacher's resolution, the code resizes it with bicubic interpolation [code: single_modality/engines/engine_for_pretraining_umt.py#L52-L58]. For L/14 at 196 px, the teacher's 196 tokens per frame match the student's 224-px tokens.
• Data paths. The Kinetics-710 index path is a placeholder [code: single_modality/exp/pretraining/b16_ptk710_f8_res224.sh#L7], so the index file must be supplied by the user.

Pitfalls

• Teacher row labels in the ablations. The paper's teacher table uses "CLIP-ST" as the teacher, and its best row (70.3 SSv2, 85.2 K400) matches the default row of the earlier ablation table, which is labelled "CLIP-S" tab. 3 tab. 4. The method says the teacher processes frames individually, so the default is the spatial teacher §3.1. Read the "CLIP-ST" label in tab. 4 as a likely typo, not a finding.
• Results text and tables disagree. The §4.4 text quotes 72.2% R@1 for zero-shot MSVD, but tab. 13 lists 49.9 and 49.0 for UMT-L. The text also quotes 80.3% for fine-tuned MSVD, while tab. 14 lists 58.2 R@1 for the same setting. The text's DiDeMo and MSRVTT zero-shot figures also mix rows from 17M and 25M. Use the tables.
• Stage 1 needs a frozen teacher forward pass every step. The teacher sees the full, unmasked clip on each step, and it is not masked [code: single_modality/engines/engine_for_pretraining_umt.py#L63-L66]. The memory figures in tab. 1 are the paper's own benchmark. Wall-clock cost per step is not reported, so the teacher's overhead is not measured.
• Sampling without replacement. The mask is drawn per frame from the teacher's attention, so it is only as good as that attention map. The code draws a full permutation of patches and keeps the first visible ones [code: single_modality/engines/engine_for_pretraining_umt.py#L71-L75]. The paper leaves the sampling procedure unstated.
• Teacher features are used as fixed targets. The paper says the teacher is frozen §3.1. The code runs it under no_grad and autocast [code: single_modality/engines/engine_for_pretraining_umt.py#L47-L66]. Mixed precision is therefore part of the target computation.
• Pixel reconstruction is removed. Setting the pixel term to zero [code: single_modality/engines/engine_for_pretraining_umt.py#L90] is what makes the method "no decoder". The pixel-reconstruction baseline in tab. 1 is in a separate script, which is not the code path used for the method [code: single_modality/run_mae_pretraining.py].
• Stage-2 init depends on Stage 1. The Stage-2 configs load the Stage-1 Kinetics-710 checkpoint l16_25m.py, so Stage 2 cannot run without a Stage-1 model.
• Table numbers in this card. Appendix tables are numbered after the main text by caption order in the LaTeX source tab. 17–tab. 24. The numbers are a source reading and have not been checked against the PDF.
• Headline costs. The 70× carbon reduction against CoCa is quoted in the paper's Figure 1 caption and introduction, with CLIP's pre-training cost ignored fig. 1 §1. It is not computed in the text.
• Fine-tuned rows and intermediate fine-tuning. The top K400 and K600/K700 numbers use intermediate fine-tuning on K710 tab. 8 tab. 9. Compare these only with other rows marked the same way.

Lineage

• Builds on: VideoMAE, which the paper extends by replacing pixel reconstruction with alignment to an image-model teacher and dropping the extra decoder §3.1 §1.
• Builds on: CLIP ViT, used as a frozen teacher §3.1. Its frame-wise, spatial architecture is kept, and the teacher's visual projection is applied to targets §3.1.
• Stage 2 follows Singularity's recipe for sampling, text-encoder split, and co-training data §4.1 app..
• Compared against: UniFormerV2, MTV, CoCa, InternVideo, VINDLU, Singularity and others on the tables above §4 tab. 8–tab. 16.
• Superseded by: not stated in the wiki.