VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

In plain words

A common way to train vision models without labels is to hide most of an input and train the network to fill in what is missing. For video, this means hiding most of the frames' small blocks ("cubes") and reconstructing them. This paper asks how far that approach scales: it trains a video model with about one billion parameters on a pool of about 1.35 million unlabeled clips.

The bottleneck is cost. The network that fills in the hidden parts also has to process every block, which is slow and memory-hungry. The authors hide part of that decoder input as well, so it reconstructs only some blocks. Training then runs in stages: first on the unlabeled clips, then on a labeled mix of Kinetics datasets, and finally on the target dataset. The reported results are strong on action recognition and on action detection, though they come from the authors' own large-scale runs.

TL;DR

• Scales VideoMAE, a masked autoencoder for video, to a ViT-g backbone with 1,011M parameters §4.1.
• Dual masking: the encoder sees about 10% of the cubes (tube masking at 90%), and the decoder reconstructs a running-cell subset that keeps 50% of cubes §3.2 tab. 10.
• Progressive training: unlabeled pre-training on 1.35M clips, supervised post-pre-training on the 710-class labeled Kinetics mix (K710), then fine-tuning on each target §3.3 §4.2.
• Author-reported: 90.0% top-1 on Kinetics-400, 77.0% on Something-Something V2 and 68.7% on Something-Something V1 tab. 6.
• The official code is MIT-licensed and matches the paper on most architecture settings. Learning rate, warmup and epoch count differ from the paper's tables (see Paper vs code).

Problem

Scaling masked video pre-training runs into compute and memory limits. The authors report that pre-training ViT-g with VideoMAE took more than two weeks on 64 A100 GPUs §1. Kinetics-400 has only about 0.24M videos, which is small next to ImageNet-22k, so a billion-parameter model may overfit if it is trained on that data alone §1. Fine-tuning such a model directly on a small target set is also expected to be suboptimal §1. The paper assumes that video is redundant enough that the decoder can reconstruct a subset of the cubes without losing much of the training signal §3.2.

Method

Starting point: VideoMAE §3.1. A clip with frame stride τ is split into cubes by a cube embedding Φ_emb. A high ratio ρ of the tokens is dropped with tube masking (ρ = 90% in the default setting). A ViT encoder runs only on the visible tokens, which number N^e ≈ 0.1N. A second ViT decoder receives the encoder output together with learnable mask tokens at the full token length and reconstructs the hidden pixels. The paper gives the original encoder-only loss inline in §3.1 as a mean squared error over the masked cubes, computed on normalised pixels.

Dual masking §3.2 fig. 1. The encoder mask M_e is the same tube mask as VideoMAE, at ratio ρ^e. A second mask M_d is applied to the decoder input, using running-cell masking at ratio ρ^d = 0.5 tab. 10 tab. 1. The paper picks running-cell masking from several alternatives tested in its ablation (frame, random, running cell) tab. 1 §4.2. Its reasoning is that tube masking in the encoder prevents information leakage across time, while the decoder mask should instead keep diverse cubes so that little information is lost. Steps:

1. Encode the visible cubes given by M_e with the ViT encoder §3.2.
2. Build the decoder input Z^c as the encoder output together with a learnable mask token and its position embedding for each position kept by M_d eq. 1.
3. Decode the combined sequence. Only the positions kept by M_d are reconstructed.
4. Compute the MSE loss only over cubes that the encoder hid and the decoder kept, as in eq. 2 §3.2 eq. 2.

Architecture §3.3 §4.1 tab. 7. The backbones are vanilla ViT models (ViT-B, ViT-L, ViT-H and ViT-g) with joint space-time attention. ViT-g uses 14×14 patches, 40 blocks, 1408 channels and 16 heads §4.1. Its cube embedding uses 2×14×14 cubes tab. 7. The decoder is kept small at 4 blocks with 512 channels in every model size app. Model Architecture.

Data §3.3 app. Datasets tab. 8. UnlabeledHybrid combines Kinetics, Something-Something V2, AVA, WebVid2M and a self-collected Instagram set, about 1.35M clips in total. Labeled hybrid (K710) takes the union of Kinetics-400, 600 and 700 with duplicates and validation overlap removed. It has 710 classes and about 0.66M clips §4.1 app. Datasets.

Progressive training §3.3. (1) Masked pre-training on UnlabeledHybrid. (2) Supervised post-pre-training on K710, following the intermediate fine-tuning used in image models. (3) Specific fine-tuning on each target dataset §3.3 app. Implementation Details.

Pre-training target normalisation §3.1. The paper says the loss uses normalised masked pixels but does not give the normalisation. The code normalises each cube per channel over its own pixels [code: engine_for_pretraining.py#L77-L88].

Pseudocode of one dual-masked pre-training step, written from the paper and checked against the code. The paper gives no algorithm block.

# Dual-masked VideoMAE V2 pre-training step  [§3.2] [fig. 1]
# Input:  clip (3, 16, 224, 224); encoder_mask M_e (True = hidden)
#         decoder_keep M_d (True = decoder reconstructs this cube)
# Output: scalar loss

def pretrain_step(clip, encoder_mask, decoder_keep):
    # Target pixels, normalised per cube and channel  [§3.1]
    # (N, 2*14*14, 3) with N = 8 * 16 * 16 = 2048  [tab. 7]
    targets = normalise_cubes(clip)
    # Cube embedding and positional embedding  [§3.1] [tab. 7]
    tokens = cube_embed(clip) + pos_embed           # (N, 1408)
    # Encoder reads only the cubes that M_e does not hide  [§3.1]
    latents = encoder(tokens[~encoder_mask])        # (N_e, 1408)
    latents = encoder_to_decoder(latents)           # (N_e, 512)
    # Decoder input: encoder latents, then a learnable mask token  [eq. 1]
    # plus its position embedding for each position kept by M_d
    # [code: models/modeling_pretrain.py#L340-L360]
    mask_tokens = mask_token + pos_embed[decoder_keep]
    decoder_input = concat(latents, mask_tokens)
    # Decode, then keep the outputs at the mask-token positions  [§3.2]
    predicted = decoder(decoder_input)[-count(decoder_keep):]
    # Loss only on cubes hidden by the encoder and kept by the decoder  [eq. 2]
    selected = encoder_mask[decoder_keep]
    # eq. 2 divides by (1 - rho_d) N; the code divides by the actual
    # count of selected cubes  [code: engine_for_pretraining.py#L102-L105]
    loss = mean((predicted[selected] - targets[decoder_keep][selected]) ** 2)
    return loss

Hyperparameters

The table numbers below follow the order in which the LaTeX source defines them. Check them against the PDF before citing them elsewhere.

Architecture: ViT-g tab. 7 §4.1.

Pre-training tab. 10. The code runs differ in some rows (see Paper vs code).

Post-pre-training on K710 tab. 11.

Specific fine-tuning, action classification tab. 9. Columns are the settings in the paper: Kinetics at 16 and 64 frames, Something-Something V2, UCF101 and HMDB51.

Source for all columns: tab. 9. The fine-tuning recipe uses the K710 post-pre-trained model as its starting point app. Specific fine-tuning.

Action detection tab. 12 app. Action detection. AdamW, base learning rate 3e-4, weight decay 0.05, β = 0.9/0.999, batch 128, cosine decay, 2 warmup epochs, 10 epochs, no repeated augmentation, drop path 0.3, flip yes, layer decay 0.9, no clip. Training uses ground-truth person boxes; testing uses boxes from AIA app. Action detection.

Knowledge distillation to ViT-B tab. 16 app. More Results. AdamW, base learning rate 1e-3 (K710) or 5e-4 (SSv2), batch 1024 (K710) or 512 (SSv2), cosine decay, 5 warmup epochs, 100 epochs, drop path 0.1, layer decay 0.75, clip gradient 1.0, temperature 3.0. The student starts from the post-pre-trained ViT-B and minimises KL divergence to the ViT-g teacher app. More Results.

Results

Author-reported. The setup for each row is given in the Setting column. The baseline is the best prior number in the same table of the paper, not a re-run.

Notes: the Kinetics ViT-g row labelled "after K710" uses the same numbers as the K710 row in tab. 5. The matching is inferred from identical values in the two tables. Rows with gray entries in tab. 6 use in-house labeled data and are omitted here.

Paper vs code

Code: https://github.com/OpenGVLab/VideoMAEv2 at 29eab1e8a588d1b3ec0cdec7b03a86cca491b74b, MIT license. Its repository is linked from the paper's abstract.

• Learning rate. The paper gives one base learning rate, 1.5e-4, for batch 8192 tab. 10. The ViT-g pre-training script uses --lr 6e-4 with a 32-clip batch per GPU on 64 GPUs vit_g_hybrid_pt.sh. The code scales the rate by total_batch/256, where total_batch = 32 × 64 = 2048 and the repeated-augmentation factor is not included [code: run_mae_pretraining.py#L302-L357]. Both routes give an effective rate of 4.8e-3 (6e-4 × 2048/256 and 1.5e-4 × 8192/256). The ViT-H script uses --lr 1e-3, which gives 8e-3 under the same scaling vit_h_hybrid_pt.sh. The paper does not report a different rate for ViT-H.
• Warmup. The paper lists 120 warmup epochs tab. 10. Both pre-training scripts pass --warmup_epochs 30 vit_g_hybrid_pt.sh.
• Epochs. The paper says 1200 epochs tab. 10. The scripts pass --epochs 300 vit_g_hybrid_pt.sh, and the log directory is named 1200e vit_g_hybrid_pt.sh. This card cannot tell whether the authors ran 1200 epochs with a different flag.
• Weight decay. The code default is 0.05 run_mae_pretraining.py, which matches the paper.
• Visible-token count. The appendix gives ⌊256(1−ρ)⌋ = 25 visible cubes per frame tab. 7. The code's tube generator masks int(0.9 * 256) = 230 cubes per frame, which leaves 26 visible masking_generator.py. The window is 8 × 16 × 16 run_mae_pretraining.py.
• Decoder mask and encoder-visible tokens. The decoder mask is generated independently of the encoder mask. The decoder places a mask token at every position kept by M_d, including positions the encoder also saw [code: models/modeling_pretrain.py#L340-L356]. The paper's eq. 1 does not say whether M_d is restricted to encoder-hidden cubes.
• Loss normalisation. Eq. 2 divides by (1−ρ^d)N eq. 2. The code divides by the number of selected cubes engine_for_pretraining.py.
• Target normalisation. The paper says "normalized" without detail §3.1. The code normalises each cube per channel over its own pixels [code: engine_for_pretraining.py#L77-L88].
• Precision. The appendix says the encoder uses FP16 and the decoder uses FP32 app. Pre-training. The code wraps the whole forward pass in torch.cuda.amp.autocast() and does not cast the decoder to FP32 [code: engine_for_pretraining.py#L107-L108].
• Giant backbone. MLP ratio 48/11 gives a hidden size of 6144, matching tab. 7 modeling_pretrain.py. The decoder has 4 blocks (script --decoder_depth 4) and an MLP ratio of 4, which gives 2048 vit_g_hybrid_pt.sh modeling_pretrain.py.
• Sampling stride. The code sets stride 2 for SSv2 [code: dataset/pretrain_datasets.py#L212-L214], as the appendix states. The code comments that frames decoded at 12 fps need stride 2, and that 24 fps would need stride 4 pretrain_datasets.py.
• K400 specific fine-tuning. The paper's Table 9 K400 column lists drop path 0.3, warmup 0, weight decay 0.05 and epochs 3 tab. 9. The K710-to-K400 script, which starts from the K710 post-pre-trained checkpoint, uses drop path 0.25, warmup 1, weight decay 0.1 and epochs 3 at lr 1e-5 [code: scripts/finetune/vit_g_k710_it_k400_ft.sh#L47-L53].
• K710 post-pre-training batch. The paper lists batch 128 tab. 11. The ViT-g K710 script uses 3 per GPU, two views and a default of 32 GPUs [code: scripts/finetune/vit_g_k710_ft.sh#L14] vit_g_k710_ft.sh. The two counts are not reconciled here, since the paper does not say how repeated views count toward batch size.
• Other rows. Post-pre-training ViT-g settings (lr 1e-3, drop path 0.3, clip 5.0, layer decay 0.9, weight decay 0.1, warmup 5, epochs 35) match tab. 11 vit_g_k710_ft.sh.

Pitfalls

• Schedule mismatch. The paper's pre-training schedule (1200 epochs, 120 warmup epochs, base lr 1.5e-4) differs from the published scripts (300 epochs passed, 30 warmup, base lr 6e-4 or 1e-3). An implementer who uses the table values gets a different run from the released code. Start from the script values or measure the difference first.
• Cube-count arithmetic. The paper's 25 visible cubes per frame come from flooring the visible fraction. The code floors the masked count instead, which leaves 26 visible cubes tab. 7 masking_generator.py. Use the code's generator if you need to match its output.
• Decoder token overlap. Mask tokens are placed on decoder-kept positions without removing those the encoder saw modeling_pretrain.py. Whether this matches the paper's intent is not stated.
• Loss scale. Eq. 2 normalises by a fixed count, while the code normalises by the actual number of selected cubes. The second count depends on the random masks, so the loss scale varies from batch to batch under the code.
• Precision. The paper's FP16/FP32 split is not in the code. Training in full FP16 autocast can differ from the paper's stated setup, and the paper notes precision overflow as a risk app. Pre-training.
• Table 2 timing. The caption says the time is for 1200 epochs on 64 GPUs. Table 1 uses 800 epochs. The ViT-g encoder-only time (356h) is estimated from 5 epochs tab. 2.
• Benchmarks. Table 1 uses a fine-tuning recipe the authors describe as better than VideoMAE's original tab. 1, so its numbers are not comparable to the VideoMAE paper's. Table 6 mixes view counts, input sizes and TFLOPs, so only compare rows with the same setting.
• Data availability. UnlabeledHybrid includes a self-collected Instagram set (about 250k clips) tab. 8. Whether it or the exact clip lists will be released is not stated. The code reads hybrid_train.csv [code: scripts/pretrain/vit_g_hybrid_pt.sh#L8].
• Unstated details. The decoder head count and the post-pre-training GPU count are not stated in the paper.
• Compute. Pre-training ViT-g used 64 80G A100 GPUs app. Pre-training. Dual masking reduces memory from 1753M to 1050M, and the estimated time for the ViT-g run goes from 356h to 241h tab. 2. Repeated augmentation, checkpointing and mixed precision are used throughout app. Pre-training.
• Post-pre-training is not uniformly helpful. It helps ViT-H and ViT-g on K400 but was worse on SSv2 §4.2.
• Appendix citations. The LaTeX source does not state the appendix section letters, so appendix text is cited by section title, for example app. Pre-training.

Lineage

• Builds on: VideoMAE, the masked video autoencoder with tube masking and an asymmetric encoder-decoder §2 §3.1. The arXiv id is not given in the source.
• Builds on: MAE-style masked autoencoding for images §2. Not compiled here.
• Decoder masking takes its running-cell idea from an efficient action-recognition transformer, MAR §3.2. Not compiled here.
• Compared against: MaskFeat, MAE-ST, VideoMAE V1, MViTv2, UniFormer and other methods listed in tab. 6.
• Superseded by: not stated.