VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

In plain words

Training a video transformer from scratch works poorly on small labelled datasets, and the usual shortcut is to start from a model trained on millions of still images. VideoMAE takes a different route: it learns from the videos themselves, without labels. It hides most of each clip and asks the network to fill in what is missing.

Video makes this harder than images. Neighbouring frames look almost the same, so a hidden patch can often be copied from the frame before or after it. To stop that shortcut, VideoMAE hides the same spatial spots in every moment of the clip, so the network cannot find the missing piece in another frame. It also hides a very large share of the clip, so the job is hard enough to force it to learn how objects and motion fit together.

Only the visible part goes through the main network, which keeps training cheaper. A small second network then rebuilds the hidden parts. After this pre-training, the main network is fine-tuned on a labelled dataset and used to recognise actions. The authors report that this works on small datasets without any extra data.

TL;DR

• VideoMAE applies masked autoencoding to video: 16 frames are sampled with a temporal stride, cut into 2×16×16 cubes, and 90% of the cubes are hidden with tube masking. A plain ViT-B encodes only the visible cubes and a 4-block decoder reconstructs the hidden ones with MSE §3.3 §4.2 app. Architectures.
• Without extra data, the author-reported ViT-B results are 81.5% top-1 on Kinetics-400 and 70.8% on Something-Something V2 tab. 7 tab. 6. The abstract's 87.4% and 75.4% come from larger models at more frames tab. 7 tab. 6.
• Pre-training on the unlabelled training split of a small dataset works: 62.6% on HMDB51 (about 3.5k videos) and 91.3% on UCF101 tab. 2.
• The official code is CC BY-NC 4.0. The batch size, flip augmentation and fine-tuning epochs in the scripts differ from the paper (see Paper vs code).

Problem

Vanilla video transformers trained from scratch do poorly unless they are pre-trained on very large data, and most published video transformers start from image-pretrained weights §1. Self-supervised pre-training could avoid this, but video has two properties that make masked reconstruction too easy. Frames are densely sampled and change slowly, so most of the content is redundant §3.2. Adjacent frames also correspond to each other, so a masked cube can be recovered by copying from an unmasked cube in a neighbouring frame, which leaks information and encourages shortcuts that do not transfer §1 §3.2. The paper's question is how to train a vanilla ViT effectively on the video data itself, without pre-trained models or extra image data §1.

Method

VideoMAE follows the image MAE design: an asymmetric encoder-decoder, a very high masking ratio, an encoder that sees only visible tokens, and an MSE loss on the masked positions §3.1. The changes for video are a temporal stride, a spacetime cube embedding, tube masking, and a plain ViT encoder with joint space-time attention §3.3. The paper has no algorithm box. The steps and the pseudocode below are written from §3.3 and the official code.

1. Temporal sampling. Draw a clip of T = 16 frames with stride τ. The paper sets τ = 4 for Kinetics and 2 for Something-Something V2 §3.3 app. Implementation Details. The code takes 16 frames spaced by the stride kinetics.py.
2. Augmentation and normalisation. Apply multi-scale crop to 224×224 and normalise with ImageNet mean and standard deviation. The crop scales are not in the paper's table and are taken from the code tab. 9 datasets.py.
3. Cube embedding. Split the clip into non-overlapping cubes of 2×16×16 and project each to 768 dimensions. A 224×224, 16-frame clip gives 8×14×14 = 1568 tokens §3.3 app. Architectures modeling_finetune.py.
4. Tube masking. Hide a fraction ρ of the 14×14 spatial cells, and hide the same cells at every temporal position. The paper calls this tube masking with the mask shared across time §3.3. The code draws the cells once and tiles them over the 8 temporal positions masking_generator.py.
5. Encoder. A ViT-B (12 blocks, 768 dimensions, 12 heads) runs on the visible tokens only. A fixed sine-cosine position embedding is added before the tokens are dropped. Attention is joint over space and time §3.3 app. Architectures modeling_pretrain.py modeling_pretrain.py.
6. Decoder input. Project the encoder output to 384 dimensions with a linear layer without bias. Append one learnable mask token per hidden cube, and add decoder position embeddings to every token. The order of visible tokens is not restored; positions are attached instead app. Architectures modeling_pretrain.py modeling_pretrain.py.
7. Decoder. Run 4 blocks of width 384 over the full sequence, then keep only the outputs at the hidden positions. A linear head maps each cube to 1536 = 3 × 2 × 16 × 16 values §4.2 app. Architectures modeling_pretrain.py.
8. Target. For each cube, normalise its pixels by subtracting the mean and dividing by the standard deviation over the cube's 512 voxels, separately for each colour channel. This is what the captions of tab. 6 and tab. 7 call "normalized cube pixels". The paper does not say how the normalisation is computed engine_for_pretraining.py.
9. Loss. Mean squared error between prediction and target, over the hidden cubes only. The paper states MSE is the default and beats L1 and smooth L1 §4.2 tab. 1. The code uses a single MSE over all hidden elements engine_for_pretraining.py engine_for_pretraining.py.
10. Fine-tuning. Discard the decoder and fine-tune the encoder on full tokens with the labelled recipe in tab. 10 app. Architectures.

The loss equation in the paper, eq. 1, is the image MAE loss from §3.1. The paper gives no separate equation for VideoMAE's loss.

# One pre-training step on a batch of clips  [§3.3]
# Input:  clips (B, 3, 16, 224, 224); mask ratio rho
# Output: scalar loss over the hidden cubes  [§4.2]

def pretrain_step(clips, rho):
    # Hide rho * 196 cells on the 14x14 grid  [§3.3]
    hidden_cells = sample_cells(196, int(rho * 196))  # (176,) at rho = 0.9
    # Hide the same cells at all 8 temporal positions  [§3.3]
    hidden = tile(hidden_cells, 8)  # (1568,) boolean, True = hidden
    # Target: pixels of each cube, normalised per cube and channel  [§3.3]
    targets = normalise_cubes(split_cubes(clips))  # (B, 1568, 1536)
    # Encode only the visible cubes, with fixed sine-cosine positions  [§3.3]
    tokens = cube_embed(clips) + sincos_pos  # (B, 1568, 768)
    visible = encoder(tokens[:, ~hidden])  # (B, 160, 768) at rho = 0.9
    # Project to decoder width, then append mask tokens for hidden cells
    # [app. Architectures]
    decoder_in = concat(
        project(visible) + decoder_pos[~hidden],
        mask_token + decoder_pos[hidden],
    )  # (B, 1568, 384)
    # Decode, then keep predictions for the hidden cubes only  [§4.2]
    predicted = decoder(decoder_in)[:, -hidden.sum():]  # (B, 1408, 1536)
    # Mean squared error on the hidden cubes  [§4.2] [tab. 1]
    loss = mean((predicted - targets[:, hidden]) ** 2)
    return loss

Hyperparameters

Pre-training values for ViT-B unless marked. "not stated" means the paper does not give it. Code values are the defaults and the scripts for ViT-B run_mae_pretraining.py.

Fine-tuning for the main results (ViT-B, tab. 10):

Fine-tuning values are for the main results; the appendix tables give ViT-S, ViT-L and ViT-H values and the AVA setting tab. 10 tab. 12.

Results

Author-reported. Ablations use ViT-B, 16 frames, 800 pre-training epochs and ρ = 90%, fine-tuned on Something-Something V2 (SSv2) and Kinetics-400 (K400) with the protocols in the table caption tab. 1. The main results use the protocol in the table captions tab. 6 tab. 7.

Pre-training on the target dataset, no labels, ViT-B, 16 frames tab. 2

Transfer from Kinetics-400 to smaller datasets, ViT-B tab. 4

Efficiency, SSv2, ViT-B tab. 3

The paper reports a 3.2× speed-up in wall-clock pre-training time tab. 3.

Main results, Kinetics-400 top-1 (no external data) tab. 7

Main results, Something-Something V2 top-1 (no external data) tab. 6

The paper says the prior state of the art on SSv2 all uses external data for pre-training §4.4. The SSv2 rows above come from models pre-trained for 2400 epochs; the Kinetics-400 rows come from models pre-trained for 1600 epochs tab. 6 tab. 7.

UCF101 and HMDB51, fine-tuned top-1, ViT-B, no external data tab. 13

Action detection, AVA v2.2 mAP tab. 5

Ablations, ViT-B, 16 frames tab. 1 — SSv2 / K400 top-1.

The paper also reports a data-efficiency run. With 42k SSv2 pre-training videos and more iterations, VideoMAE reaches 68.7% on SSv2, above the 68.5% of the model pre-trained on 240k Kinetics videos §4.3 fig. 4. The authors read this as data quality mattering more than quantity under domain shift §4.3.

The paper states that reconstructions remain plausible at ρ = 95% and that the best ratio is extremely high §4.3 fig. 3. Figure values are not given in the text.

Paper vs code

Repository: https://github.com/MCG-NJU/VideoMAE, checked at commit 14ef8d856287c94ef1f985fe30f958eb4ec2c55d. The paper links this repository in its abstract and §1 app. Implementation Details.

• Batch size. The paper gives 1024 tab. 9. The SSv2 and Kinetics ViT-B pre-training scripts use --batch_size 32 on 64 GPUs, which is 2048 in total pretrain.sh pretrain.sh. The UCF101 script uses 24 × 8 = 192, which matches the paper's UCF101 setting pretrain.sh app. Implementation Details.
• Learning rate after scaling. The code scales the learning rate by batch / 256 run_mae_pretraining.py. The Kinetics and SSv2 scripts do not set --lr, so the default 1.5e-4 is used run_mae_pretraining.py. At batch 2048 the effective peak is 1.2e-3. At the paper's stated batch of 1024, the same rule gives 6e-4. Which of these the authors used is not stated.
• Flip augmentation. The paper's table says flip is off for SSv2 and on for Kinetics during pre-training tab. 9. The pre-training transform applies only multi-scale crop, with no flip datasets.py. Flip appears only in the fine-tuning sampler kinetics.py.
• Epochs. The paper says 800 epochs app. Implementation Details. The scripts pass --epochs 801, and the cosine schedule covers those 801 epochs pretrain.sh.
• Default masking ratio. The code defaults to 0.75 run_mae_pretraining.py. The paper's default is 90%, and the Kinetics and SSv2 scripts pass --mask_ratio 0.9 pretrain.sh.
• Fine-tuning epochs. The learning rates match the paper (5e-4 for SSv2, 1e-3 for Kinetics) finetune.sh finetune.sh. The scripts train for 50 and 100 epochs, while the paper's ViT-B values are 40 and 75 tab. 10 finetune.sh finetune.sh. These are example scripts, so the paper's table may still be the intended recipe.
• Random and frame masking. The paper reports them in ablation tab. 1. The dataset builds a generator only for tube masking datasets.py, and the masking code has no frame mode. The code accepts --mask_type random but, as far as this code shows, random masking cannot run through this path.
• Decoder and encoder shapes. The decoder depth, width and encoder sizes in the paper match the code's ViT-B configuration modeling_pretrain.py run_mae_pretraining.py.
• Target normalisation. The paper does not say how the normalisation is computed. The code normalises each cube per channel over its own voxels, with an unbiased variance and an epsilon of 1e-6 engine_for_pretraining.py.

No discrepancies found in the decoder, the encoder, the masking ratio that the scripts pass, the AdamW betas or the weight decay.

Pitfalls

• Learning rate depends on batch size. The code multiplies the base learning rate by batch / 256 run_mae_pretraining.py. Set the total batch first, then read the effective learning rate. The paper's table gives the base value only tab. 9.
• Tube mask granularity. The paper describes the mask as shared across "different frames" §3.3. The code shares it across the 8 temporal token positions, each of which covers 2 frames, so the mask is defined on the 8×14×14 grid masking_generator.py. The two readings differ only in how frames are grouped, but implementers must use the token grid.
• Visible token count. Integer truncation gives 176 hidden cells per frame at ρ = 0.9 and 147 at ρ = 0.75 masking_generator.py. The encoder then sees 160 tokens (ρ = 0.9) or 392 (ρ = 0.75) per 16-frame clip. These counts are derived from the code, not stated in the paper.
• Default flags can be wrong. --mask_ratio defaults to 0.75 run_mae_pretraining.py and --normlize_target is spelt that way in the code run_mae_pretraining.py. Pass both explicitly.
• Headline numbers use larger models. The abstract's 87.4% (Kinetics-400) is ViT-H at 320 resolution with 32 frames, and its 75.4% (SSv2) is ViT-L with 32 frames tab. 7 tab. 6. ViT-B gives 81.5% and 70.8% tab. 7 tab. 6.
• Only the 91.3% and 62.6% use the target dataset's own unlabelled split. The Kinetics-to-UCF101 and Kinetics-to-HMDB51 numbers (96.1, 73.3) use Kinetics-400 pre-training tab. 4 tab. 13.
• Dataset licences. The paper's appendix says Kinetics-400, HMDB51 and AVA are CC BY-NC 4.0, and that the licences of SSv2 and UCF101 are custom app. License of Data. A model pre-trained on those datasets may carry those terms. Check them before using the weights in a product.
• Code licence. The repository is CC BY-NC 4.0 LICENSE. This card contains no code copied from it; the pseudocode is written from the logic. Non-commercial terms apply to the code, so do not copy it into a product.
• Mixed precision and clipping are unstated. The code uses fp16 autocast with a gradient scaler and no clipping by default engine_for_pretraining.py run_mae_pretraining.py. The paper says nothing about either.
• Table and figure numbers. The paper has no \appendix command, so the appendix numbers are not labelled A, B, and so on. The table and figure numbers above are counted from the order in the LaTeX source. Check them against the PDF before citing them elsewhere.
• Compute. Pre-training is the expensive stage: 19.5 hours on 64 V100s for ViT-B on SSv2 at 800 epochs tab. 3. The decoder depth raises memory use sharply tab. 1.

Lineage

• Builds on: MAE, the image masked autoencoder, with an asymmetric encoder and a light decoder §3.1 (MAE card). The plain ViT §1. Joint space-time attention from ViViT and Video Swin §3.3.
• Related masked visual modelling: BEiT, BEVT, VIMPAC and MaskFeat §2. VideoMAE is described as the first masked video pre-training method to use a plain ViT and direct pixel reconstruction §2.
• Compared against: MoCo v3 [arXiv:2104.02057], trained from scratch, ImageNet-21k supervised pre-training tab. 1 tab. 2, and the self-supervised video methods in tab. 5 and tab. 13.
• Superseded by: not known from content in this wiki.