Masked image modeling

Overview

Masked image modeling (MIM) takes an image, hides part of it, and trains a network to recover the hidden part. No labels are needed, so the same recipe can pre-train a backbone that is later fine-tuned, probed or used as a frozen feature extractor. The idea mirrors masked language modeling in NLP 2111.09886 §1.

Images make the task harder than text in three ways, according to the SimMIM authors: neighbouring pixels are highly correlated, so a network can fill a hole by copying its surroundings; pixels are low-level signals, not word-like units; and pixels are continuous, whereas masked language modeling predicts discrete tokens 2111.09886 §1. The MAE authors make the redundancy point too, and conclude that a high masking ratio is needed to make the task non-trivial 2111.06377 §1 2111.06377 §3. Earlier MIM methods answered these with extra machinery, such as a pre-trained tokenizer or block-wise masks 2111.09886 §1 2111.09886 §2.

Every variant shares four components: a masking strategy, an encoder, a prediction head and a prediction target 2111.09886 §3.1. The encoder is usually a Vision Transformer or Swin Transformer 2111.09886 §3.1 2111.06377 §3; see vision transformers for that backbone. Two choices separate most variants: what the target is, and whether the encoder processes the hidden positions at all.

Twenty-eight cards feed this page, covering natural images, robot control and navigation, image-text pretraining, text-to-image generation, satellite time series, video, tactile sensing, a generalist that prompts dense vision tasks from example pairs, human skeleton sequences and traffic sensor series. The last one is not an image: it is the only card here that masks a one-dimensional signal. They use different targets, domains and evaluation protocols, so their numbers are not interchangeable (see Contradictions). The variants below are the design choices they make, not one method's settings.

Variants

Raw pixel regression (SimMIM)

Cut the image into a grid of 32 x 32 pixel patches, hide a random subset at ratio 0.6, and replace each hidden token with one learnable mask vector after patch embedding 2111.09886 §3.2 2111.09886 §4.1.1. The encoder processes all tokens, hidden ones included; the MAE variant below does the opposite. A single 1x1 convolution (a linear layer per token) maps each feature to the RGB values of its own pixel block. The loss is the L1 distance on hidden pixels only 2111.09886 §3.4 2111.09886 eq. 1. Card: SimMIM.

The authors report 83.8% ImageNet-1K fine-tuning top-1 for ViT-B, against 83.2% for BEiT 2111.09886 tab. 6. With SwinV2-H at 512 x 512 they report 87.1% 2111.09886 tab. 7. These are author-reported and use ImageNet-1K only, except for the SwinV2-G result 2111.09886 §4.3.

Visible-only encoder (MAE)

Split the image into regular, non-overlapping patches and hide 75% of them at random. The encoder sees only the visible patches. A shallow decoder then receives the encoded visible tokens plus one shared, learned mask token for each hidden position, and predicts the pixels of every patch 2111.06377 §3. The loss is the mean squared error on hidden patches only. The main ImageNet models normalise each target patch by its own mean and standard deviation, while the ablations use raw pixels 2111.06377 §3 2111.06377 tab. 3. After pre-training the decoder is discarded, and the encoder runs on full, unmasked images 2111.06377 fig. 1. Card: Masked Autoencoders Are Scalable Vision Learners.

The ablations, run with ViT-L/16 for 800 epochs, show why the encoder design matters 2111.06377 tab. 1. Putting mask tokens into the encoder costs 3.3x the FLOPs and drops linear probing from 73.5% to 59.6%, while fine-tuning falls only from 84.9% to 84.2% 2111.06377 tab. 1c. A one-block decoder fine-tunes at 84.8%, against 84.9% for eight blocks, but probes at 65.5% against 73.5% 2111.06377 tab. 1a. Random sampling at 75% beats block-wise masks at 50% and 75%, and grid masks at 75%, on both metrics 2111.06377 tab. 1f. On the authors' 128-core TPU setup, the one-block decoder pre-trains ViT-L in 11.6 hours, against 42.4 hours for the mask-token encoder 2111.06377 tab. 2.

Author-reported ImageNet-1K results with IN-1K data only reach 86.9% for ViT-H at 224 px and 87.8% at 448 px 2111.06377 tab. 3. Linear probing is weaker: 68.0, 75.8 and 76.6 for ViT-B, L and H 2111.06377 app. B. The authors argue that linear probing is a poor proxy for transfer, and they favour partial or full fine-tuning 2111.06377 app. B 2111.06377 §4.3. Tuning a single ViT-L block lifts linear probing from 73.5% to 81.0% 2111.06377 §4.3.

Per-image masking over time and bands (SatMAE)

SatMAE applies the MAE recipe to satellite data with a time axis or many spectral bands. A sequence of images of one place shares one patch embedding, so a sequence of length T gives L x T tokens 2207.08051 §4.1. Each token gets a positional code and a temporal code built from the year, month and hour of its image. The two codes together fill the embedding width 2207.08051 §4.1.1 2207.08051 eq. 2. Masking is drawn separately for each image, so the hidden locations differ across the sequence. The paper contrasts this with consistent masking, where every image hides the same locations, as in VideoMAE 2207.08051 §4.1.2. The temporal experiments use T = 3 and a mask ratio of 0.75 2207.08051 app. A.3.2.

For multi-band Sentinel-2 input, the 13 bands are reduced to 10 by dropping B1, B9 and B10. The rest are split into three groups of similar wavelength and resolution. Each group has its own patch embedding, so G x L tokens reach the encoder, and a sinusoidal group code is concatenated to the positional code 2207.08051 §4.2 2207.08051 §5.4 2207.08051 eq. 3. Splitting the bands into ten single-band groups gave 1,440 tokens, a large memory footprint, and training that often produced NaN loss 2207.08051 app. A.3.3.

On fMoW, the temporal ablation reaches 79.90 top-1 with temporal encoding and independent masking, against 78.45 with temporal encoding and consistent masking. These two rows also differ in cropping, so the masking effect is not isolated 2207.08051 §5.3. The full temporal model reaches 81.49 with test-time augmentation and 79.69 without it, against 78.90 for MAE on three copies of the image with the same augmentation 2207.08051 §5.3. Its non-temporal fMoW RGB result is 77.84 fine-tuned and 65.94 linear-probed, against 71.55 and 68.32 for GASSL, a ResNet50 method. The linear probe therefore favours GASSL 2207.08051 §5.2. Card: SatMAE.

Tube masking on a plain video ViT (VideoMAE)

VideoMAE applies the MAE recipe to clips of 16 frames. Frames are taken with a temporal stride of 4 for Kinetics and 2 for Something-Something V2. Each clip is cut into non-overlapping cubes of 2 x 16 x 16 pixels, which gives 1,568 tokens at 224 px 2203.12602 §3.3. The mask is a tube: a share of the spatial cells is hidden, and the same cells are hidden at every time position, so a missing cube cannot be copied from the next frame. The default share is 90% 2203.12602 §4.2 2203.12602 tab. 1.

A plain ViT-B encodes only the visible cubes, and a four-block decoder of width 384 reconstructs the hidden ones 2203.12602 §3.3 2203.12602 §4.2. The loss is mean squared error on the hidden cubes, against each cube's pixels normalised per colour channel over its own voxels 2203.12602 code: engine_for_pretraining.py#L43-L48. The paper's captions name this target but do not give the formula, and it reports MSE ahead of L1 and smooth L1 2203.12602 tab. 1 2203.12602 tab. 6.

On UCF101 and HMDB51, pre-trained on each dataset's own unlabelled split, ViT-B reaches 91.3% and 62.6%. MoCo v3 reaches 81.7% and 39.2% on the same split, and training from scratch reaches 51.4% and 18.0% 2203.12602 tab. 2. On Something-Something V2 the same recipe gives 69.6%, against 54.2% for MoCo v3. VideoMAE takes 800 epochs and 19.5 hours on 64 V100 GPUs, and MoCo v3 takes 300 epochs and 61.7 hours 2203.12602 tab. 3.

In the paper's ViT-B ablation on Something-Something V2, tube masking at 90% scores 69.6%, random masking at 90% scores 68.3%, and tube masking at 75% scores 68.0%. Hiding 14 of the 16 frames drops the score to 61.5% 2203.12602 tab. 1. The main ViT-B results without external data are 81.5% on Kinetics-400 and 70.8% on Something-Something V2, at 16 frames 2203.12602 tab. 7 2203.12602 tab. 6. The 87.4% and 75.4% in the abstract use larger models at 32 frames, so do not compare them with ViT-B rows 2203.12602 tab. 7 2203.12602 tab. 6.

The released code builds only tube masks, so the random-mask row does not appear to be reproducible from it 2203.12602 code: datasets.py#L21-L24. The code is CC BY-NC 4.0, and the Kinetics-400, HMDB51 and AVA datasets carry the same non-commercial terms, so check those before reusing weights 2203.12602 app. License of Data. Card: VideoMAE.

Dual masking for video (VideoMAE V2)

VideoMAE V2 masks twice. The encoder works as in VideoMAE above. A clip is split into space-time cubes, 90% of them are hidden with tube masking, and only the remaining ~10% are encoded 2303.16727 §3.1 2303.16727 tab. 10. The decoder then receives the encoder output and a learnable mask token for each position kept by a second mask. That mask is running-cell masking at 50%. The decoder reconstructs only those positions, and the loss covers cubes that the encoder hid and the decoder kept 2303.16727 §3.2 2303.16727 eq. 1 2303.16727 eq. 2. Running-cell masking was picked over frame and random masks in the paper's ablation. The authors argue that tube masking in the encoder prevents information leaking across time, while the decoder mask should keep diverse cubes so that little information is lost 2303.16727 tab. 1 2303.16727 §4.2.

The reason for the second mask is cost. On ViT-B with Something-Something V2 pre-training, dual masking gives 70.15 top-1 at 25.87G FLOPs. Encoder-only masking gives 70.28 at 35.48G 2303.16727 tab. 1. On ViT-g, memory falls from 1,753M to 1,050M and speed rises 1.48x 2303.16727 tab. 2. At ViT-g scale the model has 1,011M parameters and is pre-trained on 1.35M unlabeled clips. Supervised post-pre-training on a 710-class Kinetics mix follows, then fine-tuning on each target 2303.16727 §4.1 2303.16727 §3.3. Author-reported, with ViT-g: 90.0% top-1 on Kinetics-400 and 77.0% on Something-Something V2, against 75.5 for MAE-ST. The baselines use different view counts and input sizes 2303.16727 tab. 6. Post-pre-training on the labeled mix helps on Kinetics-400 but was worse on Something-Something V2 2303.16727 §4.2. Card: VideoMAE V2.

Masked video encoder fused with a video-text encoder (InternVideo)

InternVideo trains two encoders separately and fuses them afterwards. The first is a masked video ViT-H. The clip is cut into 3D patches, 90% of them are hidden by tube masking, and an asymmetric encoder-decoder reconstructs the hidden content 2212.03191 §3.1 2212.03191 §4.2. The second is a CLIP-initialised video-text encoder, trained with a contrastive loss and a caption decoder 2212.03191 §3.1. Both get supervised post-pretraining on Kinetics-710 2212.03191 §3.2. Both backbones are then frozen, and a cross-model attention module with a learned linear combination of the two prediction scores is trained on top 2212.03191 §3.3. Card: InternVideo.

The fused model is what the results measure. The card does not report the masked encoder on its own, so the contribution of the masking step is not isolated 2212.03191 tab. 2. Author-reported, the 1.3B headline model reaches 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2 2212.03191 tab. 2 2212.03191 tab. 3. VideoMAE V2's ViT-g reports 90.0% and 77.0% on the same two sets 2303.16727 tab. 6. The models differ in size, data and fusion, so these are not matched results. The learned fusion is not in the released code. The repository's ensemble script only averages per-clip scores from two sets of saved predictions 2212.03191 code: InternVideo1/Pretrain/VideoMAE/ensemble.py#L8-L75.

Masked feature alignment in a state-space video model (VideoMamba)

VideoMamba replaces attention with bidirectional Mamba blocks, a linear-time state-space layer 2403.06977 §3.1. Its masked pretraining follows UMT. The student is trained from scratch on video, hides 80% of the video tokens, and aligns the final outputs of the unmasked tokens to CLIP ViT-B features. Only the final layer is aligned, where UMT aligns several 2403.06977 §3.4 2403.06977 §4.2. Card: VideoMamba.

The target here is a CLIP feature, not pixels or codes, and the loss sits on the visible positions. That is the reverse of the hidden-position loss in most variants above, so check the reading against the paper before reuse. The paper does not say whether masked tokens are dropped or replaced by a learned token, and the row and attention masks are described only in words 2403.06977 §3.4 2403.06977 fig. 5. Check the masked-pretraining code before implementing either.

Author-reported results: on Kinetics-400 at 16 x 224 px, the masked VideoMamba-M reaches 83.4% with 800 epochs and the CLIP teacher, against 81.5% for VideoMAE-B at 1600 epochs 2403.06977 tab. 3. At 8 x 224 px the masked row reaches 82.0%, against 85.7% for UMT-B, so the comparison depends on the setting 2403.06977 tab. 3. On Something-Something V2 at 16 x 288 px, the masked row reaches 71.4% against 70.8% for VideoMAE-B 2403.06977 tab. 4. In the authors' SSv2 ablation, 80% is the best of four mask ratios 2403.06977 tab. 5. Tube masking is the lowest of five mask types, at 66.3%, against 68.5% for attention masking 2403.06977 tab. 5.

Classification targets instead of pixels

Instead of regressing colour values, the SimMIM paper also trains with discrete targets: iGPT's 512 colour clusters, BEiT's discrete visual tokens (from a separately trained dVAE) and equal-width colour bins 2111.09886 §3.4. On the ablation setup these reach ImageNet top-1 between 82.3% and 82.7% at 48 x 48 targets, while the L1 pixel target reaches 82.8% at full resolution 2111.09886 tab. 5. The gaps are a few tenths of a point, and the paper reports no variance over seeds 2111.09886 §4.1.2. BEiT's dVAE pre-training is not counted in its cost figure 2111.09886 tab. 6.

CAE compares its own targets under its regressor-and-decoder design, for ViT-B at 1600 epochs. Raw RGB pixels give 72.4 linear, 77.0 attentive and 50.4 ADE20K mIoU. The d-VAE token gives 71.4, 77.4 and 50.1, and the DALL-E token gives 70.4, 77.1 and 50.2. The authors conclude that the target matters little 2202.03026 tab. 8. Here the pixel target is the best linear probe, which is the opposite of the ordering that the discrete-token papers above would suggest.

Discrete token targets (MAE-VQGAN, Unified-IO, MAGVIT, FLAVA, Muse)

Here the target is an index into a fixed codebook, so the output is a softmax over codes rather than a regression. Five cards use this, for different goals.

MAE-VQGAN is a ViT-L/16 masked autoencoder whose decoder predicts one of 1024 codes per patch. The codes come from a frozen, ImageNet-trained VQGAN, and the cross-entropy is computed on hidden patches only 2209.00647 §3.1 2209.00647 §4.1. Its pre-training set is 88,645 figures from arXiv computer-vision papers 2209.00647 §3.3. Card: Visual Prompting via Image Inpainting. Its use at test time is covered below.

Unified-IO maps image inputs to 16 x 16 patches of a 384 x 384 image, and image outputs to codes from a 16,384-entry VQGAN codebook 2206.08916 §3.1 2206.08916 §3.2. Masked image denoising masks 75% of the patches and asks for the whole image back, following BEiT and MAE. It is one of two denoising objectives in pre-training 2206.08916 §3.3. The card's ablations hold out task groups rather than this objective, so its own contribution is not measured 2206.08916 tab. 5. Card: Unified-IO.

MAGVIT is a video model. A 3D VQ tokenizer maps a 16-frame 128 x 128 clip to 4 x 16 x 16 = 1024 tokens from a 1024-entry codebook, and a BERT-style transformer is trained on those tokens with masked token modeling 2212.05199 §3.1 2212.05199 §3.2. Known frames or regions are tokenised and written into the masked positions, so one loss covers ten tasks 2212.05199 §3.2 2212.05199 app. B.1. The masking rate follows a cosine schedule, and generation takes 12 parallel steps 2212.05199 app. B.2 2212.05199 alg. 1. In its BAIR ablation, the 12-step model reaches debiased FVD 48, against 56 for a 1024-step autoregressive model on the same tokens 2212.05199 §4.5. Card: MAGVIT. Here masking is a generation mechanism, not only a pre-training task.

FLAVA uses the same kind of target inside a joint image-text model. Its unimodal masked image loss runs on unpaired images and predicts dVAE token ids for block-masked patches, in the BEiT style 2112.04482 §3.3. A masked multimodal loss uses the same block-masked image tokens on image-text pairs, and the fused encoder predicts them there 2112.04482 §3.2. Masked image modeling is one of five pre-training losses, alongside contrastive, masked language and matching terms. Its ablation row for masked image modeling alone reports only a vision average, so its effect on multimodal scores is not shown 2112.04482 tab. 3. Card: FLAVA.

Muse is a text-to-image generator built on the same masked-token objective. A VQGAN maps images to codes from an 8192-entry codebook, and a transformer is trained to predict randomly masked codes, with cross-entropy at the masked positions 2301.00704 §2.2 2301.00704 §2.3. The masking rate for each example is drawn from a fixed arccosine density, so most training steps hide most tokens 2301.00704 §2.6. Generation starts with every code masked and fills them in parallel over a few steps, keeping the most confident codes on each pass 2301.00704 §2.8. A frozen T5-XXL encoder supplies the text, with classifier-free guidance at sampling time 2301.00704 §2.1 2301.00704 §2.7. Author-reported: CC3M FID 6.06 for the base model with super-resolution 2301.00704 tab. 1, zero-shot COCO FID 7.88 for the 3B model 2301.00704 tab. 2, and 0.5 s per 256 px batch against 9.1 s for Imagen, both timed on internal TPU-v4 hardware with the batch size not stated 2301.00704 tab. 3. No code or weights are released 2301.00704 §5. Card: Muse.

Heavier prediction heads

A two-layer MLP or an inverse Swin block as head gives roughly the same ImageNet accuracy as the linear head, at 1.2x to 2.3x the training cost 2111.09886 tab. 2. Heavier heads reach a slightly lower pre-training loss but transfer worse to downstream tasks 2111.09886 §4.1.3. MAE makes the same kind of point from the decoder side: decoder depth changes what the features are good for, so choose it for the downstream protocol 2111.06377 tab. 1a.

Full-image reconstruction

Adding a loss on visible pixels, so the model also reconstructs what it can see, drops fine-tuning accuracy from 82.8% to 81.7% 2111.09886 tab. 4. The paper's visualisations of the visible area show artifacts because that area is never trained 2111.09886 §4.4. The MAE card also reports that leaving visible patches out of the loss helps slightly, by about 0.5% 2111.06377 §3.

Self-distilled targets from an EMA teacher (iBOT)

iBOT keeps the masked-patch objective but replaces pixel or token targets with the output of a teacher. The teacher is an exponential moving average (EMA) of the student and sees the unmasked views. For each hidden patch, the student's distribution is trained to match the teacher's distribution at the same position 2111.07832 §3.1 2111.07832 alg. 1. The teacher output is a softmax over K = 8192 dimensions, not a one-hot label 2111.07832 §3.2 2111.07832 tab. 18. Both teacher outputs are centred and sharpened before the cross-entropy 2111.07832 §3.1. Card: iBOT.

A second loss on the [CLS] token compares two views of one image, and the two losses are summed with weight 1 each 2111.07832 §2.2 2111.07832 §3.2. In the paper's own ablation (ViT-S/16, 300 epochs, no multi-crop), removing the [CLS] loss drops k-NN accuracy from 69.1 to 9.5 and linear probing from 74.2 to 29.8, while fine-tuning stays at 79.4 against 81.5 2111.07832 tab. 9. The patch loss alone does not give usable frozen features in that setting.

The reported results are strong on frozen features and on fine-tuning. For ViT-B/16 the authors report 79.5% linear probing and 77.1% k-NN, both under the DINO protocol 2111.07832 tab. 1 2111.07832 §4.1. For ViT-B/16 they report 84.0% fine-tuning on ImageNet-1K, against 83.4% for BEiT at 800 epochs in the same table 2111.07832 tab. 2. On COCO with Cascade Mask R-CNN, ViT-B/16 reaches 51.2 box AP, against 50.1 for DINO 2111.07832 tab. 6. The ablation also shows a trade-off: a frozen DINO model used as the tokenizer gives higher fine-tuning accuracy (81.7) than iBOT's own targets (81.5), while iBOT is ahead on k-NN and linear probing 2111.07832 tab. 9.

The cost is higher than DINO alone. For ViT-S/16 with multi-crop over 800 epochs, the authors report 193.4 h for iBOT against 180.0 h for DINO, on two 8-GPU machines at batch size 1024. Peak memory is 19.5 GB against 15.4 GB 2111.07832 tab. 19.

Averaged block targets from an EMA teacher (data2vec)

data2vec uses the same student-teacher layout as iBOT, but the target is not a single top-layer output. The teacher encodes the unmasked input, its top K Transformer blocks are each normalised and then averaged into one target per patch, and the student regresses those targets at the masked positions with a smooth L1 loss 2202.03555 §3.3 2202.03555 §3.4. For vision the paper uses K = 6 and beta = 2 2202.03555 §4.1. Blocks are masked in groups of at least 16 patches, following BEiT, at 60% of patches 2202.03555 §4.1. Card: data2vec.

The same objective is used for speech and text, with modality-specific input encoders and masks 2202.03555 §3.1 2202.03555 §3.2. This page covers the vision side only.

The authors' ablation sweeps K on ViT-B. Vision top-1 rises from 83.0 at K = 1, the BYOL-style top-layer target, to 84.0 at K = 3, and then stays between 83.6 and 83.9 2202.03555 fig. 2. The gain from averaging is therefore small for vision. The paper also varies how much of the input the teacher sees. Vision top-1 is 77.0 when the teacher sees 1% of the input and 83.8 with full context 2202.03555 fig. 3. The author-reported ImageNet-1K result for a single ViT-B is 84.2, against 83.8 for SimMIM and iBOT and 83.6 for MAE in the same table 2202.03555 tab. 1. For a single ViT-L, the paper reports 86.6 2202.03555 tab. 1.

Separate regressor and token decoder (CAE)

CAE splits the pretext task into three modules: an encoder, a regressor and a decoder 2202.03026 §3.1 2202.03026 fig. 1. The encoder sees only the visible patches, as in MAE. A cross-attention regressor predicts the encoder's representation of each hidden patch from the visible tokens. Its queries are one learned mask token per hidden position, plus that position's embedding. A decoder then maps each predicted hidden representation to a discrete token 2202.03026 §3.1. Half of the 16 x 16 patches are hidden, in random blocks as in BEiT 2202.03026 §3.2. Only the encoder is kept afterwards 2202.03026 §3.1. Card: Context Autoencoder.

The loss has two terms. One is cross-entropy on the decoder's tokens. The other is a mean-squared error between the regressor's output and the encoder's own representation of the hidden patches, weighted by 2. That representation is taken with gradient stopped, so it acts as a target 2202.03026 eq. 1 2202.03026 §3.2. The authors report that training without the reconstruction term collapses to a trivial solution 2202.03026 §5.5.

The ViT-B ablation at 300 epochs separates the two terms 2202.03026 tab. 5. With neither, linear probing is 60.3. The decoder alone gives 63.1, the alignment term alone gives 62.0, and both together give 64.1, at 1.24x the training time of the baseline. Author-reported ViT-B results at 1600 epochs are 71.4 linear, 77.4 attentive and 83.9 fine-tuning 2202.03026 tab. 1. These are below iBOT's 79.5 linear and 79.8 attentive in the same table, so CAE does not lead on frozen features there.

Masked patch loss added to a vision-language model (METER)

METER is a transformer-only image-text model. Its default pre-training uses masked language modeling and image-text matching 2111.02387 §3.3. The paper also tests two masked image losses, each hiding 15% of the image patches 2111.02387 §3.2. One asks the model to pick the original patch out of all patches in the batch, using a softmax over dot products with its output. The other replaces each patch with a token from a VQ-VAE codebook and predicts that token 2111.02387 §3.2 2111.02387 eq. 1. Card: METER.

The released code at the card's commit contains neither loss 2111.02387 code: meter/modules/objectives.py, so the results with these losses cannot be reproduced from it. In the paper's ablation, adding either loss lowers VQAv2 test-dev from 74.98 to 74.01 (in-batch) or 74.21 (discrete codes). Flickr zero-shot image retrieval R@1 falls from 66.08 to 62.12 and 59.80 2111.02387 tab. 7. The authors read this as a sign that conclusions from region-based models may not carry over to patch-based models 2111.02387 §4.4.

SimMIM as a stage before supervised scaling (Swin Transformer V2)

Swin Transformer V2 uses SimMIM as a pretraining stage before supervised training, at a size of 3 billion parameters 2111.09883 §3.4 2111.09883 §4.2. The 3B model is first pretrained on about 70 million ImageNet-22K-extension images with the masked objective, then trained again with labels on the same images. The authors report this uses 1/40 of the labelled images used by JFT-3B 2111.09883 §3.4 2111.09883 §4.2. The masked stage for the 3B model runs for 20 epochs at 192 px, with batch 9216 2111.09883 app. A2.

The paper's headline results come from the full recipe. They include 84.0% ImageNet-V2 top-1 with 640 px fine-tuning, 63.1 COCO test-dev box AP and 59.9 ADE20K mIoU, all author-reported 2111.09883 tab. 2 2111.09883 tab. 3 2111.09883 tab. 4. The ablations in the card vary the normalisation, attention and position bias, not the pretraining stage, so they do not show how much SimMIM contributes at this scale 2111.09883 tab. 6 2111.09883 tab. 7 2111.09883 tab. 8. Card: Swin Transformer V2.

The masked stage is what makes the 3B recipe hard to reproduce. The 70 million image set is private, and no configs for the H or G model are in the repository (see Contradictions).

Prototype matching without pixels (MSN)

MSN does not reconstruct anything. An unmasked target view is passed through an EMA teacher, and its soft assignment over 1024 learned prototypes (dimension 256) is the target for a masked anchor view 2204.07141 §3 2204.07141 app. A.1. The target is sharpened, and a Sinkhorn step is applied over the batch. An entropy term rewards using all prototypes 2204.07141 app. A.1 2204.07141 eq. 1. The paper proves a no-collapse property when the sharpened target is not uniform 2204.07141 app. B. Each image has one large anchor, with random patch dropping, and ten small focal crops that are not masked 2204.07141 §5 2204.07141 app. A.1. The mask ratio is set per architecture, from 0.15 for ViT-S/16 to 0.7 for ViT-L/7 and ViT-B/4 2204.07141 app. A.1. Card: Masked Siamese Networks.

In the low-label regime this is the strongest variant on the cards. ViT-B/4 reaches 72.4% top-1 with 5 labels per class and 75.7% with 1% of labels 2204.07141 tab. 1 2204.07141 tab. 2. At one label per class, ViT-S/16 reaches 47.1%, against 40.4% for iBOT's ViT-S/16 in the same table 2204.07141 tab. 1. Adding focal anchors to random masking raises 1%-label accuracy for ViT-B/16 from 52.3% to 59.8% 2204.07141 tab. 7. The advantage does not carry to full labels. Linear probing at 100% labels gives 76.9% for ViT-S/16, below iBOT's 77.9% and DINO's 77.0% 2204.07141 tab. 3. Full-label fine-tuning of ViT-B/16 reaches 83.4%, against 83.6% for DINO 2204.07141 tab. 4.

Two details need care. The paper's objective includes the entropy term with weight 1.0, but the released ViT-S/16 config sets its weight to zero 2204.07141 eq. 1 2204.07141 code: configs/pretrain/msn_vits16.yaml#L10. The paper also says masking 70% "uses half the computation and memory" 2204.07141 §1, while its own table shows memory falling from 26 GB to 17 GB per GPU and throughput rising from 415 to 600 img/s 2204.07141 tab. 10. With 70% of patches missing at evaluation, DINO ViT-B/16 loses 3.9 points at 1% labels, against 2.4 for MSN ViT-B/16 pre-trained at 0.3 masking 2204.07141 tab. 15.

Test-time masked reconstruction (TTT-MAE)

TTT-MAE does not pre-train a single fixed model for all inputs. For each test image it starts from the pre-trained weights, runs a few SGD steps on masked-patch reconstruction of that image, predicts on the unmasked image, and then discards the adapted weights 2209.07522 §3. The encoder and decoder are a masked autoencoder, and the main head is a ViT-Base classifier trained on top of the frozen encoder 2209.07522 §3 2209.07522 §4.1. Card: Test-Time Training with Masked Autoencoders.

Per test image the paper uses 128 identical copies of the crop, each with its own random 75% patch mask, and 20 SGD steps with momentum 0.9 at learning rate 5e-3 2209.07522 §4.1. The paper's reconstruction equation averages over all patches, while the released code averages over the removed patches only 2209.07522 eq. 2 2209.07522 code: models_mae_shared.py#L265-L282.

On ImageNet-C at severity 5, the authors report that TTT-MAE beats the fixed ViT-probing baseline on all 15 corruptions. Examples from their table: JPEG 51.2 to 63.0, elastic 31.6 to 50.7, and contrast 6.4 to 9.8 2209.07522 tab. 2. On ImageNet-A the paper reports 15.3 to 21.3, and on ImageNet-R 31.3 to 38.9 2209.07522 tab. 3. These are author-reported; no mean over the corruptions is reported in the card.

The method is evaluated with a masked autoencoder, not with SimMIM or iBOT. The paper's theory section covers only linear PCA with a known covariance and an orthogonal corruption, and does not carry over to the deep model by itself 2209.07522 §5.

Frozen features for robot control (MVP)

A ViT-S/16 masked autoencoder is pre-trained on about 700k images of hands and objects, then frozen for the whole RL run 2203.06173 §2.1 2203.06173 §2.2 2203.06173 §4. A small PPO controller reads the classification-token feature, projected to 128 dimensions, together with the joint positions and velocities, and it outputs joint-angle deltas 2203.06173 §4. Card: Masked Visual Pre-training for Motor Control.

The authors report that this beats a supervised ImageNet ViT-S on 7 of 8 PixMC tasks in sample efficiency, and that it comes close to an oracle policy that sees the true object state on 5 of 8 2203.06173 §5.1 2203.06173 fig. 5. Freezing matters in their ablations: with a random frozen encoder, 6 of 8 tasks reach zero or near zero success, and an unfrozen encoder stays at zero on Franka pick for both random and MAE initialisation 2203.06173 §5.3 2203.06173 fig. 9. Each image-based run takes about 5 hours on one 2080 Ti GPU, for about 8 million environment steps 2203.06173 §3.2. The pre-training code is not in the released repository, which holds only the ViT-S weights 2203.06173 code: mvp/backbones/model_zoo.py#L18-L24. Per-task numbers are only in figures, so read them from fig. 5 rather than from the text 2203.06173 §5.

Frozen MAE tokens inside a navigation model (AMR)

Aim My Robot runs a frozen MAE-Base encoder on the current RGB frame and on the reference image of the target object. The resulting tokens are fused with LiDAR and goal tokens in one transformer 2411.14770 §III-B1 2411.14770 §III-B2. The MAE is used only as a feature extractor here. The navigation policy itself is trained by behaviour cloning on simulated trajectories, with DAgger 2411.14770 §IV. Depth is not concatenated to the image; each depth patch is turned into a 3D position, embedded, and added to its matching RGB token 2411.14770 §III-B1. Card: Aim My Robot.

The authors report a 3 cm and 1 degree median error on unseen objects in simulation 2411.14770 §I. Their ablation removes one input at a time and reports collision rates. The full model scores 3.8, against 17.7 without depth and 25.6 without LiDAR. The table does not state its units 2411.14770 tab. II. The main baseline uses a CAD model and a ground-truth occupancy map, which AMR does not have, so the comparison is not like-for-like 2411.14770 §V-A.

Language-conditioned reconstruction (Voltron)

Voltron reconstructs masked patches of a video clip, and the caption enters the same model. The paper does not state a mask ratio; the code uses 0.75 2302.12766 §3.1 2302.12766 code: voltron/conf/models.py#L48. For each example, with probability alpha the caption is the target of a causal decoder, and the encoder sees only a null token. Otherwise the caption is a condition for pixel reconstruction 2302.12766 §3.1 2302.12766 §3.2. The code adds the caption loss with weight 0.5, which the paper's objective does not 2302.12766 code: voltron/conf/models.py#L405. Card: Language-Driven Representation Learning for Robotics.

The tables report the conditioning variant (alpha 0, one frame). On referring-expression grounding (OCID-Ref) it reaches 89.38 total for ViT-S, against 63.30 for R-R3M and 49.58 for R-MVP 2302.12766 tab. 3. On grasp affordance it reaches 85.15 top-1 for ViT-S, against 72.94 for R-MVP 2302.12766 tab. 2. The no-language variant falls to 53.44 on referring grounding 2302.12766 tab. 4. The baselines are the authors' reimplementations on Something-Something-v2, not the released MVP and R3M models 2302.12766 §4. The generation variant appears only in a control figure, where the authors report that it beats single-frame models with low absolute success rates 2302.12766 fig. 5 2302.12766 §5.3.

Masked tactile reconstruction, conditioned on vision (Touch in the Wild)

Here the masked signal is touch. Two fingertip arrays of 12 x 32 pressure sensors are stacked into one 24 x 32 tactile image and colour-mapped. In 95% of samples, 60% to 80% of its 4 x 4 patches are replaced by one learnable mask token 2507.15062 §4.2. A 3-layer CNN embeds the masked tactile image. A CLIP ViT-B/16, unmasked and fine-tuned, embeds the RGB frame. Cross-attention in each direction then lets each embedding attend to the other, and the fused representation is the concatenation of the two 2507.15062 eq. 2 2507.15062 eq. 3 2507.15062 eq. 4. A two-layer MLP with a sigmoid output decodes the full tactile image 2507.15062 §4.2.

The loss is the squared error over the whole tactile image, not only the masked patches 2507.15062 eq. 1. This departs from the MAE convention. The authors pick reconstruction over contrastive alignment, arguing that contrastive alignment suppresses the fine-grained tactile signal 2507.15062 §4.2. The target encoder is an exponential moving average of the online weights, with decay 0.9995 2507.15062 §4.2. Pre-training runs for 5 epochs at batch size 128 on the authors' own collection of over 2,700 handheld demonstrations, recorded in stores and outdoor markets 2507.15062 §1 2507.15062 app. E. Downstream, the pretrained encoder output and proprioception condition a diffusion policy, with no masking 2507.15062 §4.3.

Author-reported, with 20 trials per cell. On the test-tube task (whole task), success is 0.85 with pre-training, 0.70 for the same encoder without pre-training, and 0.25 for vision only 2507.15062 tab. 1. On the whiteboard task, the whole-task numbers are 0.70 with pre-training and 0.60 without. One trial moves a rate by 0.05, so several gaps here are one or two trials 2507.15062 tab. 1. When the task's own demonstrations are removed from pre-training, the test-tube whole-task rate is 0.80, against 0.85 with them 2507.15062 tab. 3. Card: Touch in the Wild.

Entity pseudo-labels on video crops (ALPRO)

ALPRO adds a region-level prediction term to a video-text model. A random crop covering 30% to 50% of the frame is embedded by a frozen prompter and compared with prompts for about 1,000 entity names. The softmax over those similarities is the target, and the main model predicts it from features pooled at the crop's positions, with a soft cross-entropy loss 2112.09583 §3.2.2 2112.09583 eq. 4 2112.09583 eq. 5. The target is a distribution over words, not pixels or codes, and no detector or human label is needed 2112.09583 §3.2.2. Nothing in this description hides the crop from the encoder, so it is closer to region-level prediction than to masked reconstruction. Card: Align and Prompt.

In the ablation, adding this term to masked language modeling and video-text matching raises MSVD-QA from 43.3 to 46.3, and MSRVTT-QA from 40.9 to 41.8 2112.09583 tab. 1. The reported full model reaches 33.9 R@1 on MSRVTT 1k after fine-tuning 2112.09583 tab. 2. The released code applies the 0.2 pseudo-label threshold to the index of the top entity rather than to its score, so it discards only crops whose top entity has index 0 2112.09583 code: src/modeling/alpro_models.py#L527.

Inpainting as an in-context prompt (visual prompting)

MAE-VQGAN can be given a task without changing its weights. An example input-output pair and a query are tiled into a 2 x 2 grid, with the pair in the top row and the query bottom-left. The bottom-right cell is hidden, and the model's predicted codes there are decoded by the VQGAN decoder 2209.00647 §3.1 2209.00647 §3.2 2209.00647 §4.2.

Pre-training data matters for this use. On Pascal-5i segmentation, MAE-VQGAN trained on ImageNet alone scores 2.22 to 7.07 mIoU across the four splits, against 24.25 to 30.44 when trained on figures 2209.00647 tab. 1. Prompt layout matters too: a vertical black-and-white layout reaches 31.57 mIoU, against 27.17 for the horizontal one 2209.00647 §4.4. Adding a vertical layout, and then a vertical layout with its rows swapped, to the horizontal one raises colour-aware mIoU on the synthetic study from 39.97 to 44.14 2209.00647 §4.4. The authors find the results competitive with other pretrained inpainting models, but not with supervised task-specific models 2209.00647 §5 2209.00647 §4.4. For comparison, a labelled one-shot CyCTR reaches 67.2 on split 0 2209.00647 tab. 3.

The released evaluation hides a fixed bottom-right block, while pre-training hides random patches 2209.00647 code: evaluate/mae_utils.py#L51-L57 2209.00647 code: models_mae.py#L123-L148. The paper does not discuss this gap. Card: Visual Prompting via Image Inpainting.

Stitched prompt pairs with a masked output (Painter)

Painter writes the output of each dense task as an RGB image, so depth, segmentation, keypoints and restoration all become pictures 2212.02499 §3.1. It stacks two input-output pairs of the same task and hides output patches with one learned mask token. The loss is smooth L1 on the masked output pixels 2212.02499 §3.2. The paper gives 75% block-wise masking 2212.02499 §3.2, while the released script masks 784 of 1568 patches, about 50% 2212.02499 code: Painter/train_painter_vit_large.sh#L12. At inference a solved pair is placed above the query, and the query's empty output half is painted 2212.02499 §3.3. No head is added for a task. Adding one needs only new training pairs 2212.02499 §3.1. Card: Painter.

Author-reported, with one ViT-L: NYUv2 depth RMSE 0.288, against 0.330 for the best listed specialist. The same model reaches 49.9 ADE-20K mIoU against 57.7, and 72.1 COCO keypoint AP against 77.2 2212.02499 tab. 1. The choice of prompt moves little. ADE-20K mIoU is 49.3 with a random prompt and 49.9 with a searched one, and the paper reports no variance, so that gap may be noise 2212.02499 tab. 3. On open-vocabulary few-shot segmentation the card reports 62.3 mIoU, against 58.3 for MAE-VQGAN, the inpainting method above 2212.02499 tab. 4.

Masked 2D skeleton sequences lifted to 3D (MotionBERT)

MotionBERT applies the masked-recovery recipe to human motion rather than to images. Its input is a sequence of 2D joints with confidence, and the pretext task is to recover 3D motion from a corrupted copy 2210.06551 §3.3. The corruption masks 5% of joints and 10% of frames, and adds noise 2210.06551 app. A.1. Masked entries are multiplied by zero in every channel, with no learned mask token 2210.06551 code: lib/data/augmentation.py#L71-L74. Mocap clips get a 3D position loss plus a velocity term 2210.06551 eq. 6. Video has no 3D target, so the model's 3D estimate is projected back to 2D and compared with the detected joints 2210.06551 eq. 7. The encoder, DSTformer, runs spatial-then-temporal and temporal-then-spatial attention in two streams, and blends them with learned weights 2210.06551 §3.2 2210.06551 eq. 5. Card: MotionBERT.

On Human3.6M with detected 2D poses, the authors report 39.2 mm MPJPE from scratch and 37.5 mm after finetuning 2210.06551 tab. 1. In the cumulative pretraining ablation, adding the masks to noisy 2D-to-3D lifting moves MPJPE from 38.1 mm to 37.4 mm 2210.06551 tab. 5. With a frozen encoder, a random one gives 404.4 mm and a pretrained one 40.3 mm 2210.06551 tab. 4. Pretraining helps finetuning only while the mask ratio stays below about 45% 2210.06551 fig. 7, so MAE's 75% does not carry over.

Masked patches of traffic time series (STEP)

Masked patch reconstruction is not limited to images. STEP applies it to a long history of traffic readings. Each sensor's series is cut into patches, 75% of them are hidden, and a Transformer (TSFormer) is trained to reconstruct the hidden patches, with a mean absolute error over those patches only 2206.09113 §3.1 2206.09113 eq. 1. As in MAE, the encoder sees only the visible patches, and a decoder inserts a shared mask token at the hidden positions 2206.09113 §3.1. The pre-trained encoder is then frozen. Its output supplies a representation of the most recent short patch, and a kNN-based term regularises a sparse graph that is learned from the data, with a Graph WaveNet forecaster on top 2206.09113 §3.2 2206.09113 eq. 5. Card: STEP.

The authors report that STEP is best on MAE, RMSE and MAPE in nearly all cells on METR-LA, PEMS-BAY and PEMS04, at horizons 3, 6 and 12, against 12 baselines 2206.09113 §4.2 2206.09113 tab. 2. The comparison is against forecasting models as a whole, so it does not isolate the masked pre-training stage. The masking ratio here, 75%, is the same as MAE's, but the signal is a series of traffic readings, not pixels.

Multi-temporal satellite data (Prithvi-EO-2.0)

Prithvi-EO-2.0 is a ViT masked autoencoder for sequences of four satellite images, each with six spectral bands at 30 m, in 300M and 600M sizes 2412.02732 §3.1 2412.02732 §3.2. Its changes to image MIM are structural. The patch embedding is 3D with a temporal extent of 1, so each image keeps its own time slice 2412.02732 §3.2. Date and centre location are added as learned-weight biases, and that metadata is dropped with probability 0.1 during pre-training so the model also works without it 2412.02732 §3.2. The mask ratio is not stated in the paper or the repository 2412.02732 §3.2. Pre-training used about 21,000 GPU-hours for the 300M model and about 58,000 for the 600M model, on A100 GPUs 2412.02732 §3.2. Card: Prithvi-EO-2.0.

Downstream, the authors use LoRA on some tasks 2412.02732 §4.2.3 2412.02732 §4.3.3 2412.02732 §4.4.1 and a frozen encoder for gross primary productivity 2412.02732 §4.4.2. Author-reported: the 600M-TL model reaches 90.3 mIoU on Sen1Floods11, against 88.3 for Prithvi-EO-1.0-100M 2412.02732 tab. 4, and 90.5 on wildfire burn scars, against 86.9 2412.02732 tab. 5. Biomass estimation is the exception: the challenge baseline, at RMSE 27.49, beats every Prithvi configuration, whose best is 33.40 2412.02732 §4.4.1 2412.02732 tab. 13. The README describes the metadata encoding differently from the paper, so check a released checkpoint's architecture before reusing the code 2412.02732 code: README.md.

When to use which

Pre-training data beyond ImageNet-1K is not evaluated in the main SimMIM ablations, and its small-dataset behaviour is not stated 2111.09886 §4.1.1. iBOT's ImageNet-22K results show that the best data for linear probing is not the best for k-NN: ViT-L/16 gets 82.3 linear but 72.9 k-NN when pre-trained on 22K 2111.07832 tab. 1.

Implementation sketch

The common core is the same for every variant. Only the target and the loss change, as noted after the block. Pseudocode in the same style as the paper card's algorithm 2111.09886 §3.2 2111.09886 §3.4.

# [arxiv:2111.09886 §3.2] split into tokens, then pick hidden ones
tokens = patch_embed(images)                  # (B, L, C)
hide = random_mask(B, L, ratio=0.6)           # (B, L) bool, True = hidden
# [arxiv:2111.09886 §3.2] swap each hidden token for one learnable vector
tokens = where(hide, mask_token, tokens)
# [arxiv:2111.09886 §3.2] encoder sees every token, hidden ones included
feats = encoder(tokens)                       # (B, L, C')
# [arxiv:2111.09886 §3.4] one linear map per token to its pixel block
pred = linear_head(feats)                     # (B, L, s*s*3)
# [arxiv:2111.09886 §3.4] L1 only over hidden pixels, averaged (eq. 1)
pix_hide = expand(hide, s)                    # (B, H, W), hidden pixels
loss = (abs(pred - target) * pix_hide).sum()
loss = loss / (pix_hide.sum() + 1e-5)

Where the variants differ:

• Encoder input. SimMIM and MAGVIT keep every position in the encoder and put a mask token at each hidden one 2111.09886 §3.2 2212.05199 §3.2. MAE drops hidden patches before the encoder, so the encoder runs on the visible quarter only 2111.06377 §3. VideoMAE drops hidden cubes in the same way 2203.12602 §3.3. MSN drops patches on its large anchor but not on its focal crops 2204.07141 app. A.1. data2vec's student keeps every position and swaps masked ones for a learned mask embedding, as SimMIM does 2202.03555 §3.2. SatMAE encodes only the visible tokens of each image in a sequence, with the mask drawn per image 2207.08051 §3 2207.08051 §4.1.2. VideoMAE V2 encodes about 10% of the cubes, and its decoder sees mask tokens at the positions its own mask keeps 2303.16727 §3.2. Touch in the Wild keeps every tactile position in its CNN input and swaps the masked patches for one learned token 2507.15062 §4.2. InternVideo's masked encoder is described as encoding visible tokens only, since its tube-masked clip is reconstructed by an asymmetric decoder 2212.03191 §3.1. VideoMamba's masked stage does not say whether hidden tokens are dropped or replaced 2403.06977 §3.4. CAE's encoder sees only the visible patches, as in MAE, and its regressor supplies the hidden positions 2202.03026 §3.1. Painter runs the input half and the masked output half in parallel streams, and merges them after three blocks 2212.02499 §3.2. MotionBERT keeps every position and zeroes the masked joints and frames, with no learned token 2210.06551 code: lib/data/augmentation.py#L71-L74. STEP's TSFormer drops hidden patches before its encoder, as MAE does 2206.09113 §3.1.
• Mask grid. The hidden set is drawn on a grid of 32-pixel cells. It is then expanded to the encoder's token grid, which is 4 x 4 for Swin and 16 x 16 for ViT with 16-pixel patches 2111.09886 §3.2 2111.09886 code: data/data_simmim.py#L28-L43.
• Target. Raw pixels or normalised RGB for the default. MAE normalises each patch by its own statistics in its main results 2111.06377 tab. 3. Discrete targets replace the regression target with a softmax over a codebook: 1024 codes for MAE-VQGAN and MAGVIT, 16,384 for Unified-IO 2209.00647 §3.1 2212.05199 §3.1 2206.08916 §3.1. The SimMIM card does not record the exact loss used for its discrete targets, so check the paper before implementing. Touch in the Wild regresses the whole tactile image rather than only its masked patches 2507.15062 eq. 1. VideoMamba's masked stage aligns to CLIP ViT-B features instead of pixels or codes 2403.06977 §3.4. Painter regresses the stitched output pixels with smooth L1 2212.02499 §3.2. MotionBERT regresses 3D joints with a velocity term, and for video compares 2D reprojections instead 2210.06551 eq. 6 2210.06551 eq. 7.
• Teacher and prototype targets. iBOT's target is the EMA teacher's softmax on the unmasked view, not a pixel value. The loss is cross-entropy with centring and sharpening, computed only at masked positions, and a [CLS] loss is added 2111.07832 §3.1 2111.07832 alg. 1. The teacher is updated as an EMA of the student after each step, so it needs a second copy of the network in memory 2111.07832 tab. 19. MSN uses the same kind of EMA teacher, but its target is a prototype assignment, and the anchor is trained by cross-entropy against it 2204.07141 §3. data2vec's target is a regression, not a softmax: the normalised outputs of the teacher's top K blocks are averaged, and the student uses a smooth L1 loss on the masked steps 2202.03555 §3.3 2202.03555 §3.4. CAE's alignment target is the same encoder run on the hidden patches with gradient stopped, not a separate EMA copy 2202.03026 §3.2. The released code uses a second module that is re-pointed at the encoder's tensors after each target pass, so the target follows the encoder's updates 2202.03026 code: models/modeling_cae.py#L98-L102.
• Decoder. MAE's default decoder has 8 blocks at width 512, and it is discarded after pre-training 2111.06377 tab. 1 caption 2111.06377 fig. 1. A one-block decoder is cheaper and fine-tunes as well, but probes worse 2111.06377 tab. 1a. CAE's decoder reads only the regressor's output at the hidden positions, not the visible tokens 2202.03026 §3.1 2202.03026 code: models/modeling_cae_helper.py#L441-L448.
• Per-image adaptation (TTT-MAE). The encoder and decoder are copied from the pre-trained weights for each test input, updated for 20 SGD steps on that input's masked reconstruction loss, used once to predict, and then discarded 2209.07522 §3 2209.07522 §4.1.
• Language input (Voltron). The caption goes through a frozen language encoder and is projected to the encoder width. In a generation example the encoder receives only a null token, and the decoder predicts the caption under a causal mask, so the encoder cannot copy the caption 2302.12766 §3.1 2302.12766 app. B.2.
• Learning rate. The released SimMIM code takes a base learning rate per 512 images and scales it by the total batch size. Copying the paper's batch-2048 value (8e-4) into such a config multiplies the rate by four 2111.09886 §4.1.1 2111.09886 code: main_simmim.py#L218-L229. MAE uses a base rate of 1.5e-4, scaled by batch size divided by 256 2111.06377 app. A.1. Check each paper's scaling rule before copying a rate.
• Masking unit. Whether the mask is drawn per image, per sequence, per batch or per sample changes the training signal. SatMAE's appendix and code differ on per-image versus sequence-wide masking 2207.08051 app. A.3.2 2207.08051 code: models_mae_temporal.py#L122-L167. Touch in the Wild draws its 95% masking decision per batch in code, but per sample in the paper 2507.15062 §4.2 2507.15062 code: pretrain_mae/pretrain_mae_model.py#L405-L424. Choose one and state it.

Contradictions

Not resolved here. Each side is cited, and the choice is left to the reader.

• Linear probing. SimMIM reports 56.7% for ViT-B and states that it does not optimise for linear evaluation 2111.09886 tab. 6 2111.09886 §3.5. Its linear probe takes the best intermediate layer, trained for 100 epochs 2111.09886 §4.2. iBOT reports 79.5% for ViT-B/16 under the DINO protocol, which sweeps learning rates 2111.07832 tab. 1 2111.07832 §4.1. MAE reports 68.0% for ViT-B 2111.06377 app. B, while CAE's table gives 67.8% for ViT-B 2202.03026 tab. 1. CAE's table takes other models' scores from the authors' own implementations unless noted 2202.03026 tab. 1. The protocols differ, so these numbers are not a like-for-like comparison.
• Mask tokens in the encoder. SimMIM feeds every position, hidden ones as a learned vector 2111.09886 §3.2. MAE reports that the same choice costs 3.3x the FLOPs and 14 points of linear probing at ViT-L/16 2111.06377 tab. 1c. The settings differ in patch size, ratio, backbone and epochs, so this is evidence about the MAE setting, not a verdict on SimMIM's. MAGVIT's transformer also takes [MASK] positions by design, because it generates 2212.05199 §3.2.
• Mask ratio. SimMIM's default is 0.6, and its best value in the ablation is 0.5 2111.09886 tab. 1 2111.09886 §4.1.2. MAE's default is 75% 2111.06377 tab. 1. MSN uses 0.15 to 0.7 depending on the architecture, and it drops patches instead of replacing them 2204.07141 app. A.1. VideoMAE uses 90% with tube masks on Kinetics and Something-Something V2, and 75% on UCF101 and HMDB51 2203.12602 tab. 1 2203.12602 app. Implementation Details. The cards do not compare these under one setup. CAE uses 50% block-wise masking. Its ViT-B ablation at 300 epochs prefers 60% on linear probing (64.8 against 64.1 at 50%), and 50% on ADE20K (48.3 against 48.1 at 60%) 2202.03026 tab. 7. MotionBERT masks 5% of joints and 10% of frames 2210.06551 app. A.1, and its finetuning gains stop above about 45% masking 2210.06551 fig. 7. Its skeleton ratio is therefore far from MAE's 75%.
• BEiT ViT-B/16 fine-tuning. SimMIM reports 83.2% 2111.09886 tab. 6. iBOT reports 83.4% for BEiT at 800 epochs 2111.07832 tab. 2. The protocols differ, so cite each number with its own paper.
• Low-label performance of MAE. MSN reports MAE ViT-B/16 at 40.5% with 5 labels per class, using partial fine-tuning, against 61.4% for DINO ViT-B/16 2204.07141 tab. 1. The MAE card emphasises full-label fine-tuning and gives no low-label numbers 2111.06377 tab. 3. Read the two as different regimes.
• Within SimMIM. The ablation's best mask ratio is 0.5 (83.0%), but the default is 0.6, which the authors chose for training stability 2111.09886 tab. 1 2111.09886 §4.1.2. The paper's text and released code also differ on schedules, which affects reproduction 2111.09886 §4.1.1 2111.09886 code: configs/vit_base__800ep/simmim_pretrain__vit_base__img224__800ep.yaml#L17-L26.
• Within TTT-MAE. The text says encoder and decoder are both optimised, with weight decay 0.2 2209.07522 §4.1. The released command freezes the decoder 2209.07522 code: README.md, and its test-time optimiser has no weight decay 2209.07522 code: engine_test_time.py#L64-L65. The appendix reports the decoder choice as negligible 2209.07522 app.. Implement the released setting, or state which one you chose.
• Within iBOT. The ablation uses 300 epochs, no multi-crop and a fixed ratio of 0.3 2111.07832 §4.4. Its headline numbers use 800 to 1600 epochs with multi-crop 2111.07832 tab. 1. Do not compare the ablation numbers with the headline ones. The paper's text and its code also differ on crop scales and the ViT-L/16 pre-training length, which the iBOT card lists in its Paper vs code section 2111.07832 §3.2 2111.07832 app. B.
• Within MSN. The paper says masking 70% "uses half the computation and memory" 2204.07141 §1, but its own compute table does not show a halving 2204.07141 tab. 10. Its ViT-S/16 pre-training length is 800 epochs in one table and 600 in another 2204.07141 tab. 1 2204.07141 tab. 3.
• Within Voltron. The paper writes its objective as an unweighted sum, while the code weights the caption loss by 0.5 2302.12766 §3.2 2302.12766 code: voltron/conf/models.py#L405. The paper also does not state the mask ratio that the code sets to 0.75 2302.12766 §3.1. Implementations that follow the text and those that follow the code train different objectives, and the reported numbers do not say which was used.
• Within ALPRO. The paper drops entity pseudo-labels whose top score is below 0.2 2112.09583 §3.4. The released code applies the threshold to the entity index, so the rule differs from the text 2112.09583 code: src/modeling/alpro_models.py#L527. Pick one rule and state it.
• Within data2vec. The paper's ViT-B recipe uses 60% block masking, K = 6, a target taken after the FFN before the last residual, a constant EMA of 0.9998 and drop path 0.2 2202.03555 §4.1 2202.03555 §3.3. The vision configs in the repository set other values for most of these, including a 75-patch default mask and an annealed EMA 2202.03555 code: examples/data2vec/models/data2vec_vision.py#L37-L40 2202.03555 code: examples/data2vec/config/vision/pretraining/base_imagenet_d2v1.yaml#L57-L64. The paper gives no ablation for the normalisation of targets, so the choice cannot be checked from the text alone 2202.03555 §3.3.
• Within SwinV2. The main text names SimMIM for the 3B model's masked stage, while the appendix cites that method with an anonymous key 2111.09883 §3.4 2111.09883 app. A2. Treat the stage as SimMIM, with details not stated in the card.
• Within SatMAE. The text says the gains are 4.48% over the previous state of the art and 2.06% over the non-temporal model. The tables give other differences, for example 81.49 minus 78.90 = 2.59 over MAE with test-time augmentation 2207.08051 §5.3. Cite the table values, and state which ones you quote. The test-time augmentation also differs, as noted above 2207.08051 app. A.3.2 2207.08051 code: engine_finetune.py#L256-L264.
• Within VideoMAE V2. The paper gives 1,200 epochs, 120 warmup epochs and a base rate of 1.5e-4 for batch 8,192 2303.16727 tab. 10. The ViT-g script passes 300 epochs and 30 warmup epochs, with a rate of 6e-4 for batch 2,048 2303.16727 code: scripts/pretrain/vit_g_hybrid_pt.sh#L37-L49. Both routes give the same effective rate of 4.8e-3, but the schedules differ, so an implementer who uses the table gets a different run from the released code.
• Within Touch in the Wild. The paper gives 60 training epochs for the policy 2507.15062 §5.2, and the default config gives 200 2507.15062 code: diffusion_policy/config/train_diffusion_unet_timm_umi_workspace.yaml#L165. The paper sets the encoder learning rate to 3e-5 throughout, while the code uses 2e-5 for CLIP 2507.15062 §4.3 2507.15062 code: diffusion_policy/workspace/train_diffusion_unet_image_workspace.py#L110-L136. The paper says "two rounds" of cross-attention, and the code runs one step in each direction, with a residual connection 2507.15062 §4.2 2507.15062 code: pretrain_mae/pretrain_mae_model.py#L61-L77.
• Within Painter. The paper gives 75% block-wise masking and says the two stream outputs are added when merged 2212.02499 §3.2. The script masks 784 of 1568 patches, and the code averages the streams 2212.02499 code: Painter/train_painter_vit_large.sh#L12 2212.02499 code: Painter/models_painter.py#L414-L415. The paper's batch of 2048 for 54K iterations does not match the script's 256 per node for 15 epochs 2212.02499 §4.1 2212.02499 code: Painter/train_painter_vit_large.sh#L5-L10. Choose one and state it.
• Within MotionBERT. The paper writes the 3D and velocity losses as sums over frames and joints, while the code uses means 2210.06551 eq. 6 2210.06551 code: lib/model/loss.py#L56-L62. The code also adds a scale term that the paper's total loss does not have 2210.06551 eq. 8 2210.06551 code: train.py#L186-L188. The paper gives Adam with no learning-rate decay, while the code uses AdamW and multiplies the rate by 0.99 after each epoch 2210.06551 app. A.1 2210.06551 code: train.py#L360-L362.
• Mask type in video. VideoMAE V2 uses tube masking in its encoder and argues that it prevents information leaking across time 2303.16727 tab. 1 2303.16727 §4.2. VideoMamba's Something-Something V2 ablation ranks tube masking last of five types 2403.06977 tab. 5. VideoMAE's own ablation puts tube masking ahead of random masking at 90%, 69.6 against 68.3 on Something-Something V2 2203.12602 tab. 1. That agrees with V2 and not with VideoMamba. The two use different architectures and targets, so the rankings do not transfer. Test the mask type on your own model.
• Within InternVideo. The text gives 1200 masked-pretraining epochs at a base learning rate of 2.5e-4, and a decoder with half the encoder's width and 4 blocks 2212.03191 §3.1 2212.03191 §4.2. The released hybrid script runs 301 epochs at 1e-3 with a 0.9 mask ratio and decoder depth 8 2212.03191 code: InternVideo1/Pretrain/VideoMAE/scripts/pretrain/slurm_train_vit_h_hybrid_pt.sh#L29-L44. Its ViT-H model uses a decoder width of 512 against an encoder width of 1280 2212.03191 code: InternVideo1/Pretrain/VideoMAE/modeling_pretrain.py#L464-L469. Implement one of these and state which.
• Within FLAVA. Table 5 gives FLAVA 72.8 on VQAv2 test-dev, 79.0 on SNLI-VE and 76.7 on Hateful Memes. The ablation table gives 72.49, 78.89 and 76.09 for the same model. The paper does not explain the gap 2112.04482 tab. 5 2112.04482 tab. 4.
• Within Muse. The text gives 1M training steps for the models, the base-model table gives 1.5M, and the super-resolution table gives 1M. The paper does not say which applies to the 3B model 2301.00704 §3 2301.00704 tab. 4 2301.00704 tab. 5.
• Masked patch losses in vision-language models. METER finds that adding either masked-patch loss lowers VQAv2 and retrieval scores in its setup 2111.02387 tab. 7. It reads this as a sign that conclusions from region-based models may not carry over 2111.02387 §4.4. CAE's ablation on image-only pre-training finds that its masked objective raises linear, attentive and ADE20K scores over the baseline 2202.03026 tab. 5. The settings differ in model, data and task, so the cards do not settle which result holds for your setup.
• Within CAE. The paper's §3.1 names the visible representations and the previous regressor output as the regressor's keys and values. The code always adds the hidden tokens to them 2202.03026 §3.1 2202.03026 code: models/modeling_cae_helper.py#L362-L366. The Table 1 CAE rows use the d-VAE tokenizer, while the code defaults to DALL-E 2202.03026 tab. 1 2202.03026 code: tools/run_pretraining.py#L28. Choose one of each and state it.

Papers

• SimMIM: A Simple Framework for Masked Image Modeling: pixel regression with a linear head, an L1 loss on hidden pixels and a 32-pixel mask. It has the most ablations on the design choices above.
• Masked Autoencoders Are Scalable Vision Learners: a visible-only encoder with 75% random masking and a light decoder, with ablations on decoder depth and mask tokens.
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training: the MAE recipe on video, with 90% tube masking on a plain ViT-B and pre-training on video alone.
• iBOT: Image BERT Pre-Training with Online Tokenizer: MIM with targets from an EMA teacher, trained jointly with a [CLS] self-distillation loss.
• Masked Siamese Networks for Label-Efficient Learning: prototype matching on a masked view with no pixel reconstruction, aimed at 1 to 5 labels per class.
• Test-Time Training with Masked Autoencoders: per-image MAE reconstruction at test time, with the weights reset after each input.
• Masked Visual Pre-training for Motor Control: a frozen MAE on hand-object images, used as the visual encoder for PPO robot controllers.
• Unified-IO: A unified model for vision, language, and multi-modal tasks: masked image denoising as one of two pre-training objectives in a single encoder-decoder over discrete tokens.
• Visual Prompting via Image Inpainting: an MAE that predicts VQGAN codes, used to solve new image tasks by inpainting a grid prompt.
• FLAVA: A Foundational Language And Vision Alignment Model: one image-text model trained with a BEiT-style masked image loss on dVAE tokens, alongside contrastive, masked language and matching losses.
• Muse: Text-To-Image Generation via Masked Generative Transformers: a masked-token transformer over VQGAN codes that generates images by parallel decoding.
• MAGVIT: Masked Generative Video Transformer: masked token modeling over 3D video tokens, with a 12-step parallel decoder for generation.
• Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications: an MAE for satellite time series, with 3D patches and optional date and location metadata.
• Language-Driven Representation Learning for Robotics: Voltron, a frame-masking autoencoder over video whose caption is either a condition for pixel reconstruction or a generation target.
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts: ALPRO, a video-text model whose extra term predicts entity pseudo-labels on random crops from a frozen prompter.
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language: one self-distillation recipe across modalities, regressing the averaged top K blocks of an EMA teacher on masked steps.
• Swin Transformer V2: Scaling Up Capacity and Resolution: scales Swin to 3B parameters, with SimMIM as the self-supervised stage before labelled training.
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery: an MAE over satellite time series and band groups, with per-image masking and temporal and group codes.
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking: video MAE scaled to ViT-g, with a second, decoder-side mask that cuts compute and memory.
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper: masked reconstruction of a tactile image, conditioned on a CLIP view, used as a pretrained encoder for a diffusion policy.
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning: a masked video ViT-H and a CLIP-initialised video-text encoder, trained separately and then fused with cross-model attention on frozen backbones.
• VideoMamba: State Space Model for Efficient Video Understanding: a bidirectional Mamba video encoder whose masked stage aligns 80%-masked video to CLIP ViT-B features, with self-distillation for scaling.
• An Empirical Study of Training End-to-End Vision-and-Language Transformers: METER, a detector-free vision-language model whose tests of masked patch losses lowered scores in its setup.
• Context Autoencoder for Self-Supervised Representation Learning: CAE, which separates encoding of visible patches, cross-attention regression of hidden features and a token decoder.
• Aim My Robot: Precision Local Navigation to Any Object: a frozen MAE encoder fused with depth, LiDAR and goal tokens in a transformer that predicts waypoints and camera tilt.
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning: one masked-output model over stitched pairs of the same task, which turns dense outputs into RGB images and is prompted with a solved pair.
• MotionBERT: A Unified Perspective on Learning Human Motion Representations: masked 2D skeleton sequences are lifted to 3D by a DSTformer encoder, then finetuned for pose, action and mesh recovery.
• Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting: STEP, masked patch pre-training of a time-series Transformer on traffic history, whose frozen output feeds a graph forecaster.