--- kind: concept slug: masked-image-modeling title: Masked image modeling summary: >- Masked image modeling hides parts of an image and trains a network to recover them, with no labels. Variants differ in the target (pixels, discrete codes, an EMA teacher or prototypes) and in whether the encoder sees the hidden parts. Pixel targets are the cheapest; teacher and code targets cost more, and none wins on every protocol. tags: [self-supervised-learning, masked-image-modeling, vision-transformer, pretraining, self-distillation, scaling, test-time-training, video-generation, video, remote-sensing, robot-learning, navigation, tactile-sensing, vision-language, text-to-image, in-context-learning, pose-estimation] papers: ["2111.09886", "2111.07832", "2209.07522", "2111.06377", "2203.06173", "2204.07141", "2206.08916", "2209.00647", "2212.05199", "2412.02732", "2112.09583", "2302.12766", "2111.09883", "2202.03555", "2207.08051", "2303.16727", "2507.15062", "2212.03191", "2403.06977", "2203.12602", "2112.04482", "2301.00704", "2111.02387", "2202.03026", "2411.14770", "2212.02499", "2210.06551", "2206.09113"] related: [vision-transformer, state-space-model, self-supervised-learning, masked-language-modeling, knowledge-distillation, domain-adaptation, test-time-training, linear-probing, in-context-learning, multi-task-learning, low-rank-adaptation, policy-gradient, vision-language-pretraining, imitation-learning, relative-positional-encoding, positional-encoding, multimodal-fusion, diffusion-models, contrastive-learning, vector-quantization, text-to-image-generation, classifier-free-guidance, robot-simulation, sim-to-real-transfer, visual-prompting, attention-mechanism, time-series-forecasting, graph-neural-network] status: draft updated_at: 2026-11-01 --- ## Overview Masked image modeling (MIM) takes an image, hides part of it, and trains a network to recover the hidden part. No labels are needed, so the same recipe can pre-train a backbone that is later fine-tuned, probed or used as a frozen feature extractor. The idea mirrors masked language modeling in NLP [arxiv:2111.09886 §1]. Images make the task harder than text in three ways, according to the SimMIM authors: neighbouring pixels are highly correlated, so a network can fill a hole by copying its surroundings; pixels are low-level signals, not word-like units; and pixels are continuous, whereas masked language modeling predicts discrete tokens [arxiv:2111.09886 §1]. The MAE authors make the redundancy point too, and conclude that a high masking ratio is needed to make the task non-trivial [arxiv:2111.06377 §1] [arxiv:2111.06377 §3]. Earlier MIM methods answered these with extra machinery, such as a pre-trained tokenizer or block-wise masks [arxiv:2111.09886 §1] [arxiv:2111.09886 §2]. Every variant shares four components: a masking strategy, an encoder, a prediction head and a prediction target [arxiv:2111.09886 §3.1]. The encoder is usually a Vision Transformer or Swin Transformer [arxiv:2111.09886 §3.1] [arxiv:2111.06377 §3]; see [vision transformers](/concepts/vision-transformer) for that backbone. Two choices separate most variants: what the target is, and whether the encoder processes the hidden positions at all. Twenty-eight cards feed this page, covering natural images, robot control and navigation, image-text pretraining, text-to-image generation, satellite time series, video, tactile sensing, a generalist that prompts dense vision tasks from example pairs, human skeleton sequences and traffic sensor series. The last one is not an image: it is the only card here that masks a one-dimensional signal. They use different targets, domains and evaluation protocols, so their numbers are not interchangeable (see Contradictions). The variants below are the design choices they make, not one method's settings. ## Variants ### Raw pixel regression (SimMIM) Cut the image into a grid of 32 x 32 pixel patches, hide a random subset at ratio 0.6, and replace each hidden token with one learnable mask vector after patch embedding [arxiv:2111.09886 §3.2] [arxiv:2111.09886 §4.1.1]. The encoder processes all tokens, hidden ones included; the MAE variant below does the opposite. A single 1x1 convolution (a linear layer per token) maps each feature to the RGB values of its own pixel block. The loss is the L1 distance on hidden pixels only [arxiv:2111.09886 §3.4] [arxiv:2111.09886 eq. 1]. Card: [SimMIM](/papers/2111.09886). The authors report 83.8% ImageNet-1K fine-tuning top-1 for ViT-B, against 83.2% for BEiT [arxiv:2111.09886 tab. 6]. With SwinV2-H at 512 x 512 they report 87.1% [arxiv:2111.09886 tab. 7]. These are author-reported and use ImageNet-1K only, except for the SwinV2-G result [arxiv:2111.09886 §4.3]. ### Visible-only encoder (MAE) Split the image into regular, non-overlapping patches and hide 75% of them at random. The encoder sees only the visible patches. A shallow decoder then receives the encoded visible tokens plus one shared, learned mask token for each hidden position, and predicts the pixels of every patch [arxiv:2111.06377 §3]. The loss is the mean squared error on hidden patches only. The main ImageNet models normalise each target patch by its own mean and standard deviation, while the ablations use raw pixels [arxiv:2111.06377 §3] [arxiv:2111.06377 tab. 3]. After pre-training the decoder is discarded, and the encoder runs on full, unmasked images [arxiv:2111.06377 fig. 1]. Card: [Masked Autoencoders Are Scalable Vision Learners](/papers/2111.06377). The ablations, run with ViT-L/16 for 800 epochs, show why the encoder design matters [arxiv:2111.06377 tab. 1]. Putting mask tokens into the encoder costs 3.3x the FLOPs and drops linear probing from 73.5% to 59.6%, while fine-tuning falls only from 84.9% to 84.2% [arxiv:2111.06377 tab. 1c]. A one-block decoder fine-tunes at 84.8%, against 84.9% for eight blocks, but probes at 65.5% against 73.5% [arxiv:2111.06377 tab. 1a]. Random sampling at 75% beats block-wise masks at 50% and 75%, and grid masks at 75%, on both metrics [arxiv:2111.06377 tab. 1f]. On the authors' 128-core TPU setup, the one-block decoder pre-trains ViT-L in 11.6 hours, against 42.4 hours for the mask-token encoder [arxiv:2111.06377 tab. 2]. Author-reported ImageNet-1K results with IN-1K data only reach 86.9% for ViT-H at 224 px and 87.8% at 448 px [arxiv:2111.06377 tab. 3]. Linear probing is weaker: 68.0, 75.8 and 76.6 for ViT-B, L and H [arxiv:2111.06377 app. B]. The authors argue that linear probing is a poor proxy for transfer, and they favour partial or full fine-tuning [arxiv:2111.06377 app. B] [arxiv:2111.06377 §4.3]. Tuning a single ViT-L block lifts linear probing from 73.5% to 81.0% [arxiv:2111.06377 §4.3]. ### Per-image masking over time and bands (SatMAE) SatMAE applies the MAE recipe to satellite data with a time axis or many spectral bands. A sequence of images of one place shares one patch embedding, so a sequence of length T gives L x T tokens [arxiv:2207.08051 §4.1]. Each token gets a positional code and a temporal code built from the year, month and hour of its image. The two codes together fill the embedding width [arxiv:2207.08051 §4.1.1] [arxiv:2207.08051 eq. 2]. Masking is drawn separately for each image, so the hidden locations differ across the sequence. The paper contrasts this with consistent masking, where every image hides the same locations, as in VideoMAE [arxiv:2207.08051 §4.1.2]. The temporal experiments use T = 3 and a mask ratio of 0.75 [arxiv:2207.08051 app. A.3.2]. For multi-band Sentinel-2 input, the 13 bands are reduced to 10 by dropping B1, B9 and B10. The rest are split into three groups of similar wavelength and resolution. Each group has its own patch embedding, so G x L tokens reach the encoder, and a sinusoidal group code is concatenated to the positional code [arxiv:2207.08051 §4.2] [arxiv:2207.08051 §5.4] [arxiv:2207.08051 eq. 3]. Splitting the bands into ten single-band groups gave 1,440 tokens, a large memory footprint, and training that often produced NaN loss [arxiv:2207.08051 app. A.3.3]. On fMoW, the temporal ablation reaches 79.90 top-1 with temporal encoding and independent masking, against 78.45 with temporal encoding and consistent masking. These two rows also differ in cropping, so the masking effect is not isolated [arxiv:2207.08051 §5.3]. The full temporal model reaches 81.49 with test-time augmentation and 79.69 without it, against 78.90 for MAE on three copies of the image with the same augmentation [arxiv:2207.08051 §5.3]. Its non-temporal fMoW RGB result is 77.84 fine-tuned and 65.94 linear-probed, against 71.55 and 68.32 for GASSL, a ResNet50 method. The linear probe therefore favours GASSL [arxiv:2207.08051 §5.2]. Card: [SatMAE](/papers/2207.08051). ### Tube masking on a plain video ViT (VideoMAE) VideoMAE applies the MAE recipe to clips of 16 frames. Frames are taken with a temporal stride of 4 for Kinetics and 2 for Something-Something V2. Each clip is cut into non-overlapping cubes of 2 x 16 x 16 pixels, which gives 1,568 tokens at 224 px [arxiv:2203.12602 §3.3]. The mask is a tube: a share of the spatial cells is hidden, and the same cells are hidden at every time position, so a missing cube cannot be copied from the next frame. The default share is 90% [arxiv:2203.12602 §4.2] [arxiv:2203.12602 tab. 1]. A plain ViT-B encodes only the visible cubes, and a four-block decoder of width 384 reconstructs the hidden ones [arxiv:2203.12602 §3.3] [arxiv:2203.12602 §4.2]. The loss is mean squared error on the hidden cubes, against each cube's pixels normalised per colour channel over its own voxels [arxiv:2203.12602 code: engine_for_pretraining.py#L43-L48]. The paper's captions name this target but do not give the formula, and it reports MSE ahead of L1 and smooth L1 [arxiv:2203.12602 tab. 1] [arxiv:2203.12602 tab. 6]. On UCF101 and HMDB51, pre-trained on each dataset's own unlabelled split, ViT-B reaches 91.3% and 62.6%. MoCo v3 reaches 81.7% and 39.2% on the same split, and training from scratch reaches 51.4% and 18.0% [arxiv:2203.12602 tab. 2]. On Something-Something V2 the same recipe gives 69.6%, against 54.2% for MoCo v3. VideoMAE takes 800 epochs and 19.5 hours on 64 V100 GPUs, and MoCo v3 takes 300 epochs and 61.7 hours [arxiv:2203.12602 tab. 3]. In the paper's ViT-B ablation on Something-Something V2, tube masking at 90% scores 69.6%, random masking at 90% scores 68.3%, and tube masking at 75% scores 68.0%. Hiding 14 of the 16 frames drops the score to 61.5% [arxiv:2203.12602 tab. 1]. The main ViT-B results without external data are 81.5% on Kinetics-400 and 70.8% on Something-Something V2, at 16 frames [arxiv:2203.12602 tab. 7] [arxiv:2203.12602 tab. 6]. The 87.4% and 75.4% in the abstract use larger models at 32 frames, so do not compare them with ViT-B rows [arxiv:2203.12602 tab. 7] [arxiv:2203.12602 tab. 6]. The released code builds only tube masks, so the random-mask row does not appear to be reproducible from it [arxiv:2203.12602 code: datasets.py#L21-L24]. The code is CC BY-NC 4.0, and the Kinetics-400, HMDB51 and AVA datasets carry the same non-commercial terms, so check those before reusing weights [arxiv:2203.12602 app. License of Data]. Card: [VideoMAE](/papers/2203.12602). ### Dual masking for video (VideoMAE V2) VideoMAE V2 masks twice. The encoder works as in VideoMAE above. A clip is split into space-time cubes, 90% of them are hidden with tube masking, and only the remaining ~10% are encoded [arxiv:2303.16727 §3.1] [arxiv:2303.16727 tab. 10]. The decoder then receives the encoder output and a learnable mask token for each position kept by a second mask. That mask is running-cell masking at 50%. The decoder reconstructs only those positions, and the loss covers cubes that the encoder hid and the decoder kept [arxiv:2303.16727 §3.2] [arxiv:2303.16727 eq. 1] [arxiv:2303.16727 eq. 2]. Running-cell masking was picked over frame and random masks in the paper's ablation. The authors argue that tube masking in the encoder prevents information leaking across time, while the decoder mask should keep diverse cubes so that little information is lost [arxiv:2303.16727 tab. 1] [arxiv:2303.16727 §4.2]. The reason for the second mask is cost. On ViT-B with Something-Something V2 pre-training, dual masking gives 70.15 top-1 at 25.87G FLOPs. Encoder-only masking gives 70.28 at 35.48G [arxiv:2303.16727 tab. 1]. On ViT-g, memory falls from 1,753M to 1,050M and speed rises 1.48x [arxiv:2303.16727 tab. 2]. At ViT-g scale the model has 1,011M parameters and is pre-trained on 1.35M unlabeled clips. Supervised post-pre-training on a 710-class Kinetics mix follows, then fine-tuning on each target [arxiv:2303.16727 §4.1] [arxiv:2303.16727 §3.3]. Author-reported, with ViT-g: 90.0% top-1 on Kinetics-400 and 77.0% on Something-Something V2, against 75.5 for MAE-ST. The baselines use different view counts and input sizes [arxiv:2303.16727 tab. 6]. Post-pre-training on the labeled mix helps on Kinetics-400 but was worse on Something-Something V2 [arxiv:2303.16727 §4.2]. Card: [VideoMAE V2](/papers/2303.16727). ### Masked video encoder fused with a video-text encoder (InternVideo) InternVideo trains two encoders separately and fuses them afterwards. The first is a masked video ViT-H. The clip is cut into 3D patches, 90% of them are hidden by tube masking, and an asymmetric encoder-decoder reconstructs the hidden content [arxiv:2212.03191 §3.1] [arxiv:2212.03191 §4.2]. The second is a CLIP-initialised video-text encoder, trained with a contrastive loss and a caption decoder [arxiv:2212.03191 §3.1]. Both get supervised post-pretraining on Kinetics-710 [arxiv:2212.03191 §3.2]. Both backbones are then frozen, and a cross-model attention module with a learned linear combination of the two prediction scores is trained on top [arxiv:2212.03191 §3.3]. Card: [InternVideo](/papers/2212.03191). The fused model is what the results measure. The card does not report the masked encoder on its own, so the contribution of the masking step is not isolated [arxiv:2212.03191 tab. 2]. Author-reported, the 1.3B headline model reaches 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2 [arxiv:2212.03191 tab. 2] [arxiv:2212.03191 tab. 3]. VideoMAE V2's ViT-g reports 90.0% and 77.0% on the same two sets [arxiv:2303.16727 tab. 6]. The models differ in size, data and fusion, so these are not matched results. The learned fusion is not in the released code. The repository's ensemble script only averages per-clip scores from two sets of saved predictions [arxiv:2212.03191 code: InternVideo1/Pretrain/VideoMAE/ensemble.py#L8-L75]. ### Masked feature alignment in a state-space video model (VideoMamba) VideoMamba replaces attention with bidirectional Mamba blocks, a linear-time state-space layer [arxiv:2403.06977 §3.1]. Its masked pretraining follows UMT. The student is trained from scratch on video, hides 80% of the video tokens, and aligns the final outputs of the unmasked tokens to CLIP ViT-B features. Only the final layer is aligned, where UMT aligns several [arxiv:2403.06977 §3.4] [arxiv:2403.06977 §4.2]. Card: [VideoMamba](/papers/2403.06977). The target here is a CLIP feature, not pixels or codes, and the loss sits on the visible positions. That is the reverse of the hidden-position loss in most variants above, so check the reading against the paper before reuse. The paper does not say whether masked tokens are dropped or replaced by a learned token, and the row and attention masks are described only in words [arxiv:2403.06977 §3.4] [arxiv:2403.06977 fig. 5]. Check the masked-pretraining code before implementing either. Author-reported results: on Kinetics-400 at 16 x 224 px, the masked VideoMamba-M reaches 83.4% with 800 epochs and the CLIP teacher, against 81.5% for VideoMAE-B at 1600 epochs [arxiv:2403.06977 tab. 3]. At 8 x 224 px the masked row reaches 82.0%, against 85.7% for UMT-B, so the comparison depends on the setting [arxiv:2403.06977 tab. 3]. On Something-Something V2 at 16 x 288 px, the masked row reaches 71.4% against 70.8% for VideoMAE-B [arxiv:2403.06977 tab. 4]. In the authors' SSv2 ablation, 80% is the best of four mask ratios [arxiv:2403.06977 tab. 5]. Tube masking is the lowest of five mask types, at 66.3%, against 68.5% for attention masking [arxiv:2403.06977 tab. 5]. ### Classification targets instead of pixels Instead of regressing colour values, the SimMIM paper also trains with discrete targets: iGPT's 512 colour clusters, BEiT's discrete visual tokens (from a separately trained dVAE) and equal-width colour bins [arxiv:2111.09886 §3.4]. On the ablation setup these reach ImageNet top-1 between 82.3% and 82.7% at 48 x 48 targets, while the L1 pixel target reaches 82.8% at full resolution [arxiv:2111.09886 tab. 5]. The gaps are a few tenths of a point, and the paper reports no variance over seeds [arxiv:2111.09886 §4.1.2]. BEiT's dVAE pre-training is not counted in its cost figure [arxiv:2111.09886 tab. 6]. CAE compares its own targets under its regressor-and-decoder design, for ViT-B at 1600 epochs. Raw RGB pixels give 72.4 linear, 77.0 attentive and 50.4 ADE20K mIoU. The d-VAE token gives 71.4, 77.4 and 50.1, and the DALL-E token gives 70.4, 77.1 and 50.2. The authors conclude that the target matters little [arxiv:2202.03026 tab. 8]. Here the pixel target is the best linear probe, which is the opposite of the ordering that the discrete-token papers above would suggest. ### Discrete token targets (MAE-VQGAN, Unified-IO, MAGVIT, FLAVA, Muse) Here the target is an index into a fixed codebook, so the output is a softmax over codes rather than a regression. Five cards use this, for different goals. MAE-VQGAN is a ViT-L/16 masked autoencoder whose decoder predicts one of 1024 codes per patch. The codes come from a frozen, ImageNet-trained VQGAN, and the cross-entropy is computed on hidden patches only [arxiv:2209.00647 §3.1] [arxiv:2209.00647 §4.1]. Its pre-training set is 88,645 figures from arXiv computer-vision papers [arxiv:2209.00647 §3.3]. Card: [Visual Prompting via Image Inpainting](/papers/2209.00647). Its use at test time is covered below. Unified-IO maps image inputs to 16 x 16 patches of a 384 x 384 image, and image outputs to codes from a 16,384-entry VQGAN codebook [arxiv:2206.08916 §3.1] [arxiv:2206.08916 §3.2]. Masked image denoising masks 75% of the patches and asks for the whole image back, following BEiT and MAE. It is one of two denoising objectives in pre-training [arxiv:2206.08916 §3.3]. The card's ablations hold out task groups rather than this objective, so its own contribution is not measured [arxiv:2206.08916 tab. 5]. Card: [Unified-IO](/papers/2206.08916). MAGVIT is a video model. A 3D VQ tokenizer maps a 16-frame 128 x 128 clip to 4 x 16 x 16 = 1024 tokens from a 1024-entry codebook, and a BERT-style transformer is trained on those tokens with masked token modeling [arxiv:2212.05199 §3.1] [arxiv:2212.05199 §3.2]. Known frames or regions are tokenised and written into the masked positions, so one loss covers ten tasks [arxiv:2212.05199 §3.2] [arxiv:2212.05199 app. B.1]. The masking rate follows a cosine schedule, and generation takes 12 parallel steps [arxiv:2212.05199 app. B.2] [arxiv:2212.05199 alg. 1]. In its BAIR ablation, the 12-step model reaches debiased FVD 48, against 56 for a 1024-step autoregressive model on the same tokens [arxiv:2212.05199 §4.5]. Card: [MAGVIT](/papers/2212.05199). Here masking is a generation mechanism, not only a pre-training task. FLAVA uses the same kind of target inside a joint image-text model. Its unimodal masked image loss runs on unpaired images and predicts dVAE token ids for block-masked patches, in the BEiT style [arxiv:2112.04482 §3.3]. A masked multimodal loss uses the same block-masked image tokens on image-text pairs, and the fused encoder predicts them there [arxiv:2112.04482 §3.2]. Masked image modeling is one of five pre-training losses, alongside contrastive, masked language and matching terms. Its ablation row for masked image modeling alone reports only a vision average, so its effect on multimodal scores is not shown [arxiv:2112.04482 tab. 3]. Card: [FLAVA](/papers/2112.04482). Muse is a text-to-image generator built on the same masked-token objective. A VQGAN maps images to codes from an 8192-entry codebook, and a transformer is trained to predict randomly masked codes, with cross-entropy at the masked positions [arxiv:2301.00704 §2.2] [arxiv:2301.00704 §2.3]. The masking rate for each example is drawn from a fixed arccosine density, so most training steps hide most tokens [arxiv:2301.00704 §2.6]. Generation starts with every code masked and fills them in parallel over a few steps, keeping the most confident codes on each pass [arxiv:2301.00704 §2.8]. A frozen T5-XXL encoder supplies the text, with classifier-free guidance at sampling time [arxiv:2301.00704 §2.1] [arxiv:2301.00704 §2.7]. Author-reported: CC3M FID 6.06 for the base model with super-resolution [arxiv:2301.00704 tab. 1], zero-shot COCO FID 7.88 for the 3B model [arxiv:2301.00704 tab. 2], and 0.5 s per 256 px batch against 9.1 s for Imagen, both timed on internal TPU-v4 hardware with the batch size not stated [arxiv:2301.00704 tab. 3]. No code or weights are released [arxiv:2301.00704 §5]. Card: [Muse](/papers/2301.00704). ### Heavier prediction heads A two-layer MLP or an inverse Swin block as head gives roughly the same ImageNet accuracy as the linear head, at 1.2x to 2.3x the training cost [arxiv:2111.09886 tab. 2]. Heavier heads reach a slightly lower pre-training loss but transfer worse to downstream tasks [arxiv:2111.09886 §4.1.3]. MAE makes the same kind of point from the decoder side: decoder depth changes what the features are good for, so choose it for the downstream protocol [arxiv:2111.06377 tab. 1a]. ### Full-image reconstruction Adding a loss on visible pixels, so the model also reconstructs what it can see, drops fine-tuning accuracy from 82.8% to 81.7% [arxiv:2111.09886 tab. 4]. The paper's visualisations of the visible area show artifacts because that area is never trained [arxiv:2111.09886 §4.4]. The MAE card also reports that leaving visible patches out of the loss helps slightly, by about 0.5% [arxiv:2111.06377 §3]. ### Self-distilled targets from an EMA teacher (iBOT) iBOT keeps the masked-patch objective but replaces pixel or token targets with the output of a teacher. The teacher is an exponential moving average (EMA) of the student and sees the unmasked views. For each hidden patch, the student's distribution is trained to match the teacher's distribution at the same position [arxiv:2111.07832 §3.1] [arxiv:2111.07832 alg. 1]. The teacher output is a softmax over K = 8192 dimensions, not a one-hot label [arxiv:2111.07832 §3.2] [arxiv:2111.07832 tab. 18]. Both teacher outputs are centred and sharpened before the cross-entropy [arxiv:2111.07832 §3.1]. Card: [iBOT](/papers/2111.07832). A second loss on the [CLS] token compares two views of one image, and the two losses are summed with weight 1 each [arxiv:2111.07832 §2.2] [arxiv:2111.07832 §3.2]. In the paper's own ablation (ViT-S/16, 300 epochs, no multi-crop), removing the [CLS] loss drops k-NN accuracy from 69.1 to 9.5 and linear probing from 74.2 to 29.8, while fine-tuning stays at 79.4 against 81.5 [arxiv:2111.07832 tab. 9]. The patch loss alone does not give usable frozen features in that setting. The reported results are strong on frozen features and on fine-tuning. For ViT-B/16 the authors report 79.5% linear probing and 77.1% k-NN, both under the DINO protocol [arxiv:2111.07832 tab. 1] [arxiv:2111.07832 §4.1]. For ViT-B/16 they report 84.0% fine-tuning on ImageNet-1K, against 83.4% for BEiT at 800 epochs in the same table [arxiv:2111.07832 tab. 2]. On COCO with Cascade Mask R-CNN, ViT-B/16 reaches 51.2 box AP, against 50.1 for DINO [arxiv:2111.07832 tab. 6]. The ablation also shows a trade-off: a frozen DINO model used as the tokenizer gives higher fine-tuning accuracy (81.7) than iBOT's own targets (81.5), while iBOT is ahead on k-NN and linear probing [arxiv:2111.07832 tab. 9]. The cost is higher than DINO alone. For ViT-S/16 with multi-crop over 800 epochs, the authors report 193.4 h for iBOT against 180.0 h for DINO, on two 8-GPU machines at batch size 1024. Peak memory is 19.5 GB against 15.4 GB [arxiv:2111.07832 tab. 19]. ### Averaged block targets from an EMA teacher (data2vec) data2vec uses the same student-teacher layout as iBOT, but the target is not a single top-layer output. The teacher encodes the unmasked input, its top K Transformer blocks are each normalised and then averaged into one target per patch, and the student regresses those targets at the masked positions with a smooth L1 loss [arxiv:2202.03555 §3.3] [arxiv:2202.03555 §3.4]. For vision the paper uses K = 6 and beta = 2 [arxiv:2202.03555 §4.1]. Blocks are masked in groups of at least 16 patches, following BEiT, at 60% of patches [arxiv:2202.03555 §4.1]. Card: [data2vec](/papers/2202.03555). The same objective is used for speech and text, with modality-specific input encoders and masks [arxiv:2202.03555 §3.1] [arxiv:2202.03555 §3.2]. This page covers the vision side only. The authors' ablation sweeps K on ViT-B. Vision top-1 rises from 83.0 at K = 1, the BYOL-style top-layer target, to 84.0 at K = 3, and then stays between 83.6 and 83.9 [arxiv:2202.03555 fig. 2]. The gain from averaging is therefore small for vision. The paper also varies how much of the input the teacher sees. Vision top-1 is 77.0 when the teacher sees 1% of the input and 83.8 with full context [arxiv:2202.03555 fig. 3]. The author-reported ImageNet-1K result for a single ViT-B is 84.2, against 83.8 for SimMIM and iBOT and 83.6 for MAE in the same table [arxiv:2202.03555 tab. 1]. For a single ViT-L, the paper reports 86.6 [arxiv:2202.03555 tab. 1]. ### Separate regressor and token decoder (CAE) CAE splits the pretext task into three modules: an encoder, a regressor and a decoder [arxiv:2202.03026 §3.1] [arxiv:2202.03026 fig. 1]. The encoder sees only the visible patches, as in MAE. A cross-attention regressor predicts the encoder's representation of each hidden patch from the visible tokens. Its queries are one learned mask token per hidden position, plus that position's embedding. A decoder then maps each predicted hidden representation to a discrete token [arxiv:2202.03026 §3.1]. Half of the 16 x 16 patches are hidden, in random blocks as in BEiT [arxiv:2202.03026 §3.2]. Only the encoder is kept afterwards [arxiv:2202.03026 §3.1]. Card: [Context Autoencoder](/papers/2202.03026). The loss has two terms. One is cross-entropy on the decoder's tokens. The other is a mean-squared error between the regressor's output and the encoder's own representation of the hidden patches, weighted by 2. That representation is taken with gradient stopped, so it acts as a target [arxiv:2202.03026 eq. 1] [arxiv:2202.03026 §3.2]. The authors report that training without the reconstruction term collapses to a trivial solution [arxiv:2202.03026 §5.5]. The ViT-B ablation at 300 epochs separates the two terms [arxiv:2202.03026 tab. 5]. With neither, linear probing is 60.3. The decoder alone gives 63.1, the alignment term alone gives 62.0, and both together give 64.1, at 1.24x the training time of the baseline. Author-reported ViT-B results at 1600 epochs are 71.4 linear, 77.4 attentive and 83.9 fine-tuning [arxiv:2202.03026 tab. 1]. These are below iBOT's 79.5 linear and 79.8 attentive in the same table, so CAE does not lead on frozen features there. ### Masked patch loss added to a vision-language model (METER) METER is a transformer-only image-text model. Its default pre-training uses masked language modeling and image-text matching [arxiv:2111.02387 §3.3]. The paper also tests two masked image losses, each hiding 15% of the image patches [arxiv:2111.02387 §3.2]. One asks the model to pick the original patch out of all patches in the batch, using a softmax over dot products with its output. The other replaces each patch with a token from a VQ-VAE codebook and predicts that token [arxiv:2111.02387 §3.2] [arxiv:2111.02387 eq. 1]. Card: [METER](/papers/2111.02387). The released code at the card's commit contains neither loss [arxiv:2111.02387 code: meter/modules/objectives.py], so the results with these losses cannot be reproduced from it. In the paper's ablation, adding either loss lowers VQAv2 test-dev from 74.98 to 74.01 (in-batch) or 74.21 (discrete codes). Flickr zero-shot image retrieval R@1 falls from 66.08 to 62.12 and 59.80 [arxiv:2111.02387 tab. 7]. The authors read this as a sign that conclusions from region-based models may not carry over to patch-based models [arxiv:2111.02387 §4.4]. ### SimMIM as a stage before supervised scaling (Swin Transformer V2) Swin Transformer V2 uses SimMIM as a pretraining stage before supervised training, at a size of 3 billion parameters [arxiv:2111.09883 §3.4] [arxiv:2111.09883 §4.2]. The 3B model is first pretrained on about 70 million ImageNet-22K-extension images with the masked objective, then trained again with labels on the same images. The authors report this uses 1/40 of the labelled images used by JFT-3B [arxiv:2111.09883 §3.4] [arxiv:2111.09883 §4.2]. The masked stage for the 3B model runs for 20 epochs at 192 px, with batch 9216 [arxiv:2111.09883 app. A2]. The paper's headline results come from the full recipe. They include 84.0% ImageNet-V2 top-1 with 640 px fine-tuning, 63.1 COCO test-dev box AP and 59.9 ADE20K mIoU, all author-reported [arxiv:2111.09883 tab. 2] [arxiv:2111.09883 tab. 3] [arxiv:2111.09883 tab. 4]. The ablations in the card vary the normalisation, attention and position bias, not the pretraining stage, so they do not show how much SimMIM contributes at this scale [arxiv:2111.09883 tab. 6] [arxiv:2111.09883 tab. 7] [arxiv:2111.09883 tab. 8]. Card: [Swin Transformer V2](/papers/2111.09883). The masked stage is what makes the 3B recipe hard to reproduce. The 70 million image set is private, and no configs for the H or G model are in the repository (see Contradictions). ### Prototype matching without pixels (MSN) MSN does not reconstruct anything. An unmasked target view is passed through an EMA teacher, and its soft assignment over 1024 learned prototypes (dimension 256) is the target for a masked anchor view [arxiv:2204.07141 §3] [arxiv:2204.07141 app. A.1]. The target is sharpened, and a Sinkhorn step is applied over the batch. An entropy term rewards using all prototypes [arxiv:2204.07141 app. A.1] [arxiv:2204.07141 eq. 1]. The paper proves a no-collapse property when the sharpened target is not uniform [arxiv:2204.07141 app. B]. Each image has one large anchor, with random patch dropping, and ten small focal crops that are not masked [arxiv:2204.07141 §5] [arxiv:2204.07141 app. A.1]. The mask ratio is set per architecture, from 0.15 for ViT-S/16 to 0.7 for ViT-L/7 and ViT-B/4 [arxiv:2204.07141 app. A.1]. Card: [Masked Siamese Networks](/papers/2204.07141). In the low-label regime this is the strongest variant on the cards. ViT-B/4 reaches 72.4% top-1 with 5 labels per class and 75.7% with 1% of labels [arxiv:2204.07141 tab. 1] [arxiv:2204.07141 tab. 2]. At one label per class, ViT-S/16 reaches 47.1%, against 40.4% for iBOT's ViT-S/16 in the same table [arxiv:2204.07141 tab. 1]. Adding focal anchors to random masking raises 1%-label accuracy for ViT-B/16 from 52.3% to 59.8% [arxiv:2204.07141 tab. 7]. The advantage does not carry to full labels. Linear probing at 100% labels gives 76.9% for ViT-S/16, below iBOT's 77.9% and DINO's 77.0% [arxiv:2204.07141 tab. 3]. Full-label fine-tuning of ViT-B/16 reaches 83.4%, against 83.6% for DINO [arxiv:2204.07141 tab. 4]. Two details need care. The paper's objective includes the entropy term with weight 1.0, but the released ViT-S/16 config sets its weight to zero [arxiv:2204.07141 eq. 1] [arxiv:2204.07141 code: configs/pretrain/msn_vits16.yaml#L10]. The paper also says masking 70% "uses half the computation and memory" [arxiv:2204.07141 §1], while its own table shows memory falling from 26 GB to 17 GB per GPU and throughput rising from 415 to 600 img/s [arxiv:2204.07141 tab. 10]. With 70% of patches missing at evaluation, DINO ViT-B/16 loses 3.9 points at 1% labels, against 2.4 for MSN ViT-B/16 pre-trained at 0.3 masking [arxiv:2204.07141 tab. 15]. ### Test-time masked reconstruction (TTT-MAE) TTT-MAE does not pre-train a single fixed model for all inputs. For each test image it starts from the pre-trained weights, runs a few SGD steps on masked-patch reconstruction of that image, predicts on the unmasked image, and then discards the adapted weights [arxiv:2209.07522 §3]. The encoder and decoder are a masked autoencoder, and the main head is a ViT-Base classifier trained on top of the frozen encoder [arxiv:2209.07522 §3] [arxiv:2209.07522 §4.1]. Card: [Test-Time Training with Masked Autoencoders](/papers/2209.07522). Per test image the paper uses 128 identical copies of the crop, each with its own random 75% patch mask, and 20 SGD steps with momentum 0.9 at learning rate 5e-3 [arxiv:2209.07522 §4.1]. The paper's reconstruction equation averages over all patches, while the released code averages over the removed patches only [arxiv:2209.07522 eq. 2] [arxiv:2209.07522 code: models_mae_shared.py#L265-L282]. On ImageNet-C at severity 5, the authors report that TTT-MAE beats the fixed ViT-probing baseline on all 15 corruptions. Examples from their table: JPEG 51.2 to 63.0, elastic 31.6 to 50.7, and contrast 6.4 to 9.8 [arxiv:2209.07522 tab. 2]. On ImageNet-A the paper reports 15.3 to 21.3, and on ImageNet-R 31.3 to 38.9 [arxiv:2209.07522 tab. 3]. These are author-reported; no mean over the corruptions is reported in the card. The method is evaluated with a masked autoencoder, not with SimMIM or iBOT. The paper's theory section covers only linear PCA with a known covariance and an orthogonal corruption, and does not carry over to the deep model by itself [arxiv:2209.07522 §5]. ### Frozen features for robot control (MVP) A ViT-S/16 masked autoencoder is pre-trained on about 700k images of hands and objects, then frozen for the whole RL run [arxiv:2203.06173 §2.1] [arxiv:2203.06173 §2.2] [arxiv:2203.06173 §4]. A small PPO controller reads the classification-token feature, projected to 128 dimensions, together with the joint positions and velocities, and it outputs joint-angle deltas [arxiv:2203.06173 §4]. Card: [Masked Visual Pre-training for Motor Control](/papers/2203.06173). The authors report that this beats a supervised ImageNet ViT-S on 7 of 8 PixMC tasks in sample efficiency, and that it comes close to an oracle policy that sees the true object state on 5 of 8 [arxiv:2203.06173 §5.1] [arxiv:2203.06173 fig. 5]. Freezing matters in their ablations: with a random frozen encoder, 6 of 8 tasks reach zero or near zero success, and an unfrozen encoder stays at zero on Franka pick for both random and MAE initialisation [arxiv:2203.06173 §5.3] [arxiv:2203.06173 fig. 9]. Each image-based run takes about 5 hours on one 2080 Ti GPU, for about 8 million environment steps [arxiv:2203.06173 §3.2]. The pre-training code is not in the released repository, which holds only the ViT-S weights [arxiv:2203.06173 code: mvp/backbones/model_zoo.py#L18-L24]. Per-task numbers are only in figures, so read them from [fig. 5] rather than from the text [arxiv:2203.06173 §5]. ### Frozen MAE tokens inside a navigation model (AMR) Aim My Robot runs a frozen MAE-Base encoder on the current RGB frame and on the reference image of the target object. The resulting tokens are fused with LiDAR and goal tokens in one transformer [arxiv:2411.14770 §III-B1] [arxiv:2411.14770 §III-B2]. The MAE is used only as a feature extractor here. The navigation policy itself is trained by behaviour cloning on simulated trajectories, with DAgger [arxiv:2411.14770 §IV]. Depth is not concatenated to the image; each depth patch is turned into a 3D position, embedded, and added to its matching RGB token [arxiv:2411.14770 §III-B1]. Card: [Aim My Robot](/papers/2411.14770). The authors report a 3 cm and 1 degree median error on unseen objects in simulation [arxiv:2411.14770 §I]. Their ablation removes one input at a time and reports collision rates. The full model scores 3.8, against 17.7 without depth and 25.6 without LiDAR. The table does not state its units [arxiv:2411.14770 tab. II]. The main baseline uses a CAD model and a ground-truth occupancy map, which AMR does not have, so the comparison is not like-for-like [arxiv:2411.14770 §V-A]. ### Language-conditioned reconstruction (Voltron) Voltron reconstructs masked patches of a video clip, and the caption enters the same model. The paper does not state a mask ratio; the code uses 0.75 [arxiv:2302.12766 §3.1] [arxiv:2302.12766 code: voltron/conf/models.py#L48]. For each example, with probability alpha the caption is the target of a causal decoder, and the encoder sees only a null token. Otherwise the caption is a condition for pixel reconstruction [arxiv:2302.12766 §3.1] [arxiv:2302.12766 §3.2]. The code adds the caption loss with weight 0.5, which the paper's objective does not [arxiv:2302.12766 code: voltron/conf/models.py#L405]. Card: [Language-Driven Representation Learning for Robotics](/papers/2302.12766). The tables report the conditioning variant (alpha 0, one frame). On referring-expression grounding (OCID-Ref) it reaches 89.38 total for ViT-S, against 63.30 for R-R3M and 49.58 for R-MVP [arxiv:2302.12766 tab. 3]. On grasp affordance it reaches 85.15 top-1 for ViT-S, against 72.94 for R-MVP [arxiv:2302.12766 tab. 2]. The no-language variant falls to 53.44 on referring grounding [arxiv:2302.12766 tab. 4]. The baselines are the authors' reimplementations on Something-Something-v2, not the released MVP and R3M models [arxiv:2302.12766 §4]. The generation variant appears only in a control figure, where the authors report that it beats single-frame models with low absolute success rates [arxiv:2302.12766 fig. 5] [arxiv:2302.12766 §5.3]. ### Masked tactile reconstruction, conditioned on vision (Touch in the Wild) Here the masked signal is touch. Two fingertip arrays of 12 x 32 pressure sensors are stacked into one 24 x 32 tactile image and colour-mapped. In 95% of samples, 60% to 80% of its 4 x 4 patches are replaced by one learnable mask token [arxiv:2507.15062 §4.2]. A 3-layer CNN embeds the masked tactile image. A CLIP ViT-B/16, unmasked and fine-tuned, embeds the RGB frame. Cross-attention in each direction then lets each embedding attend to the other, and the fused representation is the concatenation of the two [arxiv:2507.15062 eq. 2] [arxiv:2507.15062 eq. 3] [arxiv:2507.15062 eq. 4]. A two-layer MLP with a sigmoid output decodes the full tactile image [arxiv:2507.15062 §4.2]. The loss is the squared error over the whole tactile image, not only the masked patches [arxiv:2507.15062 eq. 1]. This departs from the MAE convention. The authors pick reconstruction over contrastive alignment, arguing that contrastive alignment suppresses the fine-grained tactile signal [arxiv:2507.15062 §4.2]. The target encoder is an exponential moving average of the online weights, with decay 0.9995 [arxiv:2507.15062 §4.2]. Pre-training runs for 5 epochs at batch size 128 on the authors' own collection of over 2,700 handheld demonstrations, recorded in stores and outdoor markets [arxiv:2507.15062 §1] [arxiv:2507.15062 app. E]. Downstream, the pretrained encoder output and proprioception condition a diffusion policy, with no masking [arxiv:2507.15062 §4.3]. Author-reported, with 20 trials per cell. On the test-tube task (whole task), success is 0.85 with pre-training, 0.70 for the same encoder without pre-training, and 0.25 for vision only [arxiv:2507.15062 tab. 1]. On the whiteboard task, the whole-task numbers are 0.70 with pre-training and 0.60 without. One trial moves a rate by 0.05, so several gaps here are one or two trials [arxiv:2507.15062 tab. 1]. When the task's own demonstrations are removed from pre-training, the test-tube whole-task rate is 0.80, against 0.85 with them [arxiv:2507.15062 tab. 3]. Card: [Touch in the Wild](/papers/2507.15062). ### Entity pseudo-labels on video crops (ALPRO) ALPRO adds a region-level prediction term to a video-text model. A random crop covering 30% to 50% of the frame is embedded by a frozen prompter and compared with prompts for about 1,000 entity names. The softmax over those similarities is the target, and the main model predicts it from features pooled at the crop's positions, with a soft cross-entropy loss [arxiv:2112.09583 §3.2.2] [arxiv:2112.09583 eq. 4] [arxiv:2112.09583 eq. 5]. The target is a distribution over words, not pixels or codes, and no detector or human label is needed [arxiv:2112.09583 §3.2.2]. Nothing in this description hides the crop from the encoder, so it is closer to region-level prediction than to masked reconstruction. Card: [Align and Prompt](/papers/2112.09583). In the ablation, adding this term to masked language modeling and video-text matching raises MSVD-QA from 43.3 to 46.3, and MSRVTT-QA from 40.9 to 41.8 [arxiv:2112.09583 tab. 1]. The reported full model reaches 33.9 R@1 on MSRVTT 1k after fine-tuning [arxiv:2112.09583 tab. 2]. The released code applies the 0.2 pseudo-label threshold to the index of the top entity rather than to its score, so it discards only crops whose top entity has index 0 [arxiv:2112.09583 code: src/modeling/alpro_models.py#L527]. ### Inpainting as an in-context prompt (visual prompting) MAE-VQGAN can be given a task without changing its weights. An example input-output pair and a query are tiled into a 2 x 2 grid, with the pair in the top row and the query bottom-left. The bottom-right cell is hidden, and the model's predicted codes there are decoded by the VQGAN decoder [arxiv:2209.00647 §3.1] [arxiv:2209.00647 §3.2] [arxiv:2209.00647 §4.2]. Pre-training data matters for this use. On Pascal-5i segmentation, MAE-VQGAN trained on ImageNet alone scores 2.22 to 7.07 mIoU across the four splits, against 24.25 to 30.44 when trained on figures [arxiv:2209.00647 tab. 1]. Prompt layout matters too: a vertical black-and-white layout reaches 31.57 mIoU, against 27.17 for the horizontal one [arxiv:2209.00647 §4.4]. Adding a vertical layout, and then a vertical layout with its rows swapped, to the horizontal one raises colour-aware mIoU on the synthetic study from 39.97 to 44.14 [arxiv:2209.00647 §4.4]. The authors find the results competitive with other pretrained inpainting models, but not with supervised task-specific models [arxiv:2209.00647 §5] [arxiv:2209.00647 §4.4]. For comparison, a labelled one-shot CyCTR reaches 67.2 on split 0 [arxiv:2209.00647 tab. 3]. The released evaluation hides a fixed bottom-right block, while pre-training hides random patches [arxiv:2209.00647 code: evaluate/mae_utils.py#L51-L57] [arxiv:2209.00647 code: models_mae.py#L123-L148]. The paper does not discuss this gap. Card: [Visual Prompting via Image Inpainting](/papers/2209.00647). ### Stitched prompt pairs with a masked output (Painter) Painter writes the output of each dense task as an RGB image, so depth, segmentation, keypoints and restoration all become pictures [arxiv:2212.02499 §3.1]. It stacks two input-output pairs of the same task and hides output patches with one learned mask token. The loss is smooth L1 on the masked output pixels [arxiv:2212.02499 §3.2]. The paper gives 75% block-wise masking [arxiv:2212.02499 §3.2], while the released script masks 784 of 1568 patches, about 50% [arxiv:2212.02499 code: Painter/train_painter_vit_large.sh#L12]. At inference a solved pair is placed above the query, and the query's empty output half is painted [arxiv:2212.02499 §3.3]. No head is added for a task. Adding one needs only new training pairs [arxiv:2212.02499 §3.1]. Card: [Painter](/papers/2212.02499). Author-reported, with one ViT-L: NYUv2 depth RMSE 0.288, against 0.330 for the best listed specialist. The same model reaches 49.9 ADE-20K mIoU against 57.7, and 72.1 COCO keypoint AP against 77.2 [arxiv:2212.02499 tab. 1]. The choice of prompt moves little. ADE-20K mIoU is 49.3 with a random prompt and 49.9 with a searched one, and the paper reports no variance, so that gap may be noise [arxiv:2212.02499 tab. 3]. On open-vocabulary few-shot segmentation the card reports 62.3 mIoU, against 58.3 for MAE-VQGAN, the inpainting method above [arxiv:2212.02499 tab. 4]. ### Masked 2D skeleton sequences lifted to 3D (MotionBERT) MotionBERT applies the masked-recovery recipe to human motion rather than to images. Its input is a sequence of 2D joints with confidence, and the pretext task is to recover 3D motion from a corrupted copy [arxiv:2210.06551 §3.3]. The corruption masks 5% of joints and 10% of frames, and adds noise [arxiv:2210.06551 app. A.1]. Masked entries are multiplied by zero in every channel, with no learned mask token [arxiv:2210.06551 code: lib/data/augmentation.py#L71-L74]. Mocap clips get a 3D position loss plus a velocity term [arxiv:2210.06551 eq. 6]. Video has no 3D target, so the model's 3D estimate is projected back to 2D and compared with the detected joints [arxiv:2210.06551 eq. 7]. The encoder, DSTformer, runs spatial-then-temporal and temporal-then-spatial attention in two streams, and blends them with learned weights [arxiv:2210.06551 §3.2] [arxiv:2210.06551 eq. 5]. Card: [MotionBERT](/papers/2210.06551). On Human3.6M with detected 2D poses, the authors report 39.2 mm MPJPE from scratch and 37.5 mm after finetuning [arxiv:2210.06551 tab. 1]. In the cumulative pretraining ablation, adding the masks to noisy 2D-to-3D lifting moves MPJPE from 38.1 mm to 37.4 mm [arxiv:2210.06551 tab. 5]. With a frozen encoder, a random one gives 404.4 mm and a pretrained one 40.3 mm [arxiv:2210.06551 tab. 4]. Pretraining helps finetuning only while the mask ratio stays below about 45% [arxiv:2210.06551 fig. 7], so MAE's 75% does not carry over. ### Masked patches of traffic time series (STEP) Masked patch reconstruction is not limited to images. STEP applies it to a long history of traffic readings. Each sensor's series is cut into patches, 75% of them are hidden, and a Transformer (TSFormer) is trained to reconstruct the hidden patches, with a mean absolute error over those patches only [arxiv:2206.09113 §3.1] [arxiv:2206.09113 eq. 1]. As in MAE, the encoder sees only the visible patches, and a decoder inserts a shared mask token at the hidden positions [arxiv:2206.09113 §3.1]. The pre-trained encoder is then frozen. Its output supplies a representation of the most recent short patch, and a kNN-based term regularises a sparse graph that is learned from the data, with a Graph WaveNet forecaster on top [arxiv:2206.09113 §3.2] [arxiv:2206.09113 eq. 5]. Card: [STEP](/papers/2206.09113). The authors report that STEP is best on MAE, RMSE and MAPE in nearly all cells on METR-LA, PEMS-BAY and PEMS04, at horizons 3, 6 and 12, against 12 baselines [arxiv:2206.09113 §4.2] [arxiv:2206.09113 tab. 2]. The comparison is against forecasting models as a whole, so it does not isolate the masked pre-training stage. The masking ratio here, 75%, is the same as MAE's, but the signal is a series of traffic readings, not pixels. ### Multi-temporal satellite data (Prithvi-EO-2.0) Prithvi-EO-2.0 is a ViT masked autoencoder for sequences of four satellite images, each with six spectral bands at 30 m, in 300M and 600M sizes [arxiv:2412.02732 §3.1] [arxiv:2412.02732 §3.2]. Its changes to image MIM are structural. The patch embedding is 3D with a temporal extent of 1, so each image keeps its own time slice [arxiv:2412.02732 §3.2]. Date and centre location are added as learned-weight biases, and that metadata is dropped with probability 0.1 during pre-training so the model also works without it [arxiv:2412.02732 §3.2]. The mask ratio is not stated in the paper or the repository [arxiv:2412.02732 §3.2]. Pre-training used about 21,000 GPU-hours for the 300M model and about 58,000 for the 600M model, on A100 GPUs [arxiv:2412.02732 §3.2]. Card: [Prithvi-EO-2.0](/papers/2412.02732). Downstream, the authors use LoRA on some tasks [arxiv:2412.02732 §4.2.3] [arxiv:2412.02732 §4.3.3] [arxiv:2412.02732 §4.4.1] and a frozen encoder for gross primary productivity [arxiv:2412.02732 §4.4.2]. Author-reported: the 600M-TL model reaches 90.3 mIoU on Sen1Floods11, against 88.3 for Prithvi-EO-1.0-100M [arxiv:2412.02732 tab. 4], and 90.5 on wildfire burn scars, against 86.9 [arxiv:2412.02732 tab. 5]. Biomass estimation is the exception: the challenge baseline, at RMSE 27.49, beats every Prithvi configuration, whose best is 33.40 [arxiv:2412.02732 §4.4.1] [arxiv:2412.02732 tab. 13]. The README describes the metadata encoding differently from the paper, so check a released checkpoint's architecture before reusing the code [arxiv:2412.02732 code: README.md]. ## When to use which | Situation | Use | Why | |---|---|---| | Want a cheap pre-training recipe with no tokenizer and no teacher to maintain | Raw pixel regression | Relative pre-training cost 1.0x, against 1.5x for BEiT and 2.0x for DINO in the SimMIM authors' table; no dVAE stage [arxiv:2111.09886 tab. 6]. This is the authors' cost comparison, not a measurement on iBOT's setup | | Fine-tuning is the goal and you can use the MAE recipe | Visible-only encoder | The encoder sees only the visible quarter of the patches. The one-block decoder fine-tunes at 84.8% and pre-trains ViT-L 3.7x faster than the mask-token encoder baseline [arxiv:2111.06377 tab. 1a] [arxiv:2111.06377 tab. 2] | | Backbone is hierarchical (Swin) | Raw pixel regression with 32 x 32 masked patches | The 32-pixel mask matches Swin's last-stage patch size, and the method was scaled to SwinV2-H and SwinV2-G [arxiv:2111.09886 §3.2] [arxiv:2111.09886 tab. 7] | | Dense downstream tasks on Swin | Raw pixel regression | Swin-B COCO box AP 52.3 vs 50.2 supervised; ADE20K mIoU 52.8 vs 50.4 [arxiv:2111.09886 tab. 12]. iBOT's dense results are on ViT-B, so they are not directly comparable | | Need frozen features (linear probing or k-NN) on a ViT | Self-distilled targets (iBOT), with the same protocol as your comparison | Pixel regression reaches 56.7% ViT-B linear evaluation [arxiv:2111.09886 tab. 6]; iBOT reports 79.5% under the DINO protocol [arxiv:2111.07832 tab. 1]. The protocols differ, so check them before choosing. MAE's linear probe is weaker still [arxiv:2111.06377 app. B] | | Only 1 to 5 labels per class | Prototype matching (MSN), with random and focal anchors | 72.4% at 5 labels per class for ViT-B/4 [arxiv:2204.07141 tab. 1]; random plus focal anchors beat random-only at 1% labels [arxiv:2204.07141 tab. 7]. Use the mask ratio set for your architecture [arxiv:2204.07141 app. A.1] | | Full labels, and fine-tuning is the metric | Raw pixel regression or the visible-only encoder | Fine-tuning reaches 83.8% for SimMIM and 83.6% for MAE at ViT-B. The MSN paper's own table shows no full-label gain for MSN [arxiv:2111.09886 tab. 6] [arxiv:2111.06377 tab. 3] [arxiv:2204.07141 tab. 4] | | Robot control from pixels, with a fixed budget per task | Frozen MAE pre-trained on task-like images (MVP) | Beats a supervised ImageNet ViT-S on 7 of 8 tasks, author-reported; about 5 hours per run on one 2080 Ti. The pre-training code is not released, so budget for writing it [arxiv:2203.06173 §5.1] [arxiv:2203.06173 §3.2] | | Mobile robot must reach a precise pose relative to a named object, with a camera, LiDAR and no object model or map | Frozen MAE tokens fused with depth and LiDAR (AMR) | Author-reported 3 cm and 1 degree median error on unseen objects, in simulation. The real-robot result is 3 of 3 completed runs for 5 of 6 objects [arxiv:2411.14770 §I] [arxiv:2411.14770 tab. III]. The baseline uses information AMR lacks, so the gain is not a clean comparison [arxiv:2411.14770 §V-A] | | Captioned video, and a task where the language names the target (grounding, language-conditioned control) | Language-conditioned reconstruction (Voltron), conditioning variant | Referring grounding 89.38 for ViT-S, against 63.30 for the R-R3M reimplementation, and 53.44 without language [arxiv:2302.12766 tab. 3] [arxiv:2302.12766 tab. 4]. Pre-training is 400 epochs on 168K clips, on TPU hardware [arxiv:2302.12766 §4] | | Video-text retrieval or video QA, with video-text pairs only and no detector | Entity pseudo-labels on crops (ALPRO) | 33.9 R@1 on MSRVTT 1k after fine-tuning and 42.1 MSRVTT-QA [arxiv:2112.09583 tab. 2] [arxiv:2112.09583 tab. 4]. Pre-training runs 100k iterations on 16 A100 GPUs [arxiv:2112.09583 §3.4] | | Need robustness to unseen shifts and can afford per-image adaptation | Test-time masked reconstruction (TTT-MAE) | Gains on every ImageNet-C corruption at level 5 against the fixed baseline [arxiv:2209.07522 tab. 2]. Costs 20 SGD steps with 128 copies per test image, so budget for it [arxiv:2209.07522 §4.1] | | New image-to-image task, no labels for it, no weight change allowed | Inpainting prompt (MAE-VQGAN), pre-trained on figures | Competitive with pretrained inpainting baselines and far below labelled models [arxiv:2209.00647 §5] [arxiv:2209.00647 tab. 3]. Expect 1-shot segmentation, not supervised accuracy | | Video generation with a small number of decoding steps | Masked token modeling over 3D tokens (MAGVIT) | 12 steps instead of 1024 for an autoregressive model on the same tokens, with FVD 76 on UCF-101 class-conditional for the L model [arxiv:2212.05199 §4.5] [arxiv:2212.05199 tab. 1] | | Text-to-image generation where few parallel decoding steps matter, and the output must be an image grid of codes | Masked-token generation over VQGAN codes (Muse) | Fills tokens in a few parallel passes, where an autoregressive model needs one pass per token [arxiv:2301.00704 §2.8]. Author-reported 0.5 s against 9.1 s for Imagen per batch, on internal TPU-v4 with the batch size not stated [arxiv:2301.00704 tab. 3]. No code or weights are released, so the recipe must come from the paper [arxiv:2301.00704 §5] | | Long sensor history for forecasting, with links between sensors learned rather than given by a road map | Masked patch pre-training on the history, frozen, then a graph forecaster (STEP) | Author-reported gains over 12 baselines on three traffic datasets [arxiv:2206.09113 tab. 2]. The pre-training runs once per dataset, so the cost of a long history is paid in advance [arxiv:2206.09113 §3] | | Multispectral time series where date and location are known | 3D patches with a metadata bias (Prithvi-EO-2.0) | Beats the Prithvi-EO-1.0 model on flood and burn-scar mapping [arxiv:2412.02732 tab. 4] [arxiv:2412.02732 tab. 5]. Not the best choice for biomass, where the authors' own baseline wins [arxiv:2412.02732 §4.4.1] | | Satellite or other imagery with repeated passes over one place, and a timestamp for each image | Per-image masking with temporal codes (SatMAE) | Temporal fMoW top-1 81.49 with test-time augmentation, against 78.90 for MAE on three copies of the image [arxiv:2207.08051 §5.3]. The paper says the augmentation averages scores, but the code votes, so reproduce the paper's version to match the headline number [arxiv:2207.08051 app. A.3.2] [arxiv:2207.08051 code: engine_finetune.py#L256-L264] | | Video model from a plain ViT with no image-pretrained weights, a small labelled video set, and unlabelled clips from the same domain | Tube masking at 90% on those clips (VideoMAE) | Own unlabelled split gives 91.3% on UCF101 and 62.6% on HMDB51, against 51.4% and 18.0% from scratch [arxiv:2203.12602 tab. 2]. The UCF101 and HMDB51 runs use 3,200 and 4,800 pre-training epochs, at 75% masking [arxiv:2203.12602 app. Implementation Details] | | Video, with a fixed GPU budget for pre-training | Dual masking (VideoMAE V2) | The decoder reconstructs half of the kept cubes. On the authors' ViT-g pre-training, memory falls from 1,753M to 1,050M and speed rises 1.48x [arxiv:2303.16727 tab. 2]. On ViT-B the accuracy is about the same, 70.15 against 70.28 [arxiv:2303.16727 tab. 1]. The gain is cost, not accuracy | | One video backbone for action recognition, temporal localisation and video-text retrieval, with a large compute budget | Fused masked video encoder and video-text encoder (InternVideo) | Author-reported 91.1% on Kinetics-400 and 77.2% on Something-Something V2 for the fused 1.3B headline model [arxiv:2212.03191 tab. 2] [arxiv:2212.03191 tab. 3]. Two backbones and a fusion stage must be trained, and the fusion is not in the released code [arxiv:2212.03191 code: InternVideo1/Pretrain/VideoMAE/ensemble.py#L8-L75] | | Long clips, where the quadratic cost of attention is the limit, and you can pre-train on video from scratch | Masked feature alignment with a linear-time backbone (VideoMamba) | Attention cost grows quadratically with input length [arxiv:2403.06977 §3.2]. The authors report 6x speed and 40x memory gains at 64 frames, in a figure [arxiv:2403.06977 §1] [arxiv:2403.06977 fig. 1]. On 8 x 224 px Kinetics-400 the masked row trails UMT-B, 82.0% against 85.7% [arxiv:2403.06977 tab. 3] | | Multi-band sensor data, where bands form groups of similar wavelength and resolution | Band groups with a group code (SatMAE) | 59.30 top-1 on fMoW-Sentinel with grouped tokens and independent masking, against 57.37 for stacking all bands as channels [arxiv:2207.08051 §5.4]. Ten single-band groups gave 1,440 tokens and unstable training [arxiv:2207.08051 app. A.3.3] | | Contact-rich manipulation with a tactile sensor and a camera, and a large unlabelled set of demonstrations | Masked tactile reconstruction with a CLIP image context (Touch in the Wild) | Test-tube whole-task success 0.85 with pre-training, 0.70 without, and 0.25 for vision only, at 20 trials per cell [arxiv:2507.15062 tab. 1]. The pre-training set is the authors' own collection, and the effect at fewer demonstrations is shown only as a plot [arxiv:2507.15062 fig. 7] | | Unified model with text and image outputs | Masked image denoising as one of two pre-training objectives (Unified-IO) | The card's 64.3 GRIT average is for the full multi-task model, not for this objective alone [arxiv:2206.08916 tab. 3] | | One image-text model that must also do unimodal vision and language tasks, trained only on public data | Masked image loss as one of five objectives (FLAVA) | Adding the unimodal masked image and masked language losses to the multimodal ones lifts the macro average from 74.23 to 75.85 over 35 tasks, author-reported [arxiv:2112.04482 tab. 3] [arxiv:2112.04482 tab. 4]. The masked image loss alone is not scored on multimodal tasks [arxiv:2112.04482 tab. 3] | | Small masked patches (8 or 16 pixels) | Use 32-pixel patches for pixel regression | At ratio 0.6, 4 and 8 pixel masks reach 82.0% and 82.1%, against 82.8% for 32 [arxiv:2111.09886 tab. 1] | | Pre-training only on images you can reconstruct | Keep the loss on masked pixels only | Full-image loss costs about one point of fine-tuning accuracy [arxiv:2111.09886 tab. 4] | | Compute is abundant and the head matters | Still start from the linear head | Bigger heads cost more and transfer worse in the paper's comparison [arxiv:2111.09886 tab. 2] [arxiv:2111.09886 §4.1.3] | | One masked-prediction recipe for vision, speech and text, with no tokenizer | Averaged block targets (data2vec) | Single ViT-B fine-tune at 84.2 on ImageNet-1K, against 83.8 for SimMIM and iBOT in the same table [arxiv:2202.03555 tab. 1]. The released vision configs do not match the paper's ViT-B values, so implement from the paper and state the choices [arxiv:2202.03555 §4.1] | | Scaling a hierarchical backbone to billions of parameters, with few labels | SimMIM stage before supervised training (SwinV2-G) | The 3B recipe's masked stage is on a private 70M-image set, and no G configs are released, so exact reproduction is not possible from the cards [arxiv:2111.09883 §4.1] [arxiv:2111.09883 app. A2]. Use the released SwinV2-B and -L configs to test the method at smaller scale [arxiv:2111.09883 app. A2] | | Dense downstream tasks on a plain ViT, where a regressor and a token decoder are acceptable | Context autoencoder (CAE), ViT-B at 1600 epochs | Author-reported ADE20K mIoU 50.1 to 50.2, against 48.1 for MAE, and COCO box AP 50.2 against 48.4 [arxiv:2202.03026 tab. 2] [arxiv:2202.03026 tab. 3]. iBOT reports 50.0 mIoU in the same table, so the margin over it is small. Training takes 1.24x the baseline time in the paper's ablation [arxiv:2202.03026 tab. 5]. The code defaults to the DALL-E tokenizer, so choose the tokenizer explicitly [arxiv:2202.03026 code: tools/run_pretraining.py#L28] | | Image-text model that uses a transformer for both modalities and no detector | MLM and ITM only, without masked image patch losses | In METER's tests, adding either masked-patch loss lowers VQAv2 test-dev from 74.98 to 74.01 or 74.21, and lowers Flickr retrieval too [arxiv:2111.02387 tab. 7]. Neither loss is in the released code, so the authors' setting cannot be reproduced from it [arxiv:2111.02387 code: meter/modules/objectives.py] | | Dense vision tasks from example pairs, with one model and no new head per task, where every output can be drawn as an image | Stitched prompt pair with a masked output (Painter) | Author-reported NYUv2 depth RMSE 0.288 against 0.330 for the best listed specialist, but ADE-20K mIoU 49.9 against 57.7 [arxiv:2212.02499 tab. 1]. The paper and code disagree on the mask ratio, 75% against about 50%, so state which one you use [arxiv:2212.02499 §3.2] [arxiv:2212.02499 code: Painter/train_painter_vit_large.sh#L12] | | Human motion from 2D keypoint sequences, with mocap or video available for pretraining | Masked skeleton lifting (MotionBERT) | Author-reported 37.5 mm MPJPE after finetuning on Human3.6M with detected 2D poses [arxiv:2210.06551 tab. 1]. The paper gives Adam with no learning-rate decay, while the code uses AdamW with a decay, so pick one [arxiv:2210.06551 app. A.1] [arxiv:2210.06551 code: train.py#L288-L289] | Pre-training data beyond ImageNet-1K is not evaluated in the main SimMIM ablations, and its small-dataset behaviour is not stated [arxiv:2111.09886 §4.1.1]. iBOT's ImageNet-22K results show that the best data for linear probing is not the best for k-NN: ViT-L/16 gets 82.3 linear but 72.9 k-NN when pre-trained on 22K [arxiv:2111.07832 tab. 1]. ## Implementation sketch The common core is the same for every variant. Only the target and the loss change, as noted after the block. Pseudocode in the same style as the paper card's algorithm [arxiv:2111.09886 §3.2] [arxiv:2111.09886 §3.4]. ```pseudocode # [arxiv:2111.09886 §3.2] split into tokens, then pick hidden ones tokens = patch_embed(images) # (B, L, C) hide = random_mask(B, L, ratio=0.6) # (B, L) bool, True = hidden # [arxiv:2111.09886 §3.2] swap each hidden token for one learnable vector tokens = where(hide, mask_token, tokens) # [arxiv:2111.09886 §3.2] encoder sees every token, hidden ones included feats = encoder(tokens) # (B, L, C') # [arxiv:2111.09886 §3.4] one linear map per token to its pixel block pred = linear_head(feats) # (B, L, s*s*3) # [arxiv:2111.09886 §3.4] L1 only over hidden pixels, averaged (eq. 1) pix_hide = expand(hide, s) # (B, H, W), hidden pixels loss = (abs(pred - target) * pix_hide).sum() loss = loss / (pix_hide.sum() + 1e-5) ``` Where the variants differ: - **Encoder input.** SimMIM and MAGVIT keep every position in the encoder and put a mask token at each hidden one [arxiv:2111.09886 §3.2] [arxiv:2212.05199 §3.2]. MAE drops hidden patches before the encoder, so the encoder runs on the visible quarter only [arxiv:2111.06377 §3]. VideoMAE drops hidden cubes in the same way [arxiv:2203.12602 §3.3]. MSN drops patches on its large anchor but not on its focal crops [arxiv:2204.07141 app. A.1]. data2vec's student keeps every position and swaps masked ones for a learned mask embedding, as SimMIM does [arxiv:2202.03555 §3.2]. SatMAE encodes only the visible tokens of each image in a sequence, with the mask drawn per image [arxiv:2207.08051 §3] [arxiv:2207.08051 §4.1.2]. VideoMAE V2 encodes about 10% of the cubes, and its decoder sees mask tokens at the positions its own mask keeps [arxiv:2303.16727 §3.2]. Touch in the Wild keeps every tactile position in its CNN input and swaps the masked patches for one learned token [arxiv:2507.15062 §4.2]. InternVideo's masked encoder is described as encoding visible tokens only, since its tube-masked clip is reconstructed by an asymmetric decoder [arxiv:2212.03191 §3.1]. VideoMamba's masked stage does not say whether hidden tokens are dropped or replaced [arxiv:2403.06977 §3.4]. CAE's encoder sees only the visible patches, as in MAE, and its regressor supplies the hidden positions [arxiv:2202.03026 §3.1]. Painter runs the input half and the masked output half in parallel streams, and merges them after three blocks [arxiv:2212.02499 §3.2]. MotionBERT keeps every position and zeroes the masked joints and frames, with no learned token [arxiv:2210.06551 code: lib/data/augmentation.py#L71-L74]. STEP's TSFormer drops hidden patches before its encoder, as MAE does [arxiv:2206.09113 §3.1]. - **Mask grid.** The hidden set is drawn on a grid of 32-pixel cells. It is then expanded to the encoder's token grid, which is 4 x 4 for Swin and 16 x 16 for ViT with 16-pixel patches [arxiv:2111.09886 §3.2] [arxiv:2111.09886 code: data/data_simmim.py#L28-L43]. - **Target.** Raw pixels or normalised RGB for the default. MAE normalises each patch by its own statistics in its main results [arxiv:2111.06377 tab. 3]. Discrete targets replace the regression target with a softmax over a codebook: 1024 codes for MAE-VQGAN and MAGVIT, 16,384 for Unified-IO [arxiv:2209.00647 §3.1] [arxiv:2212.05199 §3.1] [arxiv:2206.08916 §3.1]. The SimMIM card does not record the exact loss used for its discrete targets, so check the paper before implementing. Touch in the Wild regresses the whole tactile image rather than only its masked patches [arxiv:2507.15062 eq. 1]. VideoMamba's masked stage aligns to CLIP ViT-B features instead of pixels or codes [arxiv:2403.06977 §3.4]. Painter regresses the stitched output pixels with smooth L1 [arxiv:2212.02499 §3.2]. MotionBERT regresses 3D joints with a velocity term, and for video compares 2D reprojections instead [arxiv:2210.06551 eq. 6] [arxiv:2210.06551 eq. 7]. - **Teacher and prototype targets.** iBOT's target is the EMA teacher's softmax on the unmasked view, not a pixel value. The loss is cross-entropy with centring and sharpening, computed only at masked positions, and a [CLS] loss is added [arxiv:2111.07832 §3.1] [arxiv:2111.07832 alg. 1]. The teacher is updated as an EMA of the student after each step, so it needs a second copy of the network in memory [arxiv:2111.07832 tab. 19]. MSN uses the same kind of EMA teacher, but its target is a prototype assignment, and the anchor is trained by cross-entropy against it [arxiv:2204.07141 §3]. data2vec's target is a regression, not a softmax: the normalised outputs of the teacher's top K blocks are averaged, and the student uses a smooth L1 loss on the masked steps [arxiv:2202.03555 §3.3] [arxiv:2202.03555 §3.4]. CAE's alignment target is the same encoder run on the hidden patches with gradient stopped, not a separate EMA copy [arxiv:2202.03026 §3.2]. The released code uses a second module that is re-pointed at the encoder's tensors after each target pass, so the target follows the encoder's updates [arxiv:2202.03026 code: models/modeling_cae.py#L98-L102]. - **Decoder.** MAE's default decoder has 8 blocks at width 512, and it is discarded after pre-training [arxiv:2111.06377 tab. 1 caption] [arxiv:2111.06377 fig. 1]. A one-block decoder is cheaper and fine-tunes as well, but probes worse [arxiv:2111.06377 tab. 1a]. CAE's decoder reads only the regressor's output at the hidden positions, not the visible tokens [arxiv:2202.03026 §3.1] [arxiv:2202.03026 code: models/modeling_cae_helper.py#L441-L448]. - **Per-image adaptation (TTT-MAE).** The encoder and decoder are copied from the pre-trained weights for each test input, updated for 20 SGD steps on that input's masked reconstruction loss, used once to predict, and then discarded [arxiv:2209.07522 §3] [arxiv:2209.07522 §4.1]. - **Language input (Voltron).** The caption goes through a frozen language encoder and is projected to the encoder width. In a generation example the encoder receives only a null token, and the decoder predicts the caption under a causal mask, so the encoder cannot copy the caption [arxiv:2302.12766 §3.1] [arxiv:2302.12766 app. B.2]. - **Learning rate.** The released SimMIM code takes a base learning rate per 512 images and scales it by the total batch size. Copying the paper's batch-2048 value (8e-4) into such a config multiplies the rate by four [arxiv:2111.09886 §4.1.1] [arxiv:2111.09886 code: main_simmim.py#L218-L229]. MAE uses a base rate of 1.5e-4, scaled by batch size divided by 256 [arxiv:2111.06377 app. A.1]. Check each paper's scaling rule before copying a rate. - **Masking unit.** Whether the mask is drawn per image, per sequence, per batch or per sample changes the training signal. SatMAE's appendix and code differ on per-image versus sequence-wide masking [arxiv:2207.08051 app. A.3.2] [arxiv:2207.08051 code: models_mae_temporal.py#L122-L167]. Touch in the Wild draws its 95% masking decision per batch in code, but per sample in the paper [arxiv:2507.15062 §4.2] [arxiv:2507.15062 code: pretrain_mae/pretrain_mae_model.py#L405-L424]. Choose one and state it. ## Contradictions Not resolved here. Each side is cited, and the choice is left to the reader. - **Linear probing.** SimMIM reports 56.7% for ViT-B and states that it does not optimise for linear evaluation [arxiv:2111.09886 tab. 6] [arxiv:2111.09886 §3.5]. Its linear probe takes the best intermediate layer, trained for 100 epochs [arxiv:2111.09886 §4.2]. iBOT reports 79.5% for ViT-B/16 under the DINO protocol, which sweeps learning rates [arxiv:2111.07832 tab. 1] [arxiv:2111.07832 §4.1]. MAE reports 68.0% for ViT-B [arxiv:2111.06377 app. B], while CAE's table gives 67.8% for ViT-B [arxiv:2202.03026 tab. 1]. CAE's table takes other models' scores from the authors' own implementations unless noted [arxiv:2202.03026 tab. 1]. The protocols differ, so these numbers are not a like-for-like comparison. - **Mask tokens in the encoder.** SimMIM feeds every position, hidden ones as a learned vector [arxiv:2111.09886 §3.2]. MAE reports that the same choice costs 3.3x the FLOPs and 14 points of linear probing at ViT-L/16 [arxiv:2111.06377 tab. 1c]. The settings differ in patch size, ratio, backbone and epochs, so this is evidence about the MAE setting, not a verdict on SimMIM's. MAGVIT's transformer also takes [MASK] positions by design, because it generates [arxiv:2212.05199 §3.2]. - **Mask ratio.** SimMIM's default is 0.6, and its best value in the ablation is 0.5 [arxiv:2111.09886 tab. 1] [arxiv:2111.09886 §4.1.2]. MAE's default is 75% [arxiv:2111.06377 tab. 1]. MSN uses 0.15 to 0.7 depending on the architecture, and it drops patches instead of replacing them [arxiv:2204.07141 app. A.1]. VideoMAE uses 90% with tube masks on Kinetics and Something-Something V2, and 75% on UCF101 and HMDB51 [arxiv:2203.12602 tab. 1] [arxiv:2203.12602 app. Implementation Details]. The cards do not compare these under one setup. CAE uses 50% block-wise masking. Its ViT-B ablation at 300 epochs prefers 60% on linear probing (64.8 against 64.1 at 50%), and 50% on ADE20K (48.3 against 48.1 at 60%) [arxiv:2202.03026 tab. 7]. MotionBERT masks 5% of joints and 10% of frames [arxiv:2210.06551 app. A.1], and its finetuning gains stop above about 45% masking [arxiv:2210.06551 fig. 7]. Its skeleton ratio is therefore far from MAE's 75%. - **BEiT ViT-B/16 fine-tuning.** SimMIM reports 83.2% [arxiv:2111.09886 tab. 6]. iBOT reports 83.4% for BEiT at 800 epochs [arxiv:2111.07832 tab. 2]. The protocols differ, so cite each number with its own paper. - **Low-label performance of MAE.** MSN reports MAE ViT-B/16 at 40.5% with 5 labels per class, using partial fine-tuning, against 61.4% for DINO ViT-B/16 [arxiv:2204.07141 tab. 1]. The MAE card emphasises full-label fine-tuning and gives no low-label numbers [arxiv:2111.06377 tab. 3]. Read the two as different regimes. - **Within SimMIM.** The ablation's best mask ratio is 0.5 (83.0%), but the default is 0.6, which the authors chose for training stability [arxiv:2111.09886 tab. 1] [arxiv:2111.09886 §4.1.2]. The paper's text and released code also differ on schedules, which affects reproduction [arxiv:2111.09886 §4.1.1] [arxiv:2111.09886 code: configs/vit_base__800ep/simmim_pretrain__vit_base__img224__800ep.yaml#L17-L26]. - **Within TTT-MAE.** The text says encoder and decoder are both optimised, with weight decay 0.2 [arxiv:2209.07522 §4.1]. The released command freezes the decoder [arxiv:2209.07522 code: README.md], and its test-time optimiser has no weight decay [arxiv:2209.07522 code: engine_test_time.py#L64-L65]. The appendix reports the decoder choice as negligible [arxiv:2209.07522 app.]. Implement the released setting, or state which one you chose. - **Within iBOT.** The ablation uses 300 epochs, no multi-crop and a fixed ratio of 0.3 [arxiv:2111.07832 §4.4]. Its headline numbers use 800 to 1600 epochs with multi-crop [arxiv:2111.07832 tab. 1]. Do not compare the ablation numbers with the headline ones. The paper's text and its code also differ on crop scales and the ViT-L/16 pre-training length, which the iBOT card lists in its Paper vs code section [arxiv:2111.07832 §3.2] [arxiv:2111.07832 app. B]. - **Within MSN.** The paper says masking 70% "uses half the computation and memory" [arxiv:2204.07141 §1], but its own compute table does not show a halving [arxiv:2204.07141 tab. 10]. Its ViT-S/16 pre-training length is 800 epochs in one table and 600 in another [arxiv:2204.07141 tab. 1] [arxiv:2204.07141 tab. 3]. - **Within Voltron.** The paper writes its objective as an unweighted sum, while the code weights the caption loss by 0.5 [arxiv:2302.12766 §3.2] [arxiv:2302.12766 code: voltron/conf/models.py#L405]. The paper also does not state the mask ratio that the code sets to 0.75 [arxiv:2302.12766 §3.1]. Implementations that follow the text and those that follow the code train different objectives, and the reported numbers do not say which was used. - **Within ALPRO.** The paper drops entity pseudo-labels whose top score is below 0.2 [arxiv:2112.09583 §3.4]. The released code applies the threshold to the entity index, so the rule differs from the text [arxiv:2112.09583 code: src/modeling/alpro_models.py#L527]. Pick one rule and state it. - **Within data2vec.** The paper's ViT-B recipe uses 60% block masking, K = 6, a target taken after the FFN before the last residual, a constant EMA of 0.9998 and drop path 0.2 [arxiv:2202.03555 §4.1] [arxiv:2202.03555 §3.3]. The vision configs in the repository set other values for most of these, including a 75-patch default mask and an annealed EMA [arxiv:2202.03555 code: examples/data2vec/models/data2vec_vision.py#L37-L40] [arxiv:2202.03555 code: examples/data2vec/config/vision/pretraining/base_imagenet_d2v1.yaml#L57-L64]. The paper gives no ablation for the normalisation of targets, so the choice cannot be checked from the text alone [arxiv:2202.03555 §3.3]. - **Within SwinV2.** The main text names SimMIM for the 3B model's masked stage, while the appendix cites that method with an anonymous key [arxiv:2111.09883 §3.4] [arxiv:2111.09883 app. A2]. Treat the stage as SimMIM, with details not stated in the card. - **Within SatMAE.** The text says the gains are 4.48% over the previous state of the art and 2.06% over the non-temporal model. The tables give other differences, for example 81.49 minus 78.90 = 2.59 over MAE with test-time augmentation [arxiv:2207.08051 §5.3]. Cite the table values, and state which ones you quote. The test-time augmentation also differs, as noted above [arxiv:2207.08051 app. A.3.2] [arxiv:2207.08051 code: engine_finetune.py#L256-L264]. - **Within VideoMAE V2.** The paper gives 1,200 epochs, 120 warmup epochs and a base rate of 1.5e-4 for batch 8,192 [arxiv:2303.16727 tab. 10]. The ViT-g script passes 300 epochs and 30 warmup epochs, with a rate of 6e-4 for batch 2,048 [arxiv:2303.16727 code: scripts/pretrain/vit_g_hybrid_pt.sh#L37-L49]. Both routes give the same effective rate of 4.8e-3, but the schedules differ, so an implementer who uses the table gets a different run from the released code. - **Within Touch in the Wild.** The paper gives 60 training epochs for the policy [arxiv:2507.15062 §5.2], and the default config gives 200 [arxiv:2507.15062 code: diffusion_policy/config/train_diffusion_unet_timm_umi_workspace.yaml#L165]. The paper sets the encoder learning rate to 3e-5 throughout, while the code uses 2e-5 for CLIP [arxiv:2507.15062 §4.3] [arxiv:2507.15062 code: diffusion_policy/workspace/train_diffusion_unet_image_workspace.py#L110-L136]. The paper says "two rounds" of cross-attention, and the code runs one step in each direction, with a residual connection [arxiv:2507.15062 §4.2] [arxiv:2507.15062 code: pretrain_mae/pretrain_mae_model.py#L61-L77]. - **Within Painter.** The paper gives 75% block-wise masking and says the two stream outputs are added when merged [arxiv:2212.02499 §3.2]. The script masks 784 of 1568 patches, and the code averages the streams [arxiv:2212.02499 code: Painter/train_painter_vit_large.sh#L12] [arxiv:2212.02499 code: Painter/models_painter.py#L414-L415]. The paper's batch of 2048 for 54K iterations does not match the script's 256 per node for 15 epochs [arxiv:2212.02499 §4.1] [arxiv:2212.02499 code: Painter/train_painter_vit_large.sh#L5-L10]. Choose one and state it. - **Within MotionBERT.** The paper writes the 3D and velocity losses as sums over frames and joints, while the code uses means [arxiv:2210.06551 eq. 6] [arxiv:2210.06551 code: lib/model/loss.py#L56-L62]. The code also adds a scale term that the paper's total loss does not have [arxiv:2210.06551 eq. 8] [arxiv:2210.06551 code: train.py#L186-L188]. The paper gives Adam with no learning-rate decay, while the code uses AdamW and multiplies the rate by 0.99 after each epoch [arxiv:2210.06551 app. A.1] [arxiv:2210.06551 code: train.py#L360-L362]. - **Mask type in video.** VideoMAE V2 uses tube masking in its encoder and argues that it prevents information leaking across time [arxiv:2303.16727 tab. 1] [arxiv:2303.16727 §4.2]. VideoMamba's Something-Something V2 ablation ranks tube masking last of five types [arxiv:2403.06977 tab. 5]. VideoMAE's own ablation puts tube masking ahead of random masking at 90%, 69.6 against 68.3 on Something-Something V2 [arxiv:2203.12602 tab. 1]. That agrees with V2 and not with VideoMamba. The two use different architectures and targets, so the rankings do not transfer. Test the mask type on your own model. - **Within InternVideo.** The text gives 1200 masked-pretraining epochs at a base learning rate of 2.5e-4, and a decoder with half the encoder's width and 4 blocks [arxiv:2212.03191 §3.1] [arxiv:2212.03191 §4.2]. The released hybrid script runs 301 epochs at 1e-3 with a 0.9 mask ratio and decoder depth 8 [arxiv:2212.03191 code: InternVideo1/Pretrain/VideoMAE/scripts/pretrain/slurm_train_vit_h_hybrid_pt.sh#L29-L44]. Its ViT-H model uses a decoder width of 512 against an encoder width of 1280 [arxiv:2212.03191 code: InternVideo1/Pretrain/VideoMAE/modeling_pretrain.py#L464-L469]. Implement one of these and state which. - **Within FLAVA.** Table 5 gives FLAVA 72.8 on VQAv2 test-dev, 79.0 on SNLI-VE and 76.7 on Hateful Memes. The ablation table gives 72.49, 78.89 and 76.09 for the same model. The paper does not explain the gap [arxiv:2112.04482 tab. 5] [arxiv:2112.04482 tab. 4]. - **Within Muse.** The text gives 1M training steps for the models, the base-model table gives 1.5M, and the super-resolution table gives 1M. The paper does not say which applies to the 3B model [arxiv:2301.00704 §3] [arxiv:2301.00704 tab. 4] [arxiv:2301.00704 tab. 5]. - **Masked patch losses in vision-language models.** METER finds that adding either masked-patch loss lowers VQAv2 and retrieval scores in its setup [arxiv:2111.02387 tab. 7]. It reads this as a sign that conclusions from region-based models may not carry over [arxiv:2111.02387 §4.4]. CAE's ablation on image-only pre-training finds that its masked objective raises linear, attentive and ADE20K scores over the baseline [arxiv:2202.03026 tab. 5]. The settings differ in model, data and task, so the cards do not settle which result holds for your setup. - **Within CAE.** The paper's §3.1 names the visible representations and the previous regressor output as the regressor's keys and values. The code always adds the hidden tokens to them [arxiv:2202.03026 §3.1] [arxiv:2202.03026 code: models/modeling_cae_helper.py#L362-L366]. The Table 1 CAE rows use the d-VAE tokenizer, while the code defaults to DALL-E [arxiv:2202.03026 tab. 1] [arxiv:2202.03026 code: tools/run_pretraining.py#L28]. Choose one of each and state it. ## Papers - [SimMIM: A Simple Framework for Masked Image Modeling](/papers/2111.09886): pixel regression with a linear head, an L1 loss on hidden pixels and a 32-pixel mask. It has the most ablations on the design choices above. - [Masked Autoencoders Are Scalable Vision Learners](/papers/2111.06377): a visible-only encoder with 75% random masking and a light decoder, with ablations on decoder depth and mask tokens. - [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](/papers/2203.12602): the MAE recipe on video, with 90% tube masking on a plain ViT-B and pre-training on video alone. - [iBOT: Image BERT Pre-Training with Online Tokenizer](/papers/2111.07832): MIM with targets from an EMA teacher, trained jointly with a [CLS] self-distillation loss. - [Masked Siamese Networks for Label-Efficient Learning](/papers/2204.07141): prototype matching on a masked view with no pixel reconstruction, aimed at 1 to 5 labels per class. - [Test-Time Training with Masked Autoencoders](/papers/2209.07522): per-image MAE reconstruction at test time, with the weights reset after each input. - [Masked Visual Pre-training for Motor Control](/papers/2203.06173): a frozen MAE on hand-object images, used as the visual encoder for PPO robot controllers. - [Unified-IO: A unified model for vision, language, and multi-modal tasks](/papers/2206.08916): masked image denoising as one of two pre-training objectives in a single encoder-decoder over discrete tokens. - [Visual Prompting via Image Inpainting](/papers/2209.00647): an MAE that predicts VQGAN codes, used to solve new image tasks by inpainting a grid prompt. - [FLAVA: A Foundational Language And Vision Alignment Model](/papers/2112.04482): one image-text model trained with a BEiT-style masked image loss on dVAE tokens, alongside contrastive, masked language and matching losses. - [Muse: Text-To-Image Generation via Masked Generative Transformers](/papers/2301.00704): a masked-token transformer over VQGAN codes that generates images by parallel decoding. - [MAGVIT: Masked Generative Video Transformer](/papers/2212.05199): masked token modeling over 3D video tokens, with a 12-step parallel decoder for generation. - [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](/papers/2412.02732): an MAE for satellite time series, with 3D patches and optional date and location metadata. - [Language-Driven Representation Learning for Robotics](/papers/2302.12766): Voltron, a frame-masking autoencoder over video whose caption is either a condition for pixel reconstruction or a generation target. - [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](/papers/2112.09583): ALPRO, a video-text model whose extra term predicts entity pseudo-labels on random crops from a frozen prompter. - [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](/papers/2202.03555): one self-distillation recipe across modalities, regressing the averaged top K blocks of an EMA teacher on masked steps. - [Swin Transformer V2: Scaling Up Capacity and Resolution](/papers/2111.09883): scales Swin to 3B parameters, with SimMIM as the self-supervised stage before labelled training. - [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](/papers/2207.08051): an MAE over satellite time series and band groups, with per-image masking and temporal and group codes. - [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](/papers/2303.16727): video MAE scaled to ViT-g, with a second, decoder-side mask that cuts compute and memory. - [Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper](/papers/2507.15062): masked reconstruction of a tactile image, conditioned on a CLIP view, used as a pretrained encoder for a diffusion policy. - [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191): a masked video ViT-H and a CLIP-initialised video-text encoder, trained separately and then fused with cross-model attention on frozen backbones. - [VideoMamba: State Space Model for Efficient Video Understanding](/papers/2403.06977): a bidirectional Mamba video encoder whose masked stage aligns 80%-masked video to CLIP ViT-B features, with self-distillation for scaling. - [An Empirical Study of Training End-to-End Vision-and-Language Transformers](/papers/2111.02387): METER, a detector-free vision-language model whose tests of masked patch losses lowered scores in its setup. - [Context Autoencoder for Self-Supervised Representation Learning](/papers/2202.03026): CAE, which separates encoding of visible patches, cross-attention regression of hidden features and a token decoder. - [Aim My Robot: Precision Local Navigation to Any Object](/papers/2411.14770): a frozen MAE encoder fused with depth, LiDAR and goal tokens in a transformer that predicts waypoints and camera tilt. - [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](/papers/2212.02499): one masked-output model over stitched pairs of the same task, which turns dense outputs into RGB images and is prompted with a solved pair. - [MotionBERT: A Unified Perspective on Learning Human Motion Representations](/papers/2210.06551): masked 2D skeleton sequences are lifted to 3D by a DSTformer encoder, then finetuned for pose, action and mesh recovery. - [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](/papers/2206.09113): STEP, masked patch pre-training of a time-series Transformer on traffic history, whose frozen output feeds a graph forecaster.