Paper · 2022-05-26 · cs.CV, cs.AI, cs.LG
Revealing the Dark Secrets of Masked Image Modeling
Compares masked image modeling (SimMIM) with supervised pre-training in ViT, Swin and RepLKNet models. MIM adds locality and diverse attention heads, keeps layer representations similar, and transfers better to pose, depth and tracking. Author-reported headline: SwinV2-L MIM reaches 78.1 AP on COCO val and 0.287 RMSE on NYUv2.
Masked image modelingSelf-supervised learningAttention interpretabilityVision Transformer (ViT) backbonespretrainingvisionanalysismasked-modeling
arXivPDFCodeMarkdown18 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Pre-training a vision model by hiding parts of images and asking it to fill them in has become a strong alternative to pre-training on labelled ImageNet classes. This paper asks what that hiding task actually teaches the network. The authors look inside the trained models and compare them with label-trained ones: how far each attention head looks, how focused it is, whether heads look at different things, and whether the layers learn similar representations.
The main finding is that hiding-and-filling pushes the network toward looking at nearby pixels in some heads while other heads still look far away, and each head keeps its own job across all layers. Label-trained networks lose that variety in their last layers. The same models also transfer differently: the hiding-trained ones do better on tasks about where things are and how they move, such as pose, depth and tracking, and are roughly even on tasks whose categories look like ImageNet's.
TL;DR
- Masked image modeling (MIM, here SimMIM) gives ViT and Swin models local attention in part of the heads, more head diversity in every layer, and layer representations that look alike (CKA about 0.9 to 1.0) §3.
- Supervised models lose head diversity in the last layers and differ more between layers §3.1.3 §3.2.
- Author-reported: a MIM-pretrained SwinV2-L reaches 78.1 AP on COCO val tab. 2, 0.287 RMSE on NYUv2 and 1.966 on KITTI tab. 2, and 70.7 SUC on LaSOT tab. 2. The abstract's 78.9 AP (test-dev) and 0.287/1.966 depth numbers are the table's values; the abstract's 78.9 AP (test-dev) uses a stronger person detector §4.2.
- Not a new method: the paper is an analysis. Its only reusable procedures are the attention and CKA measurements and the fine-tuning recipes.
Problem
Masked image modeling works well for vision transfer, but it was unclear what it learns and where its gains come from §1. Supervised ImageNet pre-training had been the default; the paper asks (1) what internal mechanisms make MIM work and (2) which task types MIM transfers to better than supervised pre-training §1. The paper's analysis is restricted to the backbones it names: ViT-B, Swin (SwinV2) and RepLKNet-31B §2 app. B.
Method
The paper has no training method of its own. It uses SimMIM as the MIM pre-training framework §2. SimMIM, as summarised in the paper, hides random blocks of the image, using a fairly large block size (the paper's example is 32 pixels), feeds masked and visible tokens together to the encoder, uses a linear prediction head, and regresses raw RGB pixels with an ℓ1 loss §2.
The paper's analysis has three parts.
A. Attention-map analysis on ViT-B (§3.1). Three measurements per head and layer, for supervised (DeiT), contrastive (MoCo v3) and MIM (SimMIM) ViT-B:
- Averaged attention distance per head, to see local versus global attention §3.1.1 fig. 1. The paper's formula for this distance is not stated (the text gives only the qualitative definition).
- Entropy of each head's attention distribution, to see focused versus broad attention §3.1.2 fig. 3. The entropy formula is not stated.
- KL divergence between the attention distributions of different heads in the same layer, to see whether heads attend to different tokens §3.1.3 fig. 4. The KL formula is not stated (only cited to Kullback–Leibler).
The paper does not state which images or how many images are used to compute these statistics, or whether values are averaged over a batch or a validation set.
B. Layer-wise representation similarity (§3.2). Centered kernel alignment (CKA) between the feature maps of different layers §3.2 fig. 6. With features X (N × D1) and Y (N × D2) and Gram matrices K = X Xᵀ, L = Y Yᵀ, the paper defines CKA as HSIC(K, L) divided by the square root of HSIC(K, K) times HSIC(L, L) eq. 1 app. A. The HSIC estimator is cited, not written out, and the paper does not say whether the Gram matrices are centred before HSIC (the name suggests they are; the paper does not confirm it).
# Layer similarity by CKA [§3.2] [eq. 1] [app. A]
# features_a: (N, D1) layer i; features_b: (N, D2) layer j
def layer_cka(features_a, features_b):
# Gram matrices over the N examples [app. A]
gram_a = features_a @ features_a.T # (N, N)
gram_b = features_b @ features_b.T # (N, N)
# Normalised HSIC; centring not stated in the paper [eq. 1]
cross = hsic(gram_a, gram_b)
self_a = hsic(gram_a, gram_a)
self_b = hsic(gram_b, gram_b)
return cross / sqrt(self_a * self_b)C. Transfer tests (§3.1.3, §4). Two ways of testing whether heads and layers matter for downstream tasks:
- Drop the last several layers of a pretrained SwinV2-B and fine-tune on COCO pose estimation and NYUv2 depth estimation §3.1.3 fig. 5. The supervised model improves on pose when layers are dropped; the MIM model does not show this effect §3.1.3.
- Load pretrained weights of randomly shuffled layers before fine-tuning on COCO pose and NYUv2 depth §3.2. For the layer-sampling detail, see app. E: nine of the eighteen third-stage layer weights are sampled at random and loaded into the first nine layers, for the pose head at the ninth-layer feature.
# Random layer-weight test for one backbone [§3.2] [app. E]
# stage3_weights: list of 18 pretrained layer weights
def shuffled_init(stage3_weights):
# Sample 9 of the 18 pretrained layers without a stated scheme [app. E]
sampled = random_sample(stage3_weights, 9)
# Load them into the first 9 layers; run fine-tuning as usual [app. E]
return sampledD. Large-kernel ConvNet check (RepLKNet-31B, app. B). The attention analysis is repeated on RepLKNet-31B by treating each depth-wise kernel channel as an attention head, after normalising each channel over H × W app. B. The MIM model is pretrained without the re-parametrisation trick app. B.
E. Downstream benchmarks (§4). Fine-tuning comparisons on semantic classification (CoG, 12-dataset benchmark, iNat18) §4.1 tab. 1 tab. 4 tab. 5; geometric and motion tasks (pose, depth, tracking) §4.2 tab. 2; and combined tasks (COCO detection, ADE-20K segmentation) §4.3 tab. 6. Protocols and hyperparameters are in app. E tab. 7.
Hyperparameters
Values stated in the paper text or tables, with the released depth-estimation code where it adds detail. Where code and paper differ, the code value is in the Paper vs code section.
| Name | Value | Where | Notes |
|---|---|---|---|
| Pre-training framework | SimMIM | §2 | Pre-training hyperparameters (mask ratio, patch size, epochs) for the SwinV2 MIM models are not stated in this paper |
| MIM mask patch size (general) | e.g. 32 | §2 | "moderately large"; the paper's example value |
| RepLKNet MIM input | 192 × 192 | app. B | 6 × 6 patches, 60% masked, 1×1 conv before stem |
| RepLKNet MIM optimizer | AdamW, wd 5e-2, base lr 4e-4 | app. B | warm-up 10 epochs, lr to 4e-5 at epoch 260, 300 epochs total |
| RepLKNet MIM batch / drop path / clip | 2048 / 0.1 / gradient clipping (value not stated) | app. B | |
| RepLKNet MIM augmentation | RRC scale [0.67, 1], ratio [3/4, 4/3], h-flip | app. B | |
| RepLKNet fine-tune (ImageNet-1K) | AdamW, wd 5e-2, base lr 5e-3, layer decay 0.8 | app. B | cosine, 20 warm-up, 100 epochs, batch 2048, drop path 0.1 |
| Semantic fine-tune: epochs / warm-up | 100 / 20 | §4.1 app. E | input 224 × 224 |
| Semantic fine-tune: optimizer | AdamW, betas (0.9, 0.999), eps 1e-8, cosine | app. E tab. 7 | |
| Semantic fine-tune: window / patch | 14 / 4 | tab. 7 | |
| Semantic fine-tune: base lr, layer decay, wd, drop path | see table | tab. 7 | differ per pretrain and per dataset family |
| Semantic fine-tune: batch / clip / label smoothing | 2048 / 5.0 / 0.1 | tab. 7 | |
| Semantic fine-tune: augmentation | RRC (0.08, 1), ratio (3/4, 4/3), h-flip 0.5, colour jitter 0.4, RandAugment 9/0.5, erasing 0.25, mixup 0.8, cutmix 1.0 | tab. 7 | |
| Pose estimation: input | 256 × 256 | app. E | heatmaps at 4× resolution §4.2 |
| Pose: optimizer | AdamW, base lr 5e-4, wd 5e-2 | app. E | lr dropped to 5e-5 at epoch 120; 150 epochs |
| Pose: layer decay / drop path | 0.9 (Swin-B), 0.85 (Swin-L) / 0.3 (B), 0.5 (L) | app. E | |
| Pose: batch size | 512 | app. E | |
| Pose: augmentation | flip, half body, scale (0.5, 1.5), rotation (−40°, 40°), grid dropout, colour jitter (h 0.2, s 0.4, c 0.4, b 0.4) | app. E | |
| Pose: decoder | deconvolutions upsample last backbone feature to 4× | §4.2 app. E | |
| Pose: test | flip averaging; UDP for quantisation | app. E | |
| Pose: person detection | COCO: prior methods' detections; CrowdPose: cascade Mask R-CNN (Swin-B, COCO-trained) | app. E | |
| Depth: epochs | 25 | §4.2 app. E | |
| Depth: lr schedule | polynomial, factor 0.9, min 3e-5, max 5e-4 | app. E | shape of the schedule not fully stated (see pitfalls) |
| Depth: batch size | 24 | app. E | |
| Depth: optimizer / wd / layer decay / drop path | AdamW / 5e-2 / same as pose / same as pose | app. E | |
| Depth: crops | 480 × 480 (NYUv2), 352 × 352 (KITTI) | app. E | |
| Depth: max range | 10 m (NYUv2), 80 m (KITTI) | §4.2 app. E | |
| Depth: augmentation | h-flip; brightness (−0.2, 0.2); gamma (−0.2, 0.2); hue (−20, 20); saturation (−30, 30); value (−20, 20); vertical CutDepth | app. E | |
| Depth: head | three deconvolutions with BN and ReLU, then one convolution; deconv kernel 2, filter 32 | app. E | code head is different (see Paper vs code) |
| Depth: test | flip testing and sliding window (2 windows NYUv2, 16 windows KITTI) | app. E | |
| Tracking: sampling / epochs | 131072 pairs per epoch / 300 | §4.2 app. E | |
| Tracking: optimizer | AdamW, lr 5e-4 head, 5e-5 backbone, wd 1e-4 | app. E | lr × 0.1 at epoch 210; batch 160 |
| Tracking: input sizes | search 224 × 224; template 112 × 112 | §4.2 app. E | SwinTrack framework; augmentation and inference "as SwinTrack" |
| Detection (COCO): setup | Mask R-CNN, 3× schedule (36 epochs), batch 32, wd 0.05 | §4.3 app. E | |
| Detection: lr / drop path | supervised 6e-5, MIM 8e-5 / supervised 0.3, MIM 0.1 | app. E | window size 14 for both; large-jitter 1024 × 1024, scale [0.1, 2.0] |
| Segmentation (ADE-20K, UperNet) | AdamW, wd 0.05, batch 32; lr 8e-5 (sup), 1e-4 (MIM); layer decay 0.95 | app. E | 80K iterations; 640 × 640 input; window 20; drop path 0.3 / 0.1; test 2560 × 640 |
| Segmentation (ADE-20K, Mask2Former) | AdamW, wd 0.05, batch 32 (not stated); lr 1e-4 (sup), 3e-4 (MIM); backbone lr × 0.1 | app. E | 160K iterations; 512 × 512; scale [0.5, 2]; window 8; drop path 0.3; patch 16; test 2048 × 512 |
| Classification (CoG, iNat18) full grid | see tab. 7 | tab. 7 | CoG tuned on L1 and reused for L2–L5 app. E |
| K12 search | drop path 0.2 (sup); MIM 0.0 on small sets, else 0.2; wd 0.05; lr grid 1.25e-4 to 2.5e-3 (MIM), 2.5e-5 to 5e-4 (sup); layer decay grid 0.75–0.95 | app. E | final lr / layer decay / drop path adjusted near the grid optimum |
Table 7 (full classification fine-tuning settings) tab. 7, as printed in the paper. RAND = randomly initialised; columns are CoG (L1–L5) and iNat18 per pretraining.
| Hyperparameter | RAND CoG | RAND iNat | 1K-SUP CoG | 1K-SUP iNat | 1K-MIM CoG | 1K-MIM iNat |
|---|---|---|---|---|---|---|
| Input size | 224 | 224 | 224 | 224 | 224 | 224 |
| Window size | 14 | 14 | 14 | 14 | 14 | 14 |
| Patch size | 4 | 4 | 4 | 4 | 4 | 4 |
| Training epochs | 300 | 300 | 100 | 100 | 100 | 100 |
| Warm-up epochs | 20 | 20 | 20 | 20 | 20 | 20 |
| Layer decay | 1.0 | 1.0 | 0.85 | 0.9 | 0.8 | 0.75 |
| Batch size | 2048 | 2048 | 2048 | 2048 | 2048 | 2048 |
| Optimizer | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW |
| Base learning rate | 2e-3 | 4e-3 | 2e-4 | 1.6e-3 | 5e-3 | 1.6e-2 |
| Weight decay | 0.05 | 0.1 | 0.05 | 0.1 | 0.05 | 0.1 |
| Adam ε | 1e-8 | 1e-8 | 1e-8 | 1e-8 | 1e-8 | 1e-8 |
| Adam β | (0.9, 0.999) | (0.9, 0.999) | (0.9, 0.999) | (0.9, 0.999) | (0.9, 0.999) | (0.9, 0.999) |
| Learning-rate scheduler | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine |
| Gradient clipping | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 |
| Stochastic depth | 0.5 | 0.5 | 0.2 | 0.2 | 0.2 | 0.2 |
| Label smoothing | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 |
| Random crop scale | (0.08, 1) | (0.08, 1) | (0.08, 1) | (0.08, 1) | (0.08, 1) | (0.08, 1) |
| Random resize ratio | (3/4, 4/3) | (3/4, 4/3) | (3/4, 4/3) | (3/4, 4/3) | (3/4, 4/3) | (3/4, 4/3) |
| Random horizontal flip | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |
| Colour jitter | 0.4 | 0.4 | 0.4 | 0.4 | 0.4 | 0.4 |
| RandAugment | 9 / 0.5 | 9 / 0.5 | 9 / 0.5 | 9 / 0.5 | 9 / 0.5 | 9 / 0.5 |
| Random erasing probability | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 |
| Mixup probability | 0.8 | 0.8 | 0.8 | 0.8 | 0.8 | 0.8 |
| CutMix probability | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
Results
Author-reported. All numbers are from the paper; "SUP" is supervised pre-training and "MIM" is SimMIM pre-training with the stated ImageNet subset. Each result uses SwinV2 unless stated. Figures are described in words only.
Semantic understanding tab. 1 (SwinV2-B; top-1 accuracy, mean per-class for Aircraft, Pets, Caltech, Flowers §4.1):
| pre-train | CoG L1 | CoG L2 | CoG L3 | CoG L4 | CoG L5 | Food | Birdsnap | Cars | Aircraft | Average (7) | iNat18 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K-SUP | 79.4 | 76.2 | 72.7 | 72.5 | 68.4 | 93.2 | 81.8 | 88.6 | 83.0 | 89.7 | 77.7 |
| 1K-MIM | 79.6 | 77.1 | 73.6 | 73.0 | 69.1 | 94.2 | 83.7 | 89.2 | 83.5 | 86.1 | 79.6 |
Geometric, motion and depth tab. 2 (pose: AP ↑; depth: RMSE ↓; GOT10k: AO ↑; TrackingNet and LaSOT: SUC ↑). Bold in the paper marks the best of the pretrainings in the table:
| backbone | pre-train | COCO val | COCO test-dev | CrowdPose | NYUv2 | KITTI | GOT10k test | TrackingNet | LaSOT |
|---|---|---|---|---|---|---|---|---|---|
| SwinV2-B | 1K-SUP | 75.2 | 74.5 | 70.7 | 0.352 | 2.313 | 70.1 | 81.5 | 69.4 |
| SwinV2-B | 22K-SUP | 75.9 | 75.1 | 72.2 | 0.335 | 2.240 | 69.9 | 81.0 | 67.8 |
| SwinV2-B | 1K-MIM | 77.6 | 76.7 | 74.9 | 0.304 | 2.050 | 70.8 | 82.0 | 70.0 |
| SwinV2-L | 22K-SUP | 76.5 | 75.7 | 72.7 | 0.334 | 2.150 | 71.1 | 81.5 | 69.2 |
| SwinV2-L | 1K-MIM | 78.1 | 77.2 | 75.5 | 0.287 | 1.966 | 72.9 | 82.5 | 70.7 |
| representative | HRFormer / BinsFormer / MixFormer | 77.2 | 76.2 | 72.5 | 0.330 | 2.098 | 75.6 | 83.9 | 70.1 |
The representative row gives HRFormer for pose, BinsFormer for depth and MixFormer for tracking tab. 2. Paper text: MIM-pretrained SwinV2-L with a stronger detector (BigDetection) reaches 80.5 AP on COCO val, 78.9 on test-dev and 78.0 on CrowdPose §4.2; that detector setting is not the one in the table.
Large-kernel ConvNet (RepLKNet-31B) tab. 3 app. B:
| backbone | pre-train | ImageNet-1K top-1 | COCO val | COCO test | CrowdPose |
|---|---|---|---|---|---|
| RepLKNet-31B | 1K-SUP w/ reparam. | 83.5 | 74.6 | 73.9 | 70.2 |
| RepLKNet-31B | 1K-MIM w/o reparam. | 83.3 | 76.5 | 75.8 | 72.4 |
Kornblith 12-dataset benchmark, per dataset (SwinV2-B; Pascal VOC 2007 excluded) tab. 4 app. C:
| pre-train | Food101 | Birdsnap | Stanford Cars | FGVC Aircraft | Oxford Pets | Caltech101 | Flowers102 | DTD | SUN397 | CIFAR10 | CIFAR100 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K-SUP | 93.2 | 81.7 | 88.6 | 83.0 | 95.9 | 91.9 | 97.7 | 80.3 | 72.3 | 99.1 | 91.0 |
| 1K-MIM | 94.2 | 83.7 | 89.2 | 83.5 | 90.9 | 85.5 | 91.4 | 73.4 | 70.8 | 99.2 | 91.4 |
CoG and iNat18 with a random-init baseline tab. 5 app. C:
| pre-train | CoG L1 | CoG L2 | CoG L3 | CoG L4 | CoG L5 | iNat18 |
|---|---|---|---|---|---|---|
| RAND | 79.4 | 76.7 | 73.1 | 72.7 | 68.5 | 76.5 |
| 1K-SUP | 79.4 | 76.2 | 72.7 | 72.5 | 68.4 | 77.7 |
| 1K-MIM | 79.6 | 77.1 | 73.6 | 73.0 | 69.1 | 79.6 |
Combined tasks tab. 6 §4.3 app. D (SwinV2-B):
| pre-train | COCO Mask R-CNN AP box | AP mask | ADE-20K UperNet mIoU | ADE-20K Mask2Former mIoU |
|---|---|---|---|---|
| 1K-SUP | 51.9 | 45.7 | 50.9 | 52.3 |
| 1K-MIM | 52.9 | 46.7 | 49.3 (−1.6) | 51.7 (−0.6) |
Layer-shuffle test §3.2: pose AP drops by 0.3 for MIM (75.5 → 75.2) versus 0.9 for SUP (75.8 → 74.9); depth RMSE rises by 0.052 for MIM (0.382 → 0.434) versus 0.067 for SUP (0.376 → 0.443). Number of random draws and seeds are not stated.
Attention, entropy and CKA results (figures, described) fig. 1–fig. 6: MIM ViT-B heads span local and global distances in every layer, with distance slightly shrinking in deeper layers; supervised heads move from local at low layers to global at high layers fig. 1 §3.1.1. MIM entropies vary across heads in all layers fig. 3. KL divergence between heads stays large in all layers for MIM and falls toward zero in the last three layers for supervised models fig. 4 §3.1.3. Pairwise layer CKA is about 0.9–1.0 for MIM and spreads from about 0.5 to 1.0 for supervised models fig. 6 §3.2. The Swin and RepLKNet versions of these plots (figures 8–14 in source order) agree in direction app. A app. B.
Dropped-layer test fig. 5 §3.1.3: dropping two to eight final layers of a supervised SwinV2-B keeps or improves COCO pose AP and keeps NYUv2 depth about the same; the paper gives no table for these numbers.
Object detection loss fig. 7 §4.3: MIM lowers the localisation loss faster; the supervised model benefits more on the classification loss. The paper's explanation is that COCO categories are well covered by ImageNet.
Paper vs code
Official depth-estimation code: SwinTransformer/MIM-Depth-Estimation, linked
from the repository's own README as the official implementation of this paper,
at commit 59ba08d66033b9e3469f67e8c5c6f2c325398dce. The owner is the Swin
Transformer organisation; the README credits the Swin code and GLPDepth.
The paper itself does not link any code. The repo's README self-describes it as the
official implementation, but the paper names no link and the owner's members
are not checked against the author list. Ownership is therefore taken from the
repo's self-description, not from a separate author check.
What the repository contains. Monocular depth estimation only, on NYUv2 and
KITTI, with a SwinV2 backbone. It has no pose estimation, tracking, detection,
segmentation, classification, attention-analysis, CKA or layer-shuffle code.
Those experiments are not in this repository. Pretrained MIM weights are loaded
from files the README names (swin_v2_base_simmim.pth); the pretraining
itself is not in this repo.
Searched for code: the paper's text and bibliography for URLs (none);
GitHub repository search for the paper title (one hit, the depth repository
above); GitHub search for "masked image modeling attention" within the
microsoft organisation (no hits).
Discrepancies and details (paper vs code, with citations):
- Learning-rate schedule. Paper: "polynomial strategy with a factor of 0.9, min
3e-5, max 5e-4" app. E. Code: the rate rises from min to max over the first
half of training (
epochs // 2= 12 of 25 epochs), then falls back to min with the same 0.9 exponent train.py. The paper does not state the rise-and-fall shape. - Layer decay. Paper: 0.9 (Swin-B) and 0.85 (Swin-L) app. E. Code matches these values in the README commands README.md and uses default 0.9 train_options.py. The number of layers used for the decay exponent is set by the code's layer-count rule, which the paper does not state optimizer.py optimizer.py.
- Drop path, crop, batch and epochs match the paper: drop path 0.3 / 0.5
base_options.py [README.md]; crops 480 / 352 [README.md];
batch 24 base_options.py; 25 epochs
train_options.py. The
--crop_h/--crop_wargparse defaults (448 × 576) differ from the paper and are overridden by the README commands train_options.py. - Depth head. Paper: three deconvolutions with BN and ReLU, then one convolution app. E. Code: three deconvolution blocks with BN and ReLU model.py, a 3×3 convolution, BN and ReLU before the upsample model.py, and a final output head with two 3×3 convolutions and a ReLU between them model.py. The paper's "one convolution" does not match the code's two-convolution output head.
- Window sizes. The depth commands use window size 30 (NYUv2) and 22 (KITTI), pretrain window 12, and shift sizes 2 or 16 [README.md]. The paper does not state depth window sizes; its window size 14 in tab. 7 is for classification.
- Augmentation. Paper lists random vertical CutDepth app. E. The released code has horizontal flip, random crop, brightness/contrast, gamma and hue/ saturation/value transforms base_dataset.py; a CutDepth transform is not in the released depth code. The paper's brightness, gamma, hue, saturation and value ranges are not passed to those transforms in the code.
- Test-time averaging. The code averages flip and sliding-window predictions test.py train.py. The paper's counts (two windows for NYUv2, sixteen for KITTI) are not checked against the code here.
License conflict (unresolved). The repository's LICENSE is MIT (copyright
"Swin Transformer", 2023). The depth source files, including
configs/train_options.py, train.py, test.py, models/optimizer.py's
header in the GLPDepth-derived files, including train.py, test.py,
configs/train_options.py, dataset/nyudepthv2.py and dataset/kitti.py,
which say "For non-commercial purpose only (research, evaluation etc)"
train_options.py. models/optimizer.py is a SimMIM file
under MIT optimizer.py. The conflict is
between the repository's MIT licence and the non-commercial headers in files it
derives from GLPDepth. Because CLAUDE.md says to leave out anything in doubt,
this card quotes no code and records the code licence as unresolved. Facts
taken from configs and commands are cited by path. The code field should not
be treated as licence-clear for commercial reuse until a human decides.
Pitfalls
- The paper is an analysis, not a method. Nothing here can be implemented as a new model. Implementers should treat the numbers as the authors' measurements under their setup.
- The attention metrics are underdefined. The formulas for averaged attention distance, entropy and KL divergence are not stated, and neither are the images, sample size or averaging used to compute them §3.1. Only the CKA formula is given eq. 1 app. A, and it does not say whether the Gram matrices are centred.
- The K12 average favours supervised pre-training. tab. 1 shows Average (7) at 89.7 for SUP and 86.1 for MIM, while the text emphasises the fine-grained gains §4.1. The seven datasets in the average are not named. The per-dataset table tab. 4 shows MIM below SUP on Oxford Pets (90.9 vs 95.9), Caltech101 (85.5 vs 91.9), Flowers102 (91.4 vs 97.7), DTD (73.4 vs 80.3) and SUN397 (70.8 vs 72.3). The appendix table lists 11 datasets (Pascal VOC excluded).
- Birdsnap SUP differs between tables. It is 81.8 in tab. 1 and 81.7 in tab. 4. Report which one you use.
- Two different COCO detectors. The paper's headline 78.9 test-dev and 80.5 val use BigDetection person detections, not the detector in tab. 2 (78.1 val, 77.2 test-dev) §4.2 app. E.
- Model sizes do not match across comparisons. The 22K-supervised baseline exists only for SwinV2-B and SwinV2-L in tab. 2; the classification comparisons use only SwinV2-B. Compare each table within its own backbone.
- Learning rates sit outside the stated grids. The K12 grid is 1.25e-4 to 2.5e-3 for MIM and 2.5e-5 to 5e-4 for SUP app. E. The full-classification table uses MIM 5e-3 for CoG and 1.6e-2 for iNat, and SUP 1.6e-3 for iNat tab. 7. The paper does not say how the iNat values were chosen; the text describes the grid only for K12.
- Epoch counts are described inconsistently. The text says 100 epochs with 20 warm-up for the classification fine-tuning §4.1; the table tab. 7 also lists 300 epochs for RAND. The RAND rows exist only as a baseline.
- Detection and segmentation settings differ by pretrain. Learning rates, drop path (0.3 vs 0.1) and layer decay differ between SUP and MIM in app. E. Part of the claimed MIM advantage on COCO therefore comes from these tuned settings; the paper does not run a single shared setting.
- Segmentation reverses on UperNet. MIM is lower on ADE-20K for both UperNet (−1.6) and Mask2Former (−0.6) tab. 6. The paper's explanation (localisation vs recognition) is a hypothesis in the text app. D.
- Pre-training details for the SwinV2 MIM models are missing. Only SimMIM's general design is given §2; epochs, mask ratio and patch size for these specific checkpoints are not stated. The RepLKNet MIM setup app. B is the only one with those values.
- Layer-shuffle results have no seeds or variance. The number of random draws and whether the 0.3/0.9 point drops are within noise are not stated §3.2 app. E.
- Dropped-layer results have no table. The effect is described from a figure only fig. 5 §3.1.3; use the figure to decide, do not extract numbers from the text.
- Depth schedule shape and head. Use the code's up-then-down schedule train.py if you want to match the released depth results, and say so; the paper's text describes only the factor and bounds. The released head has two output convolutions model.py, which the paper's text does not describe.
- Code licence is unresolved (see Paper vs code). Do not copy code from the repository until a human clears the non-commercial headers.
- Figures are not reproduced. All figure statements are the paper's description; check the cited figure before relying on a detail.
Lineage
- Builds on: SimMIM (Xie et al., 2021) as the MIM framework §2; Swin and SwinV2 as backbones §2 app. A; ViT §2; RepLKNet for large-kernel ConvNets app. B; Simple Baseline for the pose head app. E; GLPDepth for the depth code base README.md; SwinTrack for tracking §4.2.
- Compared against: supervised ImageNet pre-training (1K and 22K) tab. 2; a contrastive model (MoCo v3) in the ViT analysis §3.1; HRFormer, BinsFormer and MixFormer as representative results tab. 2.
- Superseded by: not stated in this card's sources.
Source and license
Adapted from “Revealing the Dark Secrets of Masked Image Modeling” by Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, Yue Cao (arXiv:2205.13543), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/SwinTransformer/MIM-Depth-Estimation at commit 59ba08d66033 (MIT (LICENSE file); GLPDepth-derived files say 'For non-commercial purpose only' (unresolved, see Paper vs code)). Any quoted code is shown under that license.