wikillm

Paper · 2022-03-23 · cs.CV

What to Hide from Your Students: Attention-Guided Masked Image Modeling

AttMask replaces random token masking in distillation-based masked image modeling (iBOT, DINO) with masks from the teacher ViT's last-layer [CLS] attention: the most attended patches are hidden from the student. Author-reported: on full ImageNet-1k (ViT-S/16, 100 epochs), iBOT+AttMask raises k-NN from 71.5 to 72.5.

Masked image modelingSelf-supervised learningVision Transformer (ViT) backbonesTeacher-student learningself-supervisedmasked-image-modelingvision-transformerdistillation

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Some self-supervised vision models learn by hiding parts of a picture and trying to fill them in, much like a cloze test for images. Most methods hide random patches. This paper argues that random hiding is too easy for images, because a picture has many patches that repeat the same information, so the missing parts can often be guessed from their neighbours.

The authors propose hiding the parts the model itself pays most attention to instead. A "teacher" network looks at the full picture and shows which regions it focuses on for its summary of the image. The "student" network then sees the picture with those focus regions hidden, and must predict what the teacher thought about them. Because the teacher already computes this attention, the method adds no extra forward pass.

The idea is meant to make the training task harder in the useful way. The authors report that the learned features are better and learned faster, and that the gains are larger when data is limited and when the features are used without fine-tuning.

TL;DR

AttMask is a masking rule for distillation-based masked image modeling with vision transformers. A teacher encoder sees the whole image, and the attention of its [CLS] token over the patches decides which patches the student must reconstruct: the most attended ones by default. The authors plug it into iBOT and DINO and report better k-NN and linear accuracy, faster training, and more robust features than random block masking. All numbers are author-reported.

Problem

Masked image modeling hides a fraction of image tokens and trains a transformer to predict them. The usual choice of which tokens to hide is random, borrowed from text. The authors argue that image tokens are numerous and redundant, so random hiding often leaves enough visible context to recover the hidden content without learning much about objects §1. The question they ask is whether the hidden set can be chosen informatively §1.

The method is designed for distillation-based MIM (iBOT) and self-distillation (DINO), where a teacher with an exponential moving average of the student supplies targets §3.1. Hiding random block-wise tokens is the default of iBOT tab. 1 §4.2.

Method

The method builds on the ViT encoder and the distillation losses of iBOT and DINO. Only the masking rule is new. The image is cut into patches, the patch embeddings get a [CLS] token and position embeddings, and the transformer encoder produces output tokens §3.1 eq. 1. In distillation MIM, the student sees a masked version of each global view, in which the hidden patch embeddings are replaced by a learnable mask embedding §3.1 eq. 2. The teacher sees the unmasked view and provides targets §3.1. The MIM loss is a cross-entropy on the masked patch tokens only §3.1 eq. 3, and a global [CLS] loss compares the teacher's [CLS] output for one view with the student's for another view §3.1 eq. 4. A multi-crop scheme adds local crops that are seen only by the student, through a [CLS] loss against the teacher's global views app. B eq. A11.

The novelty is how the mask is chosen §3.2. Step by step:

  1. Attention per head. Each head produces an attention matrix over the n+1 tokens with a row-wise softmax of scaled query-key products §3.2 eq. 5.
  2. Average over heads. The matrices of the last encoder layer are averaged over the heads §3.2 eq. 6. The default uses the teacher's last layer (layer 12 of ViT-S/16) §4.1 tab. 9a.
  3. [CLS] attention vector. The row of the [CLS] token, without its own entry, gives one score per patch §3.2 eq. 7. It can be reshaped to the patch grid as an attention map §3.2; the paper visualises such maps in fig. A5.
  4. Mask ratio. A ratio r is drawn uniformly from [0.1, 0.5] by default §4.1. The top k = floor(r n) patches by score form the set M^H §3.2 eq. 8. This variant is called AttMask-High and is the default §3.2.
  5. Masking probability. Each view is masked with probability p = 0.5 §4.1 tab. 9b. An unmasked view lets the student see the full image as well.
  6. Student input. Masked patch embeddings are replaced by the learnable mask embedding §3.2 eq. 10. The teacher's global views are not masked.

Two variants are also described. AttMask-Low masks the least attended patches instead §3.2. AttMask-Hint masks the high-attention set but reveals a small random number s·n of its most attended patches, where s_max = 0.1 and s is drawn from [0.01, 0.05] §3.2 §4.1.

Pseudocode
# AttMask step on top of iBOT-style distillation  [§3.2] [§3.1]
# Input:  global_views (B, 2, 3, 224, 224), local_crops (B, 6, 3, 96, 96)
# Output: scalar loss
def training_step(global_views, local_crops):
    # Teacher sees the unmasked global views; keep its [CLS] output, patch
    # outputs and last-layer attention                         [§3.2] [§4.1]
    teacher_cls, teacher_patch, attention = teacher(global_views)
    # Mean over heads of the [CLS] row, without its own entry  [eq. 6] [eq. 7]
    cls_attention = attention[:, :, 0, 1:].mean(dim=1)   # (B*2, n)
    # One ratio r ~ U(0.1, 0.5), shared by the whole batch in the code
    #                              [§4.1] [code: main_attmask.py#L471]
    ratio = uniform(0.1, 0.5)
    # Per view, with probability p = 0.5, hide the top floor(r n) patches
    #                                        [§3.2] [eq. 8] [eq. 9] [§4.1]
    masks = attmask_high(cls_attention, ratio, masking_prob=0.5)  # (B*2, n)
    # Student sees the masked global views; hidden patch embeddings are
    # replaced by the learnable mask embedding                [eq. 2] [eq. 10]
    student_cls, student_patch = student(global_views, masks)
    # Local crops go through the student without masking
    #                                            [code: main_attmask.py#L486]
    local_student_cls = student(local_crops, masks=None)
    # Global [CLS] pairs [eq. 4] and local-crop [CLS] pairs [eq. A11];
    # the code averages all [CLS] pairs into a single term
    #                                            [code: main_attmask.py#L666]
    cls_loss = cls_cross_entropy(teacher_cls, student_cls, local_student_cls)
    # MIM loss on masked patch tokens only, teacher centred and sharpened
    # [eq. 3] [code: main_attmask.py#L656-L658]
    mim_loss = masked_patch_cross_entropy(teacher_patch, student_patch, masks)
    # Weighted sum; lambda = 1 by default                  [§4.1] [§4.4]
    return cls_loss + mim_loss
Pseudocode
# AttMask-Hint: reveal some of the most attended patches  [§3.2] [code]
# Input:  cls_attention (B*2, n), masks (B*2, n) from attmask_high
# Output: masks with a few top patches revealed
def reveal_hints(cls_attention, masks):
    # Fresh draw r' ~ U(0.1, 0.5); show ratio s = 0.1 * r'  [§4.1] [code]
    show = 0.1 * uniform(0.1, 0.5)
    # Candidate set: the top 10% most attended patches   [§3.2] [code]
    candidates = top_fraction(cls_attention, 0.1)
    # Reveal floor(s n) candidates at random: unmask them   [§3.2]
    hidden = random_subset(candidates, floor(show * n))
    masks[hidden] = False
    return masks

After the student's optimiser step, the teacher parameters follow the student by exponential moving average §3.1. In the code the momentum rises on a cosine schedule toward 1 main_attmask.py main_attmask.py, which the paper does not describe.

Key equations, reproduced from the paper:

\tilde{z}_i = (1 - m_i)\, z_i + m_i\, z^{\mathrm{mask}} eq. 2

L_{\mathrm{MIM}} = -\sum_{v \in V} \sum_{i=1}^{n} m_i^v\, f_{\theta'}(Z^v)_i \log f_\theta(\tilde{Z}^v)_i eq. 3

L_{G} = -\sum_{(u,v) \in V^2} \mathbb{1}_{u \neq v}\, f_{\theta'}(Z^u)^{\mathrm{cls}} \log f_\theta(\tilde{Z}^v)^{\mathrm{cls}} eq. 4

\bar{\mathbf{a}}^{\mathrm{cls}} = (\bar{a}_{1,2}, \bar{a}_{1,3}, \dots, \bar{a}_{1,n+1}) eq. 7

M^H = \{\sigma_\downarrow(i), \dots, \sigma_\downarrow(k)\}, \quad k = \lfloor r n \rfloor eq. 8

m_i^H = \mathbb{1}_{M^H}(i) eq. 9

\tilde{z}_i = (1 - m_i^H)\, z_i + m_i^H\, z^{\mathrm{mask}} eq. 10

L_{\mathrm{LC}} = -\sum_{v \in V} \sum_{j=1}^{m} f_{\theta'}(Z^v)^{\mathrm{cls}} \log f_\theta(Z^c_j)^{\mathrm{cls}} eq. A11

The paper contains no algorithm box; the pseudocode above is compiled from §3.2 and the code.

Hyperparameters

Values are from the paper's appendix and experiment sections unless marked "code". The official README gives the exact commands used for ImageNet pre-training, and they match the paper's values.

NameValueWhereNotes
ArchitectureViT-S/16§4.1
Attention layer for maskslast encoder layer (12)§4.1 tab. 9alayer 11 gives 49.8 vs 49.7 (k-NN), kept 12 for simplicity
Masking probability p0.5§4.1 tab. 9bper view; code draws one uniform per image attmask.py
Mask ratio rU[0.1, 0.5]§4.1code: --pred_ratio 0.3 --pred_ratio_var 0.2 README.md
Show max s_max (Hint)0.1§4.1code: --show_max 0.1 main_attmask.py
Show ratio s (Hint)U[0.01, 0.05]§4.1code: s = 0.1 * r' with fresh r' main_attmask.py
Global views2 at 224^2app. B
Local crops6 at 96^2 (20% runs); 10 at 96^2 (100% runs)app. B
Global crop scale (s, 1)s = 0.4 DINO, 0.25 iBOT; 0.32 for 300 epochsapp. Bcode: --global_crops_scale 0.25 1 README.md
Local crop scale(0.05, s)app. Bcode: --local_crops_scale 0.05 0.25
Head output dim65536 (DINO); 8192, shared [CLS] and patch head (iBOT)app. Bcode: --shared_head True --out_dim 8192 README.md
Student temperature0.1app. Bmain_attmask.py
Teacher temperaturelinear warm-up 0.04 to 0.07 over the first 30 epochsapp. Bcode: --teacher_temp 0.07 in README; patch temperature default 0.07 main_attmask.py
Teacher momentum0.99 (20% runs); 0.996 (100% runs)app. Bcosine schedule to 1 in code main_attmask.py
MIM loss weight λ1§4.1 tab. A15
OptimizerAdamWapp. Bbetas and epsilon not stated
Batch size240 (100 and 20% runs); 800 (300 epochs)app. Bcode: 60 per GPU × 4, 100 per GPU × 8 README.md README.md
Learning rate5e-4 × batch/256 (linear scaling), 10 warm-up epochs, cosineapp. Bcode: --lr 0.0005, min 1e-6 main_attmask.py
Weight decaycosine from 0.04 to 0.4app. Bmain_attmask.py
Epochs100 (20% and 100% runs); 300 (benchmark)app. B tab. 5 tab. A13
Training datafirst 20% of samples per class; full ImageNet-1k§4.1 app. Bcode: --subset 260 README.md
Clip gradientnot stated in papercodedefault 3.0 main_attmask.py
Freeze last layernot stated in papercodedefault 1 epoch main_attmask.py utils.py
Drop path (student)not stated in papercodedefault 0.1 main_attmask.py
Mixed precisionnot stated in papercodedefault on main_attmask.py
Centre momentumnot stated in papercode0.9 for both centres main_attmask.py
Mask embedding initnot stated in papercodezeros vision_transformer.py
k-NNk = 20, [CLS] of last layer§4.1 app. Bcode weights by exp(sim / 0.07) eval_knn.py eval_knn.py
Few-shot k-NNν ∈ {1, 5, 10, 20} examples per class§4.1
Linear probeSGD, batch 1024, 100 epochs, lr 0.003, no weight decayapp. BREADME uses lr 0.005 with linear scaling on 2 GPUs README.md eval_linear.py
Linear probe features[CLS] tokens of the last 4 layersapp. A.1k-NN uses the last layer only
CIFAR10, CIFAR100 fine-tune500 epochs; lr 7.5e-6; weight decay 0.05; batch 900app. Bcode: --epochs 500 --lr 7.5e-6 --batch-size 150 on 6 GPUs README.md
Oxford Flowers fine-tune1000 epochs; otherwise as aboveapp. Bcode: 1000 epochs for Flowers README.md
COCO detectionCascade Mask R-CNN; AdamW lr 1e-4; 12 epochs; weight decay 0.05; 2 images per GPUapp. BCOCO settings not in this repository
ADE20K segmentationUperNet; 160k iterations at 512^2; AdamW lr 7e-4; layer decay 0.65; weight decay 0.05app. Bnot in this repository
Retrieval, fine-grained, DAVISprotocols of DINO and the cited benchmarks; no fine-tuningapp. B

Results

Author-reported. Setup: ImageNet-1k pre-training with ViT-S/16 (k-NN and linear probing on the validation set unless noted). The baseline is the default iBOT block-wise masking; DINO is the self-distillation baseline. The paper does not state the hardware in the text read; the code README lists 4- and 8-GPU runs README.md README.md.

Table 1, masking strategies for iBOT on 20% of ImageNet, 100 epochs tab. 1:

Masking (iBOT, 20% ImageNet)Ratiok-NNLinearCIFAR10 FTCIFAR100 FTSource
Random block-wise (default)10-50%46.756.498.086.0tab. 1
Random patch75%47.355.597.785.5tab. 1
Random patch10-50%47.856.798.086.1tab. 1
AttMask-Low10-50%44.053.497.684.6tab. 1
AttMask-Hint10-50%49.557.598.186.6tab. 1
AttMask-High10-50%49.757.998.286.6tab. 1

Table 2, k-NN accuracy vs. fraction of ImageNet used for pre-training tab. 2:

Pre-training dataBlock-wise (iBOT)AttMask-HighSource
5%15.717.5tab. 2
10%31.933.8tab. 2
20%46.749.7tab. 2
100%71.572.5tab. 2

Efficiency: AttMask-High matches block-wise masking's accuracy with 42% fewer training epochs on 20% of ImageNet §4.2 fig. 4.

Table 3, DINO (self-distillation only) on 20% of ImageNet, k-NN tab. 3:

No masking (DINO)RandomAttMask-LowAttMask-HintAttMask-HighSource
43.043.442.743.643.5tab. 3

Table 4, background robustness, linear probe on ImageNet-9 (IN-9) and variants tab. 4. IN-9 full set: block-wise 89.1; AttMask-High 89.8; AttMask-Hint 89.6 tab. 4. AttMask-High is the most robust on all variants except OBB and OBT, where AttMask-Hint is higher §4.2.

Table 5, 100% ImageNet-1k, 100 epochs tab. 5:

Methodk-NNLinearν=1ν=5ν=10ν=20Source
DINO70.974.6----tab. 5
MST72.175.0----tab. 5
iBOT71.574.432.947.652.556.4tab. 5
iBOT + AttMask-High72.575.737.151.355.759.1tab. 5
iBOT + AttMask-Hint72.876.137.652.256.459.6tab. 5

Table 6, fine-tuning tab. 6. Classification accuracy on CIFAR10, CIFAR100 and Flowers; COCO detection and instance segmentation AP; ADE20K mIoU:

MethodCIFAR10CIFAR100FlowersCOCO AP^bCOCO AP^mADE20K mIoUSource
iBOT98.889.596.848.241.844.9tab. 6
iBOT + AttMask-High98.890.197.748.842.045.3tab. 6

Table 7, without fine-tuning, retrieval and video segmentation tab. 7:

MethodROxford MROxford HRParis MRParis HDAVIS (J&F)mDAVIS JmDAVIS FmSource
iBOT31.011.756.228.960.559.561.4tab. 7
iBOT + AttMask-High33.512.159.031.562.160.663.5tab. 7

Table 8, fine-grained classification, Recall@k without fine-tuning tab. 8:

MethodCUB200 R@1CARS196 R@1SOP R@1In-Shop R@1Source
iBOT51.435.657.439.1tab. 8
iBOT + AttMask-High57.239.859.040.7tab. 8

Table 9, ablations on 20% of ImageNet, k-NN accuracy tab. 9:

(a) Layer691112(b) p00.250.50.751(c) r (%)10-3010-5010-7030
k-NN48.148.149.849.7k-NN43.447.349.449.444.2k-NN49.549.748.549.1

Appendix tables continue the table counter app. A.1 app. A.2:

ResultBaselineAttMaskSource
k-NN with GAP over patch features, 100%, 100 epochs49.0 (iBOT)58.0 (iBOT + AttMask)tab. A10
Linear probe on masked validation set, 0% / 70% salient area masked74.4 / 17.075.7 / 20.5tab. A11
Linear probe, Places205 scene classification55.956.7tab. A12
300 epochs, 100%: k-NN / linear / ν=1 k-NN74.6 / 77.4 / 38.975.0 / 77.5 / 40.4tab. A13
Supervised fine-tune on ImageNet-1k, 300-epoch pre-training81.181.3tab. A14
MIM weight λ = 0.0 / 0.5 / 1.0 / 2.0 (k-NN, 20%)43.4 / 46.5 / 46.7 / 41.943.5 / 47.3 / 49.7 / 48.3tab. A15

Paper vs code

The code is the official repository (gkakogeorgiou/attmask, commit 9f62061). Its ImageNet commands in the README match the paper's pre-training values. The differences below are where the paper and the code disagree, or where the code has a setting the paper does not describe.

  • Argument defaults are not the paper's setup. Running the script without the README flags gives a different model: last-layer weight normalisation on main_attmask.py (paper: none app. B), a separate patch head main_attmask.py, output dimension 65536 main_attmask.py, local crops disabled main_attmask.py, and global crop scale (0.14, 1) main_attmask.py. The README sets --norm_last_layer False, --shared_head True, --out_dim 8192 and the local crops README.md.
  • MIM loss is averaged, not summed. The paper's Eq. 3 is a plain sum over masked tokens. The code divides each image's sum by its number of masked tokens (at least one), then averages over the views and batch main_attmask.py main_attmask.py. Images with no masked tokens therefore contribute zero to the MIM term.
  • [CLS] losses are pooled, not separate. The paper gives the global [CLS] loss (Eq. 4) and the local-crop loss (Eq. A11) as separate sums. The code adds all [CLS] pairs, global and local, and divides by the number of pairs main_attmask.py. The relative weight of local crops therefore differs from the written equations.
  • Teacher centring and sharpening. The paper's teacher output is a softmax with temperature; the code also subtracts a running centre (momentum 0.9) before the softmax, for both [CLS] and patch outputs main_attmask.py main_attmask.py main_attmask.py. The paper does not describe centring.
  • Teacher momentum is scheduled. The paper states the EMA with a fixed momentum (0.99 or 0.996). The code raises the momentum on a cosine schedule to 1 over training main_attmask.py.
  • Mask ratio is drawn once per step. The code samples one ratio r for the whole batch and both global views at each step main_attmask.py. The paper samples r from U[0.1, 0.5] without saying how often §4.1.
  • Hint show ratio uses its own draw. The code sets s = 0.1 × r' with a second, independent draw r' from the same range main_attmask.py. This matches U[0.01, 0.05] in distribution, but the paper does not describe the draw. The hint candidates are the top 10% most attended patches attmask.py. Since r ≥ 0.1, these lie inside M^H, consistent with the paper.
  • Linear probe learning rate. The paper gives lr 0.003 at batch 1024 for the linear probe app. B. The README runs lr 0.005 on 2 GPUs with 1024 per GPU, and the script applies linear scaling to the learning rate README.md eval_linear.py. The README's effective learning rate is therefore about 0.04 (0.005 × 2048 / 256). The paper does not say whether its 0.003 is scaled; unscaled, it is 0.003.
  • k-NN weighting temperature. The paper gives k = 20 only. The code weights neighbours with a temperature of 0.07 eval_knn.py eval_knn.py.
  • Fine-tuning matches. CIFAR runs use 500 epochs and Flowers 1000 app. B README.md.
  • DINO+AttMask (Table 3) cannot be reproduced from the README. The README has commands for iBOT only. I did not find the configuration of the DINO runs.
  • COCO and ADE20K settings are not in this repository. The repo contains no detection or segmentation code, so those numbers are unverified against code.

No discrepancies found in the pre-training ImageNet values (learning rate, weight decay, batch size, crops, temperatures and momentum) after reading the README commands.

Pitfalls

  • The defaults trap. Anyone who runs main_attmask.py without the README flags gets a different model (see Paper vs code). Use the README commands or pass every flag explicitly.
  • Loss normalisation changes the weights. The paper's sums and the code's averages differ for both the MIM term and the [CLS] terms. Reimplementing the equations as written will not reproduce the reported numbers. The code is the source of the reported runs.
  • Unstated but active settings. The code applies gradient clipping at 3.0, freezes the last layer for the first epoch, uses drop path 0.1, minimum learning rate 1e-6, mixed precision and a zero-initialised mask embedding. The paper does not mention any of these main_attmask.py main_attmask.py main_attmask.py main_attmask.py vision_transformer.py.
  • Last-layer freezing. Gradients of the last head layer are removed for the first epoch main_attmask.py utils.py.
  • Mixed precision scores. The teacher forward pass runs under autocast (mixed precision is on by default), so the top-k ranking uses reduced-precision attention values main_attmask.py. The order of tied scores is whatever the sort returns. The paper does not discuss either point.
  • Random ratio per step. The mask ratio is shared across the batch at each step, which changes the variance of the masking compared with per-image sampling. The paper does not say which is used.
  • Hint ratio is tied to the mask ratio's range. The show ratio is 0.1 times a second draw from the same range. Changing --pred_ratio or --show_max changes both.
  • Ablation batch size. The table caption for the masking-probability ablation says batch size 180, while the default and main runs use 240 tab. 9 app. B.
  • Probability tie. The text calls p = 0.5 the best choice §4.4, but Table 9b shows p = 0.75 with the same k-NN score (49.4).
  • Upper bound on the mask ratio matters. Raising the upper bound to 0.7 drops k-NN to 48.5. The authors suggest this may be because foreground objects are then completely masked §4.4 tab. 9.
  • MIM weight matters. λ = 2 is worse than λ = 1 (48.3 vs 49.7) tab. A15. Low-attention masking is worse than random masking, so the direction of the rule matters tab. 1.
  • The cost claim is narrow. The paper says AttMask adds no extra cost §1. It reuses the teacher attention already computed, but it still sorts tokens for each image at each step; the cost of that sort is not measured.
  • Single numbers. The tables report one value per setting and no variance. The text read does not state how many seeds were run.
  • Linear probe comparability. The linear probe uses the [CLS] tokens of the last four layers, while k-NN uses the last layer only app. A.1. Compare like with like.
  • Figures described, not reproduced. The method diagram is Figure 2 and the masking comparison is Figure 1 §1 §3. Both are described in words only.

Lineage

  • Builds on: iBOT (masked image modeling with self-distillation) and DINO (self-distillation with multi-crop) §3.1. Neither is linked here: the source bibliography gives no arXiv id, and no card exists yet.
  • Builds on: BEiT (block-wise random masking, the default that AttMask replaces) §2 §3.1.
  • Compared against: random patch masking at 10-50% and 75%, block-wise masking, MST (low-attention masking with a CNN decoder) §2 tab. 1 tab. 5.
  • Related: Attention Dropout masks the most discriminative part of an image for supervised training §2. AttMask applies a similar idea to masked image modeling with a teacher's [CLS] attention §2.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “What to Hide from Your Students: Attention-Guided Masked Image Modeling” by Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, Nikos Komodakis (arXiv:2203.12719), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/gkakogeorgiou/attmask at commit 9f62061fb3af (Apache-2.0). Any quoted code is shown under that license.