Paper · 2022-03-23 · cs.CV
What to Hide from Your Students: Attention-Guided Masked Image Modeling
AttMask replaces random token masking in distillation-based masked image modeling (iBOT, DINO) with masks from the teacher ViT's last-layer [CLS] attention: the most attended patches are hidden from the student. Author-reported: on full ImageNet-1k (ViT-S/16, 100 epochs), iBOT+AttMask raises k-NN from 71.5 to 72.5.
Masked image modelingSelf-supervised learningVision Transformer (ViT) backbonesTeacher-student learningself-supervisedmasked-image-modelingvision-transformerdistillation
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Some self-supervised vision models learn by hiding parts of a picture and trying to fill them in, much like a cloze test for images. Most methods hide random patches. This paper argues that random hiding is too easy for images, because a picture has many patches that repeat the same information, so the missing parts can often be guessed from their neighbours.
The authors propose hiding the parts the model itself pays most attention to instead. A "teacher" network looks at the full picture and shows which regions it focuses on for its summary of the image. The "student" network then sees the picture with those focus regions hidden, and must predict what the teacher thought about them. Because the teacher already computes this attention, the method adds no extra forward pass.
The idea is meant to make the training task harder in the useful way. The authors report that the learned features are better and learned faster, and that the gains are larger when data is limited and when the features are used without fine-tuning.
TL;DR
AttMask is a masking rule for distillation-based masked image modeling with vision transformers. A teacher encoder sees the whole image, and the attention of its [CLS] token over the patches decides which patches the student must reconstruct: the most attended ones by default. The authors plug it into iBOT and DINO and report better k-NN and linear accuracy, faster training, and more robust features than random block masking. All numbers are author-reported.
Problem
Masked image modeling hides a fraction of image tokens and trains a transformer to predict them. The usual choice of which tokens to hide is random, borrowed from text. The authors argue that image tokens are numerous and redundant, so random hiding often leaves enough visible context to recover the hidden content without learning much about objects §1. The question they ask is whether the hidden set can be chosen informatively §1.
The method is designed for distillation-based MIM (iBOT) and self-distillation (DINO), where a teacher with an exponential moving average of the student supplies targets §3.1. Hiding random block-wise tokens is the default of iBOT tab. 1 §4.2.
Method
The method builds on the ViT encoder and the distillation losses of iBOT and DINO. Only the masking rule is new. The image is cut into patches, the patch embeddings get a [CLS] token and position embeddings, and the transformer encoder produces output tokens §3.1 eq. 1. In distillation MIM, the student sees a masked version of each global view, in which the hidden patch embeddings are replaced by a learnable mask embedding §3.1 eq. 2. The teacher sees the unmasked view and provides targets §3.1. The MIM loss is a cross-entropy on the masked patch tokens only §3.1 eq. 3, and a global [CLS] loss compares the teacher's [CLS] output for one view with the student's for another view §3.1 eq. 4. A multi-crop scheme adds local crops that are seen only by the student, through a [CLS] loss against the teacher's global views app. B eq. A11.
The novelty is how the mask is chosen §3.2. Step by step:
- Attention per head. Each head produces an attention matrix over the n+1 tokens with a row-wise softmax of scaled query-key products §3.2 eq. 5.
- Average over heads. The matrices of the last encoder layer are averaged over the heads §3.2 eq. 6. The default uses the teacher's last layer (layer 12 of ViT-S/16) §4.1 tab. 9a.
- [CLS] attention vector. The row of the [CLS] token, without its own entry, gives one score per patch §3.2 eq. 7. It can be reshaped to the patch grid as an attention map §3.2; the paper visualises such maps in fig. A5.
- Mask ratio. A ratio r is drawn uniformly from [0.1, 0.5] by default §4.1. The top k = floor(r n) patches by score form the set M^H §3.2 eq. 8. This variant is called AttMask-High and is the default §3.2.
- Masking probability. Each view is masked with probability p = 0.5 §4.1 tab. 9b. An unmasked view lets the student see the full image as well.
- Student input. Masked patch embeddings are replaced by the learnable mask embedding §3.2 eq. 10. The teacher's global views are not masked.
Two variants are also described. AttMask-Low masks the least attended patches instead §3.2. AttMask-Hint masks the high-attention set but reveals a small random number s·n of its most attended patches, where s_max = 0.1 and s is drawn from [0.01, 0.05] §3.2 §4.1.
# AttMask step on top of iBOT-style distillation [§3.2] [§3.1]
# Input: global_views (B, 2, 3, 224, 224), local_crops (B, 6, 3, 96, 96)
# Output: scalar loss
def training_step(global_views, local_crops):
# Teacher sees the unmasked global views; keep its [CLS] output, patch
# outputs and last-layer attention [§3.2] [§4.1]
teacher_cls, teacher_patch, attention = teacher(global_views)
# Mean over heads of the [CLS] row, without its own entry [eq. 6] [eq. 7]
cls_attention = attention[:, :, 0, 1:].mean(dim=1) # (B*2, n)
# One ratio r ~ U(0.1, 0.5), shared by the whole batch in the code
# [§4.1] [code: main_attmask.py#L471]
ratio = uniform(0.1, 0.5)
# Per view, with probability p = 0.5, hide the top floor(r n) patches
# [§3.2] [eq. 8] [eq. 9] [§4.1]
masks = attmask_high(cls_attention, ratio, masking_prob=0.5) # (B*2, n)
# Student sees the masked global views; hidden patch embeddings are
# replaced by the learnable mask embedding [eq. 2] [eq. 10]
student_cls, student_patch = student(global_views, masks)
# Local crops go through the student without masking
# [code: main_attmask.py#L486]
local_student_cls = student(local_crops, masks=None)
# Global [CLS] pairs [eq. 4] and local-crop [CLS] pairs [eq. A11];
# the code averages all [CLS] pairs into a single term
# [code: main_attmask.py#L666]
cls_loss = cls_cross_entropy(teacher_cls, student_cls, local_student_cls)
# MIM loss on masked patch tokens only, teacher centred and sharpened
# [eq. 3] [code: main_attmask.py#L656-L658]
mim_loss = masked_patch_cross_entropy(teacher_patch, student_patch, masks)
# Weighted sum; lambda = 1 by default [§4.1] [§4.4]
return cls_loss + mim_loss# AttMask-Hint: reveal some of the most attended patches [§3.2] [code]
# Input: cls_attention (B*2, n), masks (B*2, n) from attmask_high
# Output: masks with a few top patches revealed
def reveal_hints(cls_attention, masks):
# Fresh draw r' ~ U(0.1, 0.5); show ratio s = 0.1 * r' [§4.1] [code]
show = 0.1 * uniform(0.1, 0.5)
# Candidate set: the top 10% most attended patches [§3.2] [code]
candidates = top_fraction(cls_attention, 0.1)
# Reveal floor(s n) candidates at random: unmask them [§3.2]
hidden = random_subset(candidates, floor(show * n))
masks[hidden] = False
return masksAfter the student's optimiser step, the teacher parameters follow the student by exponential moving average §3.1. In the code the momentum rises on a cosine schedule toward 1 main_attmask.py main_attmask.py, which the paper does not describe.
Key equations, reproduced from the paper:
\tilde{z}_i = (1 - m_i)\, z_i + m_i\, z^{\mathrm{mask}} eq. 2
L_{\mathrm{MIM}} = -\sum_{v \in V} \sum_{i=1}^{n} m_i^v\, f_{\theta'}(Z^v)_i \log f_\theta(\tilde{Z}^v)_i eq. 3
L_{G} = -\sum_{(u,v) \in V^2} \mathbb{1}_{u \neq v}\, f_{\theta'}(Z^u)^{\mathrm{cls}} \log f_\theta(\tilde{Z}^v)^{\mathrm{cls}} eq. 4
\bar{\mathbf{a}}^{\mathrm{cls}} = (\bar{a}_{1,2}, \bar{a}_{1,3}, \dots, \bar{a}_{1,n+1}) eq. 7
M^H = \{\sigma_\downarrow(i), \dots, \sigma_\downarrow(k)\}, \quad k = \lfloor r n \rfloor eq. 8
m_i^H = \mathbb{1}_{M^H}(i) eq. 9
\tilde{z}_i = (1 - m_i^H)\, z_i + m_i^H\, z^{\mathrm{mask}} eq. 10
L_{\mathrm{LC}} = -\sum_{v \in V} \sum_{j=1}^{m} f_{\theta'}(Z^v)^{\mathrm{cls}} \log f_\theta(Z^c_j)^{\mathrm{cls}} eq. A11
The paper contains no algorithm box; the pseudocode above is compiled from §3.2 and the code.
Hyperparameters
Values are from the paper's appendix and experiment sections unless marked "code". The official README gives the exact commands used for ImageNet pre-training, and they match the paper's values.
| Name | Value | Where | Notes |
|---|---|---|---|
| Architecture | ViT-S/16 | §4.1 | |
| Attention layer for masks | last encoder layer (12) | §4.1 tab. 9a | layer 11 gives 49.8 vs 49.7 (k-NN), kept 12 for simplicity |
| Masking probability p | 0.5 | §4.1 tab. 9b | per view; code draws one uniform per image attmask.py |
| Mask ratio r | U[0.1, 0.5] | §4.1 | code: --pred_ratio 0.3 --pred_ratio_var 0.2 README.md |
| Show max s_max (Hint) | 0.1 | §4.1 | code: --show_max 0.1 main_attmask.py |
| Show ratio s (Hint) | U[0.01, 0.05] | §4.1 | code: s = 0.1 * r' with fresh r' main_attmask.py |
| Global views | 2 at 224^2 | app. B | |
| Local crops | 6 at 96^2 (20% runs); 10 at 96^2 (100% runs) | app. B | |
| Global crop scale (s, 1) | s = 0.4 DINO, 0.25 iBOT; 0.32 for 300 epochs | app. B | code: --global_crops_scale 0.25 1 README.md |
| Local crop scale | (0.05, s) | app. B | code: --local_crops_scale 0.05 0.25 |
| Head output dim | 65536 (DINO); 8192, shared [CLS] and patch head (iBOT) | app. B | code: --shared_head True --out_dim 8192 README.md |
| Student temperature | 0.1 | app. B | main_attmask.py |
| Teacher temperature | linear warm-up 0.04 to 0.07 over the first 30 epochs | app. B | code: --teacher_temp 0.07 in README; patch temperature default 0.07 main_attmask.py |
| Teacher momentum | 0.99 (20% runs); 0.996 (100% runs) | app. B | cosine schedule to 1 in code main_attmask.py |
| MIM loss weight λ | 1 | §4.1 tab. A15 | |
| Optimizer | AdamW | app. B | betas and epsilon not stated |
| Batch size | 240 (100 and 20% runs); 800 (300 epochs) | app. B | code: 60 per GPU × 4, 100 per GPU × 8 README.md README.md |
| Learning rate | 5e-4 × batch/256 (linear scaling), 10 warm-up epochs, cosine | app. B | code: --lr 0.0005, min 1e-6 main_attmask.py |
| Weight decay | cosine from 0.04 to 0.4 | app. B | main_attmask.py |
| Epochs | 100 (20% and 100% runs); 300 (benchmark) | app. B tab. 5 tab. A13 | |
| Training data | first 20% of samples per class; full ImageNet-1k | §4.1 app. B | code: --subset 260 README.md |
| Clip gradient | not stated in paper | code | default 3.0 main_attmask.py |
| Freeze last layer | not stated in paper | code | default 1 epoch main_attmask.py utils.py |
| Drop path (student) | not stated in paper | code | default 0.1 main_attmask.py |
| Mixed precision | not stated in paper | code | default on main_attmask.py |
| Centre momentum | not stated in paper | code | 0.9 for both centres main_attmask.py |
| Mask embedding init | not stated in paper | code | zeros vision_transformer.py |
| k-NN | k = 20, [CLS] of last layer | §4.1 app. B | code weights by exp(sim / 0.07) eval_knn.py eval_knn.py |
| Few-shot k-NN | ν ∈ {1, 5, 10, 20} examples per class | §4.1 | |
| Linear probe | SGD, batch 1024, 100 epochs, lr 0.003, no weight decay | app. B | README uses lr 0.005 with linear scaling on 2 GPUs README.md eval_linear.py |
| Linear probe features | [CLS] tokens of the last 4 layers | app. A.1 | k-NN uses the last layer only |
| CIFAR10, CIFAR100 fine-tune | 500 epochs; lr 7.5e-6; weight decay 0.05; batch 900 | app. B | code: --epochs 500 --lr 7.5e-6 --batch-size 150 on 6 GPUs README.md |
| Oxford Flowers fine-tune | 1000 epochs; otherwise as above | app. B | code: 1000 epochs for Flowers README.md |
| COCO detection | Cascade Mask R-CNN; AdamW lr 1e-4; 12 epochs; weight decay 0.05; 2 images per GPU | app. B | COCO settings not in this repository |
| ADE20K segmentation | UperNet; 160k iterations at 512^2; AdamW lr 7e-4; layer decay 0.65; weight decay 0.05 | app. B | not in this repository |
| Retrieval, fine-grained, DAVIS | protocols of DINO and the cited benchmarks; no fine-tuning | app. B |
Results
Author-reported. Setup: ImageNet-1k pre-training with ViT-S/16 (k-NN and linear probing on the validation set unless noted). The baseline is the default iBOT block-wise masking; DINO is the self-distillation baseline. The paper does not state the hardware in the text read; the code README lists 4- and 8-GPU runs README.md README.md.
Table 1, masking strategies for iBOT on 20% of ImageNet, 100 epochs tab. 1:
| Masking (iBOT, 20% ImageNet) | Ratio | k-NN | Linear | CIFAR10 FT | CIFAR100 FT | Source |
|---|---|---|---|---|---|---|
| Random block-wise (default) | 10-50% | 46.7 | 56.4 | 98.0 | 86.0 | tab. 1 |
| Random patch | 75% | 47.3 | 55.5 | 97.7 | 85.5 | tab. 1 |
| Random patch | 10-50% | 47.8 | 56.7 | 98.0 | 86.1 | tab. 1 |
| AttMask-Low | 10-50% | 44.0 | 53.4 | 97.6 | 84.6 | tab. 1 |
| AttMask-Hint | 10-50% | 49.5 | 57.5 | 98.1 | 86.6 | tab. 1 |
| AttMask-High | 10-50% | 49.7 | 57.9 | 98.2 | 86.6 | tab. 1 |
Table 2, k-NN accuracy vs. fraction of ImageNet used for pre-training tab. 2:
| Pre-training data | Block-wise (iBOT) | AttMask-High | Source |
|---|---|---|---|
| 5% | 15.7 | 17.5 | tab. 2 |
| 10% | 31.9 | 33.8 | tab. 2 |
| 20% | 46.7 | 49.7 | tab. 2 |
| 100% | 71.5 | 72.5 | tab. 2 |
Efficiency: AttMask-High matches block-wise masking's accuracy with 42% fewer training epochs on 20% of ImageNet §4.2 fig. 4.
Table 3, DINO (self-distillation only) on 20% of ImageNet, k-NN tab. 3:
| No masking (DINO) | Random | AttMask-Low | AttMask-Hint | AttMask-High | Source |
|---|---|---|---|---|---|
| 43.0 | 43.4 | 42.7 | 43.6 | 43.5 | tab. 3 |
Table 4, background robustness, linear probe on ImageNet-9 (IN-9) and variants tab. 4. IN-9 full set: block-wise 89.1; AttMask-High 89.8; AttMask-Hint 89.6 tab. 4. AttMask-High is the most robust on all variants except OBB and OBT, where AttMask-Hint is higher §4.2.
Table 5, 100% ImageNet-1k, 100 epochs tab. 5:
| Method | k-NN | Linear | ν=1 | ν=5 | ν=10 | ν=20 | Source |
|---|---|---|---|---|---|---|---|
| DINO | 70.9 | 74.6 | - | - | - | - | tab. 5 |
| MST | 72.1 | 75.0 | - | - | - | - | tab. 5 |
| iBOT | 71.5 | 74.4 | 32.9 | 47.6 | 52.5 | 56.4 | tab. 5 |
| iBOT + AttMask-High | 72.5 | 75.7 | 37.1 | 51.3 | 55.7 | 59.1 | tab. 5 |
| iBOT + AttMask-Hint | 72.8 | 76.1 | 37.6 | 52.2 | 56.4 | 59.6 | tab. 5 |
Table 6, fine-tuning tab. 6. Classification accuracy on CIFAR10, CIFAR100 and Flowers; COCO detection and instance segmentation AP; ADE20K mIoU:
| Method | CIFAR10 | CIFAR100 | Flowers | COCO AP^b | COCO AP^m | ADE20K mIoU | Source |
|---|---|---|---|---|---|---|---|
| iBOT | 98.8 | 89.5 | 96.8 | 48.2 | 41.8 | 44.9 | tab. 6 |
| iBOT + AttMask-High | 98.8 | 90.1 | 97.7 | 48.8 | 42.0 | 45.3 | tab. 6 |
Table 7, without fine-tuning, retrieval and video segmentation tab. 7:
| Method | ROxford M | ROxford H | RParis M | RParis H | DAVIS (J&F)m | DAVIS Jm | DAVIS Fm | Source |
|---|---|---|---|---|---|---|---|---|
| iBOT | 31.0 | 11.7 | 56.2 | 28.9 | 60.5 | 59.5 | 61.4 | tab. 7 |
| iBOT + AttMask-High | 33.5 | 12.1 | 59.0 | 31.5 | 62.1 | 60.6 | 63.5 | tab. 7 |
Table 8, fine-grained classification, Recall@k without fine-tuning tab. 8:
| Method | CUB200 R@1 | CARS196 R@1 | SOP R@1 | In-Shop R@1 | Source |
|---|---|---|---|---|---|
| iBOT | 51.4 | 35.6 | 57.4 | 39.1 | tab. 8 |
| iBOT + AttMask-High | 57.2 | 39.8 | 59.0 | 40.7 | tab. 8 |
Table 9, ablations on 20% of ImageNet, k-NN accuracy tab. 9:
| (a) Layer | 6 | 9 | 11 | 12 | (b) p | 0 | 0.25 | 0.5 | 0.75 | 1 | (c) r (%) | 10-30 | 10-50 | 10-70 | 30 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| k-NN | 48.1 | 48.1 | 49.8 | 49.7 | k-NN | 43.4 | 47.3 | 49.4 | 49.4 | 44.2 | k-NN | 49.5 | 49.7 | 48.5 | 49.1 |
Appendix tables continue the table counter app. A.1 app. A.2:
| Result | Baseline | AttMask | Source |
|---|---|---|---|
| k-NN with GAP over patch features, 100%, 100 epochs | 49.0 (iBOT) | 58.0 (iBOT + AttMask) | tab. A10 |
| Linear probe on masked validation set, 0% / 70% salient area masked | 74.4 / 17.0 | 75.7 / 20.5 | tab. A11 |
| Linear probe, Places205 scene classification | 55.9 | 56.7 | tab. A12 |
| 300 epochs, 100%: k-NN / linear / ν=1 k-NN | 74.6 / 77.4 / 38.9 | 75.0 / 77.5 / 40.4 | tab. A13 |
| Supervised fine-tune on ImageNet-1k, 300-epoch pre-training | 81.1 | 81.3 | tab. A14 |
| MIM weight λ = 0.0 / 0.5 / 1.0 / 2.0 (k-NN, 20%) | 43.4 / 46.5 / 46.7 / 41.9 | 43.5 / 47.3 / 49.7 / 48.3 | tab. A15 |
Paper vs code
The code is the official repository (gkakogeorgiou/attmask, commit
9f62061). Its ImageNet commands in the README match the paper's pre-training
values. The differences below are where the paper and the code disagree, or
where the code has a setting the paper does not describe.
- Argument defaults are not the paper's setup. Running the script without
the README flags gives a different model: last-layer weight normalisation on
main_attmask.py (paper: none app. B), a separate patch head
main_attmask.py, output dimension 65536 main_attmask.py,
local crops disabled main_attmask.py, and global crop scale (0.14, 1)
main_attmask.py. The README sets
--norm_last_layer False,--shared_head True,--out_dim 8192and the local crops README.md. - MIM loss is averaged, not summed. The paper's Eq. 3 is a plain sum over masked tokens. The code divides each image's sum by its number of masked tokens (at least one), then averages over the views and batch main_attmask.py main_attmask.py. Images with no masked tokens therefore contribute zero to the MIM term.
- [CLS] losses are pooled, not separate. The paper gives the global [CLS] loss (Eq. 4) and the local-crop loss (Eq. A11) as separate sums. The code adds all [CLS] pairs, global and local, and divides by the number of pairs main_attmask.py. The relative weight of local crops therefore differs from the written equations.
- Teacher centring and sharpening. The paper's teacher output is a softmax with temperature; the code also subtracts a running centre (momentum 0.9) before the softmax, for both [CLS] and patch outputs main_attmask.py main_attmask.py main_attmask.py. The paper does not describe centring.
- Teacher momentum is scheduled. The paper states the EMA with a fixed momentum (0.99 or 0.996). The code raises the momentum on a cosine schedule to 1 over training main_attmask.py.
- Mask ratio is drawn once per step. The code samples one ratio r for the whole batch and both global views at each step main_attmask.py. The paper samples r from U[0.1, 0.5] without saying how often §4.1.
- Hint show ratio uses its own draw. The code sets s = 0.1 × r' with a second, independent draw r' from the same range main_attmask.py. This matches U[0.01, 0.05] in distribution, but the paper does not describe the draw. The hint candidates are the top 10% most attended patches attmask.py. Since r ≥ 0.1, these lie inside M^H, consistent with the paper.
- Linear probe learning rate. The paper gives lr 0.003 at batch 1024 for the linear probe app. B. The README runs lr 0.005 on 2 GPUs with 1024 per GPU, and the script applies linear scaling to the learning rate README.md eval_linear.py. The README's effective learning rate is therefore about 0.04 (0.005 × 2048 / 256). The paper does not say whether its 0.003 is scaled; unscaled, it is 0.003.
- k-NN weighting temperature. The paper gives k = 20 only. The code weights neighbours with a temperature of 0.07 eval_knn.py eval_knn.py.
- Fine-tuning matches. CIFAR runs use 500 epochs and Flowers 1000 app. B README.md.
- DINO+AttMask (Table 3) cannot be reproduced from the README. The README has commands for iBOT only. I did not find the configuration of the DINO runs.
- COCO and ADE20K settings are not in this repository. The repo contains no detection or segmentation code, so those numbers are unverified against code.
No discrepancies found in the pre-training ImageNet values (learning rate, weight decay, batch size, crops, temperatures and momentum) after reading the README commands.
Pitfalls
- The defaults trap. Anyone who runs
main_attmask.pywithout the README flags gets a different model (see Paper vs code). Use the README commands or pass every flag explicitly. - Loss normalisation changes the weights. The paper's sums and the code's averages differ for both the MIM term and the [CLS] terms. Reimplementing the equations as written will not reproduce the reported numbers. The code is the source of the reported runs.
- Unstated but active settings. The code applies gradient clipping at 3.0, freezes the last layer for the first epoch, uses drop path 0.1, minimum learning rate 1e-6, mixed precision and a zero-initialised mask embedding. The paper does not mention any of these main_attmask.py main_attmask.py main_attmask.py main_attmask.py vision_transformer.py.
- Last-layer freezing. Gradients of the last head layer are removed for the first epoch main_attmask.py utils.py.
- Mixed precision scores. The teacher forward pass runs under autocast (mixed precision is on by default), so the top-k ranking uses reduced-precision attention values main_attmask.py. The order of tied scores is whatever the sort returns. The paper does not discuss either point.
- Random ratio per step. The mask ratio is shared across the batch at each step, which changes the variance of the masking compared with per-image sampling. The paper does not say which is used.
- Hint ratio is tied to the mask ratio's range. The show ratio is 0.1 times
a second draw from the same range. Changing
--pred_ratioor--show_maxchanges both. - Ablation batch size. The table caption for the masking-probability ablation says batch size 180, while the default and main runs use 240 tab. 9 app. B.
- Probability tie. The text calls p = 0.5 the best choice §4.4, but Table 9b shows p = 0.75 with the same k-NN score (49.4).
- Upper bound on the mask ratio matters. Raising the upper bound to 0.7 drops k-NN to 48.5. The authors suggest this may be because foreground objects are then completely masked §4.4 tab. 9.
- MIM weight matters. λ = 2 is worse than λ = 1 (48.3 vs 49.7) tab. A15. Low-attention masking is worse than random masking, so the direction of the rule matters tab. 1.
- The cost claim is narrow. The paper says AttMask adds no extra cost §1. It reuses the teacher attention already computed, but it still sorts tokens for each image at each step; the cost of that sort is not measured.
- Single numbers. The tables report one value per setting and no variance. The text read does not state how many seeds were run.
- Linear probe comparability. The linear probe uses the [CLS] tokens of the last four layers, while k-NN uses the last layer only app. A.1. Compare like with like.
- Figures described, not reproduced. The method diagram is Figure 2 and the masking comparison is Figure 1 §1 §3. Both are described in words only.
Lineage
- Builds on: iBOT (masked image modeling with self-distillation) and DINO (self-distillation with multi-crop) §3.1. Neither is linked here: the source bibliography gives no arXiv id, and no card exists yet.
- Builds on: BEiT (block-wise random masking, the default that AttMask replaces) §2 §3.1.
- Compared against: random patch masking at 10-50% and 75%, block-wise masking, MST (low-attention masking with a CNN decoder) §2 tab. 1 tab. 5.
- Related: Attention Dropout masks the most discriminative part of an image for supervised training §2. AttMask applies a similar idea to masked image modeling with a teacher's [CLS] attention §2.
- Superseded by: not known from wiki content.
Source and license
Adapted from “What to Hide from Your Students: Attention-Guided Masked Image Modeling” by Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, Nikos Komodakis (arXiv:2203.12719), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/gkakogeorgiou/attmask at commit 9f62061fb3af (Apache-2.0). Any quoted code is shown under that license.