Vision Transformer (ViT) backbones

Overview

A Vision Transformer (ViT) splits an image into a grid of square patches, maps each patch to a vector, adds position information and runs a stack of self-attention and MLP blocks over the resulting sequence 2105.05633 §3.1. Every patch can therefore attend to every other patch from the first layer on, which is the main property the papers below build on. The cost is that attention grows with the square of the number of patches, so the patch size and the input resolution set the budget 2105.05633 tab. 3 2108.11084 §3.3.2. A survey groups the designs by how they change self-attention 2101.01169 §1, and notes that ViTs need large pretraining sets because they assume little about the image 2101.01169 §2.2 2101.01169 §4.2. Its reported numbers come from the cited papers, so this page uses the primary cards instead.

The cards below use the same backbone in very different roles: a per-pixel segmenter, a 3D medical network, a super-resolution network, a self-supervised pretraining target, a mixture-of-experts layer, the image tower of a vision-language model, the frozen encoder of a multimodal chat model, a frozen feature extractor for dense correspondence, an adapter target for video, a video quality scorer, a per-object encoder in a robot policy and a satellite foundation model, a depth decoder, a few-shot cost aggregator, a video adapter and a tracking adapter, a joint exemplar-search tracker, a clinical multimodal transformer and a benchmark for underwater classification. Some cards use it as a hierarchical, shifted-window backbone, as the denoiser of a diffusion model, as a teacher in self-distillation, or as the vision half of a chart or science question-answering system. Two kinds of change recur. Some papers change the pretraining objective; others change the token mixer, which is the part that replaces or reduces self-attention. One point also recurs across them: pre-training carries most of the accuracy. Several cards report that the same architecture trained from scratch loses most of its score.

Most numbers here are author-reported. The papers use different backbones, input sizes, schedules and test protocols, so a number from one row is not directly comparable with a number from another. Where a comparison is made below, the protocol is the same within the cited table.

Variants

Plain ViT encoder with a light decoder for dense prediction

SETR keeps the transformer at a fixed 1/16 resolution through every layer and only adds a decoder at the end. It tries three decoders: one that projects and upsamples in a single step, one that upsamples progressively, and one that fuses features from several encoder layers 2012.15840 §3.2 2012.15840 §3.3. With ViT-large weights pretrained on ImageNet-21k it reports 50.28% mIoU on ADE20K validation with multi-scale testing for its best decoder 2012.15840 tab. 4, and a random-initialised model reaches only 42.27% on Cityscapes against a much higher figure with pretraining 2012.15840 tab. 2. Card: SETR.

Segmenter drops the convolutional decoder too. Its linear variant maps each output patch token to class scores. Its mask variant adds one learnable class embedding per category, lets these attend to the patch tokens, and scores each patch against each class embedding to form one mask per class 2105.05633 §3.2. Under the same pretraining, the mask decoder improves the mIoU of the largest model slightly on ADE20K 2105.05633 tab. 4, and the best model reports 53.63% multi-scale mIoU on ADE20K 2105.05633 tab. 6. The paper also shows that training from scratch is far worse than starting from ImageNet-21k weights, at 4.42 to 12.51 mIoU against 45.37 2105.05633 tab. 9, and that patch size 8 gains about 1.5 mIoU over patch size 16 while running at roughly 7 instead of 92 images per second 2105.05633 tab. 3. Card: Segmenter.

The survey in remote sensing collects reported results for transformer methods on very-high-resolution optical, hyperspectral and radar images. It states that it re-runs none of them, so its tables only show what each source paper reported 2209.01206 §5 2209.01206 §6. Card: Transformers in Remote Sensing: A Survey.

Volumetric and medical segmentation

TransUNet is a hybrid. A ResNet-50 stem feeds a ViT-B/16 encoder, and a cascaded upsampler (CUP) decodes back to full resolution with skip connections from the CNN 2102.04306 §3.2. On Synapse multi-organ CT, the table has a plain ViT with the CUP decoder at 67.86% average DSC, the ResNet-50 hybrid with CUP at 71.29%, and the full TransUNet at 77.48% 2102.04306 tab. 1. It also reports 89.71% average DSC on ACDC cardiac MRI 2102.04306 tab. 5. The paper leaves the loss, the learning-rate schedule and the checkpoint choice unstated, so the card points to the released code for those values, and it flags one row of the Synapse table whose per-organ values do not average to its printed score. Card: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation.

nnFormer interleaves convolutional embedding and down-sampling with attention inside local 3D blocks, and uses global attention only in the small bottleneck. Its decoder replaces the usual skip connection with attention from the decoder to the matching encoder level 2109.03201 §3.2 2109.03201 §3.4. Against nnUNet, a strong convolutional baseline, it reports lower HD95 on all three datasets, while the Dice scores are close and on Synapse nnUNet is slightly higher 2109.03201 tab. 5. Card: nnFormer.

SAM-Med3D is a promptable segmentation model for volumes: a 3D ViT image encoder, a 3D point-prompt encoder and a 3D mask decoder 2310.15161 §4.1. It is trained on the authors' SA-Med3D-140K data 2310.15161 §3.1 in two stages, pre-training on the full 131K training masks and then fine-tuning on about 75K filtered masks 2310.15161 §4.2. Author-reported: 76.27% overall Dice with one 3D point, against 36.83% for SAM-Med2D with N points, and 80.71% with ten 3D points 2310.15161 tab. 3. The paper describes an overlapping sliding window for volumes larger than 128 voxels per side, but the released code has no sliding-window path for 3D inference 2310.15161 §4.3. Card: SAM-Med3D.

A review of Transformers in medical image analysis compiles reported results for segmentation. Its Synapse table lists ten models with parameters, FLOPs and Dice, and MERIT has the best row at 84.90% Dice and 13.22 HD95 2301.03505 tab. 6. A second table covers skin-lesion segmentation on three datasets 2301.03505 tab. 7. The authors do not say whether each row was copied from its source paper or re-run, so the rows should not be read as one protocol. The review sorts models into pure Transformer, hybrid CNN-Transformer and other placements, and it lists windowed and channel-wise attention among the ways to lower the quadratic cost 2301.03505 §2.2 2301.03505 §10. Card: Advances in Medical Image Analysis with Vision Transformers.

Efficient attention and token mixing

ESRT is a super-resolution model that runs a small CNN at half resolution and then one Transformer whose tokens are unfolded 3×3 neighbourhoods. Attention is computed inside four contiguous segments of the token sequence, which cuts the attention memory 2108.11084 §3.3. Its training GPU memory is reported at 4,191M against 16,057M for a plain Transformer in the same ablation 2108.11084 tab. 4. Card: Transformer for Single Image Super-Resolution.

EfficientFormerV2 moves a depthwise convolution into the feed-forward block and keeps self-attention only in the last two stages. Its search trades parameters against iPhone 12 latency. The smallest model reports 75.7% ImageNet top-1 at 3.5M parameters and 0.9 ms 2212.08059 tab. 2. The search code is not in the released repository, so the fixed networks are what can be reused 2212.08059 §4.2. Card: Rethinking Vision Transformers for MobileNet Size and Speed.

ResT is a four-stage pyramid that lays out a Transformer like a ResNet. Its attention shrinks keys and values with a strided depth-wise convolution, and a 1×1 convolution with instance normalisation mixes the heads 2105.13677 §2.2. Position comes from a depth-wise-conv gate, which the paper says accepts any input size 2105.13677 §2.4. ResT-Small reports 79.6% ImageNet-1k top-1 at 224 px 2105.13677 tab. 2, and 40.3 box AP for RetinaNet on COCO 2105.13677 tab. 3. In a simpler ResT-Lite ablation, the gate scores 72.88% against 71.54% with no positional encoding 2105.13677 tab. 8. Card: ResT.

XCiT replaces token-to-token attention with cross-covariance attention, which computes a d × d map between feature channels instead of an N × N map between tokens. Its cost is linear in the number of tokens. Channel-wise L2 normalisation of queries and keys is needed for stable training; without it the authors report failure for S12/16 2106.09681 §3.2 2106.09681 tab. 4. Each block also adds a depth-wise 3×3 convolution for local patch interaction, and removing it costs 1.2 top-1 points for S12/16 2106.09681 tab. 4. Author-reported: 82.0% ImageNet-1k top-1 for S12/16 at 224 px without distillation 2106.09681 tab. 1, and 47.0 box AP for S12/8 with Mask R-CNN on COCO [arxiv:2106.09681 tab. 5]. The 8 × 8 patch variants cost much more: S12/8 needs 55.6 GFLOPs at 384 px, against 4.8 for S12/16 at 224 px 2106.09681 tab. 1. Card: XCiT.

AFNO removes self-attention from the token mixer. It takes a 2D FFT of the token grid, mixes channels in each block, soft-thresholds the frequencies and inverts the FFT 2111.13587 §4. The paper lists its cost as N d²/k + N d log N FLOPs, against N²d + 3Nd² for self-attention 2111.13587 tab. 1. On Cityscapes at 1024×1024 with SegFormer-B3, AFNO-100% reports 80.9 mIoU against 79.7 for efficient self-attention 2111.13587 tab. 4. On inpainting it is within 0.01 dB PSNR of attention 2111.13587 tab. 2. Card: Adaptive Fourier Neural Operators.

Token Merging (ToMe) adds no parameters and sits inside each block of a plain ViT. In every layer it merges the r most similar token pairs, using cosine similarity on the attention keys and bipartite soft matching between two alternating token sets 2210.09461 §3 2210.09461 tab. 1. A merged token is an average weighted by the number of patches it holds, and attention adds a log-size term so that it counts as many keys 2210.09461 §3 2210.09461 eq. 1. Off the shelf, with r = 8 on ViT-L/16 MAE, the appendix sweep reports 83.92% top-1 at 183 images per second, against 85.66% at 93 without merging 2210.09461 app. A; the ablation table reports 84.25% at 182.9 images per second for the same setting 2210.09461 tab. 1. The authors also test video and audio 2210.09461 tab. 5 2210.09461 tab. 7. Card: Token Merging: Your ViT But Faster.

EfficientViT is built for measured speed rather than operation count. Each block is a sandwich with one self-attention layer between feed-forward layers, and depthwise convolutions sit before the feed-forward layers 2305.07027 §3.1 2305.07027 eq. 1. Its cascaded group attention splits the channels across heads, and each head also receives the output of the previous head, so the heads see different slices of the features 2305.07027 §3.1 2305.07027 eq. 2 2305.07027 eq. 3. The largest model, M5, reports 77.1% ImageNet-1K top-1 at 10,621 images per second on a V100, against 75.2% and 7,560 images per second for MobileNetV3-Large 2305.07027 tab. 2. Some speed ratios in the text do not reproduce from that table 2305.07027 §4.2 2305.07027 tab. 2. Card: EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention.

HiLo splits the heads of one attention layer by spatial frequency. Some heads run self-attention inside small non-overlapping windows, which carries fine detail. The rest attend from every token to window-averaged keys and values, which carries the coarse layout 2205.13213 §4.1 2205.13213 eq. 6. The LITv2 backbone uses this layer in its last two stages. Its first two stages are feed-forward blocks with a depthwise convolution, and that convolution's zero padding supplies the position information, so no position embedding is used 2205.13213 §4.2 2205.13213 §4.3. Author-reported: LITv2-S reaches 82.0% ImageNet-1K top-1 at 3.7 GFLOPs and 1,471 images per second on an RTX 3090. Plain multi-head self-attention in the same backbone reaches 82.3% at 1,293 images per second 2205.13213 tab. 4. The gain is in speed and FLOPs, not accuracy 2205.13213 tab. 4. Card: Fast Vision Transformers with HiLo Attention.

VideoMamba keeps the plain ViT layout but replaces self-attention with bidirectional Mamba state-space blocks, so cost grows linearly with clip length 2403.06977 §3.1 2403.06977 §3.2. The paper's preferred scan runs spatially first, then frame by frame 2403.06977 fig. 7. A smaller trained copy distils into the larger model, which the authors say avoids the overfitting that the pure Mamba model shows at scale 2403.06977 §3.3 2403.06977 §4.1. Author-reported: 84.0% ImageNet-1K top-1 at 576 px 2403.06977 tab. 2, 83.3% on Kinetics-400 at 64×384 px 2403.06977 tab. 3 and 68.4% on Something-Something V2 at 16×288 px 2403.06977 tab. 4. Card: VideoMamba: State Space Model for Efficient Video Understanding.

HaloNet replaces the convolutions of a ResNet-50 layout with local two-dimensional self-attention. The blocked form cuts the image into b × b query blocks and gathers one (b + 2h) × (b + 2h) memory per block, so each window is not centred on its pixel and no attention mask is needed 2103.12731 §3.2. A strided attention layer replaces the attention-then-pool downsampling of earlier designs 2103.12731 §3.2. Author-reported: H7 reaches 84.9% ImageNet top-1 when trained from scratch 2103.12731 §4.1. In Mask R-CNN on COCO, two halo layers in place of the last three convolutions of ResNet-50 add 0.7 box AP over the same baseline 2103.12731 tab. 5. Pure attention models train more slowly than matched EfficientNets on the authors' TPUs 2103.12731 §5. The card finds no official code release. Card: Scaling Local Self-Attention for Parameter Efficient Visual Backbones.

Hierarchical and shifted-window backbones

Swin Transformer builds a four-stage pyramid. Self-attention runs inside non-overlapping M×M windows, with M = 7 by default, and every second block shifts the window grid so that neighbouring windows exchange information 2103.14030 §3.2. Patch merging halves the resolution between stages 2103.14030 §3.1. Author-reported: 81.3% ImageNet-1K top-1 for Swin-T at 224 px 2103.14030 tab. 1, 58.7 box AP on COCO test-dev with a Swin-L HTC++ detector 2103.14030 tab. 2, and 53.5 mIoU on ADE20K validation with UperNet 2103.14030 tab. 3. Without shifting, the Swin-T ablation drops from 81.3 to 80.2 top-1 2103.14030 tab. 4. The released code has a window clamp in the last stage that the paper does not mention; see the Swin card.

Swin Transformer V2 scales the design to about 3 billion parameters and 1,536-pixel inputs 2111.09883 §3.5 2111.09883 §4.2. It makes three changes. Residual post-norm with scaled cosine attention keeps training stable 2111.09883 §3.2. A log-spaced continuous position bias, produced by a small network, lets a bias learned at one window size serve another 2111.09883 §3.3. SimMIM pretraining reduces the labelled data needed 2111.09883 §3.4. Author-reported for the 3B model: 84.0% ImageNet-V2 top-1 2111.09883 tab. 2, 63.1 box AP on COCO test-dev 2111.09883 tab. 3 and 59.9 mIoU on ADE20K validation 2111.09883 tab. 4. The 70M-image pretraining set is not public and the G-size configs are not in the repository 2111.09883 app. A2. Card: Swin Transformer V2.

PVT v2 keeps a four-stage pyramid without shifted windows. Its keys and values are average-pooled to a fixed 7×7 grid, so attention cost grows linearly with the number of image tokens 2106.13797 §3.2 2106.13797 eq. 2. It also uses overlapping patch embedding 2106.13797 §3.3 and a 3×3 depthwise convolution in the feed-forward block, which gives position information without a fixed position table 2106.13797 §3.4. Author-reported: 83.8% ImageNet-1K top-1 for B5 at 224 px 2106.13797 tab. 2 and 46.1 RetinaNet AP on COCO for B4 2106.13797 tab. 3. The authors present it as a stronger baseline set rather than a new idea. Card: PVT v2.

Pooling-based Vision Transformer (PiT) adds two pooling layers to a plain ViT. The token grid shrinks in three stages, 27, 14 and 7 pixels square for the small models, while the channel count grows with the head count 2103.16302 §3.2 2103.16302 tab. 1. Each pooling step applies a depth-wise convolution to the spatial tokens and a linear map to the class token 2103.16302 §3.2. Author-reported under the DeiT recipe: PiT-S reaches 80.9% ImageNet-1k top-1 at 2.9 B FLOPs, against 79.8% for ViT-S at 4.6 B [arxiv:2103.16302 tab. 2]. The text and the released code disagree on the patch size of the base model, and Table 1 matches the code 2103.16302 §3.4 2103.16302 tab. 1. Card: Rethinking Spatial Dimensions of Vision Transformers.

P2T keeps global attention but shortens the keys and values with a pyramid of average-pooled maps, not one pooled map. The four pooling ratios are 12, 16, 20 and 24 at the first attention stage and smaller at later stages 2106.12011 §III-C. Each pooled map gets a shared 3×3 depthwise convolution, and the feed-forward block is an inverted bottleneck with a depthwise convolution 2106.12011 §III-B.1 2106.12011 §III-B.2. Author-reported: P2T-Small reaches 82.4% ImageNet-1K top-1 at 3.7 GFLOPs, against 82.0% for PVTv2-B2 at 4.0 GFLOPs 2106.12011 tab. 2, and 46.7 mIoU on ADE20K with Semantic FPN 2106.12011 tab. 3. In a 100-epoch ablation, the four pooling scales lift top-1 from 70.6% for a single ratio of 24 to 74.7% 2106.12011 tab. 5. The text reports smaller detection margins than the table gives, so the table values are the ones to use 2106.12011 tab. 4. The card records the official code as non-commercial, so these numbers come from the paper alone. Card: P2T: Pyramid Pooling Transformer for Scene Understanding.

Relative position terms in attention

Instead of adding one position vector to each patch, iRPE adds a learned term to every attention logit, indexed by the offset between two patches 2107.14222 §3.2 2107.14222 eq. 13. Offsets are mapped to a fixed set of buckets. Small offsets keep their own bucket and larger ones share coarser, log-spaced buckets 2107.14222 eq. 18. The paper compares four ways to map a two-dimensional offset to a bucket. On DeiT-S with 50 buckets, the Product mapping in contextual mode, which uses one bucket per pair of row and column buckets, reaches 80.9% top-1 against 79.9% for the learned absolute baseline 2107.14222 tab. 1. Adding the term to the queries and values as well gives 81.4% 2107.14222 tab. 5. Sharing one table across heads costs little in contextual mode: 80.89% against 80.99% with separate tables 2107.14222 tab. 2. Computing the query term per bucket, not per pair, brings its cost to O(nkd) for n patches and k buckets 2107.14222 §3.2 2107.14222 eq. 28. For DETR, a 9×9 contextual table in the encoder, kept alongside the sinusoidal absolute encoding, raises AP from 39.5 to 40.8 at 150 epochs and from 40.6 to 42.3 at 300 epochs 2107.14222 tab. 6. Card: Rethinking and Improving Relative Position Encoding for Vision Transformer.

Small models and distillation

TinyViT targets small hierarchical transformers and makes large-scale pretraining cheap for them. A large teacher is run once; for each augmented view only the top-K probabilities and a 4-byte seed are stored, and the seed replays the augmentation during student training 2207.10666 §3.1. Distillation adds about 2 points for small ImageNet-21k-pretrained models, where plain pretraining added under 1 point 2207.10666 tab. 2. The label storage is large: 481 GB for ImageNet-21k with K = 100 2207.10666 §5.2. The 21M model reaches 84.8% top-1 at 224 px after distillation 2207.10666 tab. 4. Card: TinyViT.

Token labeling adds a dense loss on top of the class loss: every patch token is trained against its own label from a pretrained ImageNet-only NFNet-F6 classifier, and the dense label maps are computed once before training 2104.10858 §3.2 2104.10858 §4.1. MixToken, a CutMix-style mix of patch embeddings, is used with it 2104.10858 §3.3. Storing the full label maps would be about 2 TB, so the authors keep the five largest values per cell, which they put at about 10 GB 2104.10858 §4.1. Author-reported: 84.4% ImageNet top-1 for LV-ViT-S at 26M parameters, and 86.4% for LV-ViT-L at 512 px 2104.10858 tab. 4. Card: All Tokens Matter.

IAT is a transformer-style network for image enhancement with about 0.09M parameters 2205.14871 tab. 1. Ten learned queries, with no self-attention among them, read a low-resolution encoding of the image and predict a 3×3 colour matrix and a gamma 2205.14871 §3.2. On LOL-V2-real it reports 23.50 PSNR against 22.86 for MAXIM, but on LOL-V1 MAXIM is higher on PSNR (23.43 against 23.38) 2205.14871 tab. 1. Card: You Only Need 90K Parameters to Adapt Light.

Supervised pretraining on ImageNet-21k

ImageNet-21K-P is a cleaned ImageNet-21k. Classes with fewer than 500 images are removed, 50 images per class are held out for validation, and every image is resized to 224 px. The set falls from about 1.3 TB to 250 GB 2104.10972 §2.1. Its pretraining, semantic softmax, uses one softmax per WordNet depth. Each image trains every depth at which one of its ancestor labels exists, and the depth losses are reweighted 2104.10972 §3.3 2104.10972 eq. 3. Author-reported: this beats ImageNet-1k pretraining in all 30 model-dataset cells tested 2104.10972 tab. 4. For ViT-B-16 and Mixer-B-16 it also beats the official ImageNet-21k weights in all 14 cells 2104.10972 tab. 3. The paper's distillation variant adds a further gain on each dataset 2104.10972 tab. 2, but the released code has no training path for it, and the depth weights in the code are not those in the printed equation. See the card before reproducing those numbers. Card: ImageNet-21K Pretraining for the Masses.

Self-supervised pretraining

SimMIM hides random 32×32 pixel blocks, replaces the hidden patch embeddings with one learned mask token, and regresses the pixels of the hidden area with a single linear layer and an L1 loss 2111.09886 §3.2 2111.09886 §3.4. The authors find that computing the loss on the hidden pixels only works better than computing it on the whole image 2111.09886 tab. 4. ViT-B reaches 83.8% top-1 after fine-tuning on ImageNet-1K 2111.09886 tab. 6, but its linear-probe accuracy is 56.7%, and the authors say the features target fine-tuning rather than linear evaluation 2111.09886 §3.5. Card: SimMIM.

MAE also reconstructs masked pixels, but it drops the masked patches before the encoder, so the encoder never sees mask tokens, and a small decoder fills the gaps 2111.06377 §3. Putting mask tokens into the encoder gives 84.2% fine-tuning and 59.6% linear probing, against 84.9% and 73.5% without them, at 3.3× the FLOPs 2111.06377 tab. 1. Fine-tuning ViT-H at 448 px reaches 87.8% with ImageNet-1K data only 2111.06377 tab. 3. Linear probing stays weak: 75.8% for ViT-L 2111.06377 app. B. Card: Masked Autoencoders Are Scalable Vision Learners.

MSN does not reconstruct pixels. A randomly masked view is trained to match the prototype assignments of an unmasked view produced by an EMA teacher 2204.07141 §3. Random masking applies only to the global anchor, and small focal crops are added 2204.07141 §5.4. With one labelled image per class, ViT-B/16 reaches 49.8% 2204.07141 tab. 1, and with 1% of ImageNet labels ViT-B/4 reaches 75.7% 2204.07141 tab. 2. Card: Masked Siamese Networks for Label-Efficient Learning.

iBOT replaces the pretrained tokenizer of earlier masked-image methods with an online one. A teacher, updated as a moving average of the student, sees the unmasked views, and its per-patch distributions are the targets for the masked positions. A second loss matches the summary token across two views 2111.07832 §2.2 2111.07832 §3.1. Its ViT-L/16 pretrained on ImageNet-22K reaches 82.3% linear accuracy 2111.07832 tab. 1. On ADE20K with a UperNet head, ViT-B reaches 50.0 mIoU against 46.8 for DINO 2111.07832 tab. 6. For ViT-S/16 with multi-crop, the paper reports 193.4 h of pre-training against 180.0 h for DINO, and a higher peak memory 2111.07832 tab. 19. Card: iBOT.

Voltron is a masked autoencoder over video frames that also reads the paired caption 2302.12766 §1 2302.12766 §3. The caption either conditions pixel reconstruction or becomes a target for caption generation, and one probability decides which role each example gets. The variants fix that probability at 0 (VCond, VDual) or 0.5 (VGen) 2302.12766 §3.2. Card: Language-Driven Representation Learning for Robotics.

Generalized category discovery applies a self-supervised ViT-B/16 to a mixed set of labelled and unlabelled images. The representation is tuned with contrastive losses, and the final [CLS] features are clustered with a semi-supervised k-means whose number of clusters is picked on the labelled subset 2201.02609 §3.1 2201.02609 §3.2. Reported "All" accuracy is 91.5 on CIFAR-10 and 74.1 on ImageNet-100 2201.02609 tab. 2. Card: Generalized Category Discovery.

DINO trains a ViT without labels. A student matches the softmax output of a teacher that is an exponential moving average of the student, across several crops, and the teacher sees only the two global crops 2104.14294 §3.1. The paper uses no contrastive term and no predictor for the ViT case; centring and sharpening of the teacher output stop the outputs from collapsing 2104.14294 §3.1. Collapse depends on both operations 2104.14294 §5.3. Author-reported k-NN accuracy on ImageNet is 78.3% for ViT-S/8 and 76.1% for ViT-B/16, with linear accuracy of 78.2% for ViT-B/16 2104.14294 tab. 2. Card: Emerging Properties in Self-Supervised Vision Transformers.

data2vec applies one recipe to images, speech and text. An EMA teacher encodes the unmasked input, and the normalised outputs of its top K blocks, averaged, form the target at each position; a student that sees the masked input regresses those targets at the masked positions 2202.03555 §3.3 2202.03555 §3.4. Input encoders and masks differ by modality 2202.03555 §3.1 2202.03555 §3.2. For ImageNet-1K, the paper reports 84.2% top-1 for ViT-B and 86.6% for ViT-L, both with ImageNet-1K pretraining 2202.03555 tab. 1. Card: data2vec.

CAE splits masked-image pretraining into three parts. The encoder sees only the visible patches; a cross-attention regressor predicts the encoder representations of the hidden patches; a decoder predicts discrete tokens from those predictions 2202.03026 §3.1. The alignment target is the encoder's output on the hidden patches, with gradient stopped. The loss is the token cross-entropy plus a weighted mean-squared alignment term, with weight 2 2202.03026 §3.2 2202.03026 eq. 1. The default mask hides 98 of the 196 patches 2202.03026 §3.2. Author-reported for ViT-B at 1600 epochs: 83.9% fine-tuning, 70.4% linear probing and 77.1% attentive probing 2202.03026 tab. 1, against 67.8% and 74.2% for MAE in the same table. In the ablation, the alignment term alone lifts linear probing from 60.3% to 62.0% 2202.03026 tab. 5. The authors say training collapses without the reconstruction term 2202.03026 §5.5. The repository has no licence file, so the card takes the recipe from the paper. Card: Context Autoencoder for Self-Supervised Representation Learning.

SatMAE applies masked-autoencoder pretraining to satellite time series and multi-band images. It adds a year, month and hour encoding for each acquisition, masks each image of a sequence independently, and splits the bands into three spectral groups, each with its own patch embedding and a group encoding 2207.08051 §4.1 2207.08051 §4.2 2207.08051 eq. 3. Author-reported: 81.49% top-1 on fMoW temporal, with the authors noting that this figure uses nine random sequences and test augmentation 2207.08051 §5.3 2207.08051 app. A.3.2; 77.84% top-1 on fMoW RGB after fine-tuning 2207.08051 §5.2. Card: SatMAE.

VideoMAE V2 scales masked video pretraining to a ViT-g backbone of about one billion parameters 2303.16727 §4.1. Dual masking keeps about 10% of the cubes for the encoder and lets the decoder reconstruct a running-cell subset that keeps half of them 2303.16727 §3.2 2303.16727 tab. 10. Training runs in three stages: unlabelled clips, a labelled Kinetics mix, then the target set 2303.16727 §3.3 2303.16727 §4.2. Author-reported: 90.0% top-1 on Kinetics-400, 77.0% on Something-Something V2 and 68.7% on Something-Something V1 2303.16727 tab. 6. The authors state that their earlier ViT-g VideoMAE run took more than two weeks on 64 A100 GPUs 2303.16727 §1. Card: VideoMAE V2.

Contrastive pretraining of a ViT is harder to get right than the same recipe on a convolutional network. MoCo v3 trains a ViT with a momentum encoder and two random views per image 2104.02057 §3. The authors report that training can be unstable, with mild losses of 1 to 3% that do not show as divergence 2104.02057 §1. Keeping the random patch projection fixed, with no gradient, makes training steadier and more accurate in the settings tested 2104.02057 §4.2. Author-reported linear probing on ImageNet-1K: 76.7% for ViT-B/16 2104.02057 tab. 1, and 81.0% for a ViT-BN-L/7 variant that is not in the released code 2104.02057 §6.3. Card: An Empirical Study of Training Self-Supervised Vision Transformers. The card records the official code under a non-commercial licence, so take the recipe from the paper and its tables, not the repository.

Frozen self-supervised features as dense descriptors

Deep ViT features use the keys of a self-supervised DINO ViT as dense patch descriptors, with no training. Co-segmentation clusters the patches of all images together and keeps the clusters that a saliency map marks as foreground; correspondence keeps mutual nearest-neighbour pairs 2112.05814 §3 2112.05814 §4. Keys are the best of the four facets in the paper's ablation: 79.5 J_m on PASCAL-CO against 69.2 for tokens and 49.2 for values 2112.05814 tab. 3. Its co-segmentation result on PASCAL-CO is 79.5 against 75.8 for the best unsupervised baseline in the table 2112.05814 tab. 2. Card: Deep ViT Features as Dense Visual Descriptors.

STEGO keeps a DINO ViT-B frozen and trains a small head. The head's code correlations are pushed to follow the correlations of the frozen features, with a clamp at zero and a spatial centring step 2203.08414 §3.2. Codes are then clustered and refined with a CRF. On CocoStuff-27 it reports 28.2 unsupervised mIoU, against 14.4 for PiCIE+H 2203.08414 §4.2. Its ablation gives 24.5 for ViT-S against 28.2 for ViT-B 2203.08414 §4.3. Card: Unsupervised Semantic Segmentation by Distilling Feature Correspondences.

Splice uses a frozen DINO ViT-B/8 as a judge rather than as a feature bank. A small U-Net is trained from scratch on one structure and appearance image pair, with no adversarial loss 2201.00424 §3. The appearance loss matches the deepest class token, and the structure loss matches the self-similarity of the deepest keys 2201.00424 §3.2 2201.00424 §3.3. Author-reported: in two-alternative user studies, outputs are preferred over STROTSS on Wild-Pairs in 79.0% of cases 2201.00424 §4.1 2201.00424 tab. 1. Card: Splicing ViT Features for Semantic Appearance Transfer.

Per-image transformer head for depth

AdaBins predicts depth as a weighted sum of bin centres, and the bin widths are chosen for each image. A small transformer, the mini-ViT, cuts the decoded features into 16×16 patch tokens and outputs both the bin widths and the per-pixel range-attention maps 2011.14141 §3.3 2011.14141 tab. 1. Depth at each pixel is the sum of bin centres weighted by a softmax over the bins 2011.14141 §3.3 2011.14141 eq. 3. Author-reported: delta1 of 0.903 on NYU-Depth-v2 2011.14141 tab. 2 and RMS of 2.360 on KITTI 2011.14141 tab. 3. The released code uses a learned positional table of 500 rows and differs from the paper in several training values, so the card lists the differences and the sequence-length limit. Card: AdaBins: Depth Estimation using Adaptive Bins.

Windowed and cost-volume attention for depth and flow

NeW CRFs refines a monocular depth estimate with a fully connected CRF, but only inside non-overlapping 7×7-patch windows, and the windows are shifted every other step 2203.01502 §3.1 2203.01502 §3.2. The pairwise potential is multi-head attention 2203.01502 §3.3, on a Swin encoder 2203.01502 §3.4. Author-reported: KITTI Eigen Abs Rel 0.052 2203.01502 tab. 1 and NYUv2 Abs Rel 0.095 2203.01502 tab. 3. Card: NeW CRFs.

VAT, the Volumetric Aggregation Transformer, treats few-shot segmentation as cost aggregation over the correlation volume between query and support features 2207.10866 §3. A 4D convolutional Swin-style transformer refines the volume from coarse to fine, and a decoder that looks at the query appearance sharpens the mask 2207.10866 §3. Author-reported: 90.3 one-shot mIoU on FSS-1000 with ResNet-101, against 86.5 for HSNet 2207.10866 tab. 1 2207.10866 tab. 2 2207.10866 tab. 3. Card: Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation.

FlowFormer applies a transformer to the all-pairs cost volume of an optical-flow pair, after compressing it. Each source pixel's cost map becomes K = 8 latent tokens of width 128 2203.16194 §4.1. Three alternate-group layers mix the tokens within one pixel and across pixels 2203.16194 §3.2, and a recurrent decoder cross-attends from a 9×9 window around each pixel's current match to that memory 2203.16194 §3.3. Author-reported Sintel test AEPE is 1.16 for clean and 2.09 for final, against 1.39 and 2.47 for GMA with warm start 2203.16194 tab. 1. In an ablation under a RAFT-style schedule, three layers lower Sintel clean error from 1.53 to 1.10 2203.16194 tab. 2. The paper gives no loss formula, optimiser or number of recurrent steps, so implementations take those from the released code, and the code differs from the paper in several places 2203.16194 §4.1. Card: FlowFormer.

Metric depth and surface normals from one image

Metric3D v2 predicts metric depth and per-pixel surface normals from one RGB image, given the focal length 2404.15506 §3.1. Its encoders are ViTs pretrained with DINOv2 and registers, with DPT-style decoders 2404.15506 §4. Training images or depth labels are rescaled to one canonical focal length of 1000 px, and predictions are rescaled back at inference 2404.15506 §3.2. Depth and normals are refined jointly with recurrent updates, and a depth-normal consistency term lets normals learn from depth-only data 2404.15506 §3.3 2404.15506 §3.4. Author-reported zero-shot AbsRel on NYUv2 is 0.067 for ViT-g and 0.063 for ViT-L, and on KITTI 0.051 and 0.052 2404.15506 tab. 1. Card: Metric3D v2.

Sparse experts and analysis of the attention

Vision MoE replaces every other MLP of a ViT with a set of 32 experts and routes each patch to its top-2 experts 2106.05974 §2.2 2106.05974 §2.3. A 14.7B parameter model reaches 90.35% ImageNet top-1 after fine-tuning 2106.05974 tab. 2. Batch Prioritized Routing is applied at inference to a model trained with ordinary routing: it orders patches by their largest router weight, so the compute budget can be cut after training. The authors report that a V-MoE-H/14 model trained with ordinary routing can beat the dense ViT-H/14 while using about half its FLOPs 2106.05974 §4.2. Card: Scaling Vision with Sparse Mixture of Experts.

How Do Vision Transformers Work? argues that self-attention acts as a learned spatial smoothing that flattens the loss landscape, and that plain ViTs struggle on small data because of optimisation rather than overfitting 2202.06709 §2 2202.06709 §3. It builds a hybrid, AlterNet, with local attention blocks at the end of CNN stages. Most accuracy numbers are only in figures; the one given in text is a 2.7-point gain from global average pooling instead of the class token on CIFAR-100 2202.06709 §2. Card: How Do Vision Transformers Work?.

Transformer Interpretability Beyond Attention Visualization gives a class-specific relevance map for a ViT instead of reading the attention weights. Relevance is propagated back through the attention products, the skip connections and the linear layers, and the block matrices are multiplied together, as in attention rollout 2012.09838 §3.1 2012.09838 §3.2 2012.09838 §3.3. On ImageNet with ViT-base, the negative-perturbation AUC for the predicted class is 54.16 against 53.1 for rollout, and the positive-perturbation AUC is 17.03 against 20.05 2012.09838 tab. 1. The released code differs from the paper in the rollout range and in the matrix-product normalisation, which the card lists. Card: Transformer Interpretability Beyond Attention Visualization.

MambaVision is a hybrid backbone with two convolutional stages and two stages that mix a Mamba-style state-space block with self-attention. The authors put self-attention in the last half of the layers of each late stage. This is the best of the placements they test, at 82.3% on ImageNet-1K, against 81.5% when attention fills the first half of the layers instead 2407.08083 tab. 5. The Tiny model reports 6,298 images per second on an A100 2407.08083 tab. 1. The official code is non-commercial, so this page describes the design from the paper text only 2407.08083 §3.2.2. Card: MambaVision.

Vision-language and multimodal models

SigLIP trains a ViT image tower and a text tower with an independent sigmoid loss over every image-text pair instead of a batch softmax. The authors report that sigmoid is clearly better at small batch sizes and that the gain saturates near 32k pairs per batch 2303.15343 §4.1 2303.15343 tab. 8. A B/16 model trained from scratch reaches 73.4% zero-shot ImageNet accuracy after 5 days on 32 TPUv4 chips 2303.15343 tab. 1. Card: Sigmoid Loss for Language Image Pre-Training.

LLaVA-1.5 keeps a frozen CLIP ViT-L/14 at 336 px, connects it to a language model with a two-layer MLP, and trains the connector and then the language model on a public instruction mix 2310.03744 §3.3. The encoder has 24 by 24 patch tokens at 336 px (576 tokens), a count derived from the encoder name rather than stated in the paper. The ablation attributes much of the short-answer gain to adding academic VQA data together with explicit answer-format prompts 2310.03744 tab. 2. Card: Improved Baselines with Visual Instruction Tuning.

Video-ChatGPT feeds each frame through a frozen CLIP ViT-L/14, pools the frame features into temporal and spatial tokens and trains a single linear projection into a Vicuna language model 2306.05424 §3.1 2306.05424 §3.2. Its benchmark uses a GPT-3.5 judge, which is a closed model, and the reported correctness score is 2.40 on a 1–5 scale 2306.05424 tab. 1. Card: Video-ChatGPT.

Florence-2 uses a DaViT image encoder and a BART-style encoder-decoder. It takes an image and a text task prompt and writes the answer as text, with boxes and polygons written as quantised location tokens 2311.06242 §3. The large model reports 135.6 CIDEr zero-shot on COCO captions 2311.06242 tab. 4. Card: Florence-2.

PaLI casts every task as image and text in, text out, with no task heads; the prompt selects the task 2209.06794 §3.1. A ViT reads the image, and the text side starts from mT5 2209.06794 §3.1. The largest model pairs a 4B-parameter ViT (ViT-e) with mT5-XXL 2209.06794 §4. Author-reported: 149.1 CIDEr on COCO Karpathy test 2209.06794 tab. 1 and 53.6 CIDEr on the 35-language Crossmodal-3600 average 2209.06794 tab. 2. The pretraining data (WebLI) is not released, and the card flags an unclear place where the visual tokens enter the encoder 2209.06794 §3.1. Card: PaLI.

Multimodal-CoT splits reasoning into two fine-tuned T5 models. The first writes a rationale from the text and the image; the second answers from the text and that rationale 2302.00923 §4.1. The image enters as frozen ViT-L/32 patch features, 145 tokens of width 1024, projected to the model width and attended to by the text tokens 2302.00923 §4.2 2302.00923 tab. 12. Author-reported ScienceQA test accuracy is 85.31% for the Base model and 90.45% for the Large one 2302.00923 tab. 4. Without the picture features, the two-stage text-only model reaches 78.57% 2302.00923 tab. 3. The card lists conflicts between the paper and its released scripts on learning rate, epochs and input length. Card: Multimodal Chain-of-Thought Reasoning in Language Models.

ChartQA is a benchmark of questions about real charts, with human-written and machine-generated questions 2203.10244 §1 2203.10244 tab. 1. Its strongest baseline, VisionTaPas, encodes the chart image with a ViT-Base encoder next to a TaPas table model 2203.10244 §4.2. With the gold table it reaches 61.84% on the test split, and with a table extracted from the chart image it reaches 45.52% 2203.10244 tab. 5. Card: ChartQA.

SLIP adds a SimCLR-style image-only loss, with weight 1.0, to CLIP's image-text contrastive loss, and both losses share one ViT image encoder 2112.12750 §3.3 2112.12750 alg. 1. On YFCC15M for 25 epochs, ViT-B/16 reaches 42.8% zero-shot and 72.1% linear ImageNet accuracy, against 37.6% and 66.5% for the authors' CLIP run 2112.12750 tab. 2. A control that fine-tunes CLIP from SimCLR weights reaches 41.1% and 68.2%, below SLIP 2112.12750 tab. 7. The image-only branch adds about three times the image activations per step 2112.12750 §3.3. The CLIP baseline is the authors' own re-implementation with an adjusted augmentation recipe 2112.12750 §4. Card: SLIP: Self-supervision meets Language-Image Pre-training.

METER is a fully transformer vision-language model trained end to end, with no region detector. A ViT or Swin image encoder and a RoBERTa text encoder are joined by six co-attention layers per stream 2111.02387 §3.1 2111.02387 §3.3. Pre-training uses masked language modeling and image-text matching on about 4M images 2111.02387 §3.3 2111.02387 app. A. Adding a masked-patch objective with in-batch negatives lowers the ablation's VQAv2 test-dev score from 74.98 to 74.01 2111.02387 tab. 7. Author-reported VQAv2 test-std accuracy is 77.64 for the CLIP-ViT version with about 4M images, against 76.60 for VinVL 2111.02387 tab. 8, and 80.54 for a 14M-image version with a larger backbone 2111.02387 tab. 10. The card notes that the released configs cover the base and large Swin and CLIP models but not the 14M run. Card: An Empirical Study of Training End-to-End Vision-and-Language Transformers.

SAN keeps CLIP's ViT frozen and attaches a light side ViT: eight blocks, width 240, and 100 learnable query tokens 2302.12242 §3 2302.12242 §4.2. The side network yields mask proposals as inner products of pixel and query embeddings, and one attention bias per CLIP head 2302.12242 eq. 1 2302.12242 eq. 2. The biases steer CLIP's last blocks, which classify each proposal 2302.12242 eq. 3. Trained on COCO-Stuff, SAN reports 27.5 mIoU on ADE-150 with ViT-B/16 and 32.1 with ViT-L/14 2302.12242 tab. 2. It has 8.4M trainable parameters and runs at 15.2 FPS at 640² on one GPU 2302.12242 tab. 3. In the authors' ablation, end-to-end training beats two-stage training by 4.5 mIoU on ADE-150 2302.12242 tab. 6. The paper's ensemble rows use a model that is not in the public code, and the released code's inference step differs from the paper's formula 2302.12242 tab. 2 2302.12242 eq. 4. Card: Side Adapter Network for Open-Vocabulary Semantic Segmentation.

IRENE turns a chest radiograph, the words of a chief complaint, lab values, age and sex into one token sequence. The image tokens are 16×16 patches with a class token, and the text tokens are one per word, lab value or demographic field 2306.00864 p. 10. Two bidirectional blocks let image and text tokens attend within and across modalities, and ten self-attention blocks then run over the joined sequence 2306.00864 p. 10 2306.00864 fig. 2a. On eight pulmonary diseases the authors report a mean AUROC of 0.924, against 0.858 for Perceiver and 0.805 for an image-only ViT 2306.00864 tab. 1. On three COVID-19 outcomes the mean AUPRC is 0.592, against 0.521 for early fusion 2306.00864 tab. 2. Removing the radiograph drops the mean AUROC to 0.543 2306.00864 tab. 3. The method assumes every modality is present, and the authors list missing modalities as an open problem 2306.00864 p. 7. The paper points to a code release with the model and evaluation only. It has no training loop and no pretrained weights, so the reported numbers cannot be reproduced from it 2306.00864 p. 12. Card: A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics.

Parameter-efficient adaptation

FacT stacks the 12 weight matrices of every block of a ViT into one tensor and describes the change to that tensor with a shared low-rank factorisation, either Tensor-Train or Tucker 2212.03145 §3.1 2212.03145 §3.2. Only these factors and the head are trained, and the factors are folded back before inference 2212.03145 §1. On VTAB-1K, FacT-TK at rank up to 32 averages 75.6% with 0.069M backbone parameters, against 75.5% for NOAH with 0.361M 2212.03145 tab. 1. The reported numbers are on the test split 2212.03145 §4.1. Card: FacT.

BAT adapts a frozen dual-stream ViT tracker for RGB-thermal tracking. One adapter, shared by both streams, passes a feature prompt from each stream into the other in every layer, so neither modality is fixed as dominant 2312.10611 §1. Only the adapters are trained, 0.32M parameters in total 2312.10611 §1. Author-reported state-of-the-art results on RGBT234 and LasHeR, against 19 baselines 2312.10611 §4.2. Card: Bi-directional Adapter for Multi-modal Tracking.

Single-object tracking

SimTrack removes the Siamese branches and the transformer interaction head of a STARK-style tracker. The exemplar tokens and the search tokens go through one ViT backbone together, so attention in every layer mixes the two images 2203.05328 §3.2 2203.05328 eq. 5. The exemplar tokens get their own learned position embedding, and a 64×64 central crop of the exemplar is added as extra input tokens 2203.05328 §3.2 2203.05328 §3.3. Those foveal tokens are used only as input. Author-reported with the ViT-B/16 backbone: 69.3% AUC on LaSOT and 54.8% on TNL2K 2203.05328 tab. 1; the ViT-L/14 backbone reaches 70.5% and 55.6% 2203.05328 tab. 1. On TNL2K the joint framework adds 1.3 AUC points over the STARK-SV baseline with the same backbone, the separate position embedding adds 0.5 and the foveal window adds 0.8 2203.05328 tab. 8. Pretraining weights matter: on LaSOT, MAE weights reach 70.3% AUC against 69.3% for the CLIP weights used in the main models 2203.05328 tab. 5. Joint attention also constrains inference. The exemplar features depend on each search frame, so they cannot be cached across frames (derived from the joint attention) 2203.05328 §3.2. Restricting interaction to half the blocks lowers TNL2K AUC from 54.8 to 52.3 2203.05328 tab. 10. The released code differs from the paper in several places, including the exemplar position input and the schedule in its shipped configs, so check the card before reimplementing the training setup. Card: Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking.

Video, generation and prompting

VideoMAE pretrains a plain ViT-B on unlabelled video with masked autoencoding. It samples 16 frames, cuts them into 2×16×16 cubes and hides 90% of the cubes with a tube mask shared over time, so a hidden cube cannot be copied from a neighbouring frame 2203.12602 §3.3 2203.12602 §4.2. Only the visible cubes go through the encoder, and a 4-block decoder reconstructs the hidden ones with an MSE loss 2203.12602 tab. 1. Without extra data, author-reported ViT-B results are 81.5% top-1 on Kinetics-400 2203.12602 tab. 7 and 70.8% on Something-Something V2 2203.12602 tab. 6. Pretraining on the unlabelled training split of a small dataset also works, with 62.6% on HMDB51 and 91.3% on UCF101 2203.12602 tab. 2. Card: VideoMAE. The card records the official code under a non-commercial licence.

ST-Adapter adds a small bottleneck module to each block of a frozen image ViT. Its depth-wise 3D convolution mixes features across neighbouring frames, so only the adapters and a linear head are trained 2206.13559 §3.2 2206.13559 §4.1. With CLIP ViT-B/16 on Kinetics-400 it reports 82.0% top-1 with 7.20M trained parameters, against 81.7% for full fine-tuning with 121.57M 2206.13559 tab. 1. The paper's equation includes a non-linearity that the released code omits 2206.13559 §3.2. Card: ST-Adapter.

X-CLIP keeps a CLIP image encoder and adds cross-frame communication. Each frame emits a message token, and the message tokens of all frames are mixed inside every block 2208.02816 §3.2.1. A small temporal transformer then merges the frame vectors, and the class names are rewritten as video-specific text prompts 2208.02816 §3.2 2208.02816 §3.3.1. Author-reported: 87.1% top-1 on Kinetics-400 with ViT-L/14 at 8 frames and 4×3 views, at 658 GFLOPs per view 2208.02816 tab. 1; 83.8% with ViT-B/16 at 8 frames 2208.02816 tab. 1. Zero-shot, ViT-B/16 reaches 44.6% on HMDB-51 and 72.0% on UCF-101 2208.02816 tab. 3. Card: Expanding Language-Image Pretrained Models for General Video Recognition.

InternVideo trains a masked-video ViT-H and a CLIP-initialised UniFormerV2 video-text encoder separately, then fuses the two frozen encoders with learned cross-model attention 2212.03191 §3.1 2212.03191 §3.3. Author-reported: 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2 2212.03191 tab. 2 2212.03191 tab. 3. The paper gives the fusion learning rate as 5×10^5, with no exponent that a training run could use 2212.03191 §4.2. The card records that the released repository does not contain the fusion stage, so the fused numbers cannot be reproduced from it. Card: InternVideo: General Video Foundation Models via Generative and Discriminative Learning.

UniFormerV2 starts from an image ViT that has already been trained, typically CLIP, and adds two video modules beside each block: a local relation block over short stretches of time, and a global block in which one learned query gathers a clip-level summary by cross-attention 2211.09552 §3.1 2211.09552 §3.2 2211.09552 §3.3. Author-reported: 90.0% top-1 on Kinetics-400 with ViT-L/14 at 336 px, 64 frames, after Kinetics-710 post-pretraining 2211.09552 tab. 1. The 8-frame setting reaches 88.8 at 8.0 TFLOPs, against 75.3 TFLOPs for the 90.0 result 2211.09552 tab. 1. Card: UniFormerV2.

Video Swin Transformer applies Swin's shifted windows in 3D: each window spans a few frames as well as a patch grid, and the backbone starts from ImageNet-pretrained Swin weights 2106.13230 §3.1 2106.13230 §3.4. Author-reported: 84.9% top-1 on Kinetics-400 for Swin-L at 384 px, with ImageNet-21K pretraining and 10×5 views 2106.13230 tab. 1. The same model reaches 84.6% with 4×3 views, so the 10×5 figure depends on the number of views 2106.13230 tab. 1. Card: Video Swin Transformer.

DiT replaces the U-Net of latent diffusion with a transformer on latent patches. The timestep and class label drive adaLN-Zero, which regresses a scale, a shift and a gate for each block and starts each block as the identity 2212.09748 §3.2 2212.09748 app. A. With classifier-free guidance at scale 1.50, DiT-XL/2 reports FID 2.27 at 256×256 2212.09748 tab. 2 and 3.04 at 512×512 2212.09748 tab. 3, on class-conditional ImageNet and author-reported. Across twelve configurations, FID falls as forward-pass Gflops rise 2212.09748 §5 2212.09748 tab. 4. Card: Scalable Diffusion Models with Transformers.

W.A.L.T is a diffusion transformer that works on a causal-autoencoder latent shared by images and videos 2312.06662 §4.1. Its blocks alternate spatial windows, one latent frame each, with spatiotemporal windows across frames 2312.06662 §4.2. Without classifier-free guidance it reports ImageNet 256 FID 2.56 at 460M parameters, while VDM++ reports 2.40 at 2B parameters in the same table 2312.06662 tab. 2. The paper gives up to 2× speed-up for local window attention, measured in steps per second on UCF-101 2312.06662 §5.2. Card: Photorealistic Video Generation with Diffusion Models.

MAE-VQGAN uses a ViT-L masked autoencoder whose decoder predicts VQGAN codebook tokens 2209.00647 §3.1. A prompt tiles an example input-output pair and a query into one grid and inpaints the empty cell, with no weight updates 2209.00647 §3.2. Trained on arXiv figures, it reports Pascal-5i mIoU of 27.83, 30.44, 26.15 and 24.25 on the four splits, while the ImageNet-only variant reaches 2.22 to 7.07 2209.00647 tab. 1. Card: Visual Prompting via Image Inpainting.

FAST-VQA scores video quality with a Swin-based transformer whose input size does not grow with frame resolution. Each frame is cut into a grid, and one raw-resolution 32 × 32 patch is taken from every cell at the same position in all frames. The patches are spliced into one canvas: 7 × 7 patches, 224 pixels a side, for 32 frames 2207.02595 §3.1 2207.02595 tab. 2. The Video Swin-T backbone is changed in two places. Gated relative position biases separate pairs inside one patch from pairs across patches, and a non-linear regression head scores each patch before pooling 2207.02595 §3.2 2207.02595 eq. 5 2207.02595 eq. 7. The model is trained end to end with a PLCC-based loss 2207.02595 §4.1. The authors argue that frozen recognition features are not tuned for quality, and they compare against a full-resolution Swin-T with fixed features 2207.02595 §1 2207.02595 tab. 3. Author-reported: 0.814 PLCC on LSVQ 1080p, against 0.753 for that baseline 2207.02595 tab. 3. FLOPs are 279G at every resolution, against 11,852G for the baseline at 1080P 2207.02595 tab. 4. A single fragment covers about 2.4% of a 1080P frame, and one sample is almost as accurate as a six-sample ensemble 2207.02595 §4.7 2207.02595 tab. 8. Card: FAST-VQA.

Robotics

VIMA turns each detected object into one token: its box goes through an MLP and its crop through a small ViT trained from scratch 2210.03094 app. C.2.1. A pretrained T5 encoder reads the interleaved text and object prompt, and a decoder cross-attends from trajectory tokens to the prompt tokens 2210.03094 §4. At 200M parameters it reports 81.5% success on the easiest zero-shot level, against 58.0% for its Gato-style baseline 2210.03094 tab. 10. In the data-scaling table, with 1% of the imitation data, VIMA reports 36.3% on L1 against 11.5% for VIMA-Gato 2210.03094 tab. 11. Card: VIMA: General Robot Manipulation with Multimodal Prompts.

MEM gives a vision-language-action policy two memories 2603.03596 §3.1. The short-horizon memory is a video encoder: a ViT whose every fourth layer also attends across time, with no new learnable parameters 2603.03596 §3.3. The long-horizon memory is a text summary that the high-level policy updates from its previous summary 2603.03596 §3.2. Integrated into pi0.6, the authors report tasks of up to 15 minutes 2603.03596 §4.1. All results are author-reported. Card: MEM.

A ViT-B/16 imitation policy can also be steered by a slower model. In the driving paper, a pretrained ViT-B/16 reads a cropped camera image and outputs steering and throttle for each of three route classes, LEFT, MIDDLE and RIGHT 2411.14256 §2.1 2411.14256 app. B.1. A multimodal LLM, used without fine-tuning and prompted with a chain-of-thought template, names one of those classes, and the policy keeps the last answer until a new one arrives 2411.14256 §2.2 2411.14256 tab. 1. The LLM answers took 5.86 to 7.76 seconds on the authors' hardware 2411.14256 tab. 2. Trained on one cone in a hallway, the policy alone reaches 40% success on a harder course with cones and bins, and 63% to 83% with an LLM, depending on the model, over 30 real runs each 2411.14256 tab. 3. These are author-reported figures from one course and one car. The appendix says that in one configuration the LLM supplies the steering and throttle values, which conflicts with the main text; the card follows the main text 2411.14256 app. B.4. Card: Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs.

Frozen foundation encoders in specialised domains

The canopy-height work pretrains a huge ViT (606M parameters) with DINOv2 on 18 million unlabelled satellite tiles, freezes it, and trains a DPT decoder on airborne lidar height maps 2304.07213 §3.1 2304.07213 §3.2. It reports an average mean absolute error of 2.8 m over three lidar test sets 2304.07213 tab. 2. The pretraining cost is estimated at 16,896 GPU-hours 2304.07213 app. D. Card: Canopy height from RGB imagery.

Multimodal Fusion Transformer for hyperspectral classification turns a second sensor (LiDAR, multispectral, radar or elevation) into one classification token. In each encoder block that token alone queries the hyperspectral patch tokens 2203.16952 §III-B2 2203.16952 §III-B3. It reports 89.80% overall accuracy on Houston with LiDAR added 2203.16952 tab. II. The paper reports test-set accuracy 2203.16952 §IV-B, and the released notebook keeps the checkpoint with the best test accuracy, so the reported numbers are likely optimistic. Card: Multimodal Fusion Transformer for Remote Sensing Image Classification.

FSRA matches drone photos to satellite images for geo-localisation. A ViT-S produces patch tokens, which are ordered by mean activation and cut into three regions; each region is pooled and aligned across views with classification and triplet losses 2201.09206 §3.2. It reports 84.51% recall at 1 with three satellite repeats on University-1652 2201.09206 tab. 2. Card: FSRA drone-to-satellite matching.

Prithvi-EO-2.0 is a ViT masked autoencoder for six-band satellite time series, pretrained on 4.2M four-timestamp samples in 300M and 600M sizes 2412.02732 §3.1 2412.02732 §3.2. Its patch embedding uses a time depth of 1, and date and location enter as learned-weight bias terms, with metadata dropout during pretraining 2412.02732 §3.2. Fine-tuned on Sen1Floods11, the 600M-TL variant reports 90.3 mIoU 2412.02732 §4.2.1. For gross primary productivity it freezes the encoder and trains a small head 2412.02732 §4.4.2. Pretraining took about 58,000 GPU-hours for the 600M model 2412.02732 §3.2. Card: Prithvi-EO-2.0.

ChangeFormer is a Siamese network for change detection between two co-registered remote-sensing images. Its shared encoder is a four-stage hierarchical transformer, not a plain ViT, and its attention shortens the keys and values by a reduction ratio that the paper does not state as a number 2201.01293 §2.1 2201.01293 §2.1.1. Four difference modules compare the two feature maps at each scale, and an MLP decoder outputs the two-class change mask 2201.01293 §2.1.3 2201.01293 §2.2. Author-reported: F1 of 90.40 on LEVIR-CD and 86.67 on DSIFN-CD 2201.01293 tab. 1. The released code differs from the paper in the decoder, the difference module and the initialisation, so the card lists the changes. Card: A Transformer-Based Siamese Network for Change Detection.

Convolutional and detection alternatives

InternImage is a convolutional backbone whose core operator, DCNv3, is a 3×3 deformable convolution with learned sampling offsets, channel groups and softmax-normalised weights 2211.05778 §3.1. It is included here as the strongest convolutional competitor to ViTs. Its H variant, at 640 px, reports 89.6% ImageNet-1K top-1, against 87.5 for the closest CNN baseline in the table, a BiT ResNet at 480 px 2211.05778 tab. 2. Card: InternImage.

ConvNeXt starts from a ResNet-50 and changes its design one step at a time, using Transformer-era training recipes and layer choices, until it reaches a pure ConvNet with 7×7 depth-wise convolutions, inverted bottlenecks, LayerNorm and a single GELU per block 2201.03545 §2 2201.03545 §2.6. The authors use it to separate the effect of architecture from that of training recipe in the Swin comparison 2201.03545 §1. Author-reported: ConvNeXt-T reaches 82.1% ImageNet-1k top-1 at 224 px, against 81.3% for Swin-T 2201.03545 tab. 1. ConvNeXt-XL, pre-trained on ImageNet-22k and tested at 384 px, reaches 87.8% 2201.03545 tab. 1. On COCO with Cascade Mask R-CNN at the 3× schedule, ConvNeXt-T and Swin-T both report 50.4 box AP 2201.03545 tab. 3. Card: A ConvNet for the 2020s.

A survey of the YOLO detectors from v1 to v8, YOLO-NAS and YOLO-with-Transformer variants gives reported COCO AP in one summary table and in the text, and the two disagree for several versions 2304.00501 tab. 4 2304.00501 §3. Transformer-based detectors appear as later variants, such as YOLOS, whose 42.0 AP the survey quotes 2304.00501 §19. The card recommends using it as a reading list and checking the original papers for exact settings. Card: A Comprehensive Review of YOLO Architectures in Computer Vision.

A survey of object detectors compiles author-reported COCO and VOC AP and FPS for 17 detectors, copied from the source papers rather than re-run 2104.11892 §7. Its headline pairs are DetectoRS at 53.3 AP on COCO, about 4 FPS at 1333 input, and YOLOv4 at 43.0 AP and about 31 FPS at 512 input 2104.11892 tab. 3. It places Swin Transformer under single-stage detectors, and that row of its table gives no input size or FPS 2104.11892 §5.3 2104.11892 tab. 3. The compiled numbers come from different hardware and input sizes, so they are not directly comparable 2104.11892 §7. Card: A Survey of Modern Deep Learning based Object Detection Models.

Benchmarks that compare ViTs with CNNs

AQUA20 is a set of 8,171 underwater photographs in 20 classes, with a fixed split of 6,559 training and 1,612 test images 2506.17455 tab. 1 2506.17455 §3.1. Thirteen ImageNet-pretrained networks were fine-tuned on it, among them ViT-B/16, SwinV2-B and ConvNeXt 2506.17455 §3.2 2506.17455 §4.1. ConvNeXt scores highest, at 90.69% top-1 and 88.92% macro F1. ViT-B/16 reaches 87.34% top-1, and SwinV2-B 88.65% 2506.17455 tab. 3. The Grad-CAM and LIME maps are qualitative. They show the best model on parts of the animal that matter for identification, while InceptionV3's attention more often spreads onto the seabed 2506.17455 §4.5. The source images have no stated licence, so check them before redistributing any of them 2506.17455 §3.1. The numbers depend on choices the card flags: checkpoints are chosen on the test loss, the paper states a 32×32 input while the released notebook uses 224×224, and each model was trained once 2506.17455 §4.1 2506.17455 §4.2. Card: AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions.

Hybrid CNN and transformer branches

Conformer runs a ResNet-style convolutional branch and a ViT-style transformer branch side by side, from a shared stem. In every block a Feature Coupling Unit passes information both ways: convolutional features are pooled into patch tokens, and transformer tokens are upsampled back into the convolutional branch 2105.03889 §3.2. The authors compare this with a serial hybrid, a ResNet-26d or ResNet-50d followed by DeiT-S, and report 80.2% for the serial version at similar cost 2105.03889 §4.4 2105.03889 tab. 5. Author-reported ImageNet-1k top-1 at 224 px: Conformer-S 83.4% and Conformer-B 84.1%, against 81.8% for DeiT-B 2105.03889 tab. 2. In Faster R-CNN on COCO, Conformer-S/32 reaches 43.1 box AP against 39.4 for a ResNet-101 FPN 2105.03889 §4.3. Card: Conformer.

CvT stays a single transformer stack but adds convolutions in two places. Overlapping convolutional token embeddings build a three-stage hierarchy 2103.15808 §3.1, and depthwise convolutions compute the queries, keys and values, with stride 2 on keys and values to cut attention cost 2103.15808 §3.2. The model has no position embedding, because the convolutions supply local structure 2103.15808 tab. 5. Author-reported ImageNet-1k top-1 at 224 px: 81.6% for CvT-13, with about 20M parameters 2103.15808 tab. 3. The ImageNet-22k pretrained CvT-W24, fine-tuned at 384 px, reaches 87.7% 2103.15808 tab. 3. The card notes that the paper gives two parameter counts for CvT-W24, 276.7M in tab. 2 and 298.3M in §4.1 2103.15808 tab. 2 2103.15808 §4.1. Card: CvT.

EdgeNeXt is a mobile hybrid that keeps most of its depth in convolution blocks. Its SDTA block sits at the end of stages 2 to 4 2206.10589 §3 Overall Architecture 2206.10589 tab. 1. The block first splits channels into groups that pass through depth-wise convolutions at several scales 2206.10589 eq. 2. Its attention then runs across channels: queries and keys are L2-normalised and form a channel-by-channel score matrix, so the cost grows linearly with the number of pixels 2206.10589 eq. 3 2206.10589 eq. 4. The authors add one positional encoding before stage 2 only, since adding it in every stage raised latency and hurt dense prediction [arxiv:2206.10589 §3 Overall Architecture]. Author-reported: the S variant, with 5.6M parameters and 1.30G MAdds, reaches 79.4% ImageNet-1K top-1 at 256 px, against 78.4% for MobileViT-S at 2.01G MAdds 2206.10589 tab. 3. Removing the SDTA blocks lowers the score to 78.3% 2206.10589 tab. 7. Card: EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications.

Text-supervised grouping for segmentation

GroupViT trains a hierarchical ViT with no mask labels. Patches are merged into larger segments by learned group tokens and a differentiable hard assignment at each stage, and the only training signal is an image-text contrastive loss 2202.11094 §3.1 2202.11094 §3.2. A multi-label variant adds several short prompts per image, built from caption nouns, as extra positives; the authors report this as a key factor for grouping quality 2202.11094 §3.2 2202.11094 tab. 1. For zero-shot segmentation each segment is labelled by its similarity to a label-name text embedding, so the label list can change without retraining 2202.11094 §3.3. Author-reported zero-shot mIoU is 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context, for a model trained on CC12M and filtered YFCC 2202.11094 §4.4 2202.11094 tab. 5. The card records the official code under a non-commercial licence, so the implementation notes here rest on the paper text. Card: GroupViT.

Restoration and enhancement

The U-shape Transformer for underwater enhancement keeps a U-shaped generator but replaces its skip connections with a channel-wise transformer, CMSFFT, and puts a spatial transformer, SGFMT, at the bottleneck 2111.11843 §3.2. The channel-wise block targets the colour channels that water attenuates most 2111.11843 §1. Training is adversarial, with an RGB, LAB and LCH colour loss and a perceptual term 2111.11843 §3.3. The paper also releases LSUI, 4,279 real underwater pairs with reference images 2111.11843 §3.1. Author-reported: PSNR 24.16 and SSIM 0.93 on Test-L400, the best of ten compared methods 2111.11843 tab. 3. Card: U-shape Transformer.

Multi-agent perception for driving

V2X-ViT fuses LiDAR feature maps from connected vehicles and roadside sensors for 3D vehicle detection 2203.10638 §3. Its encoder stacks three layers, each with heterogeneous multi-agent self-attention, which treats vehicles and infrastructure differently, and multi-scale window attention 2203.10638 §3.2 2203.10638 app. B.2. A delay-aware encoding is added at every position of each shared feature map 2203.10638 eq. 8 2203.10638 eq. 10. The evaluation uses the authors' simulated V2XSet, under perfect and noisy localisation and delay 2203.10638 §4.1 2203.10638 tab. 1. In the noisy setting, AP at IoU 0.7 is 0.614, against 0.402 with no fusion 2203.10638 tab. 1. Card: V2X-ViT.

Phone-latency designs

EfficientFormer is a pure transformer tuned for phone latency. Its early stages use convolution-style pooling blocks, and only the last stage applies global multi-head self-attention 2206.01191 eq. 4 2206.01191 eq. 5. Depth and width are chosen by a latency-driven slimming step that uses a per-block latency table measured on the device 2206.01191 §4.2 2206.01191 alg. 1. The target is an iPhone 12 with CoreML 2206.01191 §1. Author-reported: EfficientFormer-L1 reaches 79.2% top-1 at 1.6 ms, and L7 reaches 83.3% at 7.0 ms 2206.01191 tab. 1. Card: EfficientFormer.

EdgeViT is a hierarchical transformer tuned for measured mobile latency. Each block mixes neighbours with convolutions, runs attention over one sampled token per r×r window, and spreads that context back to the window with a transposed convolution 2205.03436 §3.2 2205.03436 eq. 1. Author-reported ImageNet-1K top-1 is 74.4%, 77.5% and 81.0% for its three sizes, at 32.8, 54.1 and 85.3 ms on a Galaxy S21 CPU 2205.03436 tab. 2. MobileViT-XS reaches 74.7% at 150.1 ms on the same protocol 2205.03436 tab. 2. The protocol is CPU, FP32, batch 1, through TorchScript lite, which the authors chose because it ran all their models 2205.03436 §4.1. Energy was measured on a different Snapdragon device, so the latency and energy columns cannot be converted into each other 2205.03436 tab. 3. Card: EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers.

RepViT takes the opposite route: a pure CNN, built from MobileNetV3-L by adopting layout choices from small ViTs. Each block separates the spatial token mixer, a re-parameterised depthwise convolution, from the channel mixer, an FFN with expansion 2 2307.09283 §3.2. A separate stride-2 block handles downsampling 2307.09283 §3.3. With distillation, the reported ImageNet-1K top-1 on iPhone 12 with Core ML runs from 78.7% for M0.9 at 0.9 ms to 83.7% for M2.3 at 2.3 ms 2307.09283 tab. 1. Removing re-parameterisation lowers the M0.9 score from 78.74% to 78.47% 2307.09283 tab. 6. Some architecture details appear only in supplementary material, which is not in the source the card was written from. Card: RepViT.

When to use which

Implementation sketch

The shared core of most variants is a patch embedding followed by pre-norm transformer blocks. Segmenter's linear decoder is the simplest head on top 2105.05633 §3.1 2105.05633 §3.2. Other variants change the tokeniser (ESRT unfolds 3×3 neighbourhoods, MFT pools into a few tokens), the attention pattern (windows, segments, experts, Fourier mixing in AFNO 2111.13587 §4) or the training target (masked pixels, teacher tokens, prototype assignments 2204.07141 §3), and are noted in the variant sections above. MAE differs at the input rather than the loss: it removes the masked patches before the encoder 2111.06377 §3. ST-Adapter leaves the frozen weights alone and adds a residual bottleneck inside each block 2206.13559 §3.2.

# Shared ViT core for dense prediction  [arxiv:2105.05633 §3.1]
# Input: image (B, 3, H, W); patch size P; depth L; width D
# Output: one D-dim token per patch, (B, N, D) with N = H*W / P^2

def vit_encoder(image, pos, P, L):
    # Cut the image into P x P patches and project each one linearly
    tokens = conv2d(image, out=D, kernel=P, stride=P)  # (B, D, H/P, W/P)
    tokens = flatten(tokens)                           # (B, N, D)
    # Add learned position embeddings, bilinearly resized to the grid
    # [arxiv:2105.05633 §4.2]
    tokens = tokens + resize_pos(pos, tokens.grid)     # (B, N, D)
    # Pre-norm blocks: global attention over all patches, then an MLP
    # [arxiv:2105.05633 §3.1] [arxiv:2012.15840 §3.2]
    for _ in range(L):
        tokens = tokens + attention(layer_norm(tokens))
        tokens = tokens + mlp(layer_norm(tokens))
    return layer_norm(tokens)                          # (B, N, D)

# Linear decoder: one score per class per patch, then upsample
# [arxiv:2105.05633 §3.2]
# Input: tokens (B, N, D); grid H/P x W/P; K classes
# Output: logits (B, K, H, W)

def linear_head(tokens, grid, out_size):
    # Point-wise linear map from D to K classes
    logits = linear(tokens, out=K)                     # (B, N, K)
    logits = reshape(logits, (B, K) + grid)            # (B, K, H/P, W/P)
    # Bilinear upsampling to the input resolution
    return bilinear_upsample(logits, out_size)         # (B, K, H, W)

Masked-image variants replace the masked patch embeddings with one learned token before the blocks 2111.09886 §3.2, and the sparse-expert variant swaps the MLP for a routed set of experts 2106.05974 §2.2.

Contradictions

• Is global attention the source of the gains? SETR and Segmenter motivate attention in every layer by global context 2012.15840 §1 2105.05633 §1. How Do Vision Transformers Work? argues the opposite emphasis: data specificity of the attention matters more than long-range dependency, and it finds that local windows can beat global attention on small data 2202.06709 §2. The papers test different things (segmentation with pretraining versus CIFAR-scale optimisation), so neither result settles the question.
• Linear probing. SimMIM reports 56.7% linear accuracy for ViT-B and says its features are meant for fine-tuning 2111.09886 tab. 6 2111.09886 §3.5. iBOT reports 79.5% linear accuracy for ViT-B under a DINO-style protocol 2111.07832 tab. 1. MAE reports 75.8% for ViT-L and argues that linear probing is a poor proxy for transfer 2111.06377 app. B 2111.06377 §4.3. The papers use different targets and evaluation protocols, so the cards cannot say which objective produces better frozen features.
• Low-label performance of MAE features. MSN reports 8.2% for MAE ViT-B/16 with one label per class, and 49.8% for its own ViT-B/16 2204.07141 tab. 1. The MSN card says the MAE one-image row uses a linear classifier instead of partial fine-tuning, and that MSN's own low-shot rows use L2-regularised logistic regression on frozen features 2204.07141 tab. 1 2204.07141 app. A.2. The protocols are close but not identical, and the cards do not say how much of the gap is due to the protocol.
• Paper versus released code. Several cards record that the released code differs from the paper in ways that change the result. ResT's code skips the head-mixing step in stage 1, which has one head 2105.13677 §2.2. The AFNO card reads the released code as not performing the hard frequency truncation the paper describes 2111.13587 §5.6. STEGO's default config is ViT-Small, while the paper's numbers are for ViT-Base 2203.08414 §4.3. ST-Adapter's code has no non-linearity where the equation has one 2206.13559 §3.2. Implement the variant you intend to match, and check its card. SAM-Med3D's paper describes an overlapping sliding window for large volumes, which its released code does not include 2310.15161 §4.3. Conformer's detection learning rate is 2e-4 at batch 32 in the paper 2105.03889 §4.3; the released mmdetection config uses 1e-4, as the Conformer card records. PiT's text and its released code give different patch sizes for the base model 2103.16302 §3.4 2103.16302 tab. 1. XCiT's appendix algorithm and its released code take the softmax over different channel axes 2106.09681 app. C. ConvNeXt's table gives a weight-initialisation scale of 0.2, and the code uses 0.02 2201.03545 tab. 5. Semantic softmax's depth weights in the ImageNet-21K-P code differ from the printed equation 2104.10972 eq. 2 2104.10972 eq. 3; see the card.
• Test-split reporting. FacT reports its VTAB-1K numbers on the test split 2212.03145 §4.1. The MFT card judges its own reported numbers likely optimistic because the released notebook keeps the checkpoint with the best test accuracy 2203.16952 §IV-B. AQUA20 has the same issue: its paper keeps the best checkpoint seen during evaluation, and its released notebook keeps the checkpoint with the lowest test loss 2506.17455 §4.1. Compare such numbers with care.
• Is a transformer needed at all? DiT argues that the U-Net's inductive bias is not what makes diffusion models work 2212.09748 §1. InternImage argues that a convolutional operator with input-dependent sampling can reach ViT-level scale and accuracy 2211.05778 §1. ChangeFormer argues that a transformer relates distant parts of a change-detection pair better than deep ConvNets do 2201.01293 §1. ConvNeXt makes a related case: a ResNet changed step by step with Transformer-era recipes and layer choices matches Swin-T on ImageNet-1k 2201.03545 §1 2201.03545 tab. 1. The four papers test different tasks and do not share a protocol, so none of them settles the question.
• Same setting, two numbers. ToMe reports 83.92% at 183 images per second in its appendix sweep and 84.25% at 182.9 in its ablation table for ViT-L/16 MAE with r = 8 2210.09461 app. A 2210.09461 tab. 1. The unmerged baseline also differs: the table caption gives 85.96% at 93.3 images per second, the sweep 85.66% at 93, and the paper does not reconcile the two 2210.09461 tab. 1. Use one source and say which.
• Record claims depend on the comparison. Swin V2 reports new results on several vision benchmarks, but on ImageNet-1K V1 its 3B model is slightly below CoAtNet-7 (90.17% against 90.88%) 2111.09883 tab. 2. The authors did not align model size or pretraining data across rows 2111.09883 §4.2.
• Masked-patch objectives in vision-language pretraining. METER reports that adding masked-patch prediction lowers its VQAv2 score 2111.02387 tab. 7, and its card notes that the released code has no such objective. CAE finds that masked-patch reconstruction is needed to avoid a trivial solution 2202.03026 §5.5. The two settings differ, one a joint image-text model and the other an image-only encoder, so neither result settles the other.
• Synapse numbers for one model. The medical review gives TransUNet 77.49% Dice in its Synapse table 2301.03505 tab. 6, while its own discussion gives 77.48% 2301.03505 §4. The TransUNet paper reports 77.48% 2102.04306 tab. 1. The difference is small, but it shows that the review's table is not one source copied unchanged. Use the source paper's number and say which one you used.
• Relative or absolute position. iRPE finds that a relative term can replace the absolute encoding for classification: with no absolute encoding and the query, key and value terms together, DeiT-S reaches 81.3% top-1 against 79.9% for the learned absolute baseline 2107.14222 tab. 4. For DETR the same choice does not hold: without the absolute encoding, the contextual term reaches 38.7 AP, below 39.5 for the sinusoidal baseline 2107.14222 tab. 6. The paper also says earlier vision studies disagreed on whether relative encodings beat absolute ones 2107.14222 §1. The two tasks differ in resolution and training, so the card does not treat either result as general.
• Released configs for two ViT papers. iRPE's DeiT config names do not match the bucket counts in its Table 1; the card's hand calculation gives different counts 2107.14222 tab. 1 rpe_models.py. EdgeNeXt's released XXS model uses 2, 2, 6 and 2 blocks per stage, against 3, 3, 9 and 3 in its table 2206.10589 tab. 1 model.py. Take the block counts and bucket counts from the paper's tables and check the cards before reusing the configs.
• Stage depths in a mobile transformer. EdgeViT's table gives XS stage depths of 1, 1, 2 and 2, and S depths of 1, 2, 3 and 2. The released code uses 1, 1, 3 and 1 for XS and 1, 2, 5 and 3 for S 2205.03436 tab. 1. The card's numbers come from the table, so a reimplementation from the paper will have different depths from the code.

Papers

• SETR: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers: full-resolution ViT encoder with three segmentation decoders.
• Segmenter: Transformer for Semantic Segmentation: linear and mask-transformer heads on a ViT encoder; convolution-free.
• SimMIM: A Simple Framework for Masked Image Modeling: masked pixel regression with a linear head and large masked blocks.
• Transformer for Single Image Super-Resolution: lightweight super-resolution with segment-wise attention over unfolded patches.
• Multimodal Fusion Transformer for Remote Sensing Image Classification: classification token from a second sensor fused with hyperspectral patch tokens.
• Sigmoid Loss for Language Image Pre-Training: pairwise sigmoid loss for image-text pretraining with a ViT image tower.
• Improved Baselines with Visual Instruction Tuning: frozen CLIP ViT-L/14 with an MLP connector for a multimodal chat model.
• Scaling Vision with Sparse Mixture of Experts: sparse MoE ViT with batch prioritised routing at inference.
• iBOT: Image BERT Pre-Training with Online Tokenizer: masked image modeling with an EMA teacher as the online tokenizer.
• How Do Vision Transformers Work?: analysis of attention as smoothing and a local-attention hybrid, AlterNet.
• FSRA drone-to-satellite matching: patch-token regions aligned across drone and satellite views.
• Generalized Category Discovery: ViT-B/16 features clustered with semi-supervised k-means over known and new classes.
• nnFormer: Interleaved Transformer for Volumetric Segmentation: 3D transformer with local and global attention and skip attention.
• Transformers in Remote Sensing: A Survey: survey of transformer methods for optical, hyperspectral and radar imagery.
• Rethinking Vision Transformers for MobileNet Size and Speed: hybrid mobile ViT found by a size-and-speed search.
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone: hierarchical backbone mixing convolution, state-space blocks and attention.
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks: prompt-driven seq2seq vision model with location tokens.
• Canopy height from RGB imagery: frozen DINOv2 ViT-H with a DPT decoder trained on lidar heights.
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers: stored-teacher distillation and progressive model contraction for small ViTs.
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models: frame-level CLIP ViT features pooled into video tokens for a chat model.
• ResT: An Efficient Transformer for Visual Recognition: multi-scale pyramid with efficient attention and a convolutional positional gate.
• Masked Autoencoders Are Scalable Vision Learners: masked-patch pixel reconstruction with an encoder that sees only visible patches.
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers: Fourier-domain token mixer in place of self-attention.
• Rethinking Spatial Dimensions of Vision Transformers: PiT, pooling between ViT stages to shrink the token grid and widen channels.
• ImageNet-21K Pretraining for the Masses: cleaned ImageNet-21k with semantic softmax over WordNet depths.
• XCiT: Cross-Covariance Image Transformers: attention across feature channels, with cost linear in the token count.
• A ConvNet for the 2020s: ConvNeXt, a pure ConvNet modernised step by step from ResNet-50.
• Deep ViT Features as Dense Visual Descriptors: DINO ViT keys as training-free dense descriptors for co-segmentation and correspondence.
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences: frozen DINO ViT-B with a small head distilled from feature correlations, then clustered.
• Masked Siamese Networks for Label-Efficient Learning: masked joint-embedding with prototypes, no pixel reconstruction.
• IAT: a 90K-parameter transformer for image enhancement and exposure correction: tiny transformer-style image enhancer with learned queries for a global colour matrix and gamma.
• ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning: bottleneck adapters with a depth-wise temporal 3D convolution on a frozen image ViT.
• Visual Prompting via Image Inpainting: MAE-VQGAN inpaints a grid prompt of example pairs and a query.
• VIMA: General Robot Manipulation with Multimodal Prompts: object-token prompts with a cross-attention decoder for robot control.
• Photorealistic Video Generation with Diffusion Models: W.A.L.T, a window-attention diffusion transformer on a shared image-video latent.
• Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications: ViT masked autoencoder for multispectral satellite time series with metadata bias terms.
• All Tokens Matter: Token Labeling for Training Better Vision Transformers: per-patch dense loss from a pretrained annotator, with MixToken on patch embeddings.
• Splicing ViT Features for Semantic Appearance Transfer: small generator trained on one pair, judged by frozen DINO ViT-B/8 features.
• NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation: window-local fully connected CRF with learned attention potentials for depth.
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation: transformer cost aggregation over the query-support correlation volume.
• UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer: video modules added to a pretrained image ViT.
• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer: tensorised ViT weights with a shared low-rank factorised update.
• Language-Driven Representation Learning for Robotics: masked video autoencoder whose caption is a condition or a generation target.
• SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images: promptable 3D ViT segmenter for medical volumes with point prompts.
• Bi-directional Adapter for Multi-modal Tracking: shared bi-directional adapter that lets two modality streams prompt each other.
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models: video ViT memory plus a language summary memory for a robot policy.
• AdaBins: Depth Estimation using Adaptive Bins: per-image bin widths from a small transformer head for depth.
• Transformer Interpretability Beyond Attention Visualization: class-specific relevance through attention and skip connections.
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation: ResNet-50 stem, ViT-B/16 encoder and a cascaded upsampler.
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: shifted-window attention in a four-stage pyramid.
• Emerging Properties in Self-Supervised Vision Transformers: DINO, label-free self-distillation for ViTs.
• Video Swin Transformer: shifted 3D windows for video on a Swin backbone.
• Swin Transformer V2: Scaling Up Capacity and Resolution: scaled Swin with post-norm, cosine attention and a log-spaced position bias.
• A Transformer-Based Siamese Network for Change Detection: hierarchical transformer Siamese encoder for change masks.
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language: one self-distillation recipe across three modalities.
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning: chart question answering with a ViT-based table baseline.
• PaLI: A Jointly-Scaled Multilingual Language-Image Model: image and prompt in, text out, with a ViT-e encoder and mT5.
• Token Merging: Your ViT But Faster: merges similar tokens in every block without new parameters.
• InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions: convolutional backbone that competes with ViTs at scale.
• Scalable Diffusion Models with Transformers: DiT, a transformer denoiser for latent diffusion.
• Multimodal Chain-of-Thought Reasoning in Language Models: rationale and answer in two stages, with frozen ViT features.
• A Comprehensive Review of YOLO Architectures in Computer Vision: survey of YOLO detectors from v1 to v8 and their variants.
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention: sandwich blocks and cascaded group attention for measured speed.
• A Survey of Modern Deep Learning based Object Detection Models: survey of detectors, backbones and lightweight networks, with author-reported numbers compiled side by side.
• Conformer: Local Features Coupling Global Representations for Visual Recognition: CNN and transformer branches coupled in every block by a feature coupling unit.
• U-shape Transformer for Underwater Image Enhancement: channel-wise and spatial transformers inside a U-shaped enhancement GAN, with the LSUI dataset.
• GroupViT: Semantic Segmentation Emerges from Text Supervision: hierarchical grouping ViT trained on image-text pairs for zero-shot segmentation.
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer: transformer fusion of vehicle and roadside LiDAR features with delay-aware encoding.
• EfficientFormer: Vision Transformers at MobileNet Speed: pure transformer with convolutional early stages, tuned by on-device latency.
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery: masked-autoencoder pretraining with time and spectral-group encodings.
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking: dual masking for a billion-parameter video ViT, with progressive training.
• RepViT: Revisiting Mobile CNN From ViT Perspective: pure CNN that adopts lightweight-ViT block layout, measured on an iPhone.
• SLIP: Self-supervision meets Language-Image Pre-training: SimCLR-style image loss added to CLIP's contrastive loss on one shared ViT.
• Expanding Language-Image Pretrained Models for General Video Recognition: CLIP image encoder with cross-frame message tokens and video-specific text prompts.
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning: masked-video ViT-H and a CLIP-initialised UniFormerV2 fused by cross-model attention.
• VideoMamba: State Space Model for Efficient Video Understanding: bidirectional Mamba blocks in a ViT-style stack, with self-distillation from a smaller model.
• Metric3D v2: DINOv2 ViT encoders with DPT decoders for metric depth and normals from one image.
• Transformers in Vision: A Survey: survey of transformer designs in vision, grouped by self-attention block and by task.
• CvT: Introducing Convolutions to Vision Transformers: overlapping convolutional tokens and convolutional projections, with no position embedding.
• An Empirical Study of Training Self-Supervised Vision Transformers: MoCo v3 for ViTs, with training instability and a frozen patch projection.
• PVT v2: Improved Baselines with Pyramid Vision Transformer: pooled-key attention, overlapping patch embedding and depthwise convolution in a pyramid.
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training: tube-masked video autoencoding on a plain ViT, pretrained without extra data.
• Scaling Local Self-Attention for Parameter Efficient Visual Backbones: HaloNet, local 2D self-attention with blocked haloing as a convolution replacement.
• An Empirical Study of Training End-to-End Vision-and-Language Transformers: METER, a fully transformer vision-language model with co-attention and no region detector.
• Context Autoencoder for Self-Supervised Representation Learning: masked-image pretraining with a separate encoder, latent regressor and decoder.
• FlowFormer: A Transformer Architecture for Optical Flow: transformer over compressed cost-volume tokens, read by a recurrent cross-attention decoder.
• Side Adapter Network for Open-Vocabulary Semantic Segmentation: light side ViT on frozen CLIP producing mask proposals and attention biases.
• Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs: ViT-B/16 driving policy with a cached left, middle or right instruction from a zero-shot LLM.
• Rethinking and Improving Relative Position Encoding for Vision Transformer: learned relative bucket terms added to attention logits, keys, queries or values.
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications: mobile hybrid whose SDTA blocks attend across channels.
• FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling: grid mini-patch fragments with a Video Swin-T trained end to end for video quality.
• Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review: survey of ViT methods across medical imaging tasks, with compiled benchmark tables.
• P2T: Pyramid Pooling Transformer for Scene Understanding: pyramid of pooled keys and values with depthwise convolutions in attention and feed-forward.
• Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking: one ViT for exemplar and search tokens, with a foveal window of exemplar centre tokens.
• EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers: local-global-local blocks with one sampled token per window, measured on a phone CPU.
• Fast Vision Transformers with HiLo Attention: HiLo heads for local windows and pooled global attention, in the LITv2 backbone.
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics: IRENE, one transformer over radiograph, text and lab tokens for clinical prediction.
• AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions: 8,171-image underwater benchmark with fine-tuned ViT, CNN and hybrid baselines.