---
kind: concept
slug: vision-transformer
title: Vision Transformer (ViT) backbones
summary: >-
  A Vision Transformer cuts an image into patches, embeds each as a token and runs a Transformer encoder over them. Variants change the pretraining objective, the token mixer, the input or the role in a larger system. Frozen features often transfer to dense and specialised tasks. Trade-offs: results depend on pre-training, and attention cost grows with the square of the token count.
tags: [computer-vision, vision-transformer, backbone, transformer, pre-training]
papers: ["2012.15840", "2105.05633", "2111.09886", "2108.11084", "2203.16952", "2303.15343", "2310.03744", "2106.05974", "2111.07832", "2202.06709", "2201.09206", "2201.02609", "2109.03201", "2209.01206", "2212.08059", "2407.08083", "2311.06242", "2304.07213", "2207.10666", "2306.05424", "2105.13677", "2111.06377", "2111.13587", "2112.05814", "2203.08414", "2204.07141", "2205.14871", "2206.13559", "2209.00647", "2210.03094", "2312.06662", "2412.02732", "2104.10858", "2201.00424", "2203.01502", "2207.10866", "2211.09552", "2212.03145", "2302.12766", "2310.15161", "2312.10611", "2603.03596", "2011.14141", "2012.09838", "2102.04306", "2103.14030", "2104.14294", "2106.13230", "2111.09883", "2201.01293", "2202.03555", "2203.10244", "2209.06794", "2210.09461", "2211.05778", "2212.09748", "2302.00923", "2304.00501", "2305.07027", "2104.11892", "2105.03889", "2111.11843", "2202.11094", "2203.10638", "2206.01191", "2207.08051", "2303.16727", "2307.09283", "2112.12750", "2208.02816", "2212.03191", "2403.06977", "2404.15506", "2101.01169", "2103.15808", "2104.02057", "2106.13797", "2203.12602", "2103.16302", "2104.10972", "2106.09681", "2201.03545", "2103.12731", "2111.02387", "2202.03026", "2203.16194", "2302.12242", "2411.14256", "2107.14222", "2206.10589", "2207.02595", "2301.03505", "2106.12011", "2203.05328", "2205.03436", "2205.13213", "2306.00864", "2506.17455"]
related: [semantic-segmentation, optical-flow, multimodal-fusion, object-queries, efficient-vision-backbone, masked-image-modeling, self-attention, image-super-resolution, image-fusion, vision-language-pretraining, multimodal-language-model, mixture-of-experts, self-supervised-learning, novel-category-discovery, state-space-model, linear-probing, knowledge-distillation, visual-instruction-tuning, transfer-learning, parameter-efficient-fine-tuning, in-context-learning, diffusion-models, latent-diffusion, imitation-learning, robot-simulation, multi-task-learning, fourier-neural-operator, positional-encoding, relative-positional-encoding, contrastive-learning, part-based-features, k-means-clustering, low-rank-adaptation, depthwise-separable-convolution, convolutional-neural-network, deformable-convolution, monocular-depth-estimation, conditional-random-field, cost-aggregation, few-shot-learning, neural-style-transfer, image-to-image-translation, tensor-decomposition, vision-language-action-model, visual-object-tracking, promptable-segmentation, shifted-window-attention, token-merging, token-pruning, local-global-attention, attention-interpretability, feature-attribution, object-detection, label-assignment, neural-architecture-search, classifier-free-guidance, chain-of-thought-prompting, table-question-answering, chart-data-extraction, siamese-network, foundation-model, neural-scaling-laws, image-enhancement, generative-adversarial-network, open-vocabulary-segmentation, collaborative-perception, bird-eye-view-perception, channel-attention, batchnorm-folding, attention-mechanism, image-registration, image-quality-assessment, hierarchical-vision-transformer, benchmark-validity]
status: draft
updated_at: 2026-11-01
---

## Overview

A Vision Transformer (ViT) splits an image into a grid of square patches, maps each
patch to a vector, adds position information and runs a stack of self-attention and
MLP blocks over the resulting sequence [arxiv:2105.05633 §3.1]. Every patch can
therefore attend to every other patch from the first layer on, which is the main
property the papers below build on. The cost is that attention grows with the square of
the number of patches, so the patch size and the input resolution set the budget
[arxiv:2105.05633 tab. 3] [arxiv:2108.11084 §3.3.2]. A survey groups the designs by how
they change self-attention [arxiv:2101.01169 §1], and notes that ViTs need large
pretraining sets because they assume little about the image [arxiv:2101.01169 §2.2]
[arxiv:2101.01169 §4.2]. Its
reported numbers come from the cited papers, so this page uses the primary cards instead.

The cards below use the same backbone in very different roles: a per-pixel
segmenter, a 3D medical network, a super-resolution network, a self-supervised
pretraining target, a mixture-of-experts layer, the image tower of a vision-language
model, the frozen encoder of a multimodal chat model, a frozen feature extractor for
dense correspondence, an adapter target for video, a video quality scorer, a per-object encoder in a robot
policy and a satellite foundation model, a depth decoder, a few-shot cost aggregator, a video
adapter and a tracking adapter, a joint exemplar-search tracker, a clinical
multimodal transformer and a benchmark for underwater classification. Some cards use it
as a hierarchical, shifted-window
backbone, as the denoiser of a diffusion model, as a teacher in self-distillation,
or as the vision half of a chart or science question-answering system. Two kinds of
change recur. Some papers change
the pretraining objective; others change the token mixer, which is the part that
replaces or reduces self-attention. One point also recurs across them:
pre-training carries most of the accuracy. Several cards report that the same
architecture trained from scratch loses most of its score.

Most numbers here are author-reported. The papers use different backbones, input sizes,
schedules and test protocols, so a number from one row is not directly comparable with a
number from another. Where a comparison is made below, the protocol is the same within
the cited table.

## Variants

### Plain ViT encoder with a light decoder for dense prediction

SETR keeps the transformer at a fixed 1/16 resolution through every layer and only adds
a decoder at the end. It tries three decoders: one that projects and upsamples in a
single step, one that upsamples progressively, and one that fuses features from several
encoder layers [arxiv:2012.15840 §3.2] [arxiv:2012.15840 §3.3]. With ViT-large weights
pretrained on ImageNet-21k it reports 50.28% mIoU on ADE20K validation with multi-scale
testing for its best decoder [arxiv:2012.15840 tab. 4], and a random-initialised model
reaches only 42.27% on Cityscapes against a much higher figure with pretraining
[arxiv:2012.15840 tab. 2]. Card: [SETR](/papers/2012.15840).

Segmenter drops the convolutional decoder too. Its linear variant maps each output patch
token to class scores. Its mask variant adds one learnable class embedding per category,
lets these attend to the patch tokens, and scores each patch against each class embedding
to form one mask per class [arxiv:2105.05633 §3.2]. Under the same pretraining, the mask
decoder improves the mIoU of the largest model slightly on ADE20K
[arxiv:2105.05633 tab. 4], and the best model reports 53.63% multi-scale mIoU on ADE20K
[arxiv:2105.05633 tab. 6]. The paper also shows that training from scratch is far worse
than starting from ImageNet-21k weights, at 4.42 to 12.51 mIoU against 45.37
[arxiv:2105.05633 tab. 9], and that patch size 8 gains about 1.5 mIoU over patch size 16
while running at roughly 7 instead of 92 images per second
[arxiv:2105.05633 tab. 3]. Card: [Segmenter](/papers/2105.05633).

The survey in remote sensing collects reported results for transformer methods on
very-high-resolution optical, hyperspectral and radar images. It states that it re-runs
none of them, so its tables only show what each source paper reported
[arxiv:2209.01206 §5] [arxiv:2209.01206 §6]. Card: [Transformers in Remote Sensing: A Survey](/papers/2209.01206).

### Volumetric and medical segmentation

TransUNet is a hybrid. A ResNet-50 stem feeds a ViT-B/16 encoder, and a cascaded upsampler
(CUP) decodes back to full resolution with skip connections from the CNN [arxiv:2102.04306 §3.2].
On Synapse multi-organ CT, the table has a plain ViT with the CUP decoder at 67.86% average
DSC, the ResNet-50 hybrid with CUP at 71.29%, and the full TransUNet at 77.48%
[arxiv:2102.04306 tab. 1]. It also reports 89.71% average DSC on ACDC cardiac MRI
[arxiv:2102.04306 tab. 5]. The paper leaves the loss, the learning-rate schedule and the
checkpoint choice unstated, so the card points to the released code for those values, and it
flags one row of the Synapse table whose per-organ values do not average to its printed score.
Card: [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](/papers/2102.04306).

nnFormer interleaves convolutional embedding and down-sampling with attention inside
local 3D blocks, and uses global attention only in the small bottleneck. Its decoder
replaces the usual skip connection with attention from the decoder to the matching
encoder level [arxiv:2109.03201 §3.2] [arxiv:2109.03201 §3.4]. Against nnUNet, a strong
convolutional baseline, it reports lower HD95 on all three datasets, while the Dice
scores are close and on Synapse nnUNet is slightly higher [arxiv:2109.03201 tab. 5].
Card: [nnFormer](/papers/2109.03201).

SAM-Med3D is a promptable segmentation model for volumes: a 3D ViT image encoder, a
3D point-prompt encoder and a 3D mask decoder [arxiv:2310.15161 §4.1]. It is trained
on the authors' SA-Med3D-140K data [arxiv:2310.15161 §3.1] in two stages, pre-training
on the full 131K training masks and then fine-tuning on about 75K filtered masks
[arxiv:2310.15161 §4.2]. Author-reported: 76.27% overall Dice with one 3D point, against
36.83% for SAM-Med2D with N points, and 80.71% with ten 3D points [arxiv:2310.15161 tab. 3].
The paper describes an overlapping sliding window for volumes larger than 128 voxels per
side, but the released code has no sliding-window path for 3D inference [arxiv:2310.15161 §4.3]. Card: [SAM-Med3D](/papers/2310.15161).

A review of Transformers in medical image analysis compiles reported results for
segmentation. Its Synapse table lists ten models with parameters, FLOPs and Dice, and
MERIT has the best row at 84.90% Dice and 13.22 HD95 [arxiv:2301.03505 tab. 6]. A second
table covers skin-lesion segmentation on three datasets [arxiv:2301.03505 tab. 7]. The
authors do not say whether each row was copied from its source paper or re-run, so the
rows should not be read as one protocol. The review sorts models into pure Transformer,
hybrid CNN-Transformer and other placements, and it lists windowed and channel-wise
attention among the ways to lower the quadratic cost [arxiv:2301.03505 §2.2]
[arxiv:2301.03505 §10]. Card: [Advances in Medical Image Analysis with Vision Transformers](/papers/2301.03505).

### Efficient attention and token mixing

ESRT is a super-resolution model that runs a small CNN at half resolution and then one
Transformer whose tokens are unfolded 3×3 neighbourhoods. Attention is computed inside
four contiguous segments of the token sequence, which cuts the attention memory
[arxiv:2108.11084 §3.3]. Its training GPU memory is reported at 4,191M against 16,057M
for a plain Transformer in the same ablation [arxiv:2108.11084 tab. 4]. Card:
[Transformer for Single Image Super-Resolution](/papers/2108.11084).

EfficientFormerV2 moves a depthwise convolution into the feed-forward block and keeps
self-attention only in the last two stages. Its search trades parameters against
iPhone 12 latency. The smallest model reports 75.7% ImageNet top-1 at 3.5M parameters
and 0.9 ms [arxiv:2212.08059 tab. 2]. The search code is not in the released repository,
so the fixed networks are what can be reused [arxiv:2212.08059 §4.2]. Card:
[Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059).

ResT is a four-stage pyramid that lays out a Transformer like a ResNet. Its attention
shrinks keys and values with a strided depth-wise convolution, and a 1×1 convolution
with instance normalisation mixes the heads [arxiv:2105.13677 §2.2]. Position comes from
a depth-wise-conv gate, which the paper says accepts any input size
[arxiv:2105.13677 §2.4]. ResT-Small reports 79.6% ImageNet-1k top-1 at 224 px
[arxiv:2105.13677 tab. 2], and 40.3 box AP for RetinaNet on COCO
[arxiv:2105.13677 tab. 3]. In a simpler ResT-Lite ablation, the gate scores 72.88% against
71.54% with no positional encoding [arxiv:2105.13677 tab. 8]. Card:
[ResT](/papers/2105.13677).

XCiT replaces token-to-token attention with cross-covariance attention, which computes a
d × d map between feature channels instead of an N × N map between tokens. Its cost is linear
in the number of tokens. Channel-wise L2 normalisation of queries and keys is needed for stable
training; without it the authors report failure for S12/16 [arxiv:2106.09681 §3.2]
[arxiv:2106.09681 tab. 4]. Each block also adds a depth-wise 3×3 convolution for local patch
interaction, and removing it costs 1.2 top-1 points for S12/16 [arxiv:2106.09681 tab. 4].
Author-reported: 82.0% ImageNet-1k top-1 for S12/16 at 224 px without distillation
[arxiv:2106.09681 tab. 1], and 47.0 box AP for S12/8 with Mask R-CNN on COCO [arxiv:2106.09681
tab. 5]. The 8 × 8 patch variants cost much more: S12/8 needs 55.6 GFLOPs at 384 px, against
4.8 for S12/16 at 224 px [arxiv:2106.09681 tab. 1]. Card: [XCiT](/papers/2106.09681).

AFNO removes self-attention from the token mixer. It takes a 2D FFT of the token grid,
mixes channels in each block, soft-thresholds the frequencies and inverts the FFT
[arxiv:2111.13587 §4]. The paper lists its cost as N d²/k + N d log N FLOPs, against
N²d + 3Nd² for self-attention [arxiv:2111.13587 tab. 1]. On Cityscapes at 1024×1024 with
SegFormer-B3, AFNO-100% reports 80.9 mIoU against 79.7 for efficient self-attention
[arxiv:2111.13587 tab. 4]. On inpainting it is within 0.01 dB PSNR of attention
[arxiv:2111.13587 tab. 2]. Card: [Adaptive Fourier Neural Operators](/papers/2111.13587).

Token Merging (ToMe) adds no parameters and sits inside each block of a plain ViT. In
every layer it merges the r most similar token pairs, using cosine similarity on the
attention keys and bipartite soft matching between two alternating token sets
[arxiv:2210.09461 §3] [arxiv:2210.09461 tab. 1]. A merged token is an average weighted by
the number of patches it holds, and attention adds a log-size term so that it counts as
many keys [arxiv:2210.09461 §3] [arxiv:2210.09461 eq. 1]. Off the shelf, with r = 8 on
ViT-L/16 MAE, the appendix sweep reports 83.92% top-1 at 183 images per second, against
85.66% at 93 without merging [arxiv:2210.09461 app. A]; the ablation table reports 84.25%
at 182.9 images per second for the same setting [arxiv:2210.09461 tab. 1]. The authors also
test video and audio [arxiv:2210.09461 tab. 5] [arxiv:2210.09461 tab. 7]. Card:
[Token Merging: Your ViT But Faster](/papers/2210.09461).

EfficientViT is built for measured speed rather than operation count. Each block is a
sandwich with one self-attention layer between feed-forward layers, and depthwise
convolutions sit before the feed-forward layers [arxiv:2305.07027 §3.1] [arxiv:2305.07027 eq. 1].
Its cascaded group attention splits the channels across heads, and each head also receives
the output of the previous head, so the heads see different slices of the features
[arxiv:2305.07027 §3.1] [arxiv:2305.07027 eq. 2] [arxiv:2305.07027 eq. 3]. The largest model,
M5, reports 77.1% ImageNet-1K top-1 at 10,621 images per second on a V100, against 75.2% and
7,560 images per second for MobileNetV3-Large [arxiv:2305.07027 tab. 2]. Some speed ratios in
the text do not reproduce from that table [arxiv:2305.07027 §4.2] [arxiv:2305.07027 tab. 2].
Card: [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](/papers/2305.07027).

HiLo splits the heads of one attention layer by spatial frequency. Some heads run
self-attention inside small non-overlapping windows, which carries fine detail. The
rest attend from every token to window-averaged keys and values, which carries the
coarse layout [arxiv:2205.13213 §4.1] [arxiv:2205.13213 eq. 6]. The LITv2 backbone uses
this layer in its last two stages. Its first two stages are feed-forward blocks with a
depthwise convolution, and that convolution's zero padding supplies the position
information, so no position embedding is used [arxiv:2205.13213 §4.2]
[arxiv:2205.13213 §4.3]. Author-reported: LITv2-S reaches 82.0% ImageNet-1K top-1 at
3.7 GFLOPs and 1,471 images per second on an RTX 3090. Plain multi-head self-attention
in the same backbone reaches 82.3% at 1,293 images per second [arxiv:2205.13213 tab. 4].
The gain is in speed and FLOPs, not accuracy [arxiv:2205.13213 tab. 4]. Card:
[Fast Vision Transformers with HiLo Attention](/papers/2205.13213).

VideoMamba keeps the plain ViT layout but replaces self-attention with bidirectional Mamba
state-space blocks, so cost grows linearly with clip length [arxiv:2403.06977 §3.1]
[arxiv:2403.06977 §3.2]. The paper's preferred scan runs spatially first, then frame by frame
[arxiv:2403.06977 fig. 7]. A smaller trained copy distils into the larger model, which the authors
say avoids the overfitting that the pure Mamba model shows at scale [arxiv:2403.06977 §3.3]
[arxiv:2403.06977 §4.1]. Author-reported: 84.0% ImageNet-1K top-1 at 576 px [arxiv:2403.06977 tab. 2],
83.3% on Kinetics-400 at 64×384 px [arxiv:2403.06977 tab. 3] and 68.4% on Something-Something V2
at 16×288 px [arxiv:2403.06977 tab. 4]. Card: [VideoMamba: State Space Model for Efficient Video Understanding](/papers/2403.06977).

HaloNet replaces the convolutions of a ResNet-50 layout with local two-dimensional
self-attention. The blocked form cuts the image into b × b query blocks and gathers one
(b + 2h) × (b + 2h) memory per block, so each window is not centred on its pixel and no
attention mask is needed [arxiv:2103.12731 §3.2]. A strided attention layer replaces the
attention-then-pool downsampling of earlier designs [arxiv:2103.12731 §3.2]. Author-reported:
H7 reaches 84.9% ImageNet top-1 when trained from scratch [arxiv:2103.12731 §4.1]. In Mask
R-CNN on COCO, two halo layers in place of the last three convolutions of ResNet-50 add 0.7
box AP over the same baseline [arxiv:2103.12731 tab. 5]. Pure attention models train more
slowly than matched EfficientNets on the authors' TPUs [arxiv:2103.12731 §5]. The card finds
no official code release. Card: [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](/papers/2103.12731).

### Hierarchical and shifted-window backbones

Swin Transformer builds a four-stage pyramid. Self-attention runs inside non-overlapping
M×M windows, with M = 7 by default, and every second block shifts the window grid so that
neighbouring windows exchange information [arxiv:2103.14030 §3.2]. Patch merging halves the
resolution between stages [arxiv:2103.14030 §3.1]. Author-reported: 81.3% ImageNet-1K top-1
for Swin-T at 224 px [arxiv:2103.14030 tab. 1], 58.7 box AP on COCO test-dev with a Swin-L
HTC++ detector [arxiv:2103.14030 tab. 2], and 53.5 mIoU on ADE20K validation with UperNet
[arxiv:2103.14030 tab. 3]. Without shifting, the Swin-T ablation drops from 81.3 to 80.2
top-1 [arxiv:2103.14030 tab. 4]. The released code has a window clamp in the last stage that
the paper does not mention; see [the Swin card](/papers/2103.14030).

Swin Transformer V2 scales the design to about 3 billion parameters and 1,536-pixel inputs
[arxiv:2111.09883 §3.5] [arxiv:2111.09883 §4.2]. It makes three changes. Residual post-norm
with scaled cosine attention keeps training stable [arxiv:2111.09883 §3.2]. A log-spaced
continuous position bias, produced by a small network, lets a bias learned at one window
size serve another [arxiv:2111.09883 §3.3]. SimMIM pretraining reduces the labelled data
needed [arxiv:2111.09883 §3.4]. Author-reported for the 3B model: 84.0% ImageNet-V2 top-1
[arxiv:2111.09883 tab. 2], 63.1 box AP on COCO test-dev [arxiv:2111.09883 tab. 3] and 59.9
mIoU on ADE20K validation [arxiv:2111.09883 tab. 4]. The 70M-image pretraining set is not
public and the G-size configs are not in the repository [arxiv:2111.09883 app. A2]. Card:
[Swin Transformer V2](/papers/2111.09883).

PVT v2 keeps a four-stage pyramid without shifted windows. Its keys and values are
average-pooled to a fixed 7×7 grid, so attention cost grows linearly with the number of
image tokens [arxiv:2106.13797 §3.2] [arxiv:2106.13797 eq. 2]. It also uses overlapping
patch embedding [arxiv:2106.13797 §3.3] and a 3×3 depthwise convolution in the
feed-forward block, which gives position information without a fixed position table
[arxiv:2106.13797 §3.4]. Author-reported: 83.8% ImageNet-1K top-1 for B5 at 224 px
[arxiv:2106.13797 tab. 2] and 46.1 RetinaNet AP on COCO for B4 [arxiv:2106.13797 tab. 3].
The authors present it as a stronger baseline set rather than a new idea. Card:
[PVT v2](/papers/2106.13797).

Pooling-based Vision Transformer (PiT) adds two pooling layers to a plain ViT. The token
grid shrinks in three stages, 27, 14 and 7 pixels square for the small models, while the
channel count grows with the head count [arxiv:2103.16302 §3.2] [arxiv:2103.16302 tab. 1].
Each pooling step applies a depth-wise convolution to the spatial tokens and a linear map to
the class token [arxiv:2103.16302 §3.2]. Author-reported under the DeiT recipe: PiT-S reaches
80.9% ImageNet-1k top-1 at 2.9 B FLOPs, against 79.8% for ViT-S at 4.6 B [arxiv:2103.16302
tab. 2]. The text and the released code disagree on the patch size of the base model, and
Table 1 matches the code [arxiv:2103.16302 §3.4] [arxiv:2103.16302 tab. 1]. Card:
[Rethinking Spatial Dimensions of Vision Transformers](/papers/2103.16302).

P2T keeps global attention but shortens the keys and values with a pyramid of
average-pooled maps, not one pooled map. The four pooling ratios are 12, 16, 20 and
24 at the first attention stage and smaller at later stages [arxiv:2106.12011 §III-C].
Each pooled map gets a shared 3×3 depthwise convolution, and the feed-forward block is
an inverted bottleneck with a depthwise convolution [arxiv:2106.12011 §III-B.1]
[arxiv:2106.12011 §III-B.2]. Author-reported: P2T-Small reaches 82.4% ImageNet-1K top-1
at 3.7 GFLOPs, against 82.0% for PVTv2-B2 at 4.0 GFLOPs [arxiv:2106.12011 tab. 2], and
46.7 mIoU on ADE20K with Semantic FPN [arxiv:2106.12011 tab. 3]. In a 100-epoch ablation,
the four pooling scales lift top-1 from 70.6% for a single ratio of 24 to 74.7%
[arxiv:2106.12011 tab. 5]. The text reports smaller detection margins than the table
gives, so the table values are the ones to use [arxiv:2106.12011 tab. 4]. The card records
the official code as non-commercial, so these numbers come from the paper alone. Card:
[P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011).

### Relative position terms in attention

Instead of adding one position vector to each patch, iRPE adds a learned term to every
attention logit, indexed by the offset between two patches [arxiv:2107.14222 §3.2]
[arxiv:2107.14222 eq. 13]. Offsets are mapped to a fixed set of buckets. Small offsets keep
their own bucket and larger ones share coarser, log-spaced buckets [arxiv:2107.14222 eq. 18].
The paper compares four ways to map a two-dimensional offset to a bucket. On DeiT-S with
50 buckets, the Product mapping in contextual mode, which uses one bucket per pair of row
and column buckets, reaches 80.9% top-1 against 79.9% for the learned absolute baseline
[arxiv:2107.14222 tab. 1]. Adding the term to the queries and values as well gives 81.4%
[arxiv:2107.14222 tab. 5]. Sharing one table across heads costs little in contextual mode:
80.89% against 80.99% with separate tables [arxiv:2107.14222 tab. 2]. Computing the query
term per bucket, not per pair, brings its cost to O(nkd) for n patches and k buckets
[arxiv:2107.14222 §3.2] [arxiv:2107.14222 eq. 28]. For DETR, a 9×9 contextual table in the
encoder, kept alongside the sinusoidal absolute encoding, raises AP from 39.5 to 40.8 at 150
epochs and from 40.6 to 42.3 at 300 epochs [arxiv:2107.14222 tab. 6]. Card:
[Rethinking and Improving Relative Position Encoding for Vision Transformer](/papers/2107.14222).

### Small models and distillation

TinyViT targets small hierarchical transformers and makes large-scale pretraining cheap
for them. A large teacher is run once; for each augmented view only the top-K
probabilities and a 4-byte seed are stored, and the seed replays the augmentation during
student training [arxiv:2207.10666 §3.1]. Distillation adds about 2 points for small
ImageNet-21k-pretrained models, where plain pretraining added under 1 point
[arxiv:2207.10666 tab. 2]. The label storage is large: 481 GB for ImageNet-21k with
K = 100 [arxiv:2207.10666 §5.2]. The 21M model reaches 84.8% top-1 at 224 px after
distillation [arxiv:2207.10666 tab. 4]. Card: [TinyViT](/papers/2207.10666).

Token labeling adds a dense loss on top of the class loss: every patch token is trained
against its own label from a pretrained ImageNet-only NFNet-F6 classifier, and the
dense label maps are computed once before training [arxiv:2104.10858 §3.2]
[arxiv:2104.10858 §4.1]. MixToken, a CutMix-style mix of patch embeddings, is used with it
[arxiv:2104.10858 §3.3]. Storing the full label maps would be about 2 TB, so the authors
keep the five largest values per cell, which they put at about 10 GB [arxiv:2104.10858 §4.1].
Author-reported: 84.4% ImageNet top-1 for LV-ViT-S at 26M parameters, and 86.4% for LV-ViT-L
at 512 px [arxiv:2104.10858 tab. 4]. Card: [All Tokens Matter](/papers/2104.10858).

IAT is a transformer-style network for image enhancement with about 0.09M parameters
[arxiv:2205.14871 tab. 1]. Ten learned queries, with no self-attention among them, read a
low-resolution encoding of the image and predict a 3×3 colour matrix and a gamma
[arxiv:2205.14871 §3.2]. On LOL-V2-real it reports 23.50 PSNR against 22.86 for MAXIM,
but on LOL-V1 MAXIM is higher on PSNR (23.43 against 23.38)
[arxiv:2205.14871 tab. 1]. Card: [You Only Need 90K Parameters to Adapt Light](/papers/2205.14871).

### Supervised pretraining on ImageNet-21k

ImageNet-21K-P is a cleaned ImageNet-21k. Classes with fewer than 500 images are removed,
50 images per class are held out for validation, and every image is resized to 224 px. The
set falls from about 1.3 TB to 250 GB [arxiv:2104.10972 §2.1]. Its pretraining, semantic
softmax, uses one softmax per WordNet depth. Each image trains every depth at which one of its
ancestor labels exists, and the depth losses are reweighted [arxiv:2104.10972 §3.3]
[arxiv:2104.10972 eq. 3]. Author-reported: this beats ImageNet-1k pretraining in all 30
model-dataset cells tested [arxiv:2104.10972 tab. 4]. For ViT-B-16 and Mixer-B-16 it also beats
the official ImageNet-21k weights in all 14 cells [arxiv:2104.10972 tab. 3]. The paper's
distillation variant adds a further gain on each dataset [arxiv:2104.10972 tab. 2], but the
released code has no training path for it, and the depth weights in the code are not those in
the printed equation. See [the card](/papers/2104.10972) before reproducing those numbers.
Card: [ImageNet-21K Pretraining for the Masses](/papers/2104.10972).

### Self-supervised pretraining

SimMIM hides random 32×32 pixel blocks, replaces the hidden patch embeddings with one
learned mask token, and regresses the pixels of the hidden area with a single linear
layer and an L1 loss [arxiv:2111.09886 §3.2] [arxiv:2111.09886 §3.4]. The authors
find that computing the loss on the hidden pixels only works better than computing it
on the whole image [arxiv:2111.09886 tab. 4]. ViT-B reaches 83.8% top-1 after
fine-tuning on ImageNet-1K [arxiv:2111.09886 tab. 6], but its linear-probe accuracy is
56.7%, and the authors say the features target fine-tuning rather than linear evaluation
[arxiv:2111.09886 §3.5]. Card: [SimMIM](/papers/2111.09886).

MAE also reconstructs masked pixels, but it drops the masked patches before the encoder,
so the encoder never sees mask tokens, and a small decoder fills the gaps
[arxiv:2111.06377 §3]. Putting mask tokens into the encoder gives 84.2% fine-tuning
and 59.6% linear probing, against 84.9% and 73.5% without them, at 3.3× the FLOPs
[arxiv:2111.06377 tab. 1]. Fine-tuning ViT-H at 448 px reaches 87.8% with ImageNet-1K
data only [arxiv:2111.06377 tab. 3]. Linear probing stays weak: 75.8% for ViT-L
[arxiv:2111.06377 app. B]. Card: [Masked Autoencoders Are Scalable Vision Learners](/papers/2111.06377).

MSN does not reconstruct pixels. A randomly masked view is trained to match the prototype
assignments of an unmasked view produced by an EMA teacher [arxiv:2204.07141 §3].
Random masking applies only to the global anchor, and small focal crops are added
[arxiv:2204.07141 §5.4]. With one labelled image per class, ViT-B/16 reaches 49.8%
[arxiv:2204.07141 tab. 1], and with 1% of ImageNet labels ViT-B/4 reaches 75.7%
[arxiv:2204.07141 tab. 2]. Card: [Masked Siamese Networks for Label-Efficient Learning](/papers/2204.07141).

iBOT replaces the pretrained tokenizer of earlier masked-image methods with an online
one. A teacher, updated as a moving average of the student, sees the unmasked views, and
its per-patch distributions are the targets for the masked positions. A second loss
matches the summary token across two views [arxiv:2111.07832 §2.2] [arxiv:2111.07832 §3.1].
Its ViT-L/16 pretrained on ImageNet-22K reaches 82.3% linear accuracy
[arxiv:2111.07832 tab. 1]. On ADE20K with a UperNet head, ViT-B reaches 50.0 mIoU against
46.8 for DINO [arxiv:2111.07832 tab. 6]. For ViT-S/16 with multi-crop, the paper reports 193.4 h
of pre-training against 180.0 h for DINO, and a higher peak memory [arxiv:2111.07832 tab. 19]. Card:
[iBOT](/papers/2111.07832).

Voltron is a masked autoencoder over video frames that also reads the paired caption
[arxiv:2302.12766 §1] [arxiv:2302.12766 §3]. The caption either conditions pixel
reconstruction or becomes a target for caption generation, and one probability decides
which role each example gets. The variants fix that probability at 0 (VCond, VDual) or 0.5
(VGen) [arxiv:2302.12766 §3.2]. Card: [Language-Driven Representation Learning for Robotics](/papers/2302.12766).

Generalized category discovery applies a self-supervised ViT-B/16 to a mixed set of
labelled and unlabelled images. The representation is tuned with contrastive losses, and
the final [CLS] features are clustered with a semi-supervised k-means whose number of
clusters is picked on the labelled subset [arxiv:2201.02609 §3.1] [arxiv:2201.02609 §3.2].
Reported "All" accuracy is 91.5 on CIFAR-10 and 74.1 on ImageNet-100 [arxiv:2201.02609 tab. 2].
Card: [Generalized Category Discovery](/papers/2201.02609).

DINO trains a ViT without labels. A student matches the softmax output of a teacher that
is an exponential moving average of the student, across several crops, and the teacher
sees only the two global crops [arxiv:2104.14294 §3.1]. The paper uses no contrastive term
and no predictor for the ViT case; centring and sharpening of the teacher output stop the
outputs from collapsing [arxiv:2104.14294 §3.1]. Collapse depends on both operations
[arxiv:2104.14294 §5.3]. Author-reported k-NN accuracy on ImageNet is 78.3% for ViT-S/8 and
76.1% for ViT-B/16, with linear accuracy of 78.2% for ViT-B/16 [arxiv:2104.14294 tab. 2].
Card: [Emerging Properties in Self-Supervised Vision Transformers](/papers/2104.14294).

data2vec applies one recipe to images, speech and text. An EMA teacher encodes the unmasked
input, and the normalised outputs of its top K blocks, averaged, form the target at each
position; a student that sees the masked input regresses those targets at the masked
positions [arxiv:2202.03555 §3.3] [arxiv:2202.03555 §3.4]. Input encoders and masks differ
by modality [arxiv:2202.03555 §3.1] [arxiv:2202.03555 §3.2]. For ImageNet-1K, the paper
reports 84.2% top-1 for ViT-B and 86.6% for ViT-L, both with ImageNet-1K pretraining
[arxiv:2202.03555 tab. 1]. Card: [data2vec](/papers/2202.03555).

CAE splits masked-image pretraining into three parts. The encoder sees only the visible
patches; a cross-attention regressor predicts the encoder representations of the hidden
patches; a decoder predicts discrete tokens from those predictions [arxiv:2202.03026 §3.1].
The alignment target is the encoder's output on the hidden patches, with gradient stopped.
The loss is the token cross-entropy plus a weighted mean-squared alignment term, with
weight 2 [arxiv:2202.03026 §3.2] [arxiv:2202.03026 eq. 1]. The default mask hides 98 of the
196 patches [arxiv:2202.03026 §3.2]. Author-reported for ViT-B at 1600 epochs: 83.9% fine-tuning,
70.4% linear probing and 77.1% attentive probing [arxiv:2202.03026 tab. 1], against 67.8% and
74.2% for MAE in the same table. In the ablation, the alignment term alone lifts linear probing
from 60.3% to 62.0% [arxiv:2202.03026 tab. 5]. The authors say training collapses without the
reconstruction term [arxiv:2202.03026 §5.5]. The repository has no licence file, so the card
takes the recipe from the paper. Card: [Context Autoencoder for Self-Supervised Representation Learning](/papers/2202.03026).

SatMAE applies masked-autoencoder pretraining to satellite time series and multi-band
images. It adds a year, month and hour encoding for each acquisition, masks each image of a
sequence independently, and splits the bands into three spectral groups, each with its own
patch embedding and a group encoding [arxiv:2207.08051 §4.1] [arxiv:2207.08051 §4.2]
[arxiv:2207.08051 eq. 3]. Author-reported: 81.49% top-1 on fMoW temporal, with the authors
noting that this figure uses nine random sequences and test augmentation
[arxiv:2207.08051 §5.3] [arxiv:2207.08051 app. A.3.2]; 77.84% top-1 on fMoW RGB after
fine-tuning [arxiv:2207.08051 §5.2]. Card: [SatMAE](/papers/2207.08051).

VideoMAE V2 scales masked video pretraining to a ViT-g backbone of about one billion
parameters [arxiv:2303.16727 §4.1]. Dual masking keeps about 10% of the cubes for the encoder
and lets the decoder reconstruct a running-cell subset that keeps half of them
[arxiv:2303.16727 §3.2] [arxiv:2303.16727 tab. 10]. Training runs in three stages:
unlabelled clips, a labelled Kinetics mix, then the target set [arxiv:2303.16727 §3.3]
[arxiv:2303.16727 §4.2]. Author-reported: 90.0% top-1 on Kinetics-400, 77.0% on
Something-Something V2 and 68.7% on Something-Something V1 [arxiv:2303.16727 tab. 6]. The
authors state that their earlier ViT-g VideoMAE run took more than two weeks on 64 A100 GPUs
[arxiv:2303.16727 §1]. Card: [VideoMAE V2](/papers/2303.16727).

Contrastive pretraining of a ViT is harder to get right than the same recipe on a
convolutional network. MoCo v3 trains a ViT with a momentum encoder and two random views per
image [arxiv:2104.02057 §3]. The authors report that training can be unstable, with mild
losses of 1 to 3% that do not show as divergence [arxiv:2104.02057 §1]. Keeping the random
patch projection fixed, with no gradient, makes training steadier and more accurate in the
settings tested [arxiv:2104.02057 §4.2]. Author-reported linear probing on ImageNet-1K:
76.7% for ViT-B/16 [arxiv:2104.02057 tab. 1], and 81.0% for a ViT-BN-L/7 variant that is not
in the released code [arxiv:2104.02057 §6.3]. Card: [An Empirical Study of Training
Self-Supervised Vision Transformers](/papers/2104.02057). The card records the official
code under a non-commercial licence, so take the recipe from the paper and its tables, not
the repository.

### Frozen self-supervised features as dense descriptors

Deep ViT features use the keys of a self-supervised DINO ViT as dense patch descriptors,
with no training. Co-segmentation clusters the patches of all images together and keeps
the clusters that a saliency map marks as foreground; correspondence keeps mutual
nearest-neighbour pairs [arxiv:2112.05814 §3] [arxiv:2112.05814 §4]. Keys are the best
of the four facets in the paper's ablation: 79.5 J_m on PASCAL-CO against 69.2 for
tokens and 49.2 for values [arxiv:2112.05814 tab. 3]. Its co-segmentation result on
PASCAL-CO is 79.5 against 75.8 for the best unsupervised baseline in the table
[arxiv:2112.05814 tab. 2]. Card: [Deep ViT Features as Dense Visual Descriptors](/papers/2112.05814).

STEGO keeps a DINO ViT-B frozen and trains a small head. The head's code correlations are
pushed to follow the correlations of the frozen features, with a clamp at zero and a
spatial centring step [arxiv:2203.08414 §3.2]. Codes are then clustered and refined with
a CRF. On CocoStuff-27 it reports 28.2 unsupervised mIoU, against 14.4 for PiCIE+H
[arxiv:2203.08414 §4.2]. Its ablation gives 24.5 for ViT-S against 28.2 for ViT-B
[arxiv:2203.08414 §4.3]. Card: [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](/papers/2203.08414).

Splice uses a frozen DINO ViT-B/8 as a judge rather than as a feature bank. A small U-Net
is trained from scratch on one structure and appearance image pair, with no adversarial
loss [arxiv:2201.00424 §3]. The appearance loss matches the deepest class token, and the
structure loss matches the self-similarity of the deepest keys [arxiv:2201.00424 §3.2]
[arxiv:2201.00424 §3.3]. Author-reported: in two-alternative user studies, outputs are
preferred over STROTSS on Wild-Pairs in 79.0% of cases [arxiv:2201.00424 §4.1]
[arxiv:2201.00424 tab. 1]. Card: [Splicing ViT Features for Semantic Appearance Transfer](/papers/2201.00424).

### Per-image transformer head for depth

AdaBins predicts depth as a weighted sum of bin centres, and the bin widths are chosen
for each image. A small transformer, the mini-ViT, cuts the decoded features into
16×16 patch tokens and outputs both the bin widths and the per-pixel range-attention maps
[arxiv:2011.14141 §3.3] [arxiv:2011.14141 tab. 1]. Depth at each pixel is the sum of bin
centres weighted by a softmax over the bins [arxiv:2011.14141 §3.3] [arxiv:2011.14141 eq. 3].
Author-reported: delta1 of 0.903 on NYU-Depth-v2 [arxiv:2011.14141 tab. 2] and RMS of 2.360
on KITTI [arxiv:2011.14141 tab. 3]. The released code uses a learned positional table of
500 rows and differs from the paper in several training values, so the card lists the
differences and the sequence-length limit. Card: [AdaBins: Depth Estimation using Adaptive Bins](/papers/2011.14141).

### Windowed and cost-volume attention for depth and flow

NeW CRFs refines a monocular depth estimate with a fully connected CRF, but only inside
non-overlapping 7×7-patch windows, and the windows are shifted every other step
[arxiv:2203.01502 §3.1] [arxiv:2203.01502 §3.2]. The pairwise potential is multi-head
attention [arxiv:2203.01502 §3.3], on a Swin encoder [arxiv:2203.01502 §3.4]. Author-reported: KITTI Eigen Abs Rel 0.052 [arxiv:2203.01502 tab. 1]
and NYUv2 Abs Rel 0.095 [arxiv:2203.01502 tab. 3]. Card: [NeW CRFs](/papers/2203.01502).

VAT, the Volumetric Aggregation Transformer, treats few-shot segmentation as cost
aggregation over the correlation volume between query and support features
[arxiv:2207.10866 §3]. A 4D convolutional Swin-style transformer refines the volume from
coarse to fine, and a decoder that looks at the query appearance sharpens the mask
[arxiv:2207.10866 §3]. Author-reported: 90.3 one-shot mIoU on FSS-1000 with ResNet-101,
against 86.5 for HSNet [arxiv:2207.10866 tab. 1] [arxiv:2207.10866 tab. 2]
[arxiv:2207.10866 tab. 3]. Card: [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](/papers/2207.10866).

FlowFormer applies a transformer to the all-pairs cost volume of an optical-flow pair, after
compressing it. Each source pixel's cost map becomes K = 8 latent tokens of width 128
[arxiv:2203.16194 §4.1]. Three alternate-group layers mix the tokens within one pixel and
across pixels [arxiv:2203.16194 §3.2], and a recurrent decoder cross-attends from a 9×9 window
around each pixel's current match to that memory [arxiv:2203.16194 §3.3]. Author-reported
Sintel test AEPE is 1.16 for clean and 2.09 for final, against 1.39 and 2.47 for GMA with warm
start [arxiv:2203.16194 tab. 1]. In an ablation under a RAFT-style schedule, three layers lower
Sintel clean error from 1.53 to 1.10 [arxiv:2203.16194 tab. 2]. The paper gives no loss formula,
optimiser or number of recurrent steps, so implementations take those from the released code,
and the code differs from the paper in several places [arxiv:2203.16194 §4.1]. Card:
[FlowFormer](/papers/2203.16194).

### Metric depth and surface normals from one image

Metric3D v2 predicts metric depth and per-pixel surface normals from one RGB image, given the
focal length [arxiv:2404.15506 §3.1]. Its encoders are ViTs pretrained with DINOv2 and registers,
with DPT-style decoders [arxiv:2404.15506 §4]. Training images or depth labels are rescaled to one
canonical focal length of 1000 px, and predictions are rescaled back at inference
[arxiv:2404.15506 §3.2]. Depth and normals are refined jointly with recurrent updates, and a
depth-normal consistency term lets normals learn from depth-only data [arxiv:2404.15506 §3.3]
[arxiv:2404.15506 §3.4]. Author-reported zero-shot AbsRel on NYUv2 is 0.067 for ViT-g and 0.063 for
ViT-L, and on KITTI 0.051 and 0.052 [arxiv:2404.15506 tab. 1]. Card:
[Metric3D v2](/papers/2404.15506).

### Sparse experts and analysis of the attention

Vision MoE replaces every other MLP of a ViT with a set of 32 experts and routes each
patch to its top-2 experts [arxiv:2106.05974 §2.2] [arxiv:2106.05974 §2.3]. A 14.7B
parameter model reaches 90.35% ImageNet top-1 after fine-tuning
[arxiv:2106.05974 tab. 2]. Batch Prioritized Routing is applied at inference to a model
trained with ordinary routing: it orders patches by their largest router weight, so the
compute budget can be cut after training. The authors report that a V-MoE-H/14 model trained with
ordinary routing can beat the dense ViT-H/14 while using about half its FLOPs
[arxiv:2106.05974 §4.2]. Card:
[Scaling Vision with Sparse Mixture of Experts](/papers/2106.05974).

How Do Vision Transformers Work? argues that self-attention acts as a learned spatial
smoothing that flattens the loss landscape, and that plain ViTs struggle on small data
because of optimisation rather than overfitting [arxiv:2202.06709 §2] [arxiv:2202.06709 §3].
It builds a hybrid, AlterNet, with local attention blocks at the end of CNN stages. Most
accuracy numbers are only in figures; the one given in text is a 2.7-point gain from
global average pooling instead of the class token on CIFAR-100 [arxiv:2202.06709 §2].
Card: [How Do Vision Transformers Work?](/papers/2202.06709).

Transformer Interpretability Beyond Attention Visualization gives a class-specific relevance
map for a ViT instead of reading the attention weights. Relevance is propagated back through
the attention products, the skip connections and the linear layers, and the block matrices are
multiplied together, as in attention rollout [arxiv:2012.09838 §3.1] [arxiv:2012.09838 §3.2]
[arxiv:2012.09838 §3.3]. On ImageNet with ViT-base, the negative-perturbation AUC for the
predicted class is 54.16 against 53.1 for rollout, and the positive-perturbation AUC is 17.03
against 20.05 [arxiv:2012.09838 tab. 1]. The released code differs from the paper in the
rollout range and in the matrix-product normalisation, which the card lists.
Card: [Transformer Interpretability Beyond Attention Visualization](/papers/2012.09838).

MambaVision is a hybrid backbone with two convolutional stages and two stages that mix
a Mamba-style state-space block with self-attention. The authors put self-attention in
the last half of the layers of each late stage. This is the best of the placements they
test, at 82.3% on ImageNet-1K, against 81.5% when attention fills the first half of the
layers instead [arxiv:2407.08083 tab. 5]. The Tiny model reports 6,298 images per second
on an A100 [arxiv:2407.08083 tab. 1]. The official code is non-commercial, so this page
describes the design from the paper text only [arxiv:2407.08083 §3.2.2]. Card:
[MambaVision](/papers/2407.08083).

### Vision-language and multimodal models

SigLIP trains a ViT image tower and a text tower with an independent sigmoid loss over
every image-text pair instead of a batch softmax. The authors report that sigmoid is
clearly better at small batch sizes and that the gain saturates near 32k pairs per batch
[arxiv:2303.15343 §4.1] [arxiv:2303.15343 tab. 8]. A B/16 model trained from scratch
reaches 73.4% zero-shot ImageNet accuracy after 5 days on 32 TPUv4 chips
[arxiv:2303.15343 tab. 1]. Card: [Sigmoid Loss for Language Image Pre-Training](/papers/2303.15343).

LLaVA-1.5 keeps a frozen CLIP ViT-L/14 at 336 px, connects it to a language model with
a two-layer MLP, and trains the connector and then the language model on a public
instruction mix [arxiv:2310.03744 §3.3]. The encoder has 24 by 24 patch tokens at 336 px
(576 tokens), a count derived from the encoder name rather than stated in the paper.
The ablation attributes much of the short-answer gain to adding academic VQA data
together with explicit answer-format prompts [arxiv:2310.03744 tab. 2]. Card:
[Improved Baselines with Visual Instruction Tuning](/papers/2310.03744).

Video-ChatGPT feeds each frame through a frozen CLIP ViT-L/14, pools the frame features
into temporal and spatial tokens and trains a single linear projection into a Vicuna
language model [arxiv:2306.05424 §3.1] [arxiv:2306.05424 §3.2]. Its benchmark uses a
GPT-3.5 judge, which is a closed model, and the reported correctness score is 2.40 on a
1–5 scale [arxiv:2306.05424 tab. 1]. Card:
[Video-ChatGPT](/papers/2306.05424).

Florence-2 uses a DaViT image encoder and a BART-style encoder-decoder. It takes an image
and a text task prompt and writes the answer as text, with boxes and polygons written as
quantised location tokens [arxiv:2311.06242 §3]. The large model reports 135.6 CIDEr
zero-shot on COCO captions [arxiv:2311.06242 tab. 4]. Card:
[Florence-2](/papers/2311.06242).

PaLI casts every task as image and text in, text out, with no task heads; the prompt selects
the task [arxiv:2209.06794 §3.1]. A ViT reads the image, and the text side starts from mT5
[arxiv:2209.06794 §3.1]. The largest model pairs a 4B-parameter ViT (ViT-e) with mT5-XXL
[arxiv:2209.06794 §4]. Author-reported: 149.1 CIDEr on COCO Karpathy test [arxiv:2209.06794 tab. 1]
and 53.6 CIDEr on the 35-language Crossmodal-3600 average [arxiv:2209.06794 tab. 2]. The
pretraining data (WebLI) is not released, and the card flags an unclear place where the visual
tokens enter the encoder [arxiv:2209.06794 §3.1]. Card: [PaLI](/papers/2209.06794).

Multimodal-CoT splits reasoning into two fine-tuned T5 models. The first writes a rationale
from the text and the image; the second answers from the text and that rationale
[arxiv:2302.00923 §4.1]. The image enters as frozen ViT-L/32 patch features, 145 tokens of
width 1024, projected to the model width and attended to by the text tokens
[arxiv:2302.00923 §4.2] [arxiv:2302.00923 tab. 12]. Author-reported ScienceQA test accuracy is
85.31% for the Base model and 90.45% for the Large one [arxiv:2302.00923 tab. 4]. Without the
picture features, the two-stage text-only model reaches 78.57% [arxiv:2302.00923 tab. 3]. The
card lists conflicts between the paper and its released scripts on learning rate, epochs and
input length. Card: [Multimodal Chain-of-Thought Reasoning in Language Models](/papers/2302.00923).

ChartQA is a benchmark of questions about real charts, with human-written and
machine-generated questions [arxiv:2203.10244 §1] [arxiv:2203.10244 tab. 1]. Its strongest
baseline, VisionTaPas, encodes the chart image with a ViT-Base encoder next to a TaPas table
model [arxiv:2203.10244 §4.2]. With the gold table it reaches 61.84% on the test split, and with
a table extracted from the chart image it reaches 45.52% [arxiv:2203.10244 tab. 5]. Card:
[ChartQA](/papers/2203.10244).

SLIP adds a SimCLR-style image-only loss, with weight 1.0, to CLIP's image-text contrastive
loss, and both losses share one ViT image encoder [arxiv:2112.12750 §3.3] [arxiv:2112.12750 alg. 1].
On YFCC15M for 25 epochs, ViT-B/16 reaches 42.8% zero-shot and 72.1% linear ImageNet accuracy,
against 37.6% and 66.5% for the authors' CLIP run [arxiv:2112.12750 tab. 2]. A control that
fine-tunes CLIP from SimCLR weights reaches 41.1% and 68.2%, below SLIP [arxiv:2112.12750 tab. 7].
The image-only branch adds about three times the image activations per step [arxiv:2112.12750 §3.3].
The CLIP baseline is the authors' own re-implementation with an adjusted augmentation recipe
[arxiv:2112.12750 §4]. Card: [SLIP: Self-supervision meets Language-Image Pre-training](/papers/2112.12750).

METER is a fully transformer vision-language model trained end to end, with no region
detector. A ViT or Swin image encoder and a RoBERTa text encoder are joined by six co-attention
layers per stream [arxiv:2111.02387 §3.1] [arxiv:2111.02387 §3.3]. Pre-training uses masked
language modeling and image-text matching on about 4M images [arxiv:2111.02387 §3.3]
[arxiv:2111.02387 app. A]. Adding a masked-patch objective with in-batch negatives lowers the
ablation's VQAv2 test-dev score from 74.98 to 74.01 [arxiv:2111.02387 tab. 7]. Author-reported
VQAv2 test-std accuracy is 77.64 for the CLIP-ViT version with about 4M images, against 76.60 for
VinVL [arxiv:2111.02387 tab. 8], and 80.54 for a 14M-image version with a larger backbone
[arxiv:2111.02387 tab. 10]. The card notes that the released configs cover the base and large
Swin and CLIP models but not the 14M run. Card: [An Empirical Study of Training End-to-End Vision-and-Language Transformers](/papers/2111.02387).

SAN keeps CLIP's ViT frozen and attaches a light side ViT: eight blocks, width 240, and 100
learnable query tokens [arxiv:2302.12242 §3] [arxiv:2302.12242 §4.2]. The side network yields
mask proposals as inner products of pixel and query embeddings, and one attention bias per CLIP
head [arxiv:2302.12242 eq. 1] [arxiv:2302.12242 eq. 2]. The biases steer CLIP's last blocks, which
classify each proposal [arxiv:2302.12242 eq. 3]. Trained on COCO-Stuff, SAN reports 27.5 mIoU on
ADE-150 with ViT-B/16 and 32.1 with ViT-L/14 [arxiv:2302.12242 tab. 2]. It has 8.4M trainable
parameters and runs at 15.2 FPS at 640² on one GPU [arxiv:2302.12242 tab. 3]. In the authors'
ablation, end-to-end training beats two-stage training by 4.5 mIoU on ADE-150
[arxiv:2302.12242 tab. 6]. The paper's ensemble rows use a model that is not in the public code,
and the released code's inference step differs from the paper's formula
[arxiv:2302.12242 tab. 2] [arxiv:2302.12242 eq. 4]. Card: [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242).

IRENE turns a chest radiograph, the words of a chief complaint, lab values, age and sex
into one token sequence. The image tokens are 16×16 patches with a class token, and the
text tokens are one per word, lab value or demographic field [arxiv:2306.00864 p. 10].
Two bidirectional blocks let image and text tokens attend within and across modalities,
and ten self-attention blocks then run over the joined sequence [arxiv:2306.00864 p. 10]
[arxiv:2306.00864 fig. 2a]. On eight pulmonary diseases the authors report a mean AUROC
of 0.924, against 0.858 for Perceiver and 0.805 for an image-only ViT
[arxiv:2306.00864 tab. 1]. On three COVID-19 outcomes the mean AUPRC is 0.592, against
0.521 for early fusion [arxiv:2306.00864 tab. 2]. Removing the radiograph drops the mean
AUROC to 0.543 [arxiv:2306.00864 tab. 3]. The method assumes every modality is present,
and the authors list missing modalities as an open problem [arxiv:2306.00864 p. 7]. The
paper points to a code release with the model and evaluation only. It has no training
loop and no pretrained weights, so the reported numbers cannot be reproduced from it
[arxiv:2306.00864 p. 12]. Card:
[A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864).

### Parameter-efficient adaptation

FacT stacks the 12 weight matrices of every block of a ViT into one tensor and describes
the change to that tensor with a shared low-rank factorisation, either Tensor-Train or
Tucker [arxiv:2212.03145 §3.1] [arxiv:2212.03145 §3.2]. Only these factors and the head are
trained, and the factors are folded back before inference [arxiv:2212.03145 §1]. On VTAB-1K,
FacT-TK at rank up to 32 averages 75.6% with 0.069M backbone parameters, against 75.5% for
NOAH with 0.361M [arxiv:2212.03145 tab. 1]. The reported numbers are on the test split
[arxiv:2212.03145 §4.1]. Card: [FacT](/papers/2212.03145).

BAT adapts a frozen dual-stream ViT tracker for RGB-thermal tracking. One adapter, shared by
both streams, passes a feature prompt from each stream into the other in every layer, so
neither modality is fixed as dominant [arxiv:2312.10611 §1]. Only the adapters are trained,
0.32M parameters in total [arxiv:2312.10611 §1]. Author-reported state-of-the-art results on RGBT234
and LasHeR, against 19 baselines [arxiv:2312.10611 §4.2]. Card:
[Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611).

### Single-object tracking

SimTrack removes the Siamese branches and the transformer interaction head of a
STARK-style tracker. The exemplar tokens and the search tokens go through one ViT
backbone together, so attention in every layer mixes the two images
[arxiv:2203.05328 §3.2] [arxiv:2203.05328 eq. 5]. The exemplar tokens get their own
learned position embedding, and a 64×64 central crop of the exemplar is added as extra
input tokens [arxiv:2203.05328 §3.2] [arxiv:2203.05328 §3.3]. Those foveal tokens are
used only as input. Author-reported with the ViT-B/16 backbone: 69.3% AUC on LaSOT and
54.8% on TNL2K [arxiv:2203.05328 tab. 1]; the ViT-L/14 backbone reaches 70.5% and 55.6%
[arxiv:2203.05328 tab. 1]. On TNL2K the joint framework adds 1.3 AUC points over the
STARK-SV baseline with the same backbone, the separate position embedding adds 0.5 and
the foveal window adds 0.8 [arxiv:2203.05328 tab. 8]. Pretraining weights matter:
on LaSOT, MAE weights reach 70.3% AUC against 69.3% for the CLIP weights used in the
main models [arxiv:2203.05328 tab. 5]. Joint attention also constrains inference. The
exemplar features depend on each search frame, so they cannot be cached across frames
(derived from the joint attention) [arxiv:2203.05328 §3.2]. Restricting interaction to
half the blocks lowers TNL2K AUC from 54.8 to 52.3 [arxiv:2203.05328 tab. 10]. The
released code differs from the paper in several places, including the exemplar position
input and the schedule in its shipped configs, so check the card before reimplementing
the training setup. Card:
[Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328).

### Video, generation and prompting

VideoMAE pretrains a plain ViT-B on unlabelled video with masked autoencoding. It samples 16
frames, cuts them into 2×16×16 cubes and hides 90% of the cubes with a tube mask shared over
time, so a hidden cube cannot be copied from a neighbouring frame [arxiv:2203.12602 §3.3]
[arxiv:2203.12602 §4.2]. Only the visible cubes go through the encoder, and a 4-block decoder
reconstructs the hidden ones with an MSE loss [arxiv:2203.12602 tab. 1]. Without extra data,
author-reported ViT-B results are 81.5% top-1 on Kinetics-400 [arxiv:2203.12602 tab. 7] and
70.8% on Something-Something V2 [arxiv:2203.12602 tab. 6]. Pretraining on the unlabelled
training split of a small dataset also works, with 62.6% on HMDB51 and 91.3% on UCF101
[arxiv:2203.12602 tab. 2]. Card: [VideoMAE](/papers/2203.12602). The card records the
official code under a non-commercial licence.

ST-Adapter adds a small bottleneck module to each block of a frozen image ViT. Its
depth-wise 3D convolution mixes features across neighbouring frames, so only the adapters
and a linear head are trained [arxiv:2206.13559 §3.2] [arxiv:2206.13559 §4.1]. With CLIP
ViT-B/16 on Kinetics-400 it reports 82.0% top-1 with 7.20M trained parameters, against
81.7% for full fine-tuning with 121.57M [arxiv:2206.13559 tab. 1]. The paper's equation
includes a non-linearity that the released code omits [arxiv:2206.13559 §3.2]. Card:
[ST-Adapter](/papers/2206.13559).

X-CLIP keeps a CLIP image encoder and adds cross-frame communication. Each frame emits a message
token, and the message tokens of all frames are mixed inside every block [arxiv:2208.02816 §3.2.1].
A small temporal transformer then merges the frame vectors, and the class names are rewritten as
video-specific text prompts [arxiv:2208.02816 §3.2] [arxiv:2208.02816 §3.3.1]. Author-reported:
87.1% top-1 on Kinetics-400 with ViT-L/14 at 8 frames and 4×3 views, at 658 GFLOPs per view
[arxiv:2208.02816 tab. 1]; 83.8% with ViT-B/16 at 8 frames [arxiv:2208.02816 tab. 1]. Zero-shot,
ViT-B/16 reaches 44.6% on HMDB-51 and 72.0% on UCF-101 [arxiv:2208.02816 tab. 3]. Card:
[Expanding Language-Image Pretrained Models for General Video Recognition](/papers/2208.02816).

InternVideo trains a masked-video ViT-H and a CLIP-initialised UniFormerV2 video-text encoder
separately, then fuses the two frozen encoders with learned cross-model attention
[arxiv:2212.03191 §3.1] [arxiv:2212.03191 §3.3]. Author-reported: 91.1% top-1 on Kinetics-400 and
77.2% on Something-Something V2 [arxiv:2212.03191 tab. 2] [arxiv:2212.03191 tab. 3]. The paper gives
the fusion learning rate as 5×10^5, with no exponent that a training run could use
[arxiv:2212.03191 §4.2]. The [card](/papers/2212.03191) records that the released repository does
not contain the fusion stage, so the fused numbers cannot be reproduced from it. Card:
[InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191).

UniFormerV2 starts from an image ViT that has already been trained, typically CLIP, and
adds two video modules beside each block: a local relation block over short stretches of
time, and a global block in which one learned query gathers a clip-level summary by
cross-attention [arxiv:2211.09552 §3.1] [arxiv:2211.09552 §3.2] [arxiv:2211.09552 §3.3].
Author-reported: 90.0% top-1 on Kinetics-400 with ViT-L/14 at 336 px, 64 frames, after
Kinetics-710 post-pretraining [arxiv:2211.09552 tab. 1]. The 8-frame setting reaches 88.8
at 8.0 TFLOPs, against 75.3 TFLOPs for the 90.0 result [arxiv:2211.09552 tab. 1]. Card:
[UniFormerV2](/papers/2211.09552).

Video Swin Transformer applies Swin's shifted windows in 3D: each window spans a few frames
as well as a patch grid, and the backbone starts from ImageNet-pretrained Swin weights
[arxiv:2106.13230 §3.1] [arxiv:2106.13230 §3.4]. Author-reported: 84.9% top-1 on
Kinetics-400 for Swin-L at 384 px, with ImageNet-21K pretraining and 10×5 views
[arxiv:2106.13230 tab. 1]. The same model reaches 84.6% with 4×3 views, so the 10×5 figure
depends on the number of views [arxiv:2106.13230 tab. 1]. Card:
[Video Swin Transformer](/papers/2106.13230).

DiT replaces the U-Net of latent diffusion with a transformer on latent patches. The timestep
and class label drive adaLN-Zero, which regresses a scale, a shift and a gate for each block and
starts each block as the identity [arxiv:2212.09748 §3.2] [arxiv:2212.09748 app. A]. With
classifier-free guidance at scale 1.50, DiT-XL/2 reports FID 2.27 at 256×256
[arxiv:2212.09748 tab. 2] and 3.04 at 512×512 [arxiv:2212.09748 tab. 3], on class-conditional
ImageNet and author-reported. Across twelve configurations, FID falls as forward-pass Gflops
rise [arxiv:2212.09748 §5] [arxiv:2212.09748 tab. 4]. Card:
[Scalable Diffusion Models with Transformers](/papers/2212.09748).

W.A.L.T is a diffusion transformer that works on a causal-autoencoder latent shared by
images and videos [arxiv:2312.06662 §4.1]. Its blocks alternate spatial windows, one
latent frame each, with spatiotemporal windows across frames [arxiv:2312.06662 §4.2].
Without classifier-free guidance it reports ImageNet 256 FID 2.56 at 460M parameters,
while VDM++ reports 2.40 at 2B parameters in the same table [arxiv:2312.06662 tab. 2].
The paper gives up to 2× speed-up for local window attention, measured in steps per
second on UCF-101 [arxiv:2312.06662 §5.2]. Card:
[Photorealistic Video Generation with Diffusion Models](/papers/2312.06662).

MAE-VQGAN uses a ViT-L masked autoencoder whose decoder predicts VQGAN codebook tokens
[arxiv:2209.00647 §3.1]. A prompt tiles an example input-output pair and a query into one
grid and inpaints the empty cell, with no weight updates [arxiv:2209.00647 §3.2]. Trained
on arXiv figures, it reports Pascal-5i mIoU of 27.83, 30.44, 26.15 and 24.25 on the four
splits, while the ImageNet-only variant reaches 2.22 to 7.07 [arxiv:2209.00647 tab. 1].
Card: [Visual Prompting via Image Inpainting](/papers/2209.00647).

FAST-VQA scores video quality with a Swin-based transformer whose input size does not grow
with frame resolution. Each frame is cut into a grid, and one raw-resolution 32 × 32 patch is
taken from every cell at the same position in all frames. The patches are spliced into one
canvas: 7 × 7 patches, 224 pixels a side, for 32 frames [arxiv:2207.02595 §3.1]
[arxiv:2207.02595 tab. 2]. The Video Swin-T backbone is changed in two places. Gated relative
position biases separate pairs inside one patch from pairs across patches, and a non-linear
regression head scores each patch before pooling [arxiv:2207.02595 §3.2]
[arxiv:2207.02595 eq. 5] [arxiv:2207.02595 eq. 7]. The model is trained end to end with a
PLCC-based loss [arxiv:2207.02595 §4.1]. The authors argue that frozen recognition features
are not tuned for quality, and they compare against a full-resolution Swin-T with fixed
features [arxiv:2207.02595 §1] [arxiv:2207.02595 tab. 3]. Author-reported: 0.814 PLCC on
LSVQ 1080p, against 0.753 for that baseline [arxiv:2207.02595 tab. 3]. FLOPs are 279G at
every resolution, against 11,852G for the baseline at 1080P [arxiv:2207.02595 tab. 4]. A
single fragment covers about 2.4% of a 1080P frame, and one sample is almost as accurate as a
six-sample ensemble [arxiv:2207.02595 §4.7] [arxiv:2207.02595 tab. 8]. Card:
[FAST-VQA](/papers/2207.02595).

### Robotics

VIMA turns each detected object into one token: its box goes through an MLP and its crop
through a small ViT trained from scratch [arxiv:2210.03094 app. C.2.1]. A pretrained T5
encoder reads the interleaved text and object prompt, and a decoder cross-attends from
trajectory tokens to the prompt tokens [arxiv:2210.03094 §4]. At 200M parameters it
reports 81.5% success on the easiest zero-shot level, against 58.0% for its Gato-style
baseline [arxiv:2210.03094 tab. 10]. In the data-scaling table, with 1% of the imitation
data, VIMA reports 36.3% on L1 against 11.5% for VIMA-Gato [arxiv:2210.03094 tab. 11].
Card:
[VIMA: General Robot Manipulation with Multimodal Prompts](/papers/2210.03094).

MEM gives a vision-language-action policy two memories [arxiv:2603.03596 §3.1]. The
short-horizon memory is a video encoder: a ViT whose every fourth layer also attends across
time, with no new learnable parameters [arxiv:2603.03596 §3.3]. The long-horizon memory is a
text summary that the high-level policy updates from its previous summary [arxiv:2603.03596 §3.2].
Integrated into pi0.6, the authors report tasks of up to 15 minutes [arxiv:2603.03596 §4.1].
All results are author-reported. Card: [MEM](/papers/2603.03596).

A ViT-B/16 imitation policy can also be steered by a slower model. In the driving paper,
a pretrained ViT-B/16 reads a cropped camera image and outputs steering and throttle for each
of three route classes, LEFT, MIDDLE and RIGHT [arxiv:2411.14256 §2.1] [arxiv:2411.14256 app. B.1].
A multimodal LLM, used without fine-tuning and prompted with a chain-of-thought template, names
one of those classes, and the policy keeps the last answer until a new one arrives
[arxiv:2411.14256 §2.2] [arxiv:2411.14256 tab. 1]. The LLM answers took 5.86 to 7.76 seconds on
the authors' hardware [arxiv:2411.14256 tab. 2]. Trained on one cone in a hallway, the policy
alone reaches 40% success on a harder course with cones and bins, and 63% to 83% with an LLM,
depending on the model, over 30 real runs each [arxiv:2411.14256 tab. 3]. These are
author-reported figures from one course and one car. The appendix says that in one configuration
the LLM supplies the steering and throttle values, which conflicts with the main text; the card
follows the main text [arxiv:2411.14256 app. B.4]. Card: [Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs](/papers/2411.14256).

### Frozen foundation encoders in specialised domains

The canopy-height work pretrains a huge ViT (606M parameters) with DINOv2 on 18 million
unlabelled satellite tiles, freezes it, and trains a DPT decoder on airborne lidar height
maps [arxiv:2304.07213 §3.1] [arxiv:2304.07213 §3.2]. It reports an average mean absolute
error of 2.8 m over three lidar test sets [arxiv:2304.07213 tab. 2]. The pretraining cost
is estimated at 16,896 GPU-hours [arxiv:2304.07213 app. D]. Card:
[Canopy height from RGB imagery](/papers/2304.07213).

Multimodal Fusion Transformer for hyperspectral classification turns a second sensor
(LiDAR, multispectral, radar or elevation) into one classification token. In each encoder
block that token alone queries the hyperspectral patch tokens [arxiv:2203.16952 §III-B2]
[arxiv:2203.16952 §III-B3]. It reports 89.80% overall accuracy on Houston with LiDAR added
[arxiv:2203.16952 tab. II]. The paper reports test-set accuracy [arxiv:2203.16952 §IV-B],
and the released notebook keeps the checkpoint with the best test accuracy, so the
reported numbers are likely optimistic. Card:
[Multimodal Fusion Transformer for Remote Sensing Image Classification](/papers/2203.16952).

FSRA matches drone photos to satellite images for geo-localisation. A ViT-S produces patch
tokens, which are ordered by mean activation and cut into three regions; each region is
pooled and aligned across views with classification and triplet losses
[arxiv:2201.09206 §3.2]. It reports 84.51% recall at 1 with three satellite repeats on
University-1652 [arxiv:2201.09206 tab. 2]. Card:
[FSRA drone-to-satellite matching](/papers/2201.09206).

Prithvi-EO-2.0 is a ViT masked autoencoder for six-band satellite time series, pretrained
on 4.2M four-timestamp samples in 300M and 600M sizes [arxiv:2412.02732 §3.1]
[arxiv:2412.02732 §3.2]. Its patch embedding uses a time depth of 1, and date and location
enter as learned-weight bias terms, with metadata dropout during pretraining
[arxiv:2412.02732 §3.2]. Fine-tuned on Sen1Floods11, the 600M-TL variant reports 90.3
mIoU [arxiv:2412.02732 §4.2.1]. For gross primary productivity it freezes the encoder and
trains a small head [arxiv:2412.02732 §4.4.2]. Pretraining took about 58,000 GPU-hours for
the 600M model [arxiv:2412.02732 §3.2]. Card: [Prithvi-EO-2.0](/papers/2412.02732).

ChangeFormer is a Siamese network for change detection between two co-registered
remote-sensing images. Its shared encoder is a four-stage hierarchical transformer, not a plain
ViT, and its attention shortens the keys and values by a reduction ratio that the paper does not
state as a number [arxiv:2201.01293 §2.1] [arxiv:2201.01293 §2.1.1]. Four difference modules
compare the two feature maps at each scale, and an MLP decoder outputs the two-class change mask
[arxiv:2201.01293 §2.1.3] [arxiv:2201.01293 §2.2]. Author-reported: F1 of 90.40 on LEVIR-CD and
86.67 on DSIFN-CD [arxiv:2201.01293 tab. 1]. The released code differs from the paper in the
decoder, the difference module and the initialisation, so the card lists the changes. Card:
[A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293).

### Convolutional and detection alternatives

InternImage is a convolutional backbone whose core operator, DCNv3, is a 3×3 deformable
convolution with learned sampling offsets, channel groups and softmax-normalised weights
[arxiv:2211.05778 §3.1]. It is included here as the strongest convolutional competitor to ViTs.
Its H variant, at 640 px, reports 89.6% ImageNet-1K top-1, against 87.5 for the closest CNN
baseline in the table, a BiT ResNet at 480 px [arxiv:2211.05778 tab. 2]. Card:
[InternImage](/papers/2211.05778).

ConvNeXt starts from a ResNet-50 and changes its design one step at a time, using
Transformer-era training recipes and layer choices, until it reaches a pure ConvNet with 7×7
depth-wise convolutions, inverted bottlenecks, LayerNorm and a single GELU per block
[arxiv:2201.03545 §2] [arxiv:2201.03545 §2.6]. The authors use it to separate the effect of
architecture from that of training recipe in the Swin comparison [arxiv:2201.03545 §1].
Author-reported: ConvNeXt-T reaches 82.1% ImageNet-1k top-1 at 224 px, against 81.3% for
Swin-T [arxiv:2201.03545 tab. 1]. ConvNeXt-XL, pre-trained on ImageNet-22k and tested at 384 px,
reaches 87.8% [arxiv:2201.03545 tab. 1]. On COCO with Cascade Mask R-CNN at the 3× schedule,
ConvNeXt-T and Swin-T both report 50.4 box AP [arxiv:2201.03545 tab. 3]. Card:
[A ConvNet for the 2020s](/papers/2201.03545).

A survey of the YOLO detectors from v1 to v8, YOLO-NAS and YOLO-with-Transformer variants
gives reported COCO AP in one summary table and in the text, and the two disagree for several
versions [arxiv:2304.00501 tab. 4] [arxiv:2304.00501 §3]. Transformer-based detectors appear as
later variants, such as YOLOS, whose 42.0 AP the survey quotes [arxiv:2304.00501 §19]. The card
recommends using it as a reading list and checking the original papers for exact settings.
Card: [A Comprehensive Review of YOLO Architectures in Computer Vision](/papers/2304.00501).

A survey of object detectors compiles author-reported COCO and VOC AP and FPS for 17
detectors, copied from the source papers rather than re-run [arxiv:2104.11892 §7]. Its
headline pairs are DetectoRS at 53.3 AP on COCO, about 4 FPS at 1333 input, and YOLOv4 at
43.0 AP and about 31 FPS at 512 input [arxiv:2104.11892 tab. 3]. It places Swin Transformer
under single-stage detectors, and that row of its table gives no input size or FPS
[arxiv:2104.11892 §5.3] [arxiv:2104.11892 tab. 3]. The compiled numbers come from different
hardware and input sizes, so they are not directly comparable [arxiv:2104.11892 §7]. Card:
[A Survey of Modern Deep Learning based Object Detection Models](/papers/2104.11892).

### Benchmarks that compare ViTs with CNNs

AQUA20 is a set of 8,171 underwater photographs in 20 classes, with a fixed split of
6,559 training and 1,612 test images [arxiv:2506.17455 tab. 1] [arxiv:2506.17455 §3.1].
Thirteen ImageNet-pretrained networks were fine-tuned on it, among them ViT-B/16, SwinV2-B
and ConvNeXt [arxiv:2506.17455 §3.2] [arxiv:2506.17455 §4.1]. ConvNeXt scores highest, at
90.69% top-1 and 88.92% macro F1. ViT-B/16 reaches 87.34% top-1, and SwinV2-B 88.65%
[arxiv:2506.17455 tab. 3]. The Grad-CAM and LIME maps are qualitative. They show the
best model on parts of the animal that matter for identification, while InceptionV3's
attention more often spreads onto the seabed [arxiv:2506.17455 §4.5]. The source images
have no stated licence, so check them before redistributing any of them
[arxiv:2506.17455 §3.1]. The numbers depend on choices the card flags: checkpoints are
chosen on the test loss, the paper states a 32×32 input while the released notebook uses
224×224, and each model was trained once [arxiv:2506.17455 §4.1] [arxiv:2506.17455 §4.2].
Card: [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455).

### Hybrid CNN and transformer branches

Conformer runs a ResNet-style convolutional branch and a ViT-style transformer branch side by
side, from a shared stem. In every block a Feature Coupling Unit passes information both ways:
convolutional features are pooled into patch tokens, and transformer tokens are upsampled back
into the convolutional branch [arxiv:2105.03889 §3.2]. The authors compare this with a serial
hybrid, a ResNet-26d or ResNet-50d followed by DeiT-S, and report 80.2% for the serial version
at similar cost [arxiv:2105.03889 §4.4] [arxiv:2105.03889 tab. 5]. Author-reported ImageNet-1k
top-1 at 224 px: Conformer-S 83.4% and Conformer-B 84.1%, against 81.8% for DeiT-B
[arxiv:2105.03889 tab. 2]. In Faster R-CNN on COCO, Conformer-S/32 reaches 43.1 box AP against
39.4 for a ResNet-101 FPN [arxiv:2105.03889 §4.3]. Card: [Conformer](/papers/2105.03889).

CvT stays a single transformer stack but adds convolutions in two places. Overlapping
convolutional token embeddings build a three-stage hierarchy [arxiv:2103.15808 §3.1], and
depthwise convolutions compute the queries, keys and values, with stride 2 on keys and values
to cut attention cost [arxiv:2103.15808 §3.2]. The model has no position embedding, because
the convolutions supply local structure [arxiv:2103.15808 tab. 5]. Author-reported ImageNet-1k
top-1 at 224 px: 81.6% for CvT-13, with about 20M parameters [arxiv:2103.15808 tab. 3]. The
ImageNet-22k pretrained CvT-W24, fine-tuned at 384 px, reaches 87.7% [arxiv:2103.15808 tab. 3].
The card notes that the paper gives two parameter counts for CvT-W24, 276.7M in tab. 2 and
298.3M in §4.1 [arxiv:2103.15808 tab. 2] [arxiv:2103.15808 §4.1]. Card: [CvT](/papers/2103.15808).

EdgeNeXt is a mobile hybrid that keeps most of its depth in convolution blocks. Its SDTA
block sits at the end of stages 2 to 4 [arxiv:2206.10589 §3 Overall Architecture]
[arxiv:2206.10589 tab. 1]. The block first splits channels into groups that pass through
depth-wise convolutions at several scales [arxiv:2206.10589 eq. 2]. Its attention then runs
across channels: queries and keys are L2-normalised and form a channel-by-channel score
matrix, so the cost grows linearly with the number of pixels [arxiv:2206.10589 eq. 3]
[arxiv:2206.10589 eq. 4]. The authors add one positional encoding before stage 2 only, since
adding it in every stage raised latency and hurt dense prediction [arxiv:2206.10589 §3 Overall
Architecture]. Author-reported: the S variant, with 5.6M parameters and 1.30G MAdds, reaches
79.4% ImageNet-1K top-1 at 256 px, against 78.4% for MobileViT-S at 2.01G MAdds
[arxiv:2206.10589 tab. 3]. Removing the SDTA blocks lowers the score to 78.3%
[arxiv:2206.10589 tab. 7]. Card: [EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications](/papers/2206.10589).

### Text-supervised grouping for segmentation

GroupViT trains a hierarchical ViT with no mask labels. Patches are merged into larger
segments by learned group tokens and a differentiable hard assignment at each stage, and the
only training signal is an image-text contrastive loss [arxiv:2202.11094 §3.1]
[arxiv:2202.11094 §3.2]. A multi-label variant adds several short prompts per image, built
from caption nouns, as extra positives; the authors report this as a key factor for grouping
quality [arxiv:2202.11094 §3.2] [arxiv:2202.11094 tab. 1]. For zero-shot segmentation each
segment is labelled by its similarity to a label-name text embedding, so the label list can
change without retraining [arxiv:2202.11094 §3.3]. Author-reported zero-shot mIoU is 52.3 on
PASCAL VOC 2012 and 22.4 on PASCAL Context, for a model trained on CC12M and filtered YFCC
[arxiv:2202.11094 §4.4] [arxiv:2202.11094 tab. 5]. The card records the official code under a
non-commercial licence, so the implementation notes here rest on the paper text. Card:
[GroupViT](/papers/2202.11094).

### Restoration and enhancement

The U-shape Transformer for underwater enhancement keeps a U-shaped generator but replaces
its skip connections with a channel-wise transformer, CMSFFT, and puts a spatial transformer,
SGFMT, at the bottleneck [arxiv:2111.11843 §3.2]. The channel-wise block targets the colour
channels that water attenuates most [arxiv:2111.11843 §1]. Training is adversarial, with an
RGB, LAB and LCH colour loss and a perceptual term [arxiv:2111.11843 §3.3]. The paper also
releases LSUI, 4,279 real underwater pairs with reference images [arxiv:2111.11843 §3.1].
Author-reported: PSNR 24.16 and SSIM 0.93 on Test-L400, the best of ten compared methods
[arxiv:2111.11843 tab. 3]. Card: [U-shape Transformer](/papers/2111.11843).

### Multi-agent perception for driving

V2X-ViT fuses LiDAR feature maps from connected vehicles and roadside sensors for 3D vehicle
detection [arxiv:2203.10638 §3]. Its encoder stacks three layers, each with heterogeneous
multi-agent self-attention, which treats vehicles and infrastructure differently, and
multi-scale window attention [arxiv:2203.10638 §3.2] [arxiv:2203.10638 app. B.2]. A
delay-aware encoding is added at every position of each shared feature map
[arxiv:2203.10638 eq. 8] [arxiv:2203.10638 eq. 10]. The evaluation uses the authors' simulated
V2XSet, under perfect and noisy localisation and delay [arxiv:2203.10638 §4.1]
[arxiv:2203.10638 tab. 1]. In the noisy setting, AP at IoU 0.7 is 0.614, against 0.402 with no
fusion [arxiv:2203.10638 tab. 1]. Card: [V2X-ViT](/papers/2203.10638).

### Phone-latency designs

EfficientFormer is a pure transformer tuned for phone latency. Its early stages use
convolution-style pooling blocks, and only the last stage applies global multi-head
self-attention [arxiv:2206.01191 eq. 4] [arxiv:2206.01191 eq. 5]. Depth and width are chosen
by a latency-driven slimming step that uses a per-block latency table measured on the device
[arxiv:2206.01191 §4.2] [arxiv:2206.01191 alg. 1]. The target is an iPhone 12 with CoreML
[arxiv:2206.01191 §1]. Author-reported: EfficientFormer-L1 reaches 79.2% top-1 at 1.6 ms, and
L7 reaches 83.3% at 7.0 ms [arxiv:2206.01191 tab. 1]. Card: [EfficientFormer](/papers/2206.01191).

EdgeViT is a hierarchical transformer tuned for measured mobile latency. Each block
mixes neighbours with convolutions, runs attention over one sampled token per r×r window,
and spreads that context back to the window with a transposed convolution
[arxiv:2205.03436 §3.2] [arxiv:2205.03436 eq. 1]. Author-reported ImageNet-1K top-1 is
74.4%, 77.5% and 81.0% for its three sizes, at 32.8, 54.1 and 85.3 ms on a Galaxy S21
CPU [arxiv:2205.03436 tab. 2]. MobileViT-XS reaches 74.7% at 150.1 ms on the same
protocol [arxiv:2205.03436 tab. 2]. The protocol is CPU, FP32, batch 1, through
TorchScript lite, which the authors chose because it ran all their models
[arxiv:2205.03436 §4.1]. Energy was measured on a different Snapdragon device, so the
latency and energy columns cannot be converted into each other [arxiv:2205.03436 tab. 3].
Card: [EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers](/papers/2205.03436).

RepViT takes the opposite route: a pure CNN, built from MobileNetV3-L by adopting layout
choices from small ViTs. Each block separates the spatial token mixer, a re-parameterised
depthwise convolution, from the channel mixer, an FFN with expansion 2 [arxiv:2307.09283 §3.2].
A separate stride-2 block handles downsampling [arxiv:2307.09283 §3.3]. With distillation, the
reported ImageNet-1K top-1 on iPhone 12 with Core ML runs from 78.7% for M0.9 at 0.9 ms to
83.7% for M2.3 at 2.3 ms [arxiv:2307.09283 tab. 1]. Removing re-parameterisation lowers the
M0.9 score from 78.74% to 78.47% [arxiv:2307.09283 tab. 6]. Some architecture details appear
only in supplementary material, which is not in the source the card was written from. Card:
[RepViT](/papers/2307.09283).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Dense labels, ImageNet-21k checkpoint available, want a simple baseline | Segmenter with a linear head | At the largest size it is 0.6 mIoU behind the mask head on ADE20K [arxiv:2105.05633 tab. 4] |
| Dense labels, want one mask per class from learned class embeddings | Segmenter mask decoder | Gains over linear at the largest size, but is worse on large objects at the base size [arxiv:2105.05633 tab. 4] |
| Little or no pretraining data | Do not train from scratch | Scratch training is far worse in both segmentation and classification [arxiv:2105.05633 tab. 9] [arxiv:2012.15840 tab. 2] |
| Need smaller model or lower latency on phones | EfficientFormerV2 for fixed networks | Reports its own iPhone latency and accuracy [arxiv:2212.08059 tab. 2] |
| Small ViT, large pretraining set, storage budget available | TinyViT-style stored-teacher distillation | Gains of about 2 points over plain pretraining for small models [arxiv:2207.10666 tab. 2] |
| Unlabelled images, fine-tuning is the goal | SimMIM | Simple recipe, strong fine-tuning [arxiv:2111.09886 tab. 6] |
| Unlabelled images, linear probing or dense transfer | iBOT | Higher linear accuracy and ADE20K mIoU [arxiv:2111.07832 tab. 1] [arxiv:2111.07832 tab. 6] |
| Unlabelled images, fine-tuning budget available, linear probing not needed | MAE, with mask tokens kept out of the encoder | 87.8% for ViT-H at 448 px with ImageNet-1K data only [arxiv:2111.06377 tab. 3] |
| Very few labels per class, unlabelled pretraining images | MSN | 49.8% with one label per class for ViT-B/16 [arxiv:2204.07141 tab. 1] |
| Dense descriptors for co-segmentation or matching, no training budget | Keys of a frozen DINO ViT, clustered or matched | 79.5 J_m on PASCAL-CO with keys [arxiv:2112.05814 tab. 3] |
| Unlabelled pixel classes on a frozen DINO ViT-B | STEGO-style correspondence distillation | 28.2 unsupervised mIoU on CocoStuff-27 [arxiv:2203.08414 §4.2] |
| Policy that must act at sensor rate while a large model is too slow to run every step | ViT-B/16 policy with a cached LLM instruction, LLM used zero-shot | 63% to 83% success against 40% for the policy alone on the authors' harder course, 30 real runs each [arxiv:2411.14256 tab. 3]; LLM answers take 5.86 to 7.76 s [arxiv:2411.14256 tab. 2] |
| Long sequences with high resolution, memory bound | Local or windowed attention, or a hybrid | Attention memory drops with local segments [arxiv:2108.11084 tab. 4] [arxiv:2407.08083 tab. 1] |
| Attention cost limits resolution in segmentation, global mixing wanted | AFNO token mixer | 80.9 against 79.7 mIoU at 1024×1024 [arxiv:2111.13587 tab. 4] |
| Detection with a smaller backbone than PVT-S | ResT-Small | 40.3 box AP at 23.4M parameters, against 40.4 for PVT-S at 34.2M [arxiv:2105.13677 tab. 3] |
| Scale capacity without scaling per-patch compute | Sparse experts with batch prioritised routing | Reported near dense accuracy at about half the FLOPs [arxiv:2106.05974 §4.2] |
| Image-text retrieval with a large batch budget | Sigmoid pairwise loss | Better at small batch and saturates near 32k [arxiv:2303.15343 tab. 8] |
| Multimodal chat on a modest budget | Frozen CLIP ViT, MLP connector, public instruction mix | Trains in about a day on 8 GPUs [arxiv:2310.03744 §3.3] |
| Volumetric medical, boundary accuracy matters | nnFormer | Lower HD95 than nnUNet on all three datasets [arxiv:2109.03201 tab. 5] |
| Satellite heights from RGB, lidar labels exist | Frozen SSL encoder with a decoder | Average MAE 2.8 m, but the training data is US-only [arxiv:2304.07213 §3] |
| One image backbone reused for video tasks, small storage per task | ST-Adapter | 7.20M trained parameters against 121.57M for full fine-tuning [arxiv:2206.13559 tab. 1] |
| Image-to-image task, no labels and no weight update | Visual prompting with an inpainting MAE | Pascal-5i mIoU of 24 to 30 for the figure-pretrained model [arxiv:2209.00647 tab. 1] |
| Phone-class image enhancement with a tight parameter budget | IAT | About 0.09M parameters, 23.50 PSNR on LOL-V2-real [arxiv:2205.14871 tab. 1] |
| Robot task given as mixed text and object images | VIMA-style object tokens with cross-attention | 81.5% on L1 at 200M against 58.0% for the Gato-style baseline [arxiv:2210.03094 tab. 10] |
| Multi-temporal satellite data with few labels | Prithvi-EO-2.0 fine-tuning | 67.0 mIoU with about 1% of Landslide4Sense labels, 600M, against 59.7 for U-Net [arxiv:2412.02732 §4.2.4] |
| Small ViT where dense per-patch supervision is wanted and a label store fits the budget | Token labeling with a pretrained dense annotator | 84.4% ImageNet top-1 for LV-ViT-S at 26M [arxiv:2104.10858 tab. 4] |
| Many tasks on one frozen ViT, few stored parameters per task | FacT tensor factorisation | 75.6 VTAB-1K average at 0.069M against 75.5 for NOAH at 0.361M [arxiv:2212.03145 tab. 1] |
| Video recognition from an image checkpoint, no video pretraining stage | UniFormerV2 modules on CLIP ViT | 90.0 on Kinetics-400 at 64 frames, and 88.8 at 8 frames for 8.0 TFLOPs [arxiv:2211.09552 tab. 1] |
| RGB-thermal tracking on a frozen ViT | BAT bi-directional adapter | 0.32M trained parameters [arxiv:2312.10611 §1]; state of the art on RGBT234 and LasHeR [arxiv:2312.10611 §4.2] |
| Few-shot segmentation where query-support matches are noisy | VAT cost aggregation | 90.3 against 86.5 for HSNet, one-shot on FSS-1000 [arxiv:2207.10866 tab. 2] |
| Monocular depth with a Swin encoder and window-local refinement | NeW CRFs | KITTI Eigen Abs Rel 0.052 [arxiv:2203.01502 tab. 1] |
| One structure-appearance image pair, no dataset, no adversarial training | Splice | Preferred over STROTSS in 79.0% of Wild-Pairs comparisons [arxiv:2201.00424 tab. 1] |
| Promptable 3D masks for CT, MR or ultrasound volumes | SAM-Med3D | 76.27% overall Dice with one 3D point [arxiv:2310.15161 tab. 3] |
| Robot tasks that need memory of earlier steps | MEM short- and long-horizon memory | Author-reported tasks up to 15 minutes [arxiv:2603.03596 §4.1] |
| Unlabelled images, frozen features for k-NN lookup or retrieval | DINO | 78.3% k-NN for ViT-S/8, 78.2% linear for ViT-B/16 [arxiv:2104.14294 tab. 2] |
| Same self-supervised recipe for images, speech and text | data2vec | 84.2% and 86.6% ImageNet-1K top-1 for ViT-B and ViT-L [arxiv:2202.03555 tab. 1] |
| Dense prediction backbone at varying input sizes | Swin (shifted windows) | 58.7 box AP on COCO test-dev and 53.5 mIoU on ADE20K for Swin-L [arxiv:2103.14030 tab. 2] [arxiv:2103.14030 tab. 3] |
| Scaling a window ViT to large size or resolution | Swin V2 (post-norm, scaled cosine attention, log-spaced bias) | 84.0% ImageNet-V2 top-1 for the 3B model [arxiv:2111.09883 tab. 2]; the ImageNet-1K V1 result is below CoAtNet-7 [arxiv:2111.09883 tab. 2] |
| Video action recognition from image-pretrained weights | Video Swin | 84.9% Kinetics-400 top-1 with 10×5 views [arxiv:2106.13230 tab. 1] |
| Bitemporal change masks from aligned remote-sensing images | ChangeFormer-style Siamese hierarchical encoder | F1 90.40 on LEVIR-CD [arxiv:2201.01293 tab. 1] |
| Monocular depth with a per-image bin-width head | AdaBins mini-ViT head | delta1 0.903 on NYU-Depth-v2 [arxiv:2011.14141 tab. 2] |
| Explain one ViT prediction for one class | Relevance propagation through attention and skips | Negative-perturbation AUC 54.16 against 53.1 for rollout [arxiv:2012.09838 tab. 1] |
| Speed up a pretrained ViT without retraining | Token merging (ToMe) | 84.25% at 182.9 images per second against 85.96% at 93.3 without merging, ViT-L/16 MAE with r = 8 [arxiv:2210.09461 tab. 1] |
| Fast ViT-style classifier on GPU and CPU | EfficientViT (M5) | 77.1% ImageNet-1K top-1 at 10,621 images per second on a V100 [arxiv:2305.07027 tab. 2] |
| Diffusion model with a transformer denoiser | DiT (adaLN-Zero) | FID 2.27 at 256×256 with classifier-free guidance 1.50 [arxiv:2212.09748 tab. 2] |
| Multilingual image-text generation with large compute | PaLI | 53.6 CIDEr on Crossmodal-3600 for the 17B model [arxiv:2209.06794 tab. 2]; the 17B run used 1,024 TPUv4 chips for 7 days [arxiv:2209.06794 app. A.1] |
| Chart questions that need arithmetic, table read from the picture | VisionTaPas-style chart table extraction | 45.52% with an extracted table against 61.84% with the gold table [arxiv:2203.10244 tab. 5] |
| Science questions with a diagram, models under 1B parameters | Multimodal-CoT with frozen ViT-L/32 features | 90.45% ScienceQA test for the Large model [arxiv:2302.00923 tab. 4] |
| 2D multi-organ abdominal CT with skip connections wanted, 18 training cases | TransUNet (ResNet-50 stem, ViT-B/16, CUP decoder) | 77.48% average DSC on Synapse, with 12 evaluation cases [arxiv:2102.04306 tab. 1] [arxiv:2102.04306 §4.1] |
| Convolutional backbone to compare against a ViT at scale | InternImage (DCNv3) | 89.6% ImageNet-1K top-1 for the H variant at 640 px [arxiv:2211.05778 tab. 2] |
| ImageNet classification where a hybrid CNN and transformer is acceptable | Conformer | 83.4% top-1 at 224 px for Conformer-S, against 81.8% for DeiT-B [arxiv:2105.03889 tab. 2] |
| Phone latency is the binding constraint, transformer layout preferred | EfficientFormer-L1 | 79.2% top-1 at 1.6 ms on iPhone 12 with CoreML [arxiv:2206.01191 tab. 1] |
| Phone latency is the binding constraint, CNN layout preferred | RepViT-M0.9 | 78.7% top-1 at 0.9 ms on iPhone 12 with Core ML, with distillation [arxiv:2307.09283 tab. 1] |
| Segmentation labels unavailable, label list may change | GroupViT | 52.3 zero-shot mIoU on PASCAL VOC 2012, trained on image-text pairs only [arxiv:2202.11094 tab. 5]; the [card](/papers/2202.11094) records the official code as non-commercial |
| Underwater photos with colour casts, paired reference images available | U-shape Transformer, trained on LSUI | PSNR 24.16 on Test-L400, the best of ten compared methods [arxiv:2111.11843 tab. 3] |
| Vehicles and roadside sensors share LiDAR features, poses and delays are noisy | V2X-ViT | AP at IoU 0.7 of 0.614 against 0.402 with no fusion, on the authors' simulated V2XSet [arxiv:2203.10638 tab. 1] |
| Satellite time series or many spectral bands, unlabelled pretraining | SatMAE | 81.49% top-1 on fMoW temporal, with nine random sequences and test augmentation [arxiv:2207.08051 §5.3]; the [card](/papers/2207.08051) records the released code as non-commercial |
| Language-image pretraining on unlabelled images with a fixed budget, extra image-only compute acceptable | SLIP (CLIP loss plus SimCLR loss) | 42.8% zero-shot against 37.6% for CLIP at 25 epochs on YFCC15M [arxiv:2112.12750 tab. 2]; about three times the image activations per step [arxiv:2112.12750 §3.3] |
| Zero-shot or few-shot action labels from a CLIP checkpoint, short clips | X-CLIP (cross-frame message tokens) | 83.8% Kinetics-400 top-1 for ViT-B/16 at 8 frames [arxiv:2208.02816 tab. 1] |
| Long clips where attention memory is the bound | VideoMamba (bidirectional Mamba) | 83.3% Kinetics-400 top-1 at 64×384 px for M [arxiv:2403.06977 tab. 3]; linear-time scan [arxiv:2403.06977 §3.1] |
| Metric distances from a single image with a known focal length, no per-dataset scale fit | Metric3D v2 with a ViT-L or ViT-g encoder | 0.063 and 0.067 zero-shot AbsRel on NYUv2 [arxiv:2404.15506 tab. 1] |
| One video backbone for action recognition, localisation and video-text retrieval | InternVideo fused model | 91.1% Kinetics-400 top-1 [arxiv:2212.03191 tab. 2]; the fusion stage is not in the released code [card](/papers/2212.03191) |
| Dense prediction backbone where attention cost must grow linearly with tokens | PVT v2 (pooled keys and values) | 83.8% ImageNet-1K top-1 for B5 at 224 px [arxiv:2106.13797 tab. 2]; 46.1 RetinaNet AP on COCO for B4 [arxiv:2106.13797 tab. 3] |
| ViT-style classifier with convolutional locality and no position embedding | CvT-13 | 81.6% ImageNet-1k top-1 with about 20M parameters [arxiv:2103.15808 tab. 3] |
| Unlabelled video only, no extra image or video data | VideoMAE (tube masking, 90% ratio) | 62.6% on HMDB51 and 91.3% on UCF101 after pretraining on each unlabelled training split [arxiv:2203.12602 tab. 2] |
| Contrastive self-supervised ViT pretraining on unlabelled images | MoCo v3 with a frozen random patch projection | 76.7% linear probing for ViT-B/16 [arxiv:2104.02057 tab. 1]; the paper reports training instability that the freeze reduces [arxiv:2104.02057 §4.2] |
| Video recognition at billion-parameter scale, large compute budget | VideoMAE V2 (ViT-g) | 90.0% top-1 on Kinetics-400 [arxiv:2303.16727 tab. 6]; the authors' earlier ViT-g run took over two weeks on 64 A100 GPUs [arxiv:2303.16727 §1] |
| Fixed compute budget, ViT-style backbone with a shrinking token grid | PiT (pooling between stages) | 80.9% ImageNet-1k top-1 at 2.9 B FLOPs for PiT-S, against 79.8% for ViT-S at 4.6 B, under the DeiT recipe [arxiv:2103.16302 tab. 2] |
| Input resolution is high or varies, linear cost in tokens wanted, channel-level attention acceptable | XCiT (cross-covariance attention) | 82.0% ImageNet-1k top-1 for S12/16 at 224 px without distillation [arxiv:2106.09681 tab. 1]; the 8 × 8 variants cost far more compute [arxiv:2106.09681 tab. 1] |
| Pure convolutional backbone at Swin-level accuracy, simple to implement and fine-tune | ConvNeXt | 82.1% against 81.3% for Swin-T at 224 px [arxiv:2201.03545 tab. 1]; 50.4 box AP for both on COCO Cascade 3× with T size [arxiv:2201.03545 tab. 3] |
| Supervised pretraining where ImageNet-21k data is available and the downstream task is classification or transfer | ImageNet-21K-P with semantic softmax | Beats ImageNet-1k pretraining in 30 of 30 cells tested [arxiv:2104.10972 tab. 4]; about 54 hours for 80 epochs on 8 V100 GPUs with ImageNet-1k initialisation [arxiv:2104.10972 §3.1] |
| Open-vocabulary segmentation with a frozen CLIP and few trainable parameters | SAN side adapter | 8.4M trainable parameters and 15.2 FPS at 640² [arxiv:2302.12242 tab. 3]; 27.5 mIoU on ADE-150 with ViT-B/16 [arxiv:2302.12242 tab. 2] |
| Image-text model with no region detector, pretraining budget near 4M images | METER (co-attention, MLM and ITM) | 77.64 VQAv2 test-std with CLIP-ViT [arxiv:2111.02387 tab. 8]; the paper's masked-patch variant scored lower [arxiv:2111.02387 tab. 7] |
| Optical flow, transformer over a compressed cost volume | FlowFormer | 1.16 AEPE on Sintel test clean, against 1.39 for GMA with warm start [arxiv:2203.16194 tab. 1]; loss and optimiser not in the paper |
| Masked-image pretraining where the encoder should only build features | CAE (regressor and decoder kept apart) | 77.1% attentive probing for ViT-B at 1600 epochs [arxiv:2202.03026 tab. 1] |
| Image backbone from local attention windows, parameter budget matters, slow training acceptable | HaloNet | 84.9% ImageNet top-1 from scratch for H7 [arxiv:2103.12731 §4.1]; slower to train than matched EfficientNets [arxiv:2103.12731 §5] |
| Add a relative position term to an existing ViT or DETR | iRPE, contextual Product table on the keys | DeiT-S from 79.9 to 80.9 top-1 [arxiv:2107.14222 tab. 1]; DETR from 39.5 to 40.8 AP at 150 epochs [arxiv:2107.14222 tab. 6] |
| Mobile classifier with a small MAdds budget and global context wanted | EdgeNeXt-S | 79.4% top-1 at 256 px with 1.30G MAdds, against 78.4% for MobileViT-S at 2.01G [arxiv:2206.10589 tab. 3] |
| Video quality scoring on high-resolution clips, fixed compute per clip | FAST-VQA fragments with end-to-end Video Swin-T | 279 GFLOPs at every resolution, against 11,852 for full-resolution Swin-T features at 1080P [arxiv:2207.02595 tab. 4]; 0.814 PLCC on LSVQ 1080p [arxiv:2207.02595 tab. 3] |
| Pure transformer with a pooled-key pyramid, dense prediction on ADE20K or COCO | P2T | 82.4% ImageNet-1K top-1 at 3.7 GFLOPs for P2T-Small [arxiv:2106.12011 tab. 2]; 46.7 mIoU on ADE20K with Semantic FPN [arxiv:2106.12011 tab. 3] |
| Global mixing needed, attention speed on the target GPU or CPU matters more than the last point of accuracy | HiLo attention in LITv2 | 82.0% ImageNet-1K top-1 at 1,471 images per second against 82.3% for full self-attention at 1,293 [arxiv:2205.13213 tab. 4] |
| Phone CPU latency is the binding constraint, hierarchical transformer preferred | EdgeViT-XXS or EdgeViT-XS | 74.4% at 32.8 ms and 77.5% at 54.1 ms on a Galaxy S21 CPU [arxiv:2205.03436 tab. 2]; MobileViT-XS is 74.7% at 150.1 ms [arxiv:2205.03436 tab. 2] |
| Single-object tracking with one shared backbone for search and exemplar | SimTrack, ViT-B/16 with joint attention | 69.3% LaSOT AUC and 54.8% TNL2K AUC [arxiv:2203.05328 tab. 1]; exemplar features are recomputed on every frame [arxiv:2203.05328 §4.1] |
| Clinical prediction from a radiograph, free text and labs, all modalities present | IRENE-style unified token sequence | Mean AUROC 0.924 on eight pulmonary diseases against 0.858 for Perceiver [arxiv:2306.00864 tab. 1]; missing modalities are an open problem [arxiv:2306.00864 p. 7] |
| Small domain image set with few images per class, backbones to compare | AQUA20 as a reference point, with repeats added | ConvNeXt 90.69% and SwinV2-B 88.65% top-1 [arxiv:2506.17455 tab. 3]; one run per model, so gaps near one point are not settled [arxiv:2506.17455 §4.1] |
| Choosing a medical ViT architecture, want the field mapped before picking a model | The medical review as a map, then the primary cards | Taxonomy by task and by Transformer placement; benchmark rows have unclear provenance [arxiv:2301.03505 tab. 6] [arxiv:2301.03505 §2.2] |

## Implementation sketch

The shared core of most variants is a patch embedding followed by pre-norm transformer
blocks. Segmenter's linear decoder is the simplest head on top [arxiv:2105.05633 §3.1]
[arxiv:2105.05633 §3.2]. Other variants change the tokeniser (ESRT unfolds 3×3
neighbourhoods, MFT pools into a few tokens), the attention pattern (windows, segments,
experts, Fourier mixing in AFNO [arxiv:2111.13587 §4]) or the training target (masked
pixels, teacher tokens, prototype assignments [arxiv:2204.07141 §3]), and are noted in
the variant sections above. MAE differs at the input rather than the loss: it removes the
masked patches before the encoder [arxiv:2111.06377 §3]. ST-Adapter leaves the frozen
weights alone and adds a residual bottleneck inside each block [arxiv:2206.13559 §3.2].

```pseudocode
# Shared ViT core for dense prediction  [arxiv:2105.05633 §3.1]
# Input: image (B, 3, H, W); patch size P; depth L; width D
# Output: one D-dim token per patch, (B, N, D) with N = H*W / P^2

def vit_encoder(image, pos, P, L):
    # Cut the image into P x P patches and project each one linearly
    tokens = conv2d(image, out=D, kernel=P, stride=P)  # (B, D, H/P, W/P)
    tokens = flatten(tokens)                           # (B, N, D)
    # Add learned position embeddings, bilinearly resized to the grid
    # [arxiv:2105.05633 §4.2]
    tokens = tokens + resize_pos(pos, tokens.grid)     # (B, N, D)
    # Pre-norm blocks: global attention over all patches, then an MLP
    # [arxiv:2105.05633 §3.1] [arxiv:2012.15840 §3.2]
    for _ in range(L):
        tokens = tokens + attention(layer_norm(tokens))
        tokens = tokens + mlp(layer_norm(tokens))
    return layer_norm(tokens)                          # (B, N, D)
```

```pseudocode
# Linear decoder: one score per class per patch, then upsample
# [arxiv:2105.05633 §3.2]
# Input: tokens (B, N, D); grid H/P x W/P; K classes
# Output: logits (B, K, H, W)

def linear_head(tokens, grid, out_size):
    # Point-wise linear map from D to K classes
    logits = linear(tokens, out=K)                     # (B, N, K)
    logits = reshape(logits, (B, K) + grid)            # (B, K, H/P, W/P)
    # Bilinear upsampling to the input resolution
    return bilinear_upsample(logits, out_size)         # (B, K, H, W)
```

Masked-image variants replace the masked patch embeddings with one learned token before
the blocks [arxiv:2111.09886 §3.2], and the sparse-expert variant swaps the MLP for a
routed set of experts [arxiv:2106.05974 §2.2].

## Contradictions

- **Is global attention the source of the gains?** SETR and Segmenter motivate
  attention in every layer by global context [arxiv:2012.15840 §1]
  [arxiv:2105.05633 §1]. How Do Vision Transformers Work? argues the opposite
  emphasis: data specificity of the attention matters more than long-range dependency,
  and it finds that local windows can beat global attention on small data
  [arxiv:2202.06709 §2]. The papers test different things (segmentation with
  pretraining versus CIFAR-scale optimisation), so neither result settles the question.
- **Linear probing.** SimMIM reports 56.7% linear accuracy for ViT-B and says its
  features are meant for fine-tuning [arxiv:2111.09886 tab. 6] [arxiv:2111.09886 §3.5].
  iBOT reports 79.5% linear accuracy for ViT-B under a DINO-style protocol
  [arxiv:2111.07832 tab. 1]. MAE reports 75.8% for ViT-L and argues that linear probing
  is a poor proxy for transfer [arxiv:2111.06377 app. B] [arxiv:2111.06377 §4.3]. The
  papers use different targets and evaluation protocols, so the cards cannot say which
  objective produces better frozen features.
- **Low-label performance of MAE features.** MSN reports 8.2% for MAE ViT-B/16 with one
  label per class, and 49.8% for its own ViT-B/16 [arxiv:2204.07141 tab. 1]. The MSN
  card says the MAE one-image row uses a linear classifier instead of partial
  fine-tuning, and that MSN's own low-shot rows use L2-regularised logistic regression
  on frozen features [arxiv:2204.07141 tab. 1] [arxiv:2204.07141 app. A.2]. The protocols
  are close but not identical, and the cards do not say how much of the gap is due to
  the protocol.
- **Paper versus released code.** Several cards record that the released code differs
  from the paper in ways that change the result. ResT's code skips the head-mixing step
  in stage 1, which has one head [arxiv:2105.13677 §2.2]. The AFNO card reads the released
  code as not performing the hard frequency truncation the paper describes
  [arxiv:2111.13587 §5.6]. STEGO's default config is ViT-Small, while the paper's
  numbers are for ViT-Base [arxiv:2203.08414 §4.3]. ST-Adapter's code has no
  non-linearity where the equation has one [arxiv:2206.13559 §3.2]. Implement the
  variant you intend to match, and check its card. SAM-Med3D's paper describes an
  overlapping sliding window for large volumes, which its released code does not
  include [arxiv:2310.15161 §4.3]. Conformer's detection learning rate is 2e-4 at batch
  32 in the paper [arxiv:2105.03889 §4.3]; the released mmdetection config uses 1e-4, as
  the [Conformer card](/papers/2105.03889) records.
  PiT's text and its released code give different patch sizes for the base model
  [arxiv:2103.16302 §3.4] [arxiv:2103.16302 tab. 1]. XCiT's appendix algorithm and its released
  code take the softmax over different channel axes [arxiv:2106.09681 app. C]. ConvNeXt's
  table gives a weight-initialisation scale of 0.2, and the code uses 0.02
  [arxiv:2201.03545 tab. 5]. Semantic softmax's depth weights in the ImageNet-21K-P code differ
  from the printed equation [arxiv:2104.10972 eq. 2] [arxiv:2104.10972 eq. 3]; see
  [the card](/papers/2104.10972).
- **Test-split reporting.** FacT reports its VTAB-1K numbers on the test split
  [arxiv:2212.03145 §4.1]. The MFT card judges its own reported numbers likely optimistic
  because the released notebook keeps the checkpoint with the best test accuracy
  [arxiv:2203.16952 §IV-B]. AQUA20 has the same issue: its paper keeps the best checkpoint
  seen during evaluation, and its released notebook keeps the checkpoint with the lowest
  test loss [arxiv:2506.17455 §4.1]. Compare such numbers with care.
- **Is a transformer needed at all?** DiT argues that the U-Net's inductive bias is not what
  makes diffusion models work [arxiv:2212.09748 §1]. InternImage argues that a convolutional
  operator with input-dependent sampling can reach ViT-level scale and accuracy
  [arxiv:2211.05778 §1]. ChangeFormer argues that a transformer relates distant parts of a
  change-detection pair better than deep ConvNets do [arxiv:2201.01293 §1]. ConvNeXt makes a
  related case: a ResNet changed step by step with Transformer-era recipes and layer choices
  matches Swin-T on ImageNet-1k [arxiv:2201.03545 §1] [arxiv:2201.03545 tab. 1]. The four
  papers test different tasks and do not share a protocol, so none of them settles the question.
- **Same setting, two numbers.** ToMe reports 83.92% at 183 images per second in its appendix
  sweep and 84.25% at 182.9 in its ablation table for ViT-L/16 MAE with r = 8
  [arxiv:2210.09461 app. A] [arxiv:2210.09461 tab. 1]. The unmerged baseline also differs:
  the table caption gives 85.96% at 93.3 images per second, the sweep 85.66% at 93, and the
  paper does not reconcile the two [arxiv:2210.09461 tab. 1]. Use one source and say which.
- **Record claims depend on the comparison.** Swin V2 reports new results on several vision
  benchmarks, but on ImageNet-1K V1 its 3B model is slightly below CoAtNet-7
  (90.17% against 90.88%) [arxiv:2111.09883 tab. 2]. The authors did not align model size or
  pretraining data across rows [arxiv:2111.09883 §4.2].
- **Masked-patch objectives in vision-language pretraining.** METER reports that adding
  masked-patch prediction lowers its VQAv2 score [arxiv:2111.02387 tab. 7], and its card
  notes that the released code has no such objective. CAE finds that masked-patch
  reconstruction is needed to avoid a trivial solution [arxiv:2202.03026 §5.5]. The two
  settings differ, one a joint image-text model and the other an image-only encoder, so
  neither result settles the other.
- **Synapse numbers for one model.** The medical review gives TransUNet 77.49% Dice in its
  Synapse table [arxiv:2301.03505 tab. 6], while its own discussion gives 77.48%
  [arxiv:2301.03505 §4]. The TransUNet paper reports 77.48% [arxiv:2102.04306 tab. 1]. The
  difference is small, but it shows that the review's table is not one source copied
  unchanged. Use the source paper's number and say which one you used.
- **Relative or absolute position.** iRPE finds that a relative term can replace the
  absolute encoding for classification: with no absolute encoding and the query, key and
  value terms together, DeiT-S reaches 81.3% top-1 against 79.9% for the learned absolute
  baseline [arxiv:2107.14222 tab. 4]. For DETR the same choice does not hold: without the
  absolute encoding, the contextual term reaches 38.7 AP, below 39.5 for the sinusoidal
  baseline [arxiv:2107.14222 tab. 6]. The paper also says earlier vision studies disagreed
  on whether relative encodings beat absolute ones [arxiv:2107.14222 §1]. The two tasks
  differ in resolution and training, so the card does not treat either result as general.
- **Released configs for two ViT papers.** iRPE's DeiT config names do not match the
  bucket counts in its Table 1; the card's hand calculation gives different counts
  [arxiv:2107.14222 tab. 1] [code: iRPE/DeiT-with-iRPE/rpe_models.py#L70]. EdgeNeXt's
  released XXS model uses 2, 2, 6 and 2 blocks per stage, against 3, 3, 9 and 3 in its
  table [arxiv:2206.10589 tab. 1] [code: models/model.py#L20]. Take the block counts and
  bucket counts from the paper's tables and check the cards before reusing the configs.
- **Stage depths in a mobile transformer.** EdgeViT's table gives XS stage depths of 1,
  1, 2 and 2, and S depths of 1, 2, 3 and 2. The released code uses 1, 1, 3 and 1 for XS
  and 1, 2, 5 and 3 for S [arxiv:2205.03436 tab. 1]. The card's numbers come from the
  table, so a reimplementation from the paper will have different depths from the code.

## Papers

- [SETR: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](/papers/2012.15840): full-resolution ViT encoder with three segmentation decoders.
- [Segmenter: Transformer for Semantic Segmentation](/papers/2105.05633): linear and mask-transformer heads on a ViT encoder; convolution-free.
- [SimMIM: A Simple Framework for Masked Image Modeling](/papers/2111.09886): masked pixel regression with a linear head and large masked blocks.
- [Transformer for Single Image Super-Resolution](/papers/2108.11084): lightweight super-resolution with segment-wise attention over unfolded patches.
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](/papers/2203.16952): classification token from a second sensor fused with hyperspectral patch tokens.
- [Sigmoid Loss for Language Image Pre-Training](/papers/2303.15343): pairwise sigmoid loss for image-text pretraining with a ViT image tower.
- [Improved Baselines with Visual Instruction Tuning](/papers/2310.03744): frozen CLIP ViT-L/14 with an MLP connector for a multimodal chat model.
- [Scaling Vision with Sparse Mixture of Experts](/papers/2106.05974): sparse MoE ViT with batch prioritised routing at inference.
- [iBOT: Image BERT Pre-Training with Online Tokenizer](/papers/2111.07832): masked image modeling with an EMA teacher as the online tokenizer.
- [How Do Vision Transformers Work?](/papers/2202.06709): analysis of attention as smoothing and a local-attention hybrid, AlterNet.
- [FSRA drone-to-satellite matching](/papers/2201.09206): patch-token regions aligned across drone and satellite views.
- [Generalized Category Discovery](/papers/2201.02609): ViT-B/16 features clustered with semi-supervised k-means over known and new classes.
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](/papers/2109.03201): 3D transformer with local and global attention and skip attention.
- [Transformers in Remote Sensing: A Survey](/papers/2209.01206): survey of transformer methods for optical, hyperspectral and radar imagery.
- [Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059): hybrid mobile ViT found by a size-and-speed search.
- [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](/papers/2407.08083): hierarchical backbone mixing convolution, state-space blocks and attention.
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](/papers/2311.06242): prompt-driven seq2seq vision model with location tokens.
- [Canopy height from RGB imagery](/papers/2304.07213): frozen DINOv2 ViT-H with a DPT decoder trained on lidar heights.
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](/papers/2207.10666): stored-teacher distillation and progressive model contraction for small ViTs.
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](/papers/2306.05424): frame-level CLIP ViT features pooled into video tokens for a chat model.
- [ResT: An Efficient Transformer for Visual Recognition](/papers/2105.13677): multi-scale pyramid with efficient attention and a convolutional positional gate.
- [Masked Autoencoders Are Scalable Vision Learners](/papers/2111.06377): masked-patch pixel reconstruction with an encoder that sees only visible patches.
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](/papers/2111.13587): Fourier-domain token mixer in place of self-attention.
- [Rethinking Spatial Dimensions of Vision Transformers](/papers/2103.16302): PiT, pooling between ViT stages to shrink the token grid and widen channels.
- [ImageNet-21K Pretraining for the Masses](/papers/2104.10972): cleaned ImageNet-21k with semantic softmax over WordNet depths.
- [XCiT: Cross-Covariance Image Transformers](/papers/2106.09681): attention across feature channels, with cost linear in the token count.
- [A ConvNet for the 2020s](/papers/2201.03545): ConvNeXt, a pure ConvNet modernised step by step from ResNet-50.
- [Deep ViT Features as Dense Visual Descriptors](/papers/2112.05814): DINO ViT keys as training-free dense descriptors for co-segmentation and correspondence.
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](/papers/2203.08414): frozen DINO ViT-B with a small head distilled from feature correlations, then clustered.
- [Masked Siamese Networks for Label-Efficient Learning](/papers/2204.07141): masked joint-embedding with prototypes, no pixel reconstruction.
- [IAT: a 90K-parameter transformer for image enhancement and exposure correction](/papers/2205.14871): tiny transformer-style image enhancer with learned queries for a global colour matrix and gamma.
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](/papers/2206.13559): bottleneck adapters with a depth-wise temporal 3D convolution on a frozen image ViT.
- [Visual Prompting via Image Inpainting](/papers/2209.00647): MAE-VQGAN inpaints a grid prompt of example pairs and a query.
- [VIMA: General Robot Manipulation with Multimodal Prompts](/papers/2210.03094): object-token prompts with a cross-attention decoder for robot control.
- [Photorealistic Video Generation with Diffusion Models](/papers/2312.06662): W.A.L.T, a window-attention diffusion transformer on a shared image-video latent.
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](/papers/2412.02732): ViT masked autoencoder for multispectral satellite time series with metadata bias terms.
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](/papers/2104.10858): per-patch dense loss from a pretrained annotator, with MixToken on patch embeddings.
- [Splicing ViT Features for Semantic Appearance Transfer](/papers/2201.00424): small generator trained on one pair, judged by frozen DINO ViT-B/8 features.
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](/papers/2203.01502): window-local fully connected CRF with learned attention potentials for depth.
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](/papers/2207.10866): transformer cost aggregation over the query-support correlation volume.
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](/papers/2211.09552): video modules added to a pretrained image ViT.
- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](/papers/2212.03145): tensorised ViT weights with a shared low-rank factorised update.
- [Language-Driven Representation Learning for Robotics](/papers/2302.12766): masked video autoencoder whose caption is a condition or a generation target.
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](/papers/2310.15161): promptable 3D ViT segmenter for medical volumes with point prompts.
- [Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611): shared bi-directional adapter that lets two modality streams prompt each other.
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](/papers/2603.03596): video ViT memory plus a language summary memory for a robot policy.
- [AdaBins: Depth Estimation using Adaptive Bins](/papers/2011.14141): per-image bin widths from a small transformer head for depth.
- [Transformer Interpretability Beyond Attention Visualization](/papers/2012.09838): class-specific relevance through attention and skip connections.
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](/papers/2102.04306): ResNet-50 stem, ViT-B/16 encoder and a cascaded upsampler.
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](/papers/2103.14030): shifted-window attention in a four-stage pyramid.
- [Emerging Properties in Self-Supervised Vision Transformers](/papers/2104.14294): DINO, label-free self-distillation for ViTs.
- [Video Swin Transformer](/papers/2106.13230): shifted 3D windows for video on a Swin backbone.
- [Swin Transformer V2: Scaling Up Capacity and Resolution](/papers/2111.09883): scaled Swin with post-norm, cosine attention and a log-spaced position bias.
- [A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293): hierarchical transformer Siamese encoder for change masks.
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](/papers/2202.03555): one self-distillation recipe across three modalities.
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](/papers/2203.10244): chart question answering with a ViT-based table baseline.
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](/papers/2209.06794): image and prompt in, text out, with a ViT-e encoder and mT5.
- [Token Merging: Your ViT But Faster](/papers/2210.09461): merges similar tokens in every block without new parameters.
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](/papers/2211.05778): convolutional backbone that competes with ViTs at scale.
- [Scalable Diffusion Models with Transformers](/papers/2212.09748): DiT, a transformer denoiser for latent diffusion.
- [Multimodal Chain-of-Thought Reasoning in Language Models](/papers/2302.00923): rationale and answer in two stages, with frozen ViT features.
- [A Comprehensive Review of YOLO Architectures in Computer Vision](/papers/2304.00501): survey of YOLO detectors from v1 to v8 and their variants.
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](/papers/2305.07027): sandwich blocks and cascaded group attention for measured speed.
- [A Survey of Modern Deep Learning based Object Detection Models](/papers/2104.11892): survey of detectors, backbones and lightweight networks, with author-reported numbers compiled side by side.
- [Conformer: Local Features Coupling Global Representations for Visual Recognition](/papers/2105.03889): CNN and transformer branches coupled in every block by a feature coupling unit.
- [U-shape Transformer for Underwater Image Enhancement](/papers/2111.11843): channel-wise and spatial transformers inside a U-shaped enhancement GAN, with the LSUI dataset.
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094): hierarchical grouping ViT trained on image-text pairs for zero-shot segmentation.
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](/papers/2203.10638): transformer fusion of vehicle and roadside LiDAR features with delay-aware encoding.
- [EfficientFormer: Vision Transformers at MobileNet Speed](/papers/2206.01191): pure transformer with convolutional early stages, tuned by on-device latency.
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](/papers/2207.08051): masked-autoencoder pretraining with time and spectral-group encodings.
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](/papers/2303.16727): dual masking for a billion-parameter video ViT, with progressive training.
- [RepViT: Revisiting Mobile CNN From ViT Perspective](/papers/2307.09283): pure CNN that adopts lightweight-ViT block layout, measured on an iPhone.
- [SLIP: Self-supervision meets Language-Image Pre-training](/papers/2112.12750): SimCLR-style image loss added to CLIP's contrastive loss on one shared ViT.
- [Expanding Language-Image Pretrained Models for General Video Recognition](/papers/2208.02816): CLIP image encoder with cross-frame message tokens and video-specific text prompts.
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191): masked-video ViT-H and a CLIP-initialised UniFormerV2 fused by cross-model attention.
- [VideoMamba: State Space Model for Efficient Video Understanding](/papers/2403.06977): bidirectional Mamba blocks in a ViT-style stack, with self-distillation from a smaller model.
- [Metric3D v2](/papers/2404.15506): DINOv2 ViT encoders with DPT decoders for metric depth and normals from one image.
- [Transformers in Vision: A Survey](/papers/2101.01169): survey of transformer designs in vision, grouped by self-attention block and by task.
- [CvT: Introducing Convolutions to Vision Transformers](/papers/2103.15808): overlapping convolutional tokens and convolutional projections, with no position embedding.
- [An Empirical Study of Training Self-Supervised Vision Transformers](/papers/2104.02057): MoCo v3 for ViTs, with training instability and a frozen patch projection.
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](/papers/2106.13797): pooled-key attention, overlapping patch embedding and depthwise convolution in a pyramid.
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](/papers/2203.12602): tube-masked video autoencoding on a plain ViT, pretrained without extra data.
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](/papers/2103.12731): HaloNet, local 2D self-attention with blocked haloing as a convolution replacement.
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](/papers/2111.02387): METER, a fully transformer vision-language model with co-attention and no region detector.
- [Context Autoencoder for Self-Supervised Representation Learning](/papers/2202.03026): masked-image pretraining with a separate encoder, latent regressor and decoder.
- [FlowFormer: A Transformer Architecture for Optical Flow](/papers/2203.16194): transformer over compressed cost-volume tokens, read by a recurrent cross-attention decoder.
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242): light side ViT on frozen CLIP producing mask proposals and attention biases.
- [Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs](/papers/2411.14256): ViT-B/16 driving policy with a cached left, middle or right instruction from a zero-shot LLM.
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](/papers/2107.14222): learned relative bucket terms added to attention logits, keys, queries or values.
- [EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications](/papers/2206.10589): mobile hybrid whose SDTA blocks attend across channels.
- [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](/papers/2207.02595): grid mini-patch fragments with a Video Swin-T trained end to end for video quality.
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](/papers/2301.03505): survey of ViT methods across medical imaging tasks, with compiled benchmark tables.
- [P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011): pyramid of pooled keys and values with depthwise convolutions in attention and feed-forward.
- [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328): one ViT for exemplar and search tokens, with a foveal window of exemplar centre tokens.
- [EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers](/papers/2205.03436): local-global-local blocks with one sampled token per window, measured on a phone CPU.
- [Fast Vision Transformers with HiLo Attention](/papers/2205.13213): HiLo heads for local windows and pooled global attention, in the LITv2 backbone.
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864): IRENE, one transformer over radiograph, text and lab tokens for clinical prediction.
- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455): 8,171-image underwater benchmark with fine-tuned ViT, CNN and hybrid baselines.