Transformers in Vision: A Survey

In plain words

Transformers began as language models, where they read sentences by letting every word look at every other word. Researchers then asked whether the same trick works on pictures: cut an image into small patches, treat each patch as a word, and let the patches look at each other. This survey collects the ways that idea has been used in computer vision, from classifying images and finding objects to colouring photos, generating images, handling video and reading 3D point clouds.

The main lesson is a trade-off. Transformers make few assumptions about the data, so they need a lot of training data, often collected without labels first, to learn what a convolutional network knows from its design. They also get expensive as the image gets bigger. The survey sorts the designs by how they handle these costs and what they borrow from convolutions.

TL;DR

• Survey of Transformer models in computer vision, organised by the design of the self-attention block and by task §1 fig. 4.
• Transformers need large pre-training sets and have attention cost that grows with the square of the token count §2.2 §4.1 §4.2.
• The reported numbers come from the cited papers, under setups that the survey does not always state tab. 2 tab. 3.
• There is no official code and no method to reproduce; use this card as a map to the primary papers.

Problem

Convolutional networks carry prior knowledge about images, such as translation equivariance, so they need less data §3.2.1. Transformers assume little about the structure of the input and need to learn these image rules from examples §4.2. Standard self-attention scales quadratically with the number of patches, which limits high-resolution inputs such as detection, segmentation and super-resolution §4.1. Pre-training on large unlabelled corpora is the usual answer, but ImageNet-only training of ViT-L is reported to lose about 13% top-1 accuracy compared with pre-training on JFT-300M §2.2.

Method

The survey has no method of its own. It defines the building blocks and then sorts the vision literature by them §1 fig. 4.

Self-attention block

Given a sequence of n entities with embedding size d, a self-attention layer computes learned queries, keys and values, scores each query against every key, scales the scores by the square root of the query size, normalises them with softmax, and returns a weighted sum of the values §2.1 fig. 2. The survey writes this as \mathbf{Z} = \mathrm{softmax}(\mathbf{Q}\mathbf{K}^T / \sqrt{d_q})\,\mathbf{V}. The equation is not numbered in the source, so it is cited by section. The decoder variant masks future positions by setting their scores to zero before the softmax §2.1.

# Single-head self-attention  [§2.1] [fig. 2]
# Input:  X (n, d) sequence of n entities
# Output: Z (n, d_v)

def self_attention(X, W_Q, W_K, W_V):
    # Project each entity to a query, a key and a value  [§2.1]
    queries = X @ W_Q                        # (n, d_q)
    keys = X @ W_K                           # (n, d_k), with d_q = d_k
    values = X @ W_V                         # (n, d_v)

    # Score every query against every key, then scale and normalise  [§2.1]
    scores = queries @ keys.T / sqrt(d_q)    # (n, n)
    weights = softmax(scores, axis=-1)       # each row sums to 1

    # Each output is a weighted sum over all values  [§2.1]
    return weights @ values                  # (n, d_v)

Multi-head attention runs h such blocks with separate weights, concatenates their outputs and projects them back to size d §2.1 fig. 3.

Transformer and pre-training

The original Transformer is an encoder-decoder with N identical blocks on each side. Each encoder block has multi-head self-attention and a position-wise feed-forward network, with residual connections and layer normalisation. Self-attention aggregates information and the feed-forward network transforms it §2.3 fig. 3. The standard recipe is pre-training on a large dataset, then fine-tuning on the target task §2.2. BERT's pre-training objectives are summarised as masked language modelling at a 15% mask rate and next-sentence prediction with a 50% true-pair rate §2.4.

Taxonomy of vision designs

• Self-attention inside CNNs §3.1. Non-local blocks compute a dense attention map at O(N^2) cost. Criss-cross attention keeps a sparse map on a cross path, reported to use 11 times less GPU memory, at O(2\sqrt{N}) §3.1.1. Local relation layers, attention augmentation and stand-alone local attention layers are also covered §3.1.1 §3.1.2.
• Uniform-scale ViTs §3.2.1. ViT applies a Transformer encoder to flattened image patches §3.2.1. DeiT trains on ImageNet-1K by adding a distillation token that learns from a CNN teacher (RegNetY-16GF) §3.2.1. Other variants change the tokenisation (T2T), add a sub-patch attention level (TNT), label every token for the loss (token labelling), or attend across channels instead of tokens to get linear cost (XCiT) §3.2.1.
• Multi-scale ViTs §3.2.2. A hierarchy reduces the token count and raises the feature width at each stage, which suits dense prediction. Examples: PVT, PVTv2, Swin (window attention with shifted windows), Focal, CrossFormer, RegionViT and CrossViT §3.2.2.
• Hybrid ViTs §3.2.3. Convolutions are used for patch embedding or alongside attention: CvT, LeViT, ResT, CoAtNet, Twins and others §3.2.3.
• Self-supervised ViTs §3.2.4. Contrastive pre-training (MoCo v3) and self-distillation without labels (DINO) are applied to ViTs §3.2.4.
• Detection §3.3. DETR uses a CNN backbone, a Transformer encoder-decoder and object queries, and trains with bipartite matching, removing the region proposal and non-maximum suppression stages §3.3.1. Its stated weaknesses are small objects, slow convergence and cost that grows with the feature map size, O(H^2 W^2 C) §3.3.1. Anchor DETR, Pix2Seq and YOLOS are the follow-ups discussed §3.3.1.
• Other tasks. The survey also covers segmentation §3.4, image and scene generation §3.5, low-level vision §3.6, multi-modal learning §3.7, video §3.8, low-shot learning §3.9, clustering §3.10 and 3D analysis §3.11. This card does not summarise those sections.

Hyperparameters

The survey gives no training recipe of its own. The values below are the ones it states for the original Transformer, the reference point for the designs above.

Other training details for individual models are in the primary papers and are not stated here.

Results

Author-reported. The survey compiles these numbers from the cited papers and does not re-run any of them. It gives no setup for most rows of tab. 2.

Table 2 (advantages and limitations per task). Metric and dataset as listed in the survey.

The survey adds that D-DETR reaches 52.3 AP with a two-stage design and test-time augmentation, and that Oscar relies on pre-trained object detectors tab. 2.

Table 3 (ImageNet parameters, GFLOPs and top-1). The caption says these models are trained on ImageNet from scratch at 224×224 and evaluated on the validation set. Rows marked ⋆ are pure CNNs. Subset shown; the full table is in tab. 3.

Paper vs code

No official code. This is a survey with no implementation of its own. The paper text and metadata link no code repository. Survey-linked awesome lists are not treated as code here, and I did not run a separate GitHub search because the survey has no model to match.

Pitfalls

• Same model, different numbers. Swin-T is 84.5 in tab. 2 and 81.3 in tab. 3. ViT is 88.55 in tab. 2, where pre-training is not stated, and ViT-B/16 is 79.8 in tab. 3, which is ImageNet-only per its caption. Do not compare rows across the two tables. Check the primary paper for the setup.
• Setups are often missing. tab. 2 has no column for pre-training data, resolution or training schedule.
• Pre-training dominates. The survey says ViTs need JFT-scale pre-training to beat CNNs §2.2 §3.2.1. Some later uniform-scale ViTs train on ImageNet alone, but the survey notes their performance is generally lower than pre-trained counterparts §2 (footnote).
• Quadratic cost. Global self-attention costs O(n^2) in the number of patches §4.1. High-resolution inputs (detection, segmentation, super- resolution, deblurring) are the most affected §4.1.
• Equations are unnumbered in the source. Citations here use section numbers, not equation numbers.
• Figures are third-party. Several figures are taken or adapted from other papers, for example the ViT overview is credited to the ViT paper. No figure from the survey is used here, in line with the licensing rules.
• Time cut-off. The raw metadata's latest version is dated 2022-01-19. Work after that is not covered by the survey.
• Survey, not a method. Do not implement "the survey's model". Pick a primary paper from the taxonomy above and read its own card or source.

Lineage

• Builds on: the original Transformer and BERT, cited in §2.1–§2.4. Not linked here because they have no card on this site.
• Cited papers with cards on this site: DETR for detection with Transformers §3.3.1, Swin Transformer for hierarchical windowed attention §3.2.2, and DINO for self-supervised ViTs §3.2.4.
• Compared against: the survey positions itself against NLP-focused reviews and generic attention reviews §1.
• Superseded by: not stated in the source.