CvT: Introducing Convolutions to Vision Transformers

In plain words

Vision Transformers cut an image into a grid of patches and treat each patch like a word. They work very well with lots of data, but they are weaker than older convolutional networks when data is limited. Convolutional networks have a built-in sense of nearby pixels: each layer only looks at a small neighbourhood, and the layers gradually shrink the image while adding detail. Vision Transformers have no such built-in sense; they learn where things are from a position code added to each patch.

CvT keeps the Transformer but borrows these two habits from convolutional networks. First, the image is turned into patches with overlapping small windows instead of hard cut-outs, and the model shrinks the grid stage by stage while making each patch richer. Second, inside each attention block, the queries, keys and values are computed with small sliding filters instead of plain per-patch matrices. Because the local structure is now built in, the model does not need the position code at all. The aim is a model that is better and cheaper than a plain Vision Transformer on image classification.

TL;DR

CvT is a three-stage Vision Transformer. Each stage starts with an overlapping convolution that shrinks the token grid and widens the tokens. The attention block uses depthwise convolutions to compute queries, keys and values, with stride 2 on keys and values to cut attention cost. No position embedding is used. The authors report 81.6% ImageNet-1k top-1 for the 20M-parameter CvT-13 and 87.7% for the 277M-parameter CvT-W24 pre-trained on ImageNet-22k. Choose it if you need a ViT-style backbone with convolutional inductive bias and no position encoding.

Problem

Plain Vision Transformers split the image into non-overlapping patches, add a learned position embedding, and keep one token resolution through all layers §1 §2. The paper argues that this lacks the local structure, shift tolerance and multi-scale hierarchy that convolutional networks have, which makes it weaker when training data is smaller §1. The cost of full attention also grows with the number of tokens, which is a problem at high resolution §3.2.2. The paper's assumptions: ImageNet-scale classification, with pre-training on ImageNet-22k for the best results §4.1.

Method

CvT has three stages §3. Each stage has two parts §3:

1. Convolutional token embedding §3.1 eq. 1. The input is the image (stage 1) or the 2D token map from the previous stage. A convolution with kernel size s, stride and padding maps it to tokens of width C_i. The output is flattened to a token sequence and passed through LayerNorm. The output size follows eq. 1. Unlike ViT, no position embedding is added §3.
2. Convolutional Transformer blocks §3. A stack of blocks follows the embedding. Each block replaces the standard linear query, key and value projections with a convolutional projection §3.2 fig. 3. The block contains multi-head self-attention, then an MLP with expansion ratio R_i tab. 2. The paper's figure 2(b) shows the block structure; this card describes it in words only.

The convolutional projection §3.2.1 eq. 2 reshapes the tokens to a 2D grid, applies a depthwise convolution with kernel s (3 in all stages in the main models tab. 2), and flattens the result back to tokens. The paper writes the convolution as depthwise, then BatchNorm, then pointwise convolution §3.2.1. The paper uses the convolution for query, key and value separately §3. Queries use stride 1. Keys and values use stride 2 §3.2.2, which reduces the number of key and value tokens by a factor of four and therefore the cost of the attention step §3.2.2. The class token is added only in the last stage §3. Classification uses the class token at the end of the last stage, passed through an MLP head §3.

Design choices the paper explains §3.3: CvT removes the position embedding because local relations are modelled by the convolutions, which helps with inputs of varying resolution §3.3 §4.4.

# Convolutional token embedding  [§3.1] [eq. 1]
# Input:  feature map x (B, C_in, H, W); the image in stage 1
# Output: tokens (B, H'*W', C_out), plus H' and W'

def conv_token_embedding(x, kernel, stride, padding, c_out):
    # Overlapping kernel×kernel convolution; stride sets the downsampling
    x = conv2d(x, c_out, kernel, stride, padding)  # (B, c_out, H', W')
    height, width = x.shape[2], x.shape[3]
    # Flatten to a token sequence and apply LayerNorm  [§3.1]
    tokens = layer_norm(flatten(x))                # (B, H'*W', c_out)
    return tokens, height, width

# Convolutional projection for one of Q, K or V  [§3.2.1] [eq. 2]
# Input:  tokens (B, 1 + H*W, C), class token first
# Output: tokens (B, 1 + H'*W', C)

def conv_projection(tokens, height, width, stride):
    # The class token bypasses the convolution
    # [code: lib/models/cls_cvt.py#L162-L165]
    class_token, grid = tokens[:, :1], tokens[:, 1:]
    grid = reshape_2d(grid, height, width)         # (B, C, H, W)
    # Depthwise 3×3 convolution, then BatchNorm  [§3.2.1]
    grid = batch_norm(depthwise_conv(grid, 3, stride))
    tokens = concat(class_token, flatten(grid))    # (B, 1 + H'*W', C)
    # Pointwise (1×1) channel mixing; a Linear layer with bias in code
    # [code: lib/models/cls_cvt.py#L116-L118]
    return linear(tokens)

# Multi-head self-attention with convolutional projections  [§3.2]
# Input:  tokens (B, 1 + H*W, C); the token grid is H by W
# Output: tokens (B, 1 + H*W, C)

def conv_attention(tokens, height, width, num_heads):
    # Queries keep full resolution; keys and values use stride 2  [§3.2.2]
    queries = conv_projection(tokens, height, width, stride=1)
    keys = conv_projection(tokens, height, width, stride=2)
    values = conv_projection(tokens, height, width, stride=2)
    # Split channels into heads; keys and values have about H*W/4 tokens
    queries = split_heads(queries, num_heads)      # (B, heads, 1+HW, d)
    keys = split_heads(keys, num_heads)            # (B, heads, 1+HW/4, d)
    values = split_heads(values, num_heads)
    # Scaled dot-product attention  [§3.2]. The paper states no scale;
    # the code uses C ** -0.5 with the full width C
    # [code: lib/models/cls_cvt.py#L100]
    scores = queries @ transpose(keys) * channels ** -0.5
    weights = softmax(scores)
    out = merge_heads(weights @ values)            # (B, 1+HW, C)
    # Output projection; a Linear layer in code
    # [code: lib/models/cls_cvt.py#L121]
    return linear(out)

# Convolutional Transformer block  [§3.2]
# [code: lib/models/cls_cvt.py#L325-L333]
# Input and output: tokens (B, 1 + H*W, C)

def conv_transformer_block(tokens, height, width, num_heads):
    # Pre-norm residual attention; norm placement comes from the code
    normed = layer_norm(tokens)
    tokens = tokens + conv_attention(normed, height, width, num_heads)
    # Pre-norm residual MLP with expansion ratio 4  [tab. 2]
    normed = layer_norm(tokens)
    tokens = tokens + mlp(normed)
    return tokens

# Three-stage CvT forward pass  [§3] [tab. 2]
# Input:  images (B, 3, 224, 224)
# Output: class logits (B, num_classes)

def cvt_forward(images):
    grid = images
    for stage in stages:
        # Overlapping convolution embedding and LayerNorm  [§3.1]
        tokens, height, width = conv_token_embedding(grid, stage)
        # The class token exists only in the last stage  [§3]
        if stage.has_class_token:
            tokens = concat(stage.class_token, tokens)
        # Stack of convolutional Transformer blocks  [§3.2]
        for block in stage.blocks:
            tokens = conv_transformer_block(tokens, height, width, stage)
        if stage.has_class_token:
            class_token, tokens = tokens[:, :1], tokens[:, 1:]
        # Restore the 2D token map for the next stage  [§3.1]
        grid = reshape_2d(tokens, height, width)
    # Final LayerNorm on the class token, then the head  [§3]
    return linear_head(layer_norm(class_token))

The pseudocode above follows the paper's stated structure. Where the code differs, the Paper vs code section lists the difference.

Hyperparameters

Model configurations tab. 2 cvt-13-224x224.yaml.

Head size is 64 in every stage of every model (derived: width divided by heads in tab. 2).

Results

Author-reported. Setup: ImageNet-1k (1.3M images, 1k classes) and ImageNet-22k (14M images, 22k classes) for pre-training; accuracy measured on the ImageNet-1k validation set, plus ImageNet Real and ImageNet V2 §4.1 tab. 3. Transfer to CIFAR-10, CIFAR-100, Oxford-IIIT Pets and Oxford-IIIT Flowers after ImageNet-22k pre-training §4.1 tab. 4. Baselines: ResNets, ViT, DeiT, PVT, T2T-ViT, TNT and BiT tab. 3 tab. 4. Hardware is not stated in the paper.

ImageNet classification, as published tab. 3. Params in millions, FLOPs in billions, accuracy in %. "↑384" means fine-tuned at 384×384 after pre-training (rows marked 22k: pre-trained on ImageNet-22k, fine-tuned on ImageNet-1k at 384×384 tab. 3 caption).

Headline numbers (author-reported): CvT-13 reaches 81.6% and CvT-21 82.5% ImageNet-1k top-1 at 224 tab. 3. CvT-W24 pre-trained on ImageNet-22k and fine-tuned at 384 reaches 87.7% tab. 3 §4.2; the paper reports this as 2.5 points above ViT-L/16 under the same pre-training §4.2.

Transfer after ImageNet-22k pre-training, top-1 % tab. 4:

Ablations, all on ImageNet-1k top-1 tab. 5 to tab. 8:

Table 5 is the position-embedding ablation, table 6 the convolutional token embedding ablation, table 7 the stride ablation and table 8 the projection- per-stage ablation. Blocks per stage in table 8 are 1, 2 and 10 tab. 8.

Paper vs code

Code: https://github.com/leoxiaobin/CvT, commit 46ab3c832650a655becf60dd2fc5b8628ee5fbc8, MIT licence. The paper's abstract links this repository, and its README calls itself the official implementation of this paper. Searched: the abstract link only; no other repository was considered.

Discrepancies found after reading lib/models/cls_cvt.py and the ImageNet configs at this commit:

• Learning rate. The paper states an initial learning rate of 0.02 at a total batch size of 2048 §4.1. The configs use LR: 0.00025 with 256 per GPU for CvT-13 cvt-13-224x224.yaml and LR: 0.000125 with 128 per GPU for CvT-21 cvt-21-224x224.yaml. The GPU count is not stated in the paper or the configs, so the effective batch cannot be derived. The code's rates are 80 (CvT-13) to 160 (CvT-21) times smaller than the paper's.
• Per-GPU batch. The configs use 256 per GPU for CvT-13 and 128 for CvT-21 and W24 cvt-21-384x384.yaml, while the paper states a total batch of 2048 for all models §4.1.
• Attention scale. The code scales scores by dim_out ** -0.5, where dim_out is the full channel width, not the head size cls_cvt.py. For CvT-13 stage 2 that is 192 ** -0.5 rather than 64 ** -0.5 (head size 64, tab. 2). The paper does not state a scale.
• Projection pointwise step. The paper writes each projection as depthwise convolution, BatchNorm, pointwise convolution §3.2.1 eq. 2. The code has depthwise convolution and BatchNorm in _build_projection cls_cvt.py. The pointwise mixing is a separate Linear layer per branch, proj_q, proj_k, proj_v, which has a bias when QKV_BIAS is true cls_cvt.py cvt-13-224x224.yaml. A 1×1 convolution on tokens is a Linear layer, so the code matches the paper's pointwise step except for the bias, which eq. 2 does not show. Also, query, key and value are three separate depthwise convolutions with separate BatchNorms cls_cvt.py.
• Head. The paper says the classifier is an "MLP (i.e. fully connected) Head" §3. The code applies a LayerNorm to the class token and then one nn.Linear cls_cvt.py cls_cvt.py cls_cvt.py.
• Output projection. The code applies a Linear layer after the attention product cls_cvt.py cls_cvt.py. The paper's text does not mention it.
• Activation. The model uses QuickGELU, x * sigmoid(1.702 * x) cls_cvt.py cls_cvt.py. The paper does not state the activation.
• Normalisation epsilon. The code's LayerNorm has eps=1e-5 cls_cvt.py. The paper does not state it.
• Parameter count of CvT-W24. The architecture table gives 276.7 M tab. 2; §4.1 says 298.3 M; tab. 3 gives 277 M. The code's W24 widths and depths match tab. 2 cvt-w24-384x384.yaml.
• Pre-training configs. The paper pre-trains at 224 and fine-tunes at 384 §4.1. The repository has 224 configs only for CvT-13 and CvT-21. W24 has only a 384 config cvt-w24-384x384.yaml. No fine-tuning configs for downstream tasks are in the repository.
• Drop path. Stochastic depth is applied per stage: each stage's rates go linearly from 0 to the stage's maximum (0.1 for CvT-13 and CvT-21, 0.3 for W24 stage 3) cls_cvt.py cvt-w24-384x384.yaml. The paper does not mention drop path.
• Warmup and schedule. Code: warmup for 5 epochs from warmup_lr 1e-6, cosine decay to min_lr 1e-5, and cooldown_epochs: 10 cvt-13-224x224.yaml. The paper says only "cosine" §4.1. The exact meaning of cooldown_epochs depends on the timm version; requirements.txt pins timm to 0.3.2 requirements.txt.
• Weight decay. Matches the paper: 0.05 for CvT-13, 0.1 for CvT-21 and W24 §4.1 cvt-21-224x224.yaml. The config lists bn, bias and ln as excluded from decay cvt-13-224x224.yaml.
• Augmentation. The paper says "the same data augmentation and regularization methods as in ViT" and cites DeiT §4.1. The configs use Mixup α 0.8 (probability 1.0), CutMix 1.0, label smoothing 0.1, RandAugment rand-m9-mstd0.5-inc1, random erasing probability 0.25 (pixel mode), colour jitter 0.4, horizontal flip 0.5 and bicubic interpolation cvt-13-224x224.yaml.

No other discrepancies found in the blocks, stages, embedding sizes, head counts, depths, kernels or strides, which match tab. 2 and §3 cvt-13-224x224.yaml cvt-21-224x224.yaml.

Pitfalls

• Embedding padding is not in the paper. Eq. 1 uses p, but no table gives its value. The configs use 2 for the 7×7 stride-4 embedding and 1 for the 3×3 stride-2 embeddings cvt-13-224x224.yaml. With these values the stage output sizes are 56, 28 and 14 for a 224 input, matching tab. 2.
• Eq. 1 writes the stride as s−o. The tables give the stride directly (4, 2, 2). Use the table values; the overlap o is implicit.
• Keys and values have a quarter of the tokens, and the class token is not subsampled. In the code the class token skips the convolution and is concatenated back after it cls_cvt.py. For a 224 input, the stage-1 K/V grid is 28×28 (derived: 56×56 with stride 2). Stage 1 has no class token.
• Scale is not stated. The code's scale uses the full width, not the head size (see Paper vs code). Any other choice changes the attention temperature, so reproduce the code's value if you want the published numbers.
• Activation is not stated. The code uses QuickGELU cls_cvt.py. Using a different activation is a change from the code.
• Positional embedding is removed, but the code still has the loader. The model has no position embedding cls_cvt.py. The init_weights method resizes pos_embed only when loading a checkpoint that contains one cls_cvt.py.
• The learning rate is the largest risk. Using the paper's 0.02 with the code's batch setup would be far too large for the code's configs. Use the code's values, and say which you used.
• Augmentation is only in the code. The paper's pointer to ViT/DeiT is not enough to reproduce the recipe. The paper does not say whether the configs' Mixup/CutMix/RandAugment settings are the ones behind its numbers.
• Fine-tuning recipe is incomplete. The fine-tuning learning rate, weight decay and warmup are not stated, and the repository has no fine-tuning configs. Fine-tuning steps and resolution are stated §4.1.
• Results text and tables disagree in places (author-reported).
  • CvT-21 is said to be "0.5% higher" than DeiT-B §4.2; the table gives 82.5 vs 81.8, a 0.7 difference tab. 3.
  • CvT-13 is said to beat ResNet-152 by 3.2% §4.2; the table gives 3.3 tab. 3.
  • Ablation text: removing the convolutional embedding "drops 0.8%" §4.4; table 6 gives 81.6 vs 80.7, a 0.9 drop. The text calls "row d" the case with both embedding and position embedding; in table 6 that case is row c.
  • Key/value stride 2 costs "0.3%" §4.4; table 7 gives 82.3 vs 81.6, 0.7.
  • Convolutional projection in all stages "improves ... to 81.5%" §4.4; table 8 gives 81.6.
  • Downstream: CvT-W24 is said to be best "across all the downstream tasks" §4.3; on Oxford Pets ViT-H/16 is 94.82 and CvT-W24 is 94.73 tab. 4.
• Position embedding is removed for all stages. The ablation in tab. 5 shows no accuracy loss, but it is an ImageNet-only test; behaviour at other resolutions is not tested in this paper beyond the stated claim that it helps variable-resolution tasks §4.4.
• Compute. Keys and values are subsampled, but Q/K/V convolutions still run at full resolution for queries and cost O(s²C T) each §3.2.2. The stated saving is about four times in attention, with the 3×3 projections adding s²C parameters per projection §3.2.2. Hardware and wall-clock time are not reported.

Lineage

• Builds on: Vision Transformer (ViT) for the patch-token backbone and the Transformer block §1 §2; CNN-style hierarchy and local structure §1. Depthwise separable convolution §3.2.1 §3.2.2.
• Compared against: DeiT, PVT, T2T-ViT, TNT, ResNets and BiT tab. 3 tab. 4. The paper contrasts itself with prior Transformer work that adds convolution modules to the block, which it says adds complexity and cost §3.2.
• Relation to concurrent work: CPVT replaces the position embedding with a conditional position encoding §4.4; the paper's stance is that it needs no position encoding at all.
• Superseded by: not stated.