CNN-Transformer hybrid

Overview

A CNN-Transformer hybrid is a vision backbone that runs convolution blocks for most of its depth and places a smaller number of attention blocks where global context is needed. The motivation is deployment cost: the paper reports that multi-head self-attention, LayerNorm and GELU are slow on TensorRT and CoreML, while plain convolutions are fast on those runtimes but less accurate 2207.05501 §1. The shared idea across designs is that convolution does the local work and the cost, and attention is kept to a few positions.

Only one card covers this pattern so far. The variants below are therefore the design choices that one paper varies, plus its ablations, not a survey of hybrid architectures.

Variants

Late-stage attention (Next-ViT's hybrid strategy)

Stage 1 is convolution only. Stages 2 to 4 each repeat several convolution blocks and end with one attention block 2207.05501 §3.4 2207.05501 eq. 7. Attention therefore reaches every later stage, not only the last one. Card: Next-ViT.

Attention only in the last stage

The paper's comparison point puts a single attention stage at the end, with convolution in the first three stages 2207.05501 tab. 1. At the classification level it matches the late-stage design within about one point, but it scores lower on detection and segmentation 2207.05501 tab. 1.

Repeated convolution-then-attention pattern

Each stage can repeat the four-convolution, one-attention pattern more than once. The paper's two-repeat setting trades accuracy for latency against its four-repeat base model 2207.05501 tab. 2.

Convolution block choice

The paper's convolution block is a grouped-convolution token mixer with a Transformer-style MLP. In its own ablation it beats a ConvNeXt-style block at about the same latency on all three reported tasks 2207.05501 tab. 7.

Normalisation and activation

BatchNorm with ReLU runs faster than LayerNorm with GELU on the paper's runtimes, at a small accuracy difference 2207.05501 tab. 9. The paper calls that difference negligible.

Cheaper attention

The attention block reduces channels before attending and pools the keys and values, so it attends over fewer tokens than full self-attention 2207.05501 §3.3. A shrink ratio of 0.75 is faster than 1.0 at a small accuracy cost 2207.05501 tab. 8.

When to use which

Latency numbers in the paper come from one setup per runtime. Compare designs on your own target before choosing.

Implementation sketch

Common core of a late-stage hybrid stage. Variants differ in the attention block: Next-ViT adds a second branch with a convolution inside it 2207.05501 §3.3.

# Late-stage hybrid stage  [arxiv:2207.05501 eq. 7]
# Input: x (B, C, H, W); n_conv convolution blocks, then one attention block
def hybrid_stage(x, n_conv):
    # 1. Downsample the feature map before the stage  [arxiv:2207.05501 tab. 3]
    x = patch_embed(x)
    # 2. Convolution blocks mix local tokens; BatchNorm and ReLU, no LayerNorm
    #    [arxiv:2207.05501 §3.2] [arxiv:2207.05501 tab. 9]
    for _ in range(n_conv):
        x = conv_block(x)
    # 3. One attention block adds global context; keys and values are pooled
    #    before attention to cut the cost  [arxiv:2207.05501 §3.3]
    x = attention_block(x)
    return x

Contradictions

None known. Only one card covers this concept, so no two papers in the wiki disagree yet.

Papers

• Next-ViT (arXiv:2207.05501): a late-stage CNN-Transformer hybrid backbone measured on TensorRT and CoreML, with design ablations on placement, block choice, normalisation and attention cost.