---
kind: concept
slug: cnn-transformer-hybrid
title: CNN-Transformer hybrid
summary: >-
  A vision backbone that stacks convolution blocks and adds a few attention blocks for global context, aiming for Transformer-style accuracy at convolution-style latency. Covered so far by one paper, Next-ViT, measured on TensorRT and CoreML. Its ablations give the trade-offs below.
tags: [vision-backbone, hybrid-architecture, deployment-latency]
papers: ["2207.05501"]
related: [vision-transformer, hierarchical-vision-transformer, efficient-vision-backbone, convolutional-neural-network]
status: draft
updated_at: 2026-11-06
---

## Overview

A CNN-Transformer hybrid is a vision backbone that runs convolution blocks for
most of its depth and places a smaller number of attention blocks where global
context is needed. The motivation is deployment cost: the paper reports that
multi-head self-attention, LayerNorm and GELU are slow on TensorRT and CoreML,
while plain convolutions are fast on those runtimes but less accurate
[arxiv:2207.05501 §1]. The shared idea across designs is that convolution does
the local work and the cost, and attention is kept to a few positions.

Only one card covers this pattern so far. The variants below are therefore the
design choices that one paper varies, plus its ablations, not a survey of
hybrid architectures.

## Variants

### Late-stage attention (Next-ViT's hybrid strategy)

Stage 1 is convolution only. Stages 2 to 4 each repeat several convolution
blocks and end with one attention block [arxiv:2207.05501 §3.4] [arxiv:2207.05501 eq. 7].
Attention therefore reaches every later stage, not only the last one.
Card: [Next-ViT](/papers/2207.05501).

### Attention only in the last stage

The paper's comparison point puts a single attention stage at the end, with
convolution in the first three stages [arxiv:2207.05501 tab. 1]. At the
classification level it matches the late-stage design within about one point,
but it scores lower on detection and segmentation [arxiv:2207.05501 tab. 1].

### Repeated convolution-then-attention pattern

Each stage can repeat the four-convolution, one-attention pattern more than once.
The paper's two-repeat setting trades accuracy for latency against its
four-repeat base model [arxiv:2207.05501 tab. 2].

### Convolution block choice

The paper's convolution block is a grouped-convolution token mixer with a
Transformer-style MLP. In its own ablation it beats a ConvNeXt-style block at
about the same latency on all three reported tasks [arxiv:2207.05501 tab. 7].

### Normalisation and activation

BatchNorm with ReLU runs faster than LayerNorm with GELU on the paper's
runtimes, at a small accuracy difference [arxiv:2207.05501 tab. 9]. The paper
calls that difference negligible.

### Cheaper attention

The attention block reduces channels before attending and pools the keys and
values, so it attends over fewer tokens than full self-attention
[arxiv:2207.05501 §3.3].
A shrink ratio of 0.75 is faster than 1.0 at a small accuracy cost
[arxiv:2207.05501 tab. 8].

## When to use which

Latency numbers in the paper come from one setup per runtime. Compare designs
on your own target before choosing.

| Situation | Use | Why |
|---|---|---|
| Target is TensorRT or CoreML and latency is the budget | Late-stage hybrid with BatchNorm and ReLU | Next-ViT-S reaches 82.5% top-1 at 7.7 ms TensorRT at 224 px [arxiv:2207.05501 tab. 4] |
| Classification only, fixed latency | Placement matters little | Attention only in the last stage gives 82.4% top-1 at 7.8 ms, against 82.3% at 7.7 ms for the late-stage design [arxiv:2207.05501 tab. 1] |
| Detection or segmentation | Attention in stages 2 to 4 | Late-stage hybrid gives 45.5 and 46.0 against 44.7 and 45.2 for the last-stage-only design [arxiv:2207.05501 tab. 1] |
| Dense tasks under a latency budget | Measure first | Backbone latency at 512 px is higher than ResNet101 on both runtimes, 38.2 against 32.8 ms on TensorRT [arxiv:2207.05501 tab. 5] [arxiv:2207.05501 tab. 6] |
| Accuracy matters more than latency | Deeper stages | Four repeats give 83.2% at 10.5 ms, against 82.5% at 7.7 ms for two repeats [arxiv:2207.05501 tab. 2] |
| Hardware other than T4 with TensorRT or iPhone with CoreML | Measure on that hardware | The paper measures only these two targets and notes that rankings may change elsewhere [arxiv:2207.05501 §4.1.1] |

## Implementation sketch

Common core of a late-stage hybrid stage. Variants differ in the attention
block: Next-ViT adds a second branch with a convolution inside it
[arxiv:2207.05501 §3.3].

```pseudocode
# Late-stage hybrid stage  [arxiv:2207.05501 eq. 7]
# Input: x (B, C, H, W); n_conv convolution blocks, then one attention block
def hybrid_stage(x, n_conv):
    # 1. Downsample the feature map before the stage  [arxiv:2207.05501 tab. 3]
    x = patch_embed(x)
    # 2. Convolution blocks mix local tokens; BatchNorm and ReLU, no LayerNorm
    #    [arxiv:2207.05501 §3.2] [arxiv:2207.05501 tab. 9]
    for _ in range(n_conv):
        x = conv_block(x)
    # 3. One attention block adds global context; keys and values are pooled
    #    before attention to cut the cost  [arxiv:2207.05501 §3.3]
    x = attention_block(x)
    return x
```

## Contradictions

None known. Only one card covers this concept, so no two papers in the wiki
disagree yet.

## Papers

- [Next-ViT](/papers/2207.05501) (arXiv:2207.05501): a late-stage CNN-Transformer hybrid backbone measured on TensorRT and CoreML, with design ablations on placement, block choice, normalisation and attention cost.