Conformer: Local Features Coupling Global Representations for Visual Recognition

In plain words

Image models come in two families. Convolutional networks look at small neighbourhoods of pixels and stack those views. They are good at edges and texture but relate distant parts of an image poorly. Transformers let every patch of an image look at every other patch, which captures the overall layout, but they tend to lose fine local detail.

Conformer does not pick one. It runs a convolutional branch and a transformer branch side by side on the same image, and at every stage the two exchange what they have learned. The convolutional side passes local detail to the transformer, and the transformer passes its global view back. The result is meant as a general backbone for classification, detection and segmentation. The reported gains are the authors' own measurements.

TL;DR

Conformer runs a ResNet-style CNN branch and a ViT-style transformer branch in parallel, from a shared stem. A Feature Coupling Unit (FCU) at each block moves information both ways: CNN features are pooled into patch tokens, and transformer tokens are upsampled back into the CNN. Author-reported ImageNet-1k top-1: Conformer-S 83.4% and Conformer-B 84.1%, against DeiT-B 81.8% tab. 2. The official code (Apache-2.0) matches the paper's architecture, but the detection learning rate and batch size differ, and several training values are set in code only. See Paper vs code.

Problem

• CNNs build features hierarchically from local neighbourhoods but capture long-range relations poorly. Enlarging the receptive field needs pooling, which lowers spatial resolution §1.
• Visual transformers capture long-distance dependencies with self-attention over patch embeddings, but they tend to lose local detail. The paper argues this weakens the contrast between object and background §1 fig. 1.
• Earlier hybrids either add attention to a CNN, which can deteriorate local detail if fused badly §2, or stack a CNN in front of a transformer. The paper reports that its concurrent design beats such a serial hybrid (ResNet-26d/50d followed by DeiT-S) at similar cost §4.4 "Dual Structure" tab. 5.
• The ImageNet results use 224×224 inputs tab. 2. The detection results use COCO with the mmdetection library §4.3.

Method

Overview. A stem, two concurrent branches of N blocks (N = 12 in the examples given), FCUs that link the branches, and two classifiers. The branches are not stacked one after the other; they exchange features inside each block pair §3.1 fig. 2(c).

Stem. A 7×7 convolution with stride 2, then a 3×3 max pool with stride 2. At a 224 input this gives 56×56 maps §3.2 tab. 1 [code: conformer.py#L326-L331].

CNN branch. Four stages with decreasing resolution and increasing width. Each block has bottlenecks (1×1 reduce, 3×3, 1×1 expand, residual) §3.2. The paper sets n_c = 1 in the first convolution block and n_c ≥ 2 in later ones §3.2. The code builds the first block with one bottleneck and each later block with two, one before and one after the fusion step [code: conformer.py#L261-L266] conformer.py.

Transformer branch. A 4×4 convolution with stride 4 turns the stem output into 14×14 patch embeddings with no overlap. A learnable class token is prepended. There are no positional embeddings, because the CNN branch is said to supply location cues §3.2 tab. 6. Each block is pre-norm: layer norm, multi-head self-attention, residual, layer norm, MLP, residual §3.2 [code: conformer.py#L55-L72].

Feature Coupling Unit (FCU). The FCU bridges the two branches in both directions §3.2 fig. 2(a)(b).

• CNN to transformer (down). A 1×1 convolution aligns channels to the embedding size E. Average pooling then reduces the map to the token grid tab. 7. The result is added to the patch tokens. The code also applies a layer norm and an activation after pooling conformer.py.
• Transformer to CNN (up). The class token is dropped and the tokens are reshaped to the grid. A 1×1 convolution aligns channels to C/4. The code then applies batch norm and ReLU, and nearest-neighbour upsampling tab. 7 conformer.py.
• Insertion point. The upsampled features are added to the CNN features inside the block's second bottleneck, after its first 1×1 convolution and before its 3×3 convolution conformer.py. The text does not state the insertion point.
• Placement. FCU is applied from the second block, because both branches start from the same features §3.2. The code applies one per block, except the first conformer.py.

Downsampling. The CNN branch halves resolution at block 5 and block 9 (through the 3×3 convolution) and again at the last fusion step. The transformer grid stays at 14×14 throughout, so the CNN and transformer tokens stay aligned tab. 1 conformer.py. The last block's fusion bottleneck has stride 2, which gives the final 7×7 CNN map [code: conformer.py#L263-L264].

Heads. The CNN features are globally pooled and sent to a linear layer. The class token, after a final layer norm, goes to a second linear layer §3.1 conformer.py. Training uses one cross-entropy loss per head, weighted equally §3.1 engine.py. At test time the two logits are summed engine.py.

Stage schedule (S variant, 224 input, patch 16). Derived from the code and matched to tab. 1.

Variants §4.1 tab. 1 tab. 12 models.py.

Widths are from the code: base_channel 64 × channel_ratio, then doubled per stage. The S/32 variant uses 32×32 patches, giving a 7×7 token grid §4.1 models.py.

Training objective and inference.

# Two-head training loss  [§3.1] [code: engine.py#L38-L43]
# Input:  images (B, 3, 224, 224); targets (B,) or soft targets (B, K)
# Output: scalar loss

def training_loss(model, images, targets, criterion):
    # 1. Run both heads: CNN logits and class-token logits  [§3.1]
    conv_logits, trans_logits = model(images)
    # 2. Same criterion on each head, weighted 1/2 each  [§3.1]
    loss_conv = criterion(conv_logits, targets) / 2
    loss_trans = criterion(trans_logits, targets) / 2
    return loss_conv + loss_trans

# One Conformer block after the first  [§3.2] [code: conformer.py#L287-L303]
# Input:  cnn (B, C, H, W), tokens (B, 1 + h*w, E)
# Output: cnn (B, C2, H2, W2), tokens (B, 1 + h*w, E)

def conformer_block(cnn, tokens):
    # 1. First CNN bottleneck; keep its activation after the 3x3 conv  [§3.2]
    cnn, cnn_mid = bottleneck(cnn)              # cnn_mid: (B, C/4, H, W)
    # 2. CNN to transformer: 1x1 conv to E, average-pool to h x w,
    #    layer norm, GELU  [§3.2] [code: conformer.py#L159-L163]
    from_cnn = conv1x1(cnn_mid)                 # (B, E, H, W)
    from_cnn = avg_pool(from_cnn, stride_to=(h, w))
    from_cnn = gelu(layer_norm(flatten(from_cnn)))
    # 3. Prepend the class token and add to the patch tokens, then run one
    #    transformer block  [§3.2] [code: conformer.py#L165] [code: #L294]
    from_cnn = concat(tokens[:, :1], from_cnn, dim=1)
    tokens = transformer_block(from_cnn + tokens)
    # 4. Transformer to CNN: drop the class token, reshape to h x w,
    #    1x1 conv to C/4, batch norm, ReLU, nearest upsample  [§3.2]
    #    [code: conformer.py#L183-L189]
    to_cnn = tokens[:, 1:]
    to_cnn = reshape(to_cnn, (B, E, h, w))
    to_cnn = relu(batch_norm(conv1x1(to_cnn)))
    to_cnn = upsample_nearest(to_cnn, (H, W))
    # 5. Second CNN bottleneck: add to_cnn after its first 1x1 conv,
    #    before its 3x3 conv  [§3.2] [code: conformer.py#L116]
    cnn = bottleneck_with_add(cnn, to_cnn)
    return cnn, tokens

At inference the two logits are summed and the argmax is taken [code: engine.py#L100-L102]. The paper says the outputs are "simply summarized" §3.1.

Attention-based sampling (appendix, not in the released code). An alternative to pooling and interpolation. Down: patch embeddings of the transformer attend over CNN patches, using scaled dot products with queries from the transformer side, and the result is added to the tokens eq. 1. Up: the same attention weights, transposed, map the transformed CNN patches back eq. 2 app. B. This variant gives comparable accuracy with more parameters and compute tab. 7.

Hyperparameters

ImageNet-1k classification §4.2 tab. 2. The paper says it follows the DeiT augmentation and regularization recipe but does not list the values. Values below come from the released script and defaults.

COCO detection and instance segmentation §4.3. The paper reports 12 epochs (1x schedule) with decay at epochs 8 and 11, but the decay factor is not stated in the paper. The repository has only Conformer-S/32 configs mask_rcnn_conformer_small_patch32_fpn_1x_coco.py.

Results

Author-reported. Setup: ImageNet-1k train, validation top-1, 224×224 input tab. 2. COCO train2017 with evaluation on minival §4.3 tab. 3. Baselines are from the same paper's tables; the † marks ResNet numbers taken from mmdetection tab. 3 caption.

Headline numbers §4.2 §4.3.

• Classification: Conformer-S 83.4% vs ResNet-152 78.3% (+4.1) and DeiT-B 81.8% (+1.6). Conformer-B 84.1% vs DeiT-B (+2.3). Conformer converges faster in the first 50 epochs app. F.
• Detection: Conformer-S/32 in Faster R-CNN reaches 43.1 box AP vs 39.4 for ResNet-101 FPN (+3.7).
• Instance segmentation: Conformer-S/32 in Mask R-CNN reaches 39.7 mask AP vs 36.1 for ResNet-101 (+3.6).

ImageNet-1k top-1 tab. 2 (full table from the paper).

COCO val (minival) tab. 3 (full table from the paper; box AP (AP^bbox) and mask AP (AP^segm), with S/M/L sizes).

The FPN rows are the paper's label; the repo's configs for them are Faster R-CNN, and the paper does not name the detector [code: mmdetection/configs/faster_rcnn/faster_rcnn_conformer_small_patch32_fpn_1x_coco.py#L5-L6].

Ablations (ImageNet top-1 unless stated; author-reported).

Parameter-proportion results tab. 4: more CNN parameters bring the larger gain. Changing n_c also helps: n_c = 4 at C = 192 gives 83.3 (vs 82.8 at n_c = 2 with C = 192), and n_c = 3 at C = 256 gives 83.5 (vs 83.4 at n_c = 2).

Inference speed app. C tab. 9. On COCO, FPS on a V100 at batch 1: Conformer-S/32 (Faster R-CNN) 13.5 vs ResNet-101 15.9; Conformer-S/32 (Mask R-CNN) 10.9 vs 11.5. The paper calls the speed "comparable" and the parameter and GFLOP counts similar.

Robustness §4.5 fig. 5. On rotated test images, ResNet-101 drops significantly while Conformer-S holds up better. On a 224 to 112 resolution change, DeiT-S drops 25%, ResNet-50/152 drop 15%, and Conformer drops 10%. The figure values are not given in the text.

Paper vs code

Code: raw/code/2105_03889 at commit 815aaad3ef5dbdfcf1e11368891416c2d7478cb1 (official repo linked in the abstract, Apache-2.0). Its vendored mmdetection fork is also Apache-2.0.

• Architecture matches. Stem, branch widths, block counts, patch size, class token, no positional embeddings, and pre-norm blocks agree §3.2 with conformer.py conformer.py.
• FCU normalization differs in placement. The paper says LayerNorm and BatchNorm align values in both directions §3.2. The code uses LayerNorm after the CNN-to-transformer projection conformer.py and BatchNorm after the transformer-to-CNN projection conformer.py.
• FCU up-path order differs. The paper describes upsampling, then a 1×1 convolution §3.2. The code does the convolution first, then upsamples [code: conformer.py#L186-L189]. For nearest-neighbour upsampling these operations commute, so the output should match. This was not tested.
• The class token is doubled in each FCU. The FCU prepends the class token taken from the transformer side conformer.py, then the sum with the tokens is fed to the transformer block conformer.py. The class-token slot is therefore 2× its value on entry, and the residual path carries the doubled value forward. The paper does not describe class-token handling in the FCU. Effect not measured here.
• Detection learning rate and batch size differ. The paper states lr 2e-4 at batch 32 §4.3. The config uses lr 1e-4 and 2 images per GPU [code: mmdetection/configs/mask_rcnn/mask_rcnn_conformer_small_patch32_fpn_1x_coco.py#L166] mask_rcnn_conformer_small_patch32_fpn_1x_coco.py. The GPU count that gives batch 32 is not stated.
• Only S/32 detection configs are in the repo. The paper's S/16 detection rows (Table 3) have no config here tab. 3.
• Attention-based sampling and convolutional sampling are not in the code. Only average pooling and nearest-neighbour interpolation are implemented conformer.py conformer.py. The paper's variants are ablations tab. 7 app. B.
• Extra bottlenecks are an option. The code can add bottlenecks per block through num_med_block, which is set to 0 in every registered variant [code: conformer.py#L268-L272] models.py. Table 4's n_c sweep is not reproduced by the registered models.
• Training values are not in the paper. Repeated augmentation, EMA, label smoothing, warmup, the stochastic-depth rate, and the RandAugment and erasing settings are code defaults main.py main.py main.py main.py main.py. Reported accuracy uses the raw model, not the EMA main.py.
• Depth must be a multiple of 3. conformer.py.
• Batch size and LR match for classification. Batch 1024 (128 × 8 GPUs) at lr 0.001 with no scaling run.sh main.py.

Pitfalls

• FCU details are underspecified. The paper does not give the insertion point, the normalization placement, or the activations (the code uses GELU on the down path and ReLU on the up path conformer.py conformer.py). Follow the code if you need exact numbers.
• Class token doubling. Implement it if you want to match the released model, or fix it deliberately. Either choice changes the class token's scale entering every block. The paper does not say which it intends.
• Internal numerical inconsistencies in the paper. Conformer-S MACs are 10.6 G in tab. 1, tab. 2 and tab. 4, but 10.3 G in tab. 7 and tab. 8. DeiT-S MACs are 4.6 G in tab. 2 but 4.2 G in tab. 8. Conformer-S with positional embeddings is 83.4% in tab. 2 but 83.5% in tab. 6.
• Appendix B has a notation error. The sentence after eq. 2 writes P̃^i_c twice; the second symbol should be the transformer-side P̃^j_t app. B.
• Stochastic depth applies only to transformer blocks. The rate rises linearly from 0 to the maximum across the 12 blocks conformer.py conformer.py conformer.py. The CNN branch has none conformer.py.
• Mixup and label smoothing are coupled. With mixup on, the loss is soft-target cross-entropy and label smoothing is applied through the mixup targets. With mixup set to 0, the criterion becomes label-smoothing cross-entropy main.py main.py.
• Two branches cost more than MACs suggest. Detection FPS is lower than ResNet-101's tab. 9. Compare throughput on your own hardware, not MACs.
• Detection numbers depend on the training recipe. The LR, batch size, warmup and frozen stages differ between the paper's text and the repo's configs (see Paper vs code). Which setting reproduces Table 3 is not verified here.
• Baselines are partly borrowed. The ResNet detection numbers are from mmdetection tab. 3 caption.

Lineage

• Builds on: ResNet (residual bottlenecks and the CNN branch), ViT and DeiT (the transformer branch and the DeiT training recipe), and FPN for the detection neck §3.2 §4.3. None of these has a card in this wiki, so they are cited by name only.
• Compared against: ResNet and RegNetY tab. 2, ViT, T2T-ViT and DeiT tab. 2, EfficientNet for throughput app. C fig. 6, serial hybrid ResNet+DeiT tab. 5, and two-model ensembles tab. 8.
• Superseded by: not known from content in this wiki.