Paper · 2023-07-17 · cs.CV
Scale-Aware Modulation Meet Transformer
SMT is a hybrid vision backbone. Stages 1-2 use Scale-Aware Modulation: multi-head depth-wise convolutions with growing kernels, a cross-head aggregation step, then an element-wise gate on a value branch. Stage 3 alternates modulation and self-attention, and stage 4 uses attention only. Author-reported: 82.2% ImageNet-1K top-1 at 11.5M / 2.4G FLOPs, 84.3% at 32M / 7.7G.
Vision Transformer (ViT) backbonesCNN-Transformer hybridLocal-global attentionDepthwise separable convolutionvision-transformerhybrid-backboneconvolutional-modulationimage-classificationobject-detectionsemantic-segmentation
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Image models have two common building blocks. Convolutions look at small neighbourhoods and are cheap. Transformer attention can relate any two parts of an image but costs much more, and that cost grows fast with image resolution. This paper builds a backbone that uses convolution-style layers where the image is still large and detailed, and attention only where the picture is small and the model needs global context.
The new piece is a "modulation" layer. Instead of computing attention scores, it computes a gate from several convolutions that look at the image at different sizes. The gate then scales a value signal, element by element. The gate is split into groups so that each group mixes one channel from every scale, which lets the scales share information cheaply.
The network runs in four stages that shrink the image step by step. The early stages use the gate, the last stage uses attention, and the stage in between alternates the two, so the model moves from local to global reasoning as it gets deeper. The authors report that this design matches or beats several larger models on classification, object detection and segmentation.
TL;DR
SMT replaces attention in the first two stages with Scale-Aware Modulation, built from Multi-Head Mixed Convolution (depth-wise convolutions with kernels 3, 5, 7, 9 for four heads) and Scale-Aware Aggregation (a grouped, then full, 1x1 convolution) §3.2 eq. 1 eq. 2. The third stage alternates modulation blocks with self-attention blocks, and the fourth stage uses attention only §3.1 §3.3 tab. 13. Author-reported, SMT-T reaches 82.2% ImageNet-1K top-1 and SMT-B 84.3% tab. 1. The official code is MIT-licensed, but several details differ from the paper, so read the Paper vs code section before implementing.
Problem
Self-attention over visual tokens has quadratic cost in the number of tokens, which makes high-resolution inputs and dense downstream tasks expensive §1. Earlier fixes limit attention to windows, pool keys, or shrink tokens §1 §2. Hybrid CNN-Transformer networks usually replace shallow Transformer blocks with convolutions, but the authors report that plain convolution in the shallow stages does not reach the desired accuracy, and that convolutional modulation is stronger §1. They also argue that modulation alone cannot capture long-range dependencies, which motivates adding attention back at depth §1 §3.3. The paper assumes ImageNet-1K classification and COCO and ADE20K transfer as the evaluation setting §4.
Method
Overall architecture §3.1 fig. 2. The network has four stages with downsampling rates 4, 8, 16 and 32. Stages 1 and 2 contain only Scale-Aware Modulation (SAM) blocks. Stage 3 (the "penultimate" stage) stacks SAM and Multi-Head Self-Attention (MSA) blocks in alternation. Stage 4 uses MSA blocks only §3.1 §3.3. The feed-forward layer of each block is a detail-specific feed-forward network, as in Shunted §3.1. Each block is pre-norm with residual connections and stochastic depth around both its token mixer and its feed-forward layer smt.py.
Scale-Aware Modulation §3.2 fig. 3. A SAM block computes an output Z from the input X as an element-wise product of a modulator M and a value V eq. 3. The value is a linear map of X. The modulator comes from a linear map of X, followed by MHMC and then SAA eq. 3.
Multi-Head Mixed Convolution (MHMC) §3.2 eq. 1. The channels are split into N heads. Head i receives a depth-wise convolution with kernel size 3 + 2(i-1), so kernels grow by 2 per head: 3, 5, 7, 9 for four heads §3.2 app. A. The head outputs are concatenated eq. 1.
Scale-Aware Aggregation (SAA) §3.2 eq. 2 fig. 3. SAA groups the MHMC outputs so that each group holds one channel from every head. There are C / N groups. Within each group, a point-wise convolution expands the channels and a second one projects them back (an inverted-bottleneck up-down fusion). A final point-wise convolution mixes all groups eq. 2 §3.2. The paper describes the fusion as shuffle-and-group; the official code implements the grouping by channel interleaving (see Pitfalls).
# Scale-Aware Modulation (SAM) block [§3.2] [eq. 1-3]
# Input: x (B, N, C), token grid H x W with N = H * W
# Output: z (B, N, C)
def scale_aware_modulation(x, heads):
# Value branch V = W_v X [eq. 3]
values = linear_v(x) # (B, N, C)
# Modulator input W_s X, laid out on the token grid [eq. 3]
feats = reshape(linear_s(x), (B, C, H, W)) # (B, C, H, W)
# MHMC: split channels into `heads` chunks; chunk i gets a
# depth-wise conv with kernel 3 + 2 * i [§3.2] [eq. 1]
chunks = split(feats, heads, dim=1)
chunks = [dwconv(c, kernel=3 + 2 * i) for i, c in enumerate(chunks)]
# SAA: group k holds channel k of every head, C / heads groups
# [§3.2] [eq. 2] [code: models/smt.py#L116-L124]
feats = regroup(chunks, groups=C // heads) # (B, C, H, W)
# Intra-group fusion: 1x1 conv with groups = C / heads, expands x2
# [eq. 2]. BatchNorm and GELU are not in eq. 2 [code: models/smt.py#L125]
feats = gelu(batch_norm(grouped_conv1x1(feats))) # (B, 2C, H, W)
# Cross-group fusion: full 1x1 conv back to C channels [eq. 2]
modulator = conv1x1(feats) # (B, C, H, W)
modulator = reshape(modulator, (B, N, C)) # (B, N, C)
# Element-wise modulation of the values [eq. 3]
z = modulator * values # (B, N, C)
# Output linear projection: code only, not in eq. 3
# [code: models/smt.py#L140]
return linear_out(z)Modulation vs. attention §3.2. Unlike self-attention, the modulator keeps the channel dimension and does not build an N x N map. The authors argue this gives spatially and channel-specific modulation with lower memory use at high resolution §3.2.
Multi-Head Self-Attention (MSA) §3.3 tab. 13. Standard scaled dot-product attention with 8 heads in stage 3 and 16 heads in stage 4 tab. 13. The official code also adds a 3x3 depth-wise convolution of the values to the attention output smt.py. The paper text does not mention it.
Evolutionary Hybrid Network (EHN) §3.3 fig. 6. In stage 3 the paper compares two stacking rules. The chosen rule, (SAM x 1 + MSA x 1) x N/2, alternates the two block types. The alternative, SAM x N/2 followed by MSA x N/2, is the second option §3.3 tab. 9. The official code uses the alternating rule: blocks with even index in stage 3 are SAM, odd index are MSA smt.py.
Feed-forward layer §3.1 smt.py. Linear layer, then a depth-wise 3x3 convolution of the hidden features added back before the GELU, then a linear layer. The paper names the layer only as "detail-specific", so the exact form comes from the code.
Classification head smt.py. After the last stage, tokens are averaged and passed to a single linear layer. The paper does not state the pooling.
Hyperparameters
Architecture (paper values are from tab. 13 unless noted):
| Name | Value | Where | Notes |
|---|---|---|---|
| stage widths | T, S, B: 64 / 128 / 256 / 512; L: 96 / 192 / 384 / 768 | tab. 13 smt.py | |
| depths, stages 1-4 | T: 2, 2, 8, 1; S: 3, 4, 18, 2; B: 4, 6, 28, 2; L: 4, 6, 28, 3 | tab. 13 | Code uses 4 for L stage 4, see Paper vs code |
| stem conv kernel | T: 3x3; S: 7x7; B: 7x7; L: 7x7 | tab. 13 | Code uses 3x3 for S, see Paper vs code |
| MHMC heads | 4 (kernels 3, 5, 7, 9) | tab. 13 app. A | Ablation peak at 4 heads tab. 7 |
| SAA expansion ratio | 2 | tab. 13 | |
| MSA heads | 8 in stage 3, 16 in stage 4 | tab. 13 | |
| MLP ratio per stage | not stated in paper | smt.py | T, S: 4, 4, 4, 2; B, L: 8, 6, 4, 2 |
| QKV bias | not stated in paper | smt_tiny_224.yaml | True in every config |
| stochastic depth (drop path) | T 0.1; S 0.2; B 0.3; L 0.5 | app. B.1 tab. 11 | Code uses 0.4 for L, see Paper vs code |
| layer scale | not stated in paper | smt_large_224.yaml | Only on L; init 1e-4 |
Classification on ImageNet-1K §4.1 app. B.1 tab. 11:
| Name | Value | Where | Notes |
|---|---|---|---|
| optimizer | AdamW, betas (0.9, 0.999) | tab. 11 | eps not stated in paper; code uses 1e-8 config.py |
| weight decay | 0.05 | §4.1 tab. 11 | Code excludes biases and 1-D parameters optimizer.py |
| total batch size | 1024 | §4.1 tab. 11 | README: 8 GPUs x 128 [README] |
| base learning rate | text: 1e-3; table: 2e-3 | §4.1 tab. 11 | Text value is before linear scaling, see Pitfalls |
| warmup | 5 epochs, from 1e-6 | tab. 11 | Code default warmup start is 1e-7, see Paper vs code |
| schedule | cosine, minimum LR 1e-5 | tab. 11 | Code default minimum LR is 1e-6, see Paper vs code |
| epochs | 300 | §4.1 tab. 11 | |
| gradient clip (norm) | 5.0 | tab. 11 | config.py |
| label smoothing | 0.1 | tab. 11 | config.py |
| augmentation | RandAugment rand-m9-mstd0.5-inc1; color jitter 0.4; mixup alpha 0.2; CutMix alpha 1.0; random erasing 0.25 | tab. 11 | Mixup switch probability is 0.5 in code, commented out in the paper table config.py |
ImageNet-22K pre-training and fine-tuning, SMT-L app. B.2 tab. 2:
| Name | Value | Where | Notes |
|---|---|---|---|
| pre-training | 90 epochs, batch 4096, LR 1e-3, 5 warmup epochs, drop path 0.1 | app. B.2 | smt_large_224_22k.yaml smt_large_224_22k.yaml |
| fine-tuning | 30 epochs, LR 2e-5, drop path 0.1, cosine, AdamW | app. B.2 | Paper says Mixup and CutMix are disabled, see Paper vs code |
Detection and instance segmentation, COCO §4.2 app. B.3 tab. 12:
| Name | Value | Where | Notes |
|---|---|---|---|
| optimizer | AdamW, LR 2e-4, weight decay 0.05, betas (0.9, 0.999) | §4.2 tab. 12 | mask_rcnn_smt_b_fpn_mstrain-poly_3x_coco.py |
| batch size | 16 | tab. 12 | Code: 2 per GPU mask_rcnn_smt_b_fpn_mstrain-poly_3x_coco.py |
| Mask R-CNN 1x | 12 epochs, LR steps at 8 and 11 | §4.2 tab. 12 | |
| Mask R-CNN 3x | 36 epochs, LR steps at 27 and 33, shorter side randomly in [480, 800] | §4.2 tab. 12 | mask_rcnn_smt_b_fpn_mstrain-poly_3x_coco.py |
| drop path | S 0.2; B 0.3 | tab. 12 | 1x base config uses 0.2, see Paper vs code |
| warmup | 500 iterations, ratio 0.001 | tab. 12 |
Semantic segmentation, ADE20K §4.3 app. B.4 tab. 6:
| Name | Value | Where | Notes |
|---|---|---|---|
| framework | UperNet, 160k iterations, 512x512 input | §4.3 | |
| optimizer | AdamW, LR 6e-5, weight decay 0.01, batch 16 | §4.3 app. B.4 | upernet_smt_512x512_160k_ade20k.py |
| drop path | same as detection | app. B.4 |
Results
Author-reported. Setup: ImageNet-1K classification at 224 x 224 with the recipe above §4.1; COCO 2017 with Mask R-CNN §4.2; ADE20K with UperNet §4.3. Baseline names and numbers are copied from the paper's tables.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| ImageNet-1K, tiny (224) | top-1 | PVTv2-b1 78.7; Shunted-T 79.8 | SMT-T 82.2 (11.5M, 2.4G) | tab. 1 |
| ImageNet-1K, small (224) | top-1 | Conv2Former-T 83.2; CMT-S 83.5; MaxViT-T 83.6 | SMT-S 83.7 (20.5M, 4.7G) | tab. 1 |
| ImageNet-1K, base (224) | top-1 | Swin-B 83.4 (87.8M, 15.4G); ConvNeXt-B 83.8; HorNet-B 84.2 | SMT-B 84.3 (32.0M, 7.7G) | tab. 1 |
| ImageNet-1K, large, 1K-trained (224) | top-1 | Shunted-B 84.0 (39.6M, 8.1G); MaxViT-S 84.5 (69M, 11.7G); MaxViT-B 84.9 (120M, 23.4G) | SMT-L 84.6 (80.5M, 17.7G) | tab. 14 |
| ImageNet-1K, medium (224) | top-1 | PVT-T 75.1 (13M, 1.8G) | SMT-M 78.4 (6.5M, 1.3G) | tab. 14 |
| ImageNet-22K pre-train, 224 then 224 | top-1 | FocalNet-Large 86.5 (197.1M, 34.2G); Swin-Large 86.3 | SMT-L 87.1 (80.5M, 17.7G) | tab. 2 |
| ImageNet-22K pre-train, 224 then 384 | top-1 | InternImage-XL 88.0 (335M, 163G) | SMT-L 88.1 (80.5M, 54.6G) | tab. 2 |
| COCO Mask R-CNN 1x | box / mask AP | Swin-T 42.2 / 39.1; Swin-B 46.9 / 42.3 | SMT-S 47.8 / 43.0; SMT-B 49.0 / 44.0 | tab. 3 |
| COCO Mask R-CNN 3x + MS | box / mask AP | Swin-T 46.0 / 41.6; Swin-B 48.5 / 43.4 | SMT-S 49.0 / 43.4; SMT-B 49.8 / 44.0 | tab. 3 |
| COCO other detectors (3x) | box AP | Swin-T: Cascade 50.5, RetinaNet 45.0, Sparse R-CNN 47.9, ATSS 47.2, DINO 51.3 | SMT-S: Cascade 51.9, RetinaNet 47.3, Sparse R-CNN 50.2, ATSS 49.9, DINO 54.0 | tab. 4 tab. 5 |
| ADE20K UperNet, single / multi-scale | mIoU | Swin-B 48.1 / 49.7; Focal-B 49.0 / 50.5 | SMT-B 49.6 / 50.6 (61.8M, 1004G) | tab. 6 |
Ablations on ImageNet-1K §4.4 tab. 7 tab. 8 tab. 9 tab. 10:
| Study | Setting | top-1 | Source |
|---|---|---|---|
| MHMC heads | 1 / 2 / 4 / 6 / 8 | 81.8 / 82.0 / 82.2 / 81.9 / 82.0 | tab. 7 |
| Aggregation | none / single linear / two linears / IBN / SAA | 81.5 / 81.6 / 81.9 / 82.1 / 82.2 | tab. 8 |
| Stacking, penultimate stage | SAM x N / MSA x N / (SAM+MSA) x N/2 / SAM x N/2 + MSA x N/2 | 81.4 / 81.8 / 82.2 / 81.9 | tab. 9 |
| Components, cumulative | attention-free baseline / +MHMC / +SAA / +EHN | 80.0 / 80.8 / 81.6 / 82.2 | tab. 10 |
Throughput in the heads ablation was measured on a V100 GPU, following Swin: 983 images/s for 1 head and 702 for 8 heads tab. 7.
Paper vs code
Code: https://github.com/AFeng-x/SMT at commit 11e954d9b4d0f513397ba011669c7864fd94e5fb. The repository is linked from the paper's abstract [abstract]. It is MIT-licensed LICENSE. The repository's LICENSE names Microsoft Corporation as copyright holder, which is inherited from the Swin Transformer template the code builds on.
- Stage 4 depth for L: paper uses 3 blocks tab. 13; the code uses 4 smt.py and smt_large_224.yaml. The 22K pre-training and fine-tuning configs also use 4 smt_large_224_22k.yaml.
- Stem for S: paper uses a 7x7 stem tab. 13; the code uses 3x3 smt.py and the segmentation config matches upernet_smt_s.py.
- Stochastic depth for L: paper uses 0.5 app. B.1; the code uses 0.4 smt_large_224.yaml.
- Warmup and minimum LR: paper gives warmup start 1e-6 and minimum 1e-5 tab. 11. The code default is warmup start 1e-7 config.py and minimum 1e-6 config.py, both multiplied by total batch / 512 main.py. At batch 1024 that gives 2e-7 and 2e-6. The 1K YAML files do not override these values.
- Base learning rate: the text gives 1e-3 §4.1; the table gives 2e-3 tab. 11. The code uses base 1e-3 scaled by total batch / 512 main.py, so 2e-3 at batch 1024. The two paper values agree once the scaling is applied.
- Drop path in detection and segmentation backbones: the paper calls the rate a "path drop rate" app. B.4. The detection and segmentation backbones use element-wise
nn.Dropouton the residual branches, not per-sample stochastic depth smt.py smt.py. The classification model usesDropPathsmt.py. - Base Mask R-CNN 1x: the paper gives drop path 0.3 for base tab. 12; this config uses 0.2 mask_rcnn_smt_b_fpn.py. The 3x config uses 0.3 mask_rcnn_smt_b_fpn_mstrain-poly_3x_coco.py.
- 22K to 1K fine-tuning: the paper says Mixup and CutMix are disabled app. B.2. The fine-tuning YAML does not override them, so the default Mixup 0.2 and CutMix 1.0 apply unless set on the command line smt_large_224_22kto1k_finetune.yaml config.py. The README commands do not set them [README].
- Code-only details, not in the paper text: BatchNorm and GELU inside the SAA block smt.py smt.py; an output linear projection after modulation smt.py; a 3x3 depth-wise convolution on V inside MSA smt.py; the depth-wise convolution inside the feed-forward layer smt.py; and a ReLU in the stem smt.py. The code uses no positional encoding; the paper does not mention one either way.
- MLP ratios: not in the paper; set per stage in the code smt.py.
- SMT-M: the paper reports it tab. 14, but the repository defines only
smt_t,smt_s,smt_bandsmt_lsmt.py. No SMT-M configuration is in the repository. - FLOPs for SMT-L at 384: the README reports 51.6G [README] and the paper 54.6G tab. 2. The README also reports 17.6G against the paper's 17.7G for L at 224.
- Weight decay exclusions: the paper does not say which parameters are exempt. The code exempts biases and all 1-D parameters optimizer.py.
- Pooling: the code averages tokens before the classifier smt.py; the paper does not state the pooling.
The detection and segmentation learning rates, weight decay, batch sizes and LR steps match the paper tab. 12 §4.3. The mismatches that remain are the drop-path and stem items listed above.
Pitfalls
- Which "Swin" the abstract compares to. On COCO Mask R-CNN 1x, the abstract's 4.2 box-mAP gap for the base model is measured against Swin-S (44.8), not Swin-B (46.9), which is in the same table tab. 3. Against Swin-B the gap is 2.1 tab. 3. The 3x gap of 1.3 is the same against Swin-S and Swin-B (48.5) tab. 3. On ADE20K, the abstract's 2.0 and 1.1 mIoU gains are over Swin-S (47.6 / 49.5) tab. 6; the text's gains over Swin-B are 1.5 and 0.9 §4.3. Use the table rows you need.
- Ablation gains are cumulative. In tab. 10 each arrow is measured against the attention-free baseline (80.0), not the previous row. The text describes the SAA step as a "1.6% increase" over the previous modification §4.4, but that is the cumulative gain. The step increase is 0.8 (80.8 to 81.6). Likewise the EHN's "2.2%" is cumulative; the step increase is 0.6 (81.6 to 82.2).
- Learning rate is stated before scaling. The main text's learning rate of 1e-3 §4.1 is the base value; the scaled value at batch 1024 is 2e-3 tab. 11 main.py. Using 1e-3 as the final rate halves the effective step size.
- Batch scaling of schedule values. Warmup and minimum LR are also scaled by batch / 512 in code main.py, so the code's own defaults are not the values in the paper table. Implement from the table if you want the paper's schedule; reproduce from the code if you want the released results.
- Grouping in SAA. The paper says channels are "shuffled and grouped" and that each group holds one channel per head §3.2. The code achieves this by interleaving the per-head outputs, so channel c = g * heads + i, where g is the group and i the head smt.py. A contiguous split of channels would change which channels form a group.
- Config mismatches change the model. Stage 4 depth of L (4 vs 3), the stem kernel of S (3x3 vs 7x7) and the drop path of L (0.4 vs 0.5) all differ between paper and code. Each changes block count, parameters or regularisation. Parameter and FLOP counts were not recomputed here, so the reported 80.5M / 17.7G cannot be checked against the code.
- Eq. 3 is incomplete for implementation. The equation gives Z = M ⊙ V with no output projection, but the code applies a linear layer after the product smt.py. Implementing eq. 3 alone gives a different model.
- Memory and throughput. Increasing the MHMC heads from 1 to 8 reduces throughput on a V100 from 983 to 702 images/s tab. 7. The heads add larger kernels, which the authors note can slow inference §4.4.
- Test-time scales. Multi-scale testing for ADE20K uses 0.5 to 1.75 times the training resolution app. B.4. The exact scales are not stated.
- Hidden details to copy from code. The MLP ratios, the depth-wise convolution in the feed-forward layer, the MSA value convolution and the output projection are needed to match the reported parameter counts. None of them is in the paper text.
Lineage
- Builds on: convolutional modulation as in FocalNet and Conv2Former, which the paper extends with multi-head mixed kernels and cross-head aggregation §2 §3.2. Hierarchical windowed Transformers, notably Swin, for the staged design and the training recipe §4.1 app. B.1. Shunted, for the feed-forward layer §3.1. The stage-wise local-to-global view follows the analysis in UViT and ViT §3.3.
- Compared against: ResNet, RegNet, EfficientNet, PVTv2, Swin, Focal, FocalNet, Shunted, Conv2Former, CoAtNet, ConvNeXt, HorNet, InternImage, MaxViT, DeiT and ViT on ImageNet-1K and ImageNet-22K tab. 1 tab. 2 tab. 14; Mask R-CNN, Cascade Mask R-CNN, RetinaNet, Sparse R-CNN, ATSS and DINO on COCO tab. 3 tab. 4 tab. 5; UperNet on ADE20K tab. 6.
- Superseded by: not known from wiki content.
Source and license
Adapted from “Scale-Aware Modulation Meet Transformer” by Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, Lianwen Jin (arXiv:2307.08579), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/AFeng-x/SMT at commit 11e954d9b4d0 (MIT). Any quoted code is shown under that license.