A ConvNet for the 2020s

In plain words

Vision Transformers and their hierarchical successors, such as Swin, became the default image backbones after 2020. The usual explanation is that Transformers are simply better. This paper asks a narrower question: how much of the gap comes from the architecture itself, and how much from the training recipe and the many small design choices that changed at the same time?

The authors take an ordinary ResNet, first train it with the same recipe modern Transformers use, and then change its design one piece at a time, copying ideas from Transformer blocks. The changes are things like a bigger filter, fewer activation functions and normalization layers, and a different layer for normalization. None of the pieces is new to convolutional networks, and every step is checked against accuracy. The result, ConvNeXt, uses only standard convolutions, has no attention, and still matches or beats Swin Transformers on image classification, object detection and segmentation. It is useful as a simple, convolutional backbone that is easy to implement and fine-tune.

TL;DR

ConvNeXt is a family of pure ConvNets built by modernizing a ResNet-50 with Transformer-era training and design choices, one measured step at a time. On ImageNet-1K at 224 pixels the smallest variant beats Swin-T (82.1% vs 81.3%), and the largest, pre-trained on ImageNet-22K and evaluated at 384 pixels, reaches 87.8%. The paper reports equal or better COCO and ADE20K results than Swin, all author-reported. Use it when you need a convolutional backbone that is as strong as a Swin Transformer, but simpler, and check the Pitfalls: the code differs from the paper in a few places.

Problem

Hierarchical Transformers such as Swin had matched or beaten ConvNets on dense tasks, and the paper attributes this to Transformers' intrinsic strength §1. The authors argue that architecture and training recipe changed together, so the comparison is confounded §1. Their stated goal is to find which design choices from Transformers account for the difference, and to test how far a pure ConvNet can go §1. Vanilla ViT is said to be hard to use as a generic backbone because of its global attention cost and missing convolutional priors §1.

Method

The network is a four-stage hierarchy. The paper derives it by starting from a ResNet-50 baseline and applying changes in five groups: training recipe §2.1, macro design §2.2, ResNeXt-style grouped/depthwise convolution §2.3, inverted bottleneck §2.4, large kernel size §2.5 and micro design §2.6. Each group is kept only if it improves accuracy on ImageNet-1K in the ResNet-50 regime tab. 10.

Final ConvNeXt block, in words §2.4 §2.5 §2.6 fig. 3 fig. 4: a 7x7 depthwise convolution mixes space per channel; a LayerNorm follows; then two 1x1 layers expand the channels four times and project them back, with a single GELU between them; a per-channel learned scale (layer scale) multiplies that output; and the result is added to the block input. Stochastic depth is applied to the residual branch. There is one normalization layer per block and only one activation in the block.

Downsampling is a separate layer: a LayerNorm and a 2x2 stride-2 convolution before each of the three later stages. The network starts with a 4x4 stride-4 "patchify" convolution followed by LayerNorm §2.2 §2.6. The head is global average pooling, a LayerNorm and a linear classifier §2.6 convnext.py.

Pseudocode of one block, following the paper and the code:

# ConvNeXt block  [§2.4-2.6] [fig. 4] [code: models/convnext.py#L26-L48]
# Input:  x of shape (B, C, H, W)
# Output: y of shape (B, C, H, W)

def convnext_block(x):
    # Keep the input for the residual connection  [fig. 4]
    shortcut = x
    # 7x7 depthwise convolution: one filter per channel  [§2.5]
    x = depthwise_conv_7x7(x)                  # (B, C, H, W)
    # Channels-last layout for the LayerNorm and linear layers  [code: L38-L40]
    x = move_channels_last(x)                  # (B, H, W, C)
    # One LayerNorm per block  [§2.6]
    x = layer_norm(x)                          # (B, H, W, C)
    # Inverted bottleneck: expand 4x  [§2.4]
    x = linear(x, out_channels = 4 * C)        # (B, H, W, 4C)
    # Single GELU activation in the block  [§2.6]
    x = gelu(x)                                # (B, H, W, 4C)
    # Project back to C channels  [§2.4]
    x = linear(x, out_channels = C)            # (B, H, W, C)
    # Layer scale: learned per-channel factor, initial value 1e-6
    # [app. A.1] [code: models/convnext.py#L33]
    x = gamma * x                              # (B, H, W, C)
    x = move_channels_first(x)                 # (B, C, H, W)
    # Stochastic depth on the residual branch  [app. A.1]
    return shortcut + drop_path(x)

Network skeleton, with the stage widths and depths from §3.1:

# ConvNeXt forward pass  [§3.1] [tab. 9] [code: models/convnext.py#L72-L112]
# Input:  images (B, 3, 224, 224)
# Output: logits (B, num_classes)

def convnext(images):
    # Stem: 4x4 non-overlapping conv, stride 4, then LayerNorm  [§2.6]
    x = conv(images, kernel = 4, stride = 4, out_channels = C1)
    x = layer_norm(x)                          # (B, C1, 56, 56)
    for stage in range(1, 5):
        # Stages 2-4: LayerNorm, then 2x2 stride-2 conv  [§2.6]
        if stage > 1:
            x = layer_norm(x)
            x = conv(x, kernel = 2, stride = 2, out_channels = C[stage])
        # Blocks of this stage  [§3.1]
        for block in blocks[stage]:
            x = convnext_block(x)
    # Global average pool, LayerNorm, linear head  [§2.6]
    x = layer_norm(mean(x, over = (H, W)))     # (B, C4)
    return linear(x, out_channels = num_classes)

Roadmap from ResNet-50 to ConvNeXt-T §2 tab. 10

Accuracy is ImageNet-1K top-1, ResNet-50 regime (about 4.5 GFLOPs), mean of three seeds §2.

1. Training recipe §2.1: the ResNet-50 baseline trained with a modern Transformer-style recipe goes from 76.1 (torchvision) to 78.8 (+2.7). This recipe is kept fixed for all later steps.
2. Stage compute ratio §2.2: block counts change from (3, 4, 6, 3) to (3, 3, 9, 3), matching Swin-T's 1:1:3:1 ratio. Result 79.4.
3. Patchify stem §2.2: a 4x4, stride-4, non-overlapping convolution replaces the ResNet stem. Result 79.5.
4. Depthwise convolution and wider network §2.3: depthwise convolution lowers accuracy to 78.3 at 2.35 GFLOPs, and widening the network from 64 to 96 channels recovers it to 80.5 at 5.27 GFLOPs.
5. Inverted bottleneck §2.4: 80.6 at 4.64 GFLOPs.
6. Moving up the depthwise layer §2.5: placing the depthwise convolution before the expansion layers drops accuracy to 79.9 at 4.07 GFLOPs.
7. Kernel size §2.5: 7x7 is adopted. Sizes 5, 7, 9 and 11 give 80.35, 80.57, 80.57 and 80.47 tab. 10. Gains saturate at 7 in the text.
8. ReLU to GELU §2.6: 80.6, accuracy unchanged.
9. Fewer activations §2.6: only one GELU per block, between the two 1x1 layers. Result 81.3, close to Swin-T.
10. Fewer normalization layers §2.6: one normalization layer per block. Result 81.4.
11. BatchNorm to LayerNorm §2.6: result 81.5.
12. Separate downsampling layers §2.6: 2x2 stride-2 convolutions, with extra LayerNorms at each resolution change to keep training stable. Result 82.0 (table value 81.97), final ConvNeXt-T, against Swin-T 81.3.

The same sequence applied to a ResNet-200 (Swin-B regime) ends at 83.6, against Swin-B 83.5 tab. 11 §2.

Variants §3.1

Channel widths C and block counts B per stage (stage widths double each stage):

• ConvNeXt-T: C = (96, 192, 384, 768), B = (3, 3, 9, 3).
• ConvNeXt-S: C = (96, 192, 384, 768), B = (3, 3, 27, 3).
• ConvNeXt-B: C = (128, 256, 512, 1024), B = (3, 3, 27, 3).
• ConvNeXt-L: C = (192, 384, 768, 1536), B = (3, 3, 27, 3).
• ConvNeXt-XL: C = (256, 512, 1024, 2048), B = (3, 3, 27, 3).

Output feature maps for a 224 input are 56, 28, 14 and 7 pixels square tab. 9.

Hyperparameters

Pre-training on ImageNet-1K and ImageNet-22K §3.1 tab. 5. All variants use the same setting, except stochastic depth app. A.1. The paper does not state a minimum learning rate or a rule for scaling the learning rate with batch size.

Fine-tuning on ImageNet-1K app. A.2 tab. 6. Starts from the final pre-training weights, not EMA weights, except ConvNeXt-L pre-trained on ImageNet-1K, which starts from its best EMA checkpoint app. A.2.

Downstream tasks app. A.3 §4:

• COCO (Mask R-CNN and Cascade Mask R-CNN): multi-scale training, AdamW, 3x schedule §4. Lightweight sweep: learning rate {1e-4, 2e-4}, layer-wise lr decay {0.7, 0.8, 0.9, 0.95}, stochastic depth {0.3, ..., 0.8}. The paper does not state the chosen values. The code configs fix them, for example Mask R-CNN tiny: drop path 0.4, layer decay 0.95, learning rate 1e-4, 36 epochs mask_rcnn_convnext_tiny_patch4_window7_mstrain_480-800_adamw_3x_coco_in1k.py. Cascade base (22K): drop path 0.6, layer decay 0.8 cascade_mask_rcnn_convnext_base_patch4_window7_mstrain_480-800_giou_4conv1f_adamw_3x_coco_in22k.py.
• ADE20K (UperNet): 160K iterations, batch size 16, multi-scale test §4. Sweep: learning rate {8e-5, 1e-4}, layer-wise lr decay {0.8, 0.9}, stochastic depth {0.3, 0.4, 0.5} app. A.3. Code example, UperNet base 640: drop path 0.4, learning rate 1e-4, poly schedule with 1500 linear warmup iterations upernet_convnext_base_640_160k_ade20k_ms.py.

Results

Author-reported. ImageNet-1K validation top-1 at 224 pixels unless noted tab. 1; throughput measured on a V100 GPU tab. 1; parameters and FLOPs from the same table. The Swin COCO and ADE20K rows come from Swin's own repository, as the table captions say.

The ImageNet-22K ViT-B/16 and ViT-L/16 rows in tab. 1 use AugReg results that the paper says were provided by personal communication with the authors tab. 1.

Throughput and memory:

• Inference, ImageNet-1K table, V100: ConvNeXt-T 774.7 images/s, Swin-T 757.9 tab. 1.
• Inference on A100 with TF32 and channels-last layout app. E tab. 12: ConvNeXt-T 1943.5 vs Swin-T 1325.6 (+47%); ConvNeXt-S 1275.3 vs Swin-S 857.3 (+49%); ConvNeXt-B 969.0 vs Swin-B 662.8 (+46%) at 224 pixels.
• Training memory on V100, batch 32 per GPU: ConvNeXt-B (isotropic) 7.7 GB vs ViT-B 9.1 GB tab. 2. For Cascade Mask R-CNN, ConvNeXt-B needs 17.4 GB peak memory at 2 images per GPU, against a reference of 18.5 GB for Swin-B §4.

Paper vs code

Compared at commit 048efcea897d999aed302f2639b6270aedf8d4c8 of the official repository (MIT), cloned to raw/code/2201.03545.

• Weight initialization differs. Paper: truncated normal with "0.2" standard deviation tab. 5. Code: truncated normal with std 0.02 for all Conv2d and Linear weights convnext.py. The paper states no other initialization value in its text.
• EMA is off by default in code, but the paper's ImageNet-1K results use it (decay 0.9999) tab. 5. Following the given commands turns it on main.py TRAINING.md.
• Evaluation at 384 pixels. Paper: crop ratio 1.0, i.e. no cropping app. A.2. Code: for input size 384 or larger, the whole image is resized to 384x384 without preserving the aspect ratio ("warping") datasets.py. At 224 the code resizes to 256 and center-crops to 224 (ratio 0.875) datasets.py.
• Stochastic depth schedule. The paper gives one rate per variant tab. 5. The code sets the per-block rates with a linear ramp from 0 to the maximum over all blocks convnext.py.
• Code-only defaults not in the paper's table. Color jitter 0.4 main.py and a mixup/cutmix switch probability of 0.5 main.py are code defaults. The paper does not state them.
• Stage configurations match §3.1 for T, S, B, L and XL convnext.py.
• Downstream hyperparameters. The paper gives sweep ranges only. Code configs fix values per model, for example Cascade Mask R-CNN base (22K): drop path 0.6 and layer decay 0.8 cascade_mask_rcnn_convnext_base_patch4_window7_mstrain_480-800_giou_4conv1f_adamw_3x_coco_in22k.py. The paper does not say which values produced its tables.
• Pre-training commands match the paper: 22K run with 90 epochs, 5 warmup epochs and an effective batch of 4096 TRAINING.md. Isotropic S and B disable layer scale and use 50 warmup epochs, as the paper says TRAINING.md.

Pitfalls

• Table and figure numbers here were counted from the LaTeX source in order of appearance. They have not been checked against the compiled PDF. The paper has no numbered equations or algorithms, so equations are cited by section and table.
• Weight init. Use std 0.02 if you want to reproduce the released code, and 0.2 if you follow the paper's table. The two disagree and the paper does not say which was used for the reported numbers.
• Batch size and learning rate. Learning rate 4e-3 is given for batch 4096. Neither paper nor code gives a scaling rule. Changing the batch size without a rule is a guess.
• Warmup and minimum learning rate. The code starts warmup at 0 and decays to 1e-6 utils.py main.py. The paper gives neither value.
• Layer norm epsilon is 1e-6 in the code convnext.py. The paper does not state it.
• Layer scale is 1e-6, initialized per channel. It is disabled for isotropic S and B app. A.1.
• Large kernels disagree within the paper. Section 2.5 says the ResNet-200 regime also stops gaining at 7x7. Appendix C says it saturates at 5x5 instead §2.5 app. C. Table 11 agrees with the appendix: 5x5 gives 82.32 and 7x7 gives 82.30 tab. 11. The code uses 7x7 convnext.py. Larger kernels cost more FLOPs without gains tab. 10.
• Depthwise convolution costs. Under equal FLOPs, depthwise convolutions are slower and use more memory than dense convolutions §4. The paper's throughput gains come from measurements on V100 and A100 tab. 1 tab. 12 and depend on the GPU and data format (TF32, channels-last) app. E.
• Downsampling needs normalization. Without the extra LayerNorms at each resolution change, training with separate 2x2 downsampling diverged §2.6.
• Fine-tuning to 384. Use the warp-to-384 transform the code uses, or the crop ratio 1.0 the paper states. Both are documented here, and they differ. The paper does not say that the code warps.
• Swin numbers in the ImageNet, COCO and ADE20K tables come from Swin's own reports and repository tab. 1 tab. 3 tab. 4. Training conditions therefore may not match exactly.
• ViT in isotropic comparisons uses DeiT (S, B) and MAE (L) numbers §3.3. ConvNeXt isotropic models use longer warmup app. A.1. The two are not trained on identical recipes.
• Robustness results use the ImageNet-trained models without special fine-tuning app. B.
• Sensitive hyperparameters on downstream tasks. Learning rate and layer decay are swept on COCO and ADE20K app. A.3. Use the code configs, or repeat the sweep.
• Data. Pre-training uses ImageNet-22K, which the paper notes may carry data bias. The paper discusses this in its societal impact section app. G.

Lineage

• Builds on: ResNet for the baseline and its stage structure §2 §2.2; ResNeXt-style grouped convolution §2.3; depthwise convolution from MobileNet and Xception §2.3; inverted bottlenecks from MobileNetV2 §2.4; the DeiT and Swin training recipes §2.1 and Swin's multi-stage macro design §2.2; GELU and LayerNorm from Transformers §2.6.
• Compared against: Swin Transformer (Swin Transformer), DeiT and ViT, RegNet, EfficientNet and EfficientNetV2 §3.2 tab. 1.
• Superseded by: not stated in this card.