InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions

In plain words

Transformers dominate large vision models right now, and convolutional networks (CNNs) have lagged behind in size. The authors argue that CNNs can catch up. A plain CNN looks at a small neighbourhood of pixels through fixed filter weights, so it has trouble seeing across a whole image and does not adapt much to what it sees. Transformers avoid both limits by letting each position attend to others.

InternImage keeps the small 3x3 CNN window but makes it smarter. Each position learns where to look: it predicts small shifts for its sampling points, so the window can stretch to a distant part of the image when the content calls for it. It also learns how much weight each sampling point gets, and the total weight is normalised. The operator is cheap enough to build large models around it. The authors report that this design scales to very large sizes and data and performs close to or above the best transformers on image classification, object detection and segmentation.

TL;DR

InternImage is a convolutional vision backbone whose core operator is DCNv3, a deformable convolution with a 3x3 window. DCNv3 shares its projection weights across sampling points, splits channels into groups with their own sampling offsets, and normalises the modulation weights with a softmax over the points. Model sizes come from four hyper-parameters, and the largest variant (H, about 1B parameters) reaches state-of-the-art author-reported results on COCO and ADE20K. Some training details in the paper are missing or differ from the released configs (see Paper vs code and Pitfalls).

Problem

Large vision models are mostly transformers. Global self-attention has long-range reach and input-dependent weights, but its cost grows fast on large feature maps, which hurts dense tasks such as detection and segmentation §1, §2. Large-kernel CNNs such as RepLKNet extend the receptive field with dense kernels of 31x31, but the authors note that these kernels are hard to optimise and expensive §1, §3.1. Standard CNNs also have static weights and a strong locality bias, which the authors say keeps them from learning general patterns from web-scale data §3.1. The paper asks whether a CNN can reach ViT-level scale and performance when it borrows these properties §1.

Method

The paper starts from deformable convolution v2 (DCNv2) eq. 1. Its sampling offsets and modulation scalars depend on the input, which gives long-range reach and adaptive aggregation §3.1. The authors make three changes so that the operator can be trained from scratch at large scale §3.1:

1. Shared projection weights. DCNv2 has a separate projection per sampling point, so its parameter and memory cost grows with the number of points. DCNv3 uses one point-wise projection shared by all points, while the per-point modulation is predicted separately §3.1.
2. Groups. The channels are split into G groups. Each group has its own sampling offsets and modulation weights, so groups can attend to different regions §3.1, fig. 5.
3. Softmax normalisation. DCNv2 uses a sigmoid per point, so the sum of weights is unstable. DCNv3 uses a softmax over the K points of each group, so the weights sum to 1 §3.1.

The resulting operator is eq. 2. Each output position p0 sums the values sampled at p0 + p_k + offset over all groups and points, weighted by the modulation of that group and point. The paper then builds the block, stem and downsampling layers:

4. Block. The block resembles a transformer block: layer normalisation, the DCNv3 operator, a feed-forward network with GELU, and residual connections. Post-normalisation is described as the default §3.2, fig. 3. Sampling offsets and modulation are predicted from the input by a 3x3 depth-wise convolution followed by a linear projection §3.2.
5. Stem and downsampling. The stem reduces resolution by 4 with two stride-2 3x3 convolutions, two layer norms and a GELU. Between stages, a stride-2 3x3 convolution followed by a layer norm halves the resolution and doubles the channels §3.2, fig. 3 intern_image.py.
6. Stacking rules. The model has four stages. Stage-2 and stage-4 depths equal stage 1, and stage 3 is the deepest ("AABA") §3.2. The group count of each stage follows its channel count, so every group has the same channel width C' §3.2. A variant is set by four hyper-parameters (C1, C', L1, L3) §3.2, fig. 3.
7. Scaling. Depth and width grow by a composite factor with alpha = 1.09 and beta = 1.36 §3.2, app. B. The paper states the rule alpha * beta^1.99 ≈ 2 §3.2. The resulting variants are listed in tab. 1.

# DCNv3 layer  [eq. 2]
# Input:  features (N, H, W, C); G groups; K = kernel_h * kernel_w points
# Output: (N, H, W, C)
# Source: classification/ops_dcnv3/modules/dcnv3.py#L185-L216 (PyTorch reference)

def dcnv3(features):
    # Value projection before sampling; not in [eq. 2]  [code: dcnv3.py#L192]
    values = linear_value(features)                       # (N, H, W, C)

    # Depth-wise 3x3 conv, then layer norm and GELU  [§3.2] [code: dcnv3.py#L196]
    hidden = depthwise_conv3x3(features)                  # (N, H, W, C)
    hidden = gelu(layer_norm(hidden))                     # (N, H, W, C)

    # Per-group sampling offsets, (N, H, W, G*K*2)  [§3.2] [code: dcnv3.py#L197]
    offsets = linear_offset(hidden)

    # Per-group modulation, softmax over the K points  [eq. 2] [code: dcnv3.py#L198-L199]
    logits = linear_mask(hidden)                          # (N, H, W, G*K)
    weights = softmax(logits.reshape(N, H, W, G, K), axis=-1)

    # Sample each group at its shifted points and sum  [eq. 2] [code: dcnv3_func.py#L195-L216]
    outputs = aggregate(values, offsets, weights)         # (N, H, W, C)

    # Output projection after aggregation; not in [eq. 2]  [code: dcnv3.py#L216]
    return linear_output(outputs)                         # (N, H, W, C)

# Sampling and weighted sum for one DCNv3 layer  [eq. 2]
# Source: classification/ops_dcnv3/functions/dcnv3_func.py#L195-L216

def aggregate(values, offsets, weights):
    # Each group keeps C' = C / G channels  [§3.1]
    outputs = zeros_like(values)                          # (N, H, W, C)
    for group in range(G):
        # Points p0 + p_k + offset_scale * delta_p_gk, offsets normalised
        # by the padded input size  [code: dcnv3_func.py#L195-L199]
        locations = grid_points + offset_scale * offsets[group]
        # Bilinear sampling, zeros outside the map, align_corners=False
        # [code: dcnv3_func.py#L210-L211]
        sampled = bilinear_sample(values[group], locations)  # (N, H, W, K, C')
        # Weighted sum over the K points of this group  [eq. 2]
        outputs[group] = sum_over_k(weights[group][..., None] * sampled)
    return outputs

# Basic block with post-normalisation and layer scale  [§3.2] [fig. 3]
# Source: classification/models/intern_image.py#L419-L421; configs set
# LAYER_SCALE 1e-5 and POST_NORM True in the S/B/L/XL yaml files

def block(x):
    # Residual branch 1: DCNv3, then layer norm, scaled by gamma1
    x = x + drop_path(gamma1 * layer_norm1(dcnv3(x)))
    # Residual branch 2: feed-forward (4x expansion, GELU), then layer norm
    x = x + drop_path(gamma2 * layer_norm2(mlp(x)))
    return x

Hyperparameters

Model variants tab. 1

The origin model was chosen from 30 variants trained on ImageNet §3.2, app. B. Group count G_i = C_i / C' in every stage internimage_t_1k_224.yaml. The MLP ratio of 4 and a linear drop-path schedule are not stated in the paper text; they come from the code defaults config.py.

ImageNet classification app. A, detailed recipe table

The table is transcribed from app. A. The H column covers only the ImageNet-1K fine-tuning. Its pre-training on a 427M joint set (Laion-400M, YFCC-15M, CC12M) used M3I pre-training for 30 epochs, and the authors refer to M3I for those details §4.1, app. A. The L and XL models were pre-trained on ImageNet-22K for 90 epochs and fine-tuned at 384 for 20 epochs §4.1, app. A.

Object detection §4.2, app. A, code: detection/configs/coco/mask_rcnn_internimage_t_fpn_1x_coco.py

• Frameworks: Mask R-CNN and Cascade Mask R-CNN on COCO §4.2.
• Schedules: 1x (12 epochs) and 3x (36 epochs, multi-scale). Shorter side 800 (longer side at most 1333) for 1x. Shorter side 480 to 800 (longer side at most 1333) for 3x. Testing at shorter side 800 app. A.
• Batch size 16, AdamW, initial LR 1e-4 app. A.
• Weight decay 0.05 mask_rcnn_internimage_t_fpn_1x_coco.py.
• Backbone initialised from ImageNet classification weights §4.2.
• Large-scale setting app. A: DINO detector, initialised from ImageNet-22K or the 427M joint set, parameters doubled with the composite technique §4.2, app. A. Pre-training on Objects365 for 26 epochs (batch 256, LR 2e-4, shorter side 600 to 1200, LR drop at epoch 22). Fine-tuning on COCO for 12 epochs (batch 64, LR 5e-5, drop at the last epoch).

Semantic segmentation §4.3, app. A, code: segmentation/configs/ade20k/upernet_internimage_t_512_160k_ade20k.py

• UperNet on ADE20K for 160k iterations, batch 16 §4.3, app. A.
• AdamW with LR 6e-5 for T/S/B and 2e-5 for L/XL app. A. The paper writes "X/XL" for the second group app. A. Weight decay 0.05, default betas upernet_internimage_t_512_160k_ade20k.py.
• Poly decay with power 1.0, minimum LR 0 app. A, code: segmentation/configs/ade20k/upernet_internimage_t_512_160k_ade20k.py#L55-L58.
• Crop 512 for T/S/B and 640 for L/XL app. A.
• H with Mask2Former: 80k iterations of pre-training on COCO-Stuff and 80k of fine-tuning on ADE20K, crop 896, LR 1e-5 §4.3, app. A.

Pre-training data

• The 427M joint dataset is Laion-400M, YFCC-15M and CC12M §4.1, app. A. The paper does not list the exact subset or its filtering.

Results

Author-reported. Setups as stated in the cited tables. Values are copied from the paper, not re-run.

ImageNet-1K classification, top-1 tab. 2

COCO val2017, Mask R-CNN box AP / mask AP tab. 3

Cascade Mask R-CNN tab. 3: L (277M) 54.9 / 47.7 at 1x and 56.1 / 48.5 at 3x+MS. XL (387M) 55.3 / 48.1 at 1x and 56.2 / 48.8 at 3x+MS. The paper's ConvNeXt-XL baseline is 55.2 box AP at 3x+MS tab. 3.

COCO detection SOTA tab. 4: H (2.18B, with the doubled backbone) plus DINO reaches 65.0 box AP on val2017 and 65.4 on test-dev. The best prior entry in the table is FD-SwinV2-G at 64.2 test-dev with 3.00B parameters §4.2, tab. 4.

ADE20K val, mIoU, single-scale / multi-scale tab. 5

The H + Mask2Former result (62.9 MS) is above BEiT-3's 62.8 MS in the same table tab. 5.

Ablations on T tab. 6

Other ablations: sharing weights saves 42.0% of parameters and 84.2% of GPU memory at the H scale §4.3, fig. 4. A 5x5 or 7x7 kernel does not help: 83.6 and 82.8 top-1, versus 83.5 for 3x3, with more parameters and FLOPs app. B, kernel-size table. With 1% and 10% of ImageNet-1K training data, T reaches 5.9% and 56.0%, against 12.2% and 57.5% for ResNet-50 app. E, data-scale table.

Throughput and robustness app. D, app. E: the authors report that InternImage-B is slower than ConvNeXt-B, despite 84.9 versus 83.8 top-1 app. D. The throughput table app. D is not transcribed on this card. The robustness comparison is shown only in plots, and the text describes it qualitatively, with advantages on translation and large rotations app. E.

Paper vs code

Compared against the official repository at code_commit 31c962d. The code path is the PyTorch reference of DCNv3 plus the classification and detection configs.

• Post-normalisation. The paper states post-normalisation as the default §3.2. The T config keeps POST_NORM at its default of False, so the pre-norm branch runs config.py. The S, B, L and XL configs set POST_NORM: True internimage_s_1k_224.yaml.
• Layer scale. The appendix table marks layer scale as "no" for T and "yes" for S to H app. A. The code uses LAYER_SCALE: 1e-5 for S, B, L and XL internimage_s_1k_224.yaml. The T config has no layer scale config.py. The H config sets LAYER_SCALE: None internimage_h_22kto1k_640.yaml. The detection T config uses layer_scale=1.0 mask_rcnn_internimage_t_fpn_1x_coco.py. The paper gives no value.
• Warmup. The appendix table lists 5 warmup epochs for T, S, B and the 22K pre-training runs app. A. The default in code is 20 epochs config.py, and the T, S and B configs do not override it. The L and XL fine-tuning configs use 2 warmup epochs against 0 in the paper's table internimage_l_22kto1k_384.yaml. The H config also uses 2 internimage_h_22kto1k_640.yaml.
• Exponential moving average. The code enables EMA with decay 0.9999 for T, S, B and the large models internimage_t_1k_224.yaml. The paper does not mention EMA.
• Layer-wise LR decay and DCN LR multiplier. The L and XL fine-tuning configs set layer decay 0.9 and a 0.1 learning-rate multiplier for DCN parameters internimage_l_22kto1k_384.yaml. The paper does not mention either.
• Offset scale. The T, S and B configs use OFFSET_SCALE: 1.0 internimage_t_1k_224.yaml. The L and XL configs use 2.0 internimage_l_22kto1k_384.yaml, and the XL config does too. The paper's equations have no such scale. In the code it multiplies both the base grid and the predicted offset dcnv3_func.py.
• Offset and modulation predictor. The paper describes a 3x3 depth-wise convolution followed by a linear projection §3.2. The code inserts a layer norm and a GELU between the two dcnv3.py.
• Projections missing from the equation. The code adds a linear value projection before sampling and a linear output projection after aggregation dcnv3.py. Neither appears in eq. 2.
• Classification head. The code adds a 1x1 convolution that expands channels by 1.5, then batch norm and GELU, before global average pooling and the linear classifier intern_image.py. The paper does not describe this head.
• H variant. The H config sets features the paper does not describe: 5x5 depth-wise kernel, residual post-norm, post-norm at selected blocks, centre feature scaling and a use_clip_projector head internimage_h_22kto1k_640.yaml. The paper describes H only by its depths, channels and C' = 32 tab. 1.
• Learning rate. The code multiplies the base LR by batch size / 512 main.py. For T at batch 4096 this gives 5e-4 * 8 = 4e-3, which matches the table.
• Detection and segmentation. The paper says only that these follow common practice. The Mask R-CNN T config uses AdamW with LR 1e-4, weight decay 0.05, drop path 0.2, layer decay 1.0 over 30 layers, which means no decay mask_rcnn_internimage_t_fpn_1x_coco.py. The segmentation T config adds a 1500-iteration linear warmup, which the paper does not state upernet_internimage_t_512_160k_ade20k.py.
• Stage-width search. The paper's main text gives C1 in {48, 64, 80} §3.2. The appendix gives {16, 32, 64} app. B. The released configs contain only the final variants, not the search grid internimage_t_1k_224.yaml.

Pitfalls

• Use the CUDA op. The PyTorch function in the repository is marked as "for debug and test only" dcnv3_func.py. The compiled DCNv3 extension is built with classification/ops_dcnv3/make.sh. The paper does not say which implementation its speed numbers use app. D.
• Normalise the offsets. Offsets are scaled by the padded input size, not by the original size. The reference implementation pads before sampling dcnv3_func.py. An implementation that normalises by the unpadded size will differ from the released code.
• Zero padding, bilinear sampling. The sampler uses zeros outside the map and align_corners=False dcnv3_func.py. The paper does not state the padding mode.
• Sigmoid breaks training. Replacing the softmax with a sigmoid drops T top-1 from 83.5 to 65.7 tab. 6. The paper attributes this to unstable gradients §3.1.
• Group channels should be a power of 2. The code warns when C' is not a power of 2 dcnv3.py. All released configs use C' in {16, 32}.
• Bigger kernels are not better here. Replacing 3x3 with 5x5 or 7x7 does not improve top-1 app. B. The paper argues that large kernels are harder to optimise, which the authors take as a reason to keep 3x3 app. B.
• Parameter counts differ by head. H has 1.08B parameters for classification tab. 1, tab. 2, 1.12B with UperNet and 1.31B with Mask2Former tab. 5, and 2.18B for the detection model with doubled backbone tab. 4. Check which one a number refers to.
• Pre-training data is not reproducible from the paper. The H result depends on a 427M joint dataset and the M3I method. The authors say part of the gap to transformers may come from private data §4.1. The exact subset is not listed.
• Latency. The authors note that DCN operators are slower than standard convolutions at high-speed downstream tasks, and their model is slower than ConvNeXt-B despite higher accuracy §5, app. D.
• Typos. The segmentation learning rate for "X/XL" should read as L/XL, judging by the model names around it app. A.
• Layer scale without a value. The paper's table says "yes" for layer scale without a value. The code uses 1e-5 for classification and 1.0 for detection internimage_s_1k_224.yaml.

Lineage

• Builds on: Deformable ConvNets v2 (Zhu et al., CVPR 2019) and DCNv1 §1, §3.1, eq. 1. The paper's bibliography lists DCNv2 without an arXiv id, so none is given here.
• Compares against: ConvNeXt (arXiv:2201.03545) tab. 2–6, RepLKNet, SLaK, Swin, PVTv2, SwinV2-G, BEiT-3, DINO and Mask2Former (arXiv:2112.01527) tab. 2–5.
• Superseded by: not stated on this card.