Depthwise separable convolution

Overview

A regular k\times k convolution mixes space and channels in one step. A depthwise separable convolution factors that into two cheaper operators: a depthwise conv (DWConv) that filters each channel on its own with a k\times k kernel, followed by a 1×1 pointwise conv (PWConv) that mixes information across channels. This is the building block behind MobileNets and related lightweight networks, which lower FLOPs mainly through depthwise or group convolutions 2303.03667 §1. The general definition here is standard; the cards on this page study its cost 2303.03667 §3.1, apply it inside a U-Net 2007.04417 §3.1, use depthwise convs in an attention-style block 2205.14871 §3.2, use depthwise convs to build the query, key and value projections of a Transformer 2103.15808 §3.2.1, and use a 3x3 depthwise conv to predict sampling offsets for a deformable operator 2211.05778 §3.2 rather than introducing it.

The invariant every variant shares: spatial filtering and channel mixing are handled by separate operators, and the pointwise conv restores the cross-channel connections that the cheap spatial step leaves out.

The key result for practitioners is that fewer FLOPs does not mean proportionally lower latency. Latency is FLOPs divided by achieved FLOPS 2303.03667 eq. 1, and DWConv achieves very low FLOPS because it is memory-bound 2303.03667 §3.1, tab. 1. FLOPs on this page means multiply-adds, following the source 2303.03667 §1.

Costs for an input and output of c channels at h\times w 2303.03667 §3.1:

• DWConv FLOPs: h \times w \times k^2 \times c.
• Memory access of a DWConv widened to c' channels: h \times w \times 2c' + k^2 \times c' \approx h \times w \times 2c' 2303.03667 eq. 2.
• Memory access of a regular Conv: h \times w \times 2c + k^2 \times c^2 \approx h \times w \times 2c 2303.03667 eq. 3.

Because DWConv alone loses accuracy, networks usually widen it to c' > c channels (by a factor of 6 in MobileNetV2 inverted residuals). The h \times w \times 2c' input/output term then exceeds a regular conv's memory traffic, even though the FLOPs stay low 2303.03667 §3.1.

This page draws on eleven cards. Run, Don't Walk (FasterNet) argues against DWConv and proposes partial convolution as a replacement. SmaAt-UNet uses depthwise-separable blocks to shrink a U-Net for radar nowcasting, and reports parameters and accuracy but not speed. microYOLO uses a stack of depthwise-separable blocks as the backbone of a detector that runs on a microcontroller, and reports speed and memory for the whole pipeline. IAT uses depthwise convs in place of self-attention in an image-enhancement model. CvT uses depthwise-convolution projections for queries, keys and values in a Vision Transformer. InternImage uses a 3x3 depthwise conv inside its deformable operator, to predict offsets and modulation weights. ConvNeXt builds a pure ConvNet around a 7x7 depthwise conv in an inverted-bottleneck block. Lite-HRNet keeps the depthwise convs of a shuffle block and replaces its 1x1 channel-mixing convs with cheaper channel weights. EdgeNeXt uses depthwise convs with a growing kernel size in a hybrid CNN-Transformer for mobile vision. Lightweight OpenPose uses three depthwise-separable blocks in the neck of a CPU pose estimator. EdgeViT uses depthwise and pointwise convs for local mixing in a mobile vision transformer.

Variants

Depthwise conv + pointwise conv (the standard form)

A k\times k DWConv on every channel, then a PWConv. MobileNetV2 wraps this in an inverted residual that expands channels ×6 before the DWConv 2303.03667 §3.1. In FasterNet's operator benchmark, 10 stacked 3×3 DWConv layers reached on average 313 G/s on GPU, 6.42 G/s on CPU and 0.12 G/s on ARM, against 10151, 71.89 and 3.95 G/s for a regular 3×3 conv 2303.03667 tab. 1. Card: Run, Don't Walk.

Group conv + pointwise conv

Replace the depthwise step with a group conv (GConv), where each filter sees a group of channels instead of one. With 16 groups it reached 1833 / 25.93 / 1.92 G/s on GPU / CPU / ARM, between DWConv and a regular conv 2303.03667 tab. 1. ShuffleNets combine group convs with channel shuffling, which adds data-movement overhead 2303.03667 §1.

Partial conv + pointwise conv (FasterNet)

Partial convolution (PConv) runs a regular k\times k conv on a contiguous block of c_p channels (the first or the last ones) and passes the other c - c_p channels through unchanged; a PWConv follows 2303.03667 §3.2. Its cost 2303.03667 eq. 4, eq. 5:

\text{FLOPs}_{\text{PConv}} = h \times w \times k^2 \times c_p^2

\text{MemAccess}_{\text{PConv}} \approx h \times w \times 2c_p

At c_p = c/4 this is 1/16 of a regular conv's FLOPs and 1/4 of its memory access 2303.03667 §3.2. The authors describe PConv as a group conv that processes one group and leaves the others untouched 2303.03667 app. E. PConv followed by PWConv covers a T-shaped receptive field: the full window on c_p channels plus the centre position on all channels. The authors motivate the extra centre weight by showing that the centre of trained ResNet18 3×3 filters is most often the position with the largest norm 2303.03667 §3.3, fig. 6.

Measured average FLOPS of 3×3 PConv with r = 1/4: 3274 / 39.91 / 2.73 G/s on GPU / CPU / ARM, which the authors summarise as 10.5×, 6.2× and 22.8× the FLOPS of DWConv 2303.03667 tab. 1, §4.1. Card: Run, Don't Walk.

Depthwise-separable blocks in a U-Net (SmaAt-UNet)

SmaAt-UNet replaces the convolutions of a U-Net with depthwise-separable ones. Each depthwise step produces two outputs per input channel, so the depth multiplier (kernels per layer) is 2 2007.04417 §3.2. The paper's parameter table gives 3,955,185 parameters for U-Net with these blocks against 17,272,577 for plain U-Net, about 0.23× 2007.04417 tab. 1. This is the DSC-only model, without the CBAM attention the paper adds; the full SmaAt-UNet has 4,111,389 parameters 2007.04417 tab. 1. The task is 30-minute rain-map prediction on radar data, and the DSC-only model reaches NL-50 MSE 0.0127 against 0.0122 for U-Net 2007.04417 tab. 4. On the NL-20 set it reaches 0.0115 against 0.0111 2007.04417 tab. 5. These are author-reported single runs 2007.04417 §5. The paper reports no latency or FLOPs for this variant. Card: SmaAt-UNet.

Depthwise-separable backbone in a microcontroller detector (microYOLO)

microYOLO's backbone is one standard convolution (4×4, stride 2) followed by seven depthwise-separable blocks with 3×3 kernels, growing from 64 to 128 channels 2408.15865 tab. 1. The authors use depthwise-separable convolutions "to keep the parameter count low" 2408.15865 §2. The paper does not compare them with standard convolutions, so it gives no measured saving from this choice. The network is pruned by L1 norm during the last 100 training epochs and quantized to 8 bits afterwards 2408.15865 §2, §3. On an OpenMV H7 R2 (Cortex-M7, no accelerator), the 128×128 models run at 3.45–3.49 FPS in 706–771 KB of Flash and 324 KB of RAM 2408.15865 tab. 2. These are author-reported figures for the full pipeline, so they cannot be attributed to the depthwise-separable blocks alone. Card: microYOLO.

Depthwise convs in a transformer-style block (IAT)

The local branch of IAT replaces self-attention with depthwise convolution, so the branch works at full resolution on images of any size 2205.14871 §3.2. Each pixel-wise block starts with a 3×3 depthwise conv that encodes position, then passes through a pointwise, depthwise, pointwise sequence, with a learned per-channel normalisation in place of LayerNorm 2205.14871 §3.2. The paper's ablations compare normalisation layers and the global branch, but not depthwise against regular convolutions 2205.14871 tab. 7, so the card adds no evidence on the speed question. The released code uses a different middle kernel size from the text; see the IAT card before copying the layer shapes.

Depthwise-convolution projections in attention (CvT)

CvT keeps the Transformer block but computes each query, key and value with a convolutional projection instead of a linear layer. The tokens are reshaped to a 2D grid, passed through a 3×3 depthwise conv and BatchNorm, then a pointwise step mixes channels; the result is flattened back to tokens 2103.15808 §3.2.1, eq. 2. The pointwise step is a 1×1 conv in the paper's text; the authors' code implements it as a Linear layer 2103.15808 §3.2.1. Queries keep full resolution (stride 1), while keys and values use stride 2 to shorten the attention matrix 2103.15808 §3.2.2. In CvT-13, this projection beats linear projections in all three stages by 1.0 top-1 on ImageNet-1k at 224 (81.6% vs 80.6%) 2103.15808 tab. 8. The paper gives no per-projection latency or FLOPs, and its ablations do not compare depthwise against regular convolution in the projection 2103.15808 tab. 8. Card: CvT.

Depthwise conv as a predictor inside a deformable op (InternImage)

Here the depthwise conv is not a separable replacement for a conv. DCNv3 runs a 3x3 depthwise conv on its input, then predicts per-group sampling offsets and softmax-normalised modulation weights from the result 2211.05778 §3.2. The sampling and weighted sum are done by the deformable operator, not by a pointwise conv 2211.05778 §3.1, eq. 2. The paper's ablations cover shared weights, groups and softmax, but not this depthwise step 2211.05778 tab. 6, so the card gives no depthwise-versus-regular comparison. The released H config uses a 5x5 depthwise kernel where the text says 3x3, so check the InternImage card before copying shapes. Card: InternImage.

Large-kernel depthwise conv in a pure ConvNet (ConvNeXt)

ConvNeXt's block uses a 7×7 depthwise conv as its only spatial operator. A LayerNorm follows, then two 1×1 layers expand channels fourfold and project them back, with one GELU between them. A per-channel layer scale and a residual add complete the block 2201.03545 §2.4–§2.6. The depthwise layer is used at a wider width than the input: in the ResNet-50 regime, depthwise convs at 64 channels dropped accuracy to 78.3 at 2.35 GFLOPs, and widening to 96 channels recovered it to 80.5 at 5.27 GFLOPs 2201.03545 §2.3. The paper's kernel sweep gives 80.35, 80.57, 80.57 and 80.47 for 5×5, 7×7, 9×9 and 11×11 2201.03545 tab. 10, and it adopts 7×7 2201.03545 §2.5. The model reaches 82.1% ImageNet-1K top-1 at 224 pixels for ConvNeXt-T against 81.3% for Swin-T 2201.03545 tab. 1. The roadmap that leads there is a sequence of cumulative changes, so the depthwise layer's effect is measured only inside that sequence 2201.03545 §2. The paper also says that, at equal FLOPs, depthwise convs are slower and use more memory than dense convs 2201.03545 §4. Its A100 throughput advantage over Swin-T (1943.5 vs 1325.6 images/s) compares whole networks, not depthwise against dense convs 2201.03545 tab. 12. Card: ConvNeXt.

Depthwise conv with channel weights instead of pointwise convs (Lite-HRNet)

Lite-HRNet keeps the depthwise 3×3 conv of a ShuffleNet-style block and removes the two 1×1 convs that mix channels. The authors identify those 1×1 convs as the cost bottleneck of a shuffle block 2104.06403 §3.2, tab. 2. In their place, half of each branch's channels are rescaled element-wise by weights computed from the features: cross-resolution weights from pooled maps of all branches, and spatial weights from a global pooled vector 2104.06403 §3.2, eq. 2–6. The authors state that the cross-resolution weights play the same role as a 1×1 conv in exchanging information across channels 2104.06403 §3.2. In the COCO ablation, Lite-HRNet-18 with channel weights has 1.1M parameters and 205.2 MFLOPs for 64.8 AP. A wider naive model that keeps the 1×1 convs has 1.3M parameters and 311.1 MFLOPs for 65.7 AP 2104.06403 tab. 6. Lite-HRNet-30 reaches 67.2 COCO val AP at 256×192 with 1.8M parameters, without ImageNet pretraining 2104.06403 tab. 3. Card: Lite-HRNet.

Depthwise conv encoders with a chained split (EdgeNeXt)

EdgeNeXt's convolution encoder is a k×k depthwise conv, then layer norm, two pointwise linear layers and a residual. The kernel size k grows from 3 to 9 across the four stages 2206.10589 §3, tab. 1. This has the same depthwise-plus-pointwise-MLP shape as ConvNeXt, with a stage-dependent kernel. In the last block of stages 2 to 4, the SDTA encoder splits the channels into subsets. Each subset gets a depthwise 3×3 conv that also receives the previous subset's output, and the result feeds a channel-wise (transposed) attention 2206.10589 §3, eq. 2–4. The paper reports 79.4% top-1 for EdgeNeXt-S at 256 px with 1.30G MAdds 2206.10589 tab. 2. Removing the SDTA encoders also removes the chained depthwise convs. That gives 78.3% top-1 and a lower latency in the paper's own measure 2206.10589 tab. 7, so the accuracy drop cannot be assigned to the depthwise convs alone. Card: EdgeNeXt.

Depthwise-separable neck in a CPU pose estimator (Lightweight OpenPose)

The backbone is a dilated MobileNet v1, which has depthwise layers. The stride of one depthwise layer is removed and its dilation set to 2, which keeps the output stride at 8 1811.12004 §4.1.1. The neck replaces conv4_3 with three depthwise-separable convolutions that reduce 256 channels to 128 1811.12004 §4.1.2, tab. 5. The paper's per-stage table lists conv4_3 at 0.3 GFLOPs in the 2-stage network, but it gives no standard-convolution alternative for that position 1811.12004 tab. 5. The author-reported whole network needs 9 GFLOPs for 42.8 AP, against 61.7 GFLOPs for 43.4 AP in the 2-stage OpenPose baseline 1811.12004 tab. 2, tab. 5. End-to-end speed was 28 fps on an Intel NUC and 26 fps on an i7-6850K, with OpenVINO, on a video with more than 20 poses 1811.12004 tab. 6. Post-processing runs faster than the network after the change, so the speed figure mostly reflects the whole pipeline 1811.12004 tab. 6. The linked card records that the public repository's checkpoint does not reproduce the 42.8 AP, so check it before quoting the number. Card: Lightweight OpenPose.

Depthwise and pointwise local mixing in a light ViT (EdgeViT)

Each EdgeViT block starts with a local aggregation step made of depthwise and pointwise convs over the k×k neighbourhood of every token 2205.03436 §3.2. Self-attention then runs over one sampled token per window, and a transposed conv carries that context back to every token 2205.03436 §3.2. In the ablation, rescaled to about 0.5 GFLOPs, the depthwise version reaches 74.4% top-1 at 32.8 ms CPU latency. Locally grouped self-attention in the same slot reaches 74.0% at 36.1 ms, and removing the step gives 72.7% at 33.9 ms 2205.03436 tab. 4. The paper's complexity count uses k = 3 2205.03436 app. A.1, but the authors' released code uses a 5×5 depthwise conv in that step, so check the EdgeViT card before copying shapes. The positional encoding is also a 3×3 depthwise conv with zero padding and a residual connection 2205.03436 §3.3. Card: EdgeViT.

How well each approximates a regular conv

FasterNet trains each "X + PWConv" pair to reproduce the output of the first 3×3 conv of each ResNet50 stage, given that conv's input feature maps, using MSE loss 2303.03667 §4.2. Average test MSE (lower is better) 2303.03667 tab. 2:

DWConv + PWConv has the highest error in every stage. GConv beats PConv only in stage 1 (0.0065 vs 0.0069) 2303.03667 tab. 2.

When to use which

Implementation sketch

# x: [c, h, w]

depthwise_separable(x, k):
    y = depthwise_conv(x, kernel=k, groups=c)    # one k×k filter per channel
    return conv1x1(y, c -> c_out)                 # mix channels

group_separable(x, k, g):
    y = conv(x, kernel=k, groups=g)               # each filter sees c/g channels
    return conv1x1(y, c -> c_out)

partial_separable(x, k, r=1/4):                  # [arxiv:2303.03667 §3.2]
    c_p = c * r
    y_head = conv(x[0:c_p], kernel=k, c_p -> c_p, stride=1)   # contiguous slice
    y = concat(y_head, x[c_p:c])                  # untouched channels kept, not dropped
    return conv1x1(y, c -> c_out)

Notes on where variants differ:

• SmaAt-UNet uses a depth multiplier of 2: the depthwise step emits two channels per input channel, so the pointwise conv reads 2c channels 2007.04417 §3.2.
• PConv must use stride 1, because the untouched channels keep their resolution; downsample with a separate layer 2303.03667 app. F. CvT's depthwise projection is the opposite case: its stride is 2 for keys and values, which shrinks the token grid on purpose 2103.15808 §3.2.2.
• FasterNet puts no norm or activation between PConv and the following PWConv, to keep the pair close to a regular conv 2303.03667 §3.4, app. E.
• The 1/4 memory-access figure for PConv counts only the c_p processed channels 2303.03667 eq. 5. A split-and-concat implementation as sketched above also copies the untouched channels; the card notes that the paper does not say which form produced its numbers.
• When benchmarking, measure achieved speed on the target device, not FLOPs, and fold BatchNorm first (see batchnorm-folding) 2303.03667 §4.

Contradictions

Lite-HRNet's card says the 1×1 convs, not the depthwise conv, are the cost bottleneck of a shuffle block 2104.06403 §3.2, tab. 2. FasterNet says the depthwise conv runs at the lowest FLOPS of the operators it tests on GPU, CPU and ARM 2303.03667 tab. 1. Both can hold: one compares cost inside a block, the other compares achieved speed per operator. Lite-HRNet's tables report FLOPs and accuracy, not per-operator speed 2104.06403 tab. 6, so they cannot settle which part is slower on hardware. Apart from this, there is no direct numerical conflict. The cards measure different things on different tasks. The EdgeNeXt card adds a hybrid where the depthwise convs sit beside attention, and its ablation removes the attention and the depthwise convs together 2206.10589 tab. 7. The IAT card adds a third setting (low-level image enhancement) with no depthwise-versus-regular comparison 2205.14871 tab. 7. The InternImage card adds a fourth (large-scale classification and detection with a deformable operator), again with no depthwise-versus-regular comparison 2211.05778 tab. 6. The ConvNeXt card adds a fifth, a 7×7 depthwise block in a pure ConvNet. It states that depthwise convs are slower at equal FLOPs 2201.03545 §4, which agrees with FasterNet's operator-level numbers. Its speed results compare whole networks against Swin, not depthwise against dense convs 2201.03545 tab. 12, and its 7×7 kernel differs from FasterNet's 3×3 setting. The CvT card adds a fifth, a Transformer whose projections are depthwise convs; its ablation compares linear against convolutional projections, not depthwise against regular convolution 2103.15808 tab. 8. The microYOLO card adds a sixth, a complete detector on a microcontroller, which reports FPS, Flash and RAM for the whole pipeline with no depthwise-versus-standard comparison 2408.15865 tab. 2, §2. FasterNet reports approximation MSE of DWConv + PWConv against a regular conv (0.0148) and latency on CPU and ARM 2303.03667 tab. 2, tab. 3. SmaAt-UNet reports task MSE and parameter counts for a U-Net on radar data 2007.04417 tab. 4. Their numbers should not be compared. The FasterNet card also notes that its comparison against MobileNetV2, ShuffleNetV2 and GhostNet is, by the authors' own account, not fully fair, because those baselines were trained without the stronger recipe FasterNet used 2303.03667 app. A.

The Lightweight OpenPose card adds a real-time pose network, and the EdgeViT card adds a mobile transformer. Neither compares depthwise convs with standard convs. The pose paper's backbone ablation varies the backbone, not the neck 1811.12004 tab. 4, and the EdgeViT ablation compares local-aggregation operators 2205.03436 tab. 4. The EdgeViT result, where depthwise beats locally grouped attention on latency, does not conflict with the ConvNeXt statement that depthwise convs are slower than dense convs at equal FLOPs 2201.03545 §4. The two compare different alternatives. Whole-pipeline speed figures from the pose paper also cannot be assigned to its depthwise-separable neck alone 1811.12004 tab. 6.

Papers

• Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks: shows DWConv is memory-bound with low achieved FLOPS and proposes partial convolution as a faster replacement in FasterNet.
• SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture: replaces U-Net convolutions with depthwise-separable blocks (two kernels per layer) to cut parameters to about a quarter, with a small accuracy cost on radar nowcasting.
• IAT, a tiny transformer-style image enhancer (2205.14871): uses depthwise convs in place of self-attention in a 0.09 M-parameter image enhancer.
• CvT: Introducing Convolutions to Vision Transformers: replaces the linear query, key and value projections of a Vision Transformer with depthwise-convolution projections, with stride-2 keys and values.
• InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions: uses a 3x3 depthwise conv to predict the offsets and modulation weights of its DCNv3 operator in a large CNN backbone.
• A ConvNet for the 2020s: uses a 7x7 depthwise conv in an inverted-bottleneck block, modernising a ResNet-50 step by step into a pure ConvNet that matches Swin-T on ImageNet-1K.
• microYOLO: Towards Single-Shot Object Detection on Microcontrollers: uses seven depthwise-separable blocks in a YOLO-style detector for a Cortex-M7 microcontroller, with L1 pruning and 8-bit quantization, and reports FPS and memory on the device.
• Lite-HRNet: A Lightweight High-Resolution Network: replaces the 1×1 channel-mixing convs of a shuffle-based high-resolution network with conditional channel weights, for pose estimation and segmentation.
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications: uses depthwise convs with kernels growing from 3 to 9 across stages, and a chained-split depthwise block with transposed attention, for mobile image classification.
• Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose: replaces the conv4_3 stage of a dilated MobileNet v1 OpenPose backbone with three depthwise-separable convs and reports 9 GFLOPs at 42.8 AP on COCO validation, author-reported.
• EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers: uses depthwise and pointwise convs for local aggregation in a light hierarchical transformer, and reports ImageNet-1K latency on a phone CPU.