DO-Conv: Depthwise Over-parameterized Convolutional Layer

In plain words

A convolution layer is the basic building block of image models. It slides small learned filters over the picture. This paper changes how that layer is trained. Instead of learning one filter directly, the layer learns two smaller pieces: a step that treats each input channel on its own, and then an ordinary filter that mixes the channels.

Those two pieces are a longer way of writing one ordinary filter. So after training, the authors merge them into a single normal filter. The finished model has the same shape and the same cost as a plain convolution, and nothing changes at the time it is used. Only the training is different.

The authors report that this extra training structure gives small accuracy gains on image classification, object detection and segmentation. The gains are not uniform: some networks gain a little, and others stay flat or drop slightly.

TL;DR

DO-Conv replaces each non-pointwise convolution with a depthwise over-parameterized pair: a depthwise kernel D (one small kernel per input channel, D_mul outputs each) followed by an ordinary kernel W §3 eq. 3. Both are trained by gradient descent, then folded into one kernel W' = D^T ∘ W with the same shape as a normal convolution kernel, so inference cost is unchanged §3. The paper reports small gains (for example +0.03 to +1.01 top-1 on ImageNet) against baselines the authors retrained themselves §4.1 tab. 2 tab. 3. Relevant if you train CNNs and want a drop-in change with no inference cost; check the pitfalls before relying on the gains.

Problem

The authors ask whether extra linear layers, which add parameters but no new expressiveness, can speed up or improve training of existing CNNs §1. Prior work showed such over-parameterization accelerates deep linear and non-linear networks §2. Earlier convolution-level variants (ExpandNet over channels, ACNet over the kernel cross) either target compact networks or are special cases of this construction §2. The paper's aim is a general convolution replacement that adds training-time parameters and leaves inference unchanged §1 §3.

Method

Notation and the two standard operators

Tensors are indexed as in the paper: a 4D tensor can be reshaped to 3D without changing its values, and a tensor and its reshaped versions share one symbol §3. A patch P has shape (M×N) × C_in, where M×N is the spatial window and C_in the input channels §3.

A conventional convolution with kernel W of shape C_out × (M×N) × C_in computes each output channel as a dot product between one kernel and the whole patch §3 eq. 1.

A depthwise convolution with depth multiplier D_mul applies each input channel to D_mul separate kernels, with kernel shape (M×N) × D_mul × C_in, and produces a D_mul × C_in output §3 eq. 2.

The DO-Conv operator

DO-Conv is the composition of a depthwise convolution with kernel D of shape (M×N) × D_mul × C_in and an ordinary convolution with kernel W of shape C_out × D_mul × C_in, under the condition D_mul ≥ M×N §3 eq. 3. The paper says this condition is necessary: otherwise the composite cannot express every linear map that a plain convolution can §3.

The composition can be evaluated in two equivalent ways §3 eq. 3:

1. Feature composition: first apply D to the patch, then W to the result. This is the view that resembles a depthwise convolution followed by a 1×1 layer.
2. Kernel composition: first combine D and W into one composite kernel W' = D^T ∘ W, then run a single ordinary convolution with W' on the patch.

The receptive field stays M×N, because the composite kernel only sees one M×N patch at a time §3.

Over-parameterization and folding

DO-Conv over-parameterizes a convolution: for D_mul = M×N, the parameter count rises by (M×N) × D_mul × C_in compared with a plain kernel §3. Both D and W are trained with the usual gradient-based optimizer §3. After training, D and W are folded into W' = D^T ∘ W, which has the shape of a normal convolution kernel. At inference the layer is then exactly a normal convolution §3.

Choice of composition for training

The two compositions are equal in value but differ in cost. The paper gives multiply-accumulate (MACC) counts symbolically for an H × W × C_in feature map with stride 1 §3. Feature composition costs scale with H × W; kernel composition costs scale with C_out and M×N §3. Since H × W is usually much larger than C_out, and D_mul ≥ M×N holds, kernel composition typically needs fewer MACCs and less memory for training §3. The paper therefore recommends kernel composition for training and notes that W' is computed once per feature map §3.

Relation to depthwise separable convolution

Feature composition with D_mul < M×N is the same computation as a depthwise separable convolution §3, and the paper points out that kernel composition works in that case too. It suggests this can speed up training of depthwise separable layers on GPUs, which are often memory-bound, and then deploy with feature composition on edge devices. The authors call this a by-product that is not the focus of the paper §3. Note that depthwise separable convolution is normally used with D_mul < M×N, while DO-Conv requires D_mul ≥ M×N §3.

DO-DConv and DO-GConv

The same construction applies to a depthwise convolution (DO-DConv) and to a grouped convolution (DO-GConv). For DO-DConv the composition is written as W ∘ (D ∘ P) in feature form and as an equivalent kernel form §3 eq. 4. Both D and W are trained, and the folded kernel is used at inference §3. The paper says that DO-DConv can replace a depthwise convolution (as in MobileNets) and that the same principle gives DO-GConv for grouped convolutions (as in ResNeXt) §3.

Pseudocode

The first block is the layer for training, using kernel composition. Shapes follow the reference code do_conv_pytorch.py: C_in and C_out are channels, M×N is the kernel window, D_mul is the depth multiplier.

# DO-Conv layer, kernel composition  [§3] [eq. 3]
# Shapes follow the reference code:  [code: do_conv_pytorch.py#L55-L73]
#   D_prime: (C_in, M*N, D_mul)     trainable, starts at zero  [§4]
#   fixed_I: (C_in, M*N, D_mul)     identity blocks, not trained  [§4]
#   W:       (C_out, C_in, D_mul)   trainable

def init_do_conv(C_in, C_out, M, N, D_mul):
    # The paper requires D_mul >= M*N  [§3]
    # Trainable offset D' = D - I starts at zero  [§4]
    D_prime = zeros(C_in, M * N, D_mul)
    # Fixed identity: as many M*N blocks as fit, rest zero  [§4]
    fixed_I = identity_tiled(C_in, M * N, D_mul)
    # Init of W is not stated; the reference code uses a
    # uniform init  [code: do_conv_pytorch.py#L59-L60]
    W = uniform_init(C_out, C_in, D_mul)
    return D_prime, fixed_I, W

The forward pass during training, and the fold for inference, follow.

# Training forward pass  [§3] [eq. 3]
def do_conv_forward(x, D_prime, fixed_I, W, bias):
    # Effective depthwise kernel D = D' + I  [§4]
    # [code: do_conv_pytorch.py#L119]
    D = D_prime + fixed_I                  # (C_in, M*N, D_mul)
    # Kernel composition W' = D^T o W: sum over D_mul  [§3] [eq. 3]
    # [code: do_conv_pytorch.py#L120-L125]
    W_comp = einsum("ims,ois->oim", D, W)  # (C_out, C_in, M*N)
    kernel = reshape(W_comp, (C_out, C_in, M, N))
    # One ordinary convolution with the composite kernel  [§3]
    return conv2d(x, kernel, bias)

# Fold after training, for inference  [§3]
# [code: do_conv_pytorch.py#L139-L141]
def fold_for_inference(D_prime, fixed_I, W, C_out, C_in, M, N):
    # Same composition as the forward pass, computed once  [§3]
    D = D_prime + fixed_I                  # (C_in, M*N, D_mul)
    kernel = einsum("ims,ois->oim", D, W)  # (C_out, C_in, M*N)
    # Store only this kernel and the bias in a plain convolution
    return reshape(kernel, (C_out, C_in, M, N))

Hyperparameters

Results

Author-reported. All numbers are top-1 accuracy (%) for classification, mIoU for segmentation, and COCO AP for detection, unless noted. The "change" column is the DO-Conv row of the paper's table, which is a difference from the authors' retrained baseline of the same row §4.1 tab. 1 tab. 2.

Setup: CIFAR-10 and CIFAR-100 with Plain, ResNet-v1 and ResNet-v2 §4.1; ImageNet via GluonCV model-zoo settings §4.1; Cityscapes and PASCAL VOC with DeepLabv3 §4.2; COCO with Faster R-CNN §4.3. Hardware is not stated.

Table 1, CIFAR (top-1 %, tab. 1):

Table 2, ImageNet, DO-Conv on all non-pointwise layers tab. 2:

Table 3, ImageNet, depthwise and grouped variants (DO-D/GConv) tab. 3:

Table 4, semantic segmentation (mIoU %) tab. 4: rows give where DO-Conv is used: backbone pre-training and segmentation fine-tuning.

The paper states that the second and third rows are relative to the first row §4.2.

Table 5, object detection on COCO, Faster R-CNN ResNet-50 tab. 5:

The paper states that DO-Conv in the detection stage only does not improve overall performance, while DO-Conv in both stages does §4.3.

Table 6, which ResNet stages get DO-Conv tab. 6: changes relative to the baseline (CIFAR-100 with ResNet-v2-20 baseline 67.14; ImageNet with ResNet-v1b-50 baseline 77.56). A dash marks a row not run in the paper.

Table 7, initialization of D, ImageNet ResNet-v1b-50 (baseline 77.56) tab. 7: random (He) init +0.18; identity init +0.44.

Table 8, depth multiplier, CIFAR-100 ResNet-v2-20 (baseline 67.14) tab. 8: D_mul = M×N +0.55; D_mul = (M+2)×(N+2) +0.36; D_mul = (2M)×(2N) +0.47.

Figures, described in words:

• fig. 5: on ImageNet with ResNet-v1b at several depths, DO-Conv's train and validation curves over 120 epochs. The paper reports faster convergence and a lower final error, and says it is the first to report lower error on mainstream architectures §4.4.
• fig. 6: visualization of ResNet-v2-20 kernels from CIFAR-100. It plots the accumulated absolute difference H between W' and W per kernel position. Some layers show a skeleton pattern, as in ACNet, and others differ §4.4.

Paper vs code

Official code: https://github.com/yangyanli/DO-Conv, MIT License, at commit 66fa86229020d4e5795c1072f3bd857673768b03. Author Yangyan Li is a co-author of the paper LICENSE. Repository files: PyTorch (do_conv_pytorch.py, do_conv_pytorch_1_10.py), TensorFlow (do_conv_tensorflow.py), GluonCV (do_conv_gluoncv.py), and MNIST demos.

• Identity init and trainable offset: match. The paper optimizes D' = D − I, zero-initialized, and adds the identity §4. The PyTorch code keeps D at zero and adds a fixed identity buffer in the forward pass do_conv_pytorch.py do_conv_pytorch.py. TensorFlow and GluonCV do the same do_conv_tensorflow.py do_conv_gluoncv.py do_conv_gluoncv.py.
• Composition formula: match. Paper: W' = D^T ∘ W with the sum over D_mul §3 eq. 3. Code: einsum('ims,ois->oim', D, W) do_conv_pytorch.py.
• Default D_mul: match. Paper uses D_mul = M×N in experiments §4. Code sets D_mul = M*N when no value is given do_conv_pytorch.py.
• D_mul > M×N identity blocks: match. Paper fills identity blocks as many as fit and zeros the rest §4.5. Code tiles identity blocks, then pads with zeros do_conv_pytorch.py do_conv_tensorflow.py.
• 1×1 layers: match. Paper says DO-Conv is not applied when M×N = 1 §4. Code uses the plain reshape for that case do_conv_pytorch.py.
• Constraint D_mul ≥ M×N: not enforced. Paper calls it necessary §3. The code accepts any D_mul (see Pitfalls) do_conv_pytorch.py.
• Init of W: not in paper. The PyTorch code uses PyTorch's uniform init with a = √5 do_conv_pytorch.py. TensorFlow uses glorot_uniform as the default do_conv_tensorflow.py.
• ImageNet, CIFAR, segmentation and detection training: not in the repository. The paper's Tables 1–5 cannot be reproduced from this repository alone. It holds layer code and small MNIST demos. The paper cites GluonCV commit bbe4166 for its ImageNet implementation §4.1; that commit is not in this repository.
• GluonCV version supports only D_mul = M×N. Its docstring says so [code: do_conv_gluoncv.py#L16-L17], and line 69 sets D_mul = M*N do_conv_gluoncv.py. The paper's D_mul ablation §4.5 does not say which implementation was used.
• MNIST demo (not in paper). The README reports a baseline of 95.156 and a DO-Conv result of 95.674 averaged over five runs (+0.518) README.md. That demo trains a small model for one epoch, and the paper does not report it sample_pt.py.
• PyTorch 1.10 file: same layer as the PyTorch file, except the import of the container type do_conv_pytorch_1_10.py.
• Appendix proof: the equivalence proof sits in 08_appendix.tex, which main.tex leaves commented out, so it is not part of the compiled paper §3.

Pitfalls

• D_mul < M×N gives a zero kernel at initialization (read from the code, not run). The paper requires D_mul ≥ M×N §3. The PyTorch code does not check. When D_mul < M×N, the identity repeat count is zero, so the fixed identity term is all zeros do_conv_pytorch.py. With D' also zero, D is zero and the composite kernel is zero at start do_conv_pytorch.py. Gradients to W are zero at the start in that case. Add an explicit check.
• Identity init makes W' = W only for D_mul = M×N. The paper's reuse of pre-trained baseline weights as W depends on this initial equality §4. With random init (the ablation) that equality does not hold §4.5 tab. 7.
• Weight decay: value and target not stated. The paper optimizes D' rather than D because L2 regularization could suppress the diagonal of D §4. Implement decay on D' only. The reference code keeps the identity as a fixed buffer, so decay cannot reach it do_conv_pytorch.py.
• Grouped convolutions: unverified. The PyTorch forward reshapes W to (out/groups, in, D_mul) and combines it with D over all input channels do_conv_pytorch.py. I did not check that this matches grouped convolution layout for 1 < groups < C_in. The paper's DO-GConv result on ResNeXt (+0.40) tab. 3 rests on it. Test the layout before using DO-GConv with groups.
• Folding is exact only for the linear operator. The fold reproduces the convolution with the same stride, padding, dilation and groups. Changing any of them after folding changes the result §3 do_conv_pytorch.py.
• Training cost is not measured. The paper gives MACC formulas only, with symbolic H and W §3. No wall-clock training time for DO-Conv is reported.
• Results are small and mixed. ImageNet deltas range from +0.03 to +1.01 tab. 2 tab. 3. CIFAR has negative rows (−0.05, −0.06, −0.07) tab. 1. Detection has a −1.0 AP_L row tab. 5. No seed variance is given for the ImageNet numbers; CIFAR numbers are averages of the last five epochs over five runs §4.1.
• Baselines are retrained by the authors. The paper says GluonCV reference numbers are shown but the comparison is against their own retrained baseline §4.1 tab. 2. Compare against your own baseline with the same settings.
• Not stated: learning rate, batch size, optimizer, epochs for CIFAR and ImageNet runs, hardware, and the number of ImageNet runs §4.1.

Lineage

• Builds on: over-parameterization of deep linear and non-linear networks by Arora et al. (ICML 2018; no arXiv id in the paper's bibliography) §1 §2.
• Related constructions: ExpandNet (arXiv:1811.10495) over-parameterizes the channel axes §2; ACNet (ICCV 2019, no arXiv id in the bibliography) uses asymmetric kernels that the paper treats as a special case §2.
• Depthwise separable convolution: Xception (CVPR 2017, no arXiv id in the bibliography) and MobileNets (arXiv:1704.04861). DO-Conv's feature composition with D_mul < M×N matches this operator §3.
• Compared against: the baseline networks of §4.1 (ResNet-v1, ResNet-v1b, ResNet-v2, Plain, ResNeXt, MobileNet v1–v3) and DeepLabv3 and Faster R-CNN baselines §4.2 §4.3.
• Superseded by: not known from wiki content.