Over-parameterization

Overview

Over-parameterization means a model has more trainable parameters or capacity than a minimal description of the target needs. The wiki covers two senses of the term, and the cards use them differently.

1. Redundant parameters folded away after training. Extra linear parameters are added during training and then multiplied out into an ordinary layer, so inference is unchanged. The invariant is that the training-time structure folds back into the original shape 2006.12030 §3.
2. Capacity enough to fit the training data exactly (interpolation). Nothing is folded away. The question is whether a model that reaches zero training error still predicts well, and the answer depends on which exact fit training selects 2105.14368 §3.6 2105.14368 §3.7.

Variants

Training-time over-parameterization: DO-Conv

DO-Conv: depthwise over-parameterized convolution

Each non-pointwise convolution is replaced by a depthwise kernel D, with one small kernel per input channel and D_mul outputs each, followed by an ordinary kernel W. The paper requires D_mul ≥ M×N and uses D_mul = M×N in its experiments 2006.12030 §3 2006.12030 §4. Both D and W are trained, then folded into one kernel W' = D^T ∘ W with the shape of a normal convolution kernel 2006.12030 §3 eq. 3. Card: DO-Conv: Depthwise Over-parameterized Convolutional Layer.

Two details matter for reuse of existing weights. The paper optimizes an offset D' = D − I, starts it at zero and adds a fixed identity, so that with D_mul = M×N the initial W' equals W 2006.12030 §4. The paper also reports that random (He) initialization of D gives a smaller ImageNet gain than identity initialization 2006.12030 tab. 7.

Feature versus kernel composition

The same operator can be evaluated in two orders with equal values. Feature composition applies D to the patch and then W. Kernel composition first forms W' and runs one ordinary convolution. The paper gives multiply-accumulate counts that scale with the feature-map size for feature composition and with the output channels and kernel window for kernel composition, and recommends kernel composition for training 2006.12030 §3. The receptive field stays M×N in both forms 2006.12030 §3.

Depthwise and grouped variants: DO-DConv and DO-GConv

The same construction is applied to depthwise convolutions (DO-DConv, which the paper says can replace MobileNet-style depthwise layers) and to grouped convolutions (DO-GConv, as in ResNeXt) 2006.12030 §3 2006.12030 eq. 4. Reported ImageNet changes are +0.03 to +0.16 for three MobileNet versions and +0.40 for ResNeXt-50 (32×4d) 2006.12030 tab. 3. Card: DO-Conv: Depthwise Over-parameterized Convolutional Layer.

Relation to depthwise separable convolution

The paper notes that feature composition with D_mul smaller than M×N is the same computation as a depthwise separable convolution. It suggests kernel composition could speed up training of such layers on memory-bound GPUs, and then deployment with feature composition on edge devices. The authors call this a by-product and do not measure it 2006.12030 §3. See Depthwise separable convolution for that operator on its own.

Interpolating over-parameterized models

The essay by Belkin is a survey and an interpretation, not a new method. It reports no experiments of its own, and its main points are arguments and summaries of earlier work. Card: Fit without fear, a 2021 essay by Mikhail Belkin.

• Interpolation is not automatically harmful. Classical generalization bounds suggest that fitting noisy data exactly should hurt test error. The essay argues that interpolating kernel machines trained on label-noisy data can still approach the Bayes error, which it says is hard to reconcile with those bounds in general 2105.14368 §3.4 2105.14368 fig. 2.
• The inductive bias chooses the fit. Among the functions that interpolate the training data, the essay prefers the one with the smallest norm in the kernel's reproducing Hilbert space, and ties this preference to Occam's razor 2105.14368 §3.6.
• Double descent. As capacity grows past the interpolation threshold, the minimum-norm interpolant gets smoother, and test error can fall again after a peak at the threshold 2105.14368 §3.7. See Double descent.
• Linear case. For linear models with more features than samples, the least-squares minimum-norm interpolator is the solution that gradient descent started at zero reaches 2105.14368 §3.8 2105.14368 §3.9.
• Wide networks as kernels. For very wide networks with a linear output layer, the essay argues that the tangent kernel is nearly constant near initialization, so the model behaves like a kernel machine. A non-linear output layer breaks this. The argument is heuristic, and the essay says it does not settle whether deep learning is kernel learning 2105.14368 §3.10 2105.14368 §6.2. See Neural tangent kernel.
• Optimization. The essay argues that over-parameterized losses are non-convex but satisfy a Polyak-Łojasiewicz variant on most of the parameter space, which would give global minima for gradient methods. The constant in this argument is not bounded 2105.14368 §4.1 2105.14368 §4.2. For interpolated convex problems, it cites exponential convergence of SGD with a fixed step size, and a critical batch size above which larger batches stop helping 2105.14368 §4.4.

When to use which

Folding a training-time structure (DO-Conv)

The paper gives one comparison protocol: replace every non-pointwise convolution and change nothing else 2006.12030 §4. Its results support the rows below, with the caveat that gains are small and not uniform.

The paper reports no wall-clock training time for any variant. Its cost comparison is symbolic only 2006.12030 §3, so the training-speed rows above rest on operation counts, not measurements.

Models that fit the training data exactly

These rows state what the essay argues. None is backed by experiments in the essay itself.

Implementation sketch

DO-Conv layer

The shared core of DO-Conv is the composition of a depthwise kernel D and an ordinary kernel W, and the fold that removes D at the end. Shapes follow the paper's notation: C_in input channels, C_out output channels, M×N window, D_mul depth multiplier 2006.12030 §3.

# Trainable structure, one per layer  [arxiv:2006.12030 §4]
def init_over_param_conv(C_in, C_out, M, N, D_mul):
    # Paper requires D_mul >= M*N  [arxiv:2006.12030 §3]
    # Offset starts at zero; identity is fixed  [arxiv:2006.12030 §4]
    D_offset = zeros(C_in, M * N, D_mul)
    identity = identity_blocks(C_in, M * N, D_mul)
    # Ordinary kernel; the init is not stated in the paper  [arxiv:2006.12030 §4]
    W = init(C_out, C_in, D_mul)
    return D_offset, identity, W

# Training forward: compose D and W into one kernel  [arxiv:2006.12030 §3] [eq. 3]
def forward(x, D_offset, identity, W, bias):
    # Effective depthwise kernel  [arxiv:2006.12030 §4]
    D = D_offset + identity            # (C_in, M*N, D_mul)
    # Sum over the depth multiplier  [arxiv:2006.12030 §3] [eq. 3]
    K = compose(D, W)                  # (C_out, C_in, M*N)
    # One ordinary convolution with the composite kernel  [arxiv:2006.12030 §3]
    return conv2d(x, reshape(K, (C_out, C_in, M, N)), bias)

# Fold after training, for inference  [arxiv:2006.12030 §3]
def fold(D_offset, identity, W):
    # Same composition as the forward pass, computed once  [arxiv:2006.12030 §3]
    return compose(D_offset + identity, W)  # plain conv kernel, no extra cost

Notes on where variants differ. Weight decay: the paper optimizes the offset rather than D, because L2 on D could suppress its diagonal, and it does not state a decay value 2006.12030 §4. Stride, padding, dilation and groups must be the same before and after folding 2006.12030 §3. Grouped and depthwise variants change the shapes of D and W 2006.12030 §3.

Minimum-norm interpolator (linear case)

# Minimum-norm interpolator, linear model with d > n  [arxiv:2105.14368 §3.8]
# Input:  X (n, d) with d > n, labels y (n,)
# Output: beta (d,), with X @ beta equal to y

def min_norm_interpolator(X, y):
    # Smallest-norm exact fit, via the pseudo-inverse  [arxiv:2105.14368 §3.8]
    return pinv(X) @ y
    # Gradient descent from beta = 0 reaches the same solution  [arxiv:2105.14368 §3.9]

Contradictions

No claim in the wiki directly contradicts another, but the two senses of the term do not share a measure. DO-Conv's extra parameters are linear and removed at inference. The essay's extra capacity stays in the model and concerns interpolation. Their results cannot be compared on one number.

Within DO-Conv, the effect is mixed rather than uniformly positive. CIFAR-10 ResNet-v1 at depth 110 and 164 are −0.05 and −0.06, and ResNet-v2 at depth 164 is −0.07 2006.12030 tab. 1. Detection with DO-Conv only in the detection stage gives a −1.0 AP_L change 2006.12030 tab. 5. The paper's results are small and not uniform across architectures, with some rows flat or negative. The paper also says it is the first to report lower error on mainstream architectures, which is the authors' claim and not independently checked here 2006.12030 §4.4.

The essay's claims are arguments and summaries of earlier work with no experiments of its own. It says that neural networks are usually trained with early stopping rather than to exact interpolation, and that the effect of that difference is not well understood, so its results may not transfer directly to practical training 2105.14368 §6.2.

Papers

• DO-Conv: Depthwise Over-parameterized Convolutional Layer: replaces each non-pointwise convolution with a depthwise over-parameterized pair that folds into one kernel, with small mixed gains reported on CIFAR, ImageNet, segmentation and detection.
• Fit without fear: survey essay arguing that exact fits to training data need not hurt generalization, covering double descent, minimum-norm interpolants, the neural tangent kernel and optimization by gradient methods.