--- kind: concept slug: over-parameterization title: Over-parameterization summary: >- Two senses: training with extra linear parameters that fold back into one ordinary layer (DO-Conv reports small, mixed ImageNet gains and no measured training cost), and models with enough capacity to fit training data exactly (Belkin's essay argues this need not hurt generalization, depending on inductive bias). tags: [training, architecture, convolution, generalization, interpolation, over-parameterization] papers: ["2006.12030", "2105.14368"] related: [convolutional-neural-network, depthwise-separable-convolution, batchnorm-folding, double-descent, neural-tangent-kernel, inductive-bias] status: draft updated_at: 2026-11-06 --- ## Overview Over-parameterization means a model has more trainable parameters or capacity than a minimal description of the target needs. The wiki covers two senses of the term, and the cards use them differently. 1. **Redundant parameters folded away after training.** Extra linear parameters are added during training and then multiplied out into an ordinary layer, so inference is unchanged. The invariant is that the training-time structure folds back into the original shape [arxiv:2006.12030 §3]. 2. **Capacity enough to fit the training data exactly (interpolation).** Nothing is folded away. The question is whether a model that reaches zero training error still predicts well, and the answer depends on which exact fit training selects [arxiv:2105.14368 §3.6] [arxiv:2105.14368 §3.7]. ## Variants ### Training-time over-parameterization: DO-Conv #### DO-Conv: depthwise over-parameterized convolution Each non-pointwise convolution is replaced by a depthwise kernel D, with one small kernel per input channel and D_mul outputs each, followed by an ordinary kernel W. The paper requires D_mul ≥ M×N and uses D_mul = M×N in its experiments [arxiv:2006.12030 §3] [arxiv:2006.12030 §4]. Both D and W are trained, then folded into one kernel W' = D^T ∘ W with the shape of a normal convolution kernel [arxiv:2006.12030 §3] [eq. 3]. Card: [DO-Conv: Depthwise Over-parameterized Convolutional Layer](/papers/2006.12030). Two details matter for reuse of existing weights. The paper optimizes an offset D' = D − I, starts it at zero and adds a fixed identity, so that with D_mul = M×N the initial W' equals W [arxiv:2006.12030 §4]. The paper also reports that random (He) initialization of D gives a smaller ImageNet gain than identity initialization [arxiv:2006.12030 tab. 7]. #### Feature versus kernel composition The same operator can be evaluated in two orders with equal values. Feature composition applies D to the patch and then W. Kernel composition first forms W' and runs one ordinary convolution. The paper gives multiply-accumulate counts that scale with the feature-map size for feature composition and with the output channels and kernel window for kernel composition, and recommends kernel composition for training [arxiv:2006.12030 §3]. The receptive field stays M×N in both forms [arxiv:2006.12030 §3]. #### Depthwise and grouped variants: DO-DConv and DO-GConv The same construction is applied to depthwise convolutions (DO-DConv, which the paper says can replace MobileNet-style depthwise layers) and to grouped convolutions (DO-GConv, as in ResNeXt) [arxiv:2006.12030 §3] [arxiv:2006.12030 eq. 4]. Reported ImageNet changes are +0.03 to +0.16 for three MobileNet versions and +0.40 for ResNeXt-50 (32×4d) [arxiv:2006.12030 tab. 3]. Card: [DO-Conv: Depthwise Over-parameterized Convolutional Layer](/papers/2006.12030). #### Relation to depthwise separable convolution The paper notes that feature composition with D_mul smaller than M×N is the same computation as a depthwise separable convolution. It suggests kernel composition could speed up training of such layers on memory-bound GPUs, and then deployment with feature composition on edge devices. The authors call this a by-product and do not measure it [arxiv:2006.12030 §3]. See [Depthwise separable convolution](/concepts/depthwise-separable-convolution) for that operator on its own. ### Interpolating over-parameterized models The essay by Belkin is a survey and an interpretation, not a new method. It reports no experiments of its own, and its main points are arguments and summaries of earlier work. Card: [Fit without fear](/papers/2105.14368), a 2021 essay by Mikhail Belkin. - **Interpolation is not automatically harmful.** Classical generalization bounds suggest that fitting noisy data exactly should hurt test error. The essay argues that interpolating kernel machines trained on label-noisy data can still approach the Bayes error, which it says is hard to reconcile with those bounds in general [arxiv:2105.14368 §3.4] [arxiv:2105.14368 fig. 2]. - **The inductive bias chooses the fit.** Among the functions that interpolate the training data, the essay prefers the one with the smallest norm in the kernel's reproducing Hilbert space, and ties this preference to Occam's razor [arxiv:2105.14368 §3.6]. - **Double descent.** As capacity grows past the interpolation threshold, the minimum-norm interpolant gets smoother, and test error can fall again after a peak at the threshold [arxiv:2105.14368 §3.7]. See [Double descent](/concepts/double-descent). - **Linear case.** For linear models with more features than samples, the least-squares minimum-norm interpolator is the solution that gradient descent started at zero reaches [arxiv:2105.14368 §3.8] [arxiv:2105.14368 §3.9]. - **Wide networks as kernels.** For very wide networks with a linear output layer, the essay argues that the tangent kernel is nearly constant near initialization, so the model behaves like a kernel machine. A non-linear output layer breaks this. The argument is heuristic, and the essay says it does not settle whether deep learning is kernel learning [arxiv:2105.14368 §3.10] [arxiv:2105.14368 §6.2]. See [Neural tangent kernel](/concepts/neural-tangent-kernel). - **Optimization.** The essay argues that over-parameterized losses are non-convex but satisfy a Polyak-Łojasiewicz variant on most of the parameter space, which would give global minima for gradient methods. The constant in this argument is not bounded [arxiv:2105.14368 §4.1] [arxiv:2105.14368 §4.2]. For interpolated convex problems, it cites exponential convergence of SGD with a fixed step size, and a critical batch size above which larger batches stop helping [arxiv:2105.14368 §4.4]. ## When to use which ### Folding a training-time structure (DO-Conv) The paper gives one comparison protocol: replace every non-pointwise convolution and change nothing else [arxiv:2006.12030 §4]. Its results support the rows below, with the caveat that gains are small and not uniform. | Situation | Use | Why | |---|---|---| | You need the trained model to have the original inference cost | Fold after training (DO-Conv) | The folded kernel is an ordinary convolution with the same stride, padding and dilation [arxiv:2006.12030 §3] | | You want to start from existing baseline weights | D_mul = M×N, identity-initialized offset | Gives W' = W at initialization [arxiv:2006.12030 §4] | | You want the largest reported gain on CIFAR-100 | Replace stages 0–3 with D_mul = M×N (stage 4 was not run on CIFAR) | Stages 0–3 gave +0.55 on ResNet-v2-20 [arxiv:2006.12030 tab. 6] [arxiv:2006.12030 tab. 8] | | You are not sure which stages to change | Measure on your own baseline | Partial replacement is mixed: one ImageNet ResNet-v1b-50 stage row is −0.07 [arxiv:2006.12030 tab. 6] | | Your network uses depthwise layers (MobileNet-style) | DO-DConv | Reported changes +0.03 to +0.16 on three MobileNet versions [arxiv:2006.12030 tab. 3] | | Your network uses grouped convolutions (ResNeXt-style) | DO-GConv, after a layout test | Reported +0.40 on ResNeXt-50 [arxiv:2006.12030 tab. 3] | | Your 1×1 convolutions dominate | Keep them plain | The paper does not apply DO-Conv when M×N = 1 [arxiv:2006.12030 §4] | | Your GPU training is memory-bound and feature maps are large | Kernel composition for training | The paper's cost argument favours it when H×W is larger than C_out [arxiv:2006.12030 §3] | | You need a guaranteed accuracy gain | Do not rely on this | Several reported rows are flat or negative [arxiv:2006.12030 tab. 1] [arxiv:2006.12030 tab. 5] | The paper reports no wall-clock training time for any variant. Its cost comparison is symbolic only [arxiv:2006.12030 §3], so the training-speed rows above rest on operation counts, not measurements. ### Models that fit the training data exactly These rows state what the essay argues. None is backed by experiments in the essay itself. | Situation | Use | Why | |---|---|---| | Your model reaches zero training error and you expect poor test error | Do not assume it; examine which fit training selects | The essay argues interpolation need not hurt, and that the inductive bias decides [arxiv:2105.14368 §3.4] [arxiv:2105.14368 §3.6] | | Your model size is near the point where it can just fit the data | Expect a possible test-error peak at the threshold; compare sizes on both sides | Double descent is described in [arxiv:2105.14368 §3.7] | | Linear regression with more features than samples | Minimum-norm solution via the pseudo-inverse; gradient descent from zero reaches it | [arxiv:2105.14368 §3.8] [arxiv:2105.14368 §3.9] | | You want to predict generalization from a classical bound | Do not rely on it for interpolating models | The essay argues such bounds do not explain interpolating kernel machines on label-noisy data [arxiv:2105.14368 §3.3] [arxiv:2105.14368 §3.4] | | Your network has a non-linear output layer | Do not use the kernel-regime argument | The essay says the linearity argument breaks with a non-linear output layer [arxiv:2105.14368 §3.10] | | You use fixed-step SGD on an interpolated convex problem | Expect the cited exponential rate; check your batch size | Stated as cited earlier work, with a batch-size formula derived for quadratic models [arxiv:2105.14368 §4.4] | ## Implementation sketch ### DO-Conv layer The shared core of DO-Conv is the composition of a depthwise kernel D and an ordinary kernel W, and the fold that removes D at the end. Shapes follow the paper's notation: C_in input channels, C_out output channels, M×N window, D_mul depth multiplier [arxiv:2006.12030 §3]. ```pseudocode # Trainable structure, one per layer [arxiv:2006.12030 §4] def init_over_param_conv(C_in, C_out, M, N, D_mul): # Paper requires D_mul >= M*N [arxiv:2006.12030 §3] # Offset starts at zero; identity is fixed [arxiv:2006.12030 §4] D_offset = zeros(C_in, M * N, D_mul) identity = identity_blocks(C_in, M * N, D_mul) # Ordinary kernel; the init is not stated in the paper [arxiv:2006.12030 §4] W = init(C_out, C_in, D_mul) return D_offset, identity, W ``` ```pseudocode # Training forward: compose D and W into one kernel [arxiv:2006.12030 §3] [eq. 3] def forward(x, D_offset, identity, W, bias): # Effective depthwise kernel [arxiv:2006.12030 §4] D = D_offset + identity # (C_in, M*N, D_mul) # Sum over the depth multiplier [arxiv:2006.12030 §3] [eq. 3] K = compose(D, W) # (C_out, C_in, M*N) # One ordinary convolution with the composite kernel [arxiv:2006.12030 §3] return conv2d(x, reshape(K, (C_out, C_in, M, N)), bias) ``` ```pseudocode # Fold after training, for inference [arxiv:2006.12030 §3] def fold(D_offset, identity, W): # Same composition as the forward pass, computed once [arxiv:2006.12030 §3] return compose(D_offset + identity, W) # plain conv kernel, no extra cost ``` Notes on where variants differ. Weight decay: the paper optimizes the offset rather than D, because L2 on D could suppress its diagonal, and it does not state a decay value [arxiv:2006.12030 §4]. Stride, padding, dilation and groups must be the same before and after folding [arxiv:2006.12030 §3]. Grouped and depthwise variants change the shapes of D and W [arxiv:2006.12030 §3]. ### Minimum-norm interpolator (linear case) ```pseudocode # Minimum-norm interpolator, linear model with d > n [arxiv:2105.14368 §3.8] # Input: X (n, d) with d > n, labels y (n,) # Output: beta (d,), with X @ beta equal to y def min_norm_interpolator(X, y): # Smallest-norm exact fit, via the pseudo-inverse [arxiv:2105.14368 §3.8] return pinv(X) @ y # Gradient descent from beta = 0 reaches the same solution [arxiv:2105.14368 §3.9] ``` ## Contradictions No claim in the wiki directly contradicts another, but the two senses of the term do not share a measure. DO-Conv's extra parameters are linear and removed at inference. The essay's extra capacity stays in the model and concerns interpolation. Their results cannot be compared on one number. Within DO-Conv, the effect is mixed rather than uniformly positive. CIFAR-10 ResNet-v1 at depth 110 and 164 are −0.05 and −0.06, and ResNet-v2 at depth 164 is −0.07 [arxiv:2006.12030 tab. 1]. Detection with DO-Conv only in the detection stage gives a −1.0 AP_L change [arxiv:2006.12030 tab. 5]. The paper's results are small and not uniform across architectures, with some rows flat or negative. The paper also says it is the first to report lower error on mainstream architectures, which is the authors' claim and not independently checked here [arxiv:2006.12030 §4.4]. The essay's claims are arguments and summaries of earlier work with no experiments of its own. It says that neural networks are usually trained with early stopping rather than to exact interpolation, and that the effect of that difference is not well understood, so its results may not transfer directly to practical training [arxiv:2105.14368 §6.2]. ## Papers - [DO-Conv: Depthwise Over-parameterized Convolutional Layer](/papers/2006.12030): replaces each non-pointwise convolution with a depthwise over-parameterized pair that folds into one kernel, with small mixed gains reported on CIFAR, ImageNet, segmentation and detection. - [Fit without fear](/papers/2105.14368): survey essay arguing that exact fits to training data need not hurt generalization, covering double descent, minimum-norm interpolants, the neural tangent kernel and optimization by gradient methods.