Double descent

Overview

Classical learning theory expects test error to follow a U-shape: it falls as capacity grows, then rises once the model starts fitting noise. The essay argues this picture is incomplete for over-parameterized models. A model that can fit every training point, labels included, may still generalize, and the choice among the exact fits matters 2105.14368 §1 2105.14368 §3.4.

The "double descent" picture has two parts. Test risk peaks near the interpolation threshold, the capacity at which the model can first fit the training set exactly. Beyond that threshold, test risk falls again as the model grows 2105.14368 §3.7 fig. 5. The explanation is an inductive bias: among the interpolating solutions, the one with the smallest norm is the smoothest, and as capacity increases this minimum-norm interpolant becomes smoother and generalizes better 2105.14368 §3.6 2105.14368 §3.7.

Only one card covers this concept, and that card is a survey essay. It reports no experiments of its own. Its double-descent figures are credited to earlier work, and the wide-network part of the argument is heuristic 2105.14368 §3.10 2105.14368 §6.2. Read the variants below as settings the essay describes, not as independently measured results.

Variants

The essay shows the descending branch in three settings. They differ in what is swept as "capacity" and in how the interpolant is chosen.

Random-feature models swept by width

A model with fixed random features (random Fourier features, or a random ReLU-type layer) and a trained output weight. The number of features is the capacity. For random Fourier features, the essay fits by minimum-norm interpolation 2105.14368 §3.7. For the random-ReLU illustration it shows an overfitting peak near the threshold and a smooth fit at 3000 parameters, but does not state the optimizer fig. 6 2105.14368 §3.7. Card: Belkin's survey essay.

Over-parameterized linear least squares

With more features than examples, the least-squares fit is not unique. The minimum-norm interpolator is the pseudo-inverse solution. The essay states that gradient descent started at zero stays in the span of the training inputs and reaches this same solution 2105.14368 §3.8 2105.14368 §3.9. This gives a concrete, checkable case of implicit selection. Card: Belkin's survey essay.

Wide networks in the kernel regime

For very wide networks with a linear output layer, the essay argues that the tangent kernel stays nearly constant near initialization, so training behaves like a kernel machine 2105.14368 §3.10. The argument is heuristic. The Hessian bound has unspecified constants, the two-layer example uses fixed output weights, and the essay says these results do not settle whether deep learning is kernel learning 2105.14368 §3.10 2105.14368 §6.2. The essay does not show a double-descent curve for this setting. Card: Belkin's survey essay.

When to use which

The essay gives no decision rule with measured results. The table lists what its cited settings support.

Implementation sketch

A sweep to see the curve yourself. The feature map is the essay's setting: fixed random features, fit by minimum-norm interpolation 2105.14368 §3.7. The sketch is an illustration, not a reproduction of the essay's figure.

# Sweep feature count through the interpolation threshold  [arxiv:2105.14368 §3.7]
# Input:  train (X, y), test (X_test, y_test), feature counts m_list
# Output: test error for each m

def double_descent_curve(X, y, X_test, y_test, m_list):
    test_errors = []
    for m in m_list:
        # Fixed random projection, not trained  [arxiv:2105.14368 §3.7]
        V = draw_random_projection(X.shape[1], m)
        Phi = random_features(X, V)         # shape (n, m)
        Phi_test = random_features(X_test, V)
        # Minimum-norm interpolant: smallest w with Phi @ w = y  [arxiv:2105.14368 §3.8]
        w = pinv(Phi) @ y
        test_errors.append(mse(Phi_test @ w, y_test))
    # Expect a peak near the interpolation threshold and a fall beyond it  [fig. 6]
    return test_errors

Contradictions

None known. Only one card covers this concept, so there are no disagreements between papers. Inside the essay, the author flags its own heuristic steps (the wide-network link and the PL* constant) and says the trade-offs of early stopping are not understood 2105.14368 §3.10 2105.14368 §4.2 2105.14368 §6.2.

Papers

• Belkin's survey essay: survey essay by Mikhail Belkin that presents the double-descent picture, min-norm interpolation and the kernel view of wide networks, without new experiments.