---
kind: concept
slug: double-descent
title: Double descent
summary: >-
  Test error can rise near the point where a model first fits its training data exactly, then fall again as the model grows. The explanation offered is that the minimum-norm interpolant gets smoother with more capacity. One card so far, a survey essay with no experiments of its own: the descending branch is shown in illustrations and argued, not measured across architectures.
tags: [generalization, interpolation, over-parameterization, minimum-norm, double-descent]
papers: ["2105.14368"]
related: [over-parameterization, inductive-bias, neural-tangent-kernel, implicit-bias-of-gradient-descent]
status: draft
updated_at: 2026-11-06
---

## Overview

Classical learning theory expects test error to follow a U-shape: it falls as
capacity grows, then rises once the model starts fitting noise. The essay
argues this picture is incomplete for over-parameterized models. A model that
can fit every training point, labels included, may still generalize, and the
choice among the exact fits matters [arxiv:2105.14368 §1] [arxiv:2105.14368 §3.4].

The "double descent" picture has two parts. Test risk peaks near the
interpolation threshold, the capacity at which the model can first fit the
training set exactly. Beyond that threshold, test risk falls again as the model
grows [arxiv:2105.14368 §3.7] [fig. 5]. The explanation is an inductive bias:
among the interpolating solutions, the one with the smallest norm is the
smoothest, and as capacity increases this minimum-norm interpolant becomes
smoother and generalizes better [arxiv:2105.14368 §3.6] [arxiv:2105.14368 §3.7].

Only one card covers this concept, and that card is a survey essay. It reports
no experiments of its own. Its double-descent figures are credited to earlier
work, and the wide-network part of the argument is heuristic
[arxiv:2105.14368 §3.10] [arxiv:2105.14368 §6.2]. Read the variants below as
settings the essay describes, not as independently measured results.

## Variants

The essay shows the descending branch in three settings. They differ in what
is swept as "capacity" and in how the interpolant is chosen.

### Random-feature models swept by width

A model with fixed random features (random Fourier features, or a random
ReLU-type layer) and a trained output weight. The number of features is the
capacity. For random Fourier features, the essay fits by minimum-norm
interpolation [arxiv:2105.14368 §3.7]. For the random-ReLU illustration it
shows an overfitting peak near the threshold and a smooth fit at 3000
parameters, but does not state the optimizer [fig. 6] [arxiv:2105.14368 §3.7].
Card: [Belkin's survey essay](/papers/2105.14368).

### Over-parameterized linear least squares

With more features than examples, the least-squares fit is not unique. The
minimum-norm interpolator is the pseudo-inverse solution. The essay states
that gradient descent started at zero stays in the span of the training
inputs and reaches this same solution [arxiv:2105.14368 §3.8] [arxiv:2105.14368 §3.9].
This gives a concrete, checkable case of implicit selection. Card: [Belkin's survey essay](/papers/2105.14368).

### Wide networks in the kernel regime

For very wide networks with a linear output layer, the essay argues that the
tangent kernel stays nearly constant near initialization, so training behaves
like a kernel machine [arxiv:2105.14368 §3.10]. The argument is heuristic. The
Hessian bound has unspecified constants, the two-layer example uses fixed output
weights, and the essay says these results do not settle whether deep learning is
kernel learning [arxiv:2105.14368 §3.10] [arxiv:2105.14368 §6.2]. The essay does
not show a double-descent curve for this setting. Card: [Belkin's survey essay](/papers/2105.14368).

## When to use which

The essay gives no decision rule with measured results. The table lists what
its cited settings support.

| Situation | Use | Why |
|---|---|---|
| You want to test whether an exact fit must hurt test error | Sweep a model with a fixed random feature map, fit by minimum-norm interpolation, and record error across widths | The essay describes a peak near the threshold and a descending branch beyond it for this family [fig. 6] [arxiv:2105.14368 §3.7] |
| Linear over-parameterized regression started from zero | Expect the gradient-descent limit to be the minimum-norm interpolator | Stated for linear models with gradient descent from zero [arxiv:2105.14368 §3.8] [arxiv:2105.14368 §3.9] |
| A deep network trained with early stopping, as is common in practice | Do not transfer the exact-interpolation claims directly | The essay notes early stopping is usual and its trade-offs are not well understood [arxiv:2105.14368 §6.2] |
| You need a predicted shape for an architecture not in the essay | Treat the descending branch as a hypothesis to test | The essay's link from wide networks to kernels is heuristic [arxiv:2105.14368 §3.10] |
| You need a numeric guarantee on test error | Look elsewhere | The essay's bounds hold only up to constants or extra conditions that it does not spell out [arxiv:2105.14368 §3.5.2] |

## Implementation sketch

A sweep to see the curve yourself. The feature map is the essay's setting:
fixed random features, fit by minimum-norm interpolation [arxiv:2105.14368 §3.7].
The sketch is an illustration, not a reproduction of the essay's figure.

```pseudocode
# Sweep feature count through the interpolation threshold  [arxiv:2105.14368 §3.7]
# Input:  train (X, y), test (X_test, y_test), feature counts m_list
# Output: test error for each m

def double_descent_curve(X, y, X_test, y_test, m_list):
    test_errors = []
    for m in m_list:
        # Fixed random projection, not trained  [arxiv:2105.14368 §3.7]
        V = draw_random_projection(X.shape[1], m)
        Phi = random_features(X, V)         # shape (n, m)
        Phi_test = random_features(X_test, V)
        # Minimum-norm interpolant: smallest w with Phi @ w = y  [arxiv:2105.14368 §3.8]
        w = pinv(Phi) @ y
        test_errors.append(mse(Phi_test @ w, y_test))
    # Expect a peak near the interpolation threshold and a fall beyond it  [fig. 6]
    return test_errors
```

## Contradictions

None known. Only one card covers this concept, so there are no disagreements
between papers. Inside the essay, the author flags its own heuristic steps
(the wide-network link and the PL* constant) and says the trade-offs of
early stopping are not understood [arxiv:2105.14368 §3.10] [arxiv:2105.14368 §4.2] [arxiv:2105.14368 §6.2].

## Papers

- [Belkin's survey essay](/papers/2105.14368): survey essay by Mikhail Belkin that presents the double-descent picture, min-norm interpolation and the kernel view of wide networks, without new experiments.