---
kind: concept
slug: neural-tangent-kernel
title: Neural tangent kernel
summary: >-
  The neural tangent kernel (NTK) is the Gram matrix of a network's weight gradients over inputs. It is used as a label-free trainability score at initialisation, as a qualitative sign of spectral bias, and in a linearised model that removes training data. Evidence is thin, mostly author-reported, and rests on few measurements.
tags: [kernel, trainability, training-dynamics, spectral-bias, neural-architecture-search, convergence, linearization, machine-unlearning]
papers: ["2012.13431", "2102.11535", "2105.14368", "2510.25781"]
related: [neural-architecture-search, network-pruning, implicit-bias-of-gradient-descent, gradient-descent-stability, physics-informed-neural-network, learnable-activation-functions, multi-layer-perceptron, over-parameterization, double-descent, inductive-bias, machine-unlearning]
status: draft
updated_at: 2026-11-06
---

## Overview

For a network, the NTK on a pair of inputs is the product of the network's
output Jacobians with respect to its weights, evaluated at those inputs
[arxiv:2102.11535 §3.1.1]. Stacked over a set of inputs, it is a Gram matrix,
and it depends only on the network and the inputs, not on labels
[arxiv:2102.11535 §3.1.1]. The first paper writes training dynamics under
gradient descent in terms of the eigenvalues of this matrix [arxiv:2102.11535
eq. 1, eq. 2]. The second paper uses the same object, with its spectrum
shape, to reason about spectral bias and convergence [arxiv:2510.25781 §9.2].

Two other papers use the same object without a spectrum. A third paper
linearises a fine-tuned network around a fixed core, so the trainable part
enters through its Jacobian [arxiv:2012.13431 §5]. A fourth essay argues that
very wide networks with a linear output layer behave like kernel machines
near initialisation [arxiv:2105.14368 §3.10].

The invariant the variants share is that the matrix is computed at a fixed
point of the network, usually at initialisation or around a fixed core, and
the model is read through that point. The variants differ in what they do
with it.

## Variants

### Condition number as a training-free trainability score

The NTK is computed on one mini-batch. Its condition number, the ratio of
largest to smallest eigenvalue, is used as a trainability score where lower
is better [arxiv:2102.11535 eq. 3]. The paper combines it with a count of
linear regions and uses both ranks to prune a super-network without training
any candidate [arxiv:2102.11535 §3.2] [arxiv:2102.11535 alg. 1]. The paper
reports a CIFAR-10 search of 0.05 GPU days on one GTX 1080Ti and a test error
of 2.63% [arxiv:2102.11535 tab. 2]. Card:
[Four-GPU-hours NAS paper](/papers/2102.11535).

### Spectrum shape as a diagnostic for spectral bias

The review does not compute an NTK for a new model. It reports that flatter
NTK spectra, seen for KANs and for learnable-basis PINNs, indicate weaker
spectral bias, meaning less preference for low frequencies [arxiv:2510.25781
§9.2]. It adds that broader spectra can raise curvature and worsen
conditioning [arxiv:2510.25781 §9.2]. It also reports, from the literature, a
Chebyshev-based PIKAN with better NTK conditioning and faster convergence on
four PDEs [arxiv:2510.25781 §9.3] [arxiv:2510.25781 tab. 1]. Card:
[A Practitioner's Guide to Kolmogorov-Arnold Networks](/papers/2510.25781).

### Smallest eigenvalue as a convergence rate

The review gives a gradient-flow bound. If a matrix G = D^T D is positive
definite, the loss decays exponentially at a rate set by G's smallest
eigenvalue, and the same holds in expectation for SGD [arxiv:2510.25781 §9.3].
The review does not say how G relates to the NTK, and the learning-rate
condition is not stated [arxiv:2510.25781 §9.3].

### Linearised model around a fixed core, for forgetting

The network is split into a core, trained once and then frozen, and a user
part. The model is linearised around the core weights, so the user weights
enter linearly through a Jacobian-vector product [arxiv:2012.13431 eq. 3]
[arxiv:2012.13431 §5]. With a squared loss, the user weights have a closed
form, and forgetting a subset becomes one approximate Newton-type step that
the paper computes with a few SGD passes [arxiv:2012.13431 eq. 9] [arxiv:2012.13431
eq. 11]. The paper reports 2 to 3 epochs per request against 50 retraining
epochs, and readouts it describes as close to retraining [arxiv:2012.13431
fig. 2] [arxiv:2012.13431 fig. 3]. The card does not compute an NTK matrix or
its spectrum; its link to this concept is the linearisation.
Card: [Mixed-Privacy Forgetting in Deep Networks](/papers/2012.13431).

### Kernel regime of very wide networks

The essay argues that for very wide networks with a linear output layer, the
tangent kernel stays nearly constant in a ball around initialisation, so
training behaves like a kernel machine [arxiv:2105.14368 §3.10]. The bound
on the Hessian is stated only up to constants, and a non-linear output layer
breaks the argument [arxiv:2105.14368 §3.10]. The essay itself says this does
not settle whether deep learning is kernel learning [arxiv:2105.14368 §6.2].
Card: [Belkin's interpolation essay](/papers/2105.14368).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Rank many architectures in a cell-based search space, with no candidate training | Condition number of the NTK on one batch, combined with a linear-region count [arxiv:2102.11535 §3.2] | The paper's search runs in GPU days, and the NTK needs no labels [arxiv:2102.11535 tab. 2] [arxiv:2102.11535 §3.1.1] |
| Only one indicator is affordable | Pair the NTK with a second measure if you can | On NAS-Bench-201, the paper's ablations give lower CIFAR-100 accuracy with either indicator alone than with both [arxiv:2102.11535 tab. 4] |
| Explain slow convergence or spectral bias of a new architecture | Spectrum-shape reasoning, as in the review [arxiv:2510.25781 §9.2] | It is qualitative. The review gives no measurement recipe and runs no experiments of its own [arxiv:2510.25781 tab. 1] |
| Argue a convergence rate | Gradient-flow bound on the smallest eigenvalue [arxiv:2510.25781 §9.3] | It needs positive definiteness, and the learning-rate condition is not stated [arxiv:2510.25781 §9.3] |
| Remove a cohort of training data from a fine-tuned image classifier without full retraining | Linearised user layers with one approximate forgetting step [arxiv:2012.13431 §6] | Reported cost is 2 to 3 epochs per request against 50 retraining epochs [arxiv:2012.13431 fig. 2]. The paper's cost also includes one full pass over the retained data [arxiv:2012.13431 §8.2], and it has no official code |
| Reason about why interpolating models can generalise | Minimum-norm and kernel-view arguments in the essay [arxiv:2105.14368 §3.6] [arxiv:2105.14368 §3.9] | It is an expository argument; the essay reports no experiments of its own |

The NTK-based trainability score is validated only on NAS-Bench-201 in the
first paper, through a rank correlation of -0.42 for the condition number
[arxiv:2102.11535 §3.1.1]. Treat transfer to other search spaces as untested.

## Implementation sketch

The common core is the Gram matrix of per-input gradients and its spectrum.
The first paper's score is shown below. The review's variants do not define a
score, so they have no sketch. The forgetting and essay variants use a
linearised model rather than a score; their pseudocode is on their cards.

```pseudocode
# NTK condition number on one batch  [arxiv:2102.11535 §3.1.1] [eq. 3]
# Input: network, batch of N inputs
# Output: scalar; lower is better under the paper's rule

def ntk_condition(network, batch):
    # Gradient of the output with respect to the weights, one row per input
    # [arxiv:2102.11535 §3.1.1]; how several outputs are combined is not
    # stated
    grads = [sample_gradient(network, x) for x in batch]  # (N, P)
    # Gram matrix of the gradients is the NTK on this batch  [§3.1.1]
    ntk = grads @ grads.T  # (N, N)
    eig = sorted(eigvalsh(ntk))  # ascending
    # Largest over smallest eigenvalue  [eq. 3]
    return eig[-1] / eig[0]
```

The batch size and how the batch is drawn matter. The paper's appendix uses a
batch of 32 [arxiv:2102.11535 app. A], and the official code uses other sizes
(see the card's Paper vs code section). Scores from different batch sizes are
not comparable.

## Contradictions

No direct numerical conflict exists, because the review reports no NTK
measurement of its own. The papers use the object in different ways. The first
treats a small ratio of largest to smallest eigenvalue as good [arxiv:2102.11535
eq. 3]. The review says broader spectra can worsen conditioning, and it uses
"flatter" for weaker spectral bias [arxiv:2510.25781 §9.2]. The review does not
define "flatter" or "broader" numerically, so these terms cannot be mapped onto
the ratio in the first paper.

"Flat" also means something else in the essay. There it means that the kernel
changes little during training [arxiv:2105.14368 §3.10], which is a statement
about the kernel over time, not about its eigenvalue spectrum. Do not read the
essay's flatness and the review's flat spectra as the same claim.

## Pitfalls

- **The condition number depends on the batch.** The paper does not state how
  batches are drawn [arxiv:2102.11535 app. A].
- **Ranking is validated only on one benchmark.** The first paper's rank
  correlations are measured on NAS-Bench-201 [arxiv:2102.11535 §3.1.1].
- **Search cost units differ between tables.** The first paper gives seconds
  in one table and GPU days in another [arxiv:2102.11535 tab. 1]
  [arxiv:2102.11535 tab. 2].
- **Spectral claims are qualitative.** The review's NTK statements are
  illustrative and taken from other studies [arxiv:2510.25781 §9.2] [arxiv:2510.25781 tab. 1].
- **Linearisation is local.** The forgetting paper linearises only the final
  layers, around the core weights, and its bound covers only the squared user
  loss with L2 regularisation [arxiv:2012.13431 §5.1] [arxiv:2012.13431 §5].
  The essay's near-constant kernel holds only in a ball of unspecified radius
  [arxiv:2105.14368 §3.10].
- **The forgetting noise scale has three conventions.** The paper's equations
  and appendix write the noise differently, and the bound's constants do not
  agree between its statement and its proof [arxiv:2012.13431 app. C]. Check
  the derivation before quoting a privacy level.

## Papers

- [Mixed-Privacy Forgetting in Deep Networks](/papers/2012.13431): linearises a fine-tuned network around a frozen core and forgets a user cohort with one approximate Newton-type step.
- [Four-GPU-hours NAS paper](/papers/2102.11535): uses the NTK condition number, with a linear-region count, to prune a search space at initialisation.
- [Belkin's interpolation essay](/papers/2105.14368): an essay on interpolation that argues wide networks act like kernel machines near initialisation, with no experiments.
- [A Practitioner's Guide to Kolmogorov-Arnold Networks](/papers/2510.25781): a review that uses NTK spectrum shape and a gradient-flow bound to discuss spectral bias and convergence, with no experiments.