Neural tangent kernel

Overview

For a network, the NTK on a pair of inputs is the product of the network's output Jacobians with respect to its weights, evaluated at those inputs 2102.11535 §3.1.1. Stacked over a set of inputs, it is a Gram matrix, and it depends only on the network and the inputs, not on labels 2102.11535 §3.1.1. The first paper writes training dynamics under gradient descent in terms of the eigenvalues of this matrix [arxiv:2102.11535 eq. 1, eq. 2]. The second paper uses the same object, with its spectrum shape, to reason about spectral bias and convergence 2510.25781 §9.2.

Two other papers use the same object without a spectrum. A third paper linearises a fine-tuned network around a fixed core, so the trainable part enters through its Jacobian 2012.13431 §5. A fourth essay argues that very wide networks with a linear output layer behave like kernel machines near initialisation 2105.14368 §3.10.

The invariant the variants share is that the matrix is computed at a fixed point of the network, usually at initialisation or around a fixed core, and the model is read through that point. The variants differ in what they do with it.

Variants

Condition number as a training-free trainability score

The NTK is computed on one mini-batch. Its condition number, the ratio of largest to smallest eigenvalue, is used as a trainability score where lower is better 2102.11535 eq. 3. The paper combines it with a count of linear regions and uses both ranks to prune a super-network without training any candidate 2102.11535 §3.2 2102.11535 alg. 1. The paper reports a CIFAR-10 search of 0.05 GPU days on one GTX 1080Ti and a test error of 2.63% 2102.11535 tab. 2. Card: Four-GPU-hours NAS paper.

Spectrum shape as a diagnostic for spectral bias

The review does not compute an NTK for a new model. It reports that flatter NTK spectra, seen for KANs and for learnable-basis PINNs, indicate weaker spectral bias, meaning less preference for low frequencies [arxiv:2510.25781 §9.2]. It adds that broader spectra can raise curvature and worsen conditioning 2510.25781 §9.2. It also reports, from the literature, a Chebyshev-based PIKAN with better NTK conditioning and faster convergence on four PDEs 2510.25781 §9.3 2510.25781 tab. 1. Card: A Practitioner's Guide to Kolmogorov-Arnold Networks.

Smallest eigenvalue as a convergence rate

The review gives a gradient-flow bound. If a matrix G = D^T D is positive definite, the loss decays exponentially at a rate set by G's smallest eigenvalue, and the same holds in expectation for SGD 2510.25781 §9.3. The review does not say how G relates to the NTK, and the learning-rate condition is not stated 2510.25781 §9.3.

Linearised model around a fixed core, for forgetting

The network is split into a core, trained once and then frozen, and a user part. The model is linearised around the core weights, so the user weights enter linearly through a Jacobian-vector product 2012.13431 eq. 3 2012.13431 §5. With a squared loss, the user weights have a closed form, and forgetting a subset becomes one approximate Newton-type step that the paper computes with a few SGD passes 2012.13431 eq. 9 [arxiv:2012.13431 eq. 11]. The paper reports 2 to 3 epochs per request against 50 retraining epochs, and readouts it describes as close to retraining [arxiv:2012.13431 fig. 2] 2012.13431 fig. 3. The card does not compute an NTK matrix or its spectrum; its link to this concept is the linearisation. Card: Mixed-Privacy Forgetting in Deep Networks.

Kernel regime of very wide networks

The essay argues that for very wide networks with a linear output layer, the tangent kernel stays nearly constant in a ball around initialisation, so training behaves like a kernel machine 2105.14368 §3.10. The bound on the Hessian is stated only up to constants, and a non-linear output layer breaks the argument 2105.14368 §3.10. The essay itself says this does not settle whether deep learning is kernel learning 2105.14368 §6.2. Card: Belkin's interpolation essay.

When to use which

The NTK-based trainability score is validated only on NAS-Bench-201 in the first paper, through a rank correlation of -0.42 for the condition number 2102.11535 §3.1.1. Treat transfer to other search spaces as untested.

Implementation sketch

The common core is the Gram matrix of per-input gradients and its spectrum. The first paper's score is shown below. The review's variants do not define a score, so they have no sketch. The forgetting and essay variants use a linearised model rather than a score; their pseudocode is on their cards.

# NTK condition number on one batch  [arxiv:2102.11535 §3.1.1] [eq. 3]
# Input: network, batch of N inputs
# Output: scalar; lower is better under the paper's rule

def ntk_condition(network, batch):
    # Gradient of the output with respect to the weights, one row per input
    # [arxiv:2102.11535 §3.1.1]; how several outputs are combined is not
    # stated
    grads = [sample_gradient(network, x) for x in batch]  # (N, P)
    # Gram matrix of the gradients is the NTK on this batch  [§3.1.1]
    ntk = grads @ grads.T  # (N, N)
    eig = sorted(eigvalsh(ntk))  # ascending
    # Largest over smallest eigenvalue  [eq. 3]
    return eig[-1] / eig[0]

The batch size and how the batch is drawn matter. The paper's appendix uses a batch of 32 2102.11535 app. A, and the official code uses other sizes (see the card's Paper vs code section). Scores from different batch sizes are not comparable.

Contradictions

No direct numerical conflict exists, because the review reports no NTK measurement of its own. The papers use the object in different ways. The first treats a small ratio of largest to smallest eigenvalue as good [arxiv:2102.11535 eq. 3]. The review says broader spectra can worsen conditioning, and it uses "flatter" for weaker spectral bias 2510.25781 §9.2. The review does not define "flatter" or "broader" numerically, so these terms cannot be mapped onto the ratio in the first paper.

"Flat" also means something else in the essay. There it means that the kernel changes little during training 2105.14368 §3.10, which is a statement about the kernel over time, not about its eigenvalue spectrum. Do not read the essay's flatness and the review's flat spectra as the same claim.

Pitfalls

• The condition number depends on the batch. The paper does not state how batches are drawn 2102.11535 app. A.
• Ranking is validated only on one benchmark. The first paper's rank correlations are measured on NAS-Bench-201 2102.11535 §3.1.1.
• Search cost units differ between tables. The first paper gives seconds in one table and GPU days in another 2102.11535 tab. 1 2102.11535 tab. 2.
• Spectral claims are qualitative. The review's NTK statements are illustrative and taken from other studies 2510.25781 §9.2 2510.25781 tab. 1.
• Linearisation is local. The forgetting paper linearises only the final layers, around the core weights, and its bound covers only the squared user loss with L2 regularisation 2012.13431 §5.1 2012.13431 §5. The essay's near-constant kernel holds only in a ball of unspecified radius 2105.14368 §3.10.
• The forgetting noise scale has three conventions. The paper's equations and appendix write the noise differently, and the bound's constants do not agree between its statement and its proof 2012.13431 app. C. Check the derivation before quoting a privacy level.

Papers

• Mixed-Privacy Forgetting in Deep Networks: linearises a fine-tuned network around a frozen core and forgets a user cohort with one approximate Newton-type step.
• Four-GPU-hours NAS paper: uses the NTK condition number, with a linear-region count, to prune a search space at initialisation.
• Belkin's interpolation essay: an essay on interpolation that argues wide networks act like kernel machines near initialisation, with no experiments.
• A Practitioner's Guide to Kolmogorov-Arnold Networks: a review that uses NTK spectrum shape and a gradient-flow bound to discuss spectral bias and convergence, with no experiments.