KAN: Kolmogorov-Arnold Networks

In plain words

A standard neural network stacks layers of weights followed by a fixed squashing function on each neuron. KANs move the learnable part. Every connection between two neurons carries its own smooth curve, and the neurons only add up what arrives. Each curve is built from small pieces joined smoothly, so it can bend to match the data.

The payoff claimed is that a small KAN can fit a function that a much larger standard network needs many more parameters for. Because each curve is simple to inspect, the network can also be trimmed down and its curves matched to familiar functions such as sine or exponential. The authors test this mostly on small mathematics and physics problems, so it is not yet shown for large models.

TL;DR

KANs use learnable spline curves on edges instead of linear weights and fixed node activations §2.2. The authors report that much smaller KANs match or beat larger MLPs on function fitting and PDE solving, with faster scaling-law exponents §3.1, §3.4. Their pruning and symbolic-fitting tools recover compact formulas in small science examples §4. All results are author-reported and on small problems §6.

Problem

MLPs are the default approximator in deep learning, but they consume most non-embedding parameters in transformers and are hard to read without extra tools §1. Splines are accurate in low dimensions and easy to refine, but suffer the curse of dimensionality because they do not exploit compositional structure. MLPs handle high dimensions better but approximate univariate functions less accurately §1.

The Kolmogorov-Arnold representation writes a multivariate continuous function as a sum of univariate functions §2.1. Taken literally, its two-layer, width-(2n+1) form can need non-smooth inner functions that are hard to learn, so the authors generalise it to arbitrary width and depth §2.1, §2.2. They assume the target has a smooth representation of this kind §2.3.

Method

Overview. A KAN has a shape [n_0, n_1, ..., n_L]. Between layers l and l+1 there is one learnable univariate function per pair of neurons, with no linear weight matrix §2.2. Each neuron outputs the sum of the post-activations of its incoming edges §2.2. The stacked form is the composition of KAN layers §2.2.

1. Edge function. Each edge applies phi(x) = w_b * b(x) + w_s * spline(x). Here b(x) = silu(x) = x / (1 + e^(-x)) in most cases, and spline(x) is a linear combination of B-spline basis functions B_i on a grid with trainable coefficients c_i §2.2.
2. Node. Node j in layer l+1 takes the sum over its incoming edge outputs, with no further nonlinearity §2.2.
3. Initialisation. Each edge starts with w_s = 1 and a near-zero spline, obtained from small Gaussian coefficients (typically standard deviation 0.1, given in a footnote). w_b uses Xavier initialisation §2.2.
4. Grid update. Each spline grid is re-set from the input activations seen during training, because activations can leave a fixed interval §2.2.
5. Grid extension. To make a trained KAN more accurate, each spline's coarse grid is replaced by a finer one. The new coefficients come from a least-squares fit to the coarse spline over sampled inputs. This is applied to every spline independently and does not need retraining from scratch §2.4.
6. Sparsification. The loss is the prediction loss plus a regulariser lambda * (mu_1 * sum of |Phi_l|_1 + mu_2 * sum of S(Phi_l)), where |phi|_1 is the mean absolute edge output over the inputs, and S is an entropy term over the normalised edge magnitudes of each layer. mu_1 = mu_2 = 1 is usual §2.5.1.
7. Pruning. A node is kept only if its largest incoming and its largest outgoing edge magnitude both exceed a threshold, theta = 10^-2 by default §2.5.1.
8. Symbolification. An edge can be fixed to a known function f. The paper fits y ≈ c f(a x + b) + d from sampled pre- and post-activations, using a grid search over a, b and linear regression for c, d §2.5.1.
9. Human-in-the-loop workflow. The authors demonstrate training a [2,5,1] KAN with sparsity, pruning to [2,1,1], fixing the edges to symbolic functions, then training only the affine parameters §2.5.2.

# KAN forward pass: one learnable edge function per input-output pair  [§2.2]
# Input:  x of shape (B, n_0); layers with shapes [n_0, n_1, ..., n_L]
# Output: y of shape (B, n_L)

def kan_forward(x, layers):
    for layer in layers:
        # every edge applies its own univariate function  [§2.2]
        edge_outputs = edge_function(layer, x)    # (B, n_in, n_out)
        # nodes only sum the incoming edge outputs  [§2.2]
        x = edge_outputs.sum(axis=1)              # (B, n_out)
    return x

# Learnable edge function  [§2.2]
# phi(x) = w_b * b(x) + w_s * spline(x), with b(x) = silu(x)  [§2.2]

def edge_function(edge, x):
    # residual-style basis term, similar to a skip connection  [§2.2]
    base = edge.w_b * silu(x)
    # cubic B-spline on G intervals with trainable coefficients c_i  [§2.2]
    spline = sum(c_i * B_i(x) for c_i, B_i in edge.spline_terms)
    return base + edge.w_s * spline

# Grid extension: refit a finer spline to a coarse one  [§2.4]
# Coarse spline has G1 intervals with coefficients c; target grid has G2

def extend_grid(coarse_coefs, coarse_basis, fine_basis, samples):
    # values of the coarse spline on the sampled inputs  [§2.4]
    targets = coarse_basis(samples) @ coarse_coefs
    # least-squares fit of the fine spline to those values  [§2.4]
    fine_coefs = least_squares(fine_basis(samples), targets)
    return fine_coefs

# Sparsity-regularised training objective  [§2.5.1]
# mu_1 = mu_2 = 1 usually; lam sets the overall strength  [§2.5.1]

def total_loss(predictions, targets, layers, lam, mu_1, mu_2):
    loss = prediction_loss(predictions, targets)
    for layer in layers:
        # L1 of a layer: sum of mean |phi| over its edges  [§2.5.1]
        l1 = layer.sum_of_mean_abs_edge_outputs()
        # entropy of the normalised edge magnitudes  [§2.5.1]
        entropy = layer.normalised_entropy()
        loss = loss + lam * (mu_1 * l1 + mu_2 * entropy)
    return loss

# Node pruning after sparsification  [§2.5.1]
# theta = 10^-2 by default  [§2.5.1]

def prune_nodes(layers, theta):
    for each node i in layer l:
        # incoming score: largest mean |phi| over its incoming edges  [§2.5.1]
        incoming = max(mean_abs(edge(l - 1, i, k)) for k)
        # outgoing score: largest mean |phi| over its outgoing edges  [§2.5.1]
        outgoing = max(mean_abs(edge(l + 1, j, i)) for j)
        # keep the node only if both scores exceed theta  [§2.5.1]
        keep = incoming > theta and outgoing > theta

Parameter count. For depth L, equal width N, spline order k and G intervals, a KAN has O(N^2 L (G + k)) parameters, against O(N^2 L) for an MLP of the same depth and width. The authors argue that KANs usually need a much smaller N §2.2.

Theory. The approximation theorem gives a bound in the grid size G whose residual rate does not depend on the input dimension, provided the representation is smooth §2.3. This gives an expected scaling exponent of k + 1, which is 4 for cubic splines §2.3. Other theories predict (k+1)/d, (k+1)/2 or m/2 for comparable settings §2.3.

Hyperparameters

Results

Author-reported. Each setting is described in the same order as the paper. "Best" is the table entry the paper reports, not a mean over seeds.

Toy scaling (§3.1, fig. 3.1). Five smooth functions with known KAN shapes, including J_0(20x) and exp(sin(pi x) + y^2). KAN test RMSE follows the predicted exponent 4 closely, while MLPs converge more slowly and plateau. The high-dimensional exp((1/100) sum sin^2(pi x_i / 2)) example is shown with a [100,1,1] KAN as the true shape.

Special functions (§3.2, tab. 1). Fifteen special functions from scipy.special as 2-input functions. The "minimal" column is the smallest pruned KAN with test RMSE below 10^-2. The "best" column is the lowest-loss pruned KAN. MLP is the best MLP reported.

The paper reports the KAN as lower than the MLP in every row §3.2. Whether the table row is the best over seeds and depths, or a fixed choice, is not stated.

Feynman equations (§3.3, tab. 2). Two-or-more-variable equations from the Feynman dataset §3.3. The paper's example equation draws its inputs from (-1, 1) §3.3. Human-constructed KAN shapes are the paper's hand-written representations. Pruned columns come from the auto-pruning pipeline, and the MLP column is the best MLP. Bold marks the lowest loss of the four per row, as in the paper. Losses are test RMSE.

The paper's summary is that KANs and MLPs behave comparably on average here, and that auto-discovered KANs are often smaller than the hand-built ones §3.3.

PDE (§3.4, fig. 3.3). Poisson equation on [-1,1]^2 with a known solution, solved with a physics-informed loss. The paper reports that a 2-layer width-10 KAN reaches about 10^-7 MSE with about 10^2 parameters, against 10^-5 MSE and 10^4 parameters for a 4-layer width-100 MLP. The authors also say KAN training is about 10 times slower than MLP training in their implementation §3.4.

Continual learning (§3.5, fig. 3.4). A one-dimensional regression with five peaks, presented sequentially. The paper reports that KANs update only the regions with current data, while MLPs forget earlier peaks. This is a single toy example, which the authors call preliminary §3.5.

Knot theory (§4.3, tab. 3). Predicting the signature of knots from 17 invariants.

The paper also reports a symbolic formula that uses only two of the three mu and lambda inputs and reaches 77.8% accuracy §4.3, tab. 4.

Anderson localisation (§4.4, tab. 5). For two one-dimensional models the paper reports KAN-discovered mobility-edge formulas. For GAAM the auto-discovered formula reaches 99.0% against 99.2% for the analytic theory; for MAAM it reaches 97.1% against 98.6%. Manual steps raise the MAAM figure to 97.7% for one variant §4.4, tab. 5.

Paper vs code

Code: kan/MultKAN.py and kan/KANLayer.py at commit ecde4ec (2025-01-19), the default branch HEAD when the repository was cloned. The paper is version v5 (2025-02-09). Discrepancies:

• Noise on spline coefficients. Paper: Gaussian, sigma = 0.1 typical §2.2 footnote. Code: uniform (rand - 1/2) * noise_scale / G, with noise_scale defaulting to 0.3 in KAN and 0.5 in KANLayer KANLayer.py MultKAN.py.
• Initial scales. Paper: w_s = 1 and Xavier-initialised w_b §2.2. Code: scale_sp = 1/sqrt(n_in) and scale_base uniform in ±1/sqrt(n_in), which is not Xavier KANLayer.py.
• L1 penalty. Paper: mean |phi| over the N_p inputs §2.5.1. Code: the default reg_metric is edge_forward_spline_n, which uses the standard deviation of spline outputs divided by the standard deviation of inputs plus 0.1 MultKAN.py MultKAN.py.
• Entropy term. Paper: S over the normalised |phi|_1 of the whole layer, with mu_1 = mu_2 = 1 §2.5.1. Code: row and column entropies with log2, normalised by row or column sum plus 1, and a 1e-4 floor; default weights are lamb_l1 = 1 and lamb_entropy = 2 MultKAN.py MultKAN.py.
• Grid update. Paper: grids are re-set from the input activations §2.2. Code: new grid points mix sample quantiles with a uniform grid, with grid_eps = 0.02 by default KANLayer.py. In fit, grid updates stop at step 50 by default MultKAN.py MultKAN.py. The paper does not mention either detail.
• Edge pruning. Paper gives node pruning with theta = 10^-2 §2.5.1. Code also has edge pruning with default threshold 3e-2, which the paper does not mention MultKAN.py.

The repository has a Feynman dataset loader feynman.py but I found no script under kan/ that reproduces the experiments of §3.2, §3.3 or §3.4. Their settings are therefore checked only against the paper, not the code.

Pitfalls

• Shape must be chosen. The accuracy and scaling results assume the true KAN shape is known §3.1. The special-function table uses the pruning pipeline to find shapes, but the paper notes that these shapes depend on random seed, lambda, G and k [App. C] §3.2.
• Pruning is not reliable on its own. Some runs end with larger networks than needed due to unlucky seeds. An entropy penalty is needed to avoid duplicated active edges [App. C].
• Grid refinement can break training. Loss drops in steps at each grid extension. At G = 1000 on the toy problem optimisation stops working, which the authors attribute to a poor loss landscape §2.4. On PDE and regression tasks with uniform grids, training loss jumps up at large grid sizes, cause unknown [App. E].
• Scaling exponents are not as predicted. For [2,1,1] the paper measures roughly G^-3, against G^-4 predicted. The median squared error, rather than the mean, comes closer to G^-4 because a few boundary samples dominate the mean §2.4. Reporting the mean would understate the exponent.
• Cost per parameter. Each edge carries a spline, so a KAN has O(N^2 L G) parameters against O(N^2 L) for an MLP §2.2. The paper reports about 10 times slower training than MLPs in its PDE setup §3.4. The authors put this down to activation functions that cannot share one batched computation §6.
• Optimiser choice. Results use vanilla LBFGS with no switch to Adam and no boosting §3.1. The paper leaves comparisons with these advanced setups for future work, so results may not carry over to Adam-trained pipelines.
• Baselines are narrow. MLP baselines use fixed widths (5 or 100) and depths 2 to 6 §3.2 §3.3. The PDE MLP architecture is not stated beyond the comparison in §3.4.
• Symbolic fitting is manual. Snapping an edge to f needs a human to name f. The fit is c f(a x + b) + d, found by grid search over a and b and linear regression §2.5.1. The suggest_symbolic helper is only a suggestion. The authors' interpretability examples depend on this human step §2.5.2 §4.4.
• Feynman result. KANs do not beat MLPs on average in the Feynman table §3.3. The paper's interpretability claims rest on small examples §6.
• Knot baseline. The 78% MLP baseline is quoted from another group's work and not reproduced here §4.3.
• PDE loss. The loss uses alpha = 0.01 for the interior term §3.4. The paper does not state how the second derivatives are computed, nor the optimiser for this experiment.
• Continual learning is shown on one toy problem, and the authors say it is unclear whether it generalises to high dimensions §3.5.
• Affine parameters are redundant in principle, because w_b and w_s can be absorbed into the spline and base function. The paper keeps them for control over magnitude §2.2.

Lineage

• Builds on: the Kolmogorov-Arnold representation theorem §2.1. Earlier neural-network work on the same theorem is surveyed in §5. The authors link MLPs with ReLU-k activations to splines §1. No arXiv ids are given in the paper for these prior works, so none are linked here.
• Scaling-law theories compared: Sharma and Kaplan; Michaud et al. (2023); Poggio et al. (2020) §2.3.
• Compared against: MLPs of matched shape §3.1 §3.2 §3.3; a published MLP for knot signatures §4.3; physics-informed neural networks for the Poisson problem §3.4.
• Related methods: learnable activations §5; symbolic regression methods such as AI Feynman, PySR, Eureqa, EQL and OccamNet §5.
• Superseded by: not stated in this card.