---
kind: concept
slug: learnable-activation-functions
title: Learnable activation functions
summary: >-
  Learnable activation functions put a trainable univariate function on every edge instead of a fixed nonlinearity on every node. One card so far, KAN, which uses spline edges and reports small models that match larger MLPs on small function-fitting and PDE tasks. Results are author-reported, on small problems, and trained with LBFGS, so check the constraints before adopting it.
tags: [architecture, interpretability, symbolic-regression]
papers: ["2404.19756"]
related: [multi-layer-perceptron, neural-symbolic-integration]
status: draft
updated_at: 2026-10-19
---

## Overview

A standard multi-layer perceptron applies a fixed nonlinearity to each
neuron after a learned linear map [arxiv:2404.19756 §1]. The learnable variant
moves the trainable part onto the edges: each connection between two neurons
carries its own univariate function, and each neuron only sums what arrives
[arxiv:2404.19756 §2.2]. The invariant every variant shares is this split
between learnable edge functions and parameter-free summing nodes.

The motivation is twofold. The authors want better accuracy per parameter
than MLPs on smooth functions, and they want a network whose parts can be
read and simplified [arxiv:2404.19756 §1]. Both claims rest on the same
construction, described below.

## Variants

Only one variant has a card in this concept so far.

### Spline edges (Kolmogorov-Arnold networks)

Each edge applies a base term plus a B-spline term, `w_b * silu(x) + w_s *
spline(x)`, with cubic splines on a grid that can be refined during training
[arxiv:2404.19756 §2.2]. Refining the grid is done by a least-squares fit from
the coarse spline, without restarting training [arxiv:2404.19756 §2.4]. Pruning
removes weak nodes and edges, and selected edges can be replaced with named
functions such as sine or exponential [arxiv:2404.19756 §2.5.1]. Card:
[KAN: Kolmogorov-Arnold Networks](/papers/2404.19756).

Reported results: on fifteen special functions, the best KAN is lower in RMSE
than the best MLP in every row of the table [arxiv:2404.19756 tab. 1]. On the
Feynman equations, KANs and MLPs are comparable on average, and the MLP has the
lowest loss in several rows [arxiv:2404.19756 tab. 2]. On a Poisson problem, a
small KAN reaches a lower error with far fewer parameters than a larger MLP
[arxiv:2404.19756 §3.4]. On knot signatures, a KAN beats an MLP baseline quoted
from another group [arxiv:2404.19756 tab. 3].

## When to use which

With one card, the only comparison is KAN against MLP. The table lists what
the paper supports, not general advice.

| Situation | Use | Why |
|---|---|---|
| Small, smooth target function with few inputs, where a readable formula is wanted | A learnable-edge network with pruning and symbolic fitting | The paper reports compact formulas on small science examples [arxiv:2404.19756 §2.5.2] [arxiv:2404.19756 §4.4] |
| Small PDE or regression problem where parameter count matters | A learnable-edge network, checked against an MLP of similar cost | The paper reports lower error with fewer parameters on a Poisson problem [arxiv:2404.19756 §3.4] |
| Training time is the bottleneck | An MLP | The authors report KAN training about ten times slower in their PDE setup [arxiv:2404.19756 §3.4] |
| Large-scale model or standard optimiser pipeline (Adam, boosting) | An MLP until evidence appears | The paper trains with LBFGS and leaves other optimisers for later [arxiv:2404.19756 §3.1] [arxiv:2404.19756 §6] |
| High-dimensional inputs with no known shape | Choose the shape by pruning, then compare seeds | Pruned shapes depend on seed, regularisation and grid size [arxiv:2404.19756 App. C] |

## Implementation sketch

The forward pass has the same shape as an MLP, with the matrix product replaced
by a sum over per-edge functions [arxiv:2404.19756 §2.2]. The edge function is
the part that changes between variants.

```pseudocode
# Forward pass: one learnable function per input-output pair  [arxiv:2404.19756 §2.2]
# Shapes [n_0, n_1, ..., n_L]; x has shape (batch, n_0)

def forward(x, layers):
    for layer in layers:
        # each edge applies its own univariate function  [arxiv:2404.19756 §2.2]
        edge_outputs = layer.edge_functions(x)   # (batch, n_in, n_out)
        # nodes only add the incoming edge outputs  [arxiv:2404.19756 §2.2]
        x = edge_outputs.sum(axis=1)             # (batch, n_out)
    return x
```

```pseudocode
# Spline edge: base term plus trainable spline  [arxiv:2404.19756 §2.2]
# Cubic (k = 3) B-splines on a grid of G intervals

def edge(x, w_b, w_s, coefs):
    # residual-style base term, with silu as the default  [arxiv:2404.19756 §2.2]
    base = w_b * silu(x)
    # trainable combination of B-spline basis functions  [arxiv:2404.19756 §2.2]
    spline = sum(c * B(x) for c, B in zip(coefs, basis))
    return base + w_s * spline
```

```pseudocode
# Grid extension: refit a finer spline to the coarse one  [arxiv:2404.19756 §2.4]

def extend_grid(coarse_coefs, coarse_basis, fine_basis, samples):
    # values of the coarse spline on sampled inputs  [arxiv:2404.19756 §2.4]
    targets = coarse_basis(samples) @ coarse_coefs
    # least-squares fit onto the finer grid  [arxiv:2404.19756 §2.4]
    return least_squares(fine_basis(samples), targets)
```

Where variants differ: the basis (splines here; other papers may use other
families), the initialisation, and the sparsity penalty. The card's [Paper vs
code](/papers/2404.19756) section lists where the released code departs from
the paper's stated choices.

## Contradictions

None known. Only one card covers this concept. The authors' own numbers are
not checked against an independent reproduction here. The paper also does not
state the optimiser or grid for its PDE experiment [arxiv:2404.19756 §3.4].

## Papers

- [KAN: Kolmogorov-Arnold Networks](/papers/2404.19756): replaces MLP weights and fixed node activations with learnable spline edges, with pruning and symbolic fitting for interpretability.