---
kind: concept
slug: multi-layer-perceptron
title: Multi-layer perceptron (MLP)
summary: >-
  A feed-forward network of stacked layers, each a weighted sum followed by a nonlinearity. Twenty-one cards use it or test it against an alternative, from anomaly detection and choice models to click prediction, forecasting, pixel classification, terrain sensing, exoskeleton torque and microcontrollers. Most trained settings are unstated or library defaults.
tags: [neural-networks, supervised-learning, function-approximation, intrusion-detection, tabular-data, time-series, interpretability, textbook, anomaly-detection, discrete-choice, frequency-domain, remote-sensing, autonomous-driving, point-clouds, terrain-classification, biomechanics, multi-layer-perceptron]
papers: ["1704.02286", "1701.02490", "1610.04161", "2506.08274", "2012.06678", "2404.19756", "2405.14616", "2304.08424", "2106.11342", "2206.06602", "2207.12877", "2208.05233", "2303.06053", "2311.06184", "2609.40212", "2310.06753", "1611.00144", "2101.06742", "2501.03256", "2411.04700", "2412.07823"]
related: [intrusion-detection, depth-separation, universal-approximation, model-based-reinforcement-learning, convolutional-neural-network, recurrent-neural-network, support-vector-machine, random-forest, feature-scaling, ensemble-learning, self-attention, semi-supervised-learning, masked-language-modeling, learnable-activation-functions, time-series-forecasting, multi-scale-feature-fusion, residual-learning, deep-learning, isolation-forest, anomaly-detection, discrete-choice-modeling, graph-neural-network, frequency-domain-learning, image-thresholding, bipartite-matching, object-queries, bezier-curve-parameterization, continuous-convolution, semantic-segmentation, dropout, k-means-clustering]
status: draft
updated_at: 2026-10-25
---

## Overview

A multi-layer perceptron (MLP) is a feed-forward network. Each unit takes a
weighted sum of the previous layer's outputs, adds a bias and applies an
activation. The forward pass is written out layer by layer for a three-layer
sigmoid network [arxiv:1704.02286 eq. 1-4]. Training in that paper uses a
backward pass that computes error terms layer by layer and steps the weights
down the gradient [arxiv:1704.02286 §III-B] [arxiv:1704.02286 eq. 5-6].

Twenty-one cards use an MLP, or compare it with an alternative. Each bullet gives
the role of the MLP in that card:

- a supervised classifier trained by backpropagation on labelled traffic
  [arxiv:1704.02286 §III];
- a regressor for value differences inside a bidding dynamic program, with two
  tanh hidden layers [arxiv:1701.02490 §4.1] [arxiv:1701.02490 §6.2];
- a hand-built construction in a theory paper that shows what depth-limited
  ReLU and step networks can represent. It trains nothing
  [arxiv:1610.04161 §3] [arxiv:1610.04161 §4];
- an off-the-shelf MLP classifier and regressor in a benchmark of feature
  scaling on 16 tabular datasets [arxiv:2506.08274 §4.1] [arxiv:2506.08274 §4.3];
- an MLP prediction head on top of Transformer layers over categorical
  columns, with a plain MLP as the baseline [arxiv:2012.06678 §2]
  [arxiv:2012.06678 tab. 1];
- a comparison target for KANs, which keep the layered shape but replace the
  linear weights with learnable splines on edges [arxiv:2404.19756 §2.2] and
  are measured against MLPs of matched shape [arxiv:2404.19756 §3.1];
- an MLP-only time-series forecaster that mixes several downsampled scales
  [arxiv:2405.14616 §3];
- an MLP encoder-decoder for long-horizon forecasting, built from residual
  MLP blocks with a linear shortcut [arxiv:2304.08424 §4];
- a textbook whose chapter on multilayer perceptrons, with runnable code,
  teaches the model from the basics [arxiv:2106.11342 §5]. It reports no
  results of its own.
- a random, untrained MLP that reshapes the data before an isolation forest
  grows its trees on the reshaped copies [arxiv:2206.06602 §5.1];
- the utility function of a random-utility choice model, a feed-forward net
  averaged over sampled attributes [arxiv:2207.12877 §2.2] [arxiv:2207.12877 §2.3];
- a graph-free forecaster whose residual MLP blocks take learned identity
  embeddings [arxiv:2208.05233 §3];
- the time-mixing and feature-mixing MLP blocks of an all-MLP forecaster
  [arxiv:2303.06053 §4.1];
- two complex-valued MLPs run on a discrete Fourier transform of the input
  [arxiv:2311.06184 §3.1];
- a one-hidden-layer tanh classifier on RGB patches, compared with vegetation
  indices, a linear network and a CNN [arxiv:2609.40212 §2.2.2];
- two MLP heads in a query-based lane detector that score every pair of lane
  queries, and every pair of lane and traffic-element queries, as link
  predictors [arxiv:2310.06753 §3.3] [arxiv:2310.06753 §3.4];
- a click-prediction network over learned field embeddings, where a product
  layer of pairwise inner or outer products feeds ReLU layers and a sigmoid
  output [arxiv:1611.00144 §3.1] [arxiv:1611.00144 eq. 1].
- a learnable convolution whose kernel is an MLP of the relative 3D offset
  between points, so it runs on unevenly spaced lidar and depth points
  [arxiv:2101.06742 §3.1] [arxiv:2101.06742 §3.3];
- a hand-written dense network in MicroPython on a Raspberry Pi Pico, with
  weights exported from a Keras model trained off the device
  [arxiv:2501.03256 §III] [arxiv:2501.03256 tab. 2];
- a dropout-regularised classifier over statistics of force-torque and IMU
  windows, naming the terrain class of a planetary rover
  [arxiv:2411.04700 §3.2] [arxiv:2411.04700 tab. 6];
- a batch-normalised three-layer regressor on 14 features, estimating hip
  joint moment after a subset of tasks is chosen by clustering
  [arxiv:2412.07823 §II.C] [arxiv:2412.07823 §III.B].

The variants below differ in activation, output, how their weights are
obtained, and whether they are an MLP at all. The sketches show the sigmoid and
value-difference networks, and the scaling step that comes before the library
MLP. The residual MLP block of the forecasting variant is covered under its
own heading. The random, choice-model, frequency, pairwise-scorer, product-layer
and continuous-kernel variants have headings below too.

## Variants

### Sigmoid MLP classifier (6-3-1)

One hidden layer of three units, one output unit, and a bias on every unit.
Every hidden and output unit uses a sigmoid [arxiv:1704.02286 §III]
[arxiv:1704.02286 fig. 1]. The loss is the mean squared error, written per
example with a half factor; whether a batch sums or averages it is not stated
[arxiv:1704.02286 eq. 5]. The optimiser is stochastic gradient descent. The
learning rate, batch size, epochs, initialisation and regularisation are not
stated [arxiv:1704.02286 §III-B].

The data is 2313 training, 496 validation and 496 test samples; the split
procedure is not stated [arxiv:1704.02286 §V] [arxiv:1704.02286 tab. 2]. The
six input features are not named [arxiv:1704.02286 fig. 1]. The author-reported
accuracy is 99.4% on the test split (493 of 496) and 99.4% over all 3305 samples
[arxiv:1704.02286 §V] [arxiv:1704.02286 fig. 3]. Card:
[Threat analysis of IoT networks Using Artificial Neural Network Intrusion Detection System](/papers/1704.02286).

### Tanh MLP fitted to value differences

Two inputs, the auctions left t and the budget left b. Two tanh hidden layers
of 30 and 15 units, and a linear output for the value difference
D(t, b) [arxiv:1701.02490 §4.1] [arxiv:1701.02490 §6.2]. The network predicts
D(t, b) = V(t, b+1) − V(t, b) rather than V, because V is much larger than the
average click-through rate while D is on the scale of that rate
[arxiv:1701.02490 §4.1]. It is fitted on the grid {0..T0} × {0..B0} with
T0 = 10,000 [arxiv:1701.02490 §6.2]. The optimiser, learning rate, batch,
initialisation and training length are not stated [arxiv:1701.02490 §6.2].

Reported value-table error is 0.998 × 10^-6 (RMSE) on iPinYou, measured only
for states inside the trained box [arxiv:1701.02490 tab. 4]
[arxiv:1701.02490 §6.2]. The headline click gains come from the small-scale
setting (T = 1,000), where the paper uses the exact table: +16.67% over the Lin
baseline at a budget ratio of 1/8 on iPinYou [arxiv:1701.02490 tab. 2]
[arxiv:1701.02490 §6.1]. For large horizons the network results appear only as
a figure [arxiv:1701.02490 §6.2] [arxiv:1701.02490 fig. 9]. Card:
[Real-Time Bidding by Reinforcement Learning in Display Advertising](/papers/1701.02490).

### Library MLP on scaled tabular features

An off-the-shelf MLP, used as a classifier or a regressor after a feature
scaler has been fitted on the training split. The benchmark covers 16 UCI
tabular datasets, 9 for classification and 7 for regression, with 12 scalers
plus no scaling [arxiv:2506.08274 §4.1]. The paper says every model uses its
default settings [arxiv:2506.08274 §4.3]. The card does not record the layer
sizes or activation, so they are the library defaults. The authors' code sets
only an iteration cap and a seed for the MLP [/papers/2506.08274].

The scaler matters for this model. On Dry Bean, MLP accuracy was 0.2980 with
no scaling and 0.9334 with the best of the twelve scalers [arxiv:2506.08274
app. A]. The scaled-versus-unscaled Wilcoxon test is significant for the MLP
(reported p = 0.0000), and so is the Friedman test across scalers (p = 0.0006)
[arxiv:2506.08274 app. A]. For regression, the R² test for the MLP is
significant (Wilcoxon p = 0.0083) [arxiv:2506.08274 app. A]. The authors group
the MLP with LR, SVMs and TabNet as strongly affected by the scaler
[arxiv:2506.08274 §5.1].

The results come from one 70/30 split per dataset, one run per cell, with no
hyperparameter tuning and no cross-validation [arxiv:2506.08274 §6]. The
"best of twelve" figure picks the scaler after seeing the results, so it
overstates what one fixed choice would give. Card:
[The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](/papers/2506.08274).

### Deep ReLU and step networks (existence result)

No training is involved. The constructions are hand-built: binary step units
read the binary digits of the input, one per layer, and ReLU layers combine
them [arxiv:1610.04161 §3.1]. For a smooth univariate function, the paper
reaches error eps with depth O(log 1/eps), binary units O(log 1/eps) and ReLUs
O((log 1/eps)^2) [arxiv:1610.04161 thm. 4]. A shallow network, with depth below
log(1/eps), needs a number of neurons that grows like a power of 1/eps
[arxiv:1610.04161 §1]. For strongly convex functions, any network within eps
needs at least log2(mu/(16 eps)) neurons [arxiv:1610.04161 thm. 11]. Card:
[Why Deep Neural Networks for Function Approximation?](/papers/1610.04161).

### MLP head over contextual column embeddings (TabTransformer)

TabTransformer gives each categorical column its own lookup table. A column
identifier is concatenated to each class vector, and no positional encoding is
used because the columns have no order [arxiv:2012.06678 §2]. Transformer
layers then make each column's vector depend on the other columns of the same
row. The contextual vectors are flattened, joined with the numeric features,
and passed to an MLP [arxiv:2012.06678 §2] [arxiv:2012.06678 eq. 1]. The
default configuration is embedding size 32, six Transformer layers and eight
heads [arxiv:2012.06678 §3] [arxiv:2012.06678 app. B.1]. The top MLP's layer
sizes are 4·l and then 2·l, where l is the size of its input
[arxiv:2012.06678 app. B.1].

The paper's ablation removes the Transformer layers from the same model. That
plain MLP reaches 81.8 ± 0.4 mean AUC on 15 binary datasets, against 82.8 ± 0.4
for TabTransformer and 82.9 ± 0.4 for GBDT [arxiv:2012.06678 tab. 2]. The
per-dataset gain from the Transformer layers averages 1.0 points, and is
positive on 14 of 15 datasets [arxiv:2012.06678 tab. 1]. These are
author-reported results on binary classification only [arxiv:2012.06678 §3].

For semi-supervised use, the Transformer layers are first trained on unlabelled
rows, by replaced-value detection (RTD) or masking (MLM), and then fine-tuned
with the head on the labelled rows [arxiv:2012.06678 §2]. On the eight largest
datasets at 50, 200 and 500 labels, the RTD variant reaches 66.6, 70.9 and 73.1
mean AUC. The best other baseline, an MLP with entropy regularisation (ER),
reaches 65.6, 69.0 and 71.0 [arxiv:2012.06678 tab. 3]. On the twelve smaller
datasets, ER is best at 50 labels, and RTD is best at 200 and 500 labels
[arxiv:2012.06678 tab. 4]. Batch size, epochs and the pretraining learning rate
are not stated [arxiv:2012.06678 app. B.1]. Card:
[TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678).

### Learnable spline edges instead of linear weights (KAN)

A Kolmogorov-Arnold network keeps the layer-by-layer shape but has no weight
matrix and no fixed node activation. Each edge carries its own learnable
univariate function, a residual SiLU term plus a B-spline term, and each node
only sums what arrives on its incoming edges [arxiv:2404.19756 §2.2]. The
parameter count is O(N²L(G+k)) against O(N²L) for an MLP of the same depth and
width, so the authors expect a much smaller width to suffice
[arxiv:2404.19756 §2.2].

The paper compares against MLPs of matched shape on special functions,
Feynman equations and a Poisson PDE [arxiv:2404.19756 §3.1]
[arxiv:2404.19756 §3.2] [arxiv:2404.19756 §3.3] [arxiv:2404.19756 §3.4]. The
author-reported results are: KANs have lower test error than the MLP in every
row of the special-function table [arxiv:2404.19756 §3.2]; on the Feynman
equations the two are described as comparable on average
[arxiv:2404.19756 §3.3]; and on the PDE a two-layer, width-10 KAN with about
10² parameters reaches about 10^-7 MSE, against 10^-5 for a four-layer,
width-100 MLP with about 10^4 parameters. The authors say KAN training was
about ten times slower in their implementation [arxiv:2404.19756 §3.4].

The accuracy and scaling results assume the true KAN shape is known
[arxiv:2404.19756 §3.1]. The pruning and symbolic-fitting steps that produce
readable formulas need a human to name the candidate functions
[arxiv:2404.19756 §2.5.1] [arxiv:2404.19756 §2.5.2]. Card:
[KAN: Kolmogorov-Arnold Networks](/papers/2404.19756).

### MLP-only forecaster over several scales (TimeMixer)

TimeMixer uses linear layers and GELU activations only, with no attention
[arxiv:2405.14616 §3]. The past input is average-pooled into several scales.
At each scale a moving average splits the series into a seasonal part and a
trend part [arxiv:2405.14616 §3.2]. Seasonal parts are mixed from fine to
coarse and trend parts from coarse to fine, each with a two-layer mixer over
time [arxiv:2405.14616 eq. 4] [arxiv:2405.14616 eq. 5]. A linear predictor at
each scale then produces a forecast, and the forecasts are summed
[arxiv:2405.14616 §3.3] [arxiv:2405.14616 eq. 6]. The width between the two
layers of each mixer is not stated [arxiv:2405.14616 §3.2].

The paper's default setting is M = 3 down-sampling steps for long-term data and
M = 1 for short-term data [arxiv:2405.14616 §4.2] [arxiv:2405.14616 app. D].
The authors report the best average MSE and MAE on all eight long-term datasets
at input length 96 [arxiv:2405.14616 tab. 2], and the best scores on PEMS
[arxiv:2405.14616 tab. 3] and on the M4 weighted average
[arxiv:2405.14616 tab. 4]. Baselines are re-run under one setting, so their
numbers differ from the numbers in their own papers [arxiv:2405.14616 §4].
At input length 3072 the authors report 0.016 s per iteration and 1411 MiB for
TimeMixer, against 0.094 s and 16119 MiB for PatchTST. The batch size of that
table is not stated [arxiv:2405.14616 app. B] [arxiv:2405.14616 tab. 8].

The paper does not describe normalising the input. The authors' code applies
an instance normalisation to each scale and undoes it on the output, and the
reported numbers depend on it, as recorded on the card. Card:
[TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](/papers/2405.14616).

### Residual MLP encoder-decoder with a linear shortcut (TiDE)

TiDE is also MLP-only, with no attention, recurrence or convolution
[arxiv:2304.08424 §4]. Its basic layer is a residual block with one hidden
layer and a ReLU, a fully linear skip, dropout on the output layer only, and
layer norm on the sum of the output and the skip [arxiv:2304.08424 §4]. Whether
layer norm and dropout are on is set per dataset [arxiv:2304.08424 tab. 8].

The model runs per series, with weights shared across series. Past values,
projected covariates for the look-back and horizon, and static attributes are
flattened into one vector and encoded into an embedding
[arxiv:2304.08424 §4.1] [arxiv:2304.08424 eq. 4]. A dense decoder maps the
embedding to one vector per horizon step, and a temporal decoder combines each
step's vector with that step's projected covariates [arxiv:2304.08424 §4.2].
A linear map from the look-back is added to the output, so a purely linear
model is a special case of TiDE [arxiv:2304.08424 §4.2].

The paper's published settings use look-back 720 for all horizons, MSE loss
and Adam with default settings [arxiv:2304.08424 §5.1], and 512 series per batch
[arxiv:2304.08424 app. B.3]. The hidden size, depth, decoder width and
dropout are tuned per dataset [arxiv:2304.08424 tab. 8]. The number of layers
is tuned from {1, 2, 3} with encoder and decoder depth set separately
[arxiv:2304.08424 tab. 7].

The author-reported results are mixed. On Traffic, TiDE has lower MSE than
PatchTST/64 at all four horizons, with 0.336, 0.346, 0.355 and 0.386 against
0.360, 0.379, 0.392 and 0.432 [arxiv:2304.08424 tab. 2]. The paper says that,
apart from Weather, TiDE either beats PatchTST or stays within statistical
significance for most horizons [arxiv:2304.08424 §5.1]. The table in the
paper also shows PatchTST/64 with lower MSE in several rows, such as ETTh2 at
horizon 720 (0.393 against 0.419) [arxiv:2304.08424 tab. 2].

The efficiency claim is also author-reported: inference about five times
faster and training more than ten times faster than the best Transformer
[arxiv:2304.08424 §1]. The text gives no numeric timing table; the timing
evidence is the Electricity plots, which show an order-of-magnitude gap in
inference [arxiv:2304.08424 §5.3] [arxiv:2304.08424 fig. 2]. The paper's theory section bounds a linear
autoregressive class, not the full network [arxiv:2304.08424 app. A.1]. Card:
[Long-term Forecasting with TiDE: Time-series Dense Encoder](/papers/2304.08424).

### Random, untrained MLP maps before isolation trees (Deep Isolation Forest)

No network is trained. The data passes through r MLPs whose weights are drawn
at random and then kept fixed, and an isolation forest is grown on each mapped
copy [arxiv:2206.06602 §5.1] [arxiv:2206.06602 eq. 1]. Each map has a representation width d.
The paper samples weights from a widely used distribution, and does not name
the hidden widths or the activation [arxiv:2206.06602 §5.1] [arxiv:2206.06602 §5.5.3]. The
authors argue that the random maps expose anomalies that separate from normal
data only in combinations of features [arxiv:2206.06602 §1].

Author-reported: average AUC-PR 0.351 over ten tabular datasets, against 0.219
for EIF and 0.144 for iForest [arxiv:2206.06602 tab. 3]. The paper's score
formula leaves its normaliser undefined. The released code differs from the
paper's formula and standardises each mapped copy over the rows being scored,
so the card's Paper vs code section is needed before reimplementing. Card:
[Deep Isolation Forest for Anomaly Detection](/papers/2206.06602).

### Utility network inside a random-utility choice model (RUMnet)

A feed-forward network is the utility function of a random-utility model. For
each choice, the network is evaluated on K² pairs of sampled unobserved product
and customer attributes, and each evaluation gives a softmax over the offered
products. The softmaxes are averaged [arxiv:2207.12877 §2.2] [arxiv:2207.12877 §2.3]. Training
minimises the negative log-likelihood of the observed choice [arxiv:2207.12877
§4.1] [arxiv:2207.12877 eq. 2]. The paper's experiments use depth and width settings of (3,10)
and (5,20) [arxiv:2207.12877 §6.2]. Its theory allows any random-utility model
to be approximated arbitrarily closely [arxiv:2207.12877 §3] [arxiv:2207.12877 Proposition 1].
The experiments use ELU, where the theory uses ReLU [arxiv:2207.12877 §6.2]
[arxiv:2207.12877 app. E].

Author-reported: the highest accuracy on Swissmetro (0.793) and Expedia
(0.425), and the lowest Expedia loss, while a random forest has the lowest
Swissmetro loss (0.523 against 0.561) [arxiv:2207.12877 tab. 1] [arxiv:2207.12877 §6.3]. Training
is about an order of magnitude slower than a random forest, and about 36 hours
on Expedia [arxiv:2207.12877 app. E]. Card:
[Representing Random Utility Choice Models with Neural Networks](/papers/2207.12877).

### Residual MLP with learned identity embeddings (STID)

STID has no graph and no recurrence. Each series' P-step history goes through a
linear embedding, which is joined to a learned spatial identity for the series
and to learned time-of-day and day-of-week identities. The joined vector goes
through L residual MLP blocks and a linear head that outputs F steps
[arxiv:2208.05233 §3] [arxiv:2208.05233 eq. 1-4]. The paper uses D = 32 and L = 3 for all
datasets, with input length 12 for the PEMS sets and 168 for Electricity
[arxiv:2208.05233 §4.1]. It trains with MAE [arxiv:2208.05233 §3] [arxiv:2208.05233 eq. 5].

Author-reported: the lowest average MAE on all five benchmarks among the
methods in its table, and the lowest seconds per epoch [arxiv:2208.05233 tab. 3]
[arxiv:2208.05233 tab. 2]. The released Electricity configuration uses an input and output
length of 336 and a different width and depth, so the table's Electricity
column does not follow from the paper's stated setup [arxiv:2208.05233 §4.1].
Card: [Spatial-Temporal Identity: A Simple yet Effective Baseline for
Multivariate Time Series Forecasting](/papers/2208.05233).

### Time-mixing and feature-mixing MLP blocks (TSMixer)

Each block applies a linear layer over time, shared across variates, and then a
two-layer MLP over variates, shared across time steps. Each step has a residual
connection and a normalization [arxiv:2303.06053 §4.1]. The basic model uses
only the history of the target series, and one linear layer maps the input
length to the horizon [arxiv:2303.06053 app. B.3]. The long-term benchmarks use
input length 512 [arxiv:2303.06053 §5.1].

Author-reported: comparable to PatchTST on seven long-term benchmarks, with an
average MSE change of −1.53% against PatchTST [arxiv:2303.06053 tab. 3]. On M5,
test WRMSSE is 0.737 without auxiliary features and 0.640 with static and future
features, the latter using an extension [arxiv:2303.06053 tab. 4] [arxiv:2303.06053 §5.2]. The
release covers only the basic model, so the M5 numbers cannot be reproduced
from it, and its normalization and RevIN wiring differ from the paper in ways
the card's Paper vs code section lists. Card:
[TSMixer: An All-MLP Architecture for Time Series Forecasting](/papers/2303.06053).

### Frequency-domain MLPs over a Fourier transform (FreTS)

The input is passed through a discrete Fourier transform. A frequency-domain MLP
with complex weights runs along the channel axis, then along the time axis, and
each run is inverted. A two-layer feed-forward head maps the result to the
horizon in one step [arxiv:2311.06184 §3.1] [arxiv:2311.06184 eq. 1-5]. The paper writes each
frequency layer as a complex matrix product with bias, and leaves its activation
unnamed [arxiv:2311.06184 §3.2] [arxiv:2311.06184 eq. 6].

Author-reported: MAE 0.120 on Solar and 0.011 on Traffic at horizon 12
[arxiv:2311.06184 tab. 1], and the lowest value in its own multi-step tables on
METR-LA and COVID-19 [arxiv:2311.06184 tab. 8] [arxiv:2311.06184 tab. 9]. On long-term data it is
best on most rows of [arxiv:2311.06184 tab. 2]. The released code uses only
diagonal frequency weights and never runs the channel learner, so it is not
the model the paper describes. Card:
[Frequency-domain MLPs are More Effective Learners in Time Series Forecasting](/papers/2311.06184).

### Small tanh MLP on RGB patches (UAV pixel classification)

The MLP has one hidden layer of five tanh units and a two-unit softmax output.
It classifies only the centre pixel of an RGB patch, with patch sizes from 1 to
9 px [arxiv:2609.40212 §2.2.2]. It is trained with scaled conjugate gradient
and early stopping, and of three networks per structure the one with the best
validation MCC is kept [arxiv:2609.40212 §2.2.2].

Author-reported, on the WIEL test site: the 9-px MLP reaches MCC 0.8121 and
accuracy 0.9561, and a linear network with 9 px reaches the highest WIEL MCC of
0.8208. The best vegetation index, Excess Blue, reaches MCC 0.6099 on WIEL
[arxiv:2609.40212 tab. 2] [arxiv:2609.40212 tab. 3]. On the JERZ validation part the CNN is best,
while on WIEL the linear and MLP networks are slightly better than the CNN
[arxiv:2609.40212 §3.2]. Card:
[Recognition of Urbanized Areas in UAV-Derived Very-High-Resolution Visible-Light Imagery](/papers/2609.40212).

### Pairwise MLP scorers over query pairs (TopoMLP)

Here the MLPs do not map one feature vector to one output. They score pairs.
Two MLPs read each lane query's decoder features: one predicts control-point
offsets, the other a class score. Each query is one lane drawn as a Bezier
curve [arxiv:2310.06753 §3.1]. Two topology heads then score links. Each embeds
the query features, and the lane head also embeds the lane coordinates. Every
row query is concatenated with every column query, and a three-layer ReLU MLP
gives one logit per ordered pair [arxiv:2310.06753 §3.3] [arxiv:2310.06753
eq. 3]. The lane-traffic head adds an embedding of the view matrix before the
pairing [arxiv:2310.06753 §3.4] [arxiv:2310.06753 eq. 4].

Author-reported on OpenLane-V2 subset A with ResNet-50: OLS 38.2, and 41.2 with
extra YOLOv8 proposals [arxiv:2310.06753 tab. 1]. The topology terms are much
lower than the detection terms, at TOP_ll 7.2 and TOP_lt 22.8 [arxiv:2310.06753
tab. 1]. Removing the L1 term or the position embedding costs under one OLS
point each [arxiv:2310.06753 tab. 3]. These ablations are single runs with no
variance reported, so such gaps are not established. The metric is easy to
move: pushing the link scores towards 0 or 1 around 0.5 lifts OLS from 38.2 to
42.2, and the adjusted metric gives 36.3 and 34.5 [arxiv:2310.06753 tab. 4].
Card: [TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning](/papers/2310.06753).

### Product layer over field embeddings (PNN)

Each categorical field is one-hot encoded and mapped to its own learned
embedding of M = 10 values [arxiv:1611.00144 §3.1] [arxiv:1611.00144 §4.4].
The first hidden layer is a ReLU of three parts: a linear term over the
stacked embeddings, a product term over pairs of fields, and a bias
[arxiv:1611.00144 eq. 3]. The paper argues that a plain MLP's first layer,
which adds its inputs, models interactions between fields poorly
[arxiv:1611.00144 §1]. The two variants differ in the product term. IPNN takes
inner products, and a squared-norm identity reduces the cost to one value per
hidden node [arxiv:1611.00144 §3.1.1] [arxiv:1611.00144 eq. 11-12]. OPNN takes
outer products, and superposing all field vectors first gives one M × M matrix
shared by all hidden nodes [arxiv:1611.00144 §3.1.2] [arxiv:1611.00144 eq. 14].

The experiments use three hidden layers in total and a sigmoid output unit,
with dropout of 0.5 on the hidden layers [arxiv:1611.00144 §4.4]. The width of
each hidden layer is not stated [arxiv:1611.00144 §3.1]. The optimiser is
SGD, but its learning rate, batch size and momentum are not stated
[arxiv:1611.00144 §4.2]. The released TensorFlow demo uses one 500-unit hidden
layer and no dropout, so it is not the model in the paper. Implement from §3.1
and treat the release as a reference [Product-based Neural Networks for User
Response Prediction](/papers/1611.00144).

Author-reported AUC: IPNN reaches 77.79% on Criteo, against 76.71% for CCPM,
the best baseline there [arxiv:1611.00144 tab. 1]. OPNN reaches 81.74% on
iPinYou, against 76.38% for CCPM [arxiv:1611.00144 tab. 2]. The concatenated
PNN* variant is below both IPNN and OPNN on AUC on both datasets
[arxiv:1611.00144 tab. 1] [arxiv:1611.00144 tab. 2]. The tables give no error
bars or repeat counts [arxiv:1611.00144 tab. 1].

### MLP as a continuous convolution kernel (PCCN)

Here the MLP is not the layer's output map. It is the kernel. A continuous
convolution takes the relative position z between an output point and a support
point, and an MLP maps z to a weight for each output channel. The layer then
sums over the support points within a window, so it needs no grid
[arxiv:2101.06742 §3.1] [arxiv:2101.06742 §3.2]. The window is the K nearest
points or the points within a radius r [arxiv:2101.06742 §3.3].

The sum is a Monte-Carlo estimate over sampled points [arxiv:2101.06742 §3.1].
The paper factorises the weights: a linear map mixes the input channels, and
the MLP produces only one kernel value per point pair and output channel. This
avoids storing a full weight tensor per pair [arxiv:2101.06742 §3.3]. At
inference the batch-norm layer is folded into the MLP's last linear layer, which
the authors report as a 3× speed-up [arxiv:2101.06742 §3.3]. The paper says
residual links between these layers are critical for convergence [arxiv:2101.06742 §3.2].

The indoor network uses 8 such layers with width 32, and the last layer width
128, with a global max-pooled feature joined to each point before the classifier
[arxiv:2101.06742 §4.1]. The driving network uses 16 residual layers
[arxiv:2101.06742 §4.2]. Learning rate, batch size, epochs and the neighbour
count used in training are not stated [arxiv:2101.06742 §4.1] [arxiv:2101.06742 §4.2]. The
paper also notes that a regular grid gives back ordinary convolution, and a
Gaussian kernel over position and feature gives a bilateral filter
[arxiv:2101.06742 §3.3]. The paper does not test those special cases.

Author-reported: 58.27 mIOU on the indoor scene benchmark, against 48.92 for
SEGCloud and 41.09 for PointNet [arxiv:2101.06742 tab. 1]. On the in-house
driving data, the PCCN-only model reaches 46.35 mIOU, below 3D-FCN at 49.28,
but it is about 8× smaller. Adding it on top of a trained 3D-FCN gives 58.06
mIOU [arxiv:2101.06742 tab. 2]. Lidar flow end-point error is 7.810 cm against
8.161 cm for 3D-FCN [arxiv:2101.06742 tab. 3]. Runtime on a GTX 1080 Ti is
33 ms for the forward pass and 61 ms end to end with a KD-tree neighbour search
[arxiv:2101.06742 §4.2]. Card:
[Deep Parametric Continuous Convolutional Neural Networks](/papers/2101.06742).

### Dense network written by hand for a microcontroller (AI-ANNE)

The network is trained off the device in TensorFlow and Keras. Its weights and
biases are then copied into MicroPython lists on a Raspberry Pi Pico, and the
forward pass is rebuilt from hand-written neuron, layer and activation
functions, with no ML library on the board [arxiv:2501.03256 §I]
[arxiv:2501.03256 §III] [arxiv:2501.03256 app. A]. The paper lists 264 KB of
SRAM for the RP2040 and 520 KB for the RP2350 [arxiv:2501.03256 §II].

Two example nets are given, both for Iris, Versicolor against Virginica. The
first has 8 neurons in a 2-3-2-1 shape, with ReLU, tanh, softmax and sigmoid
layers, and reaches 90% accuracy. The second has 6 neurons in a 3-2-1 shape,
with ReLU, sigmoid and sigmoid layers, and reaches 95% accuracy
[arxiv:2501.03256 tab. 3] [arxiv:2501.03256 tab. 5] [arxiv:2501.03256 §IV].
Both accuracies come from a test set of 20 samples [arxiv:2501.03256 tab. 1].
The paper reports no latency, memory or energy figures and no baseline. It
names AIfES as a comparable on-device framework but gives no comparison numbers
[arxiv:2501.03256 §I]. Optimiser, loss, epochs, learning rate and batch size
are not stated [arxiv:2501.03256 §IV]. Card:
[AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems](/papers/2501.03256).

### Dropout MLP on sensor statistics for terrain classes (rover)

A feed-forward classifier over statistics of one-second windows. Each window
gives 225 inputs: 35 per force-torque sensor and 15 per IMU, with five
statistics per channel [arxiv:2411.04700 §3]. The text describes one hidden
layer of 64 units with dropout, and the figure shows two hidden layers of
64 units [arxiv:2411.04700 §3.2] [arxiv:2411.04700 fig. 8]. The released code
uses ReLU hidden units, a log-softmax output and cross-entropy loss, and it
picks the hidden size by cross-validation [/papers/2411.04700].

The paper trains for 50 epochs with batch size 32. The optimiser and learning
rate are not stated in the paper [arxiv:2411.04700 §3.2]; the code uses Adam
at 0.001 [/papers/2411.04700]. The paper places 20% dropout between hidden
layers, while the code also drops inputs at 0.1 [arxiv:2411.04700 §3.2]
[/papers/2411.04700].

Author-reported test accuracy on one random 25% split: 96.17% with
force-torque features, 79.35% with IMU only, and 96.76% with both
[arxiv:2411.04700 tab. 5] [arxiv:2411.04700 tab. 6] [arxiv:2411.04700 tab. 7].
The same features in an SVM reach 95.58% [arxiv:2411.04700 tab. 3]. The
IMU-only network recalls 28.36% of pebbles, against 88.06% with force-torque
features [arxiv:2411.04700 tab. 5] [arxiv:2411.04700 tab. 6]. Card:
[Field Assessment of Force Torque Sensors for Planetary Rover Navigation](/papers/2411.04700).

### Batch-normalised FCNN for hip joint moment (leave-one-subject-out)

A fully connected network with three hidden layers of 50 units and one output
for hip joint moment in Nm/kg [arxiv:2412.07823 §II.C] [arxiv:2412.07823 fig. 1].
Each hidden layer is linear, then batch normalization, then ReLU, then dropout
of 0.2 [arxiv:2412.07823 §II.C]. The 14 inputs are not broken down in the
text, and input standardisation is not stated [arxiv:2412.07823 §II.C]
[arxiv:2412.07823 fig. 1].

Training uses MSE loss, Adam at learning rate 0.001 and batch size 64, for at
most 100 epochs. Training stops after 10 epochs without validation-loss
improvement, and which checkpoint is restored is not stated [arxiv:2412.07823
§II.C]. Evaluation is leave-one-subject-out over 11 subjects. For each held-out
subject, the rest is split 80/20 into training and validation
[arxiv:2412.07823 §II.C] [arxiv:2412.07823 fig. 4].

Author-reported RMSE, averaged over the folds: 0.28 ± 0.05 Nm/kg with all 20
tasks, 0.30 ± 0.05 with the 8 optimized tasks, and 0.38 ± 0.08 with the 8
cyclic tasks. R² is 0.64, 0.57 and 0.32 in the same order [arxiv:2412.07823
§III.B] [arxiv:2412.07823 fig. 4]. Both the all-task and the optimized-task
sets have lower RMSE than cyclic-only, by 27.11 ± 6.18% and 20.60 ± 9.80%
(p < 0.05). The paper reports no difference between all tasks and the
optimized set, and gives no p-value for it [arxiv:2412.07823 §III.B]. The
optimized set is chosen by a clustering score, not by fitting the network
[arxiv:2412.07823 §I] [arxiv:2412.07823 §II.B]. Card:
[Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications](/papers/2412.07823).

## When to use which

The cards answer different questions, so the constraints below pick between
them. Only the cited results support each row.

| Situation | Use | Why |
|---|---|---|
| Fixed numeric feature vector per example, labelled data, need a small supervised baseline | A small MLP of the 6-3-1 shape as a starting point | The paper trains this shape on 2313 samples and reports 99.4% test accuracy [arxiv:1704.02286 §V] [arxiv:1704.02286 fig. 3]. The result comes from one simulated setup [arxiv:1704.02286 §IV]. |
| Need a comparison to other classifiers | Run your own baselines | The paper reports no comparison method [arxiv:1704.02286 §IV] [arxiv:1704.02286 §V]. |
| Class imbalance matters for the decision | Report per-class precision, recall and false-alarm rate yourself | Attacks are 64.18% of the samples, so always predicting attack scores 64.2% [arxiv:1704.02286 tab. 2]. The paper reports no precision, F1 or false-alarm rate [arxiv:1704.02286 §V]. |
| Real-time detection with a latency budget | Measure latency and throughput yourself | The paper treats the detector as an offline analysis of captured traffic and measures no latency [arxiv:1704.02286 §I] [arxiv:1704.02286 §V]. |
| A function of a few variables that is costly to tabulate exactly over a large horizon and budget | A small tanh MLP fitted to the table, as in the bidding paper | Exact value iteration needs O(TB) time and memory [arxiv:1701.02490 §4.1]. The fit is reported only inside the trained box, and the authors call generalisation "satisfactory only in small scales" [arxiv:1701.02490 §6.2]. Their scaling variants reuse a small table for larger horizons [arxiv:1701.02490 §4.1]. |
| A table that fits in memory at your horizon and budget | Keep the exact table | The paper's small-scale results use the exact table [arxiv:1701.02490 §6.1] [arxiv:1701.02490 §4.1]. |
| Tabular numeric features and an MLP as the model | Fit a scaler on the training split only, and compare with no scaling on your own data | Scaling moved MLP accuracy a lot on the benchmark's Dry Bean run, from 0.2980 to 0.9334 [arxiv:2506.08274 app. A]. One split per dataset is the only evidence, so check the effect on your own data. |
| Tabular data where you want to skip scaling | A tree ensemble, such as a random forest or boosting | The paper reports ensembles as largely insensitive to the scaler [arxiv:2506.08274 §5.1]. Random forest shows no significant change in accuracy (Wilcoxon p = 0.9710) [arxiv:2506.08274 app. A]. |
| Categorical columns, labelled rows, and unlabelled rows from the same table | A Transformer over column embeddings with an MLP head, pretrained on the unlabelled rows | Author-reported: +1.0 mean AUC points over the same model without Transformer layers [arxiv:2012.06678 tab. 1], and gains over an MLP baseline at 50 to 500 labels on the largest datasets [arxiv:2012.06678 tab. 3]. Binary classification only [arxiv:2012.06678 §3]. |
| Tabular data where streaming or end-to-end training with other encoders matters | A neural model, for the training setup rather than for accuracy | The paper's case for a neural model rests on training needs, since trees do not suit those settings [arxiv:2012.06678 §1]. Its GBDT and TabTransformer scores are level on its benchmark (82.9 vs 82.8 mean AUC) [arxiv:2012.06678 tab. 2]. |
| Multi-scale time series, a cheap MLP-only forecaster, long input windows | A TimeMixer-style model with seasonal and trend mixing across scales | Author-reported best MSE and MAE at input 96 on eight long-term datasets [arxiv:2405.14616 tab. 2], and lower time and memory than PatchTST at input length 3072 [arxiv:2405.14616 tab. 8]. The reported numbers depend on a normalisation step the paper does not describe, and M is a per-dataset choice [arxiv:2405.14616 §4.2]. |
| Small problem in a few variables where a readable formula matters, and slow training is acceptable | A KAN, with pruning and symbolic fitting | Author-reported: lower error than matched MLPs on the special-function table [arxiv:2404.19756 §3.2] and a small PDE model [arxiv:2404.19756 §3.4]. Training was about ten times slower in the authors' PDE setup [arxiv:2404.19756 §3.4], and the symbolic steps need a human [arxiv:2404.19756 §2.5.2]. |
| Long-horizon multivariate forecasting with known future covariates, where attention cost is the concern | A TiDE-style residual MLP encoder-decoder with a linear shortcut | Author-reported: lower Traffic MSE than PatchTST/64 at all four horizons [arxiv:2304.08424 tab. 2] and about five times faster inference than the best Transformer [arxiv:2304.08424 §1]. The released code differs from the paper in ways that change the model, such as a learned series-ID embedding, so check the card's Paper vs code section before reusing it. |
| Need runnable code to learn or teach MLP training from scratch | The MLP chapter of the textbook, with its training-loop code | The chapter covers multilayer perceptrons, initialization, dropout and generalization [arxiv:2106.11342 §5]. The paper reports no benchmark numbers of its own. Text is CC BY-SA and sample code is MIT-0 style, so keep the two terms separate [arxiv:2106.11342 §5]. |
| Question is whether a depth or size bound holds for a target function | Use the bounds from the expressivity paper | The bounds say what can be represented, with hand-built weights [arxiv:1610.04161 §3] [arxiv:1610.04161 §4]. They say nothing about whether training finds those weights. |
| Need a learned network that is guaranteed to reach a bound | Do not assume it from any of these papers | None of the cards gives a convergence result for training. The expressivity paper's step units are discontinuous and it gives no learning procedure [arxiv:1610.04161 §3.1]. The scaling benchmark measures accuracy on one split per dataset and makes no convergence claim [arxiv:2506.08274 §6]. |
| Unlabelled tabular data, anomaly scores wanted, and no training loop to tune | A random-map isolation forest, with the scoring fixed to the released code or to a reimplementation you check | Author-reported AUC-PR 0.351 against 0.219 for EIF on ten tabular sets [arxiv:2206.06602 tab. 3]. The paper's formula and the released code differ, and the code's scores depend on the batch scored [Deep Isolation Forest](/papers/2206.06602). |
| Choice data with many alternatives, where a fixed utility form may be too restrictive, and hours of training are acceptable | A utility network inside a random-utility model, as in RUMnet | Author-reported: highest accuracy on Swissmetro (0.793) and Expedia (0.425) [arxiv:2207.12877 tab. 1]. A random forest has the lowest Swissmetro loss, so pick the metric first [arxiv:2207.12877 §6.3]. Training is about an order of magnitude slower than a random forest [arxiv:2207.12877 app. E]. |
| Many related series, no graph available, and time per epoch matters | A graph-free MLP with learned identity embeddings, as in STID | Author-reported lowest average MAE on all five benchmarks, and the lowest seconds per epoch [arxiv:2208.05233 tab. 3] [arxiv:2208.05233 tab. 2]. The Electricity setup differs between paper and release, so state which config you used [arxiv:2208.05233 §4.1]. |
| Multivariate forecasting with an all-MLP model that is simple to train and tune | TSMixer's basic model, with its time-mixing and feature-mixing blocks | Author-reported: comparable to PatchTST on seven long-term benchmarks [arxiv:2303.06053 tab. 3]. The M5 gain needed auxiliary features, and that code is not released [arxiv:2303.06053 tab. 4] [arxiv:2303.06053 §5.2]. |
| Regularly sampled series with a repeating pattern, where a frequency view of the input is worth testing against a time-domain MLP | A frequency-domain MLP, as in FreTS | Author-reported MAE 0.120 on Solar and 0.011 on Traffic at horizon 12 [arxiv:2311.06184 tab. 1]. The released code uses diagonal weights and no channel learner, so reproduce the paper's layer rather than the release [arxiv:2311.06184 §3.2]. |
| Per-pixel classes from RGB UAV orthomosaics, with a labelled site for training and a different site for testing | A one-hidden-layer tanh MLP on 5–9 px patches, against one threshold on Excess Blue | On the second site, the 9-px MLP reaches MCC 0.8121 and Excess Blue 0.6099 [arxiv:2609.40212 tab. 2] [arxiv:2609.40212 tab. 3]. Excess Blue needs no training, and its best threshold moved from 0.100 to 0.131 across series [arxiv:2609.40212 §3.1]. Two sites only [arxiv:2609.40212 §4]. |
| Candidates (lanes, detections, items) where each pair needs a yes/no link, and the detector's quality limits the links | A pairwise MLP scorer over query pairs, as in TopoMLP, with the links trained alongside the detector | Author-reported OLS 38.2 on subset A [arxiv:2310.06753 tab. 1]. Replacing the predicted links with ground truth raised the topology scores only a little on the authors' Swin-B run, which points to detection as the limit [arxiv:2310.06753 §1]. Check the metric before comparing numbers: confidence changes alone move OLS by about 4 points [arxiv:2310.06753 tab. 4]. |
| Multi-field categorical inputs, such as ad clicks, where pairs of fields may interact and a plain MLP is the default | A product-layer network, IPNN for inner products or OPNN for outer products, as in PNN | Author-reported AUC: IPNN 77.79% on Criteo against 76.71% for CCPM [arxiv:1611.00144 tab. 1], and OPNN 81.74% on iPinYou against 76.38% for CCPM [arxiv:1611.00144 tab. 2]. Each number is one run with no variance, so repeat with seeds before relying on the gap. The released demo differs from the paper, so build from §3.1 [arxiv:1611.00144 §3.1]. |
| Points scattered in 3D without a grid, such as lidar or depth scans, that need a convolution-style layer and can change the number of points | A continuous convolution whose kernel is an MLP of relative position, as in PCCN | Author-reported: 58.27 mIOU on the indoor benchmark against 48.92 for SEGCloud [arxiv:2101.06742 tab. 1]. On driving data the PCCN-only model is below 3D-FCN on mIOU, and the gain comes from combining it with a 3D-FCN [arxiv:2101.06742 tab. 2]. No official code, and the outdoor data is in-house [arxiv:2101.06742 §4.2]. |
| Small dense classifier that must run on a microcontroller with no ML runtime, with weights trained on a desktop | Hand-written dense layers in MicroPython, as in AI-ANNE | Author-reported 90% and 95% accuracy on two Iris nets, each tested on 20 samples [arxiv:2501.03256 §IV] [arxiv:2501.03256 tab. 1]. No latency, memory or energy is measured, so measure those on your own board. The released code uses other activations than the paper's tables [arxiv:2501.03256 tab. 3], so rebuild the layers from the paper and check each one. |
| Labelled windows of sensor statistics with a few classes, where an IMU-only baseline is the comparison | A small dropout-regularised MLP, as in the rover paper, with an SVM on the same features as a check | Author-reported test accuracy 96.17% with force-torque features against 79.35% with IMU only [arxiv:2411.04700 tab. 5] [arxiv:2411.04700 tab. 6]. The SVM reaches 95.58% on the same features [arxiv:2411.04700 tab. 3], a gap of 0.59 points on one 25% split, so one split does not separate the two [arxiv:2411.04700 §3.1]. The paper's text and figure disagree on depth, so state which network you built [arxiv:2411.04700 fig. 8]. |
| Body-worn sensors estimate joint torque across many subjects, and recording every activity is costly | A batch-normalised three-layer MLP with leave-one-subject-out evaluation, trained on a subset of tasks, as in the exoskeleton paper | Author-reported RMSE 0.30 ± 0.05 Nm/kg on 8 selected tasks against 0.28 ± 0.05 on all 20 [arxiv:2412.07823 §III.B] [arxiv:2412.07823 fig. 4]. The gap is inside the reported spread. The subset comes from a clustering score, not from model results [arxiv:2412.07823 §I]. The cohort is healthy young adults only [arxiv:2412.07823 §V]. |

## Implementation sketch

The common core is the forward pass, the error terms and the update. The
shapes follow the paper's index convention, with the hidden layer first. The
paper's weight-derivative equation was not legible in the text layer, so the
weight gradients below follow the chain rule from [eq. 5-6]
[arxiv:1704.02286 §III-B].

```pseudocode
# Forward pass for one example  [arxiv:1704.02286 eq. 1-4]
# x: (6,); w1: (3, 6); b1: (3,); w2: (1, 3); b2: (1,)
def forward(x, w1, b1, w2, b2):
    # Three sigmoid hidden units  [eq. 1-3]
    a_hidden = sigmoid(w1 @ x + b1)            # (3,)
    # One sigmoid output unit  [eq. 4]
    out = sigmoid(w2 @ a_hidden + b2)          # (1,)
    return a_hidden, out
```

```pseudocode
# Error terms for one example  [arxiv:1704.02286 eq. 5-6]
# y: target; its encoding is not stated (see Contradictions)
# f'(z) = out * (1 - out) assumes the logistic sigmoid; the paper
# does not write f' out [§III]
def backward(x, y, a_hidden, out, w2):
    # Output error from the half squared error  [eq. 5]
    d_out = -(y - out) * out * (1 - out)       # (1,)
    # Hidden errors: send the output error back through w2  [eq. 6]
    d_hidden = (w2.T @ d_out) * a_hidden * (1 - a_hidden)   # (3,)
    # Bias gradients equal the error terms  [eq. 8]
    grad_b2 = d_out
    grad_b1 = d_hidden
    # Weight gradients: error times the activation or input feeding it
    grad_w2 = outer(d_out, a_hidden)           # (1, 3)
    grad_w1 = outer(d_hidden, x)               # (3, 6)
    return grad_w1, grad_b1, grad_w2, grad_b2
```

```pseudocode
# Forward pass of the value-difference network  [arxiv:1701.02490 §4.1] [§6.2]
# x: (2,) = (t, b); w1: (30, 2); w2: (15, 30); w3: (1, 15)
# Input scaling and initialisation are not stated in the paper
def forward_value_diff(x, w1, b1, w2, b2, w3, b3):
    # First tanh layer over the two inputs  [§6.2]
    h1 = tanh(w1 @ x + b1)                     # (30,)
    # Second tanh layer  [§6.2]
    h2 = tanh(w2 @ h1 + b2)                    # (15,)
    # Linear output: the value difference D(t, b)  [§4.1]
    return w3 @ h2 + b3                        # (1,)
```

```pseudocode
# Scale tabular features, then train a library MLP  [arxiv:2506.08274 §4.3]
# The split happens before this function is called  [arxiv:2506.08274 §4.2]
def scaled_mlp_fit_predict(X_train, y_train, X_test, scaler, model):
    # Statistics come from the training rows only  [arxiv:2506.08274 §4.3]
    scaler.fit(X_train)
    X_train_s = scaler.transform(X_train)      # (n_train, d)
    X_test_s = scaler.transform(X_test)        # (n_test, d)
    # Defaults apply to every setting not named here  [arxiv:2506.08274 §4.3]
    model.fit(X_train_s, y_train)
    return model.predict(X_test_s)
```

```pseudocode
# Gradient-descent step  [arxiv:1704.02286 §III-B]
# The learning rate is not stated; lr is a free choice here.
def update(params, grads, lr):
    # Move each parameter a small step against its gradient
    return [p - lr * g for p, g in zip(params, grads)]
```

Where variants differ: the sigmoid classifier has a sigmoid output, the
value-difference network a linear one. Its activation is tanh, so the error
terms use 1 − tanh² instead of the sigmoid derivative. A second hidden layer,
a softmax output or a different loss would change the shapes and error terms.
The library variant adds a scaling step before the first layer, and the choice
of scaler is a free parameter the benchmark varies [arxiv:2506.08274 §3.1].
The TabTransformer head takes the flattened contextual embeddings and the
numeric features, and its layer sizes follow from its input size
[arxiv:2012.06678 app. B.1]. The TimeMixer forecaster applies one linear
predictor per scale and sums them [arxiv:2405.14616 eq. 6]. The KAN replaces
each weight and activation with a spline on an edge, so its forward pass sums
edge functions instead of multiplying by a matrix [arxiv:2404.19756 §2.2]. The
expressivity paper's networks use ReLU and binary step units and are built by
hand, so they have no training code to sketch [arxiv:1610.04161 §3]. The
random-map variant has no backward pass: its networks are fixed draws, and only
the trees are fitted [arxiv:2206.06602 §5.1]. The choice-model variant takes a softmax over
the alternatives for each sampled attribute pair, then averages them
[arxiv:2207.12877 §2.2]. TSMixer's blocks mix along time and variates in turn,
and STID's blocks are residual MLPs over a joined identity vector, so neither
matches the sketches above [arxiv:2303.06053 §4.1] [arxiv:2208.05233 §3]. The
frequency variant applies its layer in the Fourier basis, which the
time-domain sketch does not show [arxiv:2311.06184 §3.2]. The pixel classifier
has the same layered form with tanh hidden units and a two-unit softmax output
[arxiv:2609.40212 §2.2.2]. The pairwise scorer of the lane-detection variant
is an MLP over concatenated pairs of query features, one logit per pair, so
its input is two vectors joined rather than one feature vector
[arxiv:2310.06753 §3.3]. The product-layer network's first hidden layer takes
the field embeddings and the product term, not the raw input vector
[arxiv:1611.00144 eq. 3], so the sketches above do not show it. The
continuous-kernel variant feeds the MLP the offsets between points rather
than the features, and the features enter only through a linear map and the
weighted sum [arxiv:2101.06742 §3.1] [arxiv:2101.06742 §3.3]. The AI-ANNE
layers compute the same weighted sum and activation, but in batch form, with
weights stored as inputs by units and transposed before use [arxiv:2501.03256
tab. 2] [arxiv:2501.03256 app. A]. The rover network puts dropout on its
inputs and after its hidden layer, and the exoskeleton network puts batch
normalization before each ReLU and dropout after it [arxiv:2411.04700 §3.2]
[/papers/2411.04700] [arxiv:2412.07823 §II.C].

## Contradictions

Most cards measure different quantities, so most do not contradict one another
directly. The forecasting cards are the exception: they report MSE or MAE on
overlapping benchmarks. The expressivity paper shows that small deep networks can represent
certain functions [arxiv:1610.04161 thm. 4]. The trained networks report
accuracy or error and say nothing about representation bounds
[arxiv:1704.02286 §V] [arxiv:1701.02490 §6.2] [arxiv:2506.08274 §5.1]
[arxiv:2012.06678 §3] [arxiv:2206.06602 tab. 3] [arxiv:2609.40212 §3.2].

The KAN paper reports KANs as lower than the MLP in every row of its
special-function table [arxiv:2404.19756 §3.2], but describes the two as
comparable on average for the Feynman equations [arxiv:2404.19756 §3.3]. The
MLP widths differ between those experiments: 5 or 100 for the special functions
and 5 for the Feynman equations [arxiv:2404.19756 §3.2] [arxiv:2404.19756 §3.3].
Its 78% knot-theory MLP baseline is quoted from another group's work and is not
reproduced [arxiv:2404.19756 §4.3].

The trained MLP cards that report numbers measure different things. The sigmoid
classifier is a 6-3-1 network on traffic features [arxiv:1704.02286 §III].
The scaling benchmark uses a library MLP at default settings on 16 UCI tables
[arxiv:2506.08274 §4.3]. The TabTransformer comparison uses AUC on 15 binary
datasets [arxiv:2012.06678 §3]. Their accuracies and AUCs are not comparable.
The anomaly card reports AUC-PR on ten tabular sets [arxiv:2206.06602 tab. 3],
the pixel card reports MCC on two UAV sites [arxiv:2609.40212 tab. 2], and the
choice card reports log-likelihood loss and accuracy [arxiv:2207.12877 tab. 1].
The click-prediction card reports AUC and log loss on two ad datasets
[arxiv:1611.00144 tab. 1] [arxiv:1611.00144 tab. 2]. None of these is
comparable with the others.

The forecasting cards also disagree with themselves. FreTS says it beats all
baselines on all long-term datasets, but its own table has PatchTST with lower
MAE on Traffic at two horizons and on Electricity at 336
[arxiv:2311.06184 §4.1] [arxiv:2311.06184 tab. 2]. TSMixer's M5 table prints
0.000 for three validation entries of its extension, which look like
placeholders [arxiv:2303.06053 tab. 4].

Rankings depend on the metric. In RUMnet's table the random forest has the
lowest Swissmetro log-likelihood loss, while RUMnet has the highest accuracy on
both datasets [arxiv:2207.12877 tab. 1] [arxiv:2207.12877 §6.3]. The same
holds for the product-layer network: on iPinYou, OPNN has the highest AUC while
PNN* has the lowest log loss and RMSE [arxiv:1611.00144 tab. 2].

Within one card, the paper's labels conflict with its counts. The sigmoid
classifier's text calls true positives attacks "rightly classified" and false
positives normal events "classified rightly" [arxiv:1704.02286 §V]. The
confusion matrix shows 20 attack samples labelled normal and no normal sample
labelled attack [arxiv:1704.02286 fig. 3]. Use the counts, not the paper's TP
and FP labels.

The AI-ANNE paper's text says softmax normalises across the classes of one
input [arxiv:2501.03256 §III.d]. Its neuron function applies the activation
over the samples for one unit, so as written the softmax normalises across the
batch [arxiv:2501.03256 tab. C]. The paper's own softmax listing also
does not match its formula [arxiv:2501.03256 tab. B]. The paper does not say
which behaviour it intends, so pick one and test it.

The two forecasting MLPs compare with PatchTST under different setups. TiDE
takes PatchTST/64 numbers from the PatchTST paper, corrected by the TiDE
authors for a data-loader bug [arxiv:2304.08424 §5.1] [arxiv:2304.08424 tab. 2].
TimeMixer re-runs its baselines under one common setting, so its baseline
numbers differ from the numbers in the baselines' own papers
[arxiv:2405.14616 §4]. Their PatchTST figures and win counts are therefore
not directly comparable.

This is a gap rather than a conflict. The scaling benchmark finds the MLP
strongly affected by the scaler [arxiv:2506.08274 §5.1]. The TabTransformer
paper does not state how it scales its numeric features, and only says that
rescaling options were discussed [arxiv:2012.06678 app. B.1].

The TopoMLP paper's loss weights for its lane regression term (0.2) and its
lane-traffic topology term (0.5) differ from the values in its released
configuration (0.02 and 5) [arxiv:2310.06753 §4.2]. The paper's text also
calls the lane-traffic weight "L_top_ll", a label that appears to be a typo
[arxiv:2310.06753 §4.2]. A reimplementation has to pick one of these values
and state which one [arxiv:2310.06753 §4.2].

The rover paper disagrees with its own figure on depth. The text describes one
hidden layer of 64 units [arxiv:2411.04700 §3.2], and the figure shows two
[arxiv:2411.04700 fig. 8]. Its text also says that adding the IMU to the
force-torque features gains the network about 0.1 points, while the tables
give 0.59 [arxiv:2411.04700 §3.2] [arxiv:2411.04700 tab. 6]
[arxiv:2411.04700 tab. 7].

## Papers

- [Why Deep Neural Networks for Function Approximation?](/papers/1610.04161): depth-versus-size bounds for ReLU and step networks approximating smooth functions. Hand-built constructions and lower bounds, no experiments.
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](/papers/1701.02490): a two-input tanh MLP that fits value differences for an impression-level bidding policy, with author-reported click gains over a linear bidder.
- [Threat analysis of IoT networks Using Artificial Neural Network Intrusion Detection System](/papers/1704.02286): a three-layer sigmoid MLP trained by gradient descent to separate normal from DDoS/DoS traffic on a simulated five-node IoT network.
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678): Transformer layers over categorical column embeddings with an MLP head, with masked or replaced-value pretraining. Author-reported gains over an MLP without the Transformer layers on binary tabular datasets.
- [Deep Parametric Continuous Convolutional Neural Networks](/papers/2101.06742): a convolution whose kernel is an MLP of relative 3D position, for points not on a grid. Author-reported gains on indoor segmentation and combined driving segmentation, and lower lidar flow error.
- [KAN: Kolmogorov-Arnold Networks](/papers/2404.19756): replaces linear weights and fixed activations with learnable spline functions on edges. Author-reported: smaller models than matched MLPs on small function-fitting and PDE tasks.
- [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](/papers/2405.14616): an MLP-only forecaster that mixes seasonal and trend parts across downsampled scales. Author-reported best long-term accuracy on eight benchmarks.
- [Long-term Forecasting with TiDE: Time-series Dense Encoder](/papers/2304.08424): an MLP encoder-decoder with residual blocks and a linear shortcut for long-horizon forecasting. Author-reported: lower Traffic MSE than PatchTST/64 and faster than the best Transformer.
- [Dive into Deep Learning](/papers/2106.11342): a textbook with runnable code whose MLP chapter teaches the model from the basics. Text is CC BY-SA 4.0; it reports no results of its own.
- [The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](/papers/2506.08274): a benchmark of 12 feature scalers plus no scaling across 14 algorithms, including an MLP, on 16 UCI tabular datasets. Author-reported: MLP accuracy and R² are sensitive to the scaler.
- [Deep Isolation Forest for Anomaly Detection](/papers/2206.06602): untrained random MLP maps, with isolation trees grown on each map. Author-reported AUC-PR gains on ten tabular sets.
- [Representing Random Utility Choice Models with Neural Networks](/papers/2207.12877): a feed-forward utility network averaged over sampled attributes, trained by negative log-likelihood. Author-reported accuracy gains on two choice datasets.
- [Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting](/papers/2208.05233): a graph-free residual MLP with learned identity embeddings. Author-reported lowest average MAE on five benchmarks.
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](/papers/2303.06053): time-mixing and feature-mixing MLP blocks. Author-reported comparable to PatchTST on long-term benchmarks, with an M5 extension.
- [TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning](/papers/2310.06753): query-based lane detector with two pairwise MLP link heads for lane and traffic topology. Author-reported OLS 38.2 on OpenLane-V2 subset A.
- [Frequency-domain MLPs are More Effective Learners in Time Series Forecasting](/papers/2311.06184): complex-valued MLPs on a discrete Fourier transform of the input. Author-reported gains on short-term and long-term benchmarks.
- [Recognition of Urbanized Areas in UAV-Derived Very-High-Resolution Visible-Light Imagery](/papers/2609.40212): a small tanh MLP on RGB patches, compared with vegetation-index thresholds, a linear network and a CNN. Author-reported MCC on a second site.
- [Product-based Neural Networks for User Response Prediction](/papers/1611.00144): an MLP over learned field embeddings with a product layer of pairwise inner or outer products, for click prediction. Author-reported AUC above every baseline in its tables on Criteo and iPinYou.
- [AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems](/papers/2501.03256): dense networks trained in Keras and rerun in hand-written MicroPython on a Raspberry Pi Pico. Author-reported accuracy only, 90% and 95% on two small Iris nets.
- [Field Assessment of Force Torque Sensors for Planetary Rover Navigation](/papers/2411.04700): a dropout-regularised MLP and an SVM classify four terrain types from force-torque and IMU statistics. Author-reported test accuracy 96.17% with force-torque data, against 79.35% with IMU only.
- [Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications](/papers/2412.07823): a batch-normalised three-layer MLP estimates hip joint moment after picking 8 of 20 tasks by clustering. Author-reported leave-one-subject-out RMSE 0.30 Nm/kg, against 0.28 for all tasks.