Multi-layer perceptron (MLP)

Overview

A multi-layer perceptron (MLP) is a feed-forward network. Each unit takes a weighted sum of the previous layer's outputs, adds a bias and applies an activation. The forward pass is written out layer by layer for a three-layer sigmoid network 1704.02286 eq. 1-4. Training in that paper uses a backward pass that computes error terms layer by layer and steps the weights down the gradient 1704.02286 §III-B 1704.02286 eq. 5-6.

Twenty-one cards use an MLP, or compare it with an alternative. Each bullet gives the role of the MLP in that card:

• a supervised classifier trained by backpropagation on labelled traffic 1704.02286 §III;
• a regressor for value differences inside a bidding dynamic program, with two tanh hidden layers 1701.02490 §4.1 1701.02490 §6.2;
• a hand-built construction in a theory paper that shows what depth-limited ReLU and step networks can represent. It trains nothing 1610.04161 §3 1610.04161 §4;
• an off-the-shelf MLP classifier and regressor in a benchmark of feature scaling on 16 tabular datasets 2506.08274 §4.1 2506.08274 §4.3;
• an MLP prediction head on top of Transformer layers over categorical columns, with a plain MLP as the baseline 2012.06678 §2 2012.06678 tab. 1;
• a comparison target for KANs, which keep the layered shape but replace the linear weights with learnable splines on edges 2404.19756 §2.2 and are measured against MLPs of matched shape 2404.19756 §3.1;
• an MLP-only time-series forecaster that mixes several downsampled scales 2405.14616 §3;
• an MLP encoder-decoder for long-horizon forecasting, built from residual MLP blocks with a linear shortcut 2304.08424 §4;
• a textbook whose chapter on multilayer perceptrons, with runnable code, teaches the model from the basics 2106.11342 §5. It reports no results of its own.
• a random, untrained MLP that reshapes the data before an isolation forest grows its trees on the reshaped copies 2206.06602 §5.1;
• the utility function of a random-utility choice model, a feed-forward net averaged over sampled attributes 2207.12877 §2.2 2207.12877 §2.3;
• a graph-free forecaster whose residual MLP blocks take learned identity embeddings 2208.05233 §3;
• the time-mixing and feature-mixing MLP blocks of an all-MLP forecaster 2303.06053 §4.1;
• two complex-valued MLPs run on a discrete Fourier transform of the input 2311.06184 §3.1;
• a one-hidden-layer tanh classifier on RGB patches, compared with vegetation indices, a linear network and a CNN 2609.40212 §2.2.2;
• two MLP heads in a query-based lane detector that score every pair of lane queries, and every pair of lane and traffic-element queries, as link predictors 2310.06753 §3.3 2310.06753 §3.4;
• a click-prediction network over learned field embeddings, where a product layer of pairwise inner or outer products feeds ReLU layers and a sigmoid output 1611.00144 §3.1 1611.00144 eq. 1.
• a learnable convolution whose kernel is an MLP of the relative 3D offset between points, so it runs on unevenly spaced lidar and depth points 2101.06742 §3.1 2101.06742 §3.3;
• a hand-written dense network in MicroPython on a Raspberry Pi Pico, with weights exported from a Keras model trained off the device 2501.03256 §III 2501.03256 tab. 2;
• a dropout-regularised classifier over statistics of force-torque and IMU windows, naming the terrain class of a planetary rover 2411.04700 §3.2 2411.04700 tab. 6;
• a batch-normalised three-layer regressor on 14 features, estimating hip joint moment after a subset of tasks is chosen by clustering 2412.07823 §II.C 2412.07823 §III.B.

The variants below differ in activation, output, how their weights are obtained, and whether they are an MLP at all. The sketches show the sigmoid and value-difference networks, and the scaling step that comes before the library MLP. The residual MLP block of the forecasting variant is covered under its own heading. The random, choice-model, frequency, pairwise-scorer, product-layer and continuous-kernel variants have headings below too.

Variants

Sigmoid MLP classifier (6-3-1)

One hidden layer of three units, one output unit, and a bias on every unit. Every hidden and output unit uses a sigmoid 1704.02286 §III 1704.02286 fig. 1. The loss is the mean squared error, written per example with a half factor; whether a batch sums or averages it is not stated 1704.02286 eq. 5. The optimiser is stochastic gradient descent. The learning rate, batch size, epochs, initialisation and regularisation are not stated 1704.02286 §III-B.

The data is 2313 training, 496 validation and 496 test samples; the split procedure is not stated 1704.02286 §V 1704.02286 tab. 2. The six input features are not named 1704.02286 fig. 1. The author-reported accuracy is 99.4% on the test split (493 of 496) and 99.4% over all 3305 samples 1704.02286 §V 1704.02286 fig. 3. Card: Threat analysis of IoT networks Using Artificial Neural Network Intrusion Detection System.

Tanh MLP fitted to value differences

Two inputs, the auctions left t and the budget left b. Two tanh hidden layers of 30 and 15 units, and a linear output for the value difference D(t, b) 1701.02490 §4.1 1701.02490 §6.2. The network predicts D(t, b) = V(t, b+1) − V(t, b) rather than V, because V is much larger than the average click-through rate while D is on the scale of that rate 1701.02490 §4.1. It is fitted on the grid {0..T0} × {0..B0} with T0 = 10,000 1701.02490 §6.2. The optimiser, learning rate, batch, initialisation and training length are not stated 1701.02490 §6.2.

Reported value-table error is 0.998 × 10^-6 (RMSE) on iPinYou, measured only for states inside the trained box 1701.02490 tab. 4 1701.02490 §6.2. The headline click gains come from the small-scale setting (T = 1,000), where the paper uses the exact table: +16.67% over the Lin baseline at a budget ratio of 1/8 on iPinYou 1701.02490 tab. 2 1701.02490 §6.1. For large horizons the network results appear only as a figure 1701.02490 §6.2 1701.02490 fig. 9. Card: Real-Time Bidding by Reinforcement Learning in Display Advertising.

Library MLP on scaled tabular features

An off-the-shelf MLP, used as a classifier or a regressor after a feature scaler has been fitted on the training split. The benchmark covers 16 UCI tabular datasets, 9 for classification and 7 for regression, with 12 scalers plus no scaling 2506.08274 §4.1. The paper says every model uses its default settings 2506.08274 §4.3. The card does not record the layer sizes or activation, so they are the library defaults. The authors' code sets only an iteration cap and a seed for the MLP [/papers/2506.08274].

The scaler matters for this model. On Dry Bean, MLP accuracy was 0.2980 with no scaling and 0.9334 with the best of the twelve scalers [arxiv:2506.08274 app. A]. The scaled-versus-unscaled Wilcoxon test is significant for the MLP (reported p = 0.0000), and so is the Friedman test across scalers (p = 0.0006) 2506.08274 app. A. For regression, the R² test for the MLP is significant (Wilcoxon p = 0.0083) 2506.08274 app. A. The authors group the MLP with LR, SVMs and TabNet as strongly affected by the scaler 2506.08274 §5.1.

The results come from one 70/30 split per dataset, one run per cell, with no hyperparameter tuning and no cross-validation 2506.08274 §6. The "best of twelve" figure picks the scaler after seeing the results, so it overstates what one fixed choice would give. Card: The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks.

Deep ReLU and step networks (existence result)

No training is involved. The constructions are hand-built: binary step units read the binary digits of the input, one per layer, and ReLU layers combine them 1610.04161 §3.1. For a smooth univariate function, the paper reaches error eps with depth O(log 1/eps), binary units O(log 1/eps) and ReLUs O((log 1/eps)^2) 1610.04161 thm. 4. A shallow network, with depth below log(1/eps), needs a number of neurons that grows like a power of 1/eps 1610.04161 §1. For strongly convex functions, any network within eps needs at least log2(mu/(16 eps)) neurons 1610.04161 thm. 11. Card: Why Deep Neural Networks for Function Approximation?.

MLP head over contextual column embeddings (TabTransformer)

TabTransformer gives each categorical column its own lookup table. A column identifier is concatenated to each class vector, and no positional encoding is used because the columns have no order 2012.06678 §2. Transformer layers then make each column's vector depend on the other columns of the same row. The contextual vectors are flattened, joined with the numeric features, and passed to an MLP 2012.06678 §2 2012.06678 eq. 1. The default configuration is embedding size 32, six Transformer layers and eight heads 2012.06678 §3 2012.06678 app. B.1. The top MLP's layer sizes are 4·l and then 2·l, where l is the size of its input 2012.06678 app. B.1.

The paper's ablation removes the Transformer layers from the same model. That plain MLP reaches 81.8 ± 0.4 mean AUC on 15 binary datasets, against 82.8 ± 0.4 for TabTransformer and 82.9 ± 0.4 for GBDT 2012.06678 tab. 2. The per-dataset gain from the Transformer layers averages 1.0 points, and is positive on 14 of 15 datasets 2012.06678 tab. 1. These are author-reported results on binary classification only 2012.06678 §3.

For semi-supervised use, the Transformer layers are first trained on unlabelled rows, by replaced-value detection (RTD) or masking (MLM), and then fine-tuned with the head on the labelled rows 2012.06678 §2. On the eight largest datasets at 50, 200 and 500 labels, the RTD variant reaches 66.6, 70.9 and 73.1 mean AUC. The best other baseline, an MLP with entropy regularisation (ER), reaches 65.6, 69.0 and 71.0 2012.06678 tab. 3. On the twelve smaller datasets, ER is best at 50 labels, and RTD is best at 200 and 500 labels 2012.06678 tab. 4. Batch size, epochs and the pretraining learning rate are not stated 2012.06678 app. B.1. Card: TabTransformer: Tabular Data Modeling Using Contextual Embeddings.

Learnable spline edges instead of linear weights (KAN)

A Kolmogorov-Arnold network keeps the layer-by-layer shape but has no weight matrix and no fixed node activation. Each edge carries its own learnable univariate function, a residual SiLU term plus a B-spline term, and each node only sums what arrives on its incoming edges 2404.19756 §2.2. The parameter count is O(N²L(G+k)) against O(N²L) for an MLP of the same depth and width, so the authors expect a much smaller width to suffice 2404.19756 §2.2.

The paper compares against MLPs of matched shape on special functions, Feynman equations and a Poisson PDE 2404.19756 §3.1 2404.19756 §3.2 2404.19756 §3.3 2404.19756 §3.4. The author-reported results are: KANs have lower test error than the MLP in every row of the special-function table 2404.19756 §3.2; on the Feynman equations the two are described as comparable on average 2404.19756 §3.3; and on the PDE a two-layer, width-10 KAN with about 10² parameters reaches about 10^-7 MSE, against 10^-5 for a four-layer, width-100 MLP with about 10^4 parameters. The authors say KAN training was about ten times slower in their implementation 2404.19756 §3.4.

The accuracy and scaling results assume the true KAN shape is known 2404.19756 §3.1. The pruning and symbolic-fitting steps that produce readable formulas need a human to name the candidate functions 2404.19756 §2.5.1 2404.19756 §2.5.2. Card: KAN: Kolmogorov-Arnold Networks.

MLP-only forecaster over several scales (TimeMixer)

TimeMixer uses linear layers and GELU activations only, with no attention 2405.14616 §3. The past input is average-pooled into several scales. At each scale a moving average splits the series into a seasonal part and a trend part 2405.14616 §3.2. Seasonal parts are mixed from fine to coarse and trend parts from coarse to fine, each with a two-layer mixer over time 2405.14616 eq. 4 2405.14616 eq. 5. A linear predictor at each scale then produces a forecast, and the forecasts are summed 2405.14616 §3.3 2405.14616 eq. 6. The width between the two layers of each mixer is not stated 2405.14616 §3.2.

The paper's default setting is M = 3 down-sampling steps for long-term data and M = 1 for short-term data 2405.14616 §4.2 2405.14616 app. D. The authors report the best average MSE and MAE on all eight long-term datasets at input length 96 2405.14616 tab. 2, and the best scores on PEMS 2405.14616 tab. 3 and on the M4 weighted average 2405.14616 tab. 4. Baselines are re-run under one setting, so their numbers differ from the numbers in their own papers 2405.14616 §4. At input length 3072 the authors report 0.016 s per iteration and 1411 MiB for TimeMixer, against 0.094 s and 16119 MiB for PatchTST. The batch size of that table is not stated 2405.14616 app. B 2405.14616 tab. 8.

The paper does not describe normalising the input. The authors' code applies an instance normalisation to each scale and undoes it on the output, and the reported numbers depend on it, as recorded on the card. Card: TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting.

Residual MLP encoder-decoder with a linear shortcut (TiDE)

TiDE is also MLP-only, with no attention, recurrence or convolution 2304.08424 §4. Its basic layer is a residual block with one hidden layer and a ReLU, a fully linear skip, dropout on the output layer only, and layer norm on the sum of the output and the skip 2304.08424 §4. Whether layer norm and dropout are on is set per dataset 2304.08424 tab. 8.

The model runs per series, with weights shared across series. Past values, projected covariates for the look-back and horizon, and static attributes are flattened into one vector and encoded into an embedding 2304.08424 §4.1 2304.08424 eq. 4. A dense decoder maps the embedding to one vector per horizon step, and a temporal decoder combines each step's vector with that step's projected covariates 2304.08424 §4.2. A linear map from the look-back is added to the output, so a purely linear model is a special case of TiDE 2304.08424 §4.2.

The paper's published settings use look-back 720 for all horizons, MSE loss and Adam with default settings 2304.08424 §5.1, and 512 series per batch 2304.08424 app. B.3. The hidden size, depth, decoder width and dropout are tuned per dataset 2304.08424 tab. 8. The number of layers is tuned from {1, 2, 3} with encoder and decoder depth set separately 2304.08424 tab. 7.

The author-reported results are mixed. On Traffic, TiDE has lower MSE than PatchTST/64 at all four horizons, with 0.336, 0.346, 0.355 and 0.386 against 0.360, 0.379, 0.392 and 0.432 2304.08424 tab. 2. The paper says that, apart from Weather, TiDE either beats PatchTST or stays within statistical significance for most horizons 2304.08424 §5.1. The table in the paper also shows PatchTST/64 with lower MSE in several rows, such as ETTh2 at horizon 720 (0.393 against 0.419) 2304.08424 tab. 2.

The efficiency claim is also author-reported: inference about five times faster and training more than ten times faster than the best Transformer 2304.08424 §1. The text gives no numeric timing table; the timing evidence is the Electricity plots, which show an order-of-magnitude gap in inference 2304.08424 §5.3 2304.08424 fig. 2. The paper's theory section bounds a linear autoregressive class, not the full network 2304.08424 app. A.1. Card: Long-term Forecasting with TiDE: Time-series Dense Encoder.

Random, untrained MLP maps before isolation trees (Deep Isolation Forest)

No network is trained. The data passes through r MLPs whose weights are drawn at random and then kept fixed, and an isolation forest is grown on each mapped copy 2206.06602 §5.1 2206.06602 eq. 1. Each map has a representation width d. The paper samples weights from a widely used distribution, and does not name the hidden widths or the activation 2206.06602 §5.1 2206.06602 §5.5.3. The authors argue that the random maps expose anomalies that separate from normal data only in combinations of features 2206.06602 §1.

Author-reported: average AUC-PR 0.351 over ten tabular datasets, against 0.219 for EIF and 0.144 for iForest 2206.06602 tab. 3. The paper's score formula leaves its normaliser undefined. The released code differs from the paper's formula and standardises each mapped copy over the rows being scored, so the card's Paper vs code section is needed before reimplementing. Card: Deep Isolation Forest for Anomaly Detection.

Utility network inside a random-utility choice model (RUMnet)

A feed-forward network is the utility function of a random-utility model. For each choice, the network is evaluated on K² pairs of sampled unobserved product and customer attributes, and each evaluation gives a softmax over the offered products. The softmaxes are averaged 2207.12877 §2.2 2207.12877 §2.3. Training minimises the negative log-likelihood of the observed choice [arxiv:2207.12877 §4.1] 2207.12877 eq. 2. The paper's experiments use depth and width settings of (3,10) and (5,20) 2207.12877 §6.2. Its theory allows any random-utility model to be approximated arbitrarily closely 2207.12877 §3 2207.12877 Proposition 1. The experiments use ELU, where the theory uses ReLU 2207.12877 §6.2 2207.12877 app. E.

Author-reported: the highest accuracy on Swissmetro (0.793) and Expedia (0.425), and the lowest Expedia loss, while a random forest has the lowest Swissmetro loss (0.523 against 0.561) 2207.12877 tab. 1 2207.12877 §6.3. Training is about an order of magnitude slower than a random forest, and about 36 hours on Expedia 2207.12877 app. E. Card: Representing Random Utility Choice Models with Neural Networks.

Residual MLP with learned identity embeddings (STID)

STID has no graph and no recurrence. Each series' P-step history goes through a linear embedding, which is joined to a learned spatial identity for the series and to learned time-of-day and day-of-week identities. The joined vector goes through L residual MLP blocks and a linear head that outputs F steps 2208.05233 §3 2208.05233 eq. 1-4. The paper uses D = 32 and L = 3 for all datasets, with input length 12 for the PEMS sets and 168 for Electricity 2208.05233 §4.1. It trains with MAE 2208.05233 §3 2208.05233 eq. 5.

Author-reported: the lowest average MAE on all five benchmarks among the methods in its table, and the lowest seconds per epoch 2208.05233 tab. 3 2208.05233 tab. 2. The released Electricity configuration uses an input and output length of 336 and a different width and depth, so the table's Electricity column does not follow from the paper's stated setup 2208.05233 §4.1. Card: Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting.

Time-mixing and feature-mixing MLP blocks (TSMixer)

Each block applies a linear layer over time, shared across variates, and then a two-layer MLP over variates, shared across time steps. Each step has a residual connection and a normalization 2303.06053 §4.1. The basic model uses only the history of the target series, and one linear layer maps the input length to the horizon 2303.06053 app. B.3. The long-term benchmarks use input length 512 2303.06053 §5.1.

Author-reported: comparable to PatchTST on seven long-term benchmarks, with an average MSE change of −1.53% against PatchTST 2303.06053 tab. 3. On M5, test WRMSSE is 0.737 without auxiliary features and 0.640 with static and future features, the latter using an extension 2303.06053 tab. 4 2303.06053 §5.2. The release covers only the basic model, so the M5 numbers cannot be reproduced from it, and its normalization and RevIN wiring differ from the paper in ways the card's Paper vs code section lists. Card: TSMixer: An All-MLP Architecture for Time Series Forecasting.

Frequency-domain MLPs over a Fourier transform (FreTS)

The input is passed through a discrete Fourier transform. A frequency-domain MLP with complex weights runs along the channel axis, then along the time axis, and each run is inverted. A two-layer feed-forward head maps the result to the horizon in one step 2311.06184 §3.1 2311.06184 eq. 1-5. The paper writes each frequency layer as a complex matrix product with bias, and leaves its activation unnamed 2311.06184 §3.2 2311.06184 eq. 6.

Author-reported: MAE 0.120 on Solar and 0.011 on Traffic at horizon 12 2311.06184 tab. 1, and the lowest value in its own multi-step tables on METR-LA and COVID-19 2311.06184 tab. 8 2311.06184 tab. 9. On long-term data it is best on most rows of 2311.06184 tab. 2. The released code uses only diagonal frequency weights and never runs the channel learner, so it is not the model the paper describes. Card: Frequency-domain MLPs are More Effective Learners in Time Series Forecasting.

Small tanh MLP on RGB patches (UAV pixel classification)

The MLP has one hidden layer of five tanh units and a two-unit softmax output. It classifies only the centre pixel of an RGB patch, with patch sizes from 1 to 9 px 2609.40212 §2.2.2. It is trained with scaled conjugate gradient and early stopping, and of three networks per structure the one with the best validation MCC is kept 2609.40212 §2.2.2.

Author-reported, on the WIEL test site: the 9-px MLP reaches MCC 0.8121 and accuracy 0.9561, and a linear network with 9 px reaches the highest WIEL MCC of 0.8208. The best vegetation index, Excess Blue, reaches MCC 0.6099 on WIEL 2609.40212 tab. 2 2609.40212 tab. 3. On the JERZ validation part the CNN is best, while on WIEL the linear and MLP networks are slightly better than the CNN 2609.40212 §3.2. Card: Recognition of Urbanized Areas in UAV-Derived Very-High-Resolution Visible-Light Imagery.

Pairwise MLP scorers over query pairs (TopoMLP)

Here the MLPs do not map one feature vector to one output. They score pairs. Two MLPs read each lane query's decoder features: one predicts control-point offsets, the other a class score. Each query is one lane drawn as a Bezier curve 2310.06753 §3.1. Two topology heads then score links. Each embeds the query features, and the lane head also embeds the lane coordinates. Every row query is concatenated with every column query, and a three-layer ReLU MLP gives one logit per ordered pair 2310.06753 §3.3 [arxiv:2310.06753 eq. 3]. The lane-traffic head adds an embedding of the view matrix before the pairing 2310.06753 §3.4 2310.06753 eq. 4.

Author-reported on OpenLane-V2 subset A with ResNet-50: OLS 38.2, and 41.2 with extra YOLOv8 proposals 2310.06753 tab. 1. The topology terms are much lower than the detection terms, at TOP_ll 7.2 and TOP_lt 22.8 [arxiv:2310.06753 tab. 1]. Removing the L1 term or the position embedding costs under one OLS point each 2310.06753 tab. 3. These ablations are single runs with no variance reported, so such gaps are not established. The metric is easy to move: pushing the link scores towards 0 or 1 around 0.5 lifts OLS from 38.2 to 42.2, and the adjusted metric gives 36.3 and 34.5 2310.06753 tab. 4. Card: TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning.

Product layer over field embeddings (PNN)

Each categorical field is one-hot encoded and mapped to its own learned embedding of M = 10 values 1611.00144 §3.1 1611.00144 §4.4. The first hidden layer is a ReLU of three parts: a linear term over the stacked embeddings, a product term over pairs of fields, and a bias 1611.00144 eq. 3. The paper argues that a plain MLP's first layer, which adds its inputs, models interactions between fields poorly 1611.00144 §1. The two variants differ in the product term. IPNN takes inner products, and a squared-norm identity reduces the cost to one value per hidden node 1611.00144 §3.1.1 1611.00144 eq. 11-12. OPNN takes outer products, and superposing all field vectors first gives one M × M matrix shared by all hidden nodes 1611.00144 §3.1.2 1611.00144 eq. 14.

The experiments use three hidden layers in total and a sigmoid output unit, with dropout of 0.5 on the hidden layers 1611.00144 §4.4. The width of each hidden layer is not stated 1611.00144 §3.1. The optimiser is SGD, but its learning rate, batch size and momentum are not stated 1611.00144 §4.2. The released TensorFlow demo uses one 500-unit hidden layer and no dropout, so it is not the model in the paper. Implement from §3.1 and treat the release as a reference Product-based Neural Networks for User Response Prediction.

Author-reported AUC: IPNN reaches 77.79% on Criteo, against 76.71% for CCPM, the best baseline there 1611.00144 tab. 1. OPNN reaches 81.74% on iPinYou, against 76.38% for CCPM 1611.00144 tab. 2. The concatenated PNN* variant is below both IPNN and OPNN on AUC on both datasets 1611.00144 tab. 1 1611.00144 tab. 2. The tables give no error bars or repeat counts 1611.00144 tab. 1.

MLP as a continuous convolution kernel (PCCN)

Here the MLP is not the layer's output map. It is the kernel. A continuous convolution takes the relative position z between an output point and a support point, and an MLP maps z to a weight for each output channel. The layer then sums over the support points within a window, so it needs no grid 2101.06742 §3.1 2101.06742 §3.2. The window is the K nearest points or the points within a radius r 2101.06742 §3.3.

The sum is a Monte-Carlo estimate over sampled points 2101.06742 §3.1. The paper factorises the weights: a linear map mixes the input channels, and the MLP produces only one kernel value per point pair and output channel. This avoids storing a full weight tensor per pair 2101.06742 §3.3. At inference the batch-norm layer is folded into the MLP's last linear layer, which the authors report as a 3× speed-up 2101.06742 §3.3. The paper says residual links between these layers are critical for convergence 2101.06742 §3.2.

The indoor network uses 8 such layers with width 32, and the last layer width 128, with a global max-pooled feature joined to each point before the classifier 2101.06742 §4.1. The driving network uses 16 residual layers 2101.06742 §4.2. Learning rate, batch size, epochs and the neighbour count used in training are not stated 2101.06742 §4.1 2101.06742 §4.2. The paper also notes that a regular grid gives back ordinary convolution, and a Gaussian kernel over position and feature gives a bilateral filter 2101.06742 §3.3. The paper does not test those special cases.

Author-reported: 58.27 mIOU on the indoor scene benchmark, against 48.92 for SEGCloud and 41.09 for PointNet 2101.06742 tab. 1. On the in-house driving data, the PCCN-only model reaches 46.35 mIOU, below 3D-FCN at 49.28, but it is about 8× smaller. Adding it on top of a trained 3D-FCN gives 58.06 mIOU 2101.06742 tab. 2. Lidar flow end-point error is 7.810 cm against 8.161 cm for 3D-FCN 2101.06742 tab. 3. Runtime on a GTX 1080 Ti is 33 ms for the forward pass and 61 ms end to end with a KD-tree neighbour search 2101.06742 §4.2. Card: Deep Parametric Continuous Convolutional Neural Networks.

Dense network written by hand for a microcontroller (AI-ANNE)

The network is trained off the device in TensorFlow and Keras. Its weights and biases are then copied into MicroPython lists on a Raspberry Pi Pico, and the forward pass is rebuilt from hand-written neuron, layer and activation functions, with no ML library on the board 2501.03256 §I 2501.03256 §III 2501.03256 app. A. The paper lists 264 KB of SRAM for the RP2040 and 520 KB for the RP2350 2501.03256 §II.

Two example nets are given, both for Iris, Versicolor against Virginica. The first has 8 neurons in a 2-3-2-1 shape, with ReLU, tanh, softmax and sigmoid layers, and reaches 90% accuracy. The second has 6 neurons in a 3-2-1 shape, with ReLU, sigmoid and sigmoid layers, and reaches 95% accuracy 2501.03256 tab. 3 2501.03256 tab. 5 2501.03256 §IV. Both accuracies come from a test set of 20 samples 2501.03256 tab. 1. The paper reports no latency, memory or energy figures and no baseline. It names AIfES as a comparable on-device framework but gives no comparison numbers 2501.03256 §I. Optimiser, loss, epochs, learning rate and batch size are not stated 2501.03256 §IV. Card: AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems.

Dropout MLP on sensor statistics for terrain classes (rover)

A feed-forward classifier over statistics of one-second windows. Each window gives 225 inputs: 35 per force-torque sensor and 15 per IMU, with five statistics per channel 2411.04700 §3. The text describes one hidden layer of 64 units with dropout, and the figure shows two hidden layers of 64 units 2411.04700 §3.2 2411.04700 fig. 8. The released code uses ReLU hidden units, a log-softmax output and cross-entropy loss, and it picks the hidden size by cross-validation [/papers/2411.04700].

The paper trains for 50 epochs with batch size 32. The optimiser and learning rate are not stated in the paper 2411.04700 §3.2; the code uses Adam at 0.001 [/papers/2411.04700]. The paper places 20% dropout between hidden layers, while the code also drops inputs at 0.1 2411.04700 §3.2 [/papers/2411.04700].

Author-reported test accuracy on one random 25% split: 96.17% with force-torque features, 79.35% with IMU only, and 96.76% with both 2411.04700 tab. 5 2411.04700 tab. 6 2411.04700 tab. 7. The same features in an SVM reach 95.58% 2411.04700 tab. 3. The IMU-only network recalls 28.36% of pebbles, against 88.06% with force-torque features 2411.04700 tab. 5 2411.04700 tab. 6. Card: Field Assessment of Force Torque Sensors for Planetary Rover Navigation.

Batch-normalised FCNN for hip joint moment (leave-one-subject-out)

A fully connected network with three hidden layers of 50 units and one output for hip joint moment in Nm/kg 2412.07823 §II.C 2412.07823 fig. 1. Each hidden layer is linear, then batch normalization, then ReLU, then dropout of 0.2 2412.07823 §II.C. The 14 inputs are not broken down in the text, and input standardisation is not stated 2412.07823 §II.C 2412.07823 fig. 1.

Training uses MSE loss, Adam at learning rate 0.001 and batch size 64, for at most 100 epochs. Training stops after 10 epochs without validation-loss improvement, and which checkpoint is restored is not stated [arxiv:2412.07823 §II.C]. Evaluation is leave-one-subject-out over 11 subjects. For each held-out subject, the rest is split 80/20 into training and validation 2412.07823 §II.C 2412.07823 fig. 4.

Author-reported RMSE, averaged over the folds: 0.28 ± 0.05 Nm/kg with all 20 tasks, 0.30 ± 0.05 with the 8 optimized tasks, and 0.38 ± 0.08 with the 8 cyclic tasks. R² is 0.64, 0.57 and 0.32 in the same order [arxiv:2412.07823 §III.B] 2412.07823 fig. 4. Both the all-task and the optimized-task sets have lower RMSE than cyclic-only, by 27.11 ± 6.18% and 20.60 ± 9.80% (p < 0.05). The paper reports no difference between all tasks and the optimized set, and gives no p-value for it 2412.07823 §III.B. The optimized set is chosen by a clustering score, not by fitting the network 2412.07823 §I 2412.07823 §II.B. Card: Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications.

When to use which

The cards answer different questions, so the constraints below pick between them. Only the cited results support each row.

Implementation sketch

The common core is the forward pass, the error terms and the update. The shapes follow the paper's index convention, with the hidden layer first. The paper's weight-derivative equation was not legible in the text layer, so the weight gradients below follow the chain rule from eq. 5-6 1704.02286 §III-B.

# Forward pass for one example  [arxiv:1704.02286 eq. 1-4]
# x: (6,); w1: (3, 6); b1: (3,); w2: (1, 3); b2: (1,)
def forward(x, w1, b1, w2, b2):
    # Three sigmoid hidden units  [eq. 1-3]
    a_hidden = sigmoid(w1 @ x + b1)            # (3,)
    # One sigmoid output unit  [eq. 4]
    out = sigmoid(w2 @ a_hidden + b2)          # (1,)
    return a_hidden, out

# Error terms for one example  [arxiv:1704.02286 eq. 5-6]
# y: target; its encoding is not stated (see Contradictions)
# f'(z) = out * (1 - out) assumes the logistic sigmoid; the paper
# does not write f' out [§III]
def backward(x, y, a_hidden, out, w2):
    # Output error from the half squared error  [eq. 5]
    d_out = -(y - out) * out * (1 - out)       # (1,)
    # Hidden errors: send the output error back through w2  [eq. 6]
    d_hidden = (w2.T @ d_out) * a_hidden * (1 - a_hidden)   # (3,)
    # Bias gradients equal the error terms  [eq. 8]
    grad_b2 = d_out
    grad_b1 = d_hidden
    # Weight gradients: error times the activation or input feeding it
    grad_w2 = outer(d_out, a_hidden)           # (1, 3)
    grad_w1 = outer(d_hidden, x)               # (3, 6)
    return grad_w1, grad_b1, grad_w2, grad_b2

# Forward pass of the value-difference network  [arxiv:1701.02490 §4.1] [§6.2]
# x: (2,) = (t, b); w1: (30, 2); w2: (15, 30); w3: (1, 15)
# Input scaling and initialisation are not stated in the paper
def forward_value_diff(x, w1, b1, w2, b2, w3, b3):
    # First tanh layer over the two inputs  [§6.2]
    h1 = tanh(w1 @ x + b1)                     # (30,)
    # Second tanh layer  [§6.2]
    h2 = tanh(w2 @ h1 + b2)                    # (15,)
    # Linear output: the value difference D(t, b)  [§4.1]
    return w3 @ h2 + b3                        # (1,)

# Scale tabular features, then train a library MLP  [arxiv:2506.08274 §4.3]
# The split happens before this function is called  [arxiv:2506.08274 §4.2]
def scaled_mlp_fit_predict(X_train, y_train, X_test, scaler, model):
    # Statistics come from the training rows only  [arxiv:2506.08274 §4.3]
    scaler.fit(X_train)
    X_train_s = scaler.transform(X_train)      # (n_train, d)
    X_test_s = scaler.transform(X_test)        # (n_test, d)
    # Defaults apply to every setting not named here  [arxiv:2506.08274 §4.3]
    model.fit(X_train_s, y_train)
    return model.predict(X_test_s)

# Gradient-descent step  [arxiv:1704.02286 §III-B]
# The learning rate is not stated; lr is a free choice here.
def update(params, grads, lr):
    # Move each parameter a small step against its gradient
    return [p - lr * g for p, g in zip(params, grads)]

Where variants differ: the sigmoid classifier has a sigmoid output, the value-difference network a linear one. Its activation is tanh, so the error terms use 1 − tanh² instead of the sigmoid derivative. A second hidden layer, a softmax output or a different loss would change the shapes and error terms. The library variant adds a scaling step before the first layer, and the choice of scaler is a free parameter the benchmark varies 2506.08274 §3.1. The TabTransformer head takes the flattened contextual embeddings and the numeric features, and its layer sizes follow from its input size 2012.06678 app. B.1. The TimeMixer forecaster applies one linear predictor per scale and sums them 2405.14616 eq. 6. The KAN replaces each weight and activation with a spline on an edge, so its forward pass sums edge functions instead of multiplying by a matrix 2404.19756 §2.2. The expressivity paper's networks use ReLU and binary step units and are built by hand, so they have no training code to sketch 1610.04161 §3. The random-map variant has no backward pass: its networks are fixed draws, and only the trees are fitted 2206.06602 §5.1. The choice-model variant takes a softmax over the alternatives for each sampled attribute pair, then averages them 2207.12877 §2.2. TSMixer's blocks mix along time and variates in turn, and STID's blocks are residual MLPs over a joined identity vector, so neither matches the sketches above 2303.06053 §4.1 2208.05233 §3. The frequency variant applies its layer in the Fourier basis, which the time-domain sketch does not show 2311.06184 §3.2. The pixel classifier has the same layered form with tanh hidden units and a two-unit softmax output 2609.40212 §2.2.2. The pairwise scorer of the lane-detection variant is an MLP over concatenated pairs of query features, one logit per pair, so its input is two vectors joined rather than one feature vector 2310.06753 §3.3. The product-layer network's first hidden layer takes the field embeddings and the product term, not the raw input vector 1611.00144 eq. 3, so the sketches above do not show it. The continuous-kernel variant feeds the MLP the offsets between points rather than the features, and the features enter only through a linear map and the weighted sum 2101.06742 §3.1 2101.06742 §3.3. The AI-ANNE layers compute the same weighted sum and activation, but in batch form, with weights stored as inputs by units and transposed before use [arxiv:2501.03256 tab. 2] 2501.03256 app. A. The rover network puts dropout on its inputs and after its hidden layer, and the exoskeleton network puts batch normalization before each ReLU and dropout after it 2411.04700 §3.2 [/papers/2411.04700] 2412.07823 §II.C.

Contradictions

Most cards measure different quantities, so most do not contradict one another directly. The forecasting cards are the exception: they report MSE or MAE on overlapping benchmarks. The expressivity paper shows that small deep networks can represent certain functions 1610.04161 thm. 4. The trained networks report accuracy or error and say nothing about representation bounds 1704.02286 §V 1701.02490 §6.2 2506.08274 §5.1 2012.06678 §3 2206.06602 tab. 3 2609.40212 §3.2.

The KAN paper reports KANs as lower than the MLP in every row of its special-function table 2404.19756 §3.2, but describes the two as comparable on average for the Feynman equations 2404.19756 §3.3. The MLP widths differ between those experiments: 5 or 100 for the special functions and 5 for the Feynman equations 2404.19756 §3.2 2404.19756 §3.3. Its 78% knot-theory MLP baseline is quoted from another group's work and is not reproduced 2404.19756 §4.3.

The trained MLP cards that report numbers measure different things. The sigmoid classifier is a 6-3-1 network on traffic features 1704.02286 §III. The scaling benchmark uses a library MLP at default settings on 16 UCI tables 2506.08274 §4.3. The TabTransformer comparison uses AUC on 15 binary datasets 2012.06678 §3. Their accuracies and AUCs are not comparable. The anomaly card reports AUC-PR on ten tabular sets 2206.06602 tab. 3, the pixel card reports MCC on two UAV sites 2609.40212 tab. 2, and the choice card reports log-likelihood loss and accuracy 2207.12877 tab. 1. The click-prediction card reports AUC and log loss on two ad datasets 1611.00144 tab. 1 1611.00144 tab. 2. None of these is comparable with the others.

The forecasting cards also disagree with themselves. FreTS says it beats all baselines on all long-term datasets, but its own table has PatchTST with lower MAE on Traffic at two horizons and on Electricity at 336 2311.06184 §4.1 2311.06184 tab. 2. TSMixer's M5 table prints 0.000 for three validation entries of its extension, which look like placeholders 2303.06053 tab. 4.

Rankings depend on the metric. In RUMnet's table the random forest has the lowest Swissmetro log-likelihood loss, while RUMnet has the highest accuracy on both datasets 2207.12877 tab. 1 2207.12877 §6.3. The same holds for the product-layer network: on iPinYou, OPNN has the highest AUC while PNN* has the lowest log loss and RMSE 1611.00144 tab. 2.

Within one card, the paper's labels conflict with its counts. The sigmoid classifier's text calls true positives attacks "rightly classified" and false positives normal events "classified rightly" 1704.02286 §V. The confusion matrix shows 20 attack samples labelled normal and no normal sample labelled attack 1704.02286 fig. 3. Use the counts, not the paper's TP and FP labels.

The AI-ANNE paper's text says softmax normalises across the classes of one input 2501.03256 §III.d. Its neuron function applies the activation over the samples for one unit, so as written the softmax normalises across the batch 2501.03256 tab. C. The paper's own softmax listing also does not match its formula 2501.03256 tab. B. The paper does not say which behaviour it intends, so pick one and test it.

The two forecasting MLPs compare with PatchTST under different setups. TiDE takes PatchTST/64 numbers from the PatchTST paper, corrected by the TiDE authors for a data-loader bug 2304.08424 §5.1 2304.08424 tab. 2. TimeMixer re-runs its baselines under one common setting, so its baseline numbers differ from the numbers in the baselines' own papers 2405.14616 §4. Their PatchTST figures and win counts are therefore not directly comparable.

This is a gap rather than a conflict. The scaling benchmark finds the MLP strongly affected by the scaler 2506.08274 §5.1. The TabTransformer paper does not state how it scales its numeric features, and only says that rescaling options were discussed 2012.06678 app. B.1.

The TopoMLP paper's loss weights for its lane regression term (0.2) and its lane-traffic topology term (0.5) differ from the values in its released configuration (0.02 and 5) 2310.06753 §4.2. The paper's text also calls the lane-traffic weight "L_top_ll", a label that appears to be a typo 2310.06753 §4.2. A reimplementation has to pick one of these values and state which one 2310.06753 §4.2.

The rover paper disagrees with its own figure on depth. The text describes one hidden layer of 64 units 2411.04700 §3.2, and the figure shows two 2411.04700 fig. 8. Its text also says that adding the IMU to the force-torque features gains the network about 0.1 points, while the tables give 0.59 2411.04700 §3.2 2411.04700 tab. 6 2411.04700 tab. 7.

Papers

• Why Deep Neural Networks for Function Approximation?: depth-versus-size bounds for ReLU and step networks approximating smooth functions. Hand-built constructions and lower bounds, no experiments.
• Real-Time Bidding by Reinforcement Learning in Display Advertising: a two-input tanh MLP that fits value differences for an impression-level bidding policy, with author-reported click gains over a linear bidder.
• Threat analysis of IoT networks Using Artificial Neural Network Intrusion Detection System: a three-layer sigmoid MLP trained by gradient descent to separate normal from DDoS/DoS traffic on a simulated five-node IoT network.
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings: Transformer layers over categorical column embeddings with an MLP head, with masked or replaced-value pretraining. Author-reported gains over an MLP without the Transformer layers on binary tabular datasets.
• Deep Parametric Continuous Convolutional Neural Networks: a convolution whose kernel is an MLP of relative 3D position, for points not on a grid. Author-reported gains on indoor segmentation and combined driving segmentation, and lower lidar flow error.
• KAN: Kolmogorov-Arnold Networks: replaces linear weights and fixed activations with learnable spline functions on edges. Author-reported: smaller models than matched MLPs on small function-fitting and PDE tasks.
• TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting: an MLP-only forecaster that mixes seasonal and trend parts across downsampled scales. Author-reported best long-term accuracy on eight benchmarks.
• Long-term Forecasting with TiDE: Time-series Dense Encoder: an MLP encoder-decoder with residual blocks and a linear shortcut for long-horizon forecasting. Author-reported: lower Traffic MSE than PatchTST/64 and faster than the best Transformer.
• Dive into Deep Learning: a textbook with runnable code whose MLP chapter teaches the model from the basics. Text is CC BY-SA 4.0; it reports no results of its own.
• The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks: a benchmark of 12 feature scalers plus no scaling across 14 algorithms, including an MLP, on 16 UCI tabular datasets. Author-reported: MLP accuracy and R² are sensitive to the scaler.
• Deep Isolation Forest for Anomaly Detection: untrained random MLP maps, with isolation trees grown on each map. Author-reported AUC-PR gains on ten tabular sets.
• Representing Random Utility Choice Models with Neural Networks: a feed-forward utility network averaged over sampled attributes, trained by negative log-likelihood. Author-reported accuracy gains on two choice datasets.
• Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting: a graph-free residual MLP with learned identity embeddings. Author-reported lowest average MAE on five benchmarks.
• TSMixer: An All-MLP Architecture for Time Series Forecasting: time-mixing and feature-mixing MLP blocks. Author-reported comparable to PatchTST on long-term benchmarks, with an M5 extension.
• TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning: query-based lane detector with two pairwise MLP link heads for lane and traffic topology. Author-reported OLS 38.2 on OpenLane-V2 subset A.
• Frequency-domain MLPs are More Effective Learners in Time Series Forecasting: complex-valued MLPs on a discrete Fourier transform of the input. Author-reported gains on short-term and long-term benchmarks.
• Recognition of Urbanized Areas in UAV-Derived Very-High-Resolution Visible-Light Imagery: a small tanh MLP on RGB patches, compared with vegetation-index thresholds, a linear network and a CNN. Author-reported MCC on a second site.
• Product-based Neural Networks for User Response Prediction: an MLP over learned field embeddings with a product layer of pairwise inner or outer products, for click prediction. Author-reported AUC above every baseline in its tables on Criteo and iPinYou.
• AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems: dense networks trained in Keras and rerun in hand-written MicroPython on a Raspberry Pi Pico. Author-reported accuracy only, 90% and 95% on two small Iris nets.
• Field Assessment of Force Torque Sensors for Planetary Rover Navigation: a dropout-regularised MLP and an SVM classify four terrain types from force-torque and IMU statistics. Author-reported test accuracy 96.17% with force-torque data, against 79.35% with IMU only.
• Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications: a batch-normalised three-layer MLP estimates hip joint moment after picking 8 of 20 tasks by clustering. Author-reported leave-one-subject-out RMSE 0.30 Nm/kg, against 0.28 for all tasks.