Deep learning

Overview

Deep learning is a family of machine learning built on neural networks with many layers. Its defining feature is that the layers learn the features from data, rather than relying on features a person designed 2104.05314 p. 5. The review places it in a chain of subsets: AI contains machine learning, which contains neural networks, which contain deep neural networks [arxiv:2104.05314 p. 3]. The line between shallow and deep models is a convention the review does not sharply define 2104.05314 p. 4.

The review's motivation is data. Electronic markets produce much unstructured data, such as time series, images and text. Finding patterns in it by hand would exceed human capacity, so the authors argue for automated model building 2104.05314 p. 4. Every model is built in four aspects: data input, feature extraction, model building and model assessment [arxiv:2104.05314 p. 4].

This page rests on six cards. The first is the review above, which is conceptual 2104.05314 p. 1. The second is a 2018 survey of the main families, with their core equations and training steps 1803.01164 §I. The third is a 2017 position paper that proposes an alternative to a single large network 1706.01040 §3. The fourth is an open textbook that teaches the field with runnable code. It introduces no new method 2106.11342 §1. The fifth is a PyTorch library for training deep models 2002.04688 §1. The sixth is a working paper on running a small trained network on a microcontroller, reporting accuracy only 2501.03256 §IV. All six are covered under Variants.

Variants

Modelling paradigms

The review compares three ways to build an analytical model, using the four aspects above 2104.05314 fig. 2 2104.05314 p. 4–5:

• Explicit programming. A person writes the rules. There is no learning step 2104.05314 p. 4.
• Shallow machine learning. A person designs the features, and a learning algorithm fits the model on top of them. The review names decision trees and support vector machines as examples 2104.05314 p. 5.
• Deep learning. Feature learning and model building happen in one pipeline. The features are learned hierarchically, from simple parts to complex ones 2104.05314 p. 5.

Architectures by data type

The review groups the main architectures by the data they suit and the mechanism they use 2104.05314 tab. 2 2104.05314 p. 6:

• Convolutional networks suit images, speech and data with a spatial layout. Stacked layers learn edges, then parts, then objects. See convolutional neural network.
• Recurrent networks suit sequences such as time series, event logs and language. Internal feedback keeps a memory. The review notes vanishing gradients in simple recurrent networks and names LSTMs as a more sophisticated variant 2104.05314 tab. 2. See recurrent neural network.
• Distributed representations suit language. Words or phrases become dense vectors, and similar contexts sit close together 2104.05314 tab. 2.
• Autoencoders suit any data, for dimensionality reduction and anomaly detection. The input is encoded to a compressed code and decoded back. A high reconstruction error flags an unusual case 2104.05314 tab. 2.
• Generative adversarial networks generate new samples. A generator and a discriminator train against each other 2104.05314 tab. 2.

The textbook gives a chapter to most of these families, each with runnable code. Convolutional networks, including modern variants with batch normalization and residual connections, are in §7–8. Recurrent networks, including LSTMs and GRUs, are in §9–10. Attention and transformers are in §11. Its chapters on optimization (§12) and computational performance (§13) cover the training side 2106.11342 §7–§13. Its computer vision (§14) and language (§15–16) chapters apply these models to images and text, and §17 covers reinforcement learning 2106.11342 §14–§17. See attention mechanism for the attention family.

Survey of families and training steps

The 2018 survey covers feedforward networks, convolutional networks, recurrent networks, autoencoders, restricted Boltzmann machines, GANs and deep Q-learning. For each it gives the basic formulas and a training recipe 1803.01164 §II–§VIII. Its training chapter covers weight initialization, batch normalization, activation functions, pooling, dropout and optimizers 1803.01164 §IV. Batch normalization normalizes each feature over the mini-batch, then applies a learned scale and shift 1803.01164 alg. IV.

Its suggested defaults are loose. It gives a decay factor of 0.1 per stage and a weight decay of 0.0004, and cites no source for the weight decay 1803.01164 §II.F §II.G. Treat both as starting points to tune, not as tested settings. The survey runs no experiments of its own. Its accuracy figures are copied from cited works 1803.01164 tab. II. See the paper card for the full list of equations.

Training with a layered library: fastai

The fastai paper describes a PyTorch library with three layers: one-call functions for vision, text, tabular and collaborative filtering at the top, building blocks in the middle, and transforms at the bottom. The top layer chooses defaults from the data and the model 2002.04688 §1 §2. Its training pieces are:

• A Learner that bundles model, optimizer, loss, data and metrics. A splitter divides the model into parameter groups, which can be frozen or given their own learning rates 2002.04688 §4.1.
• Callbacks that fire at each step of the training loop. A callback can read or change the data, predictions, loss, gradients or learning rate, or skip a batch or epoch 2002.04688 §4.2.
• A one-cycle schedule. The learning rate warms up while momentum falls, then anneals while momentum rises 2002.04688 §3.1 fig. 8. The paper gives no numeric values for these phases 2002.04688 §3.1.
• A learning-rate finder. It runs a short trial with a rate that grows exponentially, then suggests a starting value from the loss curve 2002.04688 §3.1 fig. 9. The caption suggests the rate at the minimum loss, divided by 10. The paper card records that the current code's default rule differs, so choose the rule explicitly.

Transfer learning is sensitive to one detail. The authors report that batch-normalization handling changed the error rate by up to 300% across their configurations, and that one approach worked on all their datasets 2002.04688 §4.1. This is an observation in the text. No table reports it.

Proposed alternative: Brain Intelligence (position only)

The position paper Brain Intelligence: Go Beyond Artificial Intelligence argues that a single large network is too narrow. It proposes many small neural sub-networks, linked so that they share parameters and structure. Each sub-network would be updated by an S-system of reproduction, selection and mutation. The design adds a memory and an "imagine" function [arxiv:1706.01040 §3]. It is compared with NEAT-style evolutionary methods 1706.01040 §3. See neuroevolution for that family.

The paper gives no equations, no algorithm and no hyperparameter values. The S-system and the imagine function are named but not defined [arxiv:1706.01040 §3]. Its claim that this design would solve four limits of current AI is not tested 1706.01040 §1 §4. Treat it as a research direction, not as a method to implement.

Small networks on microcontrollers: AI-ANNE

The AI-ANNE paper runs a trained dense network on a Raspberry Pi Pico microcontroller, which has 264 KB of memory 2501.03256 §II. Its workflow pre-trains the network off the device in TensorFlow and Keras, exports each layer's weights and biases, and rebuilds the forward pass in MicroPython 2501.03256 §IV tab. 2 tab. 4. The paper calls training on the device a complex challenge, so it keeps training off the device 2501.03256 §I. The forward pass is written as plain loops over neurons, layers and activation functions 2501.03256 app. A app. C.

Its two examples are a 2-3-2-1 network at 90% accuracy and a 3-2-1 network at 95%, both on a binary Iris task with 20 test samples 2501.03256 §IV tab. 1. These are the only results. The paper reports no latency, memory or energy measurements, though it frames the approach as low-cost [arxiv:2501.03256 §IV]. Read the claim as the authors' framing, not a measured result.

The paper's tables store weights as inputs by units, but its dense function indexes them per unit 2501.03256 tab. 2 app. A. Its softmax text normalises over classes, while the code path applies it over samples 2501.03256 §III.d. The public code also uses different activations from the paper. See the paper card before implementing.

When to use which

The review gives qualitative guidance. The rows below only state what it says.

Implementation sketch

The review gives no training procedure, loss or architecture. The sketch below restates its three points: pick the architecture by data type, let the network learn features, and assess the model on more than accuracy. It is not an algorithm from the paper.

# Build and assess a deep model  [arxiv:2104.05314 p. 4–7]
# Input:  raw data, labels for supervised tasks
# Output: a trained network and its assessment

def build_deep_model(raw_data, labels):
    # Pick an architecture by data type  [tab. 2]
    model = new_network(architecture_for(data_type(raw_data)))
    # The network learns features and the model together  [p. 5]
    model.fit(raw_data, labels)
    return model

def assess(model, data, labels, k):
    # k-fold cross-validation guards against overfitting  [p. 5]
    scores = k_fold_scores(model, data, labels, k)
    # Report performance, cost and interpretability together  [p. 5–7]
    return {
        "metrics": scores,                    # e.g. F1, RMSE  [p. 5–6]
        "memory_and_inference_time": cost(model),
        "interpretability": explain_notes(model),
    }

Contradictions

No direct conflict on a measured point. The two cards make different kinds of claim. The review says deep networks beat shallow ones on many image, text, video and speech tasks, and that shallow models can be better on low-dimensional data with little training data. It takes both points from other work [arxiv:2104.05314 p. 1] p. 4. The position paper says current systems depend heavily on big data, and that training on it takes long 1706.01040 §1 [abstract]. These fit together, but neither paper measures the data-size trade-off.

Inside the review, interpretability is stated loosely. It says some shallow models are inherently interpretable, and that most advanced models are opaque unless explained. It also says shallow models can be more interpretable than deep networks 2104.05314 p. 4. None of these claims is measured.

The textbook reports no benchmark numbers of its own. It therefore does not contradict the other two cards on results 2106.11342 §1. Its Kaggle-style chapters are walkthroughs, and their outcomes are not reported as findings 2106.11342 §14.

The library paper's results are author-reported. Its headline timing, ResNet-50 on ImageNet in 18 minutes, has no target accuracy stated 2002.04688 §1. Its batch-norm observation is untabulated 2002.04688 §4.1. Its learning-rate caption and its code's default rule differ, which the paper card records 2002.04688 fig. 9.

The AI-ANNE paper also disagrees with itself on softmax. Its text defines it over classes, and its code applies it over samples for each unit 2501.03256 §III.d tab. C. Its code also differs from its tables on activations 2501.03256 tab. 3 tab. 5. These are paper-versus-code points, and the paper card lists the code lines.

The survey and the review agree that features are learned from data 1803.01164 §I 2104.05314 p. 5. The survey also disagrees with itself on some numbers. Its text gives GoogLeNet 1.53G MACs, and its table gives 1.43G 1803.01164 §III.B.6 tab. II. Its text gives AlexNet three convolution layers, and its table gives five 1803.01164 §III.B.2 tab. II. Check the original paper before using either value.

Papers

• Machine learning and deep learning: a conceptual review that defines AI, machine learning and deep learning, contrasts three modelling paradigms, and lists deployment challenges. No experiments.
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches: a 2018 survey of feedforward, convolutional, recurrent, autoencoder, RBM, GAN and deep Q-learning models, with core equations and training steps. No new method and no code.
• Brain Intelligence: Go Beyond Artificial Intelligence: a 2017 position paper proposing networks of small linked sub-networks with memory and imagination. No equations, algorithm or experiments.
• Dive into Deep Learning: an open textbook from linear algebra and probability through CNNs, RNNs, transformers, optimization and reinforcement learning. Each concept has runnable code. No benchmark results of its own.
• fastai: A Layered API for Deep Learning: a PyTorch library with one-call training, a data block API, callbacks and a one-cycle schedule. Results are author-reported; there are no benchmark tables.
• AI-ANNE: (A) (N)eural (N)et for (E)xploration: a working paper that runs pre-trained dense networks in MicroPython on a Raspberry Pi Pico. Two Iris examples, accuracy only; no latency, memory or energy numbers.