---
kind: concept
slug: deep-learning
title: Deep learning
summary: >-
  Machine learning with neural networks of many stacked layers, which learn features from raw data instead of hand-designed ones. Covers three modelling paradigms, architectures by data type, deployment on microcontrollers, and when to prefer shallow models or transfer learning. Six sources; the library and microcontroller papers report their own experiments.
tags: [fundamentals, deep-learning, model-assessment]
papers: ["2104.05314", "1803.01164", "1706.01040", "2106.11342", "2002.04688", "2501.03256"]
related: [machine-learning, multi-layer-perceptron, transfer-learning, explainable-ai, convolutional-neural-network, recurrent-neural-network, attention-mechanism, neuroevolution, learning-rate-schedule, data-augmentation]
status: draft
updated_at: 2026-10-25
---

## Overview

Deep learning is a family of machine learning built on neural networks with
many layers. Its defining feature is that the layers learn the features from
data, rather than relying on features a person designed [arxiv:2104.05314 p. 5].
The review places it in a chain of subsets: AI contains machine learning, which
contains neural networks, which contain deep neural networks [arxiv:2104.05314
p. 3]. The line between shallow and deep models is a convention the review does not
sharply define [arxiv:2104.05314 p. 4].

The review's motivation is data. Electronic markets produce much unstructured
data, such as time series, images and text. Finding patterns in it by hand
would exceed human capacity, so the authors argue for automated model building
[arxiv:2104.05314 p. 4]. Every model is built in four aspects: data input,
feature extraction, model building and model assessment [arxiv:2104.05314
p. 4].

This page rests on six cards. The first is the review above, which is
conceptual [arxiv:2104.05314 p. 1]. The second is a 2018 survey of the main
families, with their core equations and training steps [arxiv:1803.01164 §I].
The third is a 2017 position paper that proposes an alternative to a single
large network [arxiv:1706.01040 §3]. The fourth is an open textbook that
teaches the field with runnable code. It introduces no new method
[arxiv:2106.11342 §1]. The fifth is a PyTorch library for training deep models
[arxiv:2002.04688 §1]. The sixth is a working paper on running a small trained
network on a microcontroller, reporting accuracy only [arxiv:2501.03256 §IV].
All six are covered under Variants.

## Variants

### Modelling paradigms

The review compares three ways to build an analytical model, using the four
aspects above [arxiv:2104.05314 fig. 2] [arxiv:2104.05314 p. 4–5]:

- **Explicit programming.** A person writes the rules. There is no learning
  step [arxiv:2104.05314 p. 4].
- **Shallow machine learning.** A person designs the features, and a learning
  algorithm fits the model on top of them. The review names decision trees and
  support vector machines as examples [arxiv:2104.05314 p. 5].
- **Deep learning.** Feature learning and model building happen in one
  pipeline. The features are learned hierarchically, from simple parts to
  complex ones [arxiv:2104.05314 p. 5].

### Architectures by data type

The review groups the main architectures by the data they suit and the
mechanism they use [arxiv:2104.05314 tab. 2] [arxiv:2104.05314 p. 6]:

- **Convolutional networks** suit images, speech and data with a spatial
  layout. Stacked layers learn edges, then parts, then objects. See
  [convolutional neural network](/concepts/convolutional-neural-network).
- **Recurrent networks** suit sequences such as time series, event logs and
  language. Internal feedback keeps a memory. The review notes vanishing
  gradients in simple recurrent networks and names LSTMs as a more
  sophisticated variant [arxiv:2104.05314 tab. 2]. See
  [recurrent neural network](/concepts/recurrent-neural-network).
- **Distributed representations** suit language. Words or phrases become dense
  vectors, and similar contexts sit close together [arxiv:2104.05314 tab. 2].
- **Autoencoders** suit any data, for dimensionality reduction and anomaly
  detection. The input is encoded to a compressed code and decoded back. A high
  reconstruction error flags an unusual case [arxiv:2104.05314 tab. 2].
- **Generative adversarial networks** generate new samples. A generator and a
  discriminator train against each other [arxiv:2104.05314 tab. 2].

The textbook gives a chapter to most of these families, each with runnable
code. Convolutional networks, including modern variants with batch
normalization and residual connections, are in §7–8. Recurrent networks,
including LSTMs and GRUs, are in §9–10. Attention and transformers are in §11.
Its chapters on optimization (§12) and computational performance (§13) cover
the training side [arxiv:2106.11342 §7–§13]. Its computer vision (§14) and
language (§15–16) chapters apply these models to images and text, and §17
covers reinforcement learning [arxiv:2106.11342 §14–§17]. See
[attention mechanism](/concepts/attention-mechanism) for the attention family.

### Survey of families and training steps

The 2018 survey covers feedforward networks, convolutional networks, recurrent
networks, autoencoders, restricted Boltzmann machines, GANs and deep
Q-learning. For each it gives the basic formulas and a training recipe
[arxiv:1803.01164 §II–§VIII]. Its training chapter covers weight
initialization, batch normalization, activation functions, pooling, dropout
and optimizers [arxiv:1803.01164 §IV]. Batch normalization normalizes each
feature over the mini-batch, then applies a learned scale and shift
[arxiv:1803.01164 alg. IV].

Its suggested defaults are loose. It gives a decay factor of 0.1 per stage and
a weight decay of 0.0004, and cites no source for the weight decay
[arxiv:1803.01164 §II.F] [§II.G]. Treat both as starting points to tune, not
as tested settings. The survey runs no experiments of its own. Its accuracy
figures are copied from cited works [arxiv:1803.01164 tab. II]. See the
[paper card](/papers/1803.01164) for the full list of equations.

### Training with a layered library: fastai

The fastai paper describes a PyTorch library with three layers: one-call
functions for vision, text, tabular and collaborative filtering at the top,
building blocks in the middle, and transforms at the bottom. The top layer
chooses defaults from the data and the model [arxiv:2002.04688 §1] [§2].
Its training pieces are:

- **A `Learner`** that bundles model, optimizer, loss, data and metrics. A
  splitter divides the model into parameter groups, which can be frozen or
  given their own learning rates [arxiv:2002.04688 §4.1].
- **Callbacks** that fire at each step of the training loop. A callback can
  read or change the data, predictions, loss, gradients or learning rate, or
  skip a batch or epoch [arxiv:2002.04688 §4.2].
- **A one-cycle schedule.** The learning rate warms up while momentum falls,
  then anneals while momentum rises [arxiv:2002.04688 §3.1] [fig. 8]. The paper
  gives no numeric values for these phases [arxiv:2002.04688 §3.1].
- **A learning-rate finder.** It runs a short trial with a rate that grows
  exponentially, then suggests a starting value from the loss curve
  [arxiv:2002.04688 §3.1] [fig. 9]. The caption suggests the rate at the
  minimum loss, divided by 10. The [paper card](/papers/2002.04688) records
  that the current code's default rule differs, so choose the rule explicitly.

Transfer learning is sensitive to one detail. The authors report that
batch-normalization handling changed the error rate by up to 300% across
their configurations, and that one approach worked on all their datasets
[arxiv:2002.04688 §4.1]. This is an observation in the text. No table reports
it.

### Proposed alternative: Brain Intelligence (position only)

The position paper [Brain Intelligence: Go Beyond Artificial Intelligence](/papers/1706.01040)
argues that a single large network is too narrow. It proposes many small neural
sub-networks, linked so that they share parameters and structure. Each
sub-network would be updated by an S-system of reproduction, selection and
mutation. The design adds a memory and an "imagine" function [arxiv:1706.01040
§3]. It is compared with NEAT-style evolutionary methods [arxiv:1706.01040 §3].
See [neuroevolution](/concepts/neuroevolution) for that family.

The paper gives no equations, no algorithm and no hyperparameter values. The
S-system and the imagine function are named but not defined [arxiv:1706.01040
§3]. Its claim that this design would solve four limits of current AI is not
tested [arxiv:1706.01040 §1] [§4]. Treat it as a research direction, not as a
method to implement.

### Small networks on microcontrollers: AI-ANNE

The AI-ANNE paper runs a trained dense network on a Raspberry Pi Pico
microcontroller, which has 264 KB of memory [arxiv:2501.03256 §II]. Its
workflow pre-trains the network off the device in TensorFlow and Keras, exports
each layer's weights and biases, and rebuilds the forward pass in MicroPython
[arxiv:2501.03256 §IV] [tab. 2] [tab. 4]. The paper calls training on the device
a complex challenge, so it keeps training off the device [arxiv:2501.03256 §I].
The forward pass is written as plain loops over neurons, layers and activation
functions [arxiv:2501.03256 app. A] [app. C].

Its two examples are a 2-3-2-1 network at 90% accuracy and a 3-2-1 network at
95%, both on a binary Iris task with 20 test samples [arxiv:2501.03256 §IV]
[tab. 1]. These are the only results. The paper reports no latency, memory or
energy measurements, though it frames the approach as low-cost [arxiv:2501.03256
§IV]. Read the claim as the authors' framing, not a measured result.

The paper's tables store weights as inputs by units, but its `dense` function
indexes them per unit [arxiv:2501.03256 tab. 2] [app. A]. Its softmax text
normalises over classes, while the code path applies it over samples
[arxiv:2501.03256 §III.d]. The public code also uses different activations from
the paper. See the [paper card](/papers/2501.03256) before implementing.

## When to use which

The review gives qualitative guidance. The rows below only state what it says.

| Situation | Use | Why |
|---|---|---|
| Mostly unstructured data: images, text, time series, signals | Deep learning, with an architecture matched to the data type [arxiv:2104.05314 tab. 2] | Manual feature extraction is slow, expertise-heavy and inflexible [arxiv:2104.05314 p. 5] |
| Low-dimensional data with little training data | Shallow machine learning | The review reports that shallow models can do better here, citing the literature [arxiv:2104.05314 p. 1] [arxiv:2104.05314 p. 4] |
| Reusing a pre-trained model | Transfer learning, with its risks checked; see [transfer learning](/concepts/transfer-learning) | The review lists transfer learning under resource limits, and names backdoors in pre-trained models as a risk [arxiv:2104.05314 p. 8–9] |
| High-stakes decisions that must be explained | An interpretable white-box model, or an explanation method; see [explainable AI](/concepts/explainable-ai) | The review recommends this for high-stakes systems. It does not test the choice [arxiv:2104.05314 p. 8] |
| Any model choice | Compare accuracy with memory, speed and cost together | Accuracy alone can mislead. A 99.9% model that is too slow for a time-critical task is useless [arxiv:2104.05314 p. 7] |
| Building from many small, self-updating sub-networks | No method to use yet. Read [Brain Intelligence](/papers/1706.01040) as a position | The paper gives no architecture, update rule, training procedure or results [arxiv:1706.01040 §3] [§4] |
| Training a standard model on PyTorch with few lines and sensible defaults | The fastai library, with a pinned version. See [fastai](/papers/2002.04688) | Its high-level functions pick defaults from the data and model [arxiv:2002.04688 §1] [§2]. The paper describes the pre-release v2.0.0, and names and defaults have drifted since [arxiv:2002.04688 §1] |
| Fine-tuning a pre-trained model on a small dataset | Transfer learning, with parameter groups and the batch-norm policy set on purpose | The authors report large error changes from batch-norm handling, from observation only [arxiv:2002.04688 §4.1] |
| Needing the core formulas of several families in one place | The [survey](/papers/1803.01164), then the original papers it cites for implementation | It collects equations and training steps. Its figures are copied from cited works, and it contradicts itself on some numbers [arxiv:1803.01164 tab. II] [§III.B.6] |
| Running a small dense network on a microcontroller with no ML runtime | Pre-train off the device, export weights, and write the forward pass by hand, as [AI-ANNE](/papers/2501.03256) does. Check the activations against the paper, not the code | The paper reports accuracy on two small Iris nets only, with no latency or memory numbers [arxiv:2501.03256 §IV]. It treats on-device training as a complex challenge [arxiv:2501.03256 §I] |
| Learning a standard model family with working code beside the theory | The [Dive into Deep Learning](/papers/2106.11342) textbook. Use its notebook at the pinned code commit | The book is built around runnable notebooks for several frameworks [arxiv:2106.11342 §1]. Its text is CC BY-SA and its PDF is older than its code [arxiv:2106.11342 §1] |

## Implementation sketch

The review gives no training procedure, loss or architecture. The sketch below
restates its three points: pick the architecture by data type, let the network
learn features, and assess the model on more than accuracy. It is not an
algorithm from the paper.

```pseudocode
# Build and assess a deep model  [arxiv:2104.05314 p. 4–7]
# Input:  raw data, labels for supervised tasks
# Output: a trained network and its assessment

def build_deep_model(raw_data, labels):
    # Pick an architecture by data type  [tab. 2]
    model = new_network(architecture_for(data_type(raw_data)))
    # The network learns features and the model together  [p. 5]
    model.fit(raw_data, labels)
    return model

def assess(model, data, labels, k):
    # k-fold cross-validation guards against overfitting  [p. 5]
    scores = k_fold_scores(model, data, labels, k)
    # Report performance, cost and interpretability together  [p. 5–7]
    return {
        "metrics": scores,                    # e.g. F1, RMSE  [p. 5–6]
        "memory_and_inference_time": cost(model),
        "interpretability": explain_notes(model),
    }
```

## Contradictions

No direct conflict on a measured point. The two cards make different kinds of
claim. The review says deep networks beat shallow ones on many image, text,
video and speech tasks, and that shallow models can be better on low-dimensional
data with little training data. It takes both points from other work [arxiv:2104.05314
p. 1] [p. 4]. The position paper says current systems depend heavily on big data,
and that training on it takes long [arxiv:1706.01040 §1] [abstract]. These fit
together, but neither paper measures the data-size trade-off.

Inside the review, interpretability is stated loosely. It says some shallow
models are inherently interpretable, and that most advanced models are opaque
unless explained. It also says shallow models can be more interpretable than
deep networks [arxiv:2104.05314 p. 4]. None of these claims is measured.

The textbook reports no benchmark numbers of its own. It therefore does not
contradict the other two cards on results [arxiv:2106.11342 §1]. Its
Kaggle-style chapters are walkthroughs, and their outcomes are not reported
as findings [arxiv:2106.11342 §14].

The library paper's results are author-reported. Its headline timing,
ResNet-50 on ImageNet in 18 minutes, has no target accuracy stated
[arxiv:2002.04688 §1]. Its batch-norm observation is untabulated
[arxiv:2002.04688 §4.1]. Its learning-rate caption and its code's default
rule differ, which the paper card records [arxiv:2002.04688 fig. 9].

The AI-ANNE paper also disagrees with itself on softmax. Its text defines it
over classes, and its code applies it over samples for each unit
[arxiv:2501.03256 §III.d] [tab. C]. Its code also differs from its tables on
activations [arxiv:2501.03256 tab. 3] [tab. 5]. These are paper-versus-code
points, and the [paper card](/papers/2501.03256) lists the code lines.

The survey and the review agree that features are learned from data
[arxiv:1803.01164 §I] [arxiv:2104.05314 p. 5]. The survey also disagrees with
itself on some numbers. Its text gives GoogLeNet 1.53G MACs, and its table
gives 1.43G [arxiv:1803.01164 §III.B.6] [tab. II]. Its text gives AlexNet three
convolution layers, and its table gives five [arxiv:1803.01164 §III.B.2]
[tab. II]. Check the original paper before using either value.

## Papers

- [Machine learning and deep learning](/papers/2104.05314): a conceptual
  review that defines AI, machine learning and deep learning, contrasts three
  modelling paradigms, and lists deployment challenges. No experiments.
- [The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches](/papers/1803.01164):
  a 2018 survey of feedforward, convolutional, recurrent, autoencoder, RBM,
  GAN and deep Q-learning models, with core equations and training steps. No
  new method and no code.
- [Brain Intelligence: Go Beyond Artificial Intelligence](/papers/1706.01040):
  a 2017 position paper proposing networks of small linked sub-networks with
  memory and imagination. No equations, algorithm or experiments.
- [Dive into Deep Learning](/papers/2106.11342): an open textbook from
  linear algebra and probability through CNNs, RNNs, transformers, optimization
  and reinforcement learning. Each concept has runnable code. No benchmark
  results of its own.
- [fastai: A Layered API for Deep Learning](/papers/2002.04688): a PyTorch
  library with one-call training, a data block API, callbacks and a one-cycle
  schedule. Results are author-reported; there are no benchmark tables.
- [AI-ANNE: (A) (N)eural (N)et for (E)xploration](/papers/2501.03256): a
  working paper that runs pre-trained dense networks in MicroPython on a
  Raspberry Pi Pico. Two Iris examples, accuracy only; no latency, memory or
  energy numbers.