Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review

In plain words

Microcontrollers are the tiny, cheap chips inside many sensors and gadgets. They have only a few kilobytes of working memory and, in the most constrained kinds, no hardware for decimal-point arithmetic. Most neural networks are far too large for them. This review looks at how people shrink neural networks and get them to run on such chips. The field is called TinyML.

The review goes through the usual shrinking tricks: training a small model to copy a large one, removing weights that matter little, storing numbers with fewer bits, letting many weights share one value, and replacing a big weight table with the product of two smaller ones. It also looks at the software that puts a model onto a chip. Finally, it compares how small published models are on a few standard benchmarks. The authors train no new models. They compare results that others have already published.

TL;DR

• A survey, not a new method. It has no code and no experiments of its own.
• Target: Arm Cortex-M microcontrollers with under about 8 kB of RAM and near 10 MHz §3.2.
• It recommends integer-only inference and favours uniform 8-bit post-training quantization as the default §3.3 §4.3.
• It compares flash size against accuracy for models from the literature on MNIST, ImageNet, Visual Wake Words and Google Speech Commands v2-12 §6 fig. 9.
• All accuracy and size values are author-reported or quoted from cited works. The comparison plots are not tabulated in the source.

Problem

Deep learning models are often overparameterised, and the review cites work estimating that only about 5% of parameters are critical to the output §2.4. Models that run on phones (around 10^6 parameters) are still too large for microcontrollers, which have well under 1 MB of storage §2.4. Microcontrollers in the review are on-chip only, so they cannot extend memory with external storage §3.3. The Cortex-M0+ has no floating-point support, which forces fixed-point or integer-only arithmetic §3.3 eq. 3. Memory is usually the first bottleneck, and lower bit widths raise the risk of overflow and precision loss §3.3. Tooling is immature and fragmented across hardware vendors §5.1.

Method

The review is organised as follows. Section numbers refer to the paper.

1. Neural network background and layer types, with a comparison of fully connected, convolutional, recurrent, residual and transformer layers. ResNets are called unsuitable for TinyML hardware and transformers "bad candidates" for TinyML §2.1–§2.3 tab. 1.
2. Efficient recurrent layers. GRU is favoured over LSTM for keyword spotting, and convolutional-recurrent networks are proposed for the target setting. Minimal gated unit variants MGU1, MGU2 and MGU3 remove gate inputs or biases §4 eq. 4.
3. Knowledge distillation: a small student is trained with a loss that mixes its own task loss and a divergence from a teacher's softened outputs §4.1 eq. 5.
4. Pruning. Magnitude-based unstructured pruning with a cubic sparsity schedule is the main recipe §4.2 eq. 6 fig. 6. Structured pruning removes filters, neurons or layers §4.2. Bayesian sparsity uses spike-and-slab, horseshoe and log-uniform priors §4.2 fig. 7.
5. Quantization. The review separates quantization-aware training (QAT) from post-training quantization (PTQ), uniform from non-uniform steps, and symmetric from asymmetric ranges §4.3 eq. 8 eq. 9. It restricts itself to uniform schemes because they are supported on embedded hardware §4.3.
6. Weight sharing, for example clustering weights with k-means, and low-rank matrix or tensor decomposition §4.4 §4.5.
7. Deployment tooling: CMSIS-NN as a low-level Arm library §5.2.1. TensorFlow Lite Micro (TFLM), NNoM and Edge Impulse as frameworks §5.2.2. Only TensorFlow-input frameworks targeting Arm Cortex-M are considered §5.2.
8. Benchmarks: published flash size against accuracy on MNIST, ImageNet, Visual Wake Words (VWW) and Google Speech Commands v2-12, with storage thresholds for Cortex-M0+, M4 and M7 §6 fig. 9 tab. 4.

Key equations, reproduced from the paper:

Knowledge distillation loss §4.1 eq. 5:

L_{\text{KD}}(x,y) = \alpha L_{\text{S}}(x,y) + (1-\alpha) \text{D}_{\text{KL}}\left(\text{softmax}\left(\frac{T(x,y)}{\text{temp}}\right), \text{softmax}\left(\frac{S(x,y)}{\text{temp}}\right)\right)

Polynomial sparsity schedule §4.2 eq. 6:

s_t = s_f + (s_0-s_f)\left(1- \frac{t-t_0}{n\Delta t} \right)^3

Uniform quantization function and scales, as printed §4.3 eq. 8 eq. 9:

Q(r) = \text{int}(r/S) - Z

S_{\text{sym}} = \frac{\max(|\alpha|,|\beta|)}{2^{b-1}-1}, \qquad S_{\text{asym}} = \frac{\max(|\alpha|,|\beta|)}{\left(2^{b}-1\right)/2}

Gradual magnitude pruning, as a schedule. At every Δt a gradual number of weights is set to zero by magnitude until the target sparsity is reached §4.2 eq. 6:

# Target sparsity at training step t  [§4.2] [eq. 6]
# Input:  step t; s0, sf in [0, 1]; t0; n pruning steps; dt pruning interval
# Output: sparsity to reach at step t

def target_sparsity(t, s0, sf, t0, n, dt):
    # Fraction of the pruning window already elapsed  [§4.2] [eq. 6]
    progress = (t - t0) / (n * dt)
    # Cubic decay: fast early, slow late, so most pruning happens early  [§4.2]
    return sf + (s0 - sf) * (1 - progress) ** 3

Knowledge distillation, with the student's own loss and the teacher's soft targets §4.1 eq. 5:

# Distillation loss for one batch  [§4.1] [eq. 5]
# Input:  inputs x, labels y; student S; frozen teacher T; alpha; temp
# Output: scalar loss

def distillation_loss(x, y, student, teacher, alpha, temp):
    # Student's own task loss, for example cross-entropy  [§4.1] [eq. 5]
    student_logits = student(x)
    task_loss = student_loss(student_logits, y)
    # Teacher outputs can be precomputed and stored  [§4.1]
    teacher_logits = teacher(x)
    # Softened distributions over classes  [§4.1] [eq. 5]
    teacher_probs = softmax(teacher_logits / temp)
    student_probs = softmax(student_logits / temp)
    # Divergence between teacher and student, direction as in eq. 5  [§4.1]
    divergence = kl_divergence(teacher_probs, student_probs)
    return alpha * task_loss + (1 - alpha) * divergence

Hyperparameters

The review has no experiments of its own. The values below are the examples and recommendations it gives.

Results

Author-reported, from the review's own reading of published results. There are no baseline columns because the review does not run baselines. Setup: flash model size against accuracy for models from cited works, on four datasets §6 fig. 9. The points are not tabulated in the paper; the rankings below are the authors' reading of those plots.

Paper vs code

No official code. The paper links no repository or project page; its only URL is the corresponding author's email. The arXiv metadata has no comments. A GitHub repository search for the title words returned no results.

Pitfalls

• Figure 4 labels. The caption calls floating-point layout "(b)" and fixed-point "(a)", but the subfigures are labelled the other way round: (a) is the 32-bit floating-point layout and (b) is Q16.16. Read the subfigure labels. The caption's fixed-point range "[−2^{m−1}, 2^{m−1}−2^{−n}] ≈ [−2^15, 2^16]" has an upper bound that does not match its own formula, which gives about 2^15 for m = 16.
• Asymmetric scale. Eq. 9 divides max(|α|,|β|) by (2^b−1)/2. The zero-point Z is not given a formula, and eq. 8 does not state its rounding mode; §3.3 uses round-to-nearest. Do not take eq. 9 as a complete asymmetric recipe without checking the cited quantization sources.
• Zero point. Eq. 8 defines Q(r) = int(r/S) − Z, but how Z is chosen for asymmetric ranges is not stated. The text says asymmetric schemes use the full integer range, for example [−128, +127], and Z ≠ 0 §4.3.
• Distillation. The direction of the KL term and any scaling of the gradient by temp² are not stated. α and temp are not given. Eq. 5 is written as printed; the review offers no default.
• Pruning units. Fig. 6's caption says the rate is "over epochs", but eq. 6 indexes steps t. Δt = 100 has no stated unit.
• Prior densities (§4.2, fig. 7) are shown with example hyperparameters. The review gives no recipe for choosing them, and the log-uniform prior is improper unless its interval is bounded (as the paper states).
• Cortex-M0+ ratio. §3.1 says the Cortex-M4 is "1500 times more power-hungry" than the Cortex-M0+. Using the table values (0.1 W against 70 µW) gives about 1430×, so 1500 is approximate.
• Table 4 duplicate name. "Newport" appears in two rows (Cortex-M0+ and eDMPv1) with the same clock and price. They differ only in the architecture column. The Cortex-M0+ row's 6.14 MHz clock differs from the "10 MHz" target in §3.2; the text says "approximately".
• Figure 9 is not reproducible from the paper. The points are read from plots of models in cited works. The paper gives no selection criteria, no hardware measurement conditions and no table. "Best tradeoff" is a visual judgement, and the summary that the Cortex-M4 suffices for most models is a reading of those plots.
• Cited claims. The 49× Deep Compression figure, the 90% sparsity figure and the CMSIS-NN and EON numbers are the review's summaries of other papers. They are not verified here; check the source before relying on them.
• Tool descriptions date from the arXiv version (v1, November 2023). The review's descriptions of TFLM, NNoM and Edge Impulse (for example "does not provide built-in tools to measure power") may be out of date.
• Scope. Only uniform quantization, only TensorFlow-input frameworks, and only Arm Cortex-M targets are considered §4.3 §5.2. The benchmark comparison covers four datasets only.

Lineage

• Builds on: none; this is a survey. It cites foundational work such as distillation §4.1, magnitude pruning and the lottery ticket hypothesis §4.2, and TFLM and CMSIS-NN §5.2, but no cards exist for them on this site.
• Compared against: the four benchmark rankings in §6 compare published models with each other, not with this review.
• Superseded by: not stated.