---
kind: concept
slug: on-device-training
title: On-device training
summary: >-
  Training or adapting a neural network on the target device itself, e.g. a Cortex-M microcontroller with little RAM. Fully quantized training runs forward, backward and update in 8-bit, and per-sample sparse error updates cut backward work. One paper so far; its numbers are author-reported and the best accuracy needs the last layer kept in float.
tags: [on-device-training, tinyml, microcontroller, quantized-training, sparse-gradient-updates]
papers: ["2407.10734"]
related: [quantization-aware-training, weight-quantization, transfer-learning, post-training-quantization]
status: draft
updated_at: 2026-10-25
---

## Overview

Most deployed networks are trained on a server and only run on the device.
On-device training asks whether the device can also adapt the model using data
it has just collected. The obstacles are RAM for writable weights and for the
intermediate tensors that backpropagation keeps, and compute time per sample
[arxiv:2407.10734 §1] [arxiv:2407.10734 §1.1].

The shared idea is to keep the network in the same low-bit integer form it uses
at inference, so no separate float training copy is needed, and to reduce the
work done in the backward pass. The invariant is that the weights are always
stored as quantized values, re-fitted after each update
[arxiv:2407.10734 §3.1].

## Variants

Only one paper with a card is available for this page, so the variants below
are the separate mechanisms that paper combines. Each can be used on its own
in principle, but the paper evaluates them together.

### Fully quantized training

Weights, feature maps and errors use one linear 8-bit scheme per tensor, a
scale and a zero point, the same scheme used for inference. The backward pass
reuses integer kernels with the forward structure. The weight gradient is kept
in floating point and the update is applied to the dequantized weights, after
which scale and zero point are recomputed from the new minimum and maximum
[arxiv:2407.10734 §3.1] [arxiv:2407.10734 eq. 4] [arxiv:2407.10734 eq. 5].
Card: [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734).

Batch gradients are accumulated per sample, so the extra RAM depends on the
number of trained layers and not on the batch size. The paper uses no momentum
or adaptive optimizer, to save memory [arxiv:2407.10734 §3.1].

### Dynamic sparse gradient updates

During the backward pass of each sample, each layer's error tensor is ranked by
channel or column with an L1 norm, and only the top-k are kept. The keep rate
is set per sample from the sample's loss, using two hyperparameters
lambda_min and lambda_max [arxiv:2407.10734 §3.2] [arxiv:2407.10734 eq. 9].
The paper reports a 6.64x average backward-pass speed-up per sample at
lambda_min = 0.1, in the mixed configuration on the IMXRT2062 board, averaged
over its seven transfer-learning datasets [arxiv:2407.10734 §4.3].

### Configurations: uint8, mixed and float32

The paper compares three settings: everything in uint8; uint8 except the
classification head, which stays float32 (mixed); and a float32 reference
[arxiv:2407.10734 §4]. Mixed matches float in transfer learning, and it is the
setting that closes most of the gap on the harder tasks [arxiv:2407.10734 §4.1]
[arxiv:2407.10734 fig. 4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Adapting a pretrained network, few trained layers, many classes | Mixed configuration, last layer in float | The paper reports uint8-only lagging on flowers, cifar100 and speech, while mixed matches float [arxiv:2407.10734 §4.1] |
| Adapting a pretrained network, simpler tasks, few classes | uint8 configuration | The paper reports uint8 matching or exceeding its GPU baseline on the simpler datasets [arxiv:2407.10734 §4.1] |
| Full training from scratch on a small network | uint8 or mixed, expect a gap to float32 | Full training shows a larger gap than transfer learning, with float32 highest [arxiv:2407.10734 §4.4] |
| Backward pass is the bottleneck and the model is uint8 | Dynamic sparse updates with lambda_min = 0.5 | At 0.5 the paper reports accuracy close to full updates; at 0.1 uint8 training was unstable [arxiv:2407.10734 §4.3] |
| Compare against MCUNet-5FPS on CIFAR-10 | Lower memory and latency, slightly lower accuracy | Author-reported 34.8% less memory and 49.0% lower latency, with accuracy 85.9% vs 89.5% [arxiv:2407.10734 §4.5] |
| Choosing a board | Cortex-M7 for latency, Cortex-M4 for lowest idle draw | IMXRT2062 is fastest per sample; nRF52840 has the lowest idle current [arxiv:2407.10734 §4.2] [arxiv:2407.10734 tab. 2] |

Latency is dominated by the forward pass when only the last layers are trained,
and by the backward pass for full training [arxiv:2407.10734 §4.1]
[arxiv:2407.10734 §4.4].

## Implementation sketch

Language-agnostic pseudocode of the common core. The paper's own text gives
these steps; the formulas have gaps noted in the paper card.

```pseudocode
# Once per trained layer, after a minibatch of samples  [arxiv:2407.10734 §3.1]
def update_layer(w_q, scale, zero, grad_sum, lr):
    # Gradient accumulated per sample, standardized  [arxiv:2407.10734 eq. 8]
    g = standardize(grad_sum)
    # Back to float for the step  [arxiv:2407.10734 eq. 5]
    w = (w_q - zero) * scale - lr * g
    # Refit the 8-bit grid to the new range  [arxiv:2407.10734 eq. 6-7]
    scale = (w.max() - w.min()) / 255
    zero = floor(-w.min() / scale)
    return floor(w / scale) + zero, scale, zero

# Per sample, in the backward pass  [arxiv:2407.10734 eq. 9]
def keep_count(loss, max_loss_so_far, n, lam_min, lam_max):
    eps = abs(loss - max_loss_so_far)
    rate = min(lam_min + eps * (lam_max - lam_min), 1.0)
    return floor(rate * n)    # n: count of channels or columns, see Contradictions
```

Where variants differ: the configuration decides which layers are quantized
[arxiv:2407.10734 §4]; the sparse rule is optional and only changes which error
channels enter the backward pass [arxiv:2407.10734 §3.2].

## Contradictions

Within the one paper, not between papers:

- The text says the keep rate falls toward lambda_min as the loss falls
  [arxiv:2407.10734 §3.2]. The formula as written makes the rate lowest when
  the loss equals the running maximum, which is typical early in training
  [arxiv:2407.10734 eq. 9]. The paper does not resolve this.
- The text says Table 4 shows the uint8 optimizer ahead on five of eight
  datasets [arxiv:2407.10734 §4.5]. The table shows it ahead on four, a tie on
  one and behind on three [arxiv:2407.10734 tab. 4]. The averages (73.7 and
  73.5) match the table.

## Papers

- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734): fully quantized training with per-sample sparse error updates on three Cortex-M boards, with transfer learning and full training results.