---
kind: concept
slug: on-device-inference
title: On-device inference
summary: >-
  Running a trained model on the device that needs the answer, from microcontrollers to phones and Jetson-class boards, instead of on a server. The core idea is to compress or redesign the model to fit memory, compute and power limits, then port it to a device runtime. The trade-off: small models lose capability, and reported speed and energy figures rarely carry over to other hardware.
tags: [tinyml, edge-ai, quantization, pruning, distillation, microcontrollers, edge-llm]
papers: ["2205.14942", "2211.04448", "2311.11883", "2312.12456", "2407.14514", "2410.11845", "2501.03256", "2504.09685", "2505.03238", "2601.03290"]
related: [on-device-training, computation-offloading, knowledge-distillation, network-pruning, post-training-quantization, quantization-aware-training, mixed-precision-quantization, intermittent-computing, neural-architecture-search, hardware-aware-training, speculative-decoding, llm-serving, activation-sparsity, object-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

On-device inference means the model runs on the hardware that uses its output,
not on a remote server. The hardware ranges from microcontrollers with a few
hundred kilobytes of SRAM [arxiv:2311.11883 §3.2], through Raspberry Pi Pico
boards with 264 KB to 520 KB of SRAM [arxiv:2501.03256 §II], to phones and
Jetson-class modules that run 1 to 3 billion parameter language models
[arxiv:2505.03238 tab. 3].

The sources give three reasons to keep inference local. Sending sensor data to a
distant site adds delay and confidentiality risk [arxiv:2211.04448 §I]. The
device may have no network link to rely on [arxiv:2505.03238 §1]. And the power
budget of a battery-powered sensor can be around 1 mW [arxiv:2211.04448 §I].

The common pipeline is: train on a larger machine, shrink the model, port it to
the device's runtime, and run inference on each new input
[arxiv:2211.04448 §II] [arxiv:2601.03290 §VIII-H]. The techniques fall into two
groups. One changes the model: quantization, pruning, distillation, low-rank
factorisation or a search for a compact design [arxiv:2410.11845 §3]. The other
changes how it runs: splitting work with a cloud, token reduction and early
exit, offloading between memory tiers, and hardware-specific kernels
[arxiv:2410.11845 §4].

Four of the papers are surveys. The rest report numbers from a small number of
boards or desktop machines [arxiv:2407.14514 §5.1] [arxiv:2312.12456 §8.1]
[arxiv:2205.14942 §IV-C], and the surveys quote figures from cited works
[arxiv:2601.03290 tab. 3]. Use the numbers as orientation and measure on your
own hardware.

## Variants

### Microcontroller deployment (TinyML)

The target is a microcontroller with on-chip memory only and, in the smallest
parts, no floating-point unit. The review's workflow is data collection, model
training on a resource-rich machine, then porting to the microcontroller
[arxiv:2211.04448 §II]. Three ways to install a model are named: hand-written
code, generated code, or an ML interpreter. The interpreters give the best
portability [arxiv:2211.04448 §IV.C]. The Cortex-M targets under study have
under about 8 kB of RAM and near 10 MHz, and the review favours integer-only
inference with uniform 8-bit post-training quantization as the default
[arxiv:2311.11883 §3.2] [arxiv:2311.11883 §3.3] [arxiv:2311.11883 §4.3]. Its
deployment tools are CMSIS-NN, TensorFlow Lite Micro, NNoM and Edge Impulse
[arxiv:2311.11883 §5.2]. Card: [A review of TinyML](/papers/2211.04448).
Card: [Efficient Neural Networks for Tiny Machine Learning](/papers/2311.11883).

A hand-written example runs a small dense network on a Raspberry Pi Pico in
MicroPython. Weights are exported from Keras and the layers are plain loops
[arxiv:2501.03256 §IV]. The paper reports only accuracy: 90% for an 8-neuron
network and 95% for a 6-neuron network, on an Iris task with 20 test samples
[arxiv:2501.03256 §IV] [arxiv:2501.03256 tab. 1]. It reports no timing or
memory measurements [arxiv:2501.03256 §IV]. Card:
[AI-ANNE](/papers/2501.03256).

### Architecture search under a device budget

Instead of shrinking a large model, search for a small one that already fits.
One work uses an open-weight LLM to propose five-stage CNNs. Candidates are
checked against a MAC range of 70 to 350 million and a 320 KB SRAM limit before
training, and the search keeps a Pareto front on accuracy, MACs and parameters
[arxiv:2504.09685 §3.1] [arxiv:2504.09685 §3.2] [arxiv:2504.09685 §4.1.4]. The
selected network is trained longer and distilled from a ViT-B/16 teacher
[arxiv:2504.09685 §4.1.3]. The author-reported result is 74.50% on CIFAR-100
for the chosen model, with 86 million MACs and 0.54 MB flash, at 1.5 search
days against 12.5 for MCUNet-in4 as quoted in the table [arxiv:2504.09685 tab. 2].
Card: [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685).

For batteryless devices that run on harvested energy, the budget is per power
cycle. A neural architecture search with a power-cycle model targets a 2 KB
runtime SRAM on an MSP430FR5994 [arxiv:2407.14514 §5.1] [arxiv:2407.14514 §5.3].
It distils and shrinks a large teacher first, then searches, and it proposes
processing a larger batch per power cycle [arxiv:2407.14514 §4.2]
[arxiv:2407.14514 §4.3]. The author reports +19.1 points on CIFAR-10 and
+12.0 on Tiny ImageNet against a hardware-aware NAS baseline, measured on a PC
for accuracy [arxiv:2407.14514 §5.4] [fig. 11]. Peak memory is reported as
1.33 times that of the HW-NAS baseline on CIFAR-10 [arxiv:2407.14514 §5.5]. Latency is 12%
higher than the iNAS baseline on CIFAR-10 [arxiv:2407.14514 §5.6]. The paper
also states that its reward, as printed, adds the latency term with a sign that
favours slower designs. The paper does not say whether this is a typo
[arxiv:2407.14514 §4.1]. Card:
[Accelerate Intermittent Deep Inference](/papers/2407.14514).

### Compressing a model before deployment

Five techniques recur across the surveys: quantization, pruning, distillation,
weight sharing and low-rank decomposition [arxiv:2311.11883 §4]. The edge-LLM
survey groups them the same way and adds the LLM-specific methods
[arxiv:2410.11845 §3.1] [arxiv:2410.11845 §3.2] [arxiv:2410.11845 §3.3]
[arxiv:2410.11845 §3.4]. For each, it names examples such as weight-only
quantization and weight-activation co-quantization, and structured versus
unstructured pruning [arxiv:2410.11845 §3.1.1] [arxiv:2410.11845 §3.2.1].
Card: [A Review on Edge Large Language Models](/papers/2410.11845).

The lightweight-transformer survey gives a six-step pipeline: choose the base
architecture, distil in two stages, prune heads and channels, mix FP16 and
INT8, fuse operators, then profile on the target device and repeat
[arxiv:2601.03290 §VIII-H]. Its compression data is a plot without underlying values.
At 50% of the size with FP16 it reports 99.5% relative accuracy; at 25% with
INT8, 98.7%; and at 13% with pruning and INT8, 96.5% [arxiv:2601.03290 fig. 6].
Card: [Lightweight Transformer Architectures for Edge Devices](/papers/2601.03290).

### Running a small language model locally

One robotics paper deploys a 3-billion-parameter Qwen2.5 model, quantized to
Q5 in GGUF format and run with llama.cpp, on a Jetson Orin AGX
[arxiv:2505.03238 §4.4] [arxiv:2505.03238 app. F]. Its speed table shows
38.78 tokens per second and 3.74 s average latency for that setup, against
3.55 tokens per second and 30.73 s at FP16 [arxiv:2505.03238 tab. 3]. The
quantized model scores lower on the paper's decision and control tests. The
decision accuracy falls from 86.82% at FP16 to 79.88% at Q5 [arxiv:2505.03238
tab. 4]. The control score, which is the improvement over a default controller in
percent, falls from 63.3% to 47.9% [arxiv:2505.03238 tab. 5]. The 1.5B
model's control score is higher at Q5, but with three failed outputs
[arxiv:2505.03238 tab. 5]. Card:
[RobotxR1](/papers/2505.03238).

The edge-LLM survey covers the hardware side: CPUs, GPUs and NPUs on phone and
edge SoCs, and co-design methods such as sparsity matched to a chip
[arxiv:2410.11845 §4.2] [arxiv:2410.11845 §4.3]. Many of its co-design results
were tested only in simulation or on ASIC prototypes [arxiv:2410.11845 §4.2]
[arxiv:2410.11845 §6]. For phones, the transformer survey lists per-platform
speed-up factors relative to FP32. INT8 gives 3.2 times on a mobile NPU
[arxiv:2601.03290 tab. 4]. Its combined row is the product of the three rows
above it, and the paper does not show that the effects multiply on real
hardware [arxiv:2601.03290 tab. 4].

### Splitting a large model between GPU and CPU

When a model is too large for the GPU's memory, PowerInfer divides each
feed-forward block by neuron. Neurons that fire often across profiled text are
kept on the GPU ("hot"), and the rest are computed on the CPU ("cold")
[arxiv:2312.12456 §3.1] [arxiv:2312.12456 §4]. An offline integer linear
program chooses this placement from activation counts [arxiv:2312.12456 §6.3].
At run time, a small per-layer MLP predictor selects the neurons to compute for
each token [arxiv:2312.12456 §5.1]. The authors' motivation is that for batch
sizes below 32, computing a CPU-resident slice on the CPU is faster than
transferring it to the GPU [arxiv:2312.12456 §3.2].

The reported speed-ups are for batch size one, on a desktop with an RTX 4090.
The average over the tested models is 8.32 tokens per second, or 7.23 times
llama.cpp, and the peak gain is 11.69 times for Falcon-40B [arxiv:2312.12456
§8.2] [arxiv:2312.12456 fig. 10]. On a desktop with an RTX 2080 Ti and FP16
weights, the average gain is 4.71 times [arxiv:2312.12456 fig. 11]. The method
depends on strong activation sparsity. The authors expect the largest gains
for ReLU-family models, and report roughly 50% sparsity and smaller gains for
SiLU-based ones [arxiv:2312.12456 §4] [arxiv:2312.12456 §8.4]. Their SiLU
results are in [arxiv:2312.12456 tab. 8]. Card:
[PowerInfer](/papers/2312.12456).

### Edge detector with cloud retraining

For an autonomous vehicle, Edge YOLO runs the detector on a Jetson-class board
in the car and saves its results locally. A cloud server labels the collected
data, retrains the model and sends new weights back [arxiv:2205.14942 §III-A].
The authors argue that cloud-only inference depends on network quality and adds
delay and energy cost [arxiv:2205.14942 §I]. The detector is a slimmed
YOLOv4-style network with a 17-layer backbone, a feature pyramid neck and three
output scales [arxiv:2205.14942 §III-B] [arxiv:2205.14942 tab. 1].

On COCO2017 the model is reported at 25.27 MB, with 26.6 frames per second on a
Jetson Xavier without TensorRT and 0.473 mAP@0.5. YOLOv4 reaches 0.657 in the
same table, at 2.1 frames per second [arxiv:2205.14942 tab. 5]. With TensorRT
FP16, Edge YOLO reaches 49.5 frames per second and 0.325 mAP@0.5 on the same
dataset [arxiv:2205.14942 tab. 5]. On KITTI 2D the TensorRT accuracy is 0.726
against 0.8212 without it [arxiv:2205.14942 tab. 6]. So the speed gain comes
with an accuracy loss the authors' numbers show, and the choice between the two
depends on the application. Card: [Edge YOLO](/papers/2205.14942).

### Split and collaborative inference

When a cloud or a nearby server is reachable, work can be divided. Edge YOLO
splits the other way: the device infers and the cloud retrains
[arxiv:2205.14942 §III-A]. The
edge-LLM survey describes splitting layers between cloud and edge, and
speculative decoding where a small edge model drafts tokens that a cloud model
verifies [arxiv:2410.11845 §4.1.1]. Single-device options include token
reduction, early exit once a confidence threshold is met, and offloading
across GPU, CPU and disk or flash [arxiv:2410.11845 §4.1.2]. The survey reports
no device measurements for these [arxiv:2410.11845 §7]. For
speculative decoding itself, see [speculative-decoding](/concepts/speculative-decoding).

## When to use which

The choice depends on the memory, compute, power and network constraints. Most
guidance below is qualitative. The numbers in the cited papers are
author-reported.

| Situation | Use | Why |
|---|---|---|
| Microcontroller, tens to a few hundred kB of SRAM, no FPU | Integer-only 8-bit post-training quantization, a small CNN or MLP, TensorFlow Lite Micro or CMSIS-NN | [arxiv:2311.11883 §3.2] [arxiv:2311.11883 §3.3] [arxiv:2311.11883 §5.2] [arxiv:2211.04448 §IV.C] |
| Small dense net on a Pico-class board, tiny task | Hand-written MicroPython, weights exported from a Keras model | [arxiv:2501.03256 §IV]; accuracy only, no timing |
| Fixed SRAM and MAC budget, and search compute is available | Budget-checked search with a Pareto front, then distillation | [arxiv:2504.09685 §3.2] [arxiv:2504.09685 §4.1.3] [arxiv:2504.09685 §4.1.4] |
| Batteryless device on harvested energy | Intermittent-aware search, distilled teacher, checkpointed execution | [arxiv:2407.14514 §2.1] [arxiv:2407.14514 §4.2] [arxiv:2407.14514 §5.1] |
| LLM with tight memory and moderate compute | Weight-only quantization plus structured pruning | [arxiv:2410.11845 §3.6] |
| Maximum compression, some accuracy loss accepted | Weight-activation co-quantization plus unstructured pruning | [arxiv:2410.11845 §3.6] |
| A large teacher model exists | Distillation: white-box if the teacher's internals are accessible, black-box otherwise | [arxiv:2410.11845 §3.3] [arxiv:2410.11845 §3.6] |
| Local 1 to 3B LLM on a Jetson-class board | Q5 GGUF with llama.cpp, then measure tokens per second on the board | [arxiv:2505.03238 tab. 3] [arxiv:2505.03238 app. F] |
| Latency target and a reachable cloud | Split inference, or edge-drafted speculative decoding | [arxiv:2410.11845 §4.1.1]; survey, no measurements |
| LLM too large for GPU memory, one consumer GPU, ReLU-family model | Hot and cold neuron split between GPU and CPU with activation predictors | [arxiv:2312.12456 §4] [arxiv:2312.12456 §8.2]; author-reported on two desktop PCs |
| Vehicle or robot detector on a Jetson-class board, data collected on the device | Slimmed one-stage detector, with cloud retraining; budget for the accuracy loss of TensorRT FP16 | [arxiv:2205.14942 §III-A] [arxiv:2205.14942 tab. 5] |
| Vision transformer with a 30 FPS target, on hardware of about 20 TOPS | Keep to the parameter range the survey gives for that budget, then profile on the device | [arxiv:2601.03290 §VIII-E] [arxiv:2601.03290 §VIII-H]; author estimates |

Two constraints recur in the sources. Memory is usually the first bottleneck
[arxiv:2311.11883 §3.3], and the lightweight-transformer survey assumes single-sample inference
[arxiv:2601.03290 §VIII-A]. Batch size and sequence length
change the picture, so state both when you compare numbers.

## Implementation sketch

The shared core is the same across variants. The steps differ in how they
compress, which runtime they use, and how the budget is checked.

```pseudocode
# On-device inference: from a trained model to running on the device
# Input:  trained model, device budget (SRAM, flash, MACs, power), input stream
# Output: predictions made on the device

def deploy_on_device(model, budget, device, stream):
    # Shrink: quantize, prune, distil or search  [arxiv:2311.11883 §4]
    compact = compress(model, budget)
    # Reject over-budget candidates before training  [arxiv:2504.09685 §3.1]
    if not fits(compact, budget):
        raise ValueError("candidate exceeds the device budget")
    # Port: interpreter, generated or hand code  [arxiv:2211.04448 §IV.C]
    ported = port(compact, device)
    # Run each input on device; data stays local  [arxiv:2211.04448 §I]
    for sample in stream:
        yield run(ported, sample)
    # Profile on the target, then iterate  [arxiv:2601.03290 §VIII-H]
```

`fits` should check peak activation memory as well as parameter count. The
search paper reports peak SRAM separately from model size
[arxiv:2504.09685 §4.2.2].

## Contradictions

- **Memory of a 7B model at FP16.** The edge-LLM survey gives "over 8 GB" for
  the FP16 weights of LLaMA-2 7B [arxiv:2410.11845 §1]. Two bytes per weight for
  7 billion weights is about 14 GB, and the survey does not explain the lower
  figure. The RobotxR1 table reports 8.4 GB for a Qwen2.5 7B run at FP16
  [arxiv:2505.03238 tab. 3]. The two use different models, and neither source
  defines its memory measure. Do not use either figure as a budget.
- **Size reduction of a pipeline.** The lightweight-transformer survey states
  that its six-step pipeline gives 8 to 12 times smaller models with under 2%
  accuracy loss [arxiv:2601.03290 §VIII-H]. The per-step reductions it lists
  (10 to 15% for distillation, 25 to 35% more for pruning, 50% more for
  quantization) compound to roughly 3 to 3.6 times by our arithmetic, so the
  paper's headline range is not derived from its own steps.
- **Distillation gain.** The lightweight-transformer survey reports a 1.5 to
  2.5% gain for two-stage distillation, with no experiment described
  [arxiv:2601.03290 §VIII-G]. The model-search paper measures +1.32 to +1.40
  points for logits distillation from ViT-B/16 on CIFAR-100 [arxiv:2504.09685
  tab. 4]. The setups differ, so the two figures should not be compared directly.

These are the conflicts found across the cited cards. The surveys do not
resolve them.

## Papers

- [A review of TinyML](/papers/2211.04448): workflow, deployment options and open challenges for microcontroller ML. No experiments.
- [Efficient Neural Networks for Tiny Machine Learning](/papers/2311.11883): survey of compression methods and TinyML tools for Cortex-M targets, with a published model-size comparison.
- [Accelerate Intermittent Deep Inference](/papers/2407.14514): search for networks that run on a batteryless MSP430 with 2 KB of SRAM. Author-reported results.
- [A Review on Edge Large Language Models](/papers/2410.11845): survey of compression, runtime and hardware methods for edge LLMs.
- [AI-ANNE](/papers/2501.03256): MicroPython port of Keras dense networks to a Raspberry Pi Pico. Accuracy only.
- [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685): LLM-guided search with a Pareto front for STM32-class CNNs, plus distillation.
- [RobotxR1](/papers/2505.03238): fine-tuning a small Qwen2.5 model for a robot car, with Q5 deployment on a Jetson Orin AGX.
- [Lightweight Transformer Architectures for Edge Devices](/papers/2601.03290): survey of small transformers, quantization, pruning and deployment tools for edge devices.
- [PowerInfer](/papers/2312.12456): hot and cold neuron split between a consumer GPU and CPU for LLM inference. Author-reported on two desktop PCs.
- [Edge YOLO](/papers/2205.14942): slimmed YOLOv4-style detector for Jetson boards in vehicles, with cloud retraining. Author-reported on COCO2017 and KITTI.