On-device inference

Overview

On-device inference means the model runs on the hardware that uses its output, not on a remote server. The hardware ranges from microcontrollers with a few hundred kilobytes of SRAM 2311.11883 §3.2, through Raspberry Pi Pico boards with 264 KB to 520 KB of SRAM 2501.03256 §II, to phones and Jetson-class modules that run 1 to 3 billion parameter language models 2505.03238 tab. 3.

The sources give three reasons to keep inference local. Sending sensor data to a distant site adds delay and confidentiality risk 2211.04448 §I. The device may have no network link to rely on 2505.03238 §1. And the power budget of a battery-powered sensor can be around 1 mW 2211.04448 §I.

The common pipeline is: train on a larger machine, shrink the model, port it to the device's runtime, and run inference on each new input 2211.04448 §II 2601.03290 §VIII-H. The techniques fall into two groups. One changes the model: quantization, pruning, distillation, low-rank factorisation or a search for a compact design 2410.11845 §3. The other changes how it runs: splitting work with a cloud, token reduction and early exit, offloading between memory tiers, and hardware-specific kernels 2410.11845 §4.

Four of the papers are surveys. The rest report numbers from a small number of boards or desktop machines 2407.14514 §5.1 2312.12456 §8.1 2205.14942 §IV-C, and the surveys quote figures from cited works 2601.03290 tab. 3. Use the numbers as orientation and measure on your own hardware.

Variants

Microcontroller deployment (TinyML)

The target is a microcontroller with on-chip memory only and, in the smallest parts, no floating-point unit. The review's workflow is data collection, model training on a resource-rich machine, then porting to the microcontroller 2211.04448 §II. Three ways to install a model are named: hand-written code, generated code, or an ML interpreter. The interpreters give the best portability 2211.04448 §IV.C. The Cortex-M targets under study have under about 8 kB of RAM and near 10 MHz, and the review favours integer-only inference with uniform 8-bit post-training quantization as the default 2311.11883 §3.2 2311.11883 §3.3 2311.11883 §4.3. Its deployment tools are CMSIS-NN, TensorFlow Lite Micro, NNoM and Edge Impulse 2311.11883 §5.2. Card: A review of TinyML. Card: Efficient Neural Networks for Tiny Machine Learning.

A hand-written example runs a small dense network on a Raspberry Pi Pico in MicroPython. Weights are exported from Keras and the layers are plain loops 2501.03256 §IV. The paper reports only accuracy: 90% for an 8-neuron network and 95% for a 6-neuron network, on an Iris task with 20 test samples 2501.03256 §IV 2501.03256 tab. 1. It reports no timing or memory measurements 2501.03256 §IV. Card: AI-ANNE.

Architecture search under a device budget

Instead of shrinking a large model, search for a small one that already fits. One work uses an open-weight LLM to propose five-stage CNNs. Candidates are checked against a MAC range of 70 to 350 million and a 320 KB SRAM limit before training, and the search keeps a Pareto front on accuracy, MACs and parameters 2504.09685 §3.1 2504.09685 §3.2 2504.09685 §4.1.4. The selected network is trained longer and distilled from a ViT-B/16 teacher 2504.09685 §4.1.3. The author-reported result is 74.50% on CIFAR-100 for the chosen model, with 86 million MACs and 0.54 MB flash, at 1.5 search days against 12.5 for MCUNet-in4 as quoted in the table 2504.09685 tab. 2. Card: Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?.

For batteryless devices that run on harvested energy, the budget is per power cycle. A neural architecture search with a power-cycle model targets a 2 KB runtime SRAM on an MSP430FR5994 2407.14514 §5.1 2407.14514 §5.3. It distils and shrinks a large teacher first, then searches, and it proposes processing a larger batch per power cycle 2407.14514 §4.2 2407.14514 §4.3. The author reports +19.1 points on CIFAR-10 and +12.0 on Tiny ImageNet against a hardware-aware NAS baseline, measured on a PC for accuracy 2407.14514 §5.4 fig. 11. Peak memory is reported as 1.33 times that of the HW-NAS baseline on CIFAR-10 2407.14514 §5.5. Latency is 12% higher than the iNAS baseline on CIFAR-10 2407.14514 §5.6. The paper also states that its reward, as printed, adds the latency term with a sign that favours slower designs. The paper does not say whether this is a typo 2407.14514 §4.1. Card: Accelerate Intermittent Deep Inference.

Compressing a model before deployment

Five techniques recur across the surveys: quantization, pruning, distillation, weight sharing and low-rank decomposition 2311.11883 §4. The edge-LLM survey groups them the same way and adds the LLM-specific methods 2410.11845 §3.1 2410.11845 §3.2 2410.11845 §3.3 2410.11845 §3.4. For each, it names examples such as weight-only quantization and weight-activation co-quantization, and structured versus unstructured pruning 2410.11845 §3.1.1 2410.11845 §3.2.1. Card: A Review on Edge Large Language Models.

The lightweight-transformer survey gives a six-step pipeline: choose the base architecture, distil in two stages, prune heads and channels, mix FP16 and INT8, fuse operators, then profile on the target device and repeat 2601.03290 §VIII-H. Its compression data is a plot without underlying values. At 50% of the size with FP16 it reports 99.5% relative accuracy; at 25% with INT8, 98.7%; and at 13% with pruning and INT8, 96.5% 2601.03290 fig. 6. Card: Lightweight Transformer Architectures for Edge Devices.

Running a small language model locally

One robotics paper deploys a 3-billion-parameter Qwen2.5 model, quantized to Q5 in GGUF format and run with llama.cpp, on a Jetson Orin AGX 2505.03238 §4.4 2505.03238 app. F. Its speed table shows 38.78 tokens per second and 3.74 s average latency for that setup, against 3.55 tokens per second and 30.73 s at FP16 2505.03238 tab. 3. The quantized model scores lower on the paper's decision and control tests. The decision accuracy falls from 86.82% at FP16 to 79.88% at Q5 [arxiv:2505.03238 tab. 4]. The control score, which is the improvement over a default controller in percent, falls from 63.3% to 47.9% 2505.03238 tab. 5. The 1.5B model's control score is higher at Q5, but with three failed outputs 2505.03238 tab. 5. Card: RobotxR1.

The edge-LLM survey covers the hardware side: CPUs, GPUs and NPUs on phone and edge SoCs, and co-design methods such as sparsity matched to a chip 2410.11845 §4.2 2410.11845 §4.3. Many of its co-design results were tested only in simulation or on ASIC prototypes 2410.11845 §4.2 2410.11845 §6. For phones, the transformer survey lists per-platform speed-up factors relative to FP32. INT8 gives 3.2 times on a mobile NPU 2601.03290 tab. 4. Its combined row is the product of the three rows above it, and the paper does not show that the effects multiply on real hardware 2601.03290 tab. 4.

Splitting a large model between GPU and CPU

When a model is too large for the GPU's memory, PowerInfer divides each feed-forward block by neuron. Neurons that fire often across profiled text are kept on the GPU ("hot"), and the rest are computed on the CPU ("cold") 2312.12456 §3.1 2312.12456 §4. An offline integer linear program chooses this placement from activation counts 2312.12456 §6.3. At run time, a small per-layer MLP predictor selects the neurons to compute for each token 2312.12456 §5.1. The authors' motivation is that for batch sizes below 32, computing a CPU-resident slice on the CPU is faster than transferring it to the GPU 2312.12456 §3.2.

The reported speed-ups are for batch size one, on a desktop with an RTX 4090. The average over the tested models is 8.32 tokens per second, or 7.23 times llama.cpp, and the peak gain is 11.69 times for Falcon-40B [arxiv:2312.12456 §8.2] 2312.12456 fig. 10. On a desktop with an RTX 2080 Ti and FP16 weights, the average gain is 4.71 times 2312.12456 fig. 11. The method depends on strong activation sparsity. The authors expect the largest gains for ReLU-family models, and report roughly 50% sparsity and smaller gains for SiLU-based ones 2312.12456 §4 2312.12456 §8.4. Their SiLU results are in 2312.12456 tab. 8. Card: PowerInfer.

Edge detector with cloud retraining

For an autonomous vehicle, Edge YOLO runs the detector on a Jetson-class board in the car and saves its results locally. A cloud server labels the collected data, retrains the model and sends new weights back 2205.14942 §III-A. The authors argue that cloud-only inference depends on network quality and adds delay and energy cost 2205.14942 §I. The detector is a slimmed YOLOv4-style network with a 17-layer backbone, a feature pyramid neck and three output scales 2205.14942 §III-B 2205.14942 tab. 1.

On COCO2017 the model is reported at 25.27 MB, with 26.6 frames per second on a Jetson Xavier without TensorRT and 0.473 mAP@0.5. YOLOv4 reaches 0.657 in the same table, at 2.1 frames per second 2205.14942 tab. 5. With TensorRT FP16, Edge YOLO reaches 49.5 frames per second and 0.325 mAP@0.5 on the same dataset 2205.14942 tab. 5. On KITTI 2D the TensorRT accuracy is 0.726 against 0.8212 without it 2205.14942 tab. 6. So the speed gain comes with an accuracy loss the authors' numbers show, and the choice between the two depends on the application. Card: Edge YOLO.

Split and collaborative inference

When a cloud or a nearby server is reachable, work can be divided. Edge YOLO splits the other way: the device infers and the cloud retrains 2205.14942 §III-A. The edge-LLM survey describes splitting layers between cloud and edge, and speculative decoding where a small edge model drafts tokens that a cloud model verifies 2410.11845 §4.1.1. Single-device options include token reduction, early exit once a confidence threshold is met, and offloading across GPU, CPU and disk or flash 2410.11845 §4.1.2. The survey reports no device measurements for these 2410.11845 §7. For speculative decoding itself, see speculative-decoding.

When to use which

The choice depends on the memory, compute, power and network constraints. Most guidance below is qualitative. The numbers in the cited papers are author-reported.

Two constraints recur in the sources. Memory is usually the first bottleneck 2311.11883 §3.3, and the lightweight-transformer survey assumes single-sample inference 2601.03290 §VIII-A. Batch size and sequence length change the picture, so state both when you compare numbers.

Implementation sketch

The shared core is the same across variants. The steps differ in how they compress, which runtime they use, and how the budget is checked.

# On-device inference: from a trained model to running on the device
# Input:  trained model, device budget (SRAM, flash, MACs, power), input stream
# Output: predictions made on the device

def deploy_on_device(model, budget, device, stream):
    # Shrink: quantize, prune, distil or search  [arxiv:2311.11883 §4]
    compact = compress(model, budget)
    # Reject over-budget candidates before training  [arxiv:2504.09685 §3.1]
    if not fits(compact, budget):
        raise ValueError("candidate exceeds the device budget")
    # Port: interpreter, generated or hand code  [arxiv:2211.04448 §IV.C]
    ported = port(compact, device)
    # Run each input on device; data stays local  [arxiv:2211.04448 §I]
    for sample in stream:
        yield run(ported, sample)
    # Profile on the target, then iterate  [arxiv:2601.03290 §VIII-H]

fits should check peak activation memory as well as parameter count. The search paper reports peak SRAM separately from model size 2504.09685 §4.2.2.

Contradictions

• Memory of a 7B model at FP16. The edge-LLM survey gives "over 8 GB" for the FP16 weights of LLaMA-2 7B 2410.11845 §1. Two bytes per weight for 7 billion weights is about 14 GB, and the survey does not explain the lower figure. The RobotxR1 table reports 8.4 GB for a Qwen2.5 7B run at FP16 2505.03238 tab. 3. The two use different models, and neither source defines its memory measure. Do not use either figure as a budget.
• Size reduction of a pipeline. The lightweight-transformer survey states that its six-step pipeline gives 8 to 12 times smaller models with under 2% accuracy loss 2601.03290 §VIII-H. The per-step reductions it lists (10 to 15% for distillation, 25 to 35% more for pruning, 50% more for quantization) compound to roughly 3 to 3.6 times by our arithmetic, so the paper's headline range is not derived from its own steps.
• Distillation gain. The lightweight-transformer survey reports a 1.5 to 2.5% gain for two-stage distillation, with no experiment described 2601.03290 §VIII-G. The model-search paper measures +1.32 to +1.40 points for logits distillation from ViT-B/16 on CIFAR-100 [arxiv:2504.09685 tab. 4]. The setups differ, so the two figures should not be compared directly.

These are the conflicts found across the cited cards. The surveys do not resolve them.

Papers

• A review of TinyML: workflow, deployment options and open challenges for microcontroller ML. No experiments.
• Efficient Neural Networks for Tiny Machine Learning: survey of compression methods and TinyML tools for Cortex-M targets, with a published model-size comparison.
• Accelerate Intermittent Deep Inference: search for networks that run on a batteryless MSP430 with 2 KB of SRAM. Author-reported results.
• A Review on Edge Large Language Models: survey of compression, runtime and hardware methods for edge LLMs.
• AI-ANNE: MicroPython port of Keras dense networks to a Raspberry Pi Pico. Accuracy only.
• Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?: LLM-guided search with a Pareto front for STM32-class CNNs, plus distillation.
• RobotxR1: fine-tuning a small Qwen2.5 model for a robot car, with Q5 deployment on a Jetson Orin AGX.
• Lightweight Transformer Architectures for Edge Devices: survey of small transformers, quantization, pruning and deployment tools for edge devices.
• PowerInfer: hot and cold neuron split between a consumer GPU and CPU for LLM inference. Author-reported on two desktop PCs.
• Edge YOLO: slimmed YOLOv4-style detector for Jetson boards in vehicles, with cloud retraining. Author-reported on COCO2017 and KITTI.