Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models

In plain words

AI models are often run in a data centre. This survey looks at the other option: running them on the device where the data is made, such as a phone, a camera, a sensor or a car. The paper argues that this gives quick answers, keeps personal data on the device and avoids sending everything over the network. The price is that a phone or sensor has far less memory, power and compute than a server.

The authors organise the topic like a field guide. They first say what counts as an on-device model and how the pipeline works, then list where it is used. They name the problems that come with small devices, and then group the fixes into three layers: how the data is prepared, how the model is made smaller, and how the software and hardware are set up. They close by looking at likely future developments and the social questions they raise.

The paper proposes no new method and runs no experiments of its own. It is a map of the field, and the numbers in it are copied from the works it cites.

TL;DR

• A survey (7 authors, arXiv v2, 2025) of AI models that perform inference and, in part, training on edge and terminal devices §1.1.
• Covers fundamentals, applications, device challenges, optimisation (data, model, system), and future trends §1.2.
• Its optimisation chapter is the most useful part for implementers: a catalogue of compression methods and frameworks with qualitative trade-offs §5 tab. 4 tab. 5.
• Contains no experiments, no literature-selection protocol and no code; quantitative claims are quoted from the cited works §1.2.

Problem

The paper frames the problem as a conflict. Cloud-hosted models can be large and accurate but add latency, bandwidth cost and privacy exposure, while large models are hard to fit on constrained devices §1 tab. 3. The survey treats on-device AI as a response, and identifies the main obstacles as limited compute, limited storage and memory, energy consumption, communication bandwidth, data privacy and security, and the difficulty of moving and adapting models across heterogeneous devices and changing environments §4.1–§4.6.

Method

The survey is a narrative review, not a systematic one. The paper states it reviews and analyses relevant literature, and organises the review around four research questions §1.2:

• RQ1: applications of on-device models in daily life §3.
• RQ2: technical challenges of deploying them §4.
• RQ3: optimisation and implementation methods §5.
• RQ4: future trends §6.

The paper does not state how sources were found, which databases were used, what the inclusion rules were, or what time window was covered §1.2. Section numbers above follow the compiled paper.

Taxonomy of the optimisation chapter §5 fig. 5:

• Data optimisation §5.1 fig. 3 tab. 4: data filtering, feature extraction, data aggregation, data quantisation, and edge computing frameworks.
• Model optimisation §5.2 fig. 4 tab. 5: parameter sharing, pruning, model quantisation, knowledge distillation, low-rank factorisation, hardware-aware neural architecture search (NAS), and energy-efficient model design. The section also reviews traditional (non-neural) compression methods before these, and cites Deep Compression as an example that combines pruning, quantisation and Huffman coding §5.2.
• System optimisation §5.3 fig. 5: software, split into on-device learning frameworks tab. 7 and inference frameworks tab. 8, and hardware, covering CPU, GPU, FPGA, ASIC and NPU accelerators tab. 9 with a literature summary of reported gains tab. 10.

Pipeline view §2.2 fig. 2. The paper describes deployment as data processing, then model development, then system integration. It notes that not every step is needed in practice fig. 2 caption.

# Deployment pipeline, as the survey describes it  [§2.2] [fig. 2]
# Input:  data produced on the device, a trained model
# Output: a model that runs locally within resource limits

def deploy_on_device(raw_data, model):
    # Prepare the data: filter noise, extract features, aggregate
    # sources and lower precision  [§5.1] [tab. 4]
    cleaned = filter_noise(raw_data)
    features = extract_features(cleaned)
    small_data = quantize_data(aggregate_sources(features))

    # Shrink the model with one or more methods: sharing, pruning,
    # quantisation, distillation, factorisation, or a light design
    # found by hardware-aware NAS  [§5.2] [tab. 5]
    model = share_weights(model)
    model = prune(model)
    model = quantize_weights(model)
    model = distill_into_student(model, small_data)
    model = factorize_low_rank(model)

    # Choose a software framework and target hardware  [§5.3] [tab. 7-9]
    runtime = pick_inference_framework(model)
    target = pick_hardware(runtime)

    # Check the deployment constraints the paper lists: latency,
    # memory, energy, privacy  [§4] [§2.1.3]
    return deploy(model, runtime, target)

The survey does not give a procedure with fixed steps and does not say that these methods must be combined in this order. The sketch only reflects the survey's own grouping. Function names are ours, not the paper's.

Hyperparameters

The paper proposes no training or inference procedure, so it states no hyperparameters. The only numeric settings in the text are examples quoted from other works:

Results

Author-reported. The survey runs no experiments. Its numbers are quoted from cited works and were not re-measured by the authors. The paper does not always name the dataset or hardware for each number.

Model accuracy and latency on phones tab. 6, a table of PyTorch Mobile results taken from a cited comparison [luo2020comparison]. Time is labelled "Time (ms)"; the paper does not define it. Accuracy is in percent.

Accuracy and cost of lightweight and NAS-designed models §5.2, quoted in the running text. The survey gives these as examples of trade-offs, not as a comparison with one shared setup.

Qualitative comparison of on-device and cloud models tab. 3: on-device models are rated worse on compute and scalability and on maintenance, and better on latency and privacy. The table gives no measured values.

Paper vs code

No official code.

The paper has no code link, no project page and no arXiv comment field is stored in our metadata. A GitHub repository search for "on-device ai survey" returned only unrelated projects, none by the authors. The paper is a survey, so no official implementation is expected.

Pitfalls

• No literature protocol. The paper does not say how it searched, what it excluded, or what period it covers §1.2. Do not treat its coverage as exhaustive or reproducible.
• Numbers are not checked. Figures such as the 5.3 billion MACs for EfficientNet-B0 and the 75% Gartner claim in the introduction are quoted as the paper states them §5.2 §1. Check them against the cited source before relying on one.
• Mixed setups. The NAS and efficiency numbers in §5.2 come from different works, datasets and hardware. The paper does not give the dataset for several of them (for example MobileNetV2 and EfficientNetV2). Do not compare them across rows as if they were measured together.
• Table 6 accuracies are nearly identical across phones. For each model the accuracy is the same on the first four phones and differs only on the Oppo R17. The times vary widely. The paper does not explain this. It is consistent with accuracy being reported once per model, not measured per device. Use the times with care.
• Overlapping categories. Data quantisation §5.1 and model quantisation §5.2 are separate entries. Energy-efficient design §5.2 overlaps with hardware-aware NAS §5.2. The model-type table lists transfer learning next to ML and deep learning tab. 2, which mixes a training method with model families.
• Tables not in the paper. The lightweight-models table in the source is inside a LaTeX comment block and is not in the compiled paper. It is not counted here or cited.
• Figures. Figures are described from their captions only. No figure is reproduced on this card.
• Hidden costs the survey flags but does not quantify. Pruning and NAS may need retraining or heavy search; hardware accelerators may not generalise across models or platforms tab. 5 tab. 9. Some framework limitations are listed, such as limited non-ONNX or non-NCNN model support tab. 8.
• Sustainability and ethics are discussed as open issues, with no metrics or proposed methods §6.3 §6.4.

Lineage

• Builds on: none stated. The paper positions itself against earlier surveys of edge and mobile AI, communication and computation offloading, on-device language models and on-device machine learning, and says those surveys do not cover the optimisation side in one place §1.4. Named authors only; no cards exist for those works.
• Compared against: the related surveys in §1.4.
• Superseded by: not known from content on this wiki.