A Review on Edge Large Language Models: Design, Execution, and Applications

In plain words

Phones, cars and small robots can run language models, but the big models that power chat assistants are too heavy for them. They need too much memory and compute, and the chips inside these devices differ a lot from one another.

This survey collects the ways people make such models fit. One group of methods shrinks the model before it ships: storing its numbers with fewer bits, cutting out parts that matter little, teaching a small model to copy a big one, or building a compact model from the start. Another group changes how the model runs on the device: splitting work between the device and a server, choosing which parts to compute, using the chip's special hardware, and picking the right software library. The last part lists example apps that already run this way, such as phone assistants, meeting summaries and driving aids.

The authors argue that these steps belong together as one pipeline, and that many hardware tricks look good in simulation but are hard to use on ordinary hardware. The paper does not test any method itself, so read it as a map of the field rather than as a source of benchmark numbers.

TL;DR

• A survey, not a method: no algorithm, no training run and no experiments of its own §1 §7.
• Covers the full edge pipeline: offline compression §3, runtime optimizations §4 and applications §5.
• Good for choosing which family of techniques to look at. For numbers, follow the cited original works.
• No official code. The LaTeX source contains no code repository.

Problem

The paper states that LLMs need far more memory and compute than most edge devices have, so loading and running them is slow or fails §1. It compares the growth of estimated pre-training compute with the growth of edge-device AI throughput, and says the gap is widening §2 fig. 3. Earlier surveys covered edge deep learning for CNNs and RNNs, and general efficient-LLM surveys do not cover edge deployment §2.

Method

The paper has no algorithm. Its contribution is a taxonomy and a set of comparisons. The sketch below shows the order the survey uses, with the section that covers each stage. The helper functions are not in the paper.

# Edge LLM pipeline in the order the survey gives  [§1] [fig. 1]
# Input:  a pretrained LLM and a target edge device
# Output: an application running the compact model

def edge_llm_pipeline(model, device):
    # 1. Compress before deployment: quantize, prune, distill or redesign  [§3]
    compact = offline_compress(model)
    # 2. Choose runtime techniques for this device and network  [§4]
    runtime = choose_runtime(compact, device)
    # 3. Serve the model to on-device applications  [§5]
    return build_application(compact, runtime)

The structure below follows the paper's section numbers.

Pipeline. Pre-deployment compression makes a compact model, which is then deployed and tuned at runtime, and finally used by applications §1 fig. 1. Figure 2 gives the overall taxonomy fig. 2.

Offline pre-deployment design §3.

1. Quantization §3.1: fewer bits for weights (weight-only) or for weights and activations together (weight-activation co-quantization). Weight-only examples include LLM.int8(), GPTQ and AWQ §3.1.1. Co-quantization examples include ZeroQuant and SmoothQuant §3.1.2. Outlier weights or activations are the main difficulty in both cases §3.1.1 §3.1.2.
2. Pruning §3.2: structured pruning removes whole components such as heads, channels or layers (e.g. CoFi, LLM-Pruner, SliceGPT) §3.2.1. Unstructured pruning removes single weights and leaves sparse models (e.g. Movement Pruning, oBERT, SparseGPT, Wanda, BESA) §3.2.2.
3. Knowledge distillation §3.3: white-box methods use the teacher's internal features and logits (e.g. MiniLM, TinyBERT, MiniLLM) §3.3.1. Black-box methods use only teacher outputs, often as chain-of-thought rationales (e.g. Distilling Step-by-Step, Lion) §3.3.2.
4. Low-rank approximation §3.4: replace a weight matrix W (m x n) by the product of two thin matrices U (m x r) and V^T (r x n), with r much smaller than m and n §3.4. Examples include ALBERT, FWSVD, DRONE and LoSparse.
5. Complementary design §3.5: better data curation, grouped-query or multi-query attention, rotary position embeddings, and layer-wise scaling, as used in compact model families fig. 7.

Runtime inference §4.

1. Software level §4.1.
   • Cloud and multi-edge collaboration §4.1.1: split inference divides the layers between cloud and edge. Speculative decoding has an edge model draft tokens that the cloud model verifies tab. 1 fig. 9.
   • Single-device scheduling §4.1.2: token reduction (dropping or compressing tokens), early exit (stop once a confidence threshold is met), and dynamic offloading across GPU, CPU and disk or flash tab. 2 fig. 10.
   • Framework level §4.1.3: runtimes and engines such as ExecuTorch, operator fusion methods, PowerInfer and vLLM's paged KV cache tab. 3.
2. Hardware-software co-design §4.2: sparsity matched to hardware (ASIC designs such as SpAtten and Sanger; in-memory designs such as TransPIM and TranCIM) §4.2.1, and low-bit or adaptive number formats (GOBO, Mokey, OliVe, AdaptivFloat, ANT) §4.2.2. Compared in tab. 4.
3. Hardware level §4.3: CPUs, GPUs and NPUs, with examples from phone and edge SoCs and NVIDIA Jetson modules §4.3.1 §4.3.2 §4.3.3 tab. 5.

Applications §5. Personal agents (e.g. AutoDroid), healthcare assistants, companion robots §5.1; message completion, meeting summarization and computer-operation agents §5.2; autonomous driving, fault localization and anomaly detection §5.3 fig. 11.

Open challenges §6: Mamba-style architectures on edge hardware, edge-cloud collaboration with fault tolerance, simulation-to-hardware gaps, graph reasoning, multi-agent coordination and on-device continual learning.

Selection guidance §3.6. The paper gives qualitative advice, not rules:

• Strict memory with moderate compute: weight-only quantization (e.g. LLM.int8()) and structured pruning (e.g. CoFi) §3.6.
• Maximum compression with some accuracy loss tolerated: weight-activation co-quantization (e.g. SmoothQuant) and unstructured pruning (e.g. Movement Pruning) §3.6.
• Distillation, with white-box or black-box chosen by whether the teacher's internals are accessible §3.6 §3.3.
• Low-rank methods for large, redundant matrices such as embeddings §3.6.
• Combining techniques often works better; the paper cites ZeroQuant as an example of quantization plus distillation §3.6.
• Designing an efficient architecture from scratch can be best when feasible, but the paper notes this is usually done by large companies §3.6 §3.5.

Hyperparameters

Not applicable. The paper reports no training, fine-tuning or inference configuration of its own; its scope is stated in §7. Settings that appear in the text belong to the cited works, and the paper does not reproduce them.

Results

Author-reported. Setup: a literature survey with no experiments of its own §1 §7. The figures below are quoted from the text. The paper gives no table of benchmark numbers that it measured itself.

The paper also states that LLM pre-training compute has grown much faster than edge AI performance, measured as TFLOPs against TOPS §2 fig. 3. It gives the heuristic C ≈ 6ND, where N is the parameter count and D the number of training tokens §2.

Paper vs code

No official code. The paper has no project page and its LaTeX source has no repository link; the only URL in the text is a Hugging Face home page §3.5 (footnote). The GitHub repository search for "edge llm survey zheng" returned no items. Awesome-list repositories are not official code for a survey. No discrepancies found to check.

Pitfalls

• Survey, not a method. There is nothing to implement. Anyone who needs a specific technique should read the original paper for it.
• The 8 GB figure is low. A 7B model at 2 bytes per weight is about 14 GB of weights alone (7e9 x 2 bytes), not "over 8GB" as the paper says §1. The paper does not explain the number. Do not use it for budgeting, and compute memory from the parameter count and the number format.
• Compute and memory claims are only as good as the cited sources. The paper reports results from other papers. The co-design table tab. 4 and the hardware table tab. 5 compare numbers taken from those works. Check each one against its source before relying on it.
• Simulation versus hardware. The authors say many co-design methods were tested only in simulation or on ASIC prototypes §4.2 §6. Treat such speedups as unverified on commodity devices.
• Commented-out text. The LaTeX source contains commented-out subsections (for example on FPGAs and ASICs in the runtime section). They are not in the compiled paper and should not be cited.
• Versions. The metadata record is arXiv v2 (updated 2025-02-23); the first version is dated 2024-09-29. The card's published date is the first version's date, and the license is the one recorded by fetch.
• Scope of "edge". The paper lists some frameworks as edge options that are not described with device measurements in the text (e.g. vLLM §1 §4.1.3). Check the target hardware before applying a technique.
• Ambiguous sentences. Some passages in the framework and hardware sections make claims without numbers (e.g. "low latency", "linear speed-ups"). Look for the number in the cited work before quoting the claim.

Lineage

• Builds on: the efficient-LLM and edge-computing surveys it cites, in particular Xu et al. (model compression) and Wan et al. (efficient LLMs) §2. Named, not linked: no card exists for them.
• Compared against: Qu et al. (mobile edge intelligence for LLMs), which the paper says lacks framework- and hardware-level analysis, and Li et al. (personal LLM agents), which it says lacks runtime analysis §2.
• Superseded by: not known from content in this wiki.