Lightweight Transformer Architectures for Edge Devices in Real-Time Applications

In plain words

Transformer models are large and slow, and phones, cameras and other small devices have little memory, power and compute. This survey asks which shrunken transformer designs work on such devices, and which tricks make them smaller: teaching a small model to copy a big one, cutting out parts of the network, and storing numbers with fewer bits.

The author goes through the common small transformers, the hardware and software tools vendors offer, and tables of accuracy and speed. The main takeaways are that copying a big model into a small one gives the biggest gain, that mixing lower-precision number formats works well, and that speed should be measured on the actual device rather than estimated from operation counts.

The paper's numbers are the author's summary of other work and its own analyses. Treat them as a starting point and measure on your own hardware.

TL;DR

Survey of compression and deployment options for transformers on edge hardware, including distillation, pruning, quantization, hardware-aware architecture search, and the TensorFlow Lite, ONNX Runtime, PyTorch Mobile and CoreML toolchains §I §III §IV. Its headline claim is 75-96% of full-model accuracy at 4-10x smaller size and 3-9x lower latency §VIII [abstract]. The tables are compiled from reported results; the paper describes no reproducible protocol for its own analyses. There is no official code. Read the Pitfalls before relying on any number.

Problem

Transformers have quadratic attention cost and large parameter counts, which the paper says rule them out for edge workloads with latency targets of roughly 30-100 ms, model sizes under about 100 MB and power below about 5-10 W §I-A. It lists memory, compute, power, latency and bandwidth limits of edge devices §I-B and transformer-specific challenges, including quadratic attention, parameter redundancy, quantization sensitivity, framework operator support and hardware heterogeneity §I-C.

The survey assumes single-sample (batch size 1) inference on the device §VIII-A. No assumption is given about sequence length beyond the remark that the attention cost becomes prohibitive above about 512 tokens §II-A.

Method

This is a survey, not a new method. The sections are:

1. Literature review of lightweight architectures: distilled language models (DistilBERT, TinyBERT, MobileBERT) §II-B; efficient vision transformers (EfficientFormer, EdgeFormer, MobileViT) §II-C; and efficient attention variants such as sparse, linear and dynamic token pruning §II-D.
2. Optimization techniques: INT8, FP16 and FP8 quantization §III-A, structured head and layer pruning §III-B, and latency-driven hardware-aware architecture search, citing EfficientFormer §III-C.
3. Industry tools: a comparison of edge hardware platforms tab. 1 fig. 1, four deployment frameworks §IV-B fig. 2, and optimization tools TensorRT, OpenVINO, NNCF and TVM §IV-C.
4. Benchmark tables for NLP (GLUE, SQuAD) tab. 2 fig. 3 fig. 4 and vision (ImageNet, COCO) tab. 3 fig. 5, plus a compression-accuracy tradeoff figure fig. 6.
5. Case studies and energy: cited deployments in activity recognition, drone detection and medical imaging, and energy figures §VI.
6. Open challenges: long context, multimodal models, on-device training, co-design and automated compression §VII.
7. Own analyses (§VIII): a memory-bandwidth argument for models of 15-40M parameters §VIII-A; a bit-width table §VIII-B; architecture patterns §VIII-C; speedup factors per hardware class tab. 4; real-time budgets §VIII-E; energy per inference fig. 7 §VIII-F; distillation guidance §VIII-G; and a six-step pipeline §VIII-H.
8. Recommendations §IX and conclusions §X.

The one procedure stated as steps is the six-step pipeline in §VIII-H. The paper lists it as recommended practice. It is not an algorithm with inputs, outputs or a verified implementation.

# Deployment pipeline (recommended by the paper)  [§VIII-H]
# Input:  base model, latency budget in ms, target device
# Output: compressed model (paper: 8-12x smaller, <2% accuracy loss)

def deploy(base_model, latency_budget_ms, device):
    # 1. Pick a base architecture that fits the latency budget  [§VIII-H]
    model = pick_architecture(base_model, latency_budget_ms)
    # 2. Two-stage distillation, 10-15% size reduction  [§VIII-H] [§VIII-G]
    model = distill_two_stage(model)
    # 3. Structured pruning: 30-40% of heads, 20-30% of MLP channels  [§VIII-H]
    model = prune_structured(model)
    # 4. Mixed FP16/INT8 quantization, 50% further reduction  [§VIII-H]
    model = quantize_mixed(model)
    # 5. Framework operator fusion (TensorRT, TFLite converter)  [§VIII-H]
    model = fuse_operators(model, device)
    # 6. Profile on the target device and repeat on the bottleneck  [§VIII-H]
    return profile_and_iterate(model, device, latency_budget_ms)

Hyperparameters

The paper runs no training experiments that are described, so most values below are the author's recommendations. The source column says where each value comes from. Where the paper gives no value, it is marked "not stated".

Results

Author-reported, as compiled by the survey. The paper does not state, per row, which experiments produced the values or on which hardware they were measured, except where a figure caption or table says so. Table values appear to be taken from the cited model papers.

Headline claims [abstract] §X:

• 75-96% of full-model accuracy with 4-10x smaller model size and 3-9x lower inference latency.
• Six-step pipeline with 8-12x size reduction and under 2% accuracy loss §VIII-H.
• Energy: mobile NPUs at 0.7-1.2 mJ per inference for MobileBERT-class models §VIII-F (see Pitfalls on the units).

NLP results, GLUE and SQuAD. Latency is per Fig. 4's caption, measured on Pixel 4; the caption does not give the sequence length or the runtime tab. 2 fig. 4:

tab. 2. The text cites TinyBERT-4 as 96.8% of BERT-base performance on GLUE and 9.4x faster inference §II-B. The table values give 77.0 / 79.5, about 96.9%, and 580 / 62, about 9.4x.

Vision results, ImageNet-1K and iPhone 12 latency where stated tab. 3:

The latency column mixes devices and units; see Pitfalls. The paper also reports a COCO detection gain for MobileViT over MobileNetV3 of 5.7% §II-C, with no table row.

Hardware speedup factors, relative to the FP32 baseline tab. 4:

Compression-accuracy figure fig. 6: FP16 at 50% size, 99.5% relative accuracy; INT8 at 25% size, 98.7%; pruning plus INT8 at 13% size, 96.5%. The caption reports 87% size reduction, 76% energy savings and 3.5% accuracy loss for pruning plus INT8. These values are read from a plot in the paper, which gives no underlying data.

Energy fig. 7: points for mobile NPU, mobile CPU and Jetson GPU are plotted in mJ per inference against latency.

Paper vs code

No official code. The paper gives no code repository, project page or arXiv comment. Searches for the author name and the paper title on GitHub returned no matching official repository. Several links in the bibliography go to third-party pages (SSRN, Medium, a vendor blog, market reports), not to code.

Pitfalls

• Not reproducible as written. The paper's own analyses are described as "extensive benchmarking across 50+ transformer variants", "analysis of 30+ lightweight transformer architectures", "50+ production deployments" and "extensive profiling". None of these comes with a protocol, device list, dataset, sample sizes or citation. Treat them as "not stated" for any purpose that needs evidence.
• Pipeline arithmetic does not reach 8-12x. The listed per-step reductions are 10-15% (distillation), 25-35% more (pruning) and 50% more (quantization). If they compound, the remaining size is about 28-34% of the original, so about 3-3.6x. The paper does not show how 8-12x was derived. This is our arithmetic from the paper's own steps.
• Energy units contradict each other. The text gives 0.7-1.2 mJ per inference for mobile NPUs and 3.5-6.0 mJ for mobile CPUs §VIII-F. The scatter in fig. 7 shows mobile NPU points at about 28-68 mJ per inference and mobile CPU points at about 150-240 mJ. The caption refers to "1.5-2J/s", which is a power unit. Do not use the energy numbers without checking the source.
• Speedup figure caption disagrees with its own data. fig. 4 is captioned "7-14x speedup versus BERT-base". Its data gives 2.5x for DistilBERT (580/230), 9.4x for TinyBERT-4 and MobileBERT (580/62), and 14.5x for MobileBERT-tiny (580/40). No row gives 7x.
• Memory-bound claim contradicts the table. §VIII-A says MobileBERT (25.3M) and TinyBERT-6 (67M) achieve similar latency. tab. 2 gives 62 ms and 95 ms, about 1.5x apart.
• Accuracy retention range is not supported by the tables. The abstract says 75-96% of full-model accuracy. The lowest ratio in tab. 2 and tab. 3 is about 90% (DistilBERT SQuAD, MobileViT-XS against EfficientFormer-L7). The 75% lower bound does not appear in the paper's data.
• Three different INT8 accuracy losses. The introduction gives 1-5% without calibration §I-C. §VIII-B gives 0.5-1.2% for FP32 to INT8. §IV-B says below 1% for calibrated TensorFlow Lite models. The paper does not say which setting each applies to.
• SQuAD sentence contradicts the table. §II-B says MobileBERT (F1 90.3) "match[es] TinyBERT-6". tab. 2 gives TinyBERT-6 an F1 of 87.5, a gap of 2.8 points.
• Latency columns are not comparable. tab. 3 mixes iPhone 12, Rockchip and unspecified devices, and one row gives a percentage instead of a time. The same device is not used across rows, so the comparison in fig. 5 does not show a fair speed-accuracy trade-off.
• Hardware figure and table differ. fig. 1 plots Jetson Nano at 67 TOPS and Jetson AGX at 275 TOPS. tab. 1 has no Jetson Nano row and gives Jetson AGX Orin at 40-275 TOPS.
• Abstract names COCO, but there is no COCO table. The only COCO value is one sentence in §II-C.
• Combined speedup column is a product. tab. 4 "Combined" equals the product of the three rows above it (for example 3.2 x 1.4 x 1.6 = 7.2 for the NPU). The paper does not say whether the three effects multiply on real hardware.
• Sources are weak in places. The bibliography cites a Medium post for the 5.24x JPQD compression result [intel2023jpqd], a DZone article, a vendor blog, and a market-research report for hardware claims. The EdgeFlex-Transformer entry has arXiv 2512.19741 with year 2024. The FP8 coverage numbers (92.64% vs 65.87%) and the EdgeViT++ savings (65% memory, 40% latency) rest on those sources. Check each before use.
• Hybrid depth-width ratios and temperatures have no stated source. Values in §VIII-C and §VIII-G are presented as optimal without experiments or citations.
• Section and table numbers are inferred. The card's section labels (for example §VIII-H) follow IEEEtran's automatic numbering as it appears in the LaTeX source. We did not check them against a compiled PDF.
• Figure 6 and the energy plot are read by eye. The paper gives no underlying data for them.
• Hardware speedups are per-platform averages, not per-model. No model or workload is named for tab. 4.

Lineage

• Builds on: DistilBERT (arXiv:1910.01108), Linformer (arXiv:2006.04768), EdgeFormer (arXiv:2203.03952). The paper's bibliography gives these ids. It cites TinyBERT, MobileBERT, EfficientFormer, MobileViT, Performer, BERT, ViT and the original Transformer without arXiv ids.
• Compared against: BERT-base, DistilBERT, TinyBERT, MobileBERT, EfficientFormer, MobileViT, MobileNetV2 and EdgeFormer, all within the paper's own tables.
• Superseded by: not known from content in this wiki.