---
kind: concept
slug: post-training-quantization
title: Post-training quantization
summary: >-
  Converting a trained network to integer weights, and optionally activations, without retraining. Per-channel weight ranges keep ImageNet CNNs near float accuracy; learned rounding (AdaRound) helps 4-bit weights. For LLMs, outlier features call for a 16-bit path (LLM.int8()), smoothing into the weights (SmoothQuant), or weight-only 3- to 4-bit rounding (GPTQ).
tags: [quantization, efficient-inference, edge-deployment, inference, llm-inference, microcontroller]
papers: ["1806.08342", "2004.10568", "2106.08295", "2110.00791", "2112.06126", "2204.06806", "2208.07339", "2210.16556", "2210.17323", "2211.10438", "2301.00774", "2303.13569", "2303.17568", "2311.11883", "2408.15865", "2410.11845"]
related: [weight-quantization, quantization-aware-training, batchnorm-folding, adaptive-rounding, llm-serving, post-training-pruning, mixed-precision-quantization, memory-planning-for-inference, posit-number-format, on-device-inference, network-pruning]
status: draft
updated_at: 2026-10-25
---

## Overview

Post-training quantization (PTQ) takes a trained float model and maps its weights, and often its activations, onto a small integer grid. It does not change the training loop. The only extra inputs are the trained model and, when activations are quantized, a small set of representative inputs for range estimation [arxiv:1806.08342 §3.1]. The shared mechanism is a uniform affine grid with a scale $s$ and an integer zero-point $z$, so that $x_{float} = s\,(x_Q - z)$ [arxiv:1806.08342 §2.1, eq. 3] [arxiv:2106.08295 §2.2, eq. 6].

The measurements in the first paper give the key constraint for convolutional networks. At 8 bits, PTQ on ImageNet CNNs stays close to float accuracy, but only when the weight ranges are chosen per output channel. Per-layer ranges can drop compact networks to chance-level accuracy [arxiv:1806.08342 tab. 2]. The paper traces this to batch-norm folding, which makes the kernel ranges within one layer differ widely [arxiv:1806.08342 §3.1.3, app. A].

The second paper, a white paper by Nagel et al., turns these results into a fixed order of PTQ steps. It adds cross-layer equalization, bias correction and AdaRound, and it measures detection, segmentation and BERT as well as ImageNet classifiers [arxiv:2106.08295 §3.5, tab. 6]. Its tables go below 8 bits for weights, so the first paper's 8-bit limit should not be read as general (see Lower-bit PTQ below). The AdaRound paper treats the rounding step on its own: it picks each weight's rounding direction by minimising a layer-wise reconstruction error on unlabelled inputs, which lifts 4-bit ResNet-18 from 23.99% to 68.60% top-1 [arxiv:2004.10568 tab. 6].

Large language models need a different approach: their outlier feature dimensions break per-row scales, and LLM.int8() keeps those dimensions in 16 bits [arxiv:2208.07339 §3.2]. SmoothQuant avoids the 16-bit path instead. It divides outlier activation channels by per-channel factors and multiplies the matching weight rows, so every matrix product runs in INT8 [arxiv:2211.10438 §4]. A second LLM approach leaves activations in float and rounds only the weights to 3 or 4 bits, correcting the rounding error layer by layer (GPTQ) [arxiv:2210.17323 §3, §6]. A third, simpler option appears in a 13B code model: plain 8-bit absmax on the weights, with float activations [arxiv:2303.17568 §2.4].

Other surveys place these results in a wider frame. Weng's survey defines the affine grid used here and describes PTQ as calibration on a small set of unlabelled data, with 4 bits as the lowest precision it reports for PTQ [arxiv:2112.06126 §3.1, §4.2, tab. 1]. The TinyML reviews apply the same steps under tighter limits: a few kilobytes of RAM, no floating-point unit on the smallest Cortex-M parts, and integer-only inference [arxiv:2311.11883 §3.2, §3.3]. The microcontroller and edge-LLM variants below cover them.

## Variants

### Weight-only PTQ

Weights are rounded to 8 bits and inference still runs in float. No data is needed. The first paper says this only saves storage [arxiv:1806.08342 §3.1.1, fig. 2]. With asymmetric per-channel weights, the nine networks in tab. 2 stay within about 2 points of float top-1, and the MobileNet-v2 variants are the largest drops [arxiv:1806.08342 tab. 2]. Per-layer asymmetric weights are the failure case: the three MobileNet rows drop to 0.001 to 0.004 [arxiv:1806.08342 tab. 2]. The second paper's range-setting table agrees at 8 bits: per-channel MSE ranges keep ResNet-18 at 69.66 against 69.68 float, and MobileNetV2 at 71.46 against 71.72 [arxiv:2106.08295 tab. 1]. At 4 bits the same range setting falls to 54.67 on ResNet-18 and 27.17 on MobileNetV2 [arxiv:2106.08295 tab. 1]. Card: [Quantizing deep convolutional networks for efficient inference](/papers/1806.08342). Card: [A White Paper on Neural Network Quantization](/papers/2106.08295).

### Weights and activations with calibration

Activations are also quantized to 8 bits, using per-layer ranges. Those ranges come from a moving average of per-batch minimum and maximum over about 100 mini-batches [arxiv:1806.08342 §2.5, §3.1.2]. Activation ranges are never per-channel, because per-channel activations would complicate the inner products of convolutions and matrix products [arxiv:1806.08342 §2.6]. With asymmetric per-channel weights, the same networks stay close to float: for example, MobileNet-v1 reaches 0.703 against 0.709 float [arxiv:1806.08342 tab. 3]. The first paper concludes that almost all of the accuracy loss comes from the weights [arxiv:1806.08342 §3.1.3]. The activation-only column in tab. 3 is an exception for MobileNet-v2, at 0.700 against 0.719 float, so activations are not free for every architecture [arxiv:1806.08342 tab. 3].

For the same setting, the second paper's per-channel W8A8 results are within 1.5 points of float for all eight models in its tab. 6, and per-tensor weights lose 0.73 points on MobileNetV2 (70.99 against 71.72) [arxiv:2106.08295 tab. 6]. Its default is symmetric weights and asymmetric activations, because asymmetric weights add a data-dependent term at inference [arxiv:2106.08295 §2.4.1].

### Full PTQ pipeline (with or without data)

The second paper gives a standard order of steps [arxiv:2106.08295 §3.5]:

1. Cross-layer equalization of the float model. Consecutive layers are rescaled per channel without changing the output. It is applied inside residual blocks only [arxiv:2106.08295 §3.2].
2. Add quantizers: symmetric weights, asymmetric activations, per-channel weights where the hardware allows [arxiv:2106.08295 §2.4.1].
3. Weight range setting by layer-wise MSE. Min-max may be better for per-channel weights [arxiv:2106.08295 §3.1].
4. Rounding optimisation (AdaRound) if about 500 to 1000 unlabeled images are available. The paper calls this crucial for 4-bit weights [arxiv:2106.08295 §3.4, §3.5].
5. Otherwise, analytic bias correction, when the network has batch normalization [arxiv:2106.08295 §3.3].
6. Activation range setting: MSE on a small calibration set, or BN-based ranges for a data-free pipeline [arxiv:2106.08295 §3.1].

Each step matters on its own. On MobileNetV2 with INT8 weights and activations, cross-layer equalization moves accuracy from 0.12 to 69.91, and bias absorption adds to 70.92 [arxiv:2106.08295 tab. 3]. Analytic bias correction alone moves the same model to 52.02 [arxiv:2106.08295 tab. 4]. The data-free route uses only cross-layer equalization, bias absorption and BN-based ranges. BN-based activation ranges give 71.32 on MobileNetV2 at 8 bits, against 71.72 float [arxiv:2106.08295 tab. 2], but 0.66 at 4 bits [arxiv:2106.08295 tab. 2].

Rounding by round-to-nearest is not the best choice for the task loss when weights are quantized post-training [arxiv:2106.08295 §3.4]. On ResNet-18, AdaRound raises the first-layer result from 52.29 to 69.58, and over all layers from 23.99 to 68.60. The table caption does not state the bit-width, although the text's first-layer experiment is 4-bit [arxiv:2106.08295 tab. 5]. The AdaRound paper reports the same first-layer pair, 52.29 and 69.58 [arxiv:2004.10568 tab. 2]. AdaRound's optimiser settings are not stated in the second paper [arxiv:2106.08295 §3.4].

When PTQ misses the float baseline, the second paper gives a debugging flow. Check the float model with the quantizers bypassed, quantize weights and then activations separately, and then analyse one quantizer at a time [arxiv:2106.08295 §3.7].

### Lower-bit PTQ (4 bits)

At 4 bits the results depend on the pipeline. With the first paper's methods, PTQ is unreliable on its networks. Asymmetric per-layer ranges run from 0.001 to 0.5 top-1, and symmetric per-channel ranges from 0.001 to 0.74 [arxiv:1806.08342 tab. 5]. The same models reach 0.62 to 0.76 top-1 after quantization-aware training [arxiv:1806.08342 tab. 5]. With 4-bit activations and 8-bit symmetric per-channel weights, PTQ gives 0.04 to 0.59 [arxiv:1806.08342 tab. 6].

With the full second-paper pipeline, 4-bit weights and 8-bit activations (W4A8), per-channel weights, reach within about 2 points of float on seven of eight models. ResNet-18 gives 68.91 against 69.68 [arxiv:2106.08295 tab. 6]. The exception is EfficientDet-D1, at 35.08 against 40.08 mAP [arxiv:2106.08295 tab. 6]. Per-tensor W4A8 fails on EfficientDet-D1 at 0.31 mAP [arxiv:2106.08295 tab. 6]. Quantization-aware training remains higher on the same models: ResNet-18 at W4A8 per-channel reaches 70.01 [arxiv:2106.08295 tab. 10]. Quantization-aware training is covered in [its concept page](/concepts/quantization-aware-training).

The second paper's PTQ table covers W8A8 and W4A8 only [arxiv:2106.08295 tab. 6]. Its W4A4 results are from quantization-aware training, which reaches 68.32 (per-tensor) and 68.83 (per-channel) on ResNet-18 [arxiv:2106.08295 tab. 10].

### Learned weight rounding for CNNs (AdaRound)

AdaRound decides, for each weight, whether to round down or up on the grid, instead of always taking the nearest value [arxiv:2004.10568 §3]. For one layer it minimises a reconstruction error on unlabelled inputs. The rounding choices are relaxed to continuous variables, and a regulariser pushes each one to 0 or 1 by the end of optimisation. No fine-tuning is done [arxiv:2004.10568 §3.3] [§5]. Most ablations use 1024 ImageNet images, and the authors report about 10 minutes for ResNet-18 on one GTX 1080 Ti [arxiv:2004.10568 §5].

With per-layer symmetric 4-bit weights and float activations, ResNet-18 goes from 23.99% top-1 with nearest rounding to 68.60%, against 69.68% float [arxiv:2004.10568 tab. 6] [tab. 4]. ResNet-50 goes from 35.60 to 75.23 against 76.07, and InceptionV3 from 1.67 to 75.76 against 77.40 [arxiv:2004.10568 tab. 7]. The gain depends on the grid. With min-max scales, nearest rounding gives 0.23 on ResNet-18 and AdaRound gives 61.96 [arxiv:2004.10568 tab. 6]. The method is for convolutional and linear layers. Other module types are not covered [arxiv:2004.10568 §1] [§3.3]. Card: [Up or Down? Adaptive Rounding for Post-Training Quantization](/papers/2004.10568). The regulariser weight, the beta schedule and the learning rate are not stated in the paper. See [adaptive rounding](/concepts/adaptive-rounding) for the shared idea.

### LLM-scale PTQ with outlier features

In the paper's OPT and fairseq models, a few feature dimensions with large magnitude appear from about 6.7B parameters. Their values reach about 20 times the others [arxiv:2208.07339 §1, §4.2, app. C]. Once they appear, one absmax scale per tensor, or per row of the activations, loses precision for every other value. Vector-wise absmax without the 16-bit path gives C4 perplexity 14.13 against 13.30 float at 6.7B, and 16.48 against 12.45 at 13B [arxiv:2208.07339 tab. 1].

LLM.int8() keeps the feature dimensions whose magnitude reaches 6.0 in 16 bits and runs the rest in int8, with one scale per row of the activations and per column of the weights [arxiv:2208.07339 §3.1, §3.2]. Its perplexity matches float: 13.24 at 6.7B and 12.45 at 13B [arxiv:2208.07339 tab. 1]. Weight memory is about half of 16-bit, and the decomposition adds about 0.1% memory [arxiv:2208.07339 §3.2]. The method is for inference of pretrained models. Attention products stay in 16 bits, and training at scale is not solved [arxiv:2208.07339 §6].

Speed is not a given. At model dimension 4096, the 6.7B size, LLM.int8() runs at 0.86 times the 16-bit speed, and at 5140 it reaches 1.22 times [arxiv:2208.07339 app. D.1]. The paper sets the threshold to 6.0 [arxiv:2208.07339 §3.2]. The official reference code, as checked in the card, defaults its threshold to zero, which switches the 16-bit path off, so the threshold must be set by hand. See the card's Paper vs code section: [LLM.int8()](/papers/2208.07339).

### All-INT8 matrix products with smoothing (SmoothQuant)

SmoothQuant keeps every linear layer and the two attention matrix products in W8A8, with no 16-bit outlier path [arxiv:2211.10438 §4] [fig. 6]. Before quantization it applies a per-input-channel factor $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$. The factor divides the activation channel and multiplies the matching weight row, so the product is mathematically unchanged [arxiv:2211.10438 §4, eq. 3, eq. 4]. When the activation comes from a LayerNorm, the division can be folded into that LayerNorm offline [arxiv:2211.10438 §4]. The per-channel maxima come from calibration on 512 Pile sentences [arxiv:2211.10438 §5.1].

Author-reported: naive W8A8 reaches 35.5% average accuracy on OPT-175B, against 66.9% for float. SmoothQuant at the static O3 setting gives 66.8% and WikiText perplexity 11.17, against 10.99 float [arxiv:2211.10438 tab. 3]. On BLOOM-176B the O1, O2 and O3 settings give 68.3, 68.4 and 67.4, against 68.2 float [arxiv:2211.10438 tab. 4]. Speed is up to 1.56 times FP16 in FasterTransformer, with about half the memory [arxiv:2211.10438 fig. 9, §5.3]. A 530B model runs on 8 GPUs instead of 16 [arxiv:2211.10438 tab. 10].

The migration strength $\alpha$ is the main knob. The paper's sweet spot is 0.5 for OPT and BLOOM, and it uses 0.75 for GLM-130B and 0.8 for LLaMA [arxiv:2211.10438 §4] [tab. 6]. Above about 0.6 the weights become hard to quantize, and below about 0.4 the activations do [arxiv:2211.10438 §5.5, fig. 10]. Coarser settings are faster but cost accuracy: the O3 setting lost about 0.8 points on BLOOM-176B [arxiv:2211.10438 §5.2]. Card: [SmoothQuant](/papers/2211.10438).

### Weight-only layer-wise rounding for LLMs (GPTQ)

GPTQ quantizes only the weights, to 3 or 4 bits, and leaves activations in float16 [arxiv:2210.17323 §6]. It works one linear layer at a time. For each layer it minimises the squared error between the float output and the quantized output on a small calibration set [arxiv:2210.17323 §3]. Each weight is rounded in turn, and the columns not yet rounded are adjusted to absorb the error. The method starts from Optimal Brain Quantization and fixes one column order for all rows, which lowers the per-layer cost to a level that scales to 175B-parameter models [arxiv:2210.17323 §3, §4 Step 1]. The calibration set is 128 random 2048-token segments of C4 [arxiv:2210.17323 §5 Setup]. Card: [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](/papers/2210.17323).

The paper's numbers are author-reported. At 4 bits on OPT-175B, WikiText2 perplexity is 8.37 against 8.34 float, while round-to-nearest gives 10.54 [arxiv:2210.17323 tab. 5]. At 3 bits, GPTQ gives 8.68 on OPT-175B and 8.64 on BLOOM-176B [arxiv:2210.17323 tab. 5]. Round-to-nearest at 3 bits is far worse, from about 17 to above 13,000 perplexity across the OPT and BLOOM sizes in tab. 3 and tab. 4 [arxiv:2210.17323 tab. 3, tab. 4]. At 6.7B, ungrouped 3-bit GPTQ gives 14.86 against 10.86 float [arxiv:2210.17323 tab. 3]. Grouped results are reported only for the 175B-class models [arxiv:2210.17323 tab. 5]. Group size 128 adds about 0.15 bits per weight, and group size 1024 about 0.02 [arxiv:2210.17323 §5 175 Billion Parameter Models].

Quantization time is a few hours on one GPU at the largest size: from 2.9 minutes for BLOOM-1.7B to 4.2 hours for OPT-175B on one A100 [arxiv:2210.17323 tab. 2]. At batch size 1 generating 128 tokens, 3-bit OPT-175B takes 71 ms per token on one A100, against 230 ms for FP16 on five A100s [arxiv:2210.17323 tab. 6]. That gain comes from reading fewer weight bytes and from the multi-GPU baseline, not from cheaper multiplication [arxiv:2210.17323 §6]. The paper also reports 2-bit grouped results, with OPT-175B at 8.94 perplexity at group size 32 [arxiv:2210.17323 tab. 7].

### Joint pruning and quantization (SparseGPT)

SparseGPT removes weights in one pass, without retraining. It chooses the mask block by block and updates the surviving weights with a Cholesky-based error compensation close to GPTQ's [arxiv:2301.00774 §3.2, §3.4]. Its joint mode also rounds the kept weights, so later pruning decisions see earlier rounding errors and the reverse [arxiv:2301.00774 §3.5].

On OPT-175B, 50% unstructured sparsity with 4-bit weights gives WikiText2 perplexity 8.29. GPTQ at 3 bits gives 8.68 in the paper's comparison, which the card records as the same storage [arxiv:2301.00774 §4.1] [app. C]. The same paper reports 8.55 for 2:4 with 4-bit weights and 8.85 for 4:8 with 4-bit weights [arxiv:2301.00774 §4.1]. The pruning-only result at 50% is 8.21 against 8.35 dense [arxiv:2301.00774 tab. 1]. The calibration uses 128 C4 segments of 2048 tokens, and pruning OPT-175B takes about 4 hours on one A100 [arxiv:2301.00774 §4]. Card: [SparseGPT](/papers/2301.00774).

### Weight-only absmax INT8 for a code model (CodeGeeX)

CodeGeeX, a 13B decoder-only model, quantizes each linear layer's weights from FP16 to INT8 with the absolute maximum as scale, and multiplies INT8 weights with FP16 activations in a custom kernel [arxiv:2303.17568 §2.4, eq. 4]. The paper describes no calibration step. Its reported inference memory for 128-token output on an A100 is 14.7 GB at INT8, against 26.9 GB at FP16. The authors also say FP16 runs out of memory on one RTX 3090 [arxiv:2303.17568 tab. 4, §2.4]. Its time figure, 1.61 s with FasterTransformer INT8 against 3.66 s with PyTorch FP16, changes both framework and precision, so it does not isolate the quantization effect [arxiv:2303.17568 tab. 4]. The card's results give no accuracy comparison for INT8 weights, so measure accuracy before using this approach. The released weights are under a non-commercial licence, so check that before reuse ([card](/papers/2303.17568)). Card: [CodeGeeX](/papers/2303.17568).

### Detector with a keypoint head (YOLO-Pose)

YOLO-Pose quantizes a YOLOv5-based pose detector after training, with no quantization-aware training [arxiv:2204.06806 §4.5]. Before quantizing, the authors retrained the model with ReLU in place of SiLU, which costs about 1 to 2% [arxiv:2204.06806 §4.5]. On the YOLOv5s6-ReLU model at 960 input, val2017 AP is 61.9 in float, 61.8 at 16 bits, 55.6 at 8 bits, and 60.6 when about 30% of layers are kept at 16 bits [arxiv:2204.06806 tab. 5]. The mixed setting therefore recovers most of the 8-bit loss, and it is 1.3 AP below float in the table (the text says 1.2) [arxiv:2204.06806 tab. 5, §4.5]. The card does not say which layers stay at 16 bits, nor the weight and activation granularity or the calibration method. So these numbers cannot be set directly against the per-channel results above. Card: [YOLO-Pose](/papers/2204.06806).

### Gesture CNN converted with TensorFlow Lite

A small two-convolution gesture classifier is trained, then converted with TensorFlow Lite, which reduces the precision of its parameters [arxiv:2110.00791 §3]. At 96×96 the file falls from 47.9 MB to 15.9 MB [arxiv:2110.00791 tab. 2]. Accuracy is 98.1% before conversion, measured on 100 validation images, and 96.9% after, measured on a 32-image test set [arxiv:2110.00791 tab. 2]. On the smaller set one image is 3.1 points, and the paper does not show the two sets are comparable. Loss rises about sixfold after conversion, which the paper does not explain [arxiv:2110.00791 tab. 2]. The card's own arithmetic from the layer shapes gives about 12 bytes per parameter before conversion and about 4 after, which is the storage size of float32 values. So the size drop may come from dropping training state rather than from lower precision. The paper does not say what "model size" measures [arxiv:2110.00791 tab. 1]. Pruning is mentioned but not quantified [arxiv:2110.00791 §3]. Treat this as one small experiment. Card: [Optimizing Neural Network for Computer Vision task in Edge Device](/papers/2110.00791).

### Per-tensor precision with exact-cover memory (MinUn)

MinUn compiles a model for ARM microcontrollers. It runs the model once with every tensor at the high bit width and once at the low width, ranks each tensor by how much its values change between the two runs, and promotes tensors to the high width in that order while the estimated memory stays under a limit [arxiv:2210.16556 §4.1]. Tensors that cross the limit are tried one by one and then together, and the configuration with the best measured accuracy is kept [§4.1.3]. Placement in the scratch buffer is an exact-cover problem, solved with Knuth's Algorithm X [§4.3]. On SqueezeNet, evaluated on 48,000 images, float32 reaches 55.02% at 4.42 MB peak RAM, TensorFlow Lite 8-bit zero-skew reaches 45.73% at 1.11 MB, and MinUn with 8- and 16-bit posits reaches 54.81% at 1.16 MB [arxiv:2210.16556 §6.7]. These results are author-reported. The authors also say uniform 8-bit loses too much accuracy on an RNNPool face detector [arxiv:2210.16556 §1, §6.3]. The footnoted code could not be checked, so the details come from the paper text alone [arxiv:2210.16556 §1]. Card: [MinUn](/papers/2210.16556). Per-tensor width choice is a form of [mixed-precision quantization](/concepts/mixed-precision-quantization); the posit formats are described on [posit number format](/concepts/posit-number-format).

### Uniform 8-bit defaults in the TinyML reviews

The review focused on Arm Cortex-M parts targets devices with under about 8 kB of RAM and a clock near 10 MHz [arxiv:2311.11883 §3.2]. It limits itself to uniform quantization, because those schemes are supported on embedded hardware [§4.3]. It recommends integer-only inference and uniform 8-bit PTQ as the default [§3.3] [§4.3]. The review runs no experiments. Its accuracy-against-flash comparison is read from published plots and is not tabulated [arxiv:2311.11883 fig. 9]. The broader TinyML survey describes the same pipeline in outline: train on a large machine, quantize, compile for the board, and flash [arxiv:2303.13569 §II]. It reports quantized footprints from cited works, such as 185 KB for a hand-sign classifier on a Cortex-M7, but gives no accuracy for them [arxiv:2303.13569 §III-B]. Cards: [Efficient Neural Networks for Tiny Machine Learning](/papers/2311.11883) and [TinyML](/papers/2303.13569).

### Detector pruned, then 8-bit quantized (microYOLO)

microYOLO applies iterative L1 pruning over the last part of training, then 8-bit post-training quantization [arxiv:2408.15865 §2, §3]. The authors tried quantization-aware training and did not use it, because they saw no benefit for their application [arxiv:2408.15865 §2]. The paper reports no significant degradation in mAP50 from quantization, while the fridge task loses accuracy during the pruning epochs [arxiv:2408.15865 §3.1]. On an OpenMV H7 R2 the fridge model uses 752 KB of flash and 324 KB of RAM, at 3.46 FPS [arxiv:2408.15865 tab. 2]. The card gives no before-and-after mAP50 table for quantization, and the quantization granularity and calibration data are not stated. Card: [microYOLO](/papers/2408.15865).

### Edge LLM options (survey guidance)

The edge-LLM survey lists weight-only quantization (LLM.int8(), GPTQ, AWQ) and weight-activation co-quantization (ZeroQuant, SmoothQuant) among its offline options [arxiv:2410.11845 §3.1]. Outlier weights or activations are the main difficulty in both [§3.1.1] [§3.1.2]. Its selection advice is qualitative. Weight-only quantization with structured pruning suits strict memory and moderate compute, and co-quantization suits maximum compression when some accuracy loss is acceptable [§3.6]. The survey runs no experiments of its own [§7], so treat the advice as a starting point. Card: [A Review on Edge Large Language Models](/papers/2410.11845).

### Choices inside every variant

- **Granularity of the weight range.** Per-channel means one range per output feature map of the weight tensor; per-layer means one range for the whole tensor [arxiv:1806.08342 §2.6]. The per-channel choice gives much higher per-kernel signal-to-quantization-noise ratio [arxiv:1806.08342 §3.1.3, app. A, fig. 18, fig. 19]. Per-channel activation scales are hard to implement in fixed point, so the second paper keeps activation scales per-tensor [arxiv:2106.08295 §2.2.3]. LLM.int8() uses per-row activation scales and needs GPU kernels for them [arxiv:2208.07339 §3.1]. GPTQ uses one asymmetric min-max scale per row of each weight matrix, with an optional split into groups of consecutive weights [arxiv:2210.17323 §5 Setup, §5 Additional Tricks]. SmoothQuant's O1 to O3 settings differ in activation granularity: per-token dynamic, per-tensor dynamic, or per-tensor static [arxiv:2211.10438 tab. 2]. CodeGeeX's paper writes one absolute maximum per linear layer, while its release code takes one per output row [arxiv:2303.17568 eq. 4]; see Contradictions.
- **Range criterion.** Min-max uses the observed minimum and maximum. MSE chooses the range that minimises the squared quantization error [arxiv:2106.08295 §3.1]. At 8 bits the two are within about 0.3 points in tab. 1 of the second paper. At 4 bits MSE is higher in every cell [arxiv:2106.08295 tab. 1]. The choice also changes learned rounding: AdaRound reaches 61.96% on ResNet-18 with min-max scales and 68.60% with weight-MSE scales [arxiv:2004.10568 tab. 6].
- **Symmetric or asymmetric.** Symmetric fixes $z = 0$ and clips the signed range to $\pm 127$ at 8 bits, for faster SIMD [arxiv:1806.08342 §2.2, eq. 10]. Asymmetric keeps a zero-point. On MobileNet-v1 it is clearly more accurate in tab. 2 and tab. 3 of the first paper, and the two are level on the other MobileNets. See Contradictions.
- **Zero in the range.** The range is widened to include zero, so zero padding is exact. This costs precision when a range is one-sided, for example (2.1, 3.5) [arxiv:1806.08342 §2.1].

## When to use which

Only the papers' ImageNet, detection, segmentation, GLUE, zero-shot and LLM perplexity results are used here. No latency or accuracy claims are made beyond them.

| Situation | Use | Why |
|---|---|---|
| Only need smaller model files, no data, no retraining | Weight-only 8-bit, per-channel asymmetric | Model size is 4x smaller at 8 bits; accuracy stays near float on the tab. 2 networks [arxiv:1806.08342 §1, tab. 2] |
| Need integer inference and can run a calibration pass | Weights and activations, 8-bit, per-channel weights, per-layer activations | About 100 mini-batches set activation ranges; accuracy within about 2 points on tab. 3 networks [arxiv:1806.08342 §3.1.2, tab. 3] |
| Compact network (MobileNet-like) with per-layer weight ranges | Do not use per-layer weights, or apply cross-layer equalization first | Per-layer collapses the MobileNets to 0.001 to 0.004 [arxiv:1806.08342 tab. 2]; equalization lifts per-tensor MobileNetV2 from 0.12 to 69.91 [arxiv:2106.08295 tab. 3] |
| No calibration data at all | Weight-only 8-bit, or the data-free steps (equalization, bias absorption, BN-based ranges) | Activation ranges cannot be estimated without inputs [arxiv:1806.08342 §3.1.1]; BN-based ranges give 71.32 on MobileNetV2 at 8 bits [arxiv:2106.08295 tab. 2] |
| Have 500 to 1000 unlabeled images, need 4-bit weights | Full §3.5 pipeline with AdaRound | Per-channel W4A8 is within about 2 points on seven of eight tab. 6 models [arxiv:2106.08295 tab. 6, §3.5] |
| CNN with 4-bit weights, nearest rounding loses too much, about 1024 unlabeled images | AdaRound, with the scale set by weight MSE | ResNet-18 at 68.60 against 23.99 for nearest rounding [arxiv:2004.10568 tab. 6]; the authors report about 10 minutes for ResNet-18 on one GTX 1080 Ti [arxiv:2004.10568 §5] |
| Depth-wise or detection network at 4-bit | Per-channel weights, or retrain | Per-tensor W4A8 falls to 0.31 mAP on EfficientDet-D1 [arxiv:2106.08295 tab. 6]; QAT gives 35.34 per-tensor [arxiv:2106.08295 tab. 10] |
| Need 4-bit weights and 4-bit activations | Quantization-aware training | PTQ at 4-bit activations is weak on the first paper's networks [arxiv:1806.08342 tab. 6]; QAT reaches 68.83 on ResNet-18 [arxiv:2106.08295 tab. 10] |
| Large transformer (about 6.7B and up) with outlier features, need integer products with outliers in 16 bits | Mixed-precision decomposition, as in LLM.int8() | Vector-wise absmax without it misses float perplexity at 6.7B and above [arxiv:2208.07339 tab. 1] |
| Large transformer, every matrix product must be INT8, and offline calibration with a validation-tuned α is acceptable | SmoothQuant, starting at O1 | OPT-175B stays within 0.5 points of float at O1 to O3 [arxiv:2211.10438 tab. 3]; the coarsest BLOOM-176B setting loses 0.8 [arxiv:2211.10438 tab. 4, §5.2] |
| LLM where weight memory is the bottleneck, activations can stay in float, and hours of calibration on one GPU are acceptable | GPTQ, 4 bits first; 3 bits if the loss is acceptable | 4-bit OPT-175B is 0.03 perplexity above float, and 3-bit is 0.34 above [arxiv:2210.17323 tab. 5] |
| LLM that must also be pruned, one pass, no retraining | SparseGPT in joint mode, or pruning alone | On OPT-175B, 50% sparsity with 4-bit weights gives 8.29 perplexity, against 8.35 dense for pruning alone [arxiv:2301.00774 §4.1] [tab. 1] |
| Large model where weight memory alone blocks a single-GPU run, and no calibration pass is wanted | Weight-only absmax INT8 with FP16 activations, as in CodeGeeX | Inference memory 26.9 GB to 14.7 GB on an A100 for 128-token output; no INT8 accuracy comparison in the card's results [arxiv:2303.17568 tab. 4, §2.4] |
| Smaller LLM (about 6.7B) at 3 bits, weight-only | Test before committing; expect a larger gap | Ungrouped 3-bit GPTQ gives 14.86 against 10.86 float at 6.7B; the grouped results are for 175B-class models only [arxiv:2210.17323 tab. 3, tab. 5] |
| Need integer products with quantized activations | LLM.int8() or SmoothQuant, not GPTQ | GPTQ quantizes weights only [arxiv:2210.17323 §6]; LLM.int8() keeps a 16-bit path for outliers [arxiv:2208.07339 §3.2]; SmoothQuant moves outliers into the weights [arxiv:2211.10438 §4] |
| LLM where speed is the goal and model dimension is 4096 or less | Do not assume an int8 speed-up | 0.86 times the 16-bit speed at dimension 4096 [arxiv:2208.07339 app. D.1] |
| Picking the weight quantizer when it is unclear | Test both asymmetric and symmetric per-channel | Results on MobileNet-v1 point in different directions in the first paper [arxiv:1806.08342 tab. 3, §8] |
| Detector or pose model that must run in integer arithmetic, with no quantization-aware training | Mixed precision, with about 30% of layers at 16 bits, if the plain 8-bit loss is too large | Pure 8-bit gives 55.6 AP against 61.9 float on the YOLOv5s6-ReLU model; mixed gives 60.6 [arxiv:2204.06806 tab. 5] |
| Speed-up required | Do not assume a fixed factor | The first paper summarizes 2 to 3x on a Pixel 2 CPU, but tab. 7 ratios are 1.5x to 2.6x [arxiv:1806.08342 §6, tab. 7] |
| Microcontroller with integer-only code and RAM in the tens to hundreds of KB | Uniform 8-bit PTQ first; if accuracy drops, try per-tensor 8- or 16-bit precision under a RAM limit | The TinyML review favours uniform 8-bit PTQ [arxiv:2311.11883 §3.3, §4.3]; MinUn reports a per-tensor choice at about the same peak RAM as 8-bit TensorFlow Lite, with higher accuracy [arxiv:2210.16556 §6.7] |
| Small CNN converted by a TensorFlow Lite-style converter, size is the goal | Check accuracy on one shared test set before and after conversion, and check size by counting bytes per parameter | The gesture study uses two different sets, and its size drop does not clearly come from lower precision [arxiv:2110.00791 §3, tab. 1, tab. 2] |
| Detector on a microcontroller, QAT not planned | Prune, then 8-bit PTQ, and measure mAP50 before and after | microYOLO reports no significant mAP50 loss from 8-bit quantization after pruning, and did not use QAT [arxiv:2408.15865 §2, §3.1] |
| Edge LLM with strict memory and moderate compute | Weight-only quantization, with structured pruning if needed | The edge-LLM survey's qualitative advice; it has no measurements of its own [arxiv:2410.11845 §3.6, §7] |
| Edge LLM where maximum compression is the goal and some accuracy loss is acceptable | Weight-activation co-quantization, such as SmoothQuant or ZeroQuant | The same survey's qualitative advice [arxiv:2410.11845 §3.6] |

## Implementation sketch

The rounding and range formulas below follow the first paper's equations and the second paper's grid definition. Where the papers say nothing, the sketch says so. The SmoothQuant smoothing step and the AdaRound loop are not sketched here; their cards give the steps.

```pseudocode
# Quantize trained float weights to 8 bits, no retraining  [§2, §3.1]
# Input:  W[layer] float weights, shape (out_channels, ...)
# Output: integer weights and (delta, z) per range

def quantize_weights(W, per_channel=True):
    # One (min, max) per output channel, or one per layer  [§2.6, §2.5]
    ranges = per_channel_min_max(W) if per_channel else layer_min_max(W)
    # Widen each range so that it contains zero  [§2.1]
    ranges = [(min(lo, 0.0), max(hi, 0.0)) for lo, hi in ranges]
    # Grid limits give q_max - q_min = delta * (2^b - 1), z = -q_min / delta  [§2.2]
    params = [affine_params(lo, hi, bits=8) for lo, hi in ranges]
    # Round, add z and clamp onto the integer grid  [eq. 1, eq. 2]
    return [quantize(w, p) for w, p in zip(W, params)], params
```

```pseudocode
# Estimate per-layer activation ranges on about 100 mini-batches  [§3.1.2]
def calibrate_activations(model, calib_batches, n=100):
    ranges = {}
    for batch in calib_batches[:n]:
        # Record every layer's output for this batch  [§3.1.2]
        for name, act in run_and_record(model, batch):
            lo, hi = act.min(), act.max()
            # Moving average of per-batch min and max; decay not stated  [§2.5]
            ranges[name] = blend(ranges.get(name), (lo, hi))
    # One range per layer, because activations are not per-channel  [§2.6]
    return ranges
```

```pseudocode
# Integer convolution with 8-bit weights and activations  [eq. 4-6]
def conv_int8(x_q, w_q, x_params, w_params):
    # Accumulate in integers; zero-point cross terms as in [eq. 5-6]
    acc = int_conv(w_q, x_q)
    # Rescale to float with the activation and per-channel weight deltas  [eq. 4]
    return acc * w_params.delta * x_params.delta
```

```pseudocode
# Mixed-precision linear layer for large transformers  [§3.1] [§3.2] [eq. 7] [eq. 8]
# Input:  X (s, h) float16 activations; W (h, o) float16 weights
# Output: C (s, o) float16

def mixed_int8_linear(X, W, alpha=6.0):
    # Feature dims where some position exceeds the threshold  [§3.2]
    outliers = abs(X).max(axis=0) > alpha
    # Scale each row of the regular part, and each column of W  [§3.1]
    X_reg, W_reg = X[:, ~outliers], W[~outliers, :]
    x_scale = 127 / abs(X_reg).max(axis=1, keepdims=True)
    w_scale = 127 / abs(W_reg).max(axis=0, keepdims=True)
    # Integer product, dequantized by the outer product of scales  [eq. 7]
    C_int8 = matmul(round(X_reg * x_scale), round(W_reg * w_scale))
    C_int8 = C_int8 / (x_scale * w_scale)
    # Outlier dims in 16 bits  [eq. 8]
    C_outlier = matmul(X[:, outliers], W[outliers, :])
    return C_int8 + C_outlier
```

Notes:

- The bias term, the zero-point correction and the batch-norm folding are not covered by this sketch. Batch-norm folding is covered in [batchnorm-folding](/concepts/batchnorm-folding). Cross-layer equalization and bias correction are described in the variant above, not sketched here.
- The first paper's zero-point formulas are written without a rounding mode, and the decay for the activation moving average is not stated [arxiv:1806.08342 §2.1, §2.5, §3.1.2]. The second paper does not state the calibration batch size for activation range setting [arxiv:2106.08295 §3.5].
- The mixed-precision sketch follows the paper's threshold of 6.0. The paper does not say whether outlier weights are stored in 16 bits or rounded to int8 [arxiv:2208.07339 §3.2, eq. 8].
- GPTQ's column loop and Cholesky step are written out as pseudocode in its card [arxiv:2210.17323 alg. 1] and are not repeated here. Those steps need float32 for the Cholesky factor at scale [arxiv:2210.17323 §4 Step 3].
- SmoothQuant's smoothing factor needs the per-channel input maxima from calibration [arxiv:2211.10438 §4]. The formula is written for one linear layer, while the released code shares one factor across the linears that read the same LayerNorm. Check the card's Paper vs code section before porting [arxiv:2211.10438 eq. 4].

## Contradictions

Within and between papers, several statements disagree. Both sides are cited; none is resolved here.

- **Best weight-only quantizer.** The first paper's text says asymmetric per-layer weights give the best weight-only accuracy [arxiv:1806.08342 §3.1.3]. Its table shows asymmetric per-channel weights are best, and per-layer weights collapse the MobileNets [arxiv:1806.08342 tab. 2].
- **Recommended starting quantizer.** The first paper's conclusions suggest symmetric per-channel as the PTQ starting point [arxiv:1806.08342 §8]. In its tab. 3 on MobileNet-v1, symmetric per-channel gives 0.591 against 0.703 for asymmetric per-channel [arxiv:1806.08342 tab. 3]. On MobileNet-v2 the two are level (0.698 and 0.697). The second paper defaults to symmetric weights and asymmetric activations, for inference cost reasons [arxiv:2106.08295 §2.4.1].
- **Speed.** The first paper's summary gives 2 to 3 times faster on a Pixel 2 CPU [arxiv:1806.08342 §6]. Its latency table shows 1.5 to 2.6 times across the eight networks listed [arxiv:1806.08342 tab. 7]. The card does not record whether those timed models were post-training or quantization-aware, so the two cannot be checked against each other.
- **4-bit PTQ.** The first paper reports 4-bit PTQ falling to 0.001 to 0.74 top-1 on its networks [arxiv:1806.08342 tab. 5]. The second paper reports per-channel W4A8 PTQ within about 2 points on seven of eight models [arxiv:2106.08295 tab. 6]. The second paper's pipeline is the full §3.5 one, and the first paper's 4-bit results use different quantizer and pipeline choices. The cards do not resolve which result applies to a given network.
- **8-bit accuracy claim inside the second paper.** Its text says 8-bit PTQ is within 0.7% of float for all models [arxiv:2106.08295 §3.6]. Its tab. 6 shows per-tensor MobileNetV2 at 0.73 points below float and EfficientDet-D1 at 1.79 points below [arxiv:2106.08295 tab. 6].
- **AdaRound's "within 1%" claim.** The AdaRound paper says 4-bit Resnet18 and Resnet50 are within 1% of FP32 [arxiv:2004.10568 §6]. With the 68.60% Resnet18 result of its main tables, the gap is 1.08 points. Only the 68.71% row of its literature table, which uses 2048 images and 20k iterations, is within 1% [arxiv:2004.10568 tab. 6] [tab. 7]. Resnet50 at 75.23 is 0.84 below float [arxiv:2004.10568 tab. 7].
- **Detector accuracy at 8 bits.** The second paper's per-channel W8A8 results are within 1.5 points of float for all eight models in tab. 6, and that table includes the EfficientDet-D1 detector [arxiv:2106.08295 tab. 6]. On its YOLOv5s6-ReLU model, YOLO-Pose loses 6.3 AP at 8 bits [arxiv:2204.06806 tab. 5]. The YOLO-Pose card does not record the granularity or the calibration method, so the two results cannot be compared directly.
- **Activation quantizer.** The first paper says activations are per-layer symmetric in one place [arxiv:1806.08342 §3.1.3]. It does not say which quantizer produced the activation numbers in the tables. Treat the activation type as not stated. The second paper uses asymmetric activations [arxiv:2106.08295 §2.4.1].
- **Speed of LLM methods.** LLM.int8() measures 0.86 times the 16-bit speed at model dimension 4096 [arxiv:2208.07339 app. D.1]. GPTQ reports 3.24 times for 3-bit OPT-175B at batch size 1, against a five-GPU FP16 baseline [arxiv:2210.17323 tab. 6]. SmoothQuant reports up to 1.56 times over FP16 in FasterTransformer [arxiv:2211.10438 fig. 9]. The settings differ (activation quantization or not, model size, baseline hardware, kernels), so these figures are not comparable. The cards do not resolve which applies to a given deployment. CodeGeeX reports 3.66 s against 1.61 s per 128-token output on an A100, but that pairs PyTorch FP16 with FasterTransformer INT8, so framework and precision are confounded [arxiv:2303.17568 tab. 4].
- **CodeGeeX scale granularity.** The paper's eq. 4 uses one absolute maximum per linear layer [arxiv:2303.17568 eq. 4]. Its release code takes the maximum per output row, as the card's Paper vs code section records. The two agree only when every row has the same maximum. The card does not establish which one produced tab. 4.
- **8-bit on microcontrollers.** The TinyML review favours uniform 8-bit PTQ as the default [arxiv:2311.11883 §4.3]. MinUn reports that 8-bit TensorFlow Lite loses 9.3 points against float32 on SqueezeNet, and that uniform 8-bit is too coarse for an RNNPool face detector [arxiv:2210.16556 §6.7] [§1]. The review gives no measurements of its own, and the two sets of results are on different models, so neither settles the default.
- **Size drop from conversion.** The gesture paper attributes the drop from 47.9 MB to 15.9 MB at 96×96 to precision reduction [arxiv:2110.00791 §3, tab. 2]. The card's derivation from the layer shapes gives about 4 bytes per parameter after conversion, which is the float32 size, so the drop may come from elsewhere [arxiv:2110.00791 tab. 1]. The paper does not state what "model size" measures, and this page cannot resolve it.
- **AdaRound figure.** Weng's survey lists AdaRound at 68.55% for ResNet-18 [arxiv:2112.06126 tab. 4]. The AdaRound paper's own table gives 68.60% [arxiv:2004.10568 tab. 6]. The survey copies its numbers from the cited works, so this page uses the original value.
- **Compression limit.** The survey says integer networks with minimal loss are usually limited to 8 bits, so compression is at most about 4x [arxiv:2112.06126 tab. 2]. The 4-bit AdaRound result above loses about 1 point against float32 for ResNet-18 [arxiv:2004.10568 tab. 6]. That result is weight-only and the survey's bound is an author's claim about minimal loss, so the two are not measuring the same thing.

## Papers

- [Quantizing deep convolutional networks for efficient inference: A whitepaper](/papers/1806.08342): practical PTQ and quantization-aware training recipes for 8-bit and 4-bit CNNs, with per-channel weights, activation calibration and ImageNet measurements.
- [Up or Down? Adaptive Rounding for Post-Training Quantization](/papers/2004.10568): AdaRound, a layer-wise learned rounding for 4-bit CNN weights that uses about 1024 unlabelled images and no fine-tuning.
- [A White Paper on Neural Network Quantization](/papers/2106.08295): the PTQ pipeline and debugging flow for uniform affine quantization, with cross-layer equalization, bias correction, AdaRound and measurements on classification, detection, segmentation and BERT.
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](/papers/2208.07339): vector-wise absmax quantization with a 16-bit path for outlier feature dimensions, for inference of large transformers without retraining.
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](/papers/2211.10438): per-channel smoothing that moves activation outliers into the weights, so that all LLM matrix products run in W8A8 without retraining.
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](/papers/2210.17323): weight-only 3- and 4-bit rounding for GPT-style models, with Hessian-based error compensation calibrated on 128 C4 segments.
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](/papers/2301.00774): one-pass pruning of GPT-style models, with an option to quantize the kept weights in the same pass.
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](/papers/2303.17568): a 13B code model whose INT8 weight quantization is reported only as memory and time figures, with no accuracy comparison in the card.
- [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](/papers/2204.06806): post-training 8-bit and mixed-precision results for a YOLOv5-based pose detector, with the accuracy cost of each setting.
- [Optimizing Neural Network for Computer Vision task in Edge Device](/papers/2110.00791): TensorFlow Lite precision reduction of a small gesture CNN on a Raspberry Pi, with size and accuracy before and after on one small dataset.
- [Neural Network Quantization for Efficient Inference: A Survey](/papers/2112.06126): survey of number formats, PTQ and QAT, and binary, ternary and mixed-precision schemes, with comparison tables copied from the cited works.
- [MinUn: Accurate ML Inference on Microcontrollers](/papers/2210.16556): microcontroller compiler that picks 8- or 16-bit precision per tensor under a RAM limit, with exact-cover memory placement.
- [TinyML: Tools, Applications, Challenges, and Future Research Directions](/papers/2303.13569): survey of TinyML tools, boards and applications, citing quantized footprints from other works.
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](/papers/2311.11883): review of TinyML compression for Arm Cortex-M, recommending uniform 8-bit PTQ as the default, with no experiments of its own.
- [microYOLO: Towards Single-Shot Object Detection on Microcontrollers](/papers/2408.15865): pruned and 8-bit quantized YOLO-style detector on a Cortex-M7 board.
- [A Review on Edge Large Language Models: Design, Execution, and Applications](/papers/2410.11845): survey of edge LLM compression and runtime options, with qualitative selection guidance.