--- kind: concept slug: weight-quantization title: Weight quantization summary: >- Storing weights in few bits on a grid. Per-channel 8-bit scales keep ImageNet CNNs near float accuracy. LLM.int8() keeps outlier features in 16-bit and stays within 0.2 perplexity of 32-bit. GPTQ corrects rounding error with calibration data, within 0.03 perplexity of FP16 for 4-bit OPT-175B. QLoRA's 4-bit NF4 keeps finetuned LLMs near 16-bit, but reports no speed gain. tags: [quantization, memory, inference, finetuning, efficient-inference, outliers] papers: ["1711.11294", "1806.08342", "2004.09576", "2004.10568", "2208.07339", "2210.02414", "2210.17323", "2211.10438", "2302.08469", "2305.14314", "2309.11932", "2312.12456", "2402.10373", "2403.14123", "2406.09246", "2407.10734", "2511.01866", "2605.06714"] related: [post-training-quantization, quantization-aware-training, asymmetric-quantization, batchnorm-folding, low-rank-adaptation, arithmetic-intensity, llm-serving, hardware-aware-training, network-pruning, binary-neural-network, adaptive-rounding, low-precision-arithmetic, on-device-training, on-device-inference, mixed-precision-quantization, test-time-scaling, activation-sparsity] status: draft updated_at: 2026-11-01 --- ## Overview Weight quantization stores each weight as a low-bit integer or code, plus shared scale constants, and turns it back into a number when it is used. The eighteen papers on this page cover different parts of that design space. [Quantizing deep convolutional networks](/papers/1806.08342) maps CNN weights and activations to 8-bit integers for integer-only inference on phones and DSPs [arxiv:1806.08342 §1, §2]. [Towards Accurate Binary Convolutional Neural Network](/papers/1711.11294) writes each layer's weights as a weighted sum of M binary filters and its activations as N binary thresholds, so that a convolution becomes M×N bitwise convolutions [arxiv:1711.11294 §3.1.1, §3.2, eq. 12]. [LLM.int8()](/papers/2208.07339) runs the projection-layer matrix products of a trained transformer in 8-bit integers, without retraining, and keeps a few outlier feature dimensions in 16-bit [arxiv:2208.07339 §1, §3.2]. [SmoothQuant](/papers/2211.10438) instead divides outlier activation channels by per-channel factors and multiplies the matching weight rows, so that every matrix product can run in 8 bits [arxiv:2211.10438 §4]. [QLoRA](/papers/2305.14314) stores a frozen LLM base in 4 bits so that LoRA adapters can be finetuned on top [arxiv:2305.14314 §1]. [BioMistral](/papers/2402.10373) applies AWQ 4-bit and BitsandBytes 4/8-bit to a finished 7B model and measures accuracy, memory and speed [arxiv:2402.10373 §3.4, §5.5]. [LSQ+](/papers/2004.09576) learns a scale and an offset for each activation quantizer and keeps weights symmetric, so that skewed activations use the full integer range [arxiv:2004.09576 §3.1, §3.2.1]. [OpenVLA](/papers/2406.09246) runs a 7B robot policy at bfloat16, int8 and int4 inference precision and reports memory, control rate and success for each [arxiv:2406.09246 §5.4]. [AI and Memory Wall](/papers/2403.14123) supplies the motivation: decoder inference at small batch is limited by memory transfers, so fewer bytes per weight can matter [arxiv:2403.14123 §2.2]. It does not test quantization itself [arxiv:2403.14123 §3]. [GPTQ](/papers/2210.17323) rounds the weights of a trained LLM to 3 or 4 bits one column at a time, adjusting the columns not yet rounded, using calibration inputs [arxiv:2210.17323 §3, §4]. [AdaRound](/papers/2004.10568) keeps the grid fixed and chooses each weight's rounding direction by minimising the error of one layer's output on unlabelled inputs [arxiv:2004.10568 §3]. [GLM-130B](/papers/2210.02414) uses plain row-wise absmax INT4 on its linear-layer weights, with FP16 activations, and reports results close to FP16 on its own model [arxiv:2210.02414 §4]. [The edge-deployment survey](/papers/2605.06714) collects quantisation results from other works for on-device vision and runs no experiments of its own [arxiv:2605.06714 §4.3]. [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932) places quantization under model optimization, next to pruning and distillation, and counts the cited works per period rather than measuring them [arxiv:2309.11932 §5.1.2, tab. 1]. [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734) quantizes weights, feature maps and errors to 8 bits during training, with the weight update itself in floating point [arxiv:2407.10734 §3.1]. [EdgeReasoning](/papers/2511.01866) applies AWQ 4-bit weights to reasoning models on an edge GPU and measures latency, energy and accuracy [arxiv:2511.01866 §5.6]. [PowerInfer](/papers/2312.12456) splits each feed-forward block between one consumer GPU and the CPU by neuron, and reports speed-ups at FP16 and at INT4 weights, each against llama.cpp [arxiv:2312.12456 §8.1, §8.2, fig. 13]. Five decisions recur across these papers. The first is the granularity of the scale: one range per layer, per output channel, per row or column, or per block of weights. The second is the number format: an integer grid (asymmetric or symmetric), a non-uniform 4-bit code, or a weighted sum of several binary bases [arxiv:1711.11294 §3.1.1]. The third is whether training sees the rounding. Quantization is applied to a model that is already trained in one case, and to a frozen base that is never updated in another. The fourth is what to do with outliers: a few large values can stretch a shared scale until most values lose precision [arxiv:2208.07339 §3, §4.2]. The fifth is how each rounding error is handled: left alone, or corrected by adjusting the weights that are not yet rounded, using calibration inputs [arxiv:2210.17323 §3, §4 Step 1]. AdaRound avoids the correction step: it chooses the rounding direction of each weight directly, by optimisation [arxiv:2004.10568 §3.3]. The shared mechanism in QLoRA is a storage dtype (e.g. NF4) and a compute dtype (BF16). Whenever a weight is used it is dequantized and the matmul runs in 16-bit [arxiv:2305.14314 §3]. Tensors are quantized blockwise, each block of $B$ values with its own absmax constant, so a single outlier only affects its own block [arxiv:2305.14314 §2, eq. 1]. For the adapter side, see [Low-rank adaptation](/concepts/low-rank-adaptation). For the CNN side, see [Post-training quantization](/concepts/post-training-quantization) and [Quantization-aware training](/concepts/quantization-aware-training). For why memory traffic rather than arithmetic sets decoder speed, see [Arithmetic intensity](/concepts/arithmetic-intensity) and [LLM serving](/concepts/llm-serving). ## Variants ### Granularity of the scale A scale and zero-point can be shared by a whole tensor (per-layer), by each output feature map (per-channel), or by each block of weights (QLoRA uses blocks of 64) [arxiv:1806.08342 §2.6] [arxiv:2305.14314 §3]. The per-layer choice fails on compact CNNs. On MobileNet-v1 with asymmetric 8-bit weights and no retraining, per-layer scales give 0.001 top-1 against 0.703 for per-channel [arxiv:1806.08342 tab. 3]. The paper traces this to batch-norm folding, which makes the kernels of one layer differ widely in range, so one shared grid wastes most of its levels [arxiv:1806.08342 §3.1.3, app. A]. The same failure appears for LLMs. Using one absmax scale for a whole tensor gives C4 perplexity of 87.76 at 125M parameters and 19.08 at 13B, against 25.65 and 12.45 for 32-bit [arxiv:2208.07339 tab. 1]. LLM.int8() uses one scale per row of the activations and one per column of the weights instead [arxiv:2208.07339 §3.1, eq. 7]. The CNN and LLM papers measure different things, so their numbers are not directly comparable. GPTQ fits one grid per row of each weight matrix by default, or one per group of consecutive weights within a row. Group sizes from 1024 down to 8 appear in its tables [arxiv:2210.17323 §5, tab. 5, tab. 7]. Its own bit counts show the cost: a group of 128 adds about 0.15 bits per weight and a group of 1024 about 0.02 [arxiv:2210.17323 §5 175 Billion Parameter Models]. ### Uniform affine and symmetric integer grids An asymmetric grid maps a float range to integers $0 \ldots 255$ with a scale $\Delta$ and an integer zero-point $z$. The range is first widened to include zero, so zero padding is exact [arxiv:1806.08342 §2.1, eq. 1–3]. The paper does not state how $\Delta$ and $z$ are computed from the range [arxiv:1806.08342 §2.1]. A symmetric grid fixes $z = 0$ [arxiv:1806.08342 §2.2, eq. 7–8]. Its signed 8-bit weights are clamped to $[-127, 127]$ for faster SIMD [arxiv:1806.08342 eq. 10]. With an asymmetric grid, the convolution expands into extra cross terms that need specialized kernels; the paper argues that subtracting zero-points up front cuts throughput by 2x to 4x [arxiv:1806.08342 §2.1, eq. 4–6]. QLoRA's 8-bit building block is also symmetric: each block is scaled by $127/\text{absmax}$ and rounded to Int8 (eq. 1), and dequantized by dividing by the constant (eq. 2) [arxiv:2305.14314 §2]. Its 4-bit integer analogue, Int4, gave the worst mean Pile Common Crawl perplexity of the 4-bit types tested: 34.34 [arxiv:2305.14314 tab. 2]. LLM.int8() also uses symmetric absmax grids: each tensor is multiplied by $127/\lVert X \rVert_\infty$ and rounded into $[-127, 127]$ [arxiv:2208.07339 §2.1]. Its zero-point variant is only a baseline. The zero-point equation as printed in the paper is not a usable zero point, so implementers should not copy it [arxiv:2208.07339 eq. 2]. ### Outlier-aware mixed precision (LLM.int8()) Starting at about 6.7B parameters, a few feature dimensions in every transformer layer take values up to about 20 times the others [arxiv:2208.07339 §1, §4.2]. Quantizing them with the rest makes absmax and zero-point quantization lose most of their precision [arxiv:2208.07339 §4.2, §4.3]. LLM.int8() handles this with a split: every feature dimension whose value reaches magnitude $\alpha = 6.0$ is multiplied in 16-bit, the others in int8, and the two products are summed [arxiv:2208.07339 §3.2, eq. 8]. The paper says at most 7 outlier dimensions appear in models up to 13B [arxiv:2208.07339 §3.2]. The split is necessary. Vector-wise absmax quantization without the 16-bit path reaches perplexity 14.13 at 6.7B and 16.48 at 13B, against 13.30 and 12.45 for 32-bit. With the path added, the same model reads 13.24 and 12.45 [arxiv:2208.07339 tab. 1]. Across 125M to 13B, the full method stays within 0.2 perplexity of 32-bit [arxiv:2208.07339 tab. 1]. Removing the outlier dimensions alone raises validation perplexity by 600% to 1,000%, while removing seven random dimensions changes it by about 0.1% [arxiv:2208.07339 §4.2]. Memory is about half of 16-bit, with about 0.1% extra for the decomposition [arxiv:2208.07339 §3.2]. Speed is a separate question. On the first feed-forward layer of GPT-3-sized models, the full method runs at 0.86x of 16-bit at model dimension 4096 and 1.22x at 5140, with custom kernels [arxiv:2208.07339 app. D.1]. End to end on BLOOM-176B, batch 1 takes 253 ms against 239 ms for bfloat16 on 8 A100s [arxiv:2208.07339 app. D.2]. The paper's pitfalls note that the method is slower below about 6.7B in its own timings [arxiv:2208.07339 §3.4, app. D.1]. Its training results are narrower: the authors find 8-bit feed-forward training straightforward, but say other layers need more work [arxiv:2208.07339 app. E]. ### All-INT8 matrix products with smoothing (SmoothQuant) SmoothQuant moves the outlier problem from activations into weights, so that every linear layer and both attention matrix products run in W8A8, with no 16-bit path [arxiv:2211.10438 §1, §4]. For each input channel it sets a factor $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$. The activation channel is divided by $s_j$ and the matching weight row is multiplied by it, so the product is unchanged [arxiv:2211.10438 §4, eq. 3, eq. 4]. Where the activation comes from a LayerNorm, the division folds into that LayerNorm offline [arxiv:2211.10438 §4]. The maxima come from 512 Pile sentences [arxiv:2211.10438 §5.1]. Author-reported: naive W8A8 gets 35.5% average accuracy on OPT-175B, against 66.9% for float, and SmoothQuant at O3 gets 66.8% [arxiv:2211.10438 tab. 3]. On BLOOM-176B the O1, O2 and O3 settings give 68.3, 68.4 and 67.4, against 68.2 for float [arxiv:2211.10438 tab. 4]. On GLM-130B, O3 gives 72.8 against 73.8 [arxiv:2211.10438 tab. 4]. Coarser settings are faster and cost accuracy: the paper reports about 0.8 points on BLOOM-176B for the coarsest one [arxiv:2211.10438 §5.2]. The paper reports up to 1.56 times FP16 speed in FasterTransformer and about half the memory [arxiv:2211.10438 fig. 9, §5.3]. The migration strength $\alpha$ is 0.5 for OPT and BLOOM and 0.75 for GLM-130B [arxiv:2211.10438 §4, §5.1]. The repository has no latency benchmark, so the speed figures rest on the paper alone [Card](/papers/2211.10438). For the outlier background, see [LLM.int8()](/papers/2208.07339) above and [post-training quantization](/concepts/post-training-quantization). ### 4-bit and 8-bit quantization of a finished 7B model (BioMistral) BioMistral compares AWQ 4-bit and BitsandBytes 4-bit and 8-bit on its own 7B model [arxiv:2402.10373 §3.4]. Average 3-shot accuracy over ten English medical QA tasks is 50.3 for the unquantized model, 51.7 for AWQ 4-bit with the GEMM kernel, 51.8 with GEMV, 50.0 for BitsandBytes 8-bit and 46.9 for BitsandBytes 4-bit [arxiv:2402.10373 tab. 4]. The AWQ average is driven by PubMedQA: 61.7 with GEMV against 37.6, a gain of 24.1 points. The other nine tasks lose about 1.1 points on average with GEMV [arxiv:2402.10373 tab. 4]. BitsandBytes 4-bit drops PubMedQA to 22.4 [arxiv:2402.10373 tab. 4]. The paper does not state the AWQ group size [arxiv:2402.10373 §3.4]. On one RTX 3090, the MMLU professional-medicine set took 40.94 s in 16-bit with 15.02 GB of VRAM. BitsandBytes 8-bit took 177.75 s with 8.04 GB, BitsandBytes 4-bit took 133.06 s with 5.03 GB, and AWQ with GEMM took 57.96 s with 4.68 GB [arxiv:2402.10373 app. D, tab. 7]. All four quantized runs were slower than 16-bit in that setup. The paper recommends AWQ with GEMM as the best trade-off [arxiv:2402.10373 §5.5]. Card: [BioMistral](/papers/2402.10373). ### Quantized storage with fine-grained scales (AWQ, in BioMistral) BioMistral describes AWQ as keeping the most important weights unquantized, on the observation that weights differ in importance [arxiv:2402.10373 §3.4]. The paper gives no formula for this, so the description is the only one the wiki can cite from it. Its results are in the previous variant. ### Column-wise rounding with error compensation (GPTQ) GPTQ quantizes each linear layer of a GPT-style model to 3 or 4 bits, with no retraining. It uses a calibration set of inputs to the layer, and quantizes the columns in one fixed order [arxiv:2210.17323 §3, §4 Step 1]. After a column is rounded, the error is spread over the columns that are still full precision, using the inverse of the layer's input Hessian $H = 2XX^\top$ [arxiv:2210.17323 §3]. Calibration uses 128 random 2048-token segments of C4 training text [arxiv:2210.17323 §5 Setup]. The paper's reason for the fixed order and the block updates is runtime: OBQ's greedy order costs $O(d_{row} \cdot d_{col}^3)$ per matrix, and GPTQ brings the work down to a level where OPT-175B takes 4.2 hours and BLOOM-176B 3.8 hours on one A100 80 GB [arxiv:2210.17323 §3, §4 Step 1, tab. 2]. On OPT-175B the 4-bit model reaches WikiText2 perplexity 8.37 against 8.34 for FP16 and 10.54 for round-to-nearest. At 3 bits, round-to-nearest collapses to 7.3e3 and GPTQ reaches 8.68. Grouping with 128 weights per group gives 8.45 at 3 bits [arxiv:2210.17323 tab. 5]. The 2-bit results use groups of 128, 64 or 32, with OPT-175B at 8.94 for groups of 32 [arxiv:2210.17323 tab. 7]. A ternary grid with groups of 8 reaches 9.20 on OPT-175B [arxiv:2210.17323 §5 Extreme Quantization]. The paper's 2-bit bit counts depend on how group constants are stored, so check the format before quoting one [arxiv:2210.17323 §5 Extreme Quantization]. Card: [GPTQ](/papers/2210.17323). Speed is reported for batch-1 generation of 128 tokens. 3-bit GPTQ takes 71 ms per token on one A100 against 230 ms for FP16 on five A100s (3.24x), and 130 ms on one A6000 against 589 ms on eight (4.53x) [arxiv:2210.17323 tab. 6]. The FP16 baseline uses several GPUs, so the speed-up includes the reduction in GPU count [arxiv:2210.17323 tab. 6]. ### Learned rounding direction (AdaRound) AdaRound keeps the grid fixed and chooses, for each weight, whether to round it down or up. It relaxes each choice to a continuous variable, minimises the error of one layer's output on a few unlabelled inputs, and then fixes each choice to 0 or 1 [arxiv:2004.10568 §3.3]. The method covers convolutional and linear layers [arxiv:2004.10568 §3.3]. With 4-bit weights and float activations, ResNet-18 reaches 68.60% top-1 against 69.68% for float and 23.99% for nearest rounding [arxiv:2004.10568 tab. 4] [tab. 6]. On ResNet-50 the figures are 75.23, 76.07 and 35.60 [arxiv:2004.10568 tab. 7]. The main ablations use 1024 calibration images, and the paper reports about 10 minutes for ResNet-18 on one GTX 1080 Ti [arxiv:2004.10568 §5]. The reported results are for CNNs and a segmentation model; the paper has no language-model result [arxiv:2004.10568 tab. 7] [tab. 9]. The reference implementation uses 15,000 iterations at 4 bits, against the paper's 10,000, and sets the regulariser and annealing schedule that the paper does not state; see the [card](/papers/2004.10568) and [adaptive rounding](/concepts/adaptive-rounding). The scale is fitted by weight error [arxiv:2004.10568 §5], and the gain depends on it: with min-max scales AdaRound reaches 61.96 against 0.23 for nearest rounding [arxiv:2004.10568 tab. 6]. ### Weight-only INT4 with row-wise absmax (GLM-130B) GLM-130B quantizes only the weights of the QKV, attention-output and both MLP projections in each layer, to INT4 [arxiv:2210.02414 §4]. Each output row gets one scale, its absolute maximum divided by 7, and the weights are dequantized to FP16 at run time. Activations stay in FP16 [arxiv:2210.02414 §4] [app. A]. The appendix also tests a zero-point variant, which scores 80.63 on LAMBADA against 79.47 for absmax. The authors kept absmax because it performed well and is cheaper to compute [arxiv:2210.02414 app. A]. On the 130B model, INT4 weights give LAMBADA 79.47, MMLU 44.80 and Pile bits-per-byte 0.641, against 80.21, 44.75 and 0.634 in FP16 [arxiv:2210.02414 §4]. INT8 weights give 80.21, 44.71 and 0.638 [arxiv:2210.02414 §4]. The quantized weights take about 70 GB, which fits on four RTX 3090 cards with 24 GB each [arxiv:2210.02414 §4]. The INT4 decoding times are reported for several GPU setups, but without an FP16 run on the same setups, so this variant gives no speed ratio [arxiv:2210.02414 §4]. The same scheme is reported to fail on BLOOM at INT4 [arxiv:2210.02414 app. A]. The released weights carry a non-commercial licence, while the code is Apache-2.0; the [card](/papers/2210.02414) has the details. The method itself is not restricted by this. ### INT4 weights in a neuron-split CPU-GPU engine (PowerInfer) PowerInfer keeps frequently active feed-forward neurons on the GPU and computes the rarely active ones on the CPU [arxiv:2312.12456 §4]. Its evaluation runs this engine with FP16 weights and with INT4 weights [arxiv:2312.12456 §8.1]. The INT4 speed-ups are measured against llama.cpp at INT4, not against FP16, so they do not show how much the neuron split adds on top of quantization. On PC-High (RTX 4090) the average over models is 2.89 times, with a maximum of 4.28 times. On PC-Low (RTX 2080 Ti) the average is 5.01 times and the maximum 8.00 times [arxiv:2312.12456 §8.2, fig. 13]. The paper's gains depend on activation sparsity, and it reports them as largest for ReLU-family models [arxiv:2312.12456 §4, §8.4]. These are author-reported numbers on two desktop machines [arxiv:2312.12456 §8.1]. Card: [PowerInfer](/papers/2312.12456). ### 4-bit floating point (FP4) Float4 with E2M1 or E3M0 layouts: mean perplexity 31.07 and 29.48 respectively [arxiv:2305.14314 tab. 2]. In LLaMA 7-65B finetuning, FP4 QLoRA averaged 52.2 5-shot MMLU vs 53.0 for 16-bit LoRA [arxiv:2305.14314 tab. 4]. ### 4-bit NormalFloat (NF4) Quantile quantization for a $N(0,1)$ source, rescaled to $[-1,1]$, so each bin gets an equal expected share of normally distributed weights [arxiv:2305.14314 §3, eq. 4]. The code is asymmetric so that it contains an exact zero; the 16 values are listed in the paper's appendix [arxiv:2305.14314 §3, app. E]. Block size is 64 [arxiv:2305.14314 §3]. It assumes weights are roughly zero-mean normal; 7.5% of LLaMA-7B hidden units fail a Shapiro-Wilk test at 5% [arxiv:2305.14314 app. F]. Card: [QLoRA: Efficient Finetuning of Quantized LLMs](/papers/2305.14314). ### Double quantization (DQ) The per-64-weight FP32 constants are themselves quantized to FP8 with block size 256 (after subtracting their mean, since they are all positive). Constant overhead drops from 0.5 to 0.127 bits per parameter, about 3 GB on a 65B model [arxiv:2305.14314 §1, §3]. NF4 + DQ gave the best mean perplexity in the comparison, 27.41 [arxiv:2305.14314 tab. 2]. ### Learned scale and offset for activations (LSQ+) LSQ+ learns a scale $s$ and an offset $\beta$ for each activation quantizer, and reconstructs each value as $\bar{x}s + \beta$ [arxiv:2004.09576 §3.1, eq. 2]. The offset lets a skewed activation such as Swish use all $2^b$ levels. A symmetric grid either spends one bit on a sign or clamps the negative part to zero [arxiv:2004.09576 §1, §3.1]. Weights stay symmetric and signed [arxiv:2004.09576 §3.2.1]. The offset term folds into a per-output bias at compile time [arxiv:2004.09576 eq. 5]. The paper reports zero runtime overhead for this [arxiv:2004.09576 §6]; the fold needs a layer that accepts a bias term. On EfficientNet-B0 at 4 bits for weights and activations, the asymmetric configurations reach 73.8% against 71.9% for LSQ and 76.1% at full precision [arxiv:2004.09576 tab. 2]. At 2 bits the gap is larger: 48.7% for the asymmetric unsigned configuration against 43.5% for LSQ [arxiv:2004.09576 tab. 2]. Learning the offset beats fixing it at zero or at the observed minimum: 73.8% against 71.9% and 72.5% [arxiv:2004.09576 tab. 6]. The initialisation matters for variance. At 2 bits over five runs, min-max start gives 43.8% ± 4.7 and the MSE fit on activations gives 46.8% ± 1.9 [arxiv:2004.09576 tab. 5]. The paper starts all experiments from pretrained float weights and does not test training from scratch [arxiv:2004.09576 §4]. It does not state the optimiser or learning rate used for the MSE fit [arxiv:2004.09576 §3.2.2]. Card: [LSQ+](/papers/2004.09576). ### Several binary bases per layer (ABC-Net) ABC-Net writes a layer's real weights as a weighted sum of M binary filters, W ≈ α_1 B_1 + … + α_M B_M, with each B_m equal to +1 or −1 [arxiv:1711.11294 §3.1.1, eq. 1]. Each basis is the sign of the layer's centred weights shifted by a multiple of their standard deviation, and the scales α are fitted by least squares once the bases are fixed [arxiv:1711.11294 §3.1.1, eq. 2–3]. One set of bases covers the whole layer. A separate set per output channel is described but left as future work, and with M = 1 the scheme reduces to BWN [arxiv:1711.11294 §3.1.2]. Weights stay real during training and are binary at test time [arxiv:1711.11294 §3]. On ResNet-18 with full-precision activations, Top-1 rises from 60.8% for BWN to 62.8% at M = 1 and 68.3% at M = 5, against 69.3% for full precision [arxiv:1711.11294 tab. 1]. The same paper also binarizes the activations at N thresholds, each with its own trainable shift and scale, so a convolution becomes M×N binary convolutions [arxiv:1711.11294 §3.2, eq. 11–12]. With five bases and five activation thresholds on ResNet-18, Top-1 is 65.0%, against 51.2% for XNOR-Net and 69.3% for full precision [arxiv:1711.11294 tab. 2, tab. 3]. The paper argues that bitwise operations are cheaper than fixed-point multiplies, but reports no measured speed [arxiv:1711.11294 §5.2]. Training starts from a full-precision pretrained model and uses straight-through gradients through the sign function [arxiv:1711.11294 §3.3, eq. 7]. For the one-bit case as a whole, see [Binary neural network](/concepts/binary-neural-network). Card: [Towards Accurate Binary Convolutional Neural Network](/papers/1711.11294). ### Analog in-memory weights with 8-bit converters (hardware-aware training) This variant does not store weights as low-bit integers. Each weight matrix is mapped to analog conductances on crossbar tiles of at most 512 by 512 cells, with one conductance pair per weight in the paper's model [arxiv:2302.08469 Methods: Weight programming, Methods: Crossbar tile size]. The analog product is read through an 8-bit DAC on the inputs and an 8-bit ADC on the outputs [arxiv:2302.08469 app. A.1]. The stored values are perturbed by programming error and by drift over time [arxiv:2302.08469 eq. 7–9]. The paper's question is how much retraining recovers under that noise. Direct mapping of trained weights, with no retraining, misses the iso-accuracy target for almost all of its eleven networks; only LSTM on PTB reaches 99.0% normalized accuracy at one hour [arxiv:2302.08469 §2.2, tab. 2]. Hardware-aware training runs the crossbar model in the forward pass and floating-point gradients in the backward pass, and afterwards five of the eleven networks reach iso-accuracy at one hour [arxiv:2302.08469 §2.3, tab. 3]. Output-side noise and converter resolution have the largest effect on accuracy, and the paper finds that a DAC can drop from 8 to 6 bits with little accuracy impact [arxiv:2302.08469 §2.4, §2.5, fig. 4]. Card: [Rasch et al., hardware-aware training on analog crossbars](/papers/2302.08469). ### Inference precision of a 7B robot policy (OpenVLA) OpenVLA is a 7B vision-language-action model that predicts robot actions as tokens [arxiv:2406.09246 §3.2]. The paper reports three inference precisions on eight BridgeData V2 tasks with 80 rollouts each [arxiv:2406.09246 §5.4]. Success is 71.3% ± 4.8 in bfloat16 with 16.8 GB of GPU memory, 58.1% ± 5.1 in int8 with 10.2 GB, and 71.9% ± 4.7 in int4 with 7.0 GB [arxiv:2406.09246 §5.4]. Inference rates on an A5000 are 1.2 Hz for int8 and 3 Hz for int4; the bfloat16 rate is not given as one figure [arxiv:2406.09246 §5.4]. Under blocking control the three precisions score 70.0%, 74.4% and 68.8%, and the authors read the int8 drop as a control-speed effect rather than a precision effect [arxiv:2406.09246 app. D.4]. The quantization runs use a SigLIP-only checkpoint trained on the Octo mixture, so their numbers are not comparable with the main results [arxiv:2406.09246 §5.3]. The released code also has a 4-bit NF4 option for fine-tuning, which the paper does not evaluate; see [OpenVLA](/papers/2406.09246) for the code details. ### Mixed bit widths per layer (edge survey) The survey describes mixed precision as choosing a bit width for each layer, and says quantization-aware training is advised to limit accuracy loss [arxiv:2605.06714 §4.3]. Its headline example is a mixed-precision ResNet-50 at 75.3% top-1 on ImageNet, 10x smaller than full precision [arxiv:2605.06714 §4.3]. The survey quotes this from a cited work and gives no bit widths for it, and it runs no experiments of its own [arxiv:2605.06714 §4.3]. Card: [Edge Deep Learning in Computer Vision and Medical Diagnostics](/papers/2605.06714). ### Fully quantized training on a microcontroller (FQT) Deutel et al. run the forward pass, the backward pass and the weight update on 8-bit tensors, using the same per-tensor linear scheme that inference uses [arxiv:2407.10734 §3.1]. Stored weights stay 8-bit. Each update dequantizes them, takes a floating-point step, and then re-fits the scale and zero point to the minimum and maximum of the new values, with the range split into 255 steps [arxiv:2407.10734 §3.1, eq. 5–7]. The weight gradient itself is not re-quantized [arxiv:2407.10734 §3.1, eq. 2]. Gradients accumulate per sample over the minibatch rather than being held all at once. The optimizer uses no momentum or adaptive state, to save memory [arxiv:2407.10734 §3.1]. The transfer-learning experiments use seven datasets [arxiv:2407.10734 §4.1, tab. 1]. Three boards with Cortex-M0+, M4 and M7 cores are compared for latency and energy, but only on two datasets [arxiv:2407.10734 §4.2, tab. 2]. The uint8 setting matches or exceeds the GPU baseline on the simpler datasets, and is lower on flowers, cifar100 and speech. The mixed setting, which keeps the classification head in float32, matches float [arxiv:2407.10734 §4.1, fig. 4]. On a CIFAR-10 comparison against MCUNet-5FPS, the paper reports 34.8% less memory and 49.0% lower latency for its model, at 85.9% against 89.5% accuracy. It does not say whether the memory figure is ROM or RAM [arxiv:2407.10734 §4.5, fig. 9]. Its MCUNet table puts uint8 at 73.7 average, against 73.5 for the int8 SGD+M+QAS result of Lin et al. and 73.3 for float SGD-M; the first three rows are the cited work's numbers [arxiv:2407.10734 §4.5, tab. 4]. Card: [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734). For the adapter-side and training-side context, see [On-device training](/concepts/on-device-training). ### Reasoning model on an edge GPU (AWQ W4 in EdgeReasoning) Kubwimana and Huang apply AWQ with 4-bit weights and 16-bit activations to DeepSeek-R1 distilled models at 1.5B, 8B and 14B parameters, served with vLLM on a Jetson AGX Orin [arxiv:2511.01866 §5.6]. The relative accuracy loss on MMLU-Redux is 1.04% for 1.5B, 6.16% for 8B and 0.62% for 14B [arxiv:2511.01866 §5.6]. The Orin's Ampere GPU has no INT4 support, so the 4-bit weights run through INT8 kernels on this device [arxiv:2511.01866 §5.6]. The text gives a latency gain of 2 to 5 times, larger for larger models. The appendix gives one figure for 8B, 87.16 s to 14.69 s, and none for 14B [arxiv:2511.01866 §5.6, app. B]. The quantization result sits in a study that also varies output length and parallel sampling [arxiv:2511.01866 §5]. Card: [EdgeReasoning](/papers/2511.01866). For the reasoning-length side, see [Test-time scaling](/concepts/test-time-scaling). ## When to use which Only what the cited results support. Neither the QLoRA paper nor the LLM.int8() paper measures inference speed for its 4-bit or 8-bit LLM variant, except as noted. | Situation | Use | Why | |---|---|---| | Compact CNN, 8-bit weights, no retraining | Asymmetric per-channel weights, post-training | 0.703 against 0.709 float on MobileNet-v1; per-layer gives 0.001 [arxiv:1806.08342 tab. 3] | | Choosing between symmetric and asymmetric 8-bit weights | Try asymmetric per-channel first | Symmetric per-channel gives 0.591 on MobileNet-v1 against 0.703 asymmetric [arxiv:1806.08342 tab. 3]. The paper's conclusions recommend symmetric per-channel [arxiv:1806.08342 §8]; see Contradictions | | 8-bit PTQ loses too much accuracy | Quantization-aware training with batch-norm correction and freezing | Asymmetric per-layer goes from 0.001 (PTQ) to 0.70 (QAT) on MobileNet-v1 [arxiv:1806.08342 tab. 4] | | Binary weights and binary activations, accuracy matters more than the one-bit baseline | ABC-Net with M = N = 5 | 65.0% Top-1 on ResNet-18 against 51.2% for XNOR-Net, but 4.3 points below full precision [arxiv:1711.11294 tab. 2, tab. 3]. Each layer costs M×N binary convolutions and no speed is measured [arxiv:1711.11294 eq. 12, §5.2] | | Binary weights, full-precision activations | ABC-Net with M = 5 | 68.3% Top-1 against 60.8% for BWN and 69.3% for full precision on ResNet-18 [arxiv:1711.11294 tab. 1] | | 4-bit weights for a CNN | Quantization-aware training, not PTQ | Symmetric per-channel PTQ gives 0.001 on MobileNet-v1; QAT gives 0.65. QAT reaches 0.62 to 0.76, 0.02 to 0.10 below float, across nine networks [arxiv:1806.08342 tab. 5] | | Integer inference on phone CPU or DSP | 8-bit fixed-point kernels | MobileNet-v1 latency 155 ms float, 68 ms fixed-point CPU, 16 ms on the DSP via NN-API (one Pixel 2 large core) [arxiv:1806.08342 tab. 7] | | Pretrained LLM, 8-bit weights and activations, no retraining, quality first | LLM.int8() with $\alpha = 6.0$ | Within 0.2 perplexity of 32-bit from 125M to 13B [arxiv:2208.07339 tab. 1]. Set the threshold explicitly; see Implementation sketch | | Pretrained LLM, every matrix product in INT8, no 16-bit outlier path | SmoothQuant at O1, and O3 only if static activation scales are needed | OPT-175B 66.5 at O1 and 66.8 at O3, against 66.9 for float [arxiv:2211.10438 tab. 3]. BLOOM-176B at O3 is 0.8 points below float [arxiv:2211.10438 tab. 4]. The perplexity script has no O3 switch and the repository has no latency benchmark, so test accuracy and speed on your own model | | LLM, large model, memory first, batch 1 | Measure LLM.int8() on your own kernel | Speed is 0.86x to 1.22x of 16-bit at dimensions 4096 to 5140 [arxiv:2208.07339 app. D.1]; the paper reports no speed gain for batch 1 at 176B (253 ms vs 239 ms) [app. D.2] | | Finished 7B model, memory first, 4-bit | AWQ 4-bit with the GEMM kernel | 4.68 GB VRAM against 15.02 GB, and 51.7 average against 50.3 for 16-bit; GEMM is faster than GEMV [arxiv:2402.10373 tab. 4, app. D, tab. 7]. The gain is concentrated in one task | | Finished 7B model, quality first, 8-bit | BitsandBytes 8-bit | Average within 0.3 of 16-bit (50.0 against 50.3), 8.04 GB VRAM [arxiv:2402.10373 tab. 4, app. D, tab. 7]. Runs about 4.3x slower than 16-bit in that setup | | 4-bit frozen base for finetuning, quality first | NF4 + DQ | Best 4-bit perplexity in [arxiv:2305.14314 tab. 2]; matches 16-bit LoRA on MMLU (53.1 vs 53.0 mean) [arxiv:2305.14314 tab. 4] and on Super-NaturalInstructions for T5-3B/11B [arxiv:2305.14314 tab. 3] | | Need to save the last few GB on large models | Add DQ | Saves 0.373 bits/param, about 3 GB at 65B [arxiv:2305.14314 §3] | | Choosing between 4-bit types | Prefer NF4 over FP4 and Int4 | FP4 trails NF4 + DQ on MMLU (52.2 vs 53.1) [arxiv:2305.14314 tab. 4] and in perplexity [arxiv:2305.14314 tab. 2] | | Learned low-bit quantization with skewed activations (e.g. Swish), quantization-aware training | Asymmetric activations with learned offset (LSQ+) | 73.8% against 71.9% for LSQ at W4A4 on EfficientNet-B0 [arxiv:2004.09576 tab. 2]. Requires training; the paper gives no training recipe [arxiv:2004.09576 §3.2.2] | | Learned low-bit quantization, run-to-run stability at 2 bits matters | Initialise activation scale and offset by MSE fit | 46.8% ± 1.9 against 43.8% ± 4.7 for min-max over five runs [arxiv:2004.09576 tab. 5] | | 7B robot policy, memory first, accuracy within noise | int4 inference | 71.9% ± 4.7 against 71.3% ± 4.8 for bfloat16, 7.0 GB against 16.8 GB, on eight tasks with 80 rollouts [arxiv:2406.09246 §5.4]. The gap is far inside the standard errors; the paper gives no speed figure for the bfloat16 run | | Weights programmed into analog in-memory hardware with 8-bit converters | Hardware-aware training, then check drift over time | Five of eleven networks reach iso-accuracy at one hour after retraining, against one for direct mapping [arxiv:2302.08469 §2.2, §2.3, tab. 2, tab. 3]. Convolutional networks are the hardest to recover [arxiv:2302.08469 §2.7, fig. 7] | | 7B robot policy, int8 | Measure the control rate before judging the precision | int8 scored 58.1% under non-blocking control and 74.4% under blocking control [arxiv:2406.09246 §5.4, app. D.4] | | Weights far from normal | Unknown | NF4's normality assumption is only approximately met [arxiv:2305.14314 app. F]; no evaluation on non-normal weights | | Throughput-critical LLM serving with NF4 | Not established | Every matmul dequantizes to BF16; the paper claims runtime is not degraded but gives no throughput table [arxiv:2305.14314 §1, §3]. The latency numbers above are for fixed-point CNN inference, not NF4 | | CNN, 4-bit weights, no retraining, about 1024 unlabelled images, nearest rounding too lossy | AdaRound, with the scale fitted by weight error | ResNet-18 at 68.60% against 23.99% for nearest and 69.68% for float [arxiv:2004.10568 tab. 4, tab. 6]. Not tested on language models [arxiv:2004.10568 tab. 7] | | Pretrained decoder LLM, 3 or 4-bit weights, weight-only, no retraining, calibration data available | GPTQ, row-wise or grouped | 4-bit OPT-175B 8.37 against 8.34 FP16 and 10.54 round-to-nearest; 3-bit 8.68 where round-to-nearest gives 7.3e3 [arxiv:2210.17323 tab. 5]. Quantizing the 175B model takes about 4.2 hours on one A100 [arxiv:2210.17323 tab. 2]. The calibration set is C4 training text, so C4 results are not zero-shot [arxiv:2210.17323 tab. 11, tab. 12] | | Decoder LLM at batch 1 on a fixed GPU budget, weight-only, speed is the goal | 3-bit GPTQ kernels, measured on your own hardware | 3.24x and 4.53x per token against FP16 on several GPUs, a figure that includes the reduction in GPU count [arxiv:2210.17323 tab. 6]. A bitsandbytes 4-bit run on one RTX 3090 was slower than 16-bit [arxiv:2402.10373 tab. 7] | | Mixed bit widths per layer on an edge device | Quantization-aware training, as the survey advises | The survey advises training to limit accuracy loss, and quotes 75.3% top-1 at 10x smaller for a mixed-precision ResNet-50 from a cited work [arxiv:2605.06714 §4.3]. These are quoted, not checked | | Training a small CNN on an MCU, weights and backward-pass errors in 8 bits | Fully quantized training with per-tensor uint8 and partial gradient updates; start with lambda_min 0.5 | With lambda_min 0.1 the uint8 runs became unstable, and 0.5 stayed close to 1.0 [arxiv:2407.10734 §4.3, fig. 6]. Accuracy on the harder transfer datasets falls behind the GPU baseline, so measure on your own data [arxiv:2407.10734 §4.1] | | Reasoning LLM on an edge GPU, latency-bound, 4-bit weights acceptable | AWQ W4 with vLLM, measured on your own device | Relative accuracy loss 1.04% (1.5B), 6.16% (8B), 0.62% (14B) on MMLU-Redux [arxiv:2511.01866 §5.6]. The Orin runs W4 on INT8 kernels, and the appendix's 5.9x latency figure is for 8B only [arxiv:2511.01866 §5.6, app. B] | | Large decoder LLM on consumer GPUs, 4-bit weights, FP16 activations, no calibration pass | Row-wise absmax INT4, as in GLM-130B, and measure it on your model first | LAMBADA 79.47 against 80.21 for FP16 on the 130B model, on four 24 GB cards [arxiv:2210.02414 §4]. The same scheme fails on BLOOM at INT4 [arxiv:2210.02414 app. A]. Where calibration data is available, the GPTQ row above is the alternative | | Model too large for one consumer GPU, batch 1, ReLU-family activations, INT4 acceptable | PowerInfer's neuron split with INT4 weights, measured on your hardware | 2.89 times average (4.28 max) over llama.cpp INT4 on RTX 4090, and 5.01 times average on RTX 2080 Ti [arxiv:2312.12456 §8.2, fig. 13]. The gain is smaller for SiLU-family models [arxiv:2312.12456 §4, §8.4]. These are author-reported numbers, not a comparison with FP16 [arxiv:2312.12456 §8.1] | | Decoder serving at batch 1, where memory traffic sets speed | Weight quantization is the lever the memory-wall argument points to; measure the real kernel first | Decoder inference is memory-bound at batch 1 [arxiv:2403.14123 §2.2, fn.]. But every quantized run in [arxiv:2402.10373 tab. 7] was slower than 16-bit on one RTX 3090, so the saved bytes did not translate into speed there | ## Implementation sketch Uniform affine grid, per output channel, from the CNN paper. The paper does not give the formula for the scale and zero-point, so they are left as a call. ```text # Per output channel: one scale and zero-point [arxiv:1806.08342 §2.6] quantize_channel(w_c): # w_c: weights of one output channel lo, hi = min(w_c), max(w_c) lo, hi = min(lo, 0), max(hi, 0) # range must include zero [§2.1] delta, z = grid_params(lo, hi) # not specified in the paper [§2.1] q = clamp(0, 255, round(w_c / delta) + z) # 8-bit codes [eq. 1-2] return q, delta, z dequantize_channel(q, delta, z): return (q - z) * delta # [eq. 3] ``` Outlier split for a linear layer, from LLM.int8(). The paper's Einstein-notation sum (eq. 8) is written here as two matmuls: ```text # Mixed-precision linear layer [arxiv:2208.07339 §3.1, §3.2, eq. 7-8] llm_int8_linear(X, W, alpha = 6.0): # X: (s, h) activations; W: (h, o) # Feature dims with any value of magnitude >= alpha [§3.2] outliers = dims where max_row |X| >= alpha regular = all dims except outliers # One scale per row of X and per column of W [§3.1] s_x = 127 / absmax_per_row(X[:, regular]) s_w = 127 / absmax_per_col(W[regular, :]) # Integer product, then dequantize by the outer product of scales [eq. 7] C_int = round(s_x * X[:, regular]) @ round(s_w * W[regular, :]) C = C_int / outer(s_x, s_w) # Outlier dims in 16-bit [eq. 8] C = C + X[:, outliers] @ W[outliers, :] return C ``` Two details the paper leaves open. It does not say whether outlier values are excluded from the per-row maximum, and the outlier set is defined over the data being evaluated [arxiv:2208.07339 §3.2]. Set $\alpha$ explicitly. The reference library's default threshold of 0.0 turns the 16-bit path off, and the paper shows the path without it fails above 6.7B [arxiv:2208.07339 tab. 1]. The [card](/papers/2208.07339) has the code-level details. Blockwise NF4 storage, from the QLoRA paper: ```text NF4 = [16 values from app. E] # hard-code; do not re-derive from eq. 4 quantize(W): # storage blocks = flatten(W).split(64) c2 = [absmax(b) for b in blocks] # FP32, one per 64 weights codes = [argmin_j |b_i / c2_b - NF4[j]| for each value] # 4-bit indices # double quantization of the constants mu = mean(c2); c2_blocks = (c2 - mu).split(256) c1 = [absmax(cb) for cb in c2_blocks] # FP32, one per 256 constants c2_fp8 = fp8_quantize(c2_blocks / c1) store codes, c2_fp8, c1, mu dequantize(codes, c2_fp8, c1, mu) -> W_bf16: # at every matmul, forward and backward c2 = fp8_dequant(c2_fp8) * c1 + mu W = NF4[codes] * c2 (per block) -> cast BF16 Y = X_bf16 @ dequantize(...) # no gradient stored for W ``` The rounding to the nearest NF4 value is not spelled out in the paper; it lives in `bitsandbytes` (see card) [arxiv:2305.14314 §3]. Note the paper's notation: in the DQ paragraph $c_2$ is the first-level (per-64) constant and $c_1$ the second-level one [arxiv:2305.14314 §3]. GPTQ's column loop for one weight matrix, from the paper's Algorithm 1. The paper's grid is min-max per row [arxiv:2210.17323 §5 Setup]. The reference code also puts zero inside the range and rounds the zero point, which the paper does not mention, so a grid built from the paper alone will differ. The sketch leaves `fit_row_grid` open for that reason. ```text # GPTQ for one weight matrix [alg. 1] [§4] # W: (d_row, d_col) weights; H = 2 X X^T from calibration inputs def gptq(W, H, blocksize=128, percdamp=0.01): # Dampen H by a fraction of its mean diagonal [§4 Step 3] H = H + percdamp * mean(diagonal(H)) * identity(d_col) # Only the upper Cholesky factor of inverse(H) is used [§4 Step 3] U = cholesky_upper(inverse(H)) # One min-max grid per row, fitted on the original W [§5 Setup] grid = fit_row_grid(W) Q = zeros_like(W) for start in range(0, d_col, blocksize): # lazy block updates [§4 Step 2] end = min(start + blocksize, d_col) block_err = zeros(d_row, end - start) for j in range(start, end): # fixed column order [§4 Step 1] Q[:, j] = quantize(W[:, j], grid) # Rounding error, scaled by the Cholesky diagonal [alg. 1] err = (W[:, j] - Q[:, j]) / U[j, j] block_err[:, j - start] = err # Move the error onto the rest of the block [alg. 1] W[:, j:end] -= outer(err, U[j, j:end]) # Move the block's errors onto all later columns [alg. 1] W[:, end:] -= block_err @ U[start:end, end:] return Q ``` ## Contradictions Between papers, the main point is that the measures differ, so numbers should not be set side by side: - [arxiv:2208.07339 tab. 1] reports C4 perplexity within 0.2 of 32-bit for 8-bit LLM weights and activations. [arxiv:2402.10373 tab. 4] reports BitsandBytes 8-bit at 0.3 below 16-bit on medical QA. These are different models, tasks and metrics. - [arxiv:2302.08469 tab. 3] reports normalized accuracy against a floating-point reference on a simulated analog crossbar, with noise and drift. The integer-grid papers above report perplexity or accuracy against their own 32-bit or 16-bit baselines. The two sets of numbers are not comparable. - [arxiv:2403.14123 §3.2] cites INT4 quantization as having "minimal accuracy impact" with up to 8x footprint reduction. Its authors do not test this. [arxiv:2402.10373 tab. 4] reports BitsandBytes 4-bit at 3.4 points below 16-bit average, driven by PubMedQA. Both are cited as they are; neither resolves the other. - Speed at batch 1. [arxiv:2210.17323 tab. 6] reports 3.24x per token for 3-bit OPT-175B on one A100 against FP16 on five. [arxiv:2402.10373 tab. 7] reports every quantized BioMistral run slower than 16-bit on one RTX 3090. [arxiv:2208.07339 app. D.2] reports LLM.int8() at 253 ms per token against 239 ms for bfloat16 on BLOOM-176B. The hardware, kernels and baselines differ, and GPTQ's baseline uses more GPUs than its quantized run, so the three results do not measure the same thing. - ResNet-18 at 4 bits, AdaRound against GPTQ. [arxiv:2004.10568 tab. 4] reports 68.60 for AdaRound against 69.68 for float. [arxiv:2210.17323 tab. 1] lists AdaRound at 69.34 against 69.76 for float. The two rows have different float baselines, and the GPTQ table's AdaRound setup is not stated in the text read here. Neither is resolved here. - Plain 4-bit rounding on large LLMs. [arxiv:2210.17323 tab. 5] reports round-to-nearest at 4 bits on OPT-175B as WikiText2 perplexity 10.54 against 8.34 for FP16. [arxiv:2210.02414 §4] reports row-wise absmax INT4 on GLM-130B as LAMBADA 79.47 against 80.21, and [arxiv:2210.02414 app. A] reports the same scheme failing on BLOOM at INT4. The models, metrics and grids differ, since GPTQ uses asymmetric min-max grids and GLM-130B uses symmetric absmax, so the results are not comparable. They do show that the loss at 4 bits depends on the model. - Speed of 8-bit LLM products. [arxiv:2211.10438 §5.3] reports up to 1.56 times FP16 in FasterTransformer. [arxiv:2208.07339 app. D.1] reports LLM.int8() at 0.86 times FP16 at model dimension 4096 and 1.22 times at 5140. The kernels and baselines differ, so the figures are not comparable. - Speed of 4-bit AWQ weights. [arxiv:2402.10373 tab. 7] reports the AWQ GEMM run on one RTX 3090 at 57.96 s against 40.94 s for 16-bit, so it was slower. [arxiv:2511.01866 app. B] reports 87.16 s to 14.69 s for DeepSeek-R1-Llama-8B on a Jetson AGX Orin, a 5.9x gain. The two papers differ in model, hardware, serving stack and task, so neither result transfers to the other setup. Within [arxiv:1806.08342](/papers/1806.08342): - Text against table: §3.1.3 says asymmetric per-layer gives the best weight-only accuracy. Table 2 shows asymmetric per-channel is best, and per-layer drops the MobileNets to 0.001 to 0.004 [arxiv:1806.08342 §3.1.3, tab. 2]. - Symmetric or asymmetric: the conclusions recommend symmetric per-channel as the PTQ starting point [arxiv:1806.08342 §8]. Table 3 gives 0.591 for symmetric per-channel and 0.703 for asymmetric per-channel on MobileNet-v1 [arxiv:1806.08342 tab. 3]. Within [arxiv:1711.11294](/papers/1711.11294): the introduction says the method reaches accuracy comparable to full precision [arxiv:1711.11294 §1]. That holds only with full-precision activations (68.3% against 69.3%). With binary activations at M = N = 5 the gap is 4.3 points on ResNet-18 [arxiv:1711.11294 tab. 2]. The text gives a 0.9-point gap at M = 5 with full-precision activations, while Table 1 gives 1.0 [arxiv:1711.11294 §4.1.1, tab. 1]. Use the table. Within [arxiv:2402.10373](/papers/2402.10373): the text says BitsandBytes 8-bit gains 0.65 on Clinical KG and 1.00 on Anatomy [arxiv:2402.10373 §5.5]. Table 4 gives +0.4 and +0.5 on the same tasks [arxiv:2402.10373 tab. 4]. Use the table. Within [arxiv:2004.09576](/papers/2004.09576): the introduction gives a 4.1% loss for LSQ at W4A4, while Table 2 gives 76.1 − 71.9 = 4.2 points [arxiv:2004.09576 §1, tab. 2]. The 73.8% in Table 2 is the best value over a grid search and the 73.0% in Table 5 is a five-run mean; the two are not comparable [arxiv:2004.09576 tab. 2, tab. 5]. Within [arxiv:2406.09246](/papers/2406.09246): int8 scores 58.1% under non-blocking control but 74.4% under blocking control, so the int8 drop depends on the control setup [arxiv:2406.09246 §5.4, app. D.4]. The int4 result is 0.6 points above bfloat16, well inside the reported standard errors of about 5 points [arxiv:2406.09246 §5.4]. The authors attribute the int8 difference to control speed rather than precision [arxiv:2406.09246 app. D.4]. Within [arxiv:2407.10734](/papers/2407.10734): the text says the optimizer outperforms SGD+M+QAS on five of eight MCUNet datasets, while the table shows higher values on four, a tie on one and lower values on three [arxiv:2407.10734 §4.5, tab. 4]. The average (73.7 against 73.5) matches the table. Eq. 9 is meant to lower the update rate as the loss falls, but as written it does so in the opposite direction, because the epsilon term is the gap to the largest loss seen so far [arxiv:2407.10734 §3.2, eq. 9]. The paper does not resolve this. Within [arxiv:2511.01866](/papers/2511.01866): the text gives a 2 to 5 times latency gain from quantization, and the appendix gives 5.9 times for 8B and no 14B latency [arxiv:2511.01866 §5.6, app. B]. Within [arxiv:2305.14314](/papers/2305.14314): the NF4 construction in the text (estimate $2^k+1$ quantiles), eq. 4 ($2^k$ midpoints) and the asymmetric prose description do not line up, and tail offsets are not stated; use the appendix values [arxiv:2305.14314 §3, eq. 4, app. E]. Eq. 5-6 label the constants $c_2^{\text{k-bit}}$ while the text says FP8 [arxiv:2305.14314 §3, eq. 6]. ## Papers - [Quantizing deep convolutional networks for efficient inference: A whitepaper](/papers/1806.08342): per-channel and per-layer uniform grids for CNNs, with post-training quantization and quantization-aware training using batch-norm correction and freezing. - [Towards Accurate Binary Convolutional Neural Network](/papers/1711.11294): approximates weights by a weighted sum of M binary bases and activations by N binary thresholds, so a convolution becomes M×N bitwise convolutions. - [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](/papers/2208.07339): 8-bit projection layers for pretrained LLMs with vector-wise absmax scales and a 16-bit path for outlier feature dimensions. - [QLoRA: Efficient Finetuning of Quantized LLMs](/papers/2305.14314): introduces NF4 and double quantization for a frozen 4-bit base model under LoRA finetuning. - [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](/papers/2402.10373): compares AWQ 4-bit and BitsandBytes 4/8-bit on a 7B medical model for accuracy, memory and speed. - [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](/papers/2004.09576): learned scale and offset for activations, with weight scales from weight statistics and an MSE-fitted activation start. - [Rasch et al., hardware-aware training on analog crossbars](/papers/2302.08469): retrains 11 networks with a simulated analog in-memory crossbar, with 8-bit converters, drift and noise in the forward pass. - [OpenVLA: An Open-Source Vision-Language-Action Model](/papers/2406.09246): reports bfloat16, int8 and int4 inference for a 7B robot policy, with memory, control rate and success. - [AI and Memory Wall](/papers/2403.14123): argues that memory bandwidth limits LLM inference, and surveys quantization as a cited remedy; it runs no quantization experiments. - [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](/papers/2210.17323): column-by-column 3- and 4-bit weight quantization of GPT-style models, with error compensation from calibration inputs and grouped scales. - [Edge deep learning survey for vision and medical diagnostics](/papers/2605.06714): survey of pruning, quantisation and lightweight networks for on-device vision; quotes results from cited works and runs no experiments. - [Up or Down? Adaptive Rounding for Post-Training Quantization](/papers/2004.10568): AdaRound, which chooses each weight's rounding direction per layer by minimising the layer's output error on unlabelled images. - [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414): a 130B bilingual model whose weights are quantized row-wise to INT4 with FP16 activations, evaluated on LAMBADA, MMLU and Pile. - [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](/papers/2211.10438): per-channel smoothing factors that move activation outliers into the weights, for W8A8 inference of LLMs. - [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932): PRISMA review of TinyML that sorts quantization, pruning and distillation into a taxonomy; no experiments. - [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734): fully quantized training on Cortex-M boards, with per-tensor uint8 weights and sparse per-sample error updates. - [EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs](/papers/2511.01866): measures latency, energy and accuracy of distilled reasoning models on a Jetson AGX Orin, including AWQ W4. - [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](/papers/2312.12456): hybrid GPU-CPU engine that places hot neurons on the GPU and cold ones on the CPU, tested at FP16 and INT4 weights.