Weight quantization

Overview

Weight quantization stores each weight as a low-bit integer or code, plus shared scale constants, and turns it back into a number when it is used. The eighteen papers on this page cover different parts of that design space. Quantizing deep convolutional networks maps CNN weights and activations to 8-bit integers for integer-only inference on phones and DSPs 1806.08342 §1, §2. Towards Accurate Binary Convolutional Neural Network writes each layer's weights as a weighted sum of M binary filters and its activations as N binary thresholds, so that a convolution becomes M×N bitwise convolutions 1711.11294 §3.1.1, §3.2, eq. 12. LLM.int8() runs the projection-layer matrix products of a trained transformer in 8-bit integers, without retraining, and keeps a few outlier feature dimensions in 16-bit 2208.07339 §1, §3.2. SmoothQuant instead divides outlier activation channels by per-channel factors and multiplies the matching weight rows, so that every matrix product can run in 8 bits 2211.10438 §4. QLoRA stores a frozen LLM base in 4 bits so that LoRA adapters can be finetuned on top 2305.14314 §1. BioMistral applies AWQ 4-bit and BitsandBytes 4/8-bit to a finished 7B model and measures accuracy, memory and speed 2402.10373 §3.4, §5.5. LSQ+ learns a scale and an offset for each activation quantizer and keeps weights symmetric, so that skewed activations use the full integer range 2004.09576 §3.1, §3.2.1. OpenVLA runs a 7B robot policy at bfloat16, int8 and int4 inference precision and reports memory, control rate and success for each 2406.09246 §5.4. AI and Memory Wall supplies the motivation: decoder inference at small batch is limited by memory transfers, so fewer bytes per weight can matter 2403.14123 §2.2. It does not test quantization itself 2403.14123 §3. GPTQ rounds the weights of a trained LLM to 3 or 4 bits one column at a time, adjusting the columns not yet rounded, using calibration inputs 2210.17323 §3, §4. AdaRound keeps the grid fixed and chooses each weight's rounding direction by minimising the error of one layer's output on unlabelled inputs 2004.10568 §3. GLM-130B uses plain row-wise absmax INT4 on its linear-layer weights, with FP16 activations, and reports results close to FP16 on its own model 2210.02414 §4. The edge-deployment survey collects quantisation results from other works for on-device vision and runs no experiments of its own 2605.06714 §4.3. A Machine Learning-oriented Survey on Tiny Machine Learning places quantization under model optimization, next to pruning and distillation, and counts the cited works per period rather than measuring them 2309.11932 §5.1.2, tab. 1. On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers quantizes weights, feature maps and errors to 8 bits during training, with the weight update itself in floating point 2407.10734 §3.1. EdgeReasoning applies AWQ 4-bit weights to reasoning models on an edge GPU and measures latency, energy and accuracy 2511.01866 §5.6. PowerInfer splits each feed-forward block between one consumer GPU and the CPU by neuron, and reports speed-ups at FP16 and at INT4 weights, each against llama.cpp 2312.12456 §8.1, §8.2, fig. 13.

Five decisions recur across these papers. The first is the granularity of the scale: one range per layer, per output channel, per row or column, or per block of weights. The second is the number format: an integer grid (asymmetric or symmetric), a non-uniform 4-bit code, or a weighted sum of several binary bases 1711.11294 §3.1.1. The third is whether training sees the rounding. Quantization is applied to a model that is already trained in one case, and to a frozen base that is never updated in another. The fourth is what to do with outliers: a few large values can stretch a shared scale until most values lose precision 2208.07339 §3, §4.2. The fifth is how each rounding error is handled: left alone, or corrected by adjusting the weights that are not yet rounded, using calibration inputs 2210.17323 §3, §4 Step 1. AdaRound avoids the correction step: it chooses the rounding direction of each weight directly, by optimisation 2004.10568 §3.3.

The shared mechanism in QLoRA is a storage dtype (e.g. NF4) and a compute dtype (BF16). Whenever a weight is used it is dequantized and the matmul runs in 16-bit 2305.14314 §3. Tensors are quantized blockwise, each block of B values with its own absmax constant, so a single outlier only affects its own block 2305.14314 §2, eq. 1. For the adapter side, see Low-rank adaptation. For the CNN side, see Post-training quantization and Quantization-aware training. For why memory traffic rather than arithmetic sets decoder speed, see Arithmetic intensity and LLM serving.

Variants

Granularity of the scale

A scale and zero-point can be shared by a whole tensor (per-layer), by each output feature map (per-channel), or by each block of weights (QLoRA uses blocks of 64) 1806.08342 §2.6 2305.14314 §3. The per-layer choice fails on compact CNNs. On MobileNet-v1 with asymmetric 8-bit weights and no retraining, per-layer scales give 0.001 top-1 against 0.703 for per-channel 1806.08342 tab. 3. The paper traces this to batch-norm folding, which makes the kernels of one layer differ widely in range, so one shared grid wastes most of its levels 1806.08342 §3.1.3, app. A.

The same failure appears for LLMs. Using one absmax scale for a whole tensor gives C4 perplexity of 87.76 at 125M parameters and 19.08 at 13B, against 25.65 and 12.45 for 32-bit 2208.07339 tab. 1. LLM.int8() uses one scale per row of the activations and one per column of the weights instead 2208.07339 §3.1, eq. 7. The CNN and LLM papers measure different things, so their numbers are not directly comparable.

GPTQ fits one grid per row of each weight matrix by default, or one per group of consecutive weights within a row. Group sizes from 1024 down to 8 appear in its tables 2210.17323 §5, tab. 5, tab. 7. Its own bit counts show the cost: a group of 128 adds about 0.15 bits per weight and a group of 1024 about 0.02 2210.17323 §5 175 Billion Parameter Models.

Uniform affine and symmetric integer grids

An asymmetric grid maps a float range to integers 0 \ldots 255 with a scale \Delta and an integer zero-point z. The range is first widened to include zero, so zero padding is exact 1806.08342 §2.1, eq. 1–3. The paper does not state how \Delta and z are computed from the range 1806.08342 §2.1. A symmetric grid fixes z = 0 1806.08342 §2.2, eq. 7–8. Its signed 8-bit weights are clamped to [-127, 127] for faster SIMD 1806.08342 eq. 10. With an asymmetric grid, the convolution expands into extra cross terms that need specialized kernels; the paper argues that subtracting zero-points up front cuts throughput by 2x to 4x 1806.08342 §2.1, eq. 4–6.

QLoRA's 8-bit building block is also symmetric: each block is scaled by 127/\text{absmax} and rounded to Int8 (eq. 1), and dequantized by dividing by the constant (eq. 2) 2305.14314 §2. Its 4-bit integer analogue, Int4, gave the worst mean Pile Common Crawl perplexity of the 4-bit types tested: 34.34 2305.14314 tab. 2.

LLM.int8() also uses symmetric absmax grids: each tensor is multiplied by 127/\lVert X \rVert_\infty and rounded into [-127, 127] 2208.07339 §2.1. Its zero-point variant is only a baseline. The zero-point equation as printed in the paper is not a usable zero point, so implementers should not copy it 2208.07339 eq. 2.

Outlier-aware mixed precision (LLM.int8())

Starting at about 6.7B parameters, a few feature dimensions in every transformer layer take values up to about 20 times the others 2208.07339 §1, §4.2. Quantizing them with the rest makes absmax and zero-point quantization lose most of their precision 2208.07339 §4.2, §4.3. LLM.int8() handles this with a split: every feature dimension whose value reaches magnitude \alpha = 6.0 is multiplied in 16-bit, the others in int8, and the two products are summed 2208.07339 §3.2, eq. 8. The paper says at most 7 outlier dimensions appear in models up to 13B 2208.07339 §3.2.

The split is necessary. Vector-wise absmax quantization without the 16-bit path reaches perplexity 14.13 at 6.7B and 16.48 at 13B, against 13.30 and 12.45 for 32-bit. With the path added, the same model reads 13.24 and 12.45 2208.07339 tab. 1. Across 125M to 13B, the full method stays within 0.2 perplexity of 32-bit 2208.07339 tab. 1. Removing the outlier dimensions alone raises validation perplexity by 600% to 1,000%, while removing seven random dimensions changes it by about 0.1% 2208.07339 §4.2. Memory is about half of 16-bit, with about 0.1% extra for the decomposition 2208.07339 §3.2.

Speed is a separate question. On the first feed-forward layer of GPT-3-sized models, the full method runs at 0.86x of 16-bit at model dimension 4096 and 1.22x at 5140, with custom kernels 2208.07339 app. D.1. End to end on BLOOM-176B, batch 1 takes 253 ms against 239 ms for bfloat16 on 8 A100s 2208.07339 app. D.2. The paper's pitfalls note that the method is slower below about 6.7B in its own timings 2208.07339 §3.4, app. D.1. Its training results are narrower: the authors find 8-bit feed-forward training straightforward, but say other layers need more work 2208.07339 app. E.

All-INT8 matrix products with smoothing (SmoothQuant)

SmoothQuant moves the outlier problem from activations into weights, so that every linear layer and both attention matrix products run in W8A8, with no 16-bit path 2211.10438 §1, §4. For each input channel it sets a factor s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}. The activation channel is divided by s_j and the matching weight row is multiplied by it, so the product is unchanged 2211.10438 §4, eq. 3, eq. 4. Where the activation comes from a LayerNorm, the division folds into that LayerNorm offline 2211.10438 §4. The maxima come from 512 Pile sentences 2211.10438 §5.1.

Author-reported: naive W8A8 gets 35.5% average accuracy on OPT-175B, against 66.9% for float, and SmoothQuant at O3 gets 66.8% 2211.10438 tab. 3. On BLOOM-176B the O1, O2 and O3 settings give 68.3, 68.4 and 67.4, against 68.2 for float 2211.10438 tab. 4. On GLM-130B, O3 gives 72.8 against 73.8 2211.10438 tab. 4. Coarser settings are faster and cost accuracy: the paper reports about 0.8 points on BLOOM-176B for the coarsest one 2211.10438 §5.2. The paper reports up to 1.56 times FP16 speed in FasterTransformer and about half the memory 2211.10438 fig. 9, §5.3. The migration strength \alpha is 0.5 for OPT and BLOOM and 0.75 for GLM-130B 2211.10438 §4, §5.1. The repository has no latency benchmark, so the speed figures rest on the paper alone Card. For the outlier background, see LLM.int8() above and post-training quantization.

4-bit and 8-bit quantization of a finished 7B model (BioMistral)

BioMistral compares AWQ 4-bit and BitsandBytes 4-bit and 8-bit on its own 7B model 2402.10373 §3.4. Average 3-shot accuracy over ten English medical QA tasks is 50.3 for the unquantized model, 51.7 for AWQ 4-bit with the GEMM kernel, 51.8 with GEMV, 50.0 for BitsandBytes 8-bit and 46.9 for BitsandBytes 4-bit 2402.10373 tab. 4. The AWQ average is driven by PubMedQA: 61.7 with GEMV against 37.6, a gain of 24.1 points. The other nine tasks lose about 1.1 points on average with GEMV 2402.10373 tab. 4. BitsandBytes 4-bit drops PubMedQA to 22.4 2402.10373 tab. 4. The paper does not state the AWQ group size 2402.10373 §3.4.

On one RTX 3090, the MMLU professional-medicine set took 40.94 s in 16-bit with 15.02 GB of VRAM. BitsandBytes 8-bit took 177.75 s with 8.04 GB, BitsandBytes 4-bit took 133.06 s with 5.03 GB, and AWQ with GEMM took 57.96 s with 4.68 GB 2402.10373 app. D, tab. 7. All four quantized runs were slower than 16-bit in that setup. The paper recommends AWQ with GEMM as the best trade-off 2402.10373 §5.5. Card: BioMistral.

Quantized storage with fine-grained scales (AWQ, in BioMistral)

BioMistral describes AWQ as keeping the most important weights unquantized, on the observation that weights differ in importance 2402.10373 §3.4. The paper gives no formula for this, so the description is the only one the wiki can cite from it. Its results are in the previous variant.

Column-wise rounding with error compensation (GPTQ)

GPTQ quantizes each linear layer of a GPT-style model to 3 or 4 bits, with no retraining. It uses a calibration set of inputs to the layer, and quantizes the columns in one fixed order 2210.17323 §3, §4 Step 1. After a column is rounded, the error is spread over the columns that are still full precision, using the inverse of the layer's input Hessian H = 2XX^\top 2210.17323 §3. Calibration uses 128 random 2048-token segments of C4 training text 2210.17323 §5 Setup. The paper's reason for the fixed order and the block updates is runtime: OBQ's greedy order costs O(d_{row} \cdot d_{col}^3) per matrix, and GPTQ brings the work down to a level where OPT-175B takes 4.2 hours and BLOOM-176B 3.8 hours on one A100 80 GB 2210.17323 §3, §4 Step 1, tab. 2.

On OPT-175B the 4-bit model reaches WikiText2 perplexity 8.37 against 8.34 for FP16 and 10.54 for round-to-nearest. At 3 bits, round-to-nearest collapses to 7.3e3 and GPTQ reaches 8.68. Grouping with 128 weights per group gives 8.45 at 3 bits 2210.17323 tab. 5. The 2-bit results use groups of 128, 64 or 32, with OPT-175B at 8.94 for groups of 32 2210.17323 tab. 7. A ternary grid with groups of 8 reaches 9.20 on OPT-175B 2210.17323 §5 Extreme Quantization. The paper's 2-bit bit counts depend on how group constants are stored, so check the format before quoting one 2210.17323 §5 Extreme Quantization. Card: GPTQ.

Speed is reported for batch-1 generation of 128 tokens. 3-bit GPTQ takes 71 ms per token on one A100 against 230 ms for FP16 on five A100s (3.24x), and 130 ms on one A6000 against 589 ms on eight (4.53x) 2210.17323 tab. 6. The FP16 baseline uses several GPUs, so the speed-up includes the reduction in GPU count 2210.17323 tab. 6.

Learned rounding direction (AdaRound)

AdaRound keeps the grid fixed and chooses, for each weight, whether to round it down or up. It relaxes each choice to a continuous variable, minimises the error of one layer's output on a few unlabelled inputs, and then fixes each choice to 0 or 1 2004.10568 §3.3. The method covers convolutional and linear layers 2004.10568 §3.3. With 4-bit weights and float activations, ResNet-18 reaches 68.60% top-1 against 69.68% for float and 23.99% for nearest rounding 2004.10568 tab. 4 tab. 6. On ResNet-50 the figures are 75.23, 76.07 and 35.60 2004.10568 tab. 7. The main ablations use 1024 calibration images, and the paper reports about 10 minutes for ResNet-18 on one GTX 1080 Ti 2004.10568 §5. The reported results are for CNNs and a segmentation model; the paper has no language-model result 2004.10568 tab. 7 tab. 9. The reference implementation uses 15,000 iterations at 4 bits, against the paper's 10,000, and sets the regulariser and annealing schedule that the paper does not state; see the card and adaptive rounding. The scale is fitted by weight error 2004.10568 §5, and the gain depends on it: with min-max scales AdaRound reaches 61.96 against 0.23 for nearest rounding 2004.10568 tab. 6.

Weight-only INT4 with row-wise absmax (GLM-130B)

GLM-130B quantizes only the weights of the QKV, attention-output and both MLP projections in each layer, to INT4 2210.02414 §4. Each output row gets one scale, its absolute maximum divided by 7, and the weights are dequantized to FP16 at run time. Activations stay in FP16 2210.02414 §4 app. A. The appendix also tests a zero-point variant, which scores 80.63 on LAMBADA against 79.47 for absmax. The authors kept absmax because it performed well and is cheaper to compute 2210.02414 app. A.

On the 130B model, INT4 weights give LAMBADA 79.47, MMLU 44.80 and Pile bits-per-byte 0.641, against 80.21, 44.75 and 0.634 in FP16 2210.02414 §4. INT8 weights give 80.21, 44.71 and 0.638 2210.02414 §4. The quantized weights take about 70 GB, which fits on four RTX 3090 cards with 24 GB each 2210.02414 §4. The INT4 decoding times are reported for several GPU setups, but without an FP16 run on the same setups, so this variant gives no speed ratio 2210.02414 §4. The same scheme is reported to fail on BLOOM at INT4 2210.02414 app. A. The released weights carry a non-commercial licence, while the code is Apache-2.0; the card has the details. The method itself is not restricted by this.

INT4 weights in a neuron-split CPU-GPU engine (PowerInfer)

PowerInfer keeps frequently active feed-forward neurons on the GPU and computes the rarely active ones on the CPU 2312.12456 §4. Its evaluation runs this engine with FP16 weights and with INT4 weights 2312.12456 §8.1. The INT4 speed-ups are measured against llama.cpp at INT4, not against FP16, so they do not show how much the neuron split adds on top of quantization. On PC-High (RTX 4090) the average over models is 2.89 times, with a maximum of 4.28 times. On PC-Low (RTX 2080 Ti) the average is 5.01 times and the maximum 8.00 times 2312.12456 §8.2, fig. 13. The paper's gains depend on activation sparsity, and it reports them as largest for ReLU-family models 2312.12456 §4, §8.4. These are author-reported numbers on two desktop machines 2312.12456 §8.1. Card: PowerInfer.

4-bit floating point (FP4)

Float4 with E2M1 or E3M0 layouts: mean perplexity 31.07 and 29.48 respectively 2305.14314 tab. 2. In LLaMA 7-65B finetuning, FP4 QLoRA averaged 52.2 5-shot MMLU vs 53.0 for 16-bit LoRA 2305.14314 tab. 4.

4-bit NormalFloat (NF4)

Quantile quantization for a N(0,1) source, rescaled to [-1,1], so each bin gets an equal expected share of normally distributed weights 2305.14314 §3, eq. 4. The code is asymmetric so that it contains an exact zero; the 16 values are listed in the paper's appendix 2305.14314 §3, app. E. Block size is 64 2305.14314 §3. It assumes weights are roughly zero-mean normal; 7.5% of LLaMA-7B hidden units fail a Shapiro-Wilk test at 5% 2305.14314 app. F. Card: QLoRA: Efficient Finetuning of Quantized LLMs.

Double quantization (DQ)

The per-64-weight FP32 constants are themselves quantized to FP8 with block size 256 (after subtracting their mean, since they are all positive). Constant overhead drops from 0.5 to 0.127 bits per parameter, about 3 GB on a 65B model 2305.14314 §1, §3. NF4 + DQ gave the best mean perplexity in the comparison, 27.41 2305.14314 tab. 2.

Learned scale and offset for activations (LSQ+)

LSQ+ learns a scale s and an offset \beta for each activation quantizer, and reconstructs each value as \bar{x}s + \beta 2004.09576 §3.1, eq. 2. The offset lets a skewed activation such as Swish use all 2^b levels. A symmetric grid either spends one bit on a sign or clamps the negative part to zero 2004.09576 §1, §3.1. Weights stay symmetric and signed 2004.09576 §3.2.1. The offset term folds into a per-output bias at compile time 2004.09576 eq. 5. The paper reports zero runtime overhead for this 2004.09576 §6; the fold needs a layer that accepts a bias term.

On EfficientNet-B0 at 4 bits for weights and activations, the asymmetric configurations reach 73.8% against 71.9% for LSQ and 76.1% at full precision 2004.09576 tab. 2. At 2 bits the gap is larger: 48.7% for the asymmetric unsigned configuration against 43.5% for LSQ 2004.09576 tab. 2. Learning the offset beats fixing it at zero or at the observed minimum: 73.8% against 71.9% and 72.5% 2004.09576 tab. 6. The initialisation matters for variance. At 2 bits over five runs, min-max start gives 43.8% ± 4.7 and the MSE fit on activations gives 46.8% ± 1.9 2004.09576 tab. 5. The paper starts all experiments from pretrained float weights and does not test training from scratch 2004.09576 §4. It does not state the optimiser or learning rate used for the MSE fit 2004.09576 §3.2.2. Card: LSQ+.

Several binary bases per layer (ABC-Net)

ABC-Net writes a layer's real weights as a weighted sum of M binary filters, W ≈ α_1 B_1 + … + α_M B_M, with each B_m equal to +1 or −1 1711.11294 §3.1.1, eq. 1. Each basis is the sign of the layer's centred weights shifted by a multiple of their standard deviation, and the scales α are fitted by least squares once the bases are fixed 1711.11294 §3.1.1, eq. 2–3. One set of bases covers the whole layer. A separate set per output channel is described but left as future work, and with M = 1 the scheme reduces to BWN 1711.11294 §3.1.2. Weights stay real during training and are binary at test time 1711.11294 §3. On ResNet-18 with full-precision activations, Top-1 rises from 60.8% for BWN to 62.8% at M = 1 and 68.3% at M = 5, against 69.3% for full precision 1711.11294 tab. 1.

The same paper also binarizes the activations at N thresholds, each with its own trainable shift and scale, so a convolution becomes M×N binary convolutions 1711.11294 §3.2, eq. 11–12. With five bases and five activation thresholds on ResNet-18, Top-1 is 65.0%, against 51.2% for XNOR-Net and 69.3% for full precision 1711.11294 tab. 2, tab. 3. The paper argues that bitwise operations are cheaper than fixed-point multiplies, but reports no measured speed 1711.11294 §5.2. Training starts from a full-precision pretrained model and uses straight-through gradients through the sign function 1711.11294 §3.3, eq. 7. For the one-bit case as a whole, see Binary neural network. Card: Towards Accurate Binary Convolutional Neural Network.

Analog in-memory weights with 8-bit converters (hardware-aware training)

This variant does not store weights as low-bit integers. Each weight matrix is mapped to analog conductances on crossbar tiles of at most 512 by 512 cells, with one conductance pair per weight in the paper's model 2302.08469 Methods: Weight programming, Methods: Crossbar tile size. The analog product is read through an 8-bit DAC on the inputs and an 8-bit ADC on the outputs 2302.08469 app. A.1. The stored values are perturbed by programming error and by drift over time 2302.08469 eq. 7–9. The paper's question is how much retraining recovers under that noise. Direct mapping of trained weights, with no retraining, misses the iso-accuracy target for almost all of its eleven networks; only LSTM on PTB reaches 99.0% normalized accuracy at one hour 2302.08469 §2.2, tab. 2. Hardware-aware training runs the crossbar model in the forward pass and floating-point gradients in the backward pass, and afterwards five of the eleven networks reach iso-accuracy at one hour 2302.08469 §2.3, tab. 3. Output-side noise and converter resolution have the largest effect on accuracy, and the paper finds that a DAC can drop from 8 to 6 bits with little accuracy impact 2302.08469 §2.4, §2.5, fig. 4. Card: Rasch et al., hardware-aware training on analog crossbars.

Inference precision of a 7B robot policy (OpenVLA)

OpenVLA is a 7B vision-language-action model that predicts robot actions as tokens 2406.09246 §3.2. The paper reports three inference precisions on eight BridgeData V2 tasks with 80 rollouts each 2406.09246 §5.4. Success is 71.3% ± 4.8 in bfloat16 with 16.8 GB of GPU memory, 58.1% ± 5.1 in int8 with 10.2 GB, and 71.9% ± 4.7 in int4 with 7.0 GB 2406.09246 §5.4. Inference rates on an A5000 are 1.2 Hz for int8 and 3 Hz for int4; the bfloat16 rate is not given as one figure 2406.09246 §5.4. Under blocking control the three precisions score 70.0%, 74.4% and 68.8%, and the authors read the int8 drop as a control-speed effect rather than a precision effect 2406.09246 app. D.4. The quantization runs use a SigLIP-only checkpoint trained on the Octo mixture, so their numbers are not comparable with the main results 2406.09246 §5.3. The released code also has a 4-bit NF4 option for fine-tuning, which the paper does not evaluate; see OpenVLA for the code details.

Mixed bit widths per layer (edge survey)

The survey describes mixed precision as choosing a bit width for each layer, and says quantization-aware training is advised to limit accuracy loss 2605.06714 §4.3. Its headline example is a mixed-precision ResNet-50 at 75.3% top-1 on ImageNet, 10x smaller than full precision 2605.06714 §4.3. The survey quotes this from a cited work and gives no bit widths for it, and it runs no experiments of its own 2605.06714 §4.3. Card: Edge Deep Learning in Computer Vision and Medical Diagnostics.

Fully quantized training on a microcontroller (FQT)

Deutel et al. run the forward pass, the backward pass and the weight update on 8-bit tensors, using the same per-tensor linear scheme that inference uses 2407.10734 §3.1. Stored weights stay 8-bit. Each update dequantizes them, takes a floating-point step, and then re-fits the scale and zero point to the minimum and maximum of the new values, with the range split into 255 steps 2407.10734 §3.1, eq. 5–7. The weight gradient itself is not re-quantized 2407.10734 §3.1, eq. 2. Gradients accumulate per sample over the minibatch rather than being held all at once. The optimizer uses no momentum or adaptive state, to save memory 2407.10734 §3.1.

The transfer-learning experiments use seven datasets 2407.10734 §4.1, tab. 1. Three boards with Cortex-M0+, M4 and M7 cores are compared for latency and energy, but only on two datasets 2407.10734 §4.2, tab. 2. The uint8 setting matches or exceeds the GPU baseline on the simpler datasets, and is lower on flowers, cifar100 and speech. The mixed setting, which keeps the classification head in float32, matches float 2407.10734 §4.1, fig. 4. On a CIFAR-10 comparison against MCUNet-5FPS, the paper reports 34.8% less memory and 49.0% lower latency for its model, at 85.9% against 89.5% accuracy. It does not say whether the memory figure is ROM or RAM 2407.10734 §4.5, fig. 9. Its MCUNet table puts uint8 at 73.7 average, against 73.5 for the int8 SGD+M+QAS result of Lin et al. and 73.3 for float SGD-M; the first three rows are the cited work's numbers 2407.10734 §4.5, tab. 4. Card: On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers. For the adapter-side and training-side context, see On-device training.

Reasoning model on an edge GPU (AWQ W4 in EdgeReasoning)

Kubwimana and Huang apply AWQ with 4-bit weights and 16-bit activations to DeepSeek-R1 distilled models at 1.5B, 8B and 14B parameters, served with vLLM on a Jetson AGX Orin 2511.01866 §5.6. The relative accuracy loss on MMLU-Redux is 1.04% for 1.5B, 6.16% for 8B and 0.62% for 14B 2511.01866 §5.6. The Orin's Ampere GPU has no INT4 support, so the 4-bit weights run through INT8 kernels on this device 2511.01866 §5.6. The text gives a latency gain of 2 to 5 times, larger for larger models. The appendix gives one figure for 8B, 87.16 s to 14.69 s, and none for 14B 2511.01866 §5.6, app. B. The quantization result sits in a study that also varies output length and parallel sampling 2511.01866 §5. Card: EdgeReasoning. For the reasoning-length side, see Test-time scaling.

When to use which

Only what the cited results support. Neither the QLoRA paper nor the LLM.int8() paper measures inference speed for its 4-bit or 8-bit LLM variant, except as noted.

Implementation sketch

Uniform affine grid, per output channel, from the CNN paper. The paper does not give the formula for the scale and zero-point, so they are left as a call.

# Per output channel: one scale and zero-point  [arxiv:1806.08342 §2.6]
quantize_channel(w_c):                            # w_c: weights of one output channel
    lo, hi = min(w_c), max(w_c)
    lo, hi = min(lo, 0), max(hi, 0)               # range must include zero  [§2.1]
    delta, z = grid_params(lo, hi)                # not specified in the paper  [§2.1]
    q = clamp(0, 255, round(w_c / delta) + z)     # 8-bit codes  [eq. 1-2]
    return q, delta, z

dequantize_channel(q, delta, z):
    return (q - z) * delta                        # [eq. 3]

Outlier split for a linear layer, from LLM.int8(). The paper's Einstein-notation sum (eq. 8) is written here as two matmuls:

# Mixed-precision linear layer  [arxiv:2208.07339 §3.1, §3.2, eq. 7-8]
llm_int8_linear(X, W, alpha = 6.0):          # X: (s, h) activations; W: (h, o)
    # Feature dims with any value of magnitude >= alpha  [§3.2]
    outliers = dims where max_row |X| >= alpha
    regular  = all dims except outliers
    # One scale per row of X and per column of W  [§3.1]
    s_x = 127 / absmax_per_row(X[:, regular])
    s_w = 127 / absmax_per_col(W[regular, :])
    # Integer product, then dequantize by the outer product of scales  [eq. 7]
    C_int = round(s_x * X[:, regular]) @ round(s_w * W[regular, :])
    C = C_int / outer(s_x, s_w)
    # Outlier dims in 16-bit  [eq. 8]
    C = C + X[:, outliers] @ W[outliers, :]
    return C

Two details the paper leaves open. It does not say whether outlier values are excluded from the per-row maximum, and the outlier set is defined over the data being evaluated 2208.07339 §3.2. Set \alpha explicitly. The reference library's default threshold of 0.0 turns the 16-bit path off, and the paper shows the path without it fails above 6.7B 2208.07339 tab. 1. The card has the code-level details.

Blockwise NF4 storage, from the QLoRA paper:

NF4 = [16 values from app. E]                     # hard-code; do not re-derive from eq. 4

quantize(W):                                      # storage
    blocks = flatten(W).split(64)
    c2 = [absmax(b) for b in blocks]              # FP32, one per 64 weights
    codes = [argmin_j |b_i / c2_b - NF4[j]| for each value]   # 4-bit indices
    # double quantization of the constants
    mu = mean(c2); c2_blocks = (c2 - mu).split(256)
    c1 = [absmax(cb) for cb in c2_blocks]         # FP32, one per 256 constants
    c2_fp8 = fp8_quantize(c2_blocks / c1)
    store codes, c2_fp8, c1, mu

dequantize(codes, c2_fp8, c1, mu) -> W_bf16:     # at every matmul, forward and backward
    c2 = fp8_dequant(c2_fp8) * c1 + mu
    W = NF4[codes] * c2 (per block) -> cast BF16
Y = X_bf16 @ dequantize(...)                      # no gradient stored for W

The rounding to the nearest NF4 value is not spelled out in the paper; it lives in bitsandbytes (see card) 2305.14314 §3. Note the paper's notation: in the DQ paragraph c_2 is the first-level (per-64) constant and c_1 the second-level one 2305.14314 §3.

GPTQ's column loop for one weight matrix, from the paper's Algorithm 1. The paper's grid is min-max per row 2210.17323 §5 Setup. The reference code also puts zero inside the range and rounds the zero point, which the paper does not mention, so a grid built from the paper alone will differ. The sketch leaves fit_row_grid open for that reason.

# GPTQ for one weight matrix  [alg. 1] [§4]
# W: (d_row, d_col) weights; H = 2 X X^T from calibration inputs
def gptq(W, H, blocksize=128, percdamp=0.01):
    # Dampen H by a fraction of its mean diagonal  [§4 Step 3]
    H = H + percdamp * mean(diagonal(H)) * identity(d_col)
    # Only the upper Cholesky factor of inverse(H) is used  [§4 Step 3]
    U = cholesky_upper(inverse(H))
    # One min-max grid per row, fitted on the original W  [§5 Setup]
    grid = fit_row_grid(W)
    Q = zeros_like(W)
    for start in range(0, d_col, blocksize):   # lazy block updates  [§4 Step 2]
        end = min(start + blocksize, d_col)
        block_err = zeros(d_row, end - start)
        for j in range(start, end):            # fixed column order  [§4 Step 1]
            Q[:, j] = quantize(W[:, j], grid)
            # Rounding error, scaled by the Cholesky diagonal  [alg. 1]
            err = (W[:, j] - Q[:, j]) / U[j, j]
            block_err[:, j - start] = err
            # Move the error onto the rest of the block  [alg. 1]
            W[:, j:end] -= outer(err, U[j, j:end])
        # Move the block's errors onto all later columns  [alg. 1]
        W[:, end:] -= block_err @ U[start:end, end:]
    return Q

Contradictions

Between papers, the main point is that the measures differ, so numbers should not be set side by side:

• 2208.07339 tab. 1 reports C4 perplexity within 0.2 of 32-bit for 8-bit LLM weights and activations. 2402.10373 tab. 4 reports BitsandBytes 8-bit at 0.3 below 16-bit on medical QA. These are different models, tasks and metrics.
• 2302.08469 tab. 3 reports normalized accuracy against a floating-point reference on a simulated analog crossbar, with noise and drift. The integer-grid papers above report perplexity or accuracy against their own 32-bit or 16-bit baselines. The two sets of numbers are not comparable.
• 2403.14123 §3.2 cites INT4 quantization as having "minimal accuracy impact" with up to 8x footprint reduction. Its authors do not test this. 2402.10373 tab. 4 reports BitsandBytes 4-bit at 3.4 points below 16-bit average, driven by PubMedQA. Both are cited as they are; neither resolves the other.
• Speed at batch 1. 2210.17323 tab. 6 reports 3.24x per token for 3-bit OPT-175B on one A100 against FP16 on five. 2402.10373 tab. 7 reports every quantized BioMistral run slower than 16-bit on one RTX 3090. 2208.07339 app. D.2 reports LLM.int8() at 253 ms per token against 239 ms for bfloat16 on BLOOM-176B. The hardware, kernels and baselines differ, and GPTQ's baseline uses more GPUs than its quantized run, so the three results do not measure the same thing.
• ResNet-18 at 4 bits, AdaRound against GPTQ. 2004.10568 tab. 4 reports 68.60 for AdaRound against 69.68 for float. 2210.17323 tab. 1 lists AdaRound at 69.34 against 69.76 for float. The two rows have different float baselines, and the GPTQ table's AdaRound setup is not stated in the text read here. Neither is resolved here.
• Plain 4-bit rounding on large LLMs. 2210.17323 tab. 5 reports round-to-nearest at 4 bits on OPT-175B as WikiText2 perplexity 10.54 against 8.34 for FP16. 2210.02414 §4 reports row-wise absmax INT4 on GLM-130B as LAMBADA 79.47 against 80.21, and 2210.02414 app. A reports the same scheme failing on BLOOM at INT4. The models, metrics and grids differ, since GPTQ uses asymmetric min-max grids and GLM-130B uses symmetric absmax, so the results are not comparable. They do show that the loss at 4 bits depends on the model.
• Speed of 8-bit LLM products. 2211.10438 §5.3 reports up to 1.56 times FP16 in FasterTransformer. 2208.07339 app. D.1 reports LLM.int8() at 0.86 times FP16 at model dimension 4096 and 1.22 times at 5140. The kernels and baselines differ, so the figures are not comparable.
• Speed of 4-bit AWQ weights. 2402.10373 tab. 7 reports the AWQ GEMM run on one RTX 3090 at 57.96 s against 40.94 s for 16-bit, so it was slower. 2511.01866 app. B reports 87.16 s to 14.69 s for DeepSeek-R1-Llama-8B on a Jetson AGX Orin, a 5.9x gain. The two papers differ in model, hardware, serving stack and task, so neither result transfers to the other setup.

Within arxiv:1806.08342:

• Text against table: §3.1.3 says asymmetric per-layer gives the best weight-only accuracy. Table 2 shows asymmetric per-channel is best, and per-layer drops the MobileNets to 0.001 to 0.004 1806.08342 §3.1.3, tab. 2.
• Symmetric or asymmetric: the conclusions recommend symmetric per-channel as the PTQ starting point 1806.08342 §8. Table 3 gives 0.591 for symmetric per-channel and 0.703 for asymmetric per-channel on MobileNet-v1 1806.08342 tab. 3.

Within arxiv:1711.11294: the introduction says the method reaches accuracy comparable to full precision 1711.11294 §1. That holds only with full-precision activations (68.3% against 69.3%). With binary activations at M = N = 5 the gap is 4.3 points on ResNet-18 1711.11294 tab. 2. The text gives a 0.9-point gap at M = 5 with full-precision activations, while Table 1 gives 1.0 1711.11294 §4.1.1, tab. 1. Use the table.

Within arxiv:2402.10373: the text says BitsandBytes 8-bit gains 0.65 on Clinical KG and 1.00 on Anatomy 2402.10373 §5.5. Table 4 gives +0.4 and +0.5 on the same tasks 2402.10373 tab. 4. Use the table.

Within arxiv:2004.09576: the introduction gives a 4.1% loss for LSQ at W4A4, while Table 2 gives 76.1 − 71.9 = 4.2 points 2004.09576 §1, tab. 2. The 73.8% in Table 2 is the best value over a grid search and the 73.0% in Table 5 is a five-run mean; the two are not comparable 2004.09576 tab. 2, tab. 5.

Within arxiv:2406.09246: int8 scores 58.1% under non-blocking control but 74.4% under blocking control, so the int8 drop depends on the control setup 2406.09246 §5.4, app. D.4. The int4 result is 0.6 points above bfloat16, well inside the reported standard errors of about 5 points 2406.09246 §5.4. The authors attribute the int8 difference to control speed rather than precision 2406.09246 app. D.4.

Within arxiv:2407.10734: the text says the optimizer outperforms SGD+M+QAS on five of eight MCUNet datasets, while the table shows higher values on four, a tie on one and lower values on three 2407.10734 §4.5, tab. 4. The average (73.7 against 73.5) matches the table. Eq. 9 is meant to lower the update rate as the loss falls, but as written it does so in the opposite direction, because the epsilon term is the gap to the largest loss seen so far 2407.10734 §3.2, eq. 9. The paper does not resolve this.

Within arxiv:2511.01866: the text gives a 2 to 5 times latency gain from quantization, and the appendix gives 5.9 times for 8B and no 14B latency 2511.01866 §5.6, app. B.

Within arxiv:2305.14314: the NF4 construction in the text (estimate 2^k+1 quantiles), eq. 4 (2^k midpoints) and the asymmetric prose description do not line up, and tail offsets are not stated; use the appendix values 2305.14314 §3, eq. 4, app. E. Eq. 5-6 label the constants c_2^{\text{k-bit}} while the text says FP8 2305.14314 §3, eq. 6.

Papers

• Quantizing deep convolutional networks for efficient inference: A whitepaper: per-channel and per-layer uniform grids for CNNs, with post-training quantization and quantization-aware training using batch-norm correction and freezing.
• Towards Accurate Binary Convolutional Neural Network: approximates weights by a weighted sum of M binary bases and activations by N binary thresholds, so a convolution becomes M×N bitwise convolutions.
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale: 8-bit projection layers for pretrained LLMs with vector-wise absmax scales and a 16-bit path for outlier feature dimensions.
• QLoRA: Efficient Finetuning of Quantized LLMs: introduces NF4 and double quantization for a frozen 4-bit base model under LoRA finetuning.
• BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains: compares AWQ 4-bit and BitsandBytes 4/8-bit on a 7B medical model for accuracy, memory and speed.
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization: learned scale and offset for activations, with weight scales from weight statistics and an MSE-fitted activation start.
• Rasch et al., hardware-aware training on analog crossbars: retrains 11 networks with a simulated analog in-memory crossbar, with 8-bit converters, drift and noise in the forward pass.
• OpenVLA: An Open-Source Vision-Language-Action Model: reports bfloat16, int8 and int4 inference for a 7B robot policy, with memory, control rate and success.
• AI and Memory Wall: argues that memory bandwidth limits LLM inference, and surveys quantization as a cited remedy; it runs no quantization experiments.
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: column-by-column 3- and 4-bit weight quantization of GPT-style models, with error compensation from calibration inputs and grouped scales.
• Edge deep learning survey for vision and medical diagnostics: survey of pruning, quantisation and lightweight networks for on-device vision; quotes results from cited works and runs no experiments.
• Up or Down? Adaptive Rounding for Post-Training Quantization: AdaRound, which chooses each weight's rounding direction per layer by minimising the layer's output error on unlabelled images.
• GLM-130B: An Open Bilingual Pre-trained Model: a 130B bilingual model whose weights are quantized row-wise to INT4 with FP16 activations, evaluated on LAMBADA, MMLU and Pile.
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models: per-channel smoothing factors that move activation outliers into the weights, for W8A8 inference of LLMs.
• A Machine Learning-oriented Survey on Tiny Machine Learning: PRISMA review of TinyML that sorts quantization, pruning and distillation into a taxonomy; no experiments.
• On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers: fully quantized training on Cortex-M boards, with per-tensor uint8 weights and sparse per-sample error updates.
• EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs: measures latency, energy and accuracy of distilled reasoning models on a Jetson AGX Orin, including AWQ W4.
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU: hybrid GPU-CPU engine that places hot neurons on the GPU and cold ones on the CPU, tested at FP16 and INT4 weights.