Neural Network Quantization for Efficient Inference: A Survey

In plain words

Neural networks are usually trained with 32-bit floating-point numbers, which makes them large and slow on small devices. Quantization stores and computes with fewer bits instead, for example 8-bit integers, so the model takes less memory and the arithmetic is cheaper. The catch is that fewer bits lose information, and accuracy can drop.

This paper is a survey of how people have handled that trade-off over the last decade. It explains how computers represent numbers, how a real value is mapped onto a small grid of integers, and the two usual ways to do it: retrain the network with the rounding in place, or quantize a finished network with little or no retraining. It then goes through integer and fixed-point schemes, binary and ternary schemes that use one or two bits per weight, and mixed precision, where each layer gets its own bit width. It ends with open problems.

The paper does not add a new method. Its value is the map of the field and the summary tables that let a reader compare the reported results.

TL;DR

• A survey, not a method: it explains number formats, the quantization function, QAT and PTQ, and four scheme families §1 §2 §3 §4.
• Integer/fixed-point schemes are the practical default; binary and ternary save the most arithmetic but lose the most accuracy §5 §6 tab. 2.
• Mixed precision gives each layer its own bit width, but the search space is exponential in the number of layers §7.
• All accuracy numbers are copied from the cited works and are author-reported, not re-measured here tab. 3 tab. 4.

Problem

Trained networks are over-parameterized and are often stored and run in 32-bit floating point. That makes them too large for edge hardware and expensive to compute §1. Floating-point arithmetic needs a dedicated unit, while integer arithmetic does not, so the survey asks how far precision can be cut before accuracy suffers §1 §2.1.

The survey assumes the reader knows basic neural networks and SGD training §1. It restricts its scope to inference: quantizing weights and activations. Gradient quantization for training is only touched on where it helps inference §1.

Method

The survey defines the basic quantization function and then reviews the schemes. The steps below are the core definitions an implementer needs, taken from the paper's own equations.

Number formats. A single-precision float stores a sign bit, an 8-bit exponent and a 23-bit mantissa fig. 1 §2.1. A fixed-point value stores a sign bit, integer bits and mantissa bits, with one global exponent that is shared by all values fig. 2 §2.2 eq. 4. Only the value of the global exponent differs per format.

Quantization function. A real value r maps to an integer through a scale S and an integer zero point Z §3.1 eq. 5. Rounding makes the reverse mapping inexact, so the dequantized value is only an approximation §3.2 eq. 7.

Scale. For a clipping range [alpha, beta] and bit width b, the step size is (beta - alpha) / (2^b - 1) §3.1.1 eq. 6. The clipping range is found by calibration. Symmetric quantization sets alpha = -beta and gives Z = 0; asymmetric quantization chooses alpha != -beta to fit an unbalanced range, for example after ReLU §3.1.1.

# Step size for clipping range [alpha, beta] and b bits  [§3.1.1] [eq. 6]
# Inputs: alpha, beta (floats), bits (integer)
# Output: scale (float)
def step_size(alpha, beta, bits):
    return (beta - alpha) / (2 ** bits - 1)

# Uniform quantization of one value  [§3.1] [eq. 5]
# Inputs: r (float), scale S > 0, zero_point Z (integer)
# Output: integer Q(r), not clipped to a bit range here
def quantize(r, scale, zero_point):
    # Scale the value, then round it to the integer grid  [§3.1]
    level = round(r / scale)
    # The paper subtracts the zero point  [eq. 5]
    return level - zero_point

# Reverse mapping; not equal to r because of rounding  [§3.2] [eq. 7]
def dequantize(q, scale, zero_point):
    return scale * (q + zero_point)

Two procedures §4 tab. 1.

• Quantization-aware training (QAT): quantize, run a forward pass, backpropagate with floating-point gradients, update the float weights, and repeat until convergence §4.1. The rounding has no gradient, so a straight-through estimator is used, which treats the rounding as the identity §4.1. The learned clipping range can also be trained, as in PACT §4.1.
• Post-training quantization (PTQ): calibrate the quantization parameters on a small set of unlabelled data, then quantize the model without training §4.2.

Integer and fixed point §5.

• Batch norm can be folded into the preceding weights and bias. The folded weight is gamma * w / sqrt(var + eps) and the folded bias is given in eq. 8 eq. 9 §5.
• Scale from calibration, with the maximum magnitude MAX: S = MAX / (2^b - 1) eq. 10 §5.
• Adaptive rounding (AdaRound) chooses round-up or round-down per weight to reduce layer output error, using the unlabelled data §5 §7.

Binary §6.1.

• BinaryConnect draws each weight stochastically: +1 with probability sigma(w) and -1 otherwise, where sigma is the hard sigmoid eq. 11 eq. 12.
• BinaryNet uses the sign function, with Sign(w) = +1 if w >= 0 and -1 otherwise eq. 13. With binary weights and activations, a dot product becomes XNOR and popcount eq. 14.
• Binary-weight networks and XNOR-Nets approximate each weight as alpha * w_b, where alpha is the mean absolute weight of the matrix eq. 15 §6.1.

Ternary §6.2.

• Ternary Weight Networks use a threshold Delta of about 0.7 * mean(|w|) §6.2. The weight becomes +1 above the threshold, -1 below its negative, and 0 in between eq. 16.
• The scale alpha is the mean of |w| over the weights above the threshold §6.2.
• Trained Ternary Quantization learns two separate scales, one for the positive and one for the negative values, and learns the threshold during training §6.2.

Mixed precision §7.

• HAWQ uses the Hessian spectrum as a sensitivity measure, and HAWQ-V2 uses the average Hessian trace §7.
• HAWQ-V2 scores a bit-width setting by the sum over layers of avg_trace(H_i) times the squared L2 distance between quantized and float weights eq. 17.
• HAWQ-V3 uses dyadic scale factors b / 2^c and integer linear programming under size, operation and latency limits §7.
• AdaQuant chooses step sizes per layer by minimizing the output error on a small calibration set, with an integer program for the bit widths and a further distillation step §7 eq. 18.
• Other works search bit widths with reinforcement learning, neural architecture search, or learned step sizes and bit-level sparsity §7.

Hyperparameters

The survey gives few training hyperparameters. The table lists the values it states. Nothing else is stated.

Results

Author-reported. The survey does not run experiments; it reports numbers from the works it cites, with the setup shown in the tables. Tables 3 and 4 are copied in full. The "Arithmetic" column follows the survey's check marks.

CIFAR-10 tab. 3

ImageNet tab. 4

Headline statements in the text (author-reported):

• A ternary-weight ResNet18 (TWN, 2-bit weights) incurs "only" 4% accuracy loss on ImageNet §6.2. The survey gives no float baseline for this row in its table.
• Gong et al. report less than 1% loss for 8-bit integer inference §5.
• AdaRound reports around 1% loss for 4-bit weights on ResNet18/50 §5; the table lists 68.55 (ResNet18) and 75.01 (ResNet50) without baselines.
• Zhuang et al.'s combined schemes: 4-bit AlexNet and ResNet50 marginally beat float; 2-bit AlexNet on CIFAR-100 loses under 1% §5.
• TTQ ResNets beat float ResNets on CIFAR-10 by under 0.5% §6.2.

Paper vs code

No official code. The survey has no implementation. The paper and its arXiv page link no repository or project page. I searched the LaTeX source for URLs (none apart from the ACM template), the arXiv abstract page (no code link), and GitHub for "neural network quantization survey". The three GitHub hits belong to other owners, and none is by the author. The other repositories are not official.

Pitfalls

• Exponent bias is wrong in the text. The paper gives the float32 bias as 2^8 - 1 = 255 and uses 2^(x - 255) in eq. 2, with a denormal exponent of -254 in eq. 3 §2.1. The commented-out text in the LaTeX source gives 127 and the range -126 to +127 [src: 02-background.tex, line 34], which is the value IEEE 754 single precision uses. Implement the standard 127, and treat the 255 as a typo.
• Zero-point sign. eq. 5 subtracts Z and eq. 7 adds it. Code that uses the other convention is off by 2Z on every value. Check against the framework you target.
• How to get Z and the scale is not fully given. The text defines the scale from alpha and beta eq. 6 and the zero point as "the integer that represents 0", but does not give a formula for Z from alpha and beta. Write it as not stated, or derive it from the grid you choose.
• Scale rule with signed values. eq. 10 uses MAX / (2^b - 1) with b = 7 for signed weights, which is the paper's own reading of the source method §5. The text does not show how this formula relates to eq. 6 for symmetric signed ranges. Check which convention your kernel uses.
• Threshold formula typo. The TWN sum is printed as \sum_{i + 1}^n §6.2. The index should start at i = 1. The 0.7 factor is an approximation that assumes normally distributed weights, not an exact optimum §6.2.
• Binarization details. BinaryConnect keeps float master weights that receive the float gradients, and the binary copies are used only in the forward and backward passes §6.1. Deterministic sign binarization gives +1 at zero eq. 13.
• Cross-reference error. §7 says batch-norm fusion before PTQ is problematic, citing "Equation (eq:norm)". That label is the float normalized-value equation eq. 2, not the folding equations eq. 8 eq. 9.
• AdaQuant notation. eq. 18 defines W' = W + V and then writes W_q = Q(W + V_hat). The two forms differ in the role of V. The text calls V a continuous variable that gives the weights room to absorb rounding error §7.
• AdaRound precision label. Table 4 gives AdaRound precision 4/8 (weights 4, activations 8), but the text says AdaRound quantizes weights only and leaves the remaining parameters unquantized §5. Treat the activation bits as unclear.
• Table typo. Table 4 prints "TTN" for the ternary weight network (TWN).
• Comparability. The tables mix QAT and PTQ, different networks, different first/last-layer treatment, and mixed-precision entries with no average bit width. Do not rank rows across columns without checking the source.
• Hidden costs. Ternary asymmetric scales need two floating-point multiplications per activation tab. 2. Dyadic scales remove floating point but still need an integer program over the bit widths §7. Mixed-precision search is exponential in the number of layers §7.
• Uniform precision limit. The survey says that for minimal loss, integer networks are limited by the most sensitive layer, usually 8 bits, so the compression is at most about 4x tab. 2. This is the author's claim, not a measurement.
• Tables are secondhand. Every number in tab. 3 and tab. 4 is copied from a cited paper. Check the original before using a figure.

Lineage

• Builds on: the survey cites Gholami et al., "A Survey of Quantization Methods for Efficient Neural Network Inference" (arXiv:2103.13630), for its definitions and calibration material §3.1 §4. The paper has no card in this wiki.
• Compared against: none; the survey compares the schemes it reviews, from its own tables tab. 2 tab. 3 tab. 4.
• Superseded by: not stated in this wiki's content.