Quantizing deep convolutional networks for efficient inference: A whitepaper

In plain words

Neural networks normally store every weight and intermediate value as a 32-bit floating-point number. Phones and small chips work much faster, and use less memory and power, when they compute with 8-bit integers instead. Quantization means rounding each number onto a small grid of evenly spaced levels and storing which level it landed on, plus a scale that converts it back.

This whitepaper is a practical guide to doing that for image-classification networks. Its first lesson is that the grid should be chosen separately for each filter in a layer, not once for the whole layer. Filters in the same layer can have very different value ranges, especially after batch normalization is merged into the weights. One shared grid then wastes most of its levels on a few outliers. With a grid per filter, simply converting an already trained model to 8 bits stays close to the original accuracy.

The second lesson is that training with simulated rounding helps even more, and matters most at 4 bits. During training the network sees the rounded weights in its forward pass, so it learns to tolerate them. Batch normalization needs special care here: if the rounded weights shift from batch to batch, accuracy jitters. The paper gives a correction scheme that fixes this.

It also measures the payoff: 8-bit models are four times smaller, and in the author's tests ran 2 to 3 times faster on a phone CPU and close to 10 times faster on a DSP built for low-precision vector math.

TL;DR

Uniform affine quantization maps a float range to integers with a scale \Delta and an integer zero-point z; the symmetric variant fixes z=0 §2.1, §2.2. Post-training quantization with per-channel weights and per-layer 8-bit activations stays within about 2% top-1 of float on ten ImageNet CNNs, while per-layer weights collapse MobileNets to ~0.1% top-1 tab. 2, tab. 3. Quantization-aware training (simulated quantization plus a straight-through estimator) brings even per-layer 8-bit models close to float and makes 4-bit weights usable tab. 4, tab. 5. Batch norm must be folded with a correction factor and later frozen to avoid jitter in the quantized weights §3.2.2. Read it if you deploy integer CNNs or build a QAT pipeline; it is a set of engineering recipes and measurements, not a new algorithm.

Problem

Edge devices have limited compute, memory and power, and models must be small to download §1. Lowering the precision of weights and activations works on existing architectures without redesign. It cuts model size 4x at 8 bits, reduces working memory and cache pressure for activations, enables faster 8-bit arithmetic, and lowers the energy of data movement §1. The open questions the paper addresses are which quantizer and granularity to use, when retraining is needed, and how to handle batch normalization §2, §3.

Method

Quantizers §2

Uniform affine (asymmetric) quantizer §2.1. A float range (x_{min}, x_{max}) maps to integers 0 \ldots N_{levels}-1, with N_{levels}=256 for 8 bits. The range is first relaxed to include zero, so that zero (used e.g. by zero padding) is exactly representable; z is an integer.

x_{int} = \mathrm{round}\left(\frac{x}{\Delta}\right) + z \quad \text{[eq. 1]} x_Q = \mathrm{clamp}(0, N_{levels}-1, x_{int}) \quad \text{[eq. 2]} x_{float} = (x_Q - z)\,\Delta \quad \text{[eq. 3]}

The paper does not write out how \Delta and z are computed from (x_{min}, x_{max}); it cites the NNAPI and Jacob et al. §2.1.

Cost of the zero-point §2.1, eq. 4–6. A convolution of quantized weights and activations expands to

y(k,l,n) = \Delta_w \Delta_x\, \mathrm{conv}\big(w_Q(k,l,m;n) - z_w,\; x_Q(k,l,m) - z_x\big) \quad \text{[eq. 4]} y(k,l,n) = \mathrm{conv}\big(w_Q(k,l,m;n), x_Q(k,l,m)\big) - z_w \sum_{k=0}^{K-1}\sum_{l=0}^{K-1}\sum_{m=0}^{N-1} x_Q(k,l,m) \quad \text{[eq. 5]} \quad - z_x \sum_{k=0}^{K-1}\sum_{l=0}^{K-1}\sum_{m=0}^{N-1} w_Q(k,l,m;n) + z_x z_w \quad \text{[eq. 6]}

Subtracting zero-points before the convolution forces 16/32-bit operands, which cuts throughput by 2x to 4x. Using the expansion, the last term is constant and the two cross terms each need N multiplies. The weight sum can be precomputed, and the activation sum is shared by all kernels of the same size; this needs specialized kernels (see gemmlowp) §2.1.

Uniform symmetric quantizer §2.2: z = 0.

x_{int} = \mathrm{round}(x/\Delta) \quad \text{[eq. 7]} x_Q = \mathrm{clamp}(-N_{levels}/2,\; N_{levels}/2 - 1,\; x_{int}) \;\text{(signed)} \quad \text{[eq. 8]} x_Q = \mathrm{clamp}(0,\; N_{levels} - 1,\; x_{int}) \;\text{(unsigned)} \quad \text{[eq. 9]}

For faster SIMD, the weight range is narrowed further eq. 10, eq. 11: signed \mathrm{clamp}(-(N_{levels}/2-1), N_{levels}/2-1, x_{int}), i.e. [-127, 127] at 8 bits; unsigned \mathrm{clamp}(0, N_{levels}-2, x_{int}). De-quantization is x_{out} = x_Q \Delta §2.2.

Stochastic quantizer §2.3: adds noise before rounding, x_{int} = \mathrm{round}((x+\epsilon)/\Delta) + z with \epsilon \sim \mathrm{Unif}(-\tfrac12, \tfrac12), then clamps as in eq. 2. In expectation it passes floats through, with saturation outside the range. It is not used for inference because most hardware lacks it §2.3.

Simulated quantization and its gradient §2.4. Training inserts a quantize-then-dequantize op:

x_{out} = \mathrm{SimQuant}(x) = \Delta\, \mathrm{clamp}\big(0, N_{levels}-1, \mathrm{round}(x/\Delta) - z\big) \quad \text{[eq. 12, eq. 13]}

(see Pitfalls on the sign of z). For the backward pass, the quantizer is treated as x_{out} = \mathrm{clamp}(x_{min}, x_{max}, x) eq. 14, fig. 1, which is a straight-through estimator: the incoming gradient passes where x_{min} \le x \le x_{max} and is zeroed outside eq. 15.

Quantizer parameters §2.5. Weights use their actual min and max. Activations use a moving average of per-batch min and max. KL-divergence calibration (as in TensorRT) is mentioned but not used.

Granularity §2.6. Per-layer means one (\Delta, z) for the whole tensor. Per-channel means one (\Delta, z) per convolution kernel, i.e. per output feature map of the 4-D weight tensor. Activations are always per-layer, because per-channel activations would complicate the inner products of conv and matmul.

Post-training quantization §3.1

• Weight-only (no data needed): convert weights to 8 bits and run inference in float; this only saves size §3.1.1, fig. 2.
• Weights and activations: needs calibration data to estimate activation ranges; about 100 mini-batches are typically enough for the ranges to converge §3.1.2.

Quantization-aware training §3.2

Simulated quantization is applied to weights and activations in both forward and backward passes. A float copy of the weights is kept and updated, so small updates accumulate instead of underflowing §3.2. For SGD:

w_{float} = w_{float} - \eta\, \frac{\partial L}{\partial w_{out}} \cdot I_{w_{out} \in (w_{min}, w_{max})} \quad \text{[eq. 16]} w_{out} = \mathrm{SimQuant}(w_{float}) \quad \text{[eq. 17]}

Graph rewriting rules §3.2.1:

• Element-wise add and concat need rescaling of their fixed-point inputs to a common scale, which must be modeled in training fig. 6, fig. 7.
• Ops fused at inference must be fused in training too. For example, put no fake-quant op between an add and its following ReLU §3.2.1.

Batch normalization folding with correction §3.2.2

Batch norm uses batch statistics in training, x_{bn} = \gamma (x - \mu_B)/\sigma_B + \beta eq. 18, and moving averages in inference, x_{bn} = \gamma (x - \mu)/\sigma + \beta eq. 19. At inference it is folded into the conv:

W_{inf} = \frac{\gamma W}{\sigma} \quad \text{[eq. 20]} \qquad Bias_{inf} = \beta - \frac{\gamma \mu}{\sigma} \quad \text{[eq. 21]}

Naive folding with batch statistics in training fig. 8 makes the quantized weights change every batch (jitter) and hurts accuracy. Using moving averages throughout training removes the normalization effect and destabilizes training §3.2.2. The fix fig. 9:

c = \frac{\sigma_B}{\sigma} \quad \text{[eq. 22]} \qquad w_{corrected} = c \times \frac{\gamma W}{\sigma_B} \quad \text{[eq. 23]}

so the quantized weights always use the long-term \sigma.

• Phase 1 (before freezing) eq. 24–27: y = \mathrm{conv}(Q(w_{corrected}), x), y_{corrected} = y / c, bias = \beta - \gamma \mu_B / \sigma_B, bias_{corrected} = 0. Outputs match ordinary batch norm.
• Phase 2 (after freeze_bn_delay steps) eq. 28–31: y_{corrected} = y (no division), bias = \beta - \gamma \mu_B / \sigma_B, bias_{correction} = \gamma(\mu_B/\sigma_B - \mu/\sigma). The moving mean and variance are frozen. This matches inference-time normalization.

# Training-time forward pass of a conv + batch norm block with QAT  [§3.2.2]
# Input:  x (activations), W (float conv weights), gamma, beta,
#         moving_mean, moving_std, step, freeze_bn_delay
# Output: y (pre-activation output, before activation fake-quant)

def conv_bn_qat_forward(x, W, gamma, beta, moving_mean, moving_std, step):
    # Batch statistics of the float conv output  [eq. 18]
    batch_mean, batch_std = moments(conv(x, W))
    # Correction factor towards long-term statistics  [eq. 22]
    c = batch_std / moving_std
    # Fold BN into weights, scaled to long-term sigma  [eq. 23]
    w_corrected = c * gamma * W / batch_std
    # Fake-quantize folded weights; STE in backward  [eq. 12-15]
    y = conv(sim_quant(w_corrected), x)
    # Bias from batch statistics  [eq. 26, eq. 30]
    bias = beta - gamma * batch_mean / batch_std
    if step < freeze_bn_delay:
        # Undo scaling so output equals regular batch norm  [eq. 25, eq. 27]
        y = y / c
        bias_correction = 0
    else:
        # Frozen: use long-term statistics, as at inference  [eq. 29, eq. 31]
        freeze(moving_mean, moving_std)
        bias_correction = gamma * (
            batch_mean / batch_std - moving_mean / moving_std
        )
    return y + bias + bias_correction

The paper does not say exactly how bias and bias_{correction} are combined. Adding them is what makes phase 2 equal \beta - \gamma\mu/\sigma (eq. 21); this is our reading of fig. 9, not an explicit statement.

Recommended QAT workflow §3.2

1. Start from a float checkpoint (recommended) or train from scratch.
2. Insert fake-quant ops with the rewriter (tf.contrib.quantize.create_training_graph(quant_delay=...) for training, create_eval_graph() for eval).
3. Train; the saved model then holds scale and zero-point for every weight and activation.
4. Convert with TOCO to a TFLite flatbuffer with integer weights.
5. Run with the TFLite interpreter on CPU, or on accelerators through NN-API.

Hyperparameters

Results

Author-reported. ImageNet top-1; all accuracies use simulated quantization §3.1.3. QAT columns fine-tune TF-Slim float checkpoints with BN freezing §3.2.3. Latency on one large core of a Google Pixel 2 and on Qualcomm DSP (HVX) via NN-API §6.

Networks studied tab. 1

8-bit weight-only PTQ tab. 2

8-bit weight and activation PTQ (activations per-layer) tab. 3

8-bit QAT vs PTQ tab. 4

4-bit weights, 8-bit activations tab. 5

4-bit activations (weights 8-bit symmetric per-channel; "(w,a)" bits; (4,8) column repeated for comparison) tab. 6

Latency, ms (Pixel 2, single large core; HVX via NN-API) tab. 7

Other findings:

• Per-layer weight quantization fails mainly because folded batch norm makes kernel ranges within a layer differ by a lot. Per-channel quantization gives much higher per-kernel SQNR §3.1.3, app. A, fig. 18, fig. 19. Folded weights have much longer tails than unfolded ones fig. 20.
• Almost all PTQ accuracy loss comes from weights; activations alone cost little, thanks to unscaled batch norm (Inception-v3) or ReLU6 (MobileNet-v1) §3.1.3, tab. 3.
• Stochastic quantization in training does worse than deterministic quantization §4, fig. 12. Fine-tuning from a float checkpoint beats training from scratch §4, fig. 13.
• On MobileNet-v1, BN correction plus freezing gives higher and steadier eval accuracy than naive folding, batch renormalization, or moving-average weights without freezing §4, fig. 14.
• Exponential moving averages of weights fall behind instantaneous weights late in QAT §4, fig. 15.
• ReLU instead of ReLU6 gives slightly better accuracy for float and quantized MobileNet-v1 §5, fig. 16.
• On MobileNet-v1_0.25_128, 4-bit per-channel weights at a larger depth multiplier match 8-bit accuracy with about 25% less model size §5, fig. 17.

Paper vs code

No official code pinned. The paper's tool is tf.contrib.quantize in TensorFlow 1.x, cited only via its API docs §3.2. tf.contrib was removed in TensorFlow 2, so the module is absent from the current repository head. It was not cloned at a fixed commit for this card, so no line-level comparison was done.

Pitfalls

• SimQuant sign error. Eq. 13 writes \Delta\,\mathrm{clamp}(0, N-1, \mathrm{round}(x/\Delta) - z). That disagrees with eq. 1 (+z) and eq. 3 (subtract z after clamping). A consistent fake-quant is \Delta\,(\mathrm{clamp}(0, N-1, \mathrm{round}(x/\Delta) + z) - z), i.e. eq. 1–3 composed. The two agree only when z = 0.
• Text contradicts tab. 2. §3.1.3 says asymmetric per-layer gives the best weight-only accuracy. The table shows asymmetric per-channel is best and per-layer drops MobileNets to 0.001–0.004.
• Symmetric vs asymmetric advice. The conclusions recommend symmetric per-channel as the PTQ starting point §8. Yet on MobileNet-v1, symmetric per-channel PTQ gives 0.591 against 0.703 for asymmetric tab. 3. Check asymmetric per-channel too before reaching for QAT.
• Activation quantizer type is unclear. §3.1.3 says activations use "per-layer symmetric quantization". Elsewhere the paper calls only the weight quantizers symmetric or asymmetric, and it never says which quantizer the activations used for the tables.
• tab. 2 and tab. 3 are nearly identical. The per-layer and symmetric per-channel columns match exactly between weight-only and weight+activation PTQ. Treat the small differences as noise.
• Freeze step for MobileNet-v2 is inconsistent. §3.2.2 says about 300000 steps for fig. 15, while §4 and the fig. 15 caption say 400000. The experiment API is create_experimental_training_graph(freeze_bn_delay=...), not the create_training_graph shown in the snippet.
• Bias combination is implicit. Eqs. 24–31 list bias and bias_{correction} separately. Only their sum gives the inference bias of eq. 21.
• Unstated range details. The formula from (x_{min}, x_{max}) to (\Delta, z), the rounding mode, the EMA decay for activation ranges, and QAT step count and batch size are not stated. Relaxing ranges to include zero costs precision for one-sided ranges like (2.1, 3.5) §2.1.
• Restricted weight range. For SIMD, symmetric signed weights are clamped to \pm(N/2-1), dropping the most negative level eq. 10. Match whatever range your target kernel assumes, or quantized inference will differ from training.
• Train/inference graph mismatch. Every fusion done at inference (e.g. add + ReLU) and every rescale for add and concat must be mirrored by fake-quant placement during training §3.2.1.
• EMA weights. QAT pushes float weights toward quantization decision boundaries. Small differences between EMA and live weights can then flip quantized values, so evaluate both §4.
• Small models are fragile. Networks with fewer parameters (MobileNets, Nasnet) lose the most accuracy under quantization §3.1.3, tab. 5, tab. 6.
• 4-bit activations hurt more than 4-bit weights (e.g. Resnet_v1_50 QAT 0.58 vs 0.732) tab. 6. The comparison mixes granularities, as the paper notes §3.2.4.

Lineage

• Builds on: Jacob et al. 2017, "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (arXiv:1712.05877, no card yet). This paper adds per-channel weights, batch-norm correction and freezing, and the STE backward model §3.2, §9.
• Straight-through estimator from Courbariaux et al. (BinaryConnect, arXiv:1511.00363) §2.4.
• Compared against: batch renormalization and naive BN folding for QAT fig. 14. TensorRT KL calibration is mentioned but not evaluated §2.5.