Asymmetric quantization

Overview

Asymmetric quantization maps each real value to a low-bit integer code on a grid that does not have to be centred on zero. A symmetric grid is fixed around zero, so a range that is mostly positive, or that dips only slightly below zero, wastes levels. An asymmetric grid adds an offset β next to the scale s, so its lowest level can sit at the most negative value that occurs. The shared invariant is that a quantized value is an integer code times s, plus β 2004.09576 §3.1 eq. 2. GPTQ uses the same kind of grid with a per-row minimum and maximum, and it does not learn the offset 2210.17323 §5 Setup.

Two papers are on this page so far, and they apply the idea to different targets:

• LSQ+ quantizes activations of ImageNet CNNs during quantization-aware training. It learns s and β by gradient descent with a straight-through rounding estimator 2004.09576 §3.1 eq. 3 eq. 4. The motivating case is Swish-like activations, whose negative part is bounded near −0.278 while the positive part is unbounded 2004.09576 §3.1. In that paper the weights stay symmetric and signed 2004.09576 §3.2.1.
• GPTQ quantizes the weights of GPT-style language models after training, with no retraining, using a calibration set of 128 text segments 2210.17323 §5 Setup. Its grid is uniform and asymmetric per row, set by min-max 2210.17323 §5 Setup.

Variants

Each variant is a choice of grid, of how its parameters start, or of what the grid is applied to. Activation results come from LSQ+ on EfficientNet-B0, MixNet-S and ResNet18 at W2A2, W3A3 and W4A4 2004.09576 tab. 2, tab. 3, tab. 4. Weight results come from GPTQ on OPT and BLOOM language models 2210.17323 tab. 3, tab. 4, tab. 5.

Symmetric signed grid (baseline)

No offset, with a sign bit: the grid is centred on zero and the range is −2^(b−1) to 2^(b−1) − 1 2004.09576 §3.1 tab. 1. This is Config 2 in the paper. It is the weakest of the four configurations on EfficientNet-B0, at 68.8% for W4A4 and 23.7% for W2A2, against 76.1% full precision 2004.09576 tab. 2.

Unsigned symmetric grid (LSQ)

No offset and no sign bit, so the grid covers only zero and positive values 2004.09576 §3.1 tab. 1. Negative activations are clamped to zero. This is Config 1, the LSQ reference, at 71.9% for W4A4 and 43.5% for W2A2 on EfficientNet-B0 2004.09576 tab. 2. Its loss against full precision at W4A4 is 4.2 points 2004.09576 tab. 2; see Contradictions.

Signed asymmetric grid

A learned offset with a sign bit. This is Config 3 in the paper. On EfficientNet-B0 it reaches 73.5% at W4A4, 69.9% at W3A3 and 49.1% at W2A2 2004.09576 tab. 2. On MixNet-S it reaches 71.6% at W4A4 and 42.5% at W2A2 2004.09576 tab. 3.

Unsigned asymmetric grid (LSQ+)

A learned offset and no sign bit. This is Config 4, and it is the configuration the paper uses for its analyses 2004.09576 §4.3 tab. 5 tab. 6. On EfficientNet-B0 it reaches 73.8% at W4A4, 69.3% at W3A3 and 48.7% at W2A2 2004.09576 tab. 2. Configs 3 and 4 are nearly equal in tables 2 and 3, within about 0.3 points at W4A4 on EfficientNet-B0 and MixNet-S 2004.09576 tab. 2, tab. 3. The weights in the same experiments are still symmetric and signed 2004.09576 §4.3.

Fixed offset at the observed minimum

The offset is not learned. It is set to the lowest observed value, x_min, and the scale is learned. On EfficientNet-B0 at W4A4 with Config 4 this gives 72.5%, between the symmetric LSQ baseline at 71.9% and the learned offset at 73.8% 2004.09576 tab. 6. This is the only fixed-offset row the card records, and it is for one network and one bit-width.

Initialisation of s and β

Min-max initialisation sends the observed minimum to the lowest level and the observed maximum to the highest 2004.09576 §3.2.2 eq. 6 eq. 7. The paper's proposed MSE initialisation chooses s and β to minimise the squared error between quantized and original activations, found by gradient steps over a few batches 2004.09576 §3.2.2 eq. 8. On EfficientNet-B0 with Config 4, the mean over 5 runs is 73.0% ± 0.9 at W4A4 and 46.8% ± 1.9 at W2A2 for the proposed initialisation. Min-max gives 71.3% ± 2.2 and 43.8% ± 4.7, and LSQ-style initialisation gives 72.0% ± 1.6 and 44.4% ± 2.9 2004.09576 tab. 5. The spread is the main reported effect, and it is largest for min-max.

Per-row min-max grid for weights (GPTQ)

Each row of a weight matrix gets its own grid, fitted to that row's minimum and maximum, and the grid is fixed before rounding starts 2210.17323 §5 Setup. Weights are the only quantized tensor, and activations are not quantized 2210.17323 §6. The grid is asymmetric, so it can cover a row whose values are not centred on zero. GPTQ's main contribution is not this grid. Round-to-nearest on the same grid is the baseline in the paper 2210.17323 §5 Baselines, so the gap between the two comes from how each weight is rounded, not from the grid.

Main results, OPT-175B, WikiText2 perplexity (lower is better), FP16 8.34 2210.17323 tab. 5:

• 4-bit, round-to-nearest 10.54 against GPTQ 8.37 2210.17323 tab. 5.
• 3-bit, round-to-nearest 7.3e3 against GPTQ 8.68 2210.17323 tab. 5. Round-to-nearest collapses at 3 bits on this grid.
• 3-bit with group size 128 gives 8.45, and with group size 1024 gives 8.45 2210.17323 tab. 5. Grouping adds about 0.15 bits per weight at g128 and about 0.02 at g1024 2210.17323 §5 175 Billion Parameter Models.
• 2-bit with group size 32 gives 8.94, and a ternary grid with group size 8 gives 9.20 2210.17323 tab. 7 2210.17323 §5 Extreme Quantization.

Quantizing OPT-175B takes about four hours on one A100 80 GB 2210.17323 tab. 2.

When to use which

The two papers answer different questions, and the constraints below are limited to the settings they tested.

LSQ+ tests learned offsets during fine-tuning from a pre-trained float model. Every result comes from that setting, so the claims below are limited to it 2004.09576 §4. Post-training use with no fine-tuning is not among the recorded results. GPTQ tests post-training weight quantization with no retraining, on language models with 125M to 176B parameters 2210.17323 §5 Setup.

Implementation sketch

Pseudocode in our own words. The quantizer follows eq. 2, and the initialisers follow §3.2.2. The offset fold is the part where variants most often break, so check eq. 5 before implementing it.

# Asymmetric activation quantizer  [§3.1] [eq. 2]
# Input:  x (activations); scale s > 0; offset beta; integer range n, p
# Output: x_hat, same shape as x
def quantize_activation(x, s, beta, n, p):
    # Shift by the offset, then divide by the scale  [eq. 2]
    scaled = (x - beta) / s
    # Clamp to the integer range before rounding  [eq. 2]
    codes = round(clamp(scaled, n, p))
    # Map codes back to real values  [eq. 2]
    return codes * s + beta

# Min-max initialisation of s and beta  [§3.2.2] [eq. 7]
# Input:  observed x_min, x_max; integer range n, p
# Output: s_init, beta_init
def init_minmax(x_min, x_max, n, p):
    # Stretch the observed range over the integer levels  [eq. 7]
    s_init = (x_max - x_min) / (p - n)
    # Place x_min at the lowest level n  [eq. 7]
    beta_init = x_min - n * s_init
    return s_init, beta_init

# Offset fold for a linear layer with symmetric weights  [§3.1] [eq. 5]
# Input:  weight codes, weight scale s_w, activation offset beta, bias
# Output: bias to use with integer-only matmul on the codes
def fold_offset(w_codes, s_w, beta, bias):
    # Expanding the product gives y = s * s_w * (w_codes . x_codes) + beta * s_w * sum(w_codes)
    # The second term is per output channel and depends only on the weights  [eq. 5]
    # Adding it to the bias removes it from the runtime  [eq. 5]
    correction = beta * s_w * sum(w_codes, axis=1)
    return bias + correction

Contradictions

No two papers on this page report conflicting results. LSQ+ measures activations of CNNs during fine-tuning on ImageNet, and GPTQ measures weights of language models after training on perplexity, so their numbers are not comparable.

Inside the LSQ+ paper, several reported numbers do not match each other. Each is set out in the card's Pitfalls, and the page does not resolve them:

• Gain over LSQ at W4A4. The text gives 1.6 to 1.8 points 2004.09576 §4.1. Table 2 gives 1.6 for Config 3 and 1.9 for Config 4 2004.09576 tab. 2.
• LSQ loss at W4A4 on EfficientNet-B0. Section 1 gives 4.1 points 2004.09576 §1. Table 2 gives 4.2 2004.09576 tab. 2.
• Comparability of numbers. Tables 2 to 4 report the best accuracy after a grid search, while table 5 reports a mean over 5 runs 2004.09576 tab. 5 caption. The 73.8% in table 2 and the 73.0% in table 5 are not comparable.
• Fixed β = 0 baseline. The row in table 6 is 71.9%, identical to table 2's Config 1. The paper does not say whether it was a separate run 2004.09576 tab. 6.

Between the GPTQ paper and its code, the grid differs. The paper says min-max 2210.17323 §5 Setup, and the official code also forces zero into the range and rounds the zero point. The paper does not state either step. A grid written only from the paper will differ from the released code. The details are in the GPTQ card, under Paper vs code.

Papers

• LSQ+: Improving low-bit quantization through learnable offsets and better initialization: learned offset next to the learned scale for activations, and MSE-based initialisation that narrows run-to-run spread.
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: per-row asymmetric min-max grid for weights, with error compensation that makes 3-bit and 4-bit weights usable on large language models without retraining.