GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

In plain words

Large language models store billions of numbers, called weights. At 16 bits each, the largest models need several expensive GPUs just to hold them. Quantization stores each weight with fewer bits by rounding it to a small set of allowed values. The obvious method rounds every weight to its nearest allowed value. That works at 8 bits but falls apart at 3 bits, because the rounding errors add up.

GPTQ rounds the weights one column at a time. After each column is rounded, it adjusts the columns that are not rounded yet, so they make up for the error just introduced. To decide how to adjust them, it uses how sensitive the layer's output is to each input, measured on a small set of text samples. The work is done in blocks, and with a numerically safer method, so the largest models can be compressed in a few hours on one GPU. The compressed model needs less memory and runs faster for text generation, and at 3 or 4 bits it stays close to the original accuracy.

TL;DR

• One-shot, post-training weight quantization for GPT-style models to 3 or 4 bits per weight, plus 2-bit and ternary grouped variants §1 §5.
• Calibrated on 128 random 2048-token C4 segments; no retraining and no task data §5 Setup.
• Compresses OPT-175B and BLOOM-176B in about four GPU hours on one A100 §1 tab. 2.
• Author-reported: at 4 bits GPTQ stays within 0.03 WikiText2 perplexity of FP16 on OPT-175B, where round-to-nearest loses 2.2 points §5 Language Generation; 3-bit OPT-175B runs on one 80 GB A100 and about 3.25x faster than FP16 on 5 GPUs §5 Practical Speedups.

Problem

Round-to-nearest (RTN) quantization is cheap and holds up at 8 bits for models of this scale, but it fails at 3 bits: RTN collapses at 3 bits for OPT and BLOOM models [§5 Language Generation] tab. 3 tab. 4. The more accurate post-training methods (AdaRound, BRECQ, AdaQuant, OBQ) are too slow for models with many billions of parameters §2 §5 Baselines. The paper gives OBQ a runtime of O(d_row · d_col³) per matrix, which is why it does not scale §3. ZeroQuant-LKD's reported 3 GPU hours for a 1.3B model would extrapolate to several hundred hours for a 175B model §5 Runtime. Without a method of this kind, 175B-scale models need multi-GPU inference §1.

Method

GPTQ quantizes one linear layer at a time. For each layer it minimises the squared error between the full-precision layer output and the quantized layer output on a small calibration set §3:

\text{argmin}_{\widehat{W}} \, \|W X - \widehat{W} X\|_2^2 eq. 1

The quantization grid is fixed before the process. Individual weights are free to move to any value on that grid §3. The Hessian of this objective is H = 2 X Xᵀ, and it is the same for every row of W, because it depends only on the inputs §3 §4 Step 1.

1. Start from OBQ. Optimal Brain Quantization (OBQ) quantizes one weight at a time, always picking the weight with the least added error, and updates all remaining weights to compensate §3. Its greedy update for the next weight q and the compensation of the remaining full-precision weights F is eq. 2:

   w_q = \text{argmin}_{w_q} \frac{(\text{quant}(w_q) - w_q)^2}{[\mathbf{H}_F^{-1}]_{qq}}, \quad \boldsymbol{\delta}_F = -\frac{w_q - \text{quant}(w_q)}{[\mathbf{H}_F^{-1}]_{qq}} \cdot (\mathbf{H}_F^{-1})_{:, q} eq. 2

   After a weight is removed, the inverse Hessian is updated with one Gaussian elimination step eq. 3. OBQ treats each row separately, which costs O(d_row · d_col³) §3.

   \mathbf{H}_{-q}^{-1} = \Big(\mathbf{H}^{-1} - \frac{1}{[\mathbf{H}^{-1}]_{qq}} \mathbf{H}^{-1}_{:, q} \mathbf{H}^{-1}_{q, :} \Big)_{-p} eq. 3

2. Same order for all rows (Step 1). GPTQ quantizes the columns in one fixed order for all rows. Then H_F⁻¹ is the same for all rows and needs updating only once per column, not once per weight. The cost falls to O(max{d_row · d_col², d_col³}) §4 Step 1. The paper reports that a fixed order usually costs little accuracy, compared with the greedy OBQ order, on large layers §4 Step 1.

3. Lazy block updates (Step 2). Columns are processed in blocks of B = 128 [§4 Step 2]. Errors stay inside the block, and the rest of the matrix is updated once per block with the multi-weight forms eq. 4 and eq. 5. Here Q is the set of indices in the block:

   \boldsymbol{\delta}_F = -(\mathbf{w}_Q - \text{quant}(\mathbf{w}_Q))([\mathbf{H}_F^{-1}]_{QQ})^{-1} (\mathbf{H}_F^{-1})_{:, Q} eq. 4

   \mathbf{H}_{-Q}^{-1} = \Big(\mathbf{H}^{-1} - \mathbf{H}^{-1}_{:, Q} ([\mathbf{H}^{-1}]_{QQ})^{-1} \mathbf{H}^{-1}_{Q, :} \Big)_{-Q} eq. 5

4. Dampening and Cholesky (Step 3). Before the loop, add λ = 1% of the mean diagonal of H to its diagonal §4 Step 3 §5 Setup. Then invert H through a Cholesky factorisation, and use the upper Cholesky factor of H⁻¹ as the only source of the rows the loop needs §4 Step 3. The paper states that repeated inverse updates accumulate numerical error, and that without this step H⁻¹ can become indefinite for models of a few billion parameters or more §4 Step 3.

5. Grouping (optional). Quantize groups of g consecutive weights in a row with their own scale and zero point. The group parameters are computed during the quantization, from the already-updated weights §5 175 Billion Parameter Models.

6. Sequential layers. Calibration is run layer by layer. The inputs for block i come from the already-quantized blocks before it §5 Setup. The paper's implementation loads one Transformer block, consisting of 6 layers, at a time [§5 Setup]. In the OPT code, the six linear layers of a decoder block are the ones quantized together opt.py. Embeddings and the output layer stay in FP16 §5 Practical Speedups.

The full procedure is Algorithm 1 alg. 1. It quantizes W given the inverse-Hessian factor and blocksize B. The line order below is the one in the paper.

# GPTQ for one weight matrix  [alg. 1] [§4]
# Input:  W (d_row, d_col) weights; H (d_col, d_col) = 2 X X^T over calibration
#         inputs; blocksize B (128); percdamp lambda (0.01)
# Output: Q (d_row, d_col) weights on the quantization grid

def gptq_quantize(W, H, blocksize, percdamp):
    # Dampen H by a fraction of its mean diagonal  [§4 Step 3] [alg. 1]
    damp = percdamp * mean(diagonal(H))
    H = H + damp * identity(d_col)
    # Upper Cholesky factor of H^-1: the only part of H^-1 used  [§4 Step 3]
    H_inv_chol = cholesky_upper(inverse(H))
    # Fix the grid (scale and zero per row) from the original W  [§5 Setup]
    grid = fit_row_grid(W)
    Q = zeros_like(W)
    for start in range(0, d_col, blocksize):  # Step 2  [§4 Step 2] [alg. 1]
        end = min(start + blocksize, d_col)
        # Scaled errors of this block, used for the update after it  [alg. 1]
        block_err = zeros(d_row, end - start)
        for j in range(start, end):  # one column at a time  [alg. 1]
            # Round column j to the grid  [alg. 1]
            Q[:, j] = quantize(W[:, j], grid)
            # Error of column j, scaled by the Cholesky diagonal  [alg. 1]
            err = (W[:, j] - Q[:, j]) / H_inv_chol[j, j]
            block_err[:, j - start] = err
            # Compensate the rest of the block  [alg. 1]
            W[:, j:end] -= outer(err, H_inv_chol[j, j:end])
        # Compensate all columns after the block at once  [alg. 1]
        W[:, end:] -= block_err @ H_inv_chol[start:end, end:]
    return Q

Quantization is uniform and asymmetric, per row, on a min-max grid §5 Setup [code: quant.py#L36-L76]. The exact grid is given under Pitfalls. With groups, fit_row_grid is called again at each group boundary on the current, already-updated W [code: gptq.py#L120-L123].

Hyperparameters

Results

Author-reported. Setup: OPT (125M to 175B) and BLOOM (560M to 176B) models, perplexity on WikiText2, PTB and C4, zero-shot tasks LAMBADA, ARC and PIQA §5. Baseline: FP16 and round-to-nearest (RTN) on the same grid §5 Baselines.

Quantization runtime on one A100 tab. 2:

Small-model comparison on ResNet18 and ResNet50, accuracy in % tab. 1. The full-precision accuracies (69.76 and 76.13) are from the table header:

WikiText2 perplexity, OPT family tab. 3 (lower is better):

WikiText2 perplexity, BLOOM family tab. 4:

Largest models, all three perplexity tasks and LAMBADA accuracy tab. 5:

Per-token latency, batch size 1, generating 128 tokens, 3-bit GPTQ vs FP16 tab. 6:

2-bit grouped results, OPT-175B and BLOOM-176B, WikiText2 perplexity tab. 7:

Ternary (-1, 0, +1) with group size 8 reaches 9.20 WikiText2 perplexity on OPT-175B, less than one point above FP16 §5 Extreme Quantization.

Other tables (PTB and C4 perplexity for each family tab. 9 to tab. 12; zero-shot accuracy on LAMBADA, PIQA, ARC and StoryCloze tab. 13 to tab. 22; the OBQ comparison on BERT-base and OPT-125M tab. 8) are in the appendix and are not reproduced here app. A.1 app. A.3 app. A.4.

Paper vs code

Official code: IST-DASLab/gptq, commit 2d65066eeb06a5c9ff5184d8cebdf33662c67faf, linked from the paper's abstract. Checked files: gptq.py, quant.py, opt.py, datautils.py, README.md.

• Hessian scale. The paper uses H = 2 X Xᵀ §3. The code keeps a running mean of (2/N) · X Xᵀ over calibration batches gptq.py. The dampening is relative to the mean diagonal, and the update divides by the inverse-Hessian diagonal, so a uniform rescaling of H cancels in the update rule. This is derived from the update formulas, not tested.
• Dead columns. Code sets H to 1 on zero-diagonal columns and zeros their weights gptq.py. The paper does not mention this case.
• Act-order. The code can process columns in decreasing order of diag(H) gptq.py, behind --act-order opt.py, which is off by default. The paper's Step 1 says a fixed order works, but does not describe this ordering or report it. The README reports improvements from it on OPT-66B and LLaMA README.md.
• Calibration data. The paper says "random 2048 token segments from the C4 dataset" §5 Setup. The code uses the C4 training shard 00000-of-01024 and samples documents of at least 2048 tokens datautils.py.
• Block order. The paper loads "one Transformer block, consisting of 6 layers" at a time §5 Setup. For OPT, the code collects Hessians for all six linear layers of a decoder layer in one forward pass, quantizes them all, then reruns the layer on the quantized weights opt.py. Quantizing the six layers one after another (--true-sequential) appears only in the LLaMA integration README.md.
• Grid. The paper says min-max §5 Setup. The code also forces 0 into the range quant.py and rounds the zero point to an integer quant.py.
• Speedups. The text gives 3.25x (A100) and 4.5x (A6000) §5 Practical Speedups. tab. 6 gives 3.24x and 4.53x.
• Evaluation preprocessing. The paper does not say which PTB and C4 preprocessing it uses. The code has --new-eval for updated preprocessing, which the README says is used in its updated results opt.py README.md.
• Ternary. The code's ternary path sets maxq = -1 and quantizes to a sign-based three-level grid with thresholds at half the scale and zero point quant.py quant.py quant.py. The paper describes ternary (-1, 0, +1) at group size 8 but not this mapping.

Pitfalls

• Equation 3 has a typo. The subscript on the right side says -p, but the removed index is q eq. 3. Use -q.
• Cholesky is required, not optional. Repeated inverse updates can make H⁻¹ indefinite, and the paper reports this as almost certain for some layers above a few billion parameters §4 Step 3. Dampening alone is not enough at that scale. Compute the Cholesky factor in float32; the code disables TF32 and casts to float gptq.py gptq.py.
• Grid details matter. Forcing 0 into the min-max range and rounding the zero point change the grid, and the paper does not say so quant.py quant.py. A reimplementation based only on the paper will get a different grid.
• Grouped scales use updated weights. With groups, scale and zero point are fitted when a group starts, on the current, already-updated weights gptq.py. The paper says this too §5 175 Billion. Fitting them once on the original weights gives a different result.
• Memory. Only one block is held on the GPU at a time §5 Setup. The calibration inputs and outputs are still stored in full, one copy each, at nsamples × seqlen × hidden size in the model's dtype opt.py opt.py.
• Memory of the 3-bit 175B model. The paper gives about 63 GB for the quantized weights, plus about 9 GB of KV cache at 2048 tokens. It fits on one 80 GB A100 only with dequantization done layer by layer. The 4-bit model does not fit [§5 Practical Speedups].
• Speed only from memory traffic. The kernels speed up batch-1 generation, which is memory-bound. They do not reduce multiplication compute, and activations are not quantized §6. The FP16 baseline in tab. 6 uses several GPUs, so the speedup includes a multi-GPU reduction.
• Not zero-shot on C4. The calibration data is drawn from C4's training set, so the C4 numbers are not fully zero-shot tab. 11 caption tab. 12 caption.
• Outlier model. OPT-66B has many dead units in early layers, which the paper links to its worse results §5 Language Generation footnote.
• Grouping overhead. Group size 128 costs about 0.15 extra bits per weight, and group size 1024 about 0.02 §5 175 Billion. Count these bits when comparing sizes.
• 2.2-bit claim. The paper reports 2.2 bits at group size 128 with an FP16 scale and a 2-bit zero point per group §5 Extreme Quantization. Check the storage format before using this figure.

Lineage

• Builds on: Optimal Brain Quantization (OBQ), which the paper extends with a fixed order, lazy block updates and a Cholesky reformulation §3 §4. The paper does not give an arXiv id for OBQ in its text.
• Compared against: round-to-nearest as used by ZeroQuant and LLM.int8() §5 Baselines; AdaRound, BRECQ, AdaQuant and OBQ on vision models tab. 1; ZeroQuant-LKD runtime §5 Runtime.
• Related: nuQmm, for its grouping scheme §5 Additional Tricks.