LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

In plain words

Large language models hold most of their memory in large grids of numbers, and each answer requires multiplying those grids. This paper stores most of those numbers as 8-bit integers instead of 16-bit numbers, and does the multiplications with the integers, so the model needs much less memory without giving worse answers. The difficulty is that a few positions in the model's internal vectors carry very large values. If these were squeezed into 8 bits along with everything else, the small values would be lost. So the method handles those few positions separately at full 16-bit precision, and runs everything else through the 8-bit path. Each row and column also gets its own scale factor, so one large value does not spoil the precision of its neighbours. The authors say the result lets very large models run on a single server with consumer GPUs.

TL;DR

• Replaces the 16-bit matrix products of feed-forward and attention projection layers with 8-bit integer products, using the trained model as is, with no retraining §1 app. A.
• Two parts: vector-wise absmax quantization, one scale per row of the activations and per column of the weights §3.1 eq. 7; and a mixed-precision decomposition that multiplies the feature dimensions with magnitude at least α = 6.0 in 16-bit §3.2 eq. 8.
• Author-reported: C4 validation perplexity equals the 32-bit baseline from 125M to 13B parameters tab. 1; OPT zero-shot accuracy stays at 16-bit level up to 175B fig. 1 §3.4. Weight memory is about half of 16-bit §3.2.
• Official code (bitsandbytes) defaults the outlier threshold to 0.0, which turns the 16-bit path off, and reconstructs the outlier weights from int8 rather than storing 16-bit weights. See Paper vs code.

Problem

• Inference of large transformers is limited by GPU memory. In models of 6.7B parameters and up, the feed-forward and attention projection layers make up 95% of the memory-consuming parameters and carry 65–85% of the compute; most of the rest is embeddings §1.
• Earlier 8-bit methods for transformers degraded performance, often needed tuning after training, and had only been studied below 350M parameters §1.
• Using one scale per whole tensor lets a single outlier reduce precision for every other value. Row-wise scaling helps but still fails on outliers §3 §3.1.
• Starting at about 6.7B parameters, large-magnitude feature dimensions (up to about 20× other dimensions) appear in all transformer layers, concentrated in a handful of feature dimensions §1 §4.2 app. C. Absmax and zeropoint quantization then lose most precision, and the paper ties this to the failure point §4.2 §4.3 §3.4.
• Scope assumptions: pretrained models, inference only. Training and fine-tuning are studied only in appendices and are not solved §6 app. E app. F.

Method

The method pairs per-row and per-column absmax scaling with a separate high-precision path for outlier features §3. For a linear layer with 16-bit activations X (s × h, s sequence positions) and 16-bit weights W (h × o), the steps are:

Step 1: absmax quantization of a tensor §2.1. Multiply by 127 divided by the largest absolute value, round to the nearest integer, and store as int8 in [-127, 127]. The paper writes this as one display equation (unnumbered).

\mathbf{X}_{i8} = \operatorname{round}\!\Big(\frac{127\,\mathbf{X}_{f16}}{\max_{ij}|\mathbf{X}_{f16,ij}|}\Big) = \operatorname{round}\big(s_{x}\,\mathbf{X}_{f16}\big), \quad s_x = \frac{127}{\lVert \mathbf{X}_{f16} \rVert_\infty} \qquad [§2.1]

Step 2: vector-wise quantization §3.1 eq. 7. Each row of X (one sequence position) gets its own constant c_x. Each column of W (one output unit) gets its own constant c_w. The integer product is then dequantized by the outer product of the two constant vectors. This is the same as dividing each inner product by its own pair of constants.

Step 3: mixed-precision decomposition §3.2 eq. 8. Let O be the set of feature dimensions h for which some value of X, over all sequence positions, has magnitude above the threshold α. The paper uses α = 6.0 and says this is sufficient to reduce degradation close to zero §3.2. The dimensions in O are multiplied in 16-bit with the matching rows of W. All other dimensions are quantized as in Steps 1–2 and multiplied in int8. The two results are summed in 16-bit fig. 2.

Step 4: outlier feature analysis §4.1. The analysis that motivates Step 3 uses three criteria: a magnitude of at least 6.0, present in at least 25% of layers, and present in at least 6% of sequence positions. These are analysis thresholds. The paper does not present them as inputs to the algorithm.

Equations from the Background section, which describe zeropoint quantization §2.1. The paper uses zeropoint only as a baseline; LLM.int8() itself uses absmax. Equation numbers follow the compiled PDF.

nd_{x_{f16}} = \frac{2 \cdot 127}{\max_{ij}(\mathbf{X}_{f16}^{ij}) - \min_{ij}(\mathbf{X}_{f16}^{ij})} \qquad [eq. 1]

zp_{x_{i16}} = \operatorname{round}\big(\mathbf{X}_{f16} \cdot \min_{ij}(\mathbf{X}_{f16}^{ij})\big) \qquad [eq. 2]

As printed, eq. 2 multiplies X by the minimum, which does not give a zero point. See Pitfalls.

\mathbf{X}_{i8} = \operatorname{round}\big(nd_{x_{f16}}\,\mathbf{X}_{f16}\big) \qquad [eq. 3]

C_{i32} = \mathrm{multiply}_{i16}\big(A_{zp_{a_{i16}}}, B_{zp_{b_{i16}}}\big) = \big(A_{i8} + zp_{a_{i16}}\big)\big(B_{i8} + zp_{b_{i16}}\big) \qquad [eq. 4]

C_{i32} = A_{i8}B_{i8} + A_{i8}\,zp_{b_{i16}} + B_{i8}\,zp_{a_{i16}} + zp_{a_{i16}}\,zp_{b_{i16}} \qquad [eq. 5]

Equation 5 is the expansion that is needed when the instruction in eq. 4 is not available, for example on GPUs or TPUs §2.1.

Equation 6 is the generic 8-bit matrix product with 16-bit inputs and outputs, where Q(·) is either absmax or zeropoint quantization §2.1:

\mathbf{X}_{f16}\mathbf{W}_{f16} = \mathbf{C}_{f16} \approx \frac{1}{c_{x_{f16}}c_{w_{f16}}}\mathbf{C}_{i32} = S_{f16}\cdot\mathbf{C}_{i32} \approx S_{f16}\cdot Q(\mathbf{A}_{f16})\,Q(\mathbf{B}_{f16}) \qquad [eq. 6]

Equation 7 is the vector-wise version, with c_x ∈ ℝ^s and c_w ∈ ℝ^o §3.1:

\mathbf{C}_{f16} \approx \frac{1}{\mathbf{c}_{x_{f16}} \otimes \mathbf{c}_{w_{f16}}}\,\mathbf{C}_{i32} = \mathbf{S}\cdot\mathbf{A}_{i8}\mathbf{B}_{i8} = \mathbf{S}\cdot Q(\mathbf{A}_{f16})\,Q(\mathbf{B}_{f16}) \qquad [eq. 7]

Equation 8 is the mixed-precision decomposition, written with Einstein notation where all indices are superscripts. Here S_f16 is the denormalization for the int8 inputs and weights §3.2:

\mathbf{C}_{f16} \approx \sum_{h \in O} \mathbf{X}^{h}_{f16}\,\mathbf{W}^{h}_{f16} + \mathbf{S}_{f16} \cdot \sum_{h \notin O} \mathbf{X}^{h}_{i8}\,\mathbf{W}^{h}_{i8} \qquad [eq. 8]

The following pseudocode combines Steps 2 and 3 for one linear layer. The threshold, the row and column scales and the outlier set follow §3.1 and §3.2. Where the code makes a choice the paper does not state, the comment says so and cites the code.

# LLM.int8() linear layer  [§3.1] [§3.2] [eq. 7] [eq. 8]
# Input:  X (s, h) float16 activations; W (h, o) float16 weights
# Output: C (s, o) float16

def llm_int8_linear(X, W, alpha=6.0):
    # Feature dims with any value of magnitude >= alpha  [§3.2]
    outlier_dims = dims_with_magnitude_at_least(X, alpha)
    # Remaining dims go through the int8 path  [§3.2]
    regular_dims = complement(outlier_dims)
    # One scale per row of X; outliers left out of the max (code only)
    X_regular = X[:, regular_dims]
    X_scale = 127 / absmax_per_row(X_regular)       # (s, 1)
    # One scale per output column of W  [§3.1]
    W_regular = W[regular_dims, :]
    W_scale = 127 / absmax_per_column(W_regular)    # (1, o)
    # Round to the int8 range  [§2.1]
    X_int8 = round(X_regular * X_scale)             # (s, h - |O|)
    W_int8 = round(W_regular * W_scale)             # (h - |O|, o)
    # Integer product, dequantized by the outer product of scales  [eq. 7]
    C_int8 = matmul(X_int8, W_int8) / outer(X_scale, W_scale)
    # Outlier dims: 16-bit product with 16-bit weights  [eq. 8]
    C_outlier = matmul(X[:, outlier_dims], W[outlier_dims, :])
    # Sum both parts into the 16-bit output  [fig. 2]
    return C_int8 + C_outlier

Hyperparameters

Results

Author-reported. The main setup is C4 validation perplexity for dense fairseq models from 125M to 13B parameters, with each quantization method compared against the 32-bit baseline §3.3 tab. 1. Lower is better. Values are copied from the paper.

Table 1 — C4 validation perplexity tab. 1

The caption and text say absmax, row-wise, zeropoint and vector-wise quantization all degrade as scale grows, and that LLM.int8() recovers full perplexity §3.4 tab. 1. The 13B row shows Int8 absmax (19.08) worse than the 6.7B row of the same method (14.59) tab. 1.

Zero-shot accuracy. Mean zero-shot accuracy of OPT models on WinoGrande, HellaSwag, PIQA and LAMBADA, from 125M to 175B. The 16-bit baseline, the best previous 8-bit method, and LLM.int8() are plotted. Once outliers appear at 6.7B, the baseline absmax vector-wise method falls to random performance, while LLM.int8() stays at 16-bit level fig. 1 §3.4. This figure is the only evidence in the main text for models above 13B; the 175B claim is made in the text §1 §3.4 and shown as a curve, not a table.

Memory on given hardware. Largest model that can run per hardware class, with 8-bit or 16-bit weights tab. 2:

Table 2 is in Section 6 tab. 2. It is repeated in Appendix A with the same label app. A.

Matrix-multiplication speed. Speedup over 16-bit for the first feed-forward layer of GPT-3-sized models, ratios above 1.0 are faster app. D.1. The paper states that quantization and decomposition overhead dominates for small models, and that the custom CUDA kernels are much faster than the default PyTorch and NVIDIA kernels app. D.1 §3.4:

End-to-end BLOOM-176B. Per-token generation time in milliseconds, on 8× A100 80GB unless noted app. D.2. Lower is better:

Outlier statistics (6.7B and 13B rows). Summary of the analysis in app. C for two rows. The full table has all models from 117M to 13B. Quartiles are of the most common outlier.

Source: app. C §4.2. "FSEQ" is the fairseq model family; the 6.7B and 13B quartiles are (-44, -40, -35) and (-63, -58, -45) app. C.

Outlier removal. Outlier dimensions removed (set to zero) lower the mean top-1 softmax probability from about 40% to about 20% and raise validation perplexity by 600–1000%, while removing seven random dimensions changes perplexity by about 0.1% §4.2.

Fine-tuning, GLUE, RoBERTa-large, 8-bit feed-forward layers app. F. Mean over GLUE, median of 5 seeds: 32-bit baseline 88.61, 32-bit replication 88.83, Q-BERT 86.91, Q8BERT 86.75, PSQ 88.65, vector-wise 88.81 app. F. Breakdown: 8-bit FFN with 0% decomposition 88.8; FFN and attention linear layers in 8-bit with 0% decomposition 87.7, with 2% decomposition 88.7, with 3% 88.7 app. F. The paper states that at least the top 2% magnitude dimensions must be 16-bit to avoid degradation when attention projections are 8-bit app. F.

Training, appendix E. Language model, 209M parameters, perplexity (lower is better): 16-bit baseline 16.74; 8-bit FFN 16.77; 8-bit FFN and linear 16.83 at 0% decomposition; 8-bit FFN, linear and attention with 2%, 5%, 10% decomposition: 24.33, 20.00, 19.00 app. E. At 1.1B: baseline 9.99; FFN 8-bit 9.93; FFN and linear 8-bit 10.52 (0%), 10.41 (1%) app. E. Machine translation (WMT14+16, median of 3 seeds, BLEU): 32-bit 28.9; 8-bit FFN 28.8; 8-bit FFN and linear unstable at 0%; 28.0 at 2%, 27.6 at 5%, 27.5 at 10% decomposition app. E. The paper concludes that 8-bit FFN training is straightforward, but other layers need more work app. E.

Paper vs code

The paper's footnote links https://github.com/TimDettmers/bitsandbytes §1. Its owner is the first author, so it qualifies as the official code. The code is checked out at commit 833649043474794b8fe7a4136e0c40faf077b2e0, which is the head of the default branch at compile time (dated 2026-09-03). That is about four years after the paper's arXiv version, and the layout has changed (for example, kernels now live under bitsandbytes/backends/). Numbers in the paper cannot be re-run from this commit without checking out an earlier version. I did not find the perplexity or outlier-analysis scripts in the repository: a search of *.py for c4 and perplexity only found a variable name in the test suite. Table 1 and Figures 3–4 therefore rest on the paper alone.

Differences between paper and code at this commit:

• Threshold default. The paper sets α = 6.0 §3.2. The code's Linear8bitLt defaults threshold to 0.0 modules.py, and 0.0 disables the 16-bit path: all columns go through int8 _functions.py and the docstring says the same modules.py. Callers who want the paper's method must pass 6.0.
• Outlier weights. Eq. 8 uses 16-bit weights W_f16 for outlier dimensions eq. 8. The default-backend kernel instead dequantizes the int8 weight columns for outlier dimensions ops.py. The int8 weights come from int8_vectorwise_quant(B) _functions.py. So the outlier path multiplies int8-rounded weights in 16-bit, not the original 16-bit weights. The paper does not say which path produced Table 1.
• Outlier test. The paper defines outliers as values with magnitude "larger than" α §3.2. The code uses >= ops.py and ops.py.
• Row scale excludes outliers. The code zeroes outlier entries before taking the per-row absolute maximum, so outliers do not set the int8 scale ops.py. The paper does not state this. The CUDA path does the same for its column statistics ops.py.
• Outlier set is per call. The code computes outlier columns inside each forward call from that call's activations _functions.py and _functions.py. The paper defines O over the sequence dimension of the data it evaluates §3.2; for a given input these agree only if the input is the same.
• Default CPU/reference backend. The default backend's integer matmul is a float32 matmul cast to int32 ops.py. It is a reference path, not an int8 kernel. The paper's speed results are for the CUDA kernels kernels.cu ops.cu.
• Training path. The forward pass uses int8_double_quant when the input gradient is needed, which also quantizes the transposed activations _functions.py. The paper reports that 8-bit attention projections degrade training app. E and that training at scale is not solved §6. The code exposes this path, but the paper does not validate it.
• Dequantization constant. The code uses 6.200124e-05 = 1/127² ops.py. This matches the two scale factors in eq. 7 for absmax scales of 127/max, and is consistent with the paper.
• Attention is 16-bit. Neither the paper nor the code uses int8 for the attention score and value products §6. This is a stated limitation, not a difference.

Code license: MIT, copyright notice "Facebook, Inc. and its affiliates" [LICENSE]. No code is quoted in this card.

Pitfalls

• Eq. 2 is not a zero point as printed. It multiplies X by the minimum. The zeropoint method is only used as a baseline, so this does not affect LLM.int8() itself, but an implementer should not copy eq. 2. The intended form is not stated.
• Default threshold silently gives the failing baseline. With the code's default threshold of 0.0, the layer quantizes all columns to int8. The paper shows that this vector-wise version fails at 6.7B and above tab. 1.
• Outlier weights are not 16-bit in the reference code. See Paper vs code. Choosing between the paper's 16-bit outlier weights and the code's int8-rounded ones is a memory decision. The paper's "about 0.1% additional memory" for the decomposition §3.2 does not say whether 16-bit outlier weight rows are stored separately.
• Outlier dimensions are data-dependent. The set O depends on the activations in the current call _functions.py. A short prompt can contain fewer outlier dimensions than the evaluation corpus. The paper says at most 7 dimensions for models up to 13B, but gives no bound for the dimensions in a single call §3.2.
• Scales must be computed per row and per column, not per tensor. Using one absmax for the whole activation matrix reproduces the "absmax" rows of Table 1, which fail at 6.7B and above tab. 1.
• Speed depends on size. At model dimensions up to 4096 (the 6.7B row), LLM.int8() is slower than 16-bit, from 0.14× at 768 to 0.86× at 4096 app. D.1. The text says the overhead slows "models with less than 6.7B parameters" §3.4. The table shows a slowdown at the 6.7B dimension as well (0.86×), and a speedup only at 5140 (1.22×).
• The "about two times faster" claim is loose. Section 3.4 says LLM.int8() is "about two times faster" for matrices like those of 175B models §3.4. The table gives 1.81× for LLM.int8() and 2.00× for vector-wise without decomposition app. D.1.
• "Slightly slower" end-to-end is not uniform. The text says Int8 BLOOM inference is "slightly slower but close" app. D.2. At batch size 32 the 4× and 3× A100 setups are faster than the 8× bfloat16 baseline (9.40 and 9.11 vs 9.94 ms per token) app. D.2.
• Outlier analysis numbers depend on the criteria. The counts in app. C and the "150,000 outliers per sequence" in §1 use the thresholds in §4.1. The main method only needs α = 6.0.
• No error bars. The checklist says experiments are deterministic per model and error bars are not computed [checklist]. Most comparisons are single runs; only the fine-tuning and NMT tables use multiple seeds app. E app. F.
• Hardware for timing is partly unstated. Perplexity used NVIDIA A40 GPUs §3.3. The matrix-multiplication benchmark hardware is not stated app. D.1.
• Table 2 and Appendix A duplicate one label. The memory table appears twice with the same label tab. 2 app. A. The contents match.
• Figure numbering. The wrapped figure in the introduction is Figure 1, the schematic Figure 2, and the two outlier figures Figures 3 and 4 (each with panels a and b) fig. 1 fig. 2 fig. 3 fig. 4. Appendix tables are cited by appendix section, because their numbers are not stated in the LaTeX.
• FP8 is not studied. The paper's analysis is for Int8 only §6. The authors expect many insights to carry over, but do not show it.

Lineage

• Builds on: row-wise quantization of matrix products §3 (cites FBGEMM); block-wise quantization used in optimizers §3 (cites Dettmers et al., 2022); absmax quantization, the most common method at the time §2.1.
• Compared against: Int8 absmax (tensor-wise), row-wise, zeropoint and vector-wise quantization tab. 1 §3.3; Q8BERT, Q-BERT and PSQ on GLUE finetuning app. F.
• Parallel work: nuQmm and ZeroQuant, both with finer group-wise scaling and custom CUDA kernels, and GLM-130B, which stores weights in 8-bit and computes in 16-bit §5.
• Superseded by: not known from content already in this wiki.