Paper · 2024-11-04 · cs.LG, cs.AI
"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
Empirical study of FP8 (W8A8-FP), INT8 (W8A8-INT) and INT4 weight-only (W4A16-INT) quantization of Llama-3.1 8B, 70B and 405B, plus DeepSeek-R1-Distill models, over 500,000 evaluations and vLLM speed tests. FP8 is near-lossless, tuned INT8 loses 1-3% on average, and W4A16 is cheapest for synchronous serving while W8A8 gives the most asynchronous throughput. Results are author-reported.
Weight quantizationPost-training quantizationLLM servingLLM evaluationquantizationinferencebenchmark
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Running a large language model needs a lot of memory and compute. Quantization stores the model's numbers with fewer bits, which makes the model smaller and quicker to serve, but the answers can get worse. This paper measures how much quality is lost with three common low-bit setups on the Llama 3.1 models, which come in 8, 70 and 405 billion parameter sizes. It tests them on standard exam-style benchmarks, on open-ended chat and coding tasks, on long-context retrieval and on reasoning models. It then measures how fast and how cheaply each setup runs on popular GPUs under different kinds of traffic.
The short answers are these. The 8-bit floating-point setup is almost indistinguishable from the original model. A carefully tuned 8-bit integer setup loses a few percent. Storing only the weights in 4 bits, while keeping the activations at 16 bits, holds up better than many people expected. Which setup is cheapest depends on the traffic. The 4-bit weight-only setup suits requests handled one at a time, while the 8-bit setups give more throughput when many requests run together.
This is a measurement study, not a new method. It is most useful as a guide for choosing a format, and as a list of the details that change the result.
TL;DR
- Study of FP8 (W8A8-FP), INT8 (W8A8-INT) and INT4 weight-only (W4A16-INT) quantization on Llama-3.1-Instruct 8B, 70B and 405B §3.2.
- Author-reported: the abstract calls FP8 "effectively lossless"; tuned INT8 loses 1–3% per task on average; W4A16-INT is close to 8-bit on accuracy §1.
- Cost and speed (vLLM 0.6.4.post1): W4A16 is the cheapest for synchronous, one-query-at-a-time serving; W8A8 gives the highest asynchronous throughput §1 §5.
- Use it to choose a format and to see which details (calibration data, GPTQ versus AWQ, clipping) move the accuracy. Not a new quantization method.
Problem
Serving large LLMs is expensive, and quantization is the most widely used way to cut the cost §1. The authors argue that the trade-off is unclear: systematic benchmarks and deployment guidance are scarce §1. They also cite prior work that reported drops of 10 points or more for W8A8-INT on large models §1 §2.2, and a claim that AWQ beats GPTQ for 4-bit weights §2.2. The authors say much earlier work used academic benchmarks only, and that a lack of hyperparameter tuning produced misleading conclusions about accuracy §2.2.
Method
This is an evaluation protocol, not a new algorithm. Three quantization formats are compared, each with the algorithm the authors used for it §3.2. The FP8 format quantizes all linear operators in the transformer blocks §3.2.
Basic quantizer. The paper's reference definition is asymmetric min-max
round-to-nearest (RTN) over a group of g consecutive weights, with b bits
§2.1 eq. 1. The zero point is the group minimum and the scale spreads the
group range over 2^b - 1 steps eq. 1. The three formats used in the study
are not defined by this equation; they are described in the next steps.
- W8A8-FP (FP8 weights and activations). Round-to-nearest on weights, symmetric per output channel. Activations are quantized dynamically per token. No calibration data is needed §3.2. The paper calls this "essentially lossless" §1.
- W8A8-INT (INT8 weights and activations). Weights use symmetric per-output-channel GPTQ. Activations are quantized dynamically per token §3.2. The paper applies SmoothQuant to move some activation difficulty into the weights, because the plain scheme causes noticeable drops at 70B §3.2. Calibration uses random tokens for 8B and OpenPlatypus data for the larger models §3.2.
- W4A16-INT (INT4 weights, 16-bit activations). Weights use GPTQ with MSE-optimal clipping, in groups of 128 elements §3.2. Calibration uses OpenPlatypus data, because random-token calibration degrades accuracy for this format §3.2.
- GPTQ versus AWQ for W4A16. The authors compare the two at 4 bits on the same benchmarks §3.2 tab. 1. On academic benchmarks AWQ leads by 0.23 and 0.35 points on the 0–100 scale. On real-world tasks GPTQ leads by 2.9 and 0.8 points, so the authors use GPTQ as their main INT4 method §3.2 tab. 1. They attribute the difference to three factors: MSE-optimal clipping for GPTQ (their AWQ comparison used absmax clipping), a stronger calibration set than the C4 baseline, and the inclusion of real-world benchmarks §3.2. Per-task detail is in app. A.2 tab. 8 tab. 12 tab. 13.
The evaluation covers academic benchmarks, real-world generative tasks, text similarity to the unquantized outputs, reasoning models, and inference speed §3.1 §4 §5. The benchmark definitions are in the Hyperparameters section below.
Quantization formula, for reference eq. 1:
# Asymmetric min-max round-to-nearest for one group [eq. 1] [§2.1]
# Input: group (g,) of weights; bits b
# Output: integer codes (g,), zero point, scale
def quantize_group_rtn(group, bits):
# Zero point is the group minimum [eq. 1]
zero_point = min(group)
# Scale spreads the group range over 2^b - 1 steps [eq. 1]
scale = (max(group) - min(group)) / (2 ** bits - 1)
# Round each weight to its nearest code [eq. 1]
codes = round((group - zero_point) / scale)
return codes, zero_point, scalePer-token dynamic activation quantization, used by both W8A8 formats §3.2:
# Dynamic per-token activation quantization [§3.2]
# Input: activations (tokens, d) at inference time
# Output: quantized activations (tokens, d), one scale per token
def quantize_activations_per_token(acts, qmax):
# Scale comes from the current input, not from calibration data [§3.2]
scale = max(abs(acts), axis=-1) / qmax # (tokens,)
# INT8 path: round to the nearest integer [§3.2]
quantized = round(acts / scale[:, None]) # (tokens, d)
return quantized, scaleHyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Model family | Llama-3.1-Instruct 8B, 70B, 405B | §3.2 | Instruction-tuned variants of all available sizes |
| Reasoning models | DeepSeek-R1-Distill: Llama-8B, Llama-70B, Qwen-32B, Qwen-14B, Qwen-7B, Qwen-1.5B | §4.3 tab. 4 | Quantized with the same three formats |
| W8A8-FP | FP8 for weights and activations; weights RTN, symmetric per output channel; activations dynamic per token | §3.2 | No calibration data. FP8 variant (E4M3 or E5M2) and scale formula: not stated |
| W8A8-INT | INT8 for weights and activations; weights GPTQ, symmetric per output channel; activations dynamic per token | §3.2 | SmoothQuant applied to mitigate the 70B drop; whether it is used at 8B and 405B: not stated; SmoothQuant strength: not stated |
| W4A16-INT | INT4 weights, 16-bit activations; GPTQ with MSE-optimal clipping | §3.2 app. A.2 | Group size 128 app. A.2 tab. 8 |
| Calibration data, W8A8-INT | Random tokens for 8B; OpenPlatypus for 70B and 405B | §3.2 | Number of samples and sequence length: not stated |
| Calibration data, W4A16-INT | OpenPlatypus | §3.2 | Random tokens degrade accuracy here §3.2 |
| GPTQ internals | Not stated | Dampening, block size, act-order, clipping search grid: not stated | |
| Leaderboard V1 | Meta's Llama-3.1 prompts; MMLU and ARC-C as text generation; GSM8k with CoT, 8-shot | §4.1 | Other tasks as listed in tab. 2 |
| Leaderboard V2 | Scores are normalized against chance level and mapped to 0–100 | §4.1 | Tasks: IFEval, BBH, MATH level 5, GPQA, MuSR, MMLU-Pro tab. 3 |
| Arena-Hard-Auto-v0.1 | 500 prompts; LLM judge; average of two runs; 95% confidence intervals | §4.2 tab. 7 | Judge model: not stated |
| HumanEval, HumanEval+ | pass@1 with the EvalPlus library | §4.2 | pass@10 in fig. 5 fig. 6 |
| RULER | Sequence lengths 4k to 128k | §3.1 §4.2 | Not run for 405B: too expensive for their cluster tab. 3 |
| Reasoning benchmarks | LightEval; AIME 2024, MATH-500, GPQA-Diamond | §4.3 | Temperature 0.6, top-p 0.95, 20 responses per query for pass@1 |
| Text similarity | Arena-Hard prompts, greedy sampling; ROUGE-1, ROUGE-L, BERTScore (RoBERTa-large), STS (MiniLM sentence transformers) | §3.1 §4.4 | Scores normalized to 0–1 §4.4 |
| Inference engine | vLLM 0.6.4.post1 | §1 | Setups tuned for peak BF16 and kept fixed for quantized runs §5.2 |
| GPUs | A6000, A100, H100 | §1 §5 | |
| Prices | Lambda Labs on-demand; 8×A100 price used, other A100 counts scaled proportionally | app. A.3 tab. 9 | Single price list; date of quotes not stated |
| Synchronous configurations | GPU counts per model and format | tab. 5 | Listed in the Results section |
| Workloads (prefill / decode tokens) | Code completion 256/1024; docstring generation 768/128; code fixing 1024/1024; RAG 1024/128; instruction following 256/128; multi-turn chat 512/256; summarization 4096/512 | §5 | Token counts from the text; see Pitfalls for the cost definition |
Results
Author-reported throughout. Setup: Llama-3.1-Instruct 8B, 70B and 405B, with over 500,000 evaluations in total (abstract); vLLM 0.6.4.post1 on A6000, A100 and H100 GPUs §1.
Headline claims §1 §4 §5:
- W8A8-FP: essentially lossless across benchmarks, often within the evaluation margin of error §1.
- W8A8-INT: 1–3% average accuracy loss per task §1. The paper reports 99.75% average recovery for 8-bit quantization on Leaderboard V1 §4.1.
- W4A16-INT: on par with W8A8-INT on accuracy §1. Average Leaderboard V1 recovery is 99.36% §4.1. Coding recovery is 99.9% for 8-bit and 98.9% for 4-bit §4.2, matching the 405B rows of tab. 3.
- Reasoning: recovery above 99% on average, except the smallest models at INT4 §4.3.
- Text similarity: large quantized models have ROUGE-1 about 0.7 and ROUGE-L about 0.56 against the unquantized model; 8B models have 0.62 and 0.46 §4.4 fig. 1.
- Synchronous: W4A16 cuts cost per query by 2–3× for 8B and 70B, and by 5–7× for 405B, relative to BF16 §5.1 tab. 5.
- Asynchronous: W8A8 gives the highest throughput; W4A16 is more efficient at lower latency §5.2 tab. 6 fig. 2 fig. 3.
Table 2. Open LLM Leaderboard V1, per task tab. 2. Shot counts and task settings are in the Hyperparameters section. Recovery is the quantized average divided by the BF16 average; the paper does not state the formula, but it matches (73.55 / 74.06 = 99.31 for 8B W8A8-FP).
| Size | Format | Recovery % | Avg | MMLU | MMLU-CoT | ARC-C | GSM8k | HellaSwag | Winogrande | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|---|
| 8B | BF16 | 100.00 | 74.06 | 68.3 | 72.8 | 81.4 | 82.8 | 80.5 | 78.1 | 54.5 |
| 8B | W8A8-FP | 99.31 | 73.55 | 68.0 | 71.6 | 81.2 | 82.0 | 80.0 | 77.7 | 54.3 |
| 8B | W8A8-INT | 100.31 | 74.29 | 67.8 | 72.2 | 81.7 | 84.8 | 80.3 | 78.5 | 54.7 |
| 8B | W4A16-INT | 98.72 | 73.11 | 66.9 | 71.1 | 80.2 | 82.9 | 79.9 | 78.0 | 52.8 |
| 70B | BF16 | 100.00 | 84.40 | 83.8 | 86.0 | 93.3 | 94.9 | 86.8 | 85.3 | 60.7 |
| 70B | W8A8-FP | 99.72 | 84.16 | 83.8 | 85.5 | 93.5 | 94.5 | 86.6 | 84.6 | 60.6 |
| 70B | W8A8-INT | 99.87 | 84.29 | 83.7 | 85.8 | 93.1 | 94.2 | 86.7 | 85.1 | 61.4 |
| 70B | W4A16-INT | 99.53 | 84.00 | 83.6 | 85.6 | 92.8 | 94.4 | 86.3 | 85.5 | 59.8 |
| 405B | BF16 | 100.00 | 86.79 | 87.4 | 88.1 | 95.0 | 96.0 | 88.5 | 87.2 | 65.3 |
| 405B | W8A8-FP | 100.12 | 86.89 | 87.5 | 88.1 | 95.0 | 95.8 | 88.5 | 88.0 | 65.3 |
| 405B | W8A8-INT | 99.32 | 86.20 | 87.1 | 87.7 | 94.4 | 95.5 | 88.2 | 86.1 | 64.4 |
| 405B | W4A16-INT | 99.98 | 86.78 | 87.2 | 87.7 | 95.3 | 96.3 | 88.3 | 87.4 | 65.3 |
Table 3. Leaderboard V2 average and real-world benchmarks tab. 3. Condensed: the per-task Leaderboard V2 columns (IFEval, BBH, MATH level 5, GPQA, MuSR, MMLU-Pro) are in tab. 3. RULER is not run for 405B.
| Size | Format | Recovery % | Leaderboard V2 average | Arena-Hard win-rate | HumanEval pass@1 | HumanEval+ pass@1 | RULER score |
|---|---|---|---|---|---|---|---|
| 8B | BF16 | 100.0 | 27.6 | 25.8 | 67.3 | 60.7 | 82.8 |
| 8B | W8A8-FP | 101.2 | 27.9 | 26.8 | 67.3 | 61.3 | 82.8 |
| 8B | W8A8-INT | 101.5 | 28.0 | 27.2 | 67.1 | 60.0 | 82.8 |
| 8B | W4A16-INT | 96.1 | 26.5 | 24.0 | 67.1 | 59.1 | 81.1 |
| 70B | BF16 | 100.0 | 41.7 | 57.0 | 79.7 | 74.8 | 83.3 |
| 70B | W8A8-FP | 100.0 | 41.7 | 57.7 | 80.0 | 75.0 | 83.0 |
| 70B | W8A8-INT | 97.3 | 40.5 | 57.0 | 78.7 | 74.0 | 82.5 |
| 70B | W4A16-INT | 97.4 | 40.6 | 56.3 | 80.5 | 74.2 | 82.2 |
| 405B | BF16 | 100.0 | 48.7 | 67.4 | 86.8 | 80.1 | - |
| 405B | W8A8-FP | 99.9 | 48.7 | 66.9 | 87.0 | 81.0 | - |
| 405B | W8A8-INT | 98.3 | 47.9 | 64.6 | 86.9 | 80.4 | - |
| 405B | W4A16-INT | 98.9 | 48.2 | 66.5 | 85.1 | 78.9 | - |
Table 4. Reasoning models, pass@1 over 20 samples tab. 4. The ± values are reported as given; their definition is not stated.
| Model | Format | Recovery % | Average | AIME24 | MATH-500 | GPQA-Diamond |
|---|---|---|---|---|---|---|
| Llama-8B | BF16 | 100.0 | 62.9 | 49.3 ± 6.4 | 90.2 ± 1.2 | 49.3 ± 3.1 |
| Llama-8B | W8A8-FP | 100.6 | 63.3 | 50.8 ± 9.0 | 90.2 ± 1.1 | 48.7 ± 2.5 |
| Llama-8B | W8A8-INT | 99.6 | 62.7 | 49.1 ± 6.2 | 90.0 ± 1.0 | 48.9 ± 2.0 |
| Llama-8B | W4A16-INT | 97.2 | 61.1 | 46.3 ± 6.9 | 89.9 ± 1.1 | 47.1 ± 2.6 |
| Llama-70B | BF16 | 100.0 | 76.2 | 67.8 ± 7.2 | 95.3 ± 0.7 | 65.6 ± 2.3 |
| Llama-70B | W8A8-FP | 100.3 | 76.5 | 69.2 ± 6.5 | 95.1 ± 0.5 | 65.2 ± 2.4 |
| Llama-70B | W8A8-INT | 99.7 | 76.0 | 67.8 ± 6.4 | 95.3 ± 0.5 | 65.0 ± 1.8 |
| Llama-70B | W4A16-INT | 98.3 | 75.0 | 65.6 ± 5.3 | 95.2 ± 0.6 | 64.0 ± 2.8 |
| Qwen-32B | BF16 | 100.0 | 76.3 | 69.8 ± 4.9 | 95.1 ± 0.6 | 64.1 ± 2.1 |
| Qwen-32B | W8A8-FP | 99.0 | 75.6 | 68.5 ± 4.0 | 95.3 ± 0.7 | 62.9 ± 2.6 |
| Qwen-32B | W8A8-INT | 99.6 | 76.0 | 68.2 ± 5.1 | 95.0 ± 0.8 | 64.8 ± 2.6 |
| Qwen-32B | W4A16-INT | 99.5 | 75.9 | 68.8 ± 4.2 | 95.0 ± 0.5 | 63.8 ± 1.7 |
| Qwen-14B | BF16 | 100.0 | 73.6 | 66.7 ± 5.1 | 94.7 ± 0.7 | 59.4 ± 2.3 |
| Qwen-14B | W8A8-FP | 101.0 | 74.3 | 68.1 ± 5.8 | 94.6 ± 0.6 | 60.1 ± 2.9 |
| Qwen-14B | W8A8-INT | 99.4 | 73.1 | 66.3 ± 7.1 | 94.7 ± 0.7 | 58.3 ± 2.0 |
| Qwen-14B | W4A16-INT | 99.0 | 72.8 | 66.0 ± 6.3 | 95.0 ± 0.5 | 57.5 ± 2.1 |
| Qwen-7B | BF16 | 100.0 | 65.8 | 53.2 ± 6.4 | 93.7 ± 0.8 | 50.5 ± 2.8 |
| Qwen-7B | W8A8-FP | 99.9 | 65.7 | 53.2 ± 7.5 | 93.6 ± 0.7 | 50.3 ± 2.0 |
| Qwen-7B | W8A8-INT | 100.7 | 66.3 | 55.2 ± 4.9 | 93.0 ± 1.1 | 50.7 ± 3.5 |
| Qwen-7B | W4A16-INT | 98.3 | 64.7 | 50.9 ± 7.8 | 93.3 ± 1.1 | 49.8 ± 2.8 |
| Qwen-1.5B | BF16 | 100.0 | 50.0 | 30.1 ± 5.3 | 84.7 ± 1.1 | 35.4 ± 3.0 |
| Qwen-1.5B | W8A8-FP | 100.3 | 50.2 | 29.8 ± 5.6 | 84.7 ± 1.3 | 35.9 ± 3.3 |
| Qwen-1.5B | W8A8-INT | 96.9 | 48.5 | 26.7 ± 6.3 | 84.4 ± 1.1 | 34.4 ± 2.8 |
| Qwen-1.5B | W4A16-INT | 93.5 | 46.8 | 24.6 ± 5.1 | 82.5 ± 1.1 | 33.2 ± 3.4 |
Table 5. Synchronous deployment (one query at a time), condensed tab. 5. Shown: code completion and RAG latency (s) and queries per USD. The other five use cases and the cost-reduction column are in tab. 5. # is the GPU count; CR is the cost reduction against BF16 as defined in the table footnote.
| Size | GPU | # | Format | CR | Code completion latency | Code completion Q/USD | RAG latency | RAG Q/USD |
|---|---|---|---|---|---|---|---|---|
| 8B | A6000 | 1 | BF16 | - | 24.5 | 183 | 3.3 | 1,374 |
| 8B | A6000 | 1 | INT8 | 1.54 | 15.9 | 284 | 2.1 | 2,139 |
| 8B | A6000 | 1 | INT4 | 2.39 | 9.7 | 462 | 1.4 | 3,136 |
| 70B | A6000 | 4 | BF16 | - | 61.7 | 18 | 8.1 | 138 |
| 70B | A6000 | 2 | INT8 | 1.94 | 63.4 | 35 | 8.4 | 267 |
| 70B | A6000 | 2 | INT4 | 2.96 | 39.2 | 57 | 5.8 | 390 |
| 70B | A100 | 2 | BF16 | - | 50.7 | 20 | 6.8 | 148 |
| 70B | A100 | 1 | INT8 | 1.81 | 54.3 | 37 | 7.2 | 279 |
| 70B | A100 | 1 | INT4 | 2.67 | 35.0 | 57 | 5.2 | 390 |
| 70B | H100 | 2 | BF16 | - | 31.3 | 18 | 4.1 | 138 |
| 70B | H100 | 1 | FP8 | 1.84 | 32.8 | 33 | 4.3 | 254 |
| 70B | H100 | 1 | INT4 | 2.11 | 28.6 | 38 | 3.8 | 287 |
| 405B | A100 | 16 | BF16 | - | 81.9 | 2 | 11.2 | 11 |
| 405B | A100 | 8 | INT8 | 3.27 | 50.1 | 5 | 6.8 | 37 |
| 405B | A100 | 4 | INT4 | 6.38 | 48.9 | 10 | 7.3 | 68 |
| 405B | H100 | 16 | BF16 | - | 50.6 | 1 | 6.6 | 11 |
| 405B | H100 | 8 | FP8 | 3.17 | 31.7 | 5 | 4.2 | 36 |
| 405B | H100 | 4 | INT4 | 5.15 | 37.5 | 8 | 5.1 | 57 |
Table 6. Asynchronous deployment (continuous batching), condensed tab. 6. Shown: Speedup, and code completion and RAG throughput (queries per second) and queries per USD. "Speedup" is not defined in the compiled paper (see Pitfalls).
| Size | Hardware | Format | Speedup | Code completion QPS | Code completion Q/USD | RAG QPS | RAG Q/USD |
|---|---|---|---|---|---|---|---|
| 8B | 1×A6000 | BF16 | - | 1.5 | 6.8k | 4.4 | 19.9k |
| 8B | 1×A6000 | INT8 | 1.38 | 2.2 | 9.8k | 6.1 | 27.6k |
| 8B | 1×A6000 | INT4 | 1.08 | 2.2 | 9.8k | 4.1 | 18.6k |
| 70B | 4×A6000 | BF16 | - | 0.4 | 0.4k | 1.4 | 1.6k |
| 70B | 4×A6000 | INT8 | 1.91 | 0.7 | 0.8k | 2.8 | 3.1k |
| 70B | 4×A6000 | INT4 | 1.92 | 1.2 | 1.4k | 1.9 | 2.1k |
| 70B | 4×A100 | BF16 | - | 1.4 | 0.7k | 3.3 | 1.6k |
| 70B | 4×A100 | INT8 | 1.87 | 2.4 | 1.2k | 6.1 | 3.1k |
| 70B | 4×A100 | INT4 | 1.64 | 2.3 | 1.2k | 4.3 | 2.2k |
| 70B | 4×H100 | BF16 | - | 3.5 | 1.0k | 8.0 | 2.3k |
| 70B | 4×H100 | FP8 | 1.77 | 6.9 | 2.0k | 14.3 | 4.2k |
| 70B | 4×H100 | INT4 | 1.55 | 5.9 | 1.7k | 13.0 | 3.8k |
| 405B | 16×A100 | BF16 | - | 0.8 | 59 | 2.1 | 156 |
| 405B | 16×A100 | INT8 | 2.53 | 1.3 | 98 | 3.8 | 282 |
| 405B | 16×A100 | INT4 | 2.21 | 1.9 | 144 | 2.8 | 211 |
| 405B | 16×H100 | BF16 | - | 0.7 | 52 | 4.8 | 363 |
| 405B | 16×H100 | FP8 | 3.04 | 4.4 | 329 | 7.6 | 571 |
| 405B | 16×H100 | INT4 | 3.09 | 4.0 | 304 | 8.7 | 652 |
Tables 5 and 6 label the formats INT8, INT4 and FP8, where the other tables use W8A8-INT, W4A16-INT and W8A8-FP tab. 5 tab. 6.
Text similarity §4.4 fig. 1: the figure is a heatmap; its values are given in the text only as averages (see Results headline above).
Limitations (unnumbered section): KV-cache, input embeddings and the LM head were not quantized; multilingual use was not studied [Limitations].
Paper vs code
No official code.
Searched, in order: the paper source (no repository or project URL appears in
the LaTeX); the arXiv abstract page (comment "Accepted to ACL 2025", no code
link). GitHub searches via the API: the phrase "bf16 or give me death" (0
results); "quantization eldar kurtic" (0); organisation-scoped queries on
neuralmagic ("quantization accuracy", "quantized llm evaluation", "quantization
benchmark": 0 hits relevant to this paper); IST-DASLab ("eval": one unrelated
benchmark repo, local_platinum_bench). Not used:
vllm-project/llm-compressor(Apache-2.0): the quantization library the authors' group works on, but owned by vllm-project and not linked from this paper, so it is not this paper's official code.neuralmagic/quant_kernel_benchmarks: kernel benchmarks, with no license detected; not this paper's evaluation.
Pitfalls
- Eq. 1 is not the formula used. It is asymmetric min-max, but W8A8-FP and the INT8 weights are symmetric per output channel §3.2. The paper gives no symmetric scale formula, no integer range for INT8, and no FP8 variant (E4M3 or E5M2). These are "not stated".
- Constant groups. Eq. 1 divides by
max(x) - min(x). The paper does not say how a group with equal values is handled. - Activation scheme is partly unspecified. The text says activations are quantized "dynamically per token" §3.2. It does not say whether that is symmetric, how the FP8 cast is done, or whether the scale is clipped.
- GPTQ and clipping details are missing. Calibration sample count, sequence length, dampening, block size, act-order and the search grid for "MSE-optimal" clipping are not stated §3.2. Results depend on this ("well-tuned" INT8 is the paper's own word §1). Expect a gap when reproducing.
- SmoothQuant scope. The paper applies SmoothQuant to fix the 70B drop §3.2. Whether it is used for 8B and 405B, and its migration strength, is not stated.
- Calibration data matters. For W4A16, random-token calibration degrades accuracy and OpenPlatypus is used instead §3.2. For W8A8-INT, random tokens are used at 8B only §3.2. Mixing these up changes the result.
- Text versus table. §2.2 says W8A8-INT is 0.7 points below FP8 on the 405B Leaderboard V2 average. Table 3 gives 48.7 (FP8) and 47.9 (INT8), a gap of 0.8 tab. 3. Rounding may explain it; the paper does not say.
- RULER claim. §4.2 says recovery is at least 98% for all formats. Table 3 gives 81.1 / 82.8 = 97.9% for 8B W4A16-INT tab. 3.
- Cost definition. "Q/USD" is queries per USD using Lambda's on-demand prices §5 tab. 9. How per-query time was turned into cost is not stated. For A100 the 8× price is used and smaller counts are scaled proportionally app. A.3.
- "Speedup" is undefined. The footnote that would define it in Table 6 is inside a LaTeX comment in the source, so the compiled paper does not define it tab. 6.
- Fair baseline? The asynchronous setups were tuned for the BF16 baseline and then reused unchanged for the quantized models §5.2. Tuning for BF16 may favour BF16 in the comparison; the paper does not test a separately tuned configuration per format.
- Reasoning variance. The ± values are not defined. The authors say sampling 20 responses was needed because of high variance on small datasets §4.3. Single-sample comparisons on these tasks are unreliable.
- Greedy versus sampled. Text similarity uses greedy decoding §4.4; the reasoning runs use temperature 0.6 and top-p 0.95 §4.3. Do not mix them.
- Judge model. Arena-Hard uses an LLM judge §4.2; the judge model is not named in the text.
- Scope. Only Llama-3.1 instruct models and DeepSeek-R1-Distill models are tested; multilingual use is not studied [Limitations]. The GPTQ-over-AWQ finding is for Llama-3.1 with the stated tuning; it is not a general claim §3.2 §2.2.
- Author-reported. All numbers are from the authors' runs. No independent replication is in the paper.
Lineage
- Builds on: GPTQ §2.1 §3.2 (weight quantization with calibration data), AWQ §2.1 (outlier-aware 4-bit weights, used as the comparison), SmoothQuant §1 §3.2 (moves activation outliers into weights).
- Compared against: AWQ and GPTQ for W4A16 tab. 1; prior accuracy studies that reported larger 8-bit drops and an AWQ advantage §2.2.
- Superseded by: not stated.
Source and license
Adapted from “"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization” by Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz, Dan Alistarh (arXiv:2411.02355), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.