---
kind: concept
slug: low-precision-arithmetic
title: Low-precision arithmetic
summary: >-
  Store or multiply numbers in fewer bits than FP32 to cut memory and raise throughput. Twelve wikillm papers cover it: two surveys, FP8 attention, NVFP4 and INT4 LLMs, 4-bit clipping, 16-bit storage, FP32 decoding, FP16 kernels, float32 rounding that causes loss spikes, and bfloat16 versus float32 on TPUs.
tags: [gpu-kernels, quantization, fp8, nvfp4, int4, numerics, compression, mixed-precision, reproducibility, training-stability, benchmarking]
papers: ["1805.06085", "2604.12374", "2406.17074", "2407.08608", "2408.15857", "2506.09501", "2512.02551", "2210.02414", "2112.06126", "2512.23914", "2605.06152", "1907.10701"]
related: [fused-attention-kernel, weight-quantization, post-training-quantization, quantization-aware-training, asymmetric-quantization, model-flops-utilization, gaussian-splatting, vector-quantization, object-detection, hardware-aware-training, state-space-model, llm-serving, llm-evaluation, language-model-pretraining, arithmetic-intensity, kv-cache-management, binary-neural-network, adaptive-rounding, batchnorm-folding, grokking, gradient-descent-stability, benchmark-validity]
status: draft
updated_at: 2026-11-01
---

## Overview

Low-precision arithmetic stores and multiplies numbers with fewer bits than
the usual FP16 or BF16, so the matrix units do more work per second and
move less data. For the attention kernel on NVIDIA Hopper the relevant format
is FP8 in the e4m3 layout, which has few mantissa bits and therefore loses
accuracy unless the values are scaled first [arxiv:2407.08608 §2.2]
[arxiv:2407.08608 §3.3].

The invariant the FP8 path tries to keep is that the attention output stays
close to the FP16 result. The paper's tools are one scale per block of
values, and an orthogonal matrix M applied to Q and K before quantization.
Because M M^T = I, (Q M)(K M)^T equals Q K^T in exact arithmetic, so the
rotation changes the numbers the quantizer sees without changing the scores
[arxiv:2407.08608 §3.3].

Eight wikillm papers use low precision in eight different ways: low-precision
compute in attention [arxiv:2407.08608], integer activations with a learned
clip level [arxiv:1805.06085 §4], 4-bit floating-point pretraining and
inference checkpoints of an LLM [arxiv:2604.12374 §2.2] [arxiv:2604.12374 §4],
16-bit storage of compressed 3D Gaussian Splatting attributes
[arxiv:2406.17074 §4.3], 16-bit mixed precision as one part of a
detector's training recipe [arxiv:2408.15857 §3.5], BF16 weight storage
with FP32 arithmetic to reduce run-to-run variation in LLM inference
[arxiv:2506.09501 §4], FP16 matrix-multiply kernels whose search reward
penalises numerical error [arxiv:2512.02551 §3.2.3], and FP16 training of a
130B model with INT4 weight-only inference [arxiv:2210.02414 §3]
[arxiv:2210.02414 §4]. They measure different quantities (classifier accuracy,
LLM benchmarks, attention error, file size, output-length variation, kernel
speed, perplexity-style scores), so their numbers should not be compared.

Two surveys place these methods in context. The first defines the uniform
integer quantizer, separates quantization-aware training from post-training
quantization, and surveys binary, ternary and mixed-precision schemes
[arxiv:2112.06126 §3.1] [arxiv:2112.06126 §4] [arxiv:2112.06126 §6]. The
second lists lowering numeric precision as one of its
optimisation levers [arxiv:2512.23914 §1] and gives a KV-cache size formula in
which the bytes per element is a parameter [arxiv:2512.23914 eq. 15].

One more card, arXiv:2605.06152, looks at the reverse risk: float32 rounding
inside the cross-entropy loss is argued to cause loss spikes in long training,
so there the precision is the problem to fix. See the variant below the
INT4 section.

A last card, arXiv:1907.10701, is a hardware benchmark. It does not study
precision as its subject, but its TPU runs use bfloat16 and its GPU runs use
float16, and it reports bfloat16 against float32 speedups on TPU. Those
numbers mix precision with memory traffic and batch size, so they are read
as a caution in the variant below.

## Variants

### FP16 or BF16 (reference)

The higher-precision path of the same kernel. It is the accuracy reference
for the FP8 path and reaches up to 740 TFLOPs/s, about 75% of the H100
theoretical FP16 peak, with 1.5-2.0x speedup over FlashAttention-2 in the
authors' benchmarks [arxiv:2407.08608 §4.1] [arxiv:2407.08608 §1]. Card:
[FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](/papers/2407.08608).

### FP8 (e4m3) with block quantization and incoherent processing

The paper's FP8 forward pass has four parts [arxiv:2407.08608 §3.3]:

1. V is transposed inside the kernel, because FP8 matrix instructions need
   a particular operand layout.
2. The accumulator registers of the first matrix multiply are permuted so
   they can feed the second multiply as an operand.
3. Q, K and V are split into blocks, and each block gets its own scale.
4. Q and K are rotated by the random-sign Hadamard matrix described above
   before quantization.

Reported speed: FP8 forward reaches close to 1.2 PFLOPs/s [arxiv:2407.08608 §1]
[fig. 7]. Reported accuracy: on synthetic inputs with rare large outliers,
the RMSE against an FP64 reference is 9.1e-3, against 2.4e-2 for a
per-tensor-scaled FP8 baseline, which the paper calls 2.6x lower error
[arxiv:2407.08608 tab. 3] [arxiv:2407.08608 §4.3].

The ablation separates the two accuracy tricks on that same test. Removing
block quantization moves the RMSE to 9.3e-3, a small change. Removing the
rotation brings it back to 2.4e-2, the baseline level [arxiv:2407.08608 tab. 3].
On this test the rotation does most of the work.

Card: [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](/papers/2407.08608).
The code at the pinned commit does not contain the rotation or per-block
scales; see Contradictions.

### NVFP4 (4-bit float with block scales): pretraining and inference checkpoints

Nemotron 3 Super is a hybrid Mamba-Transformer mixture-of-experts LLM. Most of
its linear layers are trained in NVFP4: E2M1 elements, FP8 E4M3 micro-block
scales and a second-level FP32 scale [arxiv:2604.12374 §2.2]. The forward
and both backward matrix multiplies use NVFP4. Random Hadamard transforms
are applied to the inputs of the weight-gradient multiply, and gradients are
rounded stochastically [arxiv:2604.12374 §2.2]. Other layers stay higher
precision, as listed in [arxiv:2604.12374 tab. 3]: the final 15% of the
network, the MTP heads, the latent projections, the QKV and attention
projections, and the embeddings are BF16, and the Mamba output projection is
MXFP8.

The paper reports that the share of zero-valued weight-gradient elements
grew to about 7% of parameters by the end of pretraining, traced to underflow
in expert layers [arxiv:2604.12374 §2.2] [fig. 6] [fig. 8]. Switching the last
phase to MXFP8 improved the loss but not downstream accuracy, so the final run
stayed in NVFP4 [arxiv:2604.12374 §2.2] [fig. 9].

The released checkpoints are post-training quantized from the BF16 model:

- **FP8 (W8A8).** MoE and Mamba linear layers and the KV cache are FP8; the
  router stays FP32. Calibration uses 256 SFT samples at 65,536 context
  [arxiv:2604.12374 §4.1] [arxiv:2604.12374 tab. 6]. On the authors' benchmark
  table the FP8 scores are within about one point of BF16 on most rows
  [arxiv:2604.12374 tab. 8].
- **NVFP4 (W4A4).** Weight block scales are chosen to minimize weight error,
  using 16-element blocks; activation block scales are computed from the block
  maximum at runtime [arxiv:2604.12374 §4.2] [app. B.1]. A sensitivity search
  (AutoQuantize) then promotes selected operators to FP8 or BF16 under an
  average budget of 4.75 bits [arxiv:2604.12374 §4.2] [app. B.2]
  [arxiv:2604.12374 tab. 7]. The paper reports 512 SFT samples at sequence
  length 4096, under two hours on one 8-GPU node, and 99.8% median accuracy
  relative to BF16 [arxiv:2604.12374 §4.2]. Per benchmark, the drop is larger
  on some rows, for example 2.4 points on Scale AI Multi-Challenge
  [arxiv:2604.12374 tab. 8].

Card: [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](/papers/2604.12374).

### Uniform integer quantization and the two training routes

The general recipe maps a real value `r` to an integer through a scale `S` and
an integer zero point `Z` [arxiv:2112.06126 §3.1] [arxiv:2112.06126 eq. 5].
For a clipping range `[alpha, beta]` and `b` bits, the step size is
`(beta - alpha) / (2^b - 1)` [arxiv:2112.06126 §3.1.1] [arxiv:2112.06126 eq. 6].
Symmetric quantization sets `alpha = -beta` and `Z = 0`. Asymmetric
quantization fits an unbalanced range, for example after ReLU [arxiv:2112.06126
§3.1.1]. The survey gives no formula for `Z` from `alpha` and `beta`
[arxiv:2112.06126 §3.1].

The two routes differ in when the rounding is learned:

- Quantization-aware training (QAT) runs the rounding in the forward pass,
  keeps float weights, and treats the rounding as the identity in the backward
  pass [arxiv:2112.06126 §4.1].
- Post-training quantization (PTQ) calibrates the ranges on a small set of
  unlabelled data and does not retrain [arxiv:2112.06126 §4.2].

The survey's summary table gives 1 bit as the minimum for QAT, citing XNOR-Net,
and 4 bits for PTQ, citing AdaRound [arxiv:2112.06126 tab. 1]. For 8-bit
integer inference on ImageNet, the survey says Gong et al. lose less than 1%
[arxiv:2112.06126 §5]. Its table gives that PTQ ResNet50 row as 71.84% without
a float baseline [arxiv:2112.06126 tab. 4].

Card: [Neural Network Quantization for Efficient Inference: A Survey](/papers/2112.06126).

### Integer activations with a learned clip level (PACT)

Here the format is a k-bit integer grid, not a float. Each layer's
activations are clipped to [0, alpha], where alpha is a trainable parameter,
and then rounded to 2^k levels spaced alpha / (2^k - 1) apart
[arxiv:1805.06085 §4] [arxiv:1805.06085 eq. 1] [arxiv:1805.06085 eq. 2]. The
backward pass uses a straight-through estimator for the rounding, so alpha
receives gradient from the loss [arxiv:1805.06085 eq. 3]. The authors add an
L2 penalty on alpha to shrink the range during training [arxiv:1805.06085 §4]
[arxiv:1805.06085 app. A.2]. Their clip level is one value per layer, and they
start it at a large value [arxiv:1805.06085 §4.2].

Accuracy with 4-bit weights (DoReFa) and 4-bit activations (PACT), trained
from scratch: CIFAR10 91.3% against 91.6% full precision, ResNet18 69.2%
against 70.2%, and ResNet50 76.5% against 76.9% [arxiv:1805.06085 tab. 1]
[arxiv:1805.06085 app. B]. The paper's first and last layers stay at full
precision in the main comparison [arxiv:1805.06085 §4.2]. Quantizing them too
costs accuracy in a CIFAR10 ResNet20 study: with middle layers at 2 bits,
validation error rises from 10.9% to 21.0% when the first and last layers are
also 2 bits [arxiv:1805.06085 tab. 2].

Card: [PACT: Parameterized Clipping Activation for Quantized Neural Networks](/papers/1805.06085).
Its hardware gains come from a model (a 14 nm MAC area estimate and a
performance simulator), not measured silicon [arxiv:1805.06085 §6]
[arxiv:1805.06085 fig. 6].

### Binary and ternary weights (1 to 2 bits)

BinaryConnect draws each weight stochastically from a hard sigmoid, keeps float
master weights that receive the gradients, and uses the binary copies only in
the forward and backward passes [arxiv:2112.06126 §6.1] [arxiv:2112.06126 eq. 11].
BinaryNet uses the sign function instead [arxiv:2112.06126 eq. 13]. With
binary weights and activations a dot product becomes XNOR and popcount
[arxiv:2112.06126 eq. 14]. Binary-weight networks and XNOR-Net scale each
binary weight by the mean absolute weight of the matrix [arxiv:2112.06126 eq. 15].

Ternary schemes use three values. Ternary Weight Networks set a threshold of
about 0.7 times the mean absolute weight, a factor the survey says assumes
normally distributed weights [arxiv:2112.06126 §6.2]. Trained Ternary
Quantization learns separate positive and negative scales [arxiv:2112.06126
§6.2].

On ImageNet with ResNet18, the table lists TWN (2-bit weights) at 61.8% and
XNOR-Net (1-bit weights and activations) at 51.2%, with no float baseline for
those rows [arxiv:2112.06126 tab. 4]. The text calls TWN's loss on that network
small, about 4% [arxiv:2112.06126 §6.2]. On CIFAR-10 the table gives BNN
(1-bit weights and activations) at 88.6% [arxiv:2112.06126 tab. 3].

Card: [Neural Network Quantization for Efficient Inference: A Survey](/papers/2112.06126).

### 16-bit float storage of attributes (not compute)

This variant lowers only the stored size. In the Gaussian Splatting compression
method, positions and k-means codebook entries are stored as 16-bit floats
instead of 32-bit, and the other attributes are stored as one-byte codebook
indices [arxiv:2406.17074 §4.3]. The authors report one ablation row for this
step on Mip-NeRF360: a 38 MB k-means file with PSNR 27.17 becomes a 29 MB half-float
file with PSNR 27.10 [arxiv:2406.17074 tab. 4]. The scene is decoded when it is
loaded, not in the render loop [arxiv:2406.17074 §5]. The authors say render-time
VRAM would fall only once the codebooks and half floats are used by the renderer
itself [arxiv:2406.17074 §6.1], so this variant saves disk and transfer size,
not GPU memory at render time.

Card: [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074).

### Recurrent state cache in FP16 (stochastic rounding)

The Mamba state cache of Nemotron 3 Super is FP32 by default. The paper
reports that storing it in plain FP16 made generations longer: up to about
40% more completion tokens on LiveCodeBench with W8A8 weights
[arxiv:2604.12374 §4.3] [arxiv:2604.12374 tab. 9]. Converting to FP16 with
stochastic rounding (Philox, 5 rounds) brought the length change to within
about 2% on the same benchmark [arxiv:2604.12374 tab. 9]. The cache study used
an early checkpoint and code benchmarks only [arxiv:2604.12374 tab. 9].

Card: [Nemotron 3 Super](/papers/2604.12374).

### 16-bit mixed-precision training (named, not measured)

The YOLOv8 review lists "mixed precision with 16-bit floats" as one of four
parts of the training recipe [arxiv:2408.15857 §3.5]. It says mixed precision
significantly accelerates training, reduces memory use and enables larger batch
sizes [arxiv:2408.15857 §3.5]. The paper gives no numbers for these claims, no
list of operations kept in 32-bit, and no batch size, learning rate or epoch
count [arxiv:2408.15857 §3.5] [arxiv:2408.15857 §3].

Card: [What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector](/papers/2408.15857).

### BF16 storage with FP32 compute (LayerCast, for reproducibility)

This variant targets run-to-run variation, not speed. The paper's analysis is
that, under BF16 with greedy decoding, the same model, prompt and seed give
different accuracy and output lengths when the GPU count, GPU type or batch
size changes. Floating-point addition is not associative, and serving changes
the order of the sums [arxiv:2506.09501 §2.2] [arxiv:2506.09501 §3.2]. For
DeepSeek-R1-Distill-Qwen-7B on AIME'24, the accuracy standard deviation across
12 configurations is 9.15% in BF16 and 0 in FP32 [arxiv:2506.09501 tab. 3].
FP32 removes most of the variation, and FP16 falls in between [arxiv:2506.09501
§3.2].

LayerCast keeps the linear-layer weights and biases in BF16. Each layer's
weight is converted to FP32 just before its matmul, so the arithmetic runs in
FP32 [arxiv:2506.09501 §4]. The paper reports memory 34% below FP32, without a
table or a stated method [arxiv:2506.09501 §4]. On the same model across five
benchmarks, the LayerCast accuracy standard deviation is close to FP32 and
below BF16 (AIME'24: 0 for LayerCast and FP32, 0.0544 for BF16, as fractions)
[arxiv:2506.09501 app. F]. The paper reports divergence below 3.4% across
configurations, shown only in a figure [arxiv:2506.09501 §4].

```pseudocode
# LayerCast for one linear layer  [arxiv:2506.09501 §4]
# Input:  x (tokens, in) in FP32
# Output: y (tokens, out) in FP32

# At rest the weight and bias are 16-bit (the paper says all linear layers)
w_bf16 = to_bfloat16(layer.weight)
b_bf16 = to_bfloat16(layer.bias)
# Just before the matmul, make a temporary FP32 copy
w_fp32 = to_float32(w_bf16)
b_fp32 = to_float32(b_bf16)
y = linear(x, w_fp32, b_fp32)         # the arithmetic is FP32
```

Card: [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501).
Its released script does not run as shipped, and its memory saving is not
explained by the code; see Contradictions.

### FP16 matrix multiply found by RL, with a numerical penalty

This variant keeps the format at FP16 and changes the kernel. CUDA-L2 uses a
reinforcement-learning loop to write half-precision matrix-multiply (HGEMM)
kernels for 1,000 (M, N, K) shapes [arxiv:2512.02551 §1]. The reward is the
mean measured speedup over a reference, minus a penalty on the largest
element-wise difference from an FP32 CPU result, minus a penalty on code
length [arxiv:2512.02551 eq. 3] [arxiv:2512.02551 §3.2.3]. The two penalty
coefficients are not stated [arxiv:2512.02551 §3.2.3]. During the search,
correctness is an exact-match test on random 0/1 inputs, which only covers
outputs below 2048 [arxiv:2512.02551 §2.3]. The paper does not state the
accumulator precision of its main A100 table [arxiv:2512.02551 §1]
[arxiv:2512.02551 §2.3]. The reported offline mean speedup is 11.4% over
cuBLASLt-AutoTuning-max and 22.0% over torch.matmul, both author-reported
[arxiv:2512.02551 tab. 1]. Card: [CUDA-L2: Surpassing cuBLAS Performance for
Matrix Multiplication through Reinforcement Learning](/papers/2512.02551).

### FP16 training at 100B scale with DeepNorm and embedding gradient shrink

The authors trained a 130B-parameter model in FP16 and report that the usual
stability fixes did not work for them: Pre-LN, Sandwich-LN and PB-Relax all
failed in their runs [arxiv:2210.02414 §3] [arxiv:2210.02414 app. A]. The fix
they kept combines DeepNorm Post-LN scaling with a gradient shrink of 0.1 on
the embedding layer, which they say wipes out most loss spikes
[arxiv:2210.02414 §2.1] [arxiv:2210.02414 §3]. Forward and backward passes are
FP16, with FP32 master weights and optimizer states [arxiv:2210.02414 §3].
BF16 avoids the overflow, but the authors measured about 15% more GPU memory
with it, and it is not supported on V100 [arxiv:2210.02414 §3]. The paper
reports three late loss divergences in its final run [arxiv:2210.02414 §3].

Card: [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414).

### INT4 weight-only inference with row-wise absmax scales

Only the linear-layer weights are stored in 4 bits. Each output row has its
own scale, its absolute maximum divided by 7, and the integers are rounded to
the range -7 to 7. Weights are converted back to FP16 at run time, and
activations stay FP16 [arxiv:2210.02414 §4] [arxiv:2210.02414 app. A].
Embeddings, norms and biases also stay FP16 [arxiv:2210.02414 app. A]. The
authors say the quantized weights take about 70 GB, which fits 4 × RTX 3090
(24 GB) cards [arxiv:2210.02414 §4]. Against FP16, the reported scores are
LAMBADA 79.47 against 80.21, MMLU 44.80 against 44.75 and Pile bits per byte
0.641 against 0.634 [arxiv:2210.02414 §4]. For 128 generated tokens on 4 ×
RTX 3090, encoding takes 0.37 s and decoding 8.16 s, with FasterTransformer
[arxiv:2210.02414 §4]. The paper does not quantize activations, because they
contain outlier dimensions, reported at up to about 30% of dimensions
[arxiv:2210.02414 §4] [arxiv:2210.02414 app. A]. An appendix table gives a
zero-point INT4 variant at 80.63% on LAMBADA against 79.47% for absmax, and
the authors chose absmax for cost and because it kept performance
[arxiv:2210.02414 app. A].

Card: [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414).

### Rounding in the loss as a cause of instability (a risk, not a speed-up)

This variant reverses the usual question: float32 rounding is the suspect.
The paper's argument is that softmax cross-entropy computes its denominator by
log-sum-exp. Once the correct logit leads the others by more than about 16,
the other terms are absorbed, and the correct-class gradient rounds to
exactly zero [arxiv:2605.06152 §3.1] [arxiv:2605.06152 eq. 1]
[arxiv:2605.06152 eq. 2]. The wrong-class gradients stay non-zero, so the
classes no longer cancel. The global classifier mean and the feature mean then
push each other and grow, which the authors call Numerical Feature Inflation
[arxiv:2605.06152 §3.2] [arxiv:2605.06152 thm. 2]. When some margins fall back
below the threshold, the correct-class gradient returns after a long quiet
period, and Adam's small second moment turns it into a very large step
[arxiv:2605.06152 §3.3].

The paper tests three changes, each of which removed the spikes in its own
small runs [arxiv:2605.06152 §4.1] [arxiv:2605.06152 §4.2]:

- Computing the loss in float64 [arxiv:2605.06152 fig. 1a].
- Projecting the logit gradient to zero sum over the classes
  [arxiv:2605.06152 §4.1].
- Placing BatchNorm directly before the classifier, tested on VGG11
  [arxiv:2605.06152 fig. 3c]. A LayerNorm in the same place did not prevent
  the spike [arxiv:2605.06152 fig. 3b].

Raising Adam's epsilon to 1e-5 removed the spikes, and 1e-6 reduced them
[arxiv:2605.06152 §4.2] [arxiv:2605.06152 fig. 3]. Card: [Grokking or
Glitching? How Low-Precision Drives Slingshot Loss Spikes](/papers/2605.06152).
Its results are author-reported on modular division, CIFAR-10 and one nanoGPT
run [arxiv:2605.06152 §4].

### bfloat16 and float16 as the default format in a hardware benchmark

This variant is not a method. It is the number format a benchmark runs in. The
ParaDnn study runs its TPU models with bfloat16 compute and float32
accumulation, and its V100 models with float16 compute and float32 accumulation
[arxiv:1907.10701 §3]. Its real-model batch sizes on TPU are the largest the platform supports with
bfloat16, for example 4k for Transformer [arxiv:1907.10701 §2.2].

The study also measures precision on its own. On TPU, bfloat16 against float32
gives 90th-percentile speedups of up to 1.8x for FC and bottleneck CNN models,
and up to 1.3x for residual CNN models [arxiv:1907.10701 §6.1]. With infeed
removed, bfloat16 speedups are 37% for RetinaNet and 180% for SqueezeNet
[arxiv:1907.10701 §4.4]. The authors note that these bfloat16 results change
memory traffic and batch size together, and that accuracy effects are out of
scope [arxiv:1907.10701 §6.1]. So these numbers should not be read as the
effect of precision alone.

Card: [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](/papers/1907.10701).
Its throughput numbers are author-reported single-device results.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Hopper GPU, throughput is the priority, and FP8 error is acceptable after testing on your own data | FP8 path with the rotation and block scales | Close to 1.2 PFLOPs/s reported [arxiv:2407.08608 §1] [fig. 7]; the error gain is measured on synthetic outliers only [arxiv:2407.08608 §4.3] |
| Accuracy must match the FP16 kernel closely, or the data is not tested for FP8 | FP16 or BF16 path | No format change; 1.5-2.0x over FlashAttention-2 in the authors' FP16 benchmarks [arxiv:2407.08608 §4.1] |
| Activations have large outliers (typical of LLM activations) | Keep the rotation; per-tensor scaling alone is the weaker choice | Without it the synthetic RMSE is back at the baseline level [arxiv:2407.08608 tab. 3] |
| Causal masking, and FP8 compared with cuDNN | Benchmark both before choosing | FP8 is behind cuDNN with a causal mask; without one it is ahead at head dimension 64 and on par at 128 and 256 [arxiv:2407.08608 §1] |
| Short sequences | Benchmark against cuDNN | The paper reports FP8 losing to cuDNN on small sequences [arxiv:2407.08608 §1] [arxiv:2407.08608 §5] |
| GPU other than Hopper | Not covered by this card | The design uses Hopper TMA and WGMMA instructions [arxiv:2407.08608 §2.2]; the card gives no results for other GPUs |
| Serving an LLM with smaller weights and activations, and you can calibrate on your SFT data | FP8 (W8A8) checkpoint recipe | Most benchmark rows within about one point of BF16 [arxiv:2604.12374 tab. 8] |
| Serving an LLM at 4-bit weights and activations, with a few layers allowed to stay higher precision | NVFP4 checkpoint with a sensitivity search | 99.8% median accuracy relative to BF16 and under two hours of calibration [arxiv:2604.12374 §4.2]; no speed figure isolates NVFP4, since the throughput comparisons mix precisions [fig. 1] |
| Pretraining an LLM from scratch in 4-bit | Keep the sensitive layers in BF16 or MXFP8 as in the paper's table | Zero-gradient share grew to about 7% and the paper traced it to underflow [arxiv:2604.12374 §2.2] [arxiv:2604.12374 tab. 3] |
| A recurrent state cache (Mamba or similar) stored in 16 bits | FP16 with stochastic rounding, and measure output length | Plain FP16 added up to 40% completion tokens; stochastic rounding brought it near zero on one benchmark [arxiv:2604.12374 tab. 9] |
| A classifier trained from scratch, 4-bit activations wanted, and a per-layer learned clip is acceptable | PACT-style learned clipping | Within about one point of full precision at 4 bits on the tested networks [arxiv:1805.06085 tab. 1] |
| Gaussian Splatting scene that must be small on disk or over the network, and a small PSNR loss is acceptable | 16-bit storage for positions and codebook entries | 38 MB to 29 MB on Mip-NeRF360 with PSNR 27.17 to 27.10 [arxiv:2406.17074 tab. 4]; no render-time memory gain yet [arxiv:2406.17074 §6.1] |
| Training a detector on tensor-core GPUs, and you want the paper's recipe | 16-bit mixed precision | The paper names it and claims speed and memory gains, but gives no numbers [arxiv:2408.15857 §3.5]; measure on your own setup |
| Greedy LLM outputs must match across GPU counts, GPU types or batch sizes, and the model is a reasoning model with long outputs | FP32 weights and compute, or LayerCast (BF16 storage, FP32 compute) | FP32 removes most of the variation under BF16 [arxiv:2506.09501 tab. 3] [arxiv:2506.09501 tab. 4]; LayerCast matches FP32 on the paper's DeepSeek-R1-Distill-Qwen-7B runs [arxiv:2506.09501 app. F], with 34% lower memory than FP32 reported without a table or stated method [arxiv:2506.09501 §4]. Its released code needs fixing first |
| Serving a large LLM on consumer GPUs, with weights only in 4 bits and FP16 activations, and a small accuracy loss acceptable | INT4 weight-only with row-wise absmax scales | Scores within about 1 point of FP16 on LAMBADA and MMLU, and 4 × RTX 3090 fit the model [arxiv:2210.02414 §4]; the speed figures are INT4 timings with no FP16 run on the same hardware [arxiv:2210.02414 §4] |
| Training a 100B-scale model in FP16 where the usual stability fixes fail | DeepNorm Post-LN with embedding gradient shrink, FP16 with FP32 master weights | The authors' runs stabilised with this combination; Pre-LN, Sandwich-LN and PB-Relax failed [arxiv:2210.02414 §3] [app. A] |
| Same training, and GPU memory is not the limit | BF16 | Avoids FP16 overflow; the authors measured about 15% more memory and no V100 support [arxiv:2210.02414 §3] |
| Writing FP16 GEMM kernels for your own GPU and shape set, with a speed target and a numerical check you control | Search with a speed reward and an error penalty, as CUDA-L2 does, starting from the vendor library as the baseline | Reported 11.4% offline mean over cuBLASLt-AutoTuning-max on A100 [arxiv:2512.02551 tab. 1]; the penalty coefficients are not stated [arxiv:2512.02551 §3.2.3], and the exact-match check ignores outputs at 2048 and above [arxiv:2512.02551 §2.3] |
| Quantizing a model you already trained, with a small unlabelled calibration set and no retraining budget | Post-training quantization | The survey defines PTQ as calibration without training [arxiv:2112.06126 §4.2]; its PTQ rows are reported without a float baseline [arxiv:2112.06126 tab. 4], so measure your own loss |
| Retraining is affordable and the bit width must go low | Quantization-aware training with a straight-through estimator | The survey's QAT route keeps float weights and rounds in the forward pass [arxiv:2112.06126 §4.1]; its minimum QAT precision is 1 bit [arxiv:2112.06126 tab. 1] |
| Weights at 1 or 2 bits, and a loss of a few percent is acceptable | Ternary (TWN, TTQ) or binary (BinaryConnect, XNOR-Net) weights | TWN's ImageNet ResNet18 loss is described as about 4% [arxiv:2112.06126 §6.2]; the table has no float baseline for that row [arxiv:2112.06126 tab. 4] |
| Each layer needs its own bit width, and the search must stay tractable | Sensitivity-guided assignment (HAWQ family) rather than exhaustive search | A search over {1, 2, 4, 8} bits for ResNet50 spans 4^50 choices [arxiv:2112.06126 §7] |
| KV cache dominates memory at long context | Store cache elements in fewer bytes, as in the FP8 KV cache of the checkpoint row above | Cache size is linear in the element size, 2 bytes for FP16 or BF16 [arxiv:2512.23914 eq. 15]; the formula counts a dense cache only [arxiv:2512.23914 §5.2] |
| Deciding whether lower precision will speed up LLM decoding | Check the roofline position first | The survey argues decode is often memory-limited [arxiv:2512.23914 §4.4] and gives the roofline bound [arxiv:2512.23914 eq. 14]; fewer bytes per value helps only when memory traffic limits the kernel |
| Long unregularised training with cross-entropy and Adam shows periodic loss spikes, and you suspect rounding | Compute the loss in float64, or put BatchNorm directly before the classifier | Float64 removed spikes in the authors' MLP, CNN and ViT runs [arxiv:2605.06152 fig. 1a]; BatchNorm removed them on VGG11 [arxiv:2605.06152 fig. 3c]. Both are small setups; test on your own model |
| Cross-entropy with label smoothing | Expect a different spike type | Label smoothing removed the precision-driven spikes but added EOS-type spikes in the authors' runs [arxiv:2605.06152 §4.2] [arxiv:2605.06152 app. A.2] |
| Comparing a benchmark's TPU and GPU throughput, where the TPU runs in bfloat16 and the GPU in float16 | Report the numeric format next to every number, and treat the speedup as a platform-plus-format result | The paper's bfloat16 gains change memory traffic and batch size together [arxiv:1907.10701 §6.1]; its TPU real-model batches are set for bfloat16 [arxiv:1907.10701 §2.2] |

## Implementation sketch

The steps follow the paper's FP8 section [arxiv:2407.08608 §3.3]. The paper
does not give a rule for choosing each block's scale, only that each block has
one scalar, so the rule is left open below.

```pseudocode
# FP8 preprocessing of Q and K  [arxiv:2407.08608 §3.3]
# Input:  Q, K (N, d) in FP16 or BF16
# Output: e4m3 blocks, one scale per block

# Random orthogonal mixing: signs is +1/-1 per dimension, H a Hadamard matrix
M = diag(signs) @ H                   # M M^T = I, so (Q M)(K M)^T = Q K^T
Q_mix = Q @ M                         # can be fused with rotary embedding
K_mix = K @ M

# One scale per block of rows: B_r rows for Q, B_c rows for K  [§3.3]
# Scale rule: not stated in the paper
Q_fp8, Q_scale = quantize_blocks(Q_mix, B_r)
K_fp8, K_scale = quantize_blocks(K_mix, B_c)

# Each block of scores is rescaled by the product of its two scales  [§3.3]
S_block = (Q_fp8 @ K_fp8^T) * Q_scale * K_scale
```

Notes on the sketch:

- The rotation costs O(d log d) per row, which the paper says is cheap enough
  to fuse with the rotary embedding [arxiv:2407.08608 §3.3].
- V needs the in-kernel transpose from the FP8 variant above before the second
  multiply [arxiv:2407.08608 §3.3].
- The paper's FP8 section describes preprocessing, layout and scaling. It
  does not give a separate FP8 version of the attention loop.
- `B_r`, `B_c` and the block-scale rule are not stated in the paper
  [arxiv:2407.08608 §3.1] [arxiv:2407.08608 §3.3].

The NVFP4 checkpoint uses a different scale rule: a search for weight scales
offline and a per-block maximum for activations at runtime [arxiv:2604.12374 §4.2].

```pseudocode
# NVFP4 post-training quantization of one linear layer  [arxiv:2604.12374 §4.2]
# Input:  weight (out, in) in BF16, activations (tokens, in) in BF16
# Output: 4-bit elements with one 8-bit scale per block of 16

# Weights are calibrated once, so an offline scale search is affordable
weight_scales = search_scales_min_mse(weight, block=16)
# Activation scales come from the block maximum, computed at runtime
act_scales = block_max(activations, block=16)
# Scales are stored as FP8 E4M3 under one FP32 scale per tensor  [app. B.1]
```

The integer variant from PACT needs a clip level per layer [arxiv:1805.06085 §4]:

```pseudocode
# Learned clipping and k-bit rounding of one activation tensor  [arxiv:1805.06085 §4]
# Input:  x (activations), alpha (trainable clip, one per layer), k (bits)
# Output: x_q (same shape, on a grid of 2^k levels in [0, alpha])

y = clip(x, 0, alpha)                 # [arxiv:1805.06085 eq. 1]
step = alpha / (2 ** k - 1)           # [arxiv:1805.06085 eq. 2]
x_q = round(y / step) * step          # straight-through estimator in backward
```

The storage variant needs only a cast, applied after training:

```pseudocode
# 16-bit storage of positions and codebook entries  [arxiv:2406.17074 §4.3]
# Input:  positions (N, 3) and codebook centres (256, d), float32
# Output: the same arrays as float16, two bytes per value

positions_f16 = to_float16(positions)
centres_f16 = to_float16(centres)
# The one-byte attribute indices are unchanged  [arxiv:2406.17074 §4.3]
```

## Contradictions

Between papers: no direct conflict. The eight method cards measure different things:
attention error and speed on Hopper, classifier accuracy with learned clipping,
LLM benchmark scores after quantization, the on-disk size of a 3DGS scene, a
training recipe with no numbers, output variation of greedy LLM decoding, kernel
speed against cuBLAS, and LLM perplexity and benchmark scores at INT4 weights.
The hardware benchmark card (arXiv:1907.10701) measures training throughput
across TPU, GPU and CPU, with bfloat16 and float16 as the default formats, so
its speedups are not precision effects alone [arxiv:1907.10701 §6.1].
The two surveys report no measurements of their own, and their tables copy
numbers from works that are not on the wiki. Those tables also mix QAT and PTQ
rows [arxiv:2112.06126 tab. 3] [arxiv:2112.06126 tab. 4], so they are not
compared with the measured rows above.
The papers also choose different layers to
keep at higher precision (PACT keeps the first and last layers in full
precision [arxiv:1805.06085 §4.2]; Nemotron 3 Super keeps the final 15% of
layers in BF16 [arxiv:2604.12374 tab. 3]). These are different models and
settings, so the choices are not in conflict.

Within arXiv:2112.06126: the zero point is subtracted in [arxiv:2112.06126 eq. 5]
and added in [arxiv:2112.06126 eq. 7]. Check which sign your framework uses
before you port a quantizer.

Within one paper (arXiv:2406.17074): the paper says most attributes tolerate
low precision while positions do not [arxiv:2406.17074 §3], yet it stores
positions as 16-bit floats [arxiv:2406.17074 §4.3]. The half-float step is one
row in [arxiv:2406.17074 tab. 4], so its effect on positions alone is not
separated.

Within arXiv:2210.02414: the appendix writes the INT4 integer range as
[-(2^b − 1), 2^b − 1], but defines the scale with 2^(b−1) − 1 [arxiv:2210.02414
app. A]. At 4 bits these give 15 and 7. The INT4 variant above uses 7, the
divisor from the scale formula; the paper's own text is not consistent on this
point.

Paper versus code (the pinned commit is newer than the paper, so the code may
differ from what was evaluated):

- The paper's rotation of Q and K does not appear in the code at this commit.
  The card found no occurrence of "hadamard" or "incoherent" anywhere in the
  repository [arxiv:2407.08608 §3.3].
- The code applies descale values per batch and KV head, not per block
  [code: hopper/mainloop_fwd_sm90_tma_gmma_ws.hpp#L1069-L1071]. The FP8 benchmark
  passes descale values of 1.0, so its timings are valid but say nothing about
  accuracy [code: hopper/benchmark_flash_attention_fp8.py#L282-L284].

So the accuracy claim in the paper cannot be reproduced from the released
code at this commit, and the FP8 speed claim is not checked against accuracy
there. [arxiv:2407.08608 §4.3]

For the Nemotron paper, the public repository has the pretraining recipe comment
that names NVFP4, but not the NVFP4 settings, the Hadamard transform or the
rounding rules, and not the quantization recipes [arxiv:2604.12374 §2.2]
[code: src/nemotron/recipes/super3/stage0_pretrain/test_train.py#L93]. The
numbers in this page come from the paper, not from a reproduced run.

For arXiv:2506.09501, the paper's numbers disagree in one place. The text
gives 2.2% of FP32 MATH500 examples diverging for DeepSeek-R1-Distill-Qwen-7B
[arxiv:2506.09501 §3.2], while the appendix table gives 5.80% for the same
setting [arxiv:2506.09501 app. C]. The paper also says FP32 doubles memory and
inference time relative to BF16, without a table [arxiv:2506.09501 §4].

Paper versus code for that paper (the code may differ from what was evaluated):

- The LayerCast evaluation script imports a module that the repository does
  not contain, so it does not run as shipped [code: evaluation/eval_layercast.py#L23].
  The LayerCast code is in a separate patch file [code: evaluation/patch_layercast.py#L174-L241].
- The patch sets the KV cache to FP32 [code: evaluation/patch_layercast.py#L132].
  The paper names the KV cache as the main place LayerCast saves memory
  [arxiv:2506.09501 §4], so the code does not show where the 34% figure comes from.
- The patch changes the attention backend and enables eager mode, while the
  baseline script does not patch either setting [code: evaluation/patch_layercast.py#L179]
  [code: evaluation/eval_layercast.py#L159-L163]. Differences between LayerCast
  and the BF16 or FP32 baselines therefore mix number format with runtime
  settings, which the paper does not mention.
- The code matches linear layers by name, so the output head may keep FP32
  weights. This is read from the code and not run.

The LayerCast numbers on this page come from the paper, not from a reproduced run.

Within arXiv:2605.06152: the update size at a spike is given as about 50 times
the previous epoch in [arxiv:2605.06152 §3.3] and as more than 40-fold in
[arxiv:2605.06152 §4.1]. Neither figure is a constant in the appendix, so
neither should be used as a fixed number.

Between that paper and earlier work: the paper argues that slingshot spikes
are numerical, while earlier work on the same spikes, Thilak et al.
(arXiv:2206.04817), points at the optimiser, and the paper also compares
against Edge of Stability (Cohen et al.) [arxiv:2605.06152 §1]
[arxiv:2605.06152 §2] [arxiv:2605.06152 app. A.2]. Those papers have no cards
here, so their side is not checked on this page.

## Papers

- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](/papers/2407.08608): FP8 e4m3 attention on Hopper with per-block scales and a random Hadamard rotation of Q and K; reports up to 740 TFLOPs/s in FP16 and close to 1.2 PFLOPs/s in FP8.
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](/papers/1805.06085): learned per-layer clip level for k-bit activations; reports accuracy within about one point of full precision at 4 bits on five networks.
- [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](/papers/2604.12374): NVFP4 pretraining of most linear layers, FP8 and NVFP4 checkpoints, and a stochastic-rounding fix for the Mamba state cache; reports 99.8% median accuracy relative to BF16 for NVFP4.
- [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074): stores positions and codebook entries as 16-bit floats alongside k-means codebooks; reports about 27x smaller scenes on disk.
- [What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector](/papers/2408.15857): a review that names 16-bit mixed-precision training as one part of YOLOv8's recipe, without numbers.
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501): BF16 greedy outputs vary with GPU count, GPU type and batch size; FP32 nearly removes it; LayerCast stores weights in 16 bits and computes in FP32; author-reported results only.
- [CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning](/papers/2512.02551): an LLM trained with staged RL writes FP16 HGEMM kernels; the reward penalises numerical error against an FP32 reference; author-reported offline mean of 11.4% over cuBLASLt-AutoTuning-max on A100.
- [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414): a 130B model trained in FP16 with DeepNorm and an embedding gradient shrink; INT4 weight-only inference with row-wise absmax scales; author-reported LAMBADA 79.47 against 80.21 for FP16.
- [Neural Network Quantization for Efficient Inference: A Survey](/papers/2112.06126): survey of uniform quantization, QAT and PTQ, and integer, binary, ternary and mixed-precision schemes; copies numbers from cited works and runs no experiments.
- [Hardware Acceleration for Neural Networks: A Comprehensive Survey](/papers/2512.23914): survey of accelerators that names lower numeric precision as one lever and gives the KV-cache size formula; runs no experiments.
- [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](/papers/2605.06152): attributes slingshot loss spikes to float32 rounding in softmax cross-entropy; float64 loss, a zero-sum logit gradient or BatchNorm before the classifier removes the spikes in the authors' small runs.
- [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](/papers/1907.10701): training throughput of TPU v2/v3, a V100 and a CPU on generated and real models, with TPU in bfloat16 and GPU in float16; reports bfloat16 against float32 speedups on TPU as author-reported results.