Low-precision arithmetic

Overview

Low-precision arithmetic stores and multiplies numbers with fewer bits than the usual FP16 or BF16, so the matrix units do more work per second and move less data. For the attention kernel on NVIDIA Hopper the relevant format is FP8 in the e4m3 layout, which has few mantissa bits and therefore loses accuracy unless the values are scaled first 2407.08608 §2.2 2407.08608 §3.3.

The invariant the FP8 path tries to keep is that the attention output stays close to the FP16 result. The paper's tools are one scale per block of values, and an orthogonal matrix M applied to Q and K before quantization. Because M M^T = I, (Q M)(K M)^T equals Q K^T in exact arithmetic, so the rotation changes the numbers the quantizer sees without changing the scores 2407.08608 §3.3.

Eight wikillm papers use low precision in eight different ways: low-precision compute in attention 2407.08608, integer activations with a learned clip level 1805.06085 §4, 4-bit floating-point pretraining and inference checkpoints of an LLM 2604.12374 §2.2 2604.12374 §4, 16-bit storage of compressed 3D Gaussian Splatting attributes 2406.17074 §4.3, 16-bit mixed precision as one part of a detector's training recipe 2408.15857 §3.5, BF16 weight storage with FP32 arithmetic to reduce run-to-run variation in LLM inference 2506.09501 §4, FP16 matrix-multiply kernels whose search reward penalises numerical error 2512.02551 §3.2.3, and FP16 training of a 130B model with INT4 weight-only inference 2210.02414 §3 2210.02414 §4. They measure different quantities (classifier accuracy, LLM benchmarks, attention error, file size, output-length variation, kernel speed, perplexity-style scores), so their numbers should not be compared.

Two surveys place these methods in context. The first defines the uniform integer quantizer, separates quantization-aware training from post-training quantization, and surveys binary, ternary and mixed-precision schemes 2112.06126 §3.1 2112.06126 §4 2112.06126 §6. The second lists lowering numeric precision as one of its optimisation levers 2512.23914 §1 and gives a KV-cache size formula in which the bytes per element is a parameter 2512.23914 eq. 15.

One more card, arXiv:2605.06152, looks at the reverse risk: float32 rounding inside the cross-entropy loss is argued to cause loss spikes in long training, so there the precision is the problem to fix. See the variant below the INT4 section.

A last card, arXiv:1907.10701, is a hardware benchmark. It does not study precision as its subject, but its TPU runs use bfloat16 and its GPU runs use float16, and it reports bfloat16 against float32 speedups on TPU. Those numbers mix precision with memory traffic and batch size, so they are read as a caution in the variant below.

Variants

FP16 or BF16 (reference)

The higher-precision path of the same kernel. It is the accuracy reference for the FP8 path and reaches up to 740 TFLOPs/s, about 75% of the H100 theoretical FP16 peak, with 1.5-2.0x speedup over FlashAttention-2 in the authors' benchmarks 2407.08608 §4.1 2407.08608 §1. Card: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision.

FP8 (e4m3) with block quantization and incoherent processing

The paper's FP8 forward pass has four parts 2407.08608 §3.3:

1. V is transposed inside the kernel, because FP8 matrix instructions need a particular operand layout.
2. The accumulator registers of the first matrix multiply are permuted so they can feed the second multiply as an operand.
3. Q, K and V are split into blocks, and each block gets its own scale.
4. Q and K are rotated by the random-sign Hadamard matrix described above before quantization.

Reported speed: FP8 forward reaches close to 1.2 PFLOPs/s 2407.08608 §1 fig. 7. Reported accuracy: on synthetic inputs with rare large outliers, the RMSE against an FP64 reference is 9.1e-3, against 2.4e-2 for a per-tensor-scaled FP8 baseline, which the paper calls 2.6x lower error 2407.08608 tab. 3 2407.08608 §4.3.

The ablation separates the two accuracy tricks on that same test. Removing block quantization moves the RMSE to 9.3e-3, a small change. Removing the rotation brings it back to 2.4e-2, the baseline level 2407.08608 tab. 3. On this test the rotation does most of the work.

Card: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. The code at the pinned commit does not contain the rotation or per-block scales; see Contradictions.

NVFP4 (4-bit float with block scales): pretraining and inference checkpoints

Nemotron 3 Super is a hybrid Mamba-Transformer mixture-of-experts LLM. Most of its linear layers are trained in NVFP4: E2M1 elements, FP8 E4M3 micro-block scales and a second-level FP32 scale 2604.12374 §2.2. The forward and both backward matrix multiplies use NVFP4. Random Hadamard transforms are applied to the inputs of the weight-gradient multiply, and gradients are rounded stochastically 2604.12374 §2.2. Other layers stay higher precision, as listed in 2604.12374 tab. 3: the final 15% of the network, the MTP heads, the latent projections, the QKV and attention projections, and the embeddings are BF16, and the Mamba output projection is MXFP8.

The paper reports that the share of zero-valued weight-gradient elements grew to about 7% of parameters by the end of pretraining, traced to underflow in expert layers 2604.12374 §2.2 fig. 6 fig. 8. Switching the last phase to MXFP8 improved the loss but not downstream accuracy, so the final run stayed in NVFP4 2604.12374 §2.2 fig. 9.

The released checkpoints are post-training quantized from the BF16 model:

• FP8 (W8A8). MoE and Mamba linear layers and the KV cache are FP8; the router stays FP32. Calibration uses 256 SFT samples at 65,536 context 2604.12374 §4.1 2604.12374 tab. 6. On the authors' benchmark table the FP8 scores are within about one point of BF16 on most rows 2604.12374 tab. 8.
• NVFP4 (W4A4). Weight block scales are chosen to minimize weight error, using 16-element blocks; activation block scales are computed from the block maximum at runtime 2604.12374 §4.2 app. B.1. A sensitivity search (AutoQuantize) then promotes selected operators to FP8 or BF16 under an average budget of 4.75 bits 2604.12374 §4.2 app. B.2 2604.12374 tab. 7. The paper reports 512 SFT samples at sequence length 4096, under two hours on one 8-GPU node, and 99.8% median accuracy relative to BF16 2604.12374 §4.2. Per benchmark, the drop is larger on some rows, for example 2.4 points on Scale AI Multi-Challenge 2604.12374 tab. 8.

Card: Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning.

Uniform integer quantization and the two training routes

The general recipe maps a real value r to an integer through a scale S and an integer zero point Z 2112.06126 §3.1 2112.06126 eq. 5. For a clipping range [alpha, beta] and b bits, the step size is (beta - alpha) / (2^b - 1) 2112.06126 §3.1.1 2112.06126 eq. 6. Symmetric quantization sets alpha = -beta and Z = 0. Asymmetric quantization fits an unbalanced range, for example after ReLU [arxiv:2112.06126 §3.1.1]. The survey gives no formula for Z from alpha and beta 2112.06126 §3.1.

The two routes differ in when the rounding is learned:

• Quantization-aware training (QAT) runs the rounding in the forward pass, keeps float weights, and treats the rounding as the identity in the backward pass 2112.06126 §4.1.
• Post-training quantization (PTQ) calibrates the ranges on a small set of unlabelled data and does not retrain 2112.06126 §4.2.

The survey's summary table gives 1 bit as the minimum for QAT, citing XNOR-Net, and 4 bits for PTQ, citing AdaRound 2112.06126 tab. 1. For 8-bit integer inference on ImageNet, the survey says Gong et al. lose less than 1% 2112.06126 §5. Its table gives that PTQ ResNet50 row as 71.84% without a float baseline 2112.06126 tab. 4.

Card: Neural Network Quantization for Efficient Inference: A Survey.

Integer activations with a learned clip level (PACT)

Here the format is a k-bit integer grid, not a float. Each layer's activations are clipped to [0, alpha], where alpha is a trainable parameter, and then rounded to 2^k levels spaced alpha / (2^k - 1) apart 1805.06085 §4 1805.06085 eq. 1 1805.06085 eq. 2. The backward pass uses a straight-through estimator for the rounding, so alpha receives gradient from the loss 1805.06085 eq. 3. The authors add an L2 penalty on alpha to shrink the range during training 1805.06085 §4 1805.06085 app. A.2. Their clip level is one value per layer, and they start it at a large value 1805.06085 §4.2.

Accuracy with 4-bit weights (DoReFa) and 4-bit activations (PACT), trained from scratch: CIFAR10 91.3% against 91.6% full precision, ResNet18 69.2% against 70.2%, and ResNet50 76.5% against 76.9% 1805.06085 tab. 1 1805.06085 app. B. The paper's first and last layers stay at full precision in the main comparison 1805.06085 §4.2. Quantizing them too costs accuracy in a CIFAR10 ResNet20 study: with middle layers at 2 bits, validation error rises from 10.9% to 21.0% when the first and last layers are also 2 bits 1805.06085 tab. 2.

Card: PACT: Parameterized Clipping Activation for Quantized Neural Networks. Its hardware gains come from a model (a 14 nm MAC area estimate and a performance simulator), not measured silicon 1805.06085 §6 1805.06085 fig. 6.

Binary and ternary weights (1 to 2 bits)

BinaryConnect draws each weight stochastically from a hard sigmoid, keeps float master weights that receive the gradients, and uses the binary copies only in the forward and backward passes 2112.06126 §6.1 2112.06126 eq. 11. BinaryNet uses the sign function instead 2112.06126 eq. 13. With binary weights and activations a dot product becomes XNOR and popcount 2112.06126 eq. 14. Binary-weight networks and XNOR-Net scale each binary weight by the mean absolute weight of the matrix 2112.06126 eq. 15.

Ternary schemes use three values. Ternary Weight Networks set a threshold of about 0.7 times the mean absolute weight, a factor the survey says assumes normally distributed weights 2112.06126 §6.2. Trained Ternary Quantization learns separate positive and negative scales [arxiv:2112.06126 §6.2].

On ImageNet with ResNet18, the table lists TWN (2-bit weights) at 61.8% and XNOR-Net (1-bit weights and activations) at 51.2%, with no float baseline for those rows 2112.06126 tab. 4. The text calls TWN's loss on that network small, about 4% 2112.06126 §6.2. On CIFAR-10 the table gives BNN (1-bit weights and activations) at 88.6% 2112.06126 tab. 3.

Card: Neural Network Quantization for Efficient Inference: A Survey.

16-bit float storage of attributes (not compute)

This variant lowers only the stored size. In the Gaussian Splatting compression method, positions and k-means codebook entries are stored as 16-bit floats instead of 32-bit, and the other attributes are stored as one-byte codebook indices 2406.17074 §4.3. The authors report one ablation row for this step on Mip-NeRF360: a 38 MB k-means file with PSNR 27.17 becomes a 29 MB half-float file with PSNR 27.10 2406.17074 tab. 4. The scene is decoded when it is loaded, not in the render loop 2406.17074 §5. The authors say render-time VRAM would fall only once the codebooks and half floats are used by the renderer itself 2406.17074 §6.1, so this variant saves disk and transfer size, not GPU memory at render time.

Card: Reducing the Memory Footprint of 3D Gaussian Splatting.

Recurrent state cache in FP16 (stochastic rounding)

The Mamba state cache of Nemotron 3 Super is FP32 by default. The paper reports that storing it in plain FP16 made generations longer: up to about 40% more completion tokens on LiveCodeBench with W8A8 weights 2604.12374 §4.3 2604.12374 tab. 9. Converting to FP16 with stochastic rounding (Philox, 5 rounds) brought the length change to within about 2% on the same benchmark 2604.12374 tab. 9. The cache study used an early checkpoint and code benchmarks only 2604.12374 tab. 9.

Card: Nemotron 3 Super.

16-bit mixed-precision training (named, not measured)

The YOLOv8 review lists "mixed precision with 16-bit floats" as one of four parts of the training recipe 2408.15857 §3.5. It says mixed precision significantly accelerates training, reduces memory use and enables larger batch sizes 2408.15857 §3.5. The paper gives no numbers for these claims, no list of operations kept in 32-bit, and no batch size, learning rate or epoch count 2408.15857 §3.5 2408.15857 §3.

Card: What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector.

BF16 storage with FP32 compute (LayerCast, for reproducibility)

This variant targets run-to-run variation, not speed. The paper's analysis is that, under BF16 with greedy decoding, the same model, prompt and seed give different accuracy and output lengths when the GPU count, GPU type or batch size changes. Floating-point addition is not associative, and serving changes the order of the sums 2506.09501 §2.2 2506.09501 §3.2. For DeepSeek-R1-Distill-Qwen-7B on AIME'24, the accuracy standard deviation across 12 configurations is 9.15% in BF16 and 0 in FP32 2506.09501 tab. 3. FP32 removes most of the variation, and FP16 falls in between [arxiv:2506.09501 §3.2].

LayerCast keeps the linear-layer weights and biases in BF16. Each layer's weight is converted to FP32 just before its matmul, so the arithmetic runs in FP32 2506.09501 §4. The paper reports memory 34% below FP32, without a table or a stated method 2506.09501 §4. On the same model across five benchmarks, the LayerCast accuracy standard deviation is close to FP32 and below BF16 (AIME'24: 0 for LayerCast and FP32, 0.0544 for BF16, as fractions) 2506.09501 app. F. The paper reports divergence below 3.4% across configurations, shown only in a figure 2506.09501 §4.

# LayerCast for one linear layer  [arxiv:2506.09501 §4]
# Input:  x (tokens, in) in FP32
# Output: y (tokens, out) in FP32

# At rest the weight and bias are 16-bit (the paper says all linear layers)
w_bf16 = to_bfloat16(layer.weight)
b_bf16 = to_bfloat16(layer.bias)
# Just before the matmul, make a temporary FP32 copy
w_fp32 = to_float32(w_bf16)
b_fp32 = to_float32(b_bf16)
y = linear(x, w_fp32, b_fp32)         # the arithmetic is FP32

Card: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. Its released script does not run as shipped, and its memory saving is not explained by the code; see Contradictions.

FP16 matrix multiply found by RL, with a numerical penalty

This variant keeps the format at FP16 and changes the kernel. CUDA-L2 uses a reinforcement-learning loop to write half-precision matrix-multiply (HGEMM) kernels for 1,000 (M, N, K) shapes 2512.02551 §1. The reward is the mean measured speedup over a reference, minus a penalty on the largest element-wise difference from an FP32 CPU result, minus a penalty on code length 2512.02551 eq. 3 2512.02551 §3.2.3. The two penalty coefficients are not stated 2512.02551 §3.2.3. During the search, correctness is an exact-match test on random 0/1 inputs, which only covers outputs below 2048 2512.02551 §2.3. The paper does not state the accumulator precision of its main A100 table 2512.02551 §1 2512.02551 §2.3. The reported offline mean speedup is 11.4% over cuBLASLt-AutoTuning-max and 22.0% over torch.matmul, both author-reported 2512.02551 tab. 1. Card: CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning.

FP16 training at 100B scale with DeepNorm and embedding gradient shrink

The authors trained a 130B-parameter model in FP16 and report that the usual stability fixes did not work for them: Pre-LN, Sandwich-LN and PB-Relax all failed in their runs 2210.02414 §3 2210.02414 app. A. The fix they kept combines DeepNorm Post-LN scaling with a gradient shrink of 0.1 on the embedding layer, which they say wipes out most loss spikes 2210.02414 §2.1 2210.02414 §3. Forward and backward passes are FP16, with FP32 master weights and optimizer states 2210.02414 §3. BF16 avoids the overflow, but the authors measured about 15% more GPU memory with it, and it is not supported on V100 2210.02414 §3. The paper reports three late loss divergences in its final run 2210.02414 §3.

Card: GLM-130B: An Open Bilingual Pre-trained Model.

INT4 weight-only inference with row-wise absmax scales

Only the linear-layer weights are stored in 4 bits. Each output row has its own scale, its absolute maximum divided by 7, and the integers are rounded to the range -7 to 7. Weights are converted back to FP16 at run time, and activations stay FP16 2210.02414 §4 2210.02414 app. A. Embeddings, norms and biases also stay FP16 2210.02414 app. A. The authors say the quantized weights take about 70 GB, which fits 4 × RTX 3090 (24 GB) cards 2210.02414 §4. Against FP16, the reported scores are LAMBADA 79.47 against 80.21, MMLU 44.80 against 44.75 and Pile bits per byte 0.641 against 0.634 2210.02414 §4. For 128 generated tokens on 4 × RTX 3090, encoding takes 0.37 s and decoding 8.16 s, with FasterTransformer 2210.02414 §4. The paper does not quantize activations, because they contain outlier dimensions, reported at up to about 30% of dimensions 2210.02414 §4 2210.02414 app. A. An appendix table gives a zero-point INT4 variant at 80.63% on LAMBADA against 79.47% for absmax, and the authors chose absmax for cost and because it kept performance 2210.02414 app. A.

Card: GLM-130B: An Open Bilingual Pre-trained Model.

Rounding in the loss as a cause of instability (a risk, not a speed-up)

This variant reverses the usual question: float32 rounding is the suspect. The paper's argument is that softmax cross-entropy computes its denominator by log-sum-exp. Once the correct logit leads the others by more than about 16, the other terms are absorbed, and the correct-class gradient rounds to exactly zero 2605.06152 §3.1 2605.06152 eq. 1 2605.06152 eq. 2. The wrong-class gradients stay non-zero, so the classes no longer cancel. The global classifier mean and the feature mean then push each other and grow, which the authors call Numerical Feature Inflation 2605.06152 §3.2 2605.06152 thm. 2. When some margins fall back below the threshold, the correct-class gradient returns after a long quiet period, and Adam's small second moment turns it into a very large step 2605.06152 §3.3.

The paper tests three changes, each of which removed the spikes in its own small runs 2605.06152 §4.1 2605.06152 §4.2:

• Computing the loss in float64 2605.06152 fig. 1a.
• Projecting the logit gradient to zero sum over the classes 2605.06152 §4.1.
• Placing BatchNorm directly before the classifier, tested on VGG11 2605.06152 fig. 3c. A LayerNorm in the same place did not prevent the spike 2605.06152 fig. 3b.

Raising Adam's epsilon to 1e-5 removed the spikes, and 1e-6 reduced them 2605.06152 §4.2 2605.06152 fig. 3. Card: Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes. Its results are author-reported on modular division, CIFAR-10 and one nanoGPT run 2605.06152 §4.

bfloat16 and float16 as the default format in a hardware benchmark

This variant is not a method. It is the number format a benchmark runs in. The ParaDnn study runs its TPU models with bfloat16 compute and float32 accumulation, and its V100 models with float16 compute and float32 accumulation 1907.10701 §3. Its real-model batch sizes on TPU are the largest the platform supports with bfloat16, for example 4k for Transformer 1907.10701 §2.2.

The study also measures precision on its own. On TPU, bfloat16 against float32 gives 90th-percentile speedups of up to 1.8x for FC and bottleneck CNN models, and up to 1.3x for residual CNN models 1907.10701 §6.1. With infeed removed, bfloat16 speedups are 37% for RetinaNet and 180% for SqueezeNet 1907.10701 §4.4. The authors note that these bfloat16 results change memory traffic and batch size together, and that accuracy effects are out of scope 1907.10701 §6.1. So these numbers should not be read as the effect of precision alone.

Card: Benchmarking TPU, GPU, and CPU Platforms for Deep Learning. Its throughput numbers are author-reported single-device results.

When to use which

Implementation sketch

The steps follow the paper's FP8 section 2407.08608 §3.3. The paper does not give a rule for choosing each block's scale, only that each block has one scalar, so the rule is left open below.

# FP8 preprocessing of Q and K  [arxiv:2407.08608 §3.3]
# Input:  Q, K (N, d) in FP16 or BF16
# Output: e4m3 blocks, one scale per block

# Random orthogonal mixing: signs is +1/-1 per dimension, H a Hadamard matrix
M = diag(signs) @ H                   # M M^T = I, so (Q M)(K M)^T = Q K^T
Q_mix = Q @ M                         # can be fused with rotary embedding
K_mix = K @ M

# One scale per block of rows: B_r rows for Q, B_c rows for K  [§3.3]
# Scale rule: not stated in the paper
Q_fp8, Q_scale = quantize_blocks(Q_mix, B_r)
K_fp8, K_scale = quantize_blocks(K_mix, B_c)

# Each block of scores is rescaled by the product of its two scales  [§3.3]
S_block = (Q_fp8 @ K_fp8^T) * Q_scale * K_scale

Notes on the sketch:

• The rotation costs O(d log d) per row, which the paper says is cheap enough to fuse with the rotary embedding 2407.08608 §3.3.
• V needs the in-kernel transpose from the FP8 variant above before the second multiply 2407.08608 §3.3.
• The paper's FP8 section describes preprocessing, layout and scaling. It does not give a separate FP8 version of the attention loop.
• B_r, B_c and the block-scale rule are not stated in the paper 2407.08608 §3.1 2407.08608 §3.3.

The NVFP4 checkpoint uses a different scale rule: a search for weight scales offline and a per-block maximum for activations at runtime 2604.12374 §4.2.

# NVFP4 post-training quantization of one linear layer  [arxiv:2604.12374 §4.2]
# Input:  weight (out, in) in BF16, activations (tokens, in) in BF16
# Output: 4-bit elements with one 8-bit scale per block of 16

# Weights are calibrated once, so an offline scale search is affordable
weight_scales = search_scales_min_mse(weight, block=16)
# Activation scales come from the block maximum, computed at runtime
act_scales = block_max(activations, block=16)
# Scales are stored as FP8 E4M3 under one FP32 scale per tensor  [app. B.1]

The integer variant from PACT needs a clip level per layer 1805.06085 §4:

# Learned clipping and k-bit rounding of one activation tensor  [arxiv:1805.06085 §4]
# Input:  x (activations), alpha (trainable clip, one per layer), k (bits)
# Output: x_q (same shape, on a grid of 2^k levels in [0, alpha])

y = clip(x, 0, alpha)                 # [arxiv:1805.06085 eq. 1]
step = alpha / (2 ** k - 1)           # [arxiv:1805.06085 eq. 2]
x_q = round(y / step) * step          # straight-through estimator in backward

The storage variant needs only a cast, applied after training:

# 16-bit storage of positions and codebook entries  [arxiv:2406.17074 §4.3]
# Input:  positions (N, 3) and codebook centres (256, d), float32
# Output: the same arrays as float16, two bytes per value

positions_f16 = to_float16(positions)
centres_f16 = to_float16(centres)
# The one-byte attribute indices are unchanged  [arxiv:2406.17074 §4.3]

Contradictions

Between papers: no direct conflict. The eight method cards measure different things: attention error and speed on Hopper, classifier accuracy with learned clipping, LLM benchmark scores after quantization, the on-disk size of a 3DGS scene, a training recipe with no numbers, output variation of greedy LLM decoding, kernel speed against cuBLAS, and LLM perplexity and benchmark scores at INT4 weights. The hardware benchmark card (arXiv:1907.10701) measures training throughput across TPU, GPU and CPU, with bfloat16 and float16 as the default formats, so its speedups are not precision effects alone 1907.10701 §6.1. The two surveys report no measurements of their own, and their tables copy numbers from works that are not on the wiki. Those tables also mix QAT and PTQ rows 2112.06126 tab. 3 2112.06126 tab. 4, so they are not compared with the measured rows above. The papers also choose different layers to keep at higher precision (PACT keeps the first and last layers in full precision 1805.06085 §4.2; Nemotron 3 Super keeps the final 15% of layers in BF16 2604.12374 tab. 3). These are different models and settings, so the choices are not in conflict.

Within arXiv:2112.06126: the zero point is subtracted in 2112.06126 eq. 5 and added in 2112.06126 eq. 7. Check which sign your framework uses before you port a quantizer.

Within one paper (arXiv:2406.17074): the paper says most attributes tolerate low precision while positions do not 2406.17074 §3, yet it stores positions as 16-bit floats 2406.17074 §4.3. The half-float step is one row in 2406.17074 tab. 4, so its effect on positions alone is not separated.

Within arXiv:2210.02414: the appendix writes the INT4 integer range as [-(2^b − 1), 2^b − 1], but defines the scale with 2^(b−1) − 1 [arxiv:2210.02414 app. A]. At 4 bits these give 15 and 7. The INT4 variant above uses 7, the divisor from the scale formula; the paper's own text is not consistent on this point.

Paper versus code (the pinned commit is newer than the paper, so the code may differ from what was evaluated):

• The paper's rotation of Q and K does not appear in the code at this commit. The card found no occurrence of "hadamard" or "incoherent" anywhere in the repository 2407.08608 §3.3.
• The code applies descale values per batch and KV head, not per block mainloop_fwd_sm90_tma_gmma_ws.hpp. The FP8 benchmark passes descale values of 1.0, so its timings are valid but say nothing about accuracy benchmark_flash_attention_fp8.py.

So the accuracy claim in the paper cannot be reproduced from the released code at this commit, and the FP8 speed claim is not checked against accuracy there. 2407.08608 §4.3

For the Nemotron paper, the public repository has the pretraining recipe comment that names NVFP4, but not the NVFP4 settings, the Hadamard transform or the rounding rules, and not the quantization recipes 2604.12374 §2.2 test_train.py. The numbers in this page come from the paper, not from a reproduced run.

For arXiv:2506.09501, the paper's numbers disagree in one place. The text gives 2.2% of FP32 MATH500 examples diverging for DeepSeek-R1-Distill-Qwen-7B 2506.09501 §3.2, while the appendix table gives 5.80% for the same setting 2506.09501 app. C. The paper also says FP32 doubles memory and inference time relative to BF16, without a table 2506.09501 §4.

Paper versus code for that paper (the code may differ from what was evaluated):

• The LayerCast evaluation script imports a module that the repository does not contain, so it does not run as shipped eval_layercast.py. The LayerCast code is in a separate patch file patch_layercast.py.
• The patch sets the KV cache to FP32 patch_layercast.py. The paper names the KV cache as the main place LayerCast saves memory 2506.09501 §4, so the code does not show where the 34% figure comes from.
• The patch changes the attention backend and enables eager mode, while the baseline script does not patch either setting patch_layercast.py eval_layercast.py. Differences between LayerCast and the BF16 or FP32 baselines therefore mix number format with runtime settings, which the paper does not mention.
• The code matches linear layers by name, so the output head may keep FP32 weights. This is read from the code and not run.

The LayerCast numbers on this page come from the paper, not from a reproduced run.

Within arXiv:2605.06152: the update size at a spike is given as about 50 times the previous epoch in 2605.06152 §3.3 and as more than 40-fold in 2605.06152 §4.1. Neither figure is a constant in the appendix, so neither should be used as a fixed number.

Between that paper and earlier work: the paper argues that slingshot spikes are numerical, while earlier work on the same spikes, Thilak et al. (arXiv:2206.04817), points at the optimiser, and the paper also compares against Edge of Stability (Cohen et al.) 2605.06152 §1 2605.06152 §2 2605.06152 app. A.2. Those papers have no cards here, so their side is not checked on this page.

Papers

• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision: FP8 e4m3 attention on Hopper with per-block scales and a random Hadamard rotation of Q and K; reports up to 740 TFLOPs/s in FP16 and close to 1.2 PFLOPs/s in FP8.
• PACT: Parameterized Clipping Activation for Quantized Neural Networks: learned per-layer clip level for k-bit activations; reports accuracy within about one point of full precision at 4 bits on five networks.
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning: NVFP4 pretraining of most linear layers, FP8 and NVFP4 checkpoints, and a stochastic-rounding fix for the Mamba state cache; reports 99.8% median accuracy relative to BF16 for NVFP4.
• Reducing the Memory Footprint of 3D Gaussian Splatting: stores positions and codebook entries as 16-bit floats alongside k-means codebooks; reports about 27x smaller scenes on disk.
• What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector: a review that names 16-bit mixed-precision training as one part of YOLOv8's recipe, without numbers.
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference: BF16 greedy outputs vary with GPU count, GPU type and batch size; FP32 nearly removes it; LayerCast stores weights in 16 bits and computes in FP32; author-reported results only.
• CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning: an LLM trained with staged RL writes FP16 HGEMM kernels; the reward penalises numerical error against an FP32 reference; author-reported offline mean of 11.4% over cuBLASLt-AutoTuning-max on A100.
• GLM-130B: An Open Bilingual Pre-trained Model: a 130B model trained in FP16 with DeepNorm and an embedding gradient shrink; INT4 weight-only inference with row-wise absmax scales; author-reported LAMBADA 79.47 against 80.21 for FP16.
• Neural Network Quantization for Efficient Inference: A Survey: survey of uniform quantization, QAT and PTQ, and integer, binary, ternary and mixed-precision schemes; copies numbers from cited works and runs no experiments.
• Hardware Acceleration for Neural Networks: A Comprehensive Survey: survey of accelerators that names lower numeric precision as one lever and gives the KV-cache size formula; runs no experiments.
• Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes: attributes slingshot loss spikes to float32 rounding in softmax cross-entropy; float64 loss, a zero-sum logit gradient or BatchNorm before the classifier removes the spikes in the authors' small runs.
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning: training throughput of TPU v2/v3, a V100 and a CPU on generated and real models, with TPU in bfloat16 and GPU in float16; reports bfloat16 against float32 speedups on TPU as author-reported results.