Model FLOPs utilization (MFU)

Overview

Teams training large models want one number for how well their system uses the accelerators. The usual number was hardware FLOPs utilization (HFU): the FLOPs a device actually executes divided by its theoretical peak. The PaLM authors identify two problems with HFU 2204.02311 §4.1:

1. The FLOPs executed depend on the system and implementation. Rematerialization (recomputing forward activations in the backward pass instead of keeping them in memory) adds hardware FLOPs. It can still be the right call because it allows larger batches. What matters is time to train, measured in tokens per second, so a system that burns extra FLOPs on recomputation looks better under HFU without being faster 2204.02311 §4.1.
2. Teams count observed hardware FLOPs in different ways. Some use analytical accounting, others hardware performance counters 2204.02311 §4.1.

Model FLOPs utilization (MFU) solves both. The numerator is the measured training throughput in tokens per second. The denominator is the throughput the hardware would reach at peak matmul FLOPs if it did only the forward and backward work the model needs, with no rematerialization 2204.02311 §4.1, app. B. The numerator is a plain measurement. The denominator depends only on the architecture and the published peak FLOPs of the hardware. That makes MFU comparable across systems, compilers and parallelism strategies 2204.02311 §4.1.

The invariant behind every MFU number: count only the FLOPs the model requires, and divide measured tokens/s by peak-FLOPs tokens/s. Recomputation, padding tricks and other implementation choices are not credited.

Variants

MFU with and without the attention term

For a dense decoder-only Transformer with N parameters, the non-attention matmuls cost 6N FLOPs per token. That is 2N in the forward pass and 4N in the backward pass: each matmul does one multiply and one add per input pair, and the backward pass runs two matmuls for each forward one 2204.02311 app. B. Dense self-attention adds 6·L·H·(2·Q·T) = 12·L·H·Q·T FLOPs per token, where L is the number of layers, H the number of heads, Q the head dimension and T the sequence length 2204.02311 app. B. With a total peak matmul throughput of P FLOP/s for the accelerator group:

R = \frac{P}{6N + 12LHQT} \qquad \textrm{MFU} = \frac{\textrm{observed tokens/s}}{R}

2204.02311 app. B

The paper calls the attention term much smaller for large models and includes it for completeness 2204.02311 app. B. It reports both forms, so state which one you use: PaLM 540B scores 45.7% without attention and 46.2% with it. Megatron-Turing NLG 530B scores 29.7% and 30.2% 2204.02311 §4.1, app. B. Card: PaLM: Scaling Language Modeling with Pathways.

Hardware FLOPs utilization (HFU), for contrast

HFU adds the FLOPs spent on rematerialization to the numerator. PaLM 540B uses rematerialization because the larger batch it permits gives higher throughput 2204.02311 app. B. Its analytically computed HFU is 57.8%, against 46.2% MFU 2204.02311 §4.1, tab. 3. Per token, PaLM 540B needs 3.28 TFLOPs of model work (non-attention plus attention) and 4.10 TFLOPs when rematerialization is included. That run recomputes 75% of the non-attention forward FLOPs and all of the attention forward FLOPs. The 8B and 62B models rematerialize only inside self-attention 2204.02311 app. B. By our arithmetic, 238.3K tokens/s × 4.10 TFLOPs / (275 TFLOP/s × 6144) ≈ 57.8%, so the HFU figure is the same throughput divided by the larger per-token count.

FLOPS utilization in ParaDnn (not MFU)

The ParaDnn benchmark runs training on Cloud TPU v2/v3, one V100 and one Skylake CPU, using synthetic fully connected, convolutional and recurrent models and six real models 1907.10701 §1, §2. It reports FLOPS utilization: average achieved FLOPS divided by the platform's peak FLOPS 1907.10701 §4.1. This is not MFU. The paper gives no formula for how it counts FLOPs for the parameterized models 1907.10701 §4.1, so its numbers cannot be recomputed from the paper. Its peaks are the authors' own figures: 125 TFLOPS for the V100, 180 TFLOPS for a TPU v2 board and 420 TFLOPS for a TPU v3 board, the last an estimate 1907.10701 §3, tab. 3, §4.5. Do not compare these values with the MFU figures above.

Reported numbers

All of these MFU figures are computed by the PaLM authors 2204.02311 tab. 3, §4.1, app. B:

The authors attribute PaLM's MFU to its parallelism strategy (no pipeline parallelism, see the card), to XLA TPU compiler optimizations, and to "parallel layers", where attention and MLP read the same input so their input matmuls can be fused 2204.02311 §4.1, §2. The paper does not ablate these factors separately.

ParaDnn reports FLOPS utilization (defined above), not MFU. Its figures use the paper's own accounting:

When to use which

Implementation sketch

inputs:
  N        # parameter count used in 6N (dense decoder; PaLM's convention)
  L, H, Q  # layers, attention heads, head dim
  T        # training sequence length
  P_chip   # peak dense matmul FLOP/s of one accelerator, from the vendor spec
  n_chips
  tokens_per_step = global_batch_sequences * T
  step_time_s     # measured wall-clock per optimizer step, averaged over many steps

flops_per_token = 6*N + 12*L*H*Q*T           # drop the second term for "w/o attention"
peak            = P_chip * n_chips
R               = peak / flops_per_token     # max tokens/s at peak
observed        = tokens_per_step / step_time_s
MFU             = observed / R

# optional, for contrast:
HFU = observed * (flops_per_token + remat_flops_per_token) / peak

Check it against PaLM's worked example: 238.3e3 × 6 × 540e9 / (275e12 × 6144) ≈ 0.457 without attention 2204.02311 app. B. With L = 118, H = 48, Q = 256, T = 2048 2204.02311 tab. 1, §5, the attention term is about 0.036 TFLOPs per token. That gives 3.28 TFLOPs per token in total, matching the paper's compute table, and an MFU of about 46.2% 2204.02311 app. B. Use H·Q, not d_model, in the attention term: for PaLM 540B, 48 × 256 = 12,288 ≠ d_model = 18,432 2204.02311 tab. 1.

Notes:

• Peak FLOPs means the accelerator group's theoretical peak matmul throughput 2204.02311 app. B. The paper uses 275 TFLOP/s per TPU v4 chip and 312 TFLOP/s per A100 2204.02311 app. B. It does not say which numeric precision these peaks assume.
• Throughput is an average over training 2204.02311 §4.1. The paper says its compute-usage figures leave out development work, evictions and other inefficiencies 2204.02311 app. B. Do not read MFU as end-to-end wall-clock efficiency.
• PaLM packs examples into full 2048-token sequences without padding 2204.02311 §5, so every counted token is a real one. The paper does not say how padded batches should be counted.
• Synthetic versus real data. ParaDnn generates its inputs on the host to avoid infeed 1907.10701 §4.4. Its utilization figures are device numbers and miss input stalls. For end-to-end efficiency, use the real data pipeline.

Contradictions

Only PaLM reports MFU, so no two papers give conflicting MFU values. ParaDnn's FLOPS utilization 1907.10701 §4.1 uses different accounting, different hardware and different peak values from PaLM's MFU 2204.02311 app. B, so do not put the two side by side as if they measured the same thing. Two caveats from within PaLM:

• The Megatron-Turing NLG MFU uses that model's reported training step times. The paper notes that these differ from the earlier benchmark numbers in the Megatron-LM repository and in Narayanan et al. (arXiv:2104.04473), so an MFU computed from those benchmarks would not match 2204.02311 §4.1.
• The Gopher figure rests on a step rate obtained by personal communication, and the GPT-3 figure on a per-GPU throughput reported by a third party. Neither can be checked against the original papers 2204.02311 §4.1.

Papers

• PaLM: Scaling Language Modeling with Pathways: defines MFU in contrast to HFU, gives the 6N + 12LHQT formula, and reports 46.2% MFU for a 540B model on 6144 TPU v4 chips.
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning: reports FLOPS utilization (average over peak) for training on TPU, GPU and CPU across a parameterized model sweep. Not MFU; the paper gives no FLOP formula for the parameterized models, and it finds that input infeed can dominate real-model training.