--- kind: concept slug: arithmetic-intensity title: Arithmetic intensity summary: >- How many floating-point operations an operation performs per byte it moves to or from memory. Low intensity means the chip waits on memory, not on arithmetic. Decoder Transformers at batch size 1 sit far below encoders on this measure, so they run slower despite similar FLOPs. Intensity depends on batch size, prompt length and precision, and a cut-off needs your own hardware numbers. tags: [inference, training, hardware, memory-bandwidth, performance] papers: ["1907.10701", "2403.14123", "2411.10291", "2512.23914"] related: [llm-serving, kv-cache-management, weight-quantization, model-flops-utilization, low-precision-arithmetic, benchmark-validity, depthwise-separable-convolution] status: draft updated_at: 2026-11-01 --- ## Overview Arithmetic intensity is the number of FLOPs an operation performs divided by the number of bytes it loads from or stores to memory [arxiv:2403.14123 §2.1] [arxiv:2403.14123 eq. 1]. It is a property of the workload, not of the chip. The paper's argument is that peak compute has grown much faster than DRAM and interconnect bandwidth, so an operation with low intensity is limited by memory traffic rather than by arithmetic [arxiv:2403.14123 §1] [arxiv:2403.14123 fig. 1]. The paper calls this the memory wall. It cites Williams et al. (2009) for the intensity definition [arxiv:2403.14123 §2.1]. The survey of accelerators gives the roofline bound that puts the metric to work. Attainable performance is capped by the lower of peak compute and intensity times peak memory bandwidth [arxiv:2512.23914 eq. 14]. It also states that, for LLM inference, prefill is compute-heavy and decode is limited by memory-bandwidth [arxiv:2512.23914 §4.4] [arxiv:2512.23914 fig. 4]. The survey does not state hardware peak-to-bandwidth ratios, so a cut-off between memory-bound and compute-bound has to come from your own hardware specifications. The TPU benchmark uses the same ratio, FLOPs per byte moved from memory, for its roofline of fully connected and convolutional layers [arxiv:1907.10701 §4.2]. The paper's central case is autoregressive Transformer decoding. Encoder models process the whole input at once and multiply large matrices. Decoder models generate one token per step, which turns most of their work into matrix-vector products with much lower intensity [arxiv:2403.14123 §2.1]. In profiling, GPT-2 moves orders of magnitude more memory operations than BERT-Base and has orders of magnitude lower intensity, and its latency is significantly longer at each sequence length measured [arxiv:2403.14123 §2.2] [arxiv:2403.14123 fig. 3]. The four sources are measurements and analyses, not new algorithms. The memory-wall paper runs one profile at batch size 1 on one CPU [arxiv:2403.14123 §2.2]. The accelerator survey runs no experiments. The driving-hardware review adds one measurement study: three end-to-end driving models on four CPU and GPU systems [arxiv:2411.10291 §V-A]. The TPU, GPU and CPU benchmark measures training throughput and FLOPS utilisation on three platforms, using generated models and six real ones [arxiv:1907.10701 §2] [arxiv:1907.10701 §5]. Every number here is an author-reported profile, a trend fit, or a figure cited from other work [arxiv:2403.14123 §1, fn.] [arxiv:2512.23914 §5]. ## Variants Five ways to apply or read the metric. They differ in what is counted and in how far the result can be trusted. ### Per-operation intensity (the definition) Count the FLOPs of one operation and the bytes it reads and writes, then divide. A multiply-add counts as two FLOPs [arxiv:2403.14123 §2.1]. The byte count assumes that local memory can hold both operand matrices, so the result is an upper bound on data reuse, and a kernel that reloads data will have lower intensity than this count gives [arxiv:2403.14123 §2.1, fn.]. Use this form to find which operations in a graph are memory-bound before any profiling. ### Whole-model profile at batch size 1 (the paper's measurement) The paper profiles BERT-Base, BERT-Large and GPT-2 on one Intel Gold 6242 CPU at batch size 1, with 8-bit parameters and activations, across several sequence lengths. It records FLOPs, memory operations, intensity and latency for each [arxiv:2403.14123 §2.2] [arxiv:2403.14123 fig. 3]. The result is specific to that setup. The paper says the decoder penalty may not carry over to larger batches [arxiv:2403.14123 §2.2, fn.]. No accelerator profile is reported in the text. ### Intensity versus hardware ridge point Compare an operation's intensity with the ratio of the hardware's peak compute to its memory bandwidth. The survey's roofline bound [arxiv:2512.23914 eq. 14] gives this ratio directly: below it, the bound is set by bandwidth; above it, by peak compute. The survey notes that a single peak-FLOP/s figure is misleading for serving, because the decode phase and the prefill phase sit on opposite sides of the bound [arxiv:2512.23914 §5.1]. To place a running workload, compare its achieved FLOP/s with peak as a fraction [arxiv:2512.23914 eq. 13]. The survey gives no ridge values, so take them from your device's datasheet and use the same precision for both sides. The ratio is a rough guide: the paper's own profile is a CPU at 8 bits, which may not match your target. The only explicit ridge value among these cards is from the TPU benchmark: an inflection point of 75 operations per byte for TPU v2, computed from the paper's stated 180 TFLOPS and 2.4 TB/s [arxiv:1907.10701 §4.5]. It applies to that board. The paper gives no roofline for the V100 GPU or the CPU, so ridge points cannot be compared across platforms [arxiv:1907.10701 §4.2]. The paper also omits the roofline for RNN models, because the TPU profiler reported incorrect memory-bandwidth figures for them [arxiv:1907.10701 §4.2]. A measured intensity therefore depends on the tool that produced it. ### Memory-bound share of training time (large batch, accelerator) The TPU benchmark reports the share of execution time spent in memory-bound operations in models it calls compute-bound: 19.9% for Transformer and 13% for ResNet-50 [arxiv:1907.10701 §4.2]. The measurements are training runs on one TPU board, with each platform's largest supported batch size [arxiv:1907.10701 §2.2] [arxiv:1907.10701 §3]. They do not test decoding, and the batch sizes differ from the paper's batch-1 profile. ### Per-model values from a driving-hardware review The review reports one FLOPs-per-byte value for each of three end-to-end driving models: 101.83 for TransFuser, 169.68 for InterFuser and 153.02 for MILE [arxiv:2411.10291 tab. VI]. It also plots the intensity of sample InterFuser layers [arxiv:2411.10291 fig. 9]. The text does not say how the byte count was made, which precision was used or which batch size applied, so these values cannot be recomputed from the paper. Treat them as a rough guide for these models on that setup, and do not rank the models from them alone. The benchmark protocol is not described in the text [arxiv:2411.10291 §V-A]. ## When to use which The sources give conditions under which intensity changes the picture. Use the table to decide whether to measure it on your own system. | Situation | Use | Why | |---|---|---| | Decoder inference at low batch size, one sequence at a time | Compute intensity per operation first; expect memory-bound behaviour | The paper reports low intensity and long latency for GPT-2 against BERT-Base at batch size 1 [arxiv:2403.14123 §2.2] [arxiv:2403.14123 fig. 3]. The survey places decode in the memory-bandwidth-bound regime [arxiv:2512.23914 §4.4] | | Serving a mix of prefill and decode | Measure each phase separately | The survey splits LLM inference into compute-heavy prefill and memory-limited decode [arxiv:2512.23914 §4.4] [arxiv:2512.23914 fig. 4] | | Batched decoding, or processing a long prompt | Re-measure; do not assume the decoder penalty holds | The paper says the penalty shrinks here because matrix-matrix products dominate [arxiv:2403.14123 §2.2, fn.] | | Encoder or other whole-input workloads | Intensity is less likely to be the bottleneck | Encoders run matrix-matrix products with higher intensity [arxiv:2403.14123 §2.1] | | Training at a large batch on an accelerator | Profile memory-bound operations per model; do not assume a compute-bound model has none | At each platform's largest batch, memory-bound operations take 19.9% of Transformer and 13% of ResNet-50 execution time on TPU [arxiv:1907.10701 §4.2] | | Training or large-batch serving of a decoder | Do not generalise the paper's inference result | The memory-wall paper's profile is batch size 1 inference only [arxiv:2403.14123 §2.2, fn.]. The survey gives no batch-size threshold for this [arxiv:2512.23914 §4.4] | | Choosing hardware for a model family | Compare measured throughput over the model's own batch and size range, not one ridge point | The benchmark finds no single platform wins everywhere, and the winner depends on batch size and model kind [arxiv:1907.10701 §5] | | Choosing between two models with similar FLOPs | Compare intensity and memory operations, not FLOPs | The paper argues FLOP counts alone mislead on run time [arxiv:2403.14123 §2.1] [arxiv:2403.14123 §2.2] | | Comparing driving or embedded perception models on fixed hardware | Measure your own model's intensity; use the review's per-model values only for orientation | The review gives one FLOPs-per-byte value per model, but the byte count, precision and batch size are not stated [arxiv:2411.10291 tab. VI] [arxiv:2411.10291 §V-A] | | Reducing bytes per value (for example, lower precision) | Expect intensity to rise, if FLOPs stay the same | Arithmetic from the definition, not a result any of these papers reports. The memory-wall paper cites quantization to INT4 as reducing footprint and latency in other work [arxiv:2403.14123 §3.2]. The benchmark notes that its bfloat16 and float16 results change memory traffic and batch size together [arxiv:1907.10701 §6.1] | None of these papers measures the effect of lower precision on intensity for the workloads in this table, so check the new value on your own hardware and confirm the output quality. ## Implementation sketch The per-operation count and the ridge comparison share one core. Hardware values are inputs you supply; no source gives them for your device. ```pseudocode # Arithmetic intensity of one operation [arxiv:2403.14123 §2.1] [eq. 1] # Input: op with shapes and dtype; batch size B (B = 1 for the # memory-wall paper's setting) # Output: FLOPs per byte moved, and whether it sits below the ridge point def intensity(op, batch_size, bytes_per_value): # Multiply-adds count as two FLOPs each [arxiv:2403.14123 §2.1] flops = count_flops(op, batch_size) # Every byte read or written, assuming operands fit in local memory # This is an upper bound on reuse [arxiv:2403.14123 §2.1, fn.] mem_bytes = count_bytes(op, batch_size, bytes_per_value) # Intensity is FLOPs divided by memory bytes [arxiv:2403.14123 eq. 1] return flops / mem_bytes def is_memory_bound(op, batch_size, bytes_per_value, peak_flops, bandwidth): # Roofline bound: attainable = min(peak, intensity * bandwidth) # [arxiv:2512.23914 eq. 14] [arxiv:1907.10701 §4.2] # peak_flops and bandwidth must come from your device, not from any source ridge = peak_flops / bandwidth # FLOPs per byte return intensity(op, batch_size, bytes_per_value) < ridge ``` ## Contradictions No direct disagreement. The two LLM-focused sources agree that decoder inference at low batch size is memory-limited [arxiv:2403.14123 §2.2] [arxiv:2512.23914 §4.4]. The memory-wall paper reports the effect for three models on one CPU, while the survey describes it qualitatively and reports no measurements of its own. The memory-wall paper does not compare its intensity figures with other work's measurements. The TPU benchmark's memory-bound shares [arxiv:1907.10701 §4.2] cover training at large batch on one accelerator and do not test decoding, so they do not conflict with the batch-1 decoder result. The driving review's per-model values [arxiv:2411.10291 tab. VI] are not directly comparable with the memory-wall paper's profile, because the review does not state its byte count, precision or batch size, and its models are not decoder-only language models. The TPU v2 ridge of 75 operations per byte [arxiv:1907.10701 §4.5] has no counterpart on the other platforms in these sources. ## Papers - [AI and Memory Wall](/papers/2403.14123): defines the metric in this setting, profiles BERT-Base, BERT-Large and GPT-2 on one CPU, and argues decoder inference is memory-bound at batch size 1. - [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](/papers/1907.10701): a training benchmark of generated fully connected, convolutional and recurrent models and six real models on a TPU, a GPU and a CPU. It gives a TPU roofline for fully connected and convolutional layers and the TPU v2 ridge point. - [Moving Forward: A Review of Autonomous Driving Software and Hardware Systems](/papers/2411.10291): a review with one benchmark of three driving models on four CPU and GPU systems. It reports a FLOPs-per-byte value for each model, with the method not stated. - [Hardware Acceleration for Neural Networks: A Comprehensive Survey](/papers/2512.23914): a survey without experiments. It gives the roofline bound, the prefill and decode split, and the utilisation formula used here.