Arithmetic intensity

Overview

Arithmetic intensity is the number of FLOPs an operation performs divided by the number of bytes it loads from or stores to memory 2403.14123 §2.1 2403.14123 eq. 1. It is a property of the workload, not of the chip. The paper's argument is that peak compute has grown much faster than DRAM and interconnect bandwidth, so an operation with low intensity is limited by memory traffic rather than by arithmetic 2403.14123 §1 [arxiv:2403.14123 fig. 1]. The paper calls this the memory wall. It cites Williams et al. (2009) for the intensity definition 2403.14123 §2.1.

The survey of accelerators gives the roofline bound that puts the metric to work. Attainable performance is capped by the lower of peak compute and intensity times peak memory bandwidth 2512.23914 eq. 14. It also states that, for LLM inference, prefill is compute-heavy and decode is limited by memory-bandwidth 2512.23914 §4.4 2512.23914 fig. 4. The survey does not state hardware peak-to-bandwidth ratios, so a cut-off between memory-bound and compute-bound has to come from your own hardware specifications. The TPU benchmark uses the same ratio, FLOPs per byte moved from memory, for its roofline of fully connected and convolutional layers 1907.10701 §4.2.

The paper's central case is autoregressive Transformer decoding. Encoder models process the whole input at once and multiply large matrices. Decoder models generate one token per step, which turns most of their work into matrix-vector products with much lower intensity 2403.14123 §2.1. In profiling, GPT-2 moves orders of magnitude more memory operations than BERT-Base and has orders of magnitude lower intensity, and its latency is significantly longer at each sequence length measured 2403.14123 §2.2 2403.14123 fig. 3.

The four sources are measurements and analyses, not new algorithms. The memory-wall paper runs one profile at batch size 1 on one CPU [arxiv:2403.14123 §2.2]. The accelerator survey runs no experiments. The driving-hardware review adds one measurement study: three end-to-end driving models on four CPU and GPU systems 2411.10291 §V-A. The TPU, GPU and CPU benchmark measures training throughput and FLOPS utilisation on three platforms, using generated models and six real ones 1907.10701 §2 1907.10701 §5. Every number here is an author-reported profile, a trend fit, or a figure cited from other work 2403.14123 §1, fn. 2512.23914 §5.

Variants

Five ways to apply or read the metric. They differ in what is counted and in how far the result can be trusted.

Per-operation intensity (the definition)

Count the FLOPs of one operation and the bytes it reads and writes, then divide. A multiply-add counts as two FLOPs 2403.14123 §2.1. The byte count assumes that local memory can hold both operand matrices, so the result is an upper bound on data reuse, and a kernel that reloads data will have lower intensity than this count gives 2403.14123 §2.1, fn.. Use this form to find which operations in a graph are memory-bound before any profiling.

Whole-model profile at batch size 1 (the paper's measurement)

The paper profiles BERT-Base, BERT-Large and GPT-2 on one Intel Gold 6242 CPU at batch size 1, with 8-bit parameters and activations, across several sequence lengths. It records FLOPs, memory operations, intensity and latency for each 2403.14123 §2.2 2403.14123 fig. 3. The result is specific to that setup. The paper says the decoder penalty may not carry over to larger batches 2403.14123 §2.2, fn.. No accelerator profile is reported in the text.

Intensity versus hardware ridge point

Compare an operation's intensity with the ratio of the hardware's peak compute to its memory bandwidth. The survey's roofline bound 2512.23914 eq. 14 gives this ratio directly: below it, the bound is set by bandwidth; above it, by peak compute. The survey notes that a single peak-FLOP/s figure is misleading for serving, because the decode phase and the prefill phase sit on opposite sides of the bound 2512.23914 §5.1. To place a running workload, compare its achieved FLOP/s with peak as a fraction 2512.23914 eq. 13. The survey gives no ridge values, so take them from your device's datasheet and use the same precision for both sides. The ratio is a rough guide: the paper's own profile is a CPU at 8 bits, which may not match your target.

The only explicit ridge value among these cards is from the TPU benchmark: an inflection point of 75 operations per byte for TPU v2, computed from the paper's stated 180 TFLOPS and 2.4 TB/s 1907.10701 §4.5. It applies to that board. The paper gives no roofline for the V100 GPU or the CPU, so ridge points cannot be compared across platforms 1907.10701 §4.2. The paper also omits the roofline for RNN models, because the TPU profiler reported incorrect memory-bandwidth figures for them 1907.10701 §4.2. A measured intensity therefore depends on the tool that produced it.

Memory-bound share of training time (large batch, accelerator)

The TPU benchmark reports the share of execution time spent in memory-bound operations in models it calls compute-bound: 19.9% for Transformer and 13% for ResNet-50 1907.10701 §4.2. The measurements are training runs on one TPU board, with each platform's largest supported batch size [arxiv:1907.10701 §2.2] 1907.10701 §3. They do not test decoding, and the batch sizes differ from the paper's batch-1 profile.

Per-model values from a driving-hardware review

The review reports one FLOPs-per-byte value for each of three end-to-end driving models: 101.83 for TransFuser, 169.68 for InterFuser and 153.02 for MILE 2411.10291 tab. VI. It also plots the intensity of sample InterFuser layers 2411.10291 fig. 9. The text does not say how the byte count was made, which precision was used or which batch size applied, so these values cannot be recomputed from the paper. Treat them as a rough guide for these models on that setup, and do not rank the models from them alone. The benchmark protocol is not described in the text 2411.10291 §V-A.

When to use which

The sources give conditions under which intensity changes the picture. Use the table to decide whether to measure it on your own system.

None of these papers measures the effect of lower precision on intensity for the workloads in this table, so check the new value on your own hardware and confirm the output quality.

Implementation sketch

The per-operation count and the ridge comparison share one core. Hardware values are inputs you supply; no source gives them for your device.

# Arithmetic intensity of one operation  [arxiv:2403.14123 §2.1] [eq. 1]
# Input:  op with shapes and dtype; batch size B (B = 1 for the
#         memory-wall paper's setting)
# Output: FLOPs per byte moved, and whether it sits below the ridge point

def intensity(op, batch_size, bytes_per_value):
    # Multiply-adds count as two FLOPs each  [arxiv:2403.14123 §2.1]
    flops = count_flops(op, batch_size)
    # Every byte read or written, assuming operands fit in local memory
    # This is an upper bound on reuse  [arxiv:2403.14123 §2.1, fn.]
    mem_bytes = count_bytes(op, batch_size, bytes_per_value)
    # Intensity is FLOPs divided by memory bytes  [arxiv:2403.14123 eq. 1]
    return flops / mem_bytes

def is_memory_bound(op, batch_size, bytes_per_value, peak_flops, bandwidth):
    # Roofline bound: attainable = min(peak, intensity * bandwidth)
    #   [arxiv:2512.23914 eq. 14] [arxiv:1907.10701 §4.2]
    # peak_flops and bandwidth must come from your device, not from any source
    ridge = peak_flops / bandwidth  # FLOPs per byte
    return intensity(op, batch_size, bytes_per_value) < ridge

Contradictions

No direct disagreement. The two LLM-focused sources agree that decoder inference at low batch size is memory-limited 2403.14123 §2.2 2512.23914 §4.4. The memory-wall paper reports the effect for three models on one CPU, while the survey describes it qualitatively and reports no measurements of its own. The memory-wall paper does not compare its intensity figures with other work's measurements. The TPU benchmark's memory-bound shares 1907.10701 §4.2 cover training at large batch on one accelerator and do not test decoding, so they do not conflict with the batch-1 decoder result. The driving review's per-model values 2411.10291 tab. VI are not directly comparable with the memory-wall paper's profile, because the review does not state its byte count, precision or batch size, and its models are not decoder-only language models. The TPU v2 ridge of 75 operations per byte 1907.10701 §4.5 has no counterpart on the other platforms in these sources.

Papers

• AI and Memory Wall: defines the metric in this setting, profiles BERT-Base, BERT-Large and GPT-2 on one CPU, and argues decoder inference is memory-bound at batch size 1.
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning: a training benchmark of generated fully connected, convolutional and recurrent models and six real models on a TPU, a GPU and a CPU. It gives a TPU roofline for fully connected and convolutional layers and the TPU v2 ridge point.
• Moving Forward: A Review of Autonomous Driving Software and Hardware Systems: a review with one benchmark of three driving models on four CPU and GPU systems. It reports a FLOPs-per-byte value for each model, with the method not stated.
• Hardware Acceleration for Neural Networks: A Comprehensive Survey: a survey without experiments. It gives the roofline bound, the prefill and decode split, and the utilisation formula used here.