LLM serving

Overview

An LLM serving engine takes a stream of requests with unknown output lengths and must decide, every decoding iteration, which requests run together, what to do when memory runs out, and how to spread work over GPUs. The PagedAttention paper argues that serving throughput is limited by batch size, and batch size by KV-cache memory 2309.06180 §3; on a 40 GB A100 serving a 13B model about 65% of memory goes to weights and close to 30% to KV cache 2309.06180 §1, fig. 1. The engine-level answer is iteration-level batching (from Orca, which the paper builds on) combined with a memory manager that lets more requests share the batch 2309.06180 §2.3, §4.3.

Batch composition also changes the numbers. Under BF16 with greedy decoding, the same model's accuracy and response length vary with GPU count, GPU type and batch size, and the spread is largest for reasoning models 2506.09501 §1, §3.2, tab. 3. Because the engine chooses the batch, its scheduling decisions can change outputs, not only speed.

The regime matters for which lever helps. An analysis paper reports that peak compute grew about 3.0x per two years, against 1.6x for DRAM bandwidth, so memory traffic limits more workloads 2403.14123 §1, fig. 1. In its CPU profile at batch size 1, GPT-2 decoding has orders of magnitude lower arithmetic intensity than BERT-Base and runs significantly slower at each sequence length 2403.14123 §2.2, fig. 3. Serving engines therefore pay off most when decoding is memory-bound. A hardware survey frames the same split: prefill is compute-heavy and batches well, while decode is often limited by memory traffic for the KV cache 2512.23914 §4.4. A second lever, speculative decoding, cuts the number of target-model passes per token; a recent paper measures it inside two serving engines and finds its speedup shrinks as concurrency rises 2602.06036 tab. 3, tab. 12.

This page covers the engine: scheduling, batching, preemption and throughput. How the cache itself is laid out, allocated and shared is on KV-cache management. The draft-and-verify method itself is on speculative decoding. Serving on a single consumer GPU with CPU memory as overflow is on on-device inference, and the power-cap lever is on GPU power capping.

Energy is a separate axis from speed. The "From Words to Watts" study runs LLaMA 7B to 65B on V100 and A100 GPUs, with batched generation and energy read from the GPUs, and reports energy per second, per token and per response 2310.03003 §3.3. Its runs use fixed batches of 64 prompts with no continuous batching, so they describe batched generation but not a scheduler that admits requests as they arrive 2310.03003 §3.3.

This page draws on twelve papers: vLLM / PagedAttention for the engine, SpecInfer for tree-based speculation inside a serving system, PowerInfer for hybrid CPU-GPU serving on a consumer GPU, AI and Memory Wall for the memory-bound regime, SmoothQuant for 8-bit weights and activations, Small Language Models are the Future of Agentic AI for the cost argument about small models, Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference for batch and precision effects on outputs, DFlash for speculative decoding in serving, From Words to Watts for batched energy and power capping on data-centre GPUs, Power Hungry Processing for measured energy per task type, and Noor carbon footprint for an energy estimate of inference, and Hardware Acceleration for Neural Networks: A Comprehensive Survey for serving metrics and the KV-cache size formula. It will grow as more serving papers are compiled.

Variants

The variants below are engine designs described or compared against in the PagedAttention paper and the SpecInfer paper, the two sources here that build serving engines. The PowerInfer paper adds a hybrid CPU-GPU engine for one consumer GPU, which is a different target from the data-centre setting of the first two. The DFlash paper adds a decoding method that it measures inside two serving engines. The analysis paper adds no variant and is used for the regime above; the numerics paper adds a weight-storage method, LayerCast, which it runs inside vLLM; the two carbon papers add only energy and carbon figures, with no serving engine. The SmoothQuant paper adds an 8-bit weight-and-activation method, measured in its own FasterTransformer and PyTorch integrations rather than in an engine. The agent position paper argues for serving small models rather than one large one; it describes no engine. The hardware survey names LPU-style chips built for token generation, but gives no numbers for them 2512.23914 §3.5.

Static reservation engines (baselines)

FasterTransformer (given a custom dynamic-batching scheduler by the authors) and the authors' reimplementation of Orca with three reservation policies: Oracle (output length known), Pow2 (up to 2x over-reservation) and Max (reserve up to 2048 tokens) 2309.06180 §6.1. Orca already batches at iteration level; its limit is that reserved KV memory caps the batch 2309.06180 §6.2, fig. 13.

vLLM engine

A centralized scheduler holds the KV cache manager; GPU workers each run a block engine 2309.06180 §4, fig. 4. Each iteration the scheduler selects requests, allocates any new blocks, concatenates all current inputs (full prompts for requests in prefill, the last token for requests in decode) into one sequence and runs the model 2309.06180 §4.3. Because the model only sees a list of physical block IDs per sequence, requests with different decoding methods (sampling, beam search, shared prefix) can share one batch 2309.06180 §4.4. Decoding algorithms are built from three engine calls: fork, append, free 2309.06180 §5.2. Card: Efficient Memory Management for Large Language Model Serving with PagedAttention.

Preemption by swapping

Scheduling is FCFS; when memory runs out the latest-arrived request is preempted first 2309.06180 §4.5. Eviction is all-or-nothing per sequence, and all sequences of one request (a sequence group, e.g. the beams) are preempted and rescheduled together 2309.06180 §4.5. In swap mode, the group's blocks go to CPU RAM, and the engine admits no new requests until all preempted sequences finish; CPU swap space never exceeds the GPU KV-cache size 2309.06180 §4.5.

Preemption by recomputation

Evicted blocks are dropped. On rescheduling, the prompt and the tokens generated so far are concatenated and processed as one prompt-phase iteration, which is cheaper than the original decode 2309.06180 §4.5.

Distributed (tensor-parallel) execution

Megatron-LM tensor parallelism with attention split across heads. One KV cache manager serves all workers: all use the same block IDs but store only their own heads' KV. The scheduler broadcasts token IDs and block tables each step; workers all-reduce among themselves and return sampled tokens 2309.06180 §4.6.

Speculative decoding inside the serving loop

A drafter proposes a block of tokens in one parallel pass, and the target verifies the whole block in one forward pass, keeping the accepted prefix and its own next token (the bonus token) 2602.06036 §3.1, §4.1. The drafter reads hidden states from a fixed set of target layers. These enter every draft layer as extra keys and values, not as inputs 2602.06036 §4.1, app. A.3. The paper calls the output lossless 2602.06036 §1. Results are author-reported. With the Transformers backend on an H200, the seven-task greedy average for Qwen3-4B is 4.91x, with an average acceptance length of 6.54 2602.06036 tab. 1. These are not serving-engine numbers. Inside SGLang on one B200, the Qwen3-4B speedup on MATH-500 falls from 4.8x at one request to 2.9x at 32 2602.06036 tab. 3. Inside vLLM with Qwen3.5-9B on Math500, it falls from 4.0x to 1.9x over the same range 2602.06036 tab. 12. Card: DFlash. Method-level detail is on speculative decoding.

Tree-based speculation in a serving system (SpecInfer)

Small draft models propose a token tree for each request rather than one sequence. The paper's evaluation uses one draft model, expanded with configuration <1, 1, 3, 1, 1, 1, 1, 1> 2305.09781 §6.1. The LLM then scores every tree node in one decoding step, using a causal mask that follows the tree, so each node sees only its ancestors and itself 2305.09781 §3, §4.2. Under greedy verification, the engine follows the child whose token matches the LLM's prediction at each node, and appends the LLM's own next token when no child matches 2305.09781 §4.3, alg. 2. Wider trees help the match rate: for LLaMA-7B with LLaMA-68M on the CIP dataset, the success rate of verifying a token rises from 70% with the top-1 draft token to 89% with the top-5 under greedy decoding 2305.09781 tab. 1. The paper also describes multi-step speculative sampling for stochastic decoding and merging trees from several draft models 2305.09781 §3, §4.3. The released inference branch implements only greedy verification and one draft model, and it caps tree width at 3 where the paper's tables use up to 5 request_manager.cc request_manager.cc batch_config.h. Results are author-reported. Against incremental-decoding systems (vLLM, HuggingFace TGI, FasterTransformer), per-token latency is 1.5-2.5x lower on one node and 2.4-2.8x lower across two nodes; with OPT-13B and OPT-30B offloaded to CPU memory on one GPU, it is 2.6-3.5x lower than FlexGen 2305.09781 §6.2, §6.3. Card: SpecInfer.

Hot and cold neurons on one consumer GPU (PowerInfer)

Feed-forward neurons are split by how often they fire on profiling text. Frequently active ("hot") neurons stay in GPU memory and the rest ("cold") are computed on the CPU 2312.12456 §3.1, §4. A small per-layer MLP predictor picks the active neurons for each input 2312.12456 §5.1, and an offline integer linear program decides which neurons go to the GPU 2312.12456 §6.3. The design depends on sparse activation, and the authors report the largest gains for ReLU-family models 2312.12456 §4, §8.4. Author-reported at batch size 1: up to 11.69x over llama.cpp for Falcon-40B on an RTX 4090, and an average of 7.23x in FP16 across models 2312.12456 §8.2, fig. 10; an average of 4.71x on an RTX 2080 Ti 2312.12456 §8.2, fig. 11. Against vLLM on an A100 at input length 1, the paper reports OPT-30B at 18% slower 2312.12456 §8.3.5, fig. 18. The released code accepts only ReLU, ReGLU and squared ReLU, and its predictor training code is not released README.md README.md. Card: PowerInfer. This is an offloading design; the cache and placement questions it raises are on on-device inference.

Mixed-precision weight storage (LayerCast)

Weights and biases of every linear layer are stored in BF16. Each layer's weight is cast back to FP32 just before its matmul, so the arithmetic runs in FP32 2506.09501 §4. Across 12 runtime configurations on DeepSeek-R1-Distill-Qwen-7B, its greedy accuracy spread is close to FP32's on all five benchmarks, and it matches FP32 exactly on AIME'24 2506.09501 app. F. The paper reports 34% less memory than FP32, with no table or method given 2506.09501 §4. The released code is a vLLM patch. Its evaluation script imports a module the repository does not contain eval_layercast.py. The patch keeps the KV cache in FP32 patch_layercast.py, so it does not shrink KV memory, and it changes the attention backend relative to the baseline patch_layercast.py. Card: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. Results are author-reported.

INT8 weights and activations (SmoothQuant)

Post-training W8A8 quantization of every linear layer and of the two attention matrix products, with no retraining. A per-input-channel factor divides the activation channel and multiplies the matching weight row, so the product is unchanged but the activation outliers are easier to store in 8 bits 2211.10438 §4 eq. 3. The migration strength α sets how the difficulty is split; the authors use 0.5 for OPT and BLOOM, 0.75 for GLM-130B and 0.8 for LLaMA 2211.10438 §4, §5.1, tab. 6. Three granularity levels trade accuracy for speed: O1 uses per-token dynamic activation scales, O2 per-tensor dynamic and O3 per-tensor static, all with per-tensor weights 2211.10438 tab. 2. On OPT-175B the three levels stay within 0.5 points of FP16 average accuracy, while naive W8A8 falls to 35.5% from 66.9% 2211.10438 tab. 3. Speed and memory come from the authors' FasterTransformer integration: up to 1.56x over FP16 and about half the memory 2211.10438 fig. 9, §5.3. On MT-NLG 530B the INT8 build runs on 8 GPUs where FP16 needed 16, with 527 GB against 1040 GB at sequence length 128, but its latency there is slightly higher, 253 ms against 232 ms 2211.10438 tab. 10. Card: SmoothQuant. Method-level detail is on post-training quantization.

Power-capped data-centre GPUs (measured, not an engine)

Lowering the GPU power limit is a setting, not an engine change. On A100 80 GB GPUs running LLaMA 65B on GSM8K at batch 64, a 175 W cap against the 250 W default gave about 23% less energy for about 7% more time, averaged over three output lengths 2310.03003 §4.5, tab. III. A 150 W cap gave 33% to 35% less energy for 15% to 22% more time, depending on output length 2310.03003 tab. III. The study covers one model, one dataset, one batch size and one GPU type, with one run per cell and no error bars, and the paper says a single static cap may not suit every workload 2310.03003 §4.5. Card: From Words to Watts. Method-level detail is on GPU power capping.

Measuring a serving engine

Report tail latency (p50, p95 and p99), time to first token (TTFT) and mean inter-token latency (ITL) together. TTFT covers queueing, prefill and the first decode step, and is an approximation; ITL is decode time divided by output tokens 2512.23914 §5.2 eq. 2 eq. 3 eq. 4. The survey does not say whether the first generated token counts toward the ITL denominator, so state which convention you use 2512.23914 §5.2. Goodput counts output tokens only from requests that meet a latency objective, divided by a fixed window 2512.23914 §5.2 eq. 6. The survey gives the objective as an example (TTFT, p99 or both) with no value, and leaves the window and batch sizes unstated 2512.23914 §5.2, so each team has to fix and report them.

The PagedAttention paper reports a different metric: normalized latency (end-to-end latency divided by output length), plotted against request rate, with a sustainable request rate read off the curve 2309.06180 §6.1. Its figures are not interchangeable with TTFT, ITL or goodput figures. The SpecInfer paper reports per-token latency against incremental-decoding baselines 2305.09781 §6.2; check how each paper defines its metric before comparing it with the others.

For energy, the "From Words to Watts" study reads GPU energy with NVIDIA DCGM on the rank-0 node, multiplies it by the node count, and divides by run time, output tokens or responses 2310.03003 §3.3. The node scaling assumes every node draws what rank 0 draws, which the paper does not check 2310.03003 §3.3. Use the same definition on both sides before comparing energy figures.

For KV memory planning, the survey gives a dense-cache estimate. Per token, the cache holds keys and values for every layer, KV head and head dimension, at 2 bytes per element for FP16 or BF16. The total is linear in context length and in the number of sequences 2512.23914 §5.2 eq. 15. It omits paging overhead, fragmentation and quantised caches, so use it only for planning 2512.23914 §5.2. For OPT-13B's shape (40 layers, 40 heads of size 128) this gives 819,200 bytes, about 800 KB, per token, the figure the PagedAttention paper states 2309.06180 §3. The arithmetic is ours, applied to the survey's formula.

When to use which

Implementation sketch

waiting = FCFS queue; running = []; swapped = []

loop each iteration:
    # 1. schedule
    if swapped: swap preempted groups back in when blocks free up; admit no new requests
    else: admit from waiting while blocks are available
    for group in running (oldest first):
        while group needs a block and none is free:
            victim = latest-arrived running group          # whole sequence group
            mode == SWAP ? move victim's blocks to CPU, push to swapped
                         : free victim's blocks, push back to waiting (recompute later)
    # 2. build batch
    tokens = concat(full prompt for prefill/recomputed groups, last token for decode groups)
    send tokens + block tables to all tensor-parallel workers
    # 3. run + sample
    workers run model, all-reduce, return sampled tokens
    apply fork/append/free per decoding algorithm; free blocks of finished requests

The paper fixes FCFS with latest-first preemption and offers both recovery modes; which mode is the default is not stated 2309.06180 §4.5.

Contradictions

The two core papers point the same way and do not conflict. The PagedAttention paper says throughput follows batch size 2309.06180 §3; the analysis says decoding at batch size 1 is memory-bound 2403.14123 §2.2. Their scopes differ. The analysis profiles one CPU at batch size 1 with 8-bit values and reports no GPU or accelerator profile 2403.14123 §2.2, fn.. The serving results come from A100 GPUs under request traces 2309.06180 §6.1. The analysis gives no ratio at which a device becomes memory-bound 2403.14123 §2.1, so measure that on your own hardware. Paper vs code, per the card: at the checked vLLM HEAD (V1 engine), preemption works by recomputation only with no swap-space setting found, beam search is driven from the entrypoint rather than by in-engine block forking, and a PRIORITY scheduling policy exists alongside FCFS 2309.06180 §4.5, §5.2. The headline throughput numbers are against the authors' own Orca reimplementation and a FasterTransformer with a custom scheduler 2309.06180 §6.1.

Batch size differs between the energy estimates and the serving results. The Noor estimate assumes batch size 1 2610.00223 §5.1.5, and the measured task-level figures run one input at a time without batching 2311.16863 §3.2. The Words to Watts runs use fixed batches of 64 with no continuous batching, on V100 shards for 65B, and its per-token figures are plotted and approximate 2310.03003 §3.3, §4.3, fig. 6. Those settings differ from the Noor estimate in batch size and hardware, so the energy figures are not comparable. The PagedAttention gains come from larger batches 2309.06180 §3. None of these papers measures continuous batching under arriving requests, so the energy of a request-driven engine remains unmeasured here.

The agent position paper and its counter-view disagree on serving cost. It argues that small models are cheaper to serve per call 2506.02153 §3.2. It also grants that, at scale, centralised LLM serving is cheaper per token than running many small models, and that balancing load across many small endpoints is harder 2506.02153 §4.2. It calls the counter-view valid but case-specific and gives no measurement for either side. Its cost figures are not serving-engine measurements, so they cannot be checked against the PagedAttention or DFlash tables above.

The survey and the PagedAttention paper measure serving differently, as described under Measuring a serving engine, so their numbers cannot be compared directly 2512.23914 §5.2 2309.06180 §6.1. The survey also says that in a naive LLM serving system compute can sit mostly idle during a long decode, about 90% of the time 2512.23914 §2.5. It gives no source or setup for that figure, and the PagedAttention paper measures wasted KV memory, not idle compute 2309.06180 §1, fig. 2. Treat the 90% as unverified.

The DFlash speedups are author-reported and their serving tables use different backends and hardware from the main table, so do not compare them across tables 2602.06036 §5. The repository contains inference code only. Its training recipe, which the speedups depend on, cannot be checked against code model.py 2602.06036 §4.2. The PagedAttention and DFlash results are not directly comparable: one measures a memory manager and scheduler, the other a decoding method. The PagedAttention baselines are the authors' own reimplementations 2309.06180 §6.1.

The DFlash paper calls its output lossless, meaning it keeps the target model's output distribution 2602.06036 §1. The numerics paper finds that the target's own greedy output under BF16 varies with batch size and GPU count 2506.09501 §3.2, tab. 3. The two claims are not in conflict as stated, because lossless is defined against the target's output. But a speedup or equality check is only meaningful if the baseline and drafted runs share a batch configuration and numerics. The DFlash card does not record whether they did 2602.06036 §5. Check this on your own engine before comparing outputs across runs.

The SpecInfer paper says its greedy verification gives the same output as incremental decoding for all prompts 2305.09781 §6.2. Tree-parallel decoding runs a different batch shape from incremental decoding, and the numerics paper shows that batch shape can change greedy outputs under BF16 2506.09501 §3.2, tab. 3. The SpecInfer evaluation uses half precision for LLaMA-7B and OPT-30B only, and does not state precision for LLaMA-65B or the offloading runs 2305.09781 §6.2. Its equality claim therefore cannot be checked from the paper alone. The two speculative serving papers also measure different things: SpecInfer compares per-token latency with incremental-decoding systems on A10 GPUs 2305.09781 §6.1, and DFlash reports speedups on an H200 with Transformers and on a B200 inside SGLang 2602.06036 tab. 1, tab. 3. Their speedups are not comparable.

The SmoothQuant speed and memory gains come from FasterTransformer and PyTorch integrations that its repository does not include. The repository simulates quantization for perplexity and has no latency benchmark 2211.10438 §5.1 fake_quant.py. Neither that paper nor the PagedAttention paper measures INT8 weights together with a paged KV cache in a serving engine. SmoothQuant says activation quantization could reduce KV-cache memory but does not measure it 2211.10438 app. A. Treat its 1.56x as a kernel-level figure, not an engine-level one.

Papers

• Efficient Memory Management for Large Language Model Serving with PagedAttention: vLLM, an iteration-level serving engine with paged KV cache, FCFS scheduling and swap/recompute preemption.
• SmoothQuant: post-training W8A8 quantization with per-channel smoothing of activation outliers; author-reported speed-ups from FasterTransformer, no serving engine.
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification: tree-shaped speculation with small draft models, verified in one LLM pass inside a serving system; author-reported speedups over incremental decoding; released code is greedy-only.
• AI and Memory Wall: analysis of the widening gap between compute and memory bandwidth, and why low-batch decoding is memory-bound; no serving engine or algorithm.
• Small Language Models are the Future of Agentic AI: position paper arguing small models should handle most narrow agent calls, with a five-step conversion procedure; no experiments of its own, so its figures are cited or estimated.
• Hardware Acceleration for Neural Networks: A Comprehensive Survey: survey of accelerators and LLM-serving chips; gives serving metrics, the roofline bound and a dense KV-cache size formula; runs no experiments.
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference: measures how BF16, FP16 and FP32 change greedy outputs across GPU count, GPU type and batch size, and proposes LayerCast, BF16 storage with FP32 compute.
• DFlash: Block Diffusion for Flash Speculative Decoding: a small block-diffusion drafter that proposes a block in one pass, with target features as keys and values; author-reported speedups in Transformers and in SGLang and vLLM.
• Power Hungry Processing: Watts Driving the Cost of AI Deployment?: measures energy and CO2eq of 1,000 inferences for 88 models on ten tasks, one input at a time; no serving engine.
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model: carbon accounting of a model family, including a prospective per-token inference energy estimate at batch size 1; no serving engine.
• From Words to Watts: measures tokens per second and GPU energy for LLaMA 7B to 65B on V100 and A100 nodes at batch 64, plus a power-cap test; no serving engine, and no quantisation or continuous batching.
• PowerInfer: hybrid CPU-GPU engine that keeps hot FFN neurons on one consumer GPU and computes cold ones on the CPU, with MLP predictors; author-reported speedups over llama.cpp at batch size 1.