---
kind: concept
slug: llm-serving
title: LLM serving
summary: >-
  System design for serving many concurrent LLM generation requests at high throughput: iteration-level batching, request scheduling, preemption with swap or recompute, and multi-GPU execution. Throughput is largely set by how many requests fit in a batch, so serving engines are built around KV-cache memory; gains appear mainly when serving is memory-bound.
tags: [inference, serving, systems, scheduling, memory-bandwidth, numerics, energy, offloading]
papers: ["2211.10438", "2305.09781", "2309.06180", "2310.03003", "2311.16863", "2312.12456", "2403.14123", "2506.02153", "2506.09501", "2512.23914", "2602.06036", "2610.00223"]
related: [kv-cache-management, speculative-decoding, arithmetic-intensity, weight-quantization, post-training-quantization, low-precision-arithmetic, llm-agent, on-device-inference, activation-sparsity, gpu-power-capping]
status: draft
updated_at: 2026-11-01
---

## Overview

An LLM serving engine takes a stream of requests with unknown output lengths and must decide, every decoding iteration, which requests run together, what to do when memory runs out, and how to spread work over GPUs. The PagedAttention paper argues that serving throughput is limited by batch size, and batch size by KV-cache memory [arxiv:2309.06180 §3]; on a 40 GB A100 serving a 13B model about 65% of memory goes to weights and close to 30% to KV cache [arxiv:2309.06180 §1, fig. 1]. The engine-level answer is iteration-level batching (from Orca, which the paper builds on) combined with a memory manager that lets more requests share the batch [arxiv:2309.06180 §2.3, §4.3].

Batch composition also changes the numbers. Under BF16 with greedy decoding, the same model's accuracy and response length vary with GPU count, GPU type and batch size, and the spread is largest for reasoning models [arxiv:2506.09501 §1, §3.2, tab. 3]. Because the engine chooses the batch, its scheduling decisions can change outputs, not only speed.

The regime matters for which lever helps. An analysis paper reports that peak compute grew about 3.0x per two years, against 1.6x for DRAM bandwidth, so memory traffic limits more workloads [arxiv:2403.14123 §1, fig. 1]. In its CPU profile at batch size 1, GPT-2 decoding has orders of magnitude lower arithmetic intensity than BERT-Base and runs significantly slower at each sequence length [arxiv:2403.14123 §2.2, fig. 3]. Serving engines therefore pay off most when decoding is memory-bound. A hardware survey frames the same split: prefill is compute-heavy and batches well, while decode is often limited by memory traffic for the KV cache [arxiv:2512.23914 §4.4]. A second lever, speculative decoding, cuts the number of target-model passes per token; a recent paper measures it inside two serving engines and finds its speedup shrinks as concurrency rises [arxiv:2602.06036 tab. 3, tab. 12].

This page covers the engine: scheduling, batching, preemption and throughput. How the cache itself is laid out, allocated and shared is on [KV-cache management](/concepts/kv-cache-management). The draft-and-verify method itself is on [speculative decoding](/concepts/speculative-decoding). Serving on a single consumer GPU with CPU memory as overflow is on [on-device inference](/concepts/on-device-inference), and the power-cap lever is on [GPU power capping](/concepts/gpu-power-capping).

Energy is a separate axis from speed. The "From Words to Watts" study runs LLaMA 7B to 65B on V100 and A100 GPUs, with batched generation and energy read from the GPUs, and reports energy per second, per token and per response [arxiv:2310.03003 §3.3]. Its runs use fixed batches of 64 prompts with no continuous batching, so they describe batched generation but not a scheduler that admits requests as they arrive [arxiv:2310.03003 §3.3].

This page draws on twelve papers: [vLLM / PagedAttention](/papers/2309.06180) for the engine, [SpecInfer](/papers/2305.09781) for tree-based speculation inside a serving system, [PowerInfer](/papers/2312.12456) for hybrid CPU-GPU serving on a consumer GPU, [AI and Memory Wall](/papers/2403.14123) for the memory-bound regime, [SmoothQuant](/papers/2211.10438) for 8-bit weights and activations, [Small Language Models are the Future of Agentic AI](/papers/2506.02153) for the cost argument about small models, [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501) for batch and precision effects on outputs, [DFlash](/papers/2602.06036) for speculative decoding in serving, [From Words to Watts](/papers/2310.03003) for batched energy and power capping on data-centre GPUs, [Power Hungry Processing](/papers/2311.16863) for measured energy per task type, and [Noor carbon footprint](/papers/2610.00223) for an energy estimate of inference, and [Hardware Acceleration for Neural Networks: A Comprehensive Survey](/papers/2512.23914) for serving metrics and the KV-cache size formula. It will grow as more serving papers are compiled.

## Variants

The variants below are engine designs described or compared against in the PagedAttention paper and the SpecInfer paper, the two sources here that build serving engines. The PowerInfer paper adds a hybrid CPU-GPU engine for one consumer GPU, which is a different target from the data-centre setting of the first two. The DFlash paper adds a decoding method that it measures inside two serving engines. The analysis paper adds no variant and is used for the regime above; the numerics paper adds a weight-storage method, LayerCast, which it runs inside vLLM; the two carbon papers add only energy and carbon figures, with no serving engine. The SmoothQuant paper adds an 8-bit weight-and-activation method, measured in its own FasterTransformer and PyTorch integrations rather than in an engine. The agent position paper argues for serving small models rather than one large one; it describes no engine. The hardware survey names LPU-style chips built for token generation, but gives no numbers for them [arxiv:2512.23914 §3.5].

### Static reservation engines (baselines)

FasterTransformer (given a custom dynamic-batching scheduler by the authors) and the authors' reimplementation of Orca with three reservation policies: Oracle (output length known), Pow2 (up to 2x over-reservation) and Max (reserve up to 2048 tokens) [arxiv:2309.06180 §6.1]. Orca already batches at iteration level; its limit is that reserved KV memory caps the batch [arxiv:2309.06180 §6.2, fig. 13].

### vLLM engine

A centralized scheduler holds the KV cache manager; GPU workers each run a block engine [arxiv:2309.06180 §4, fig. 4]. Each iteration the scheduler selects requests, allocates any new blocks, concatenates all current inputs (full prompts for requests in prefill, the last token for requests in decode) into one sequence and runs the model [arxiv:2309.06180 §4.3]. Because the model only sees a list of physical block IDs per sequence, requests with different decoding methods (sampling, beam search, shared prefix) can share one batch [arxiv:2309.06180 §4.4]. Decoding algorithms are built from three engine calls: `fork`, `append`, `free` [arxiv:2309.06180 §5.2]. Card: [Efficient Memory Management for Large Language Model Serving with PagedAttention](/papers/2309.06180).

### Preemption by swapping

Scheduling is FCFS; when memory runs out the latest-arrived request is preempted first [arxiv:2309.06180 §4.5]. Eviction is all-or-nothing per sequence, and all sequences of one request (a sequence group, e.g. the beams) are preempted and rescheduled together [arxiv:2309.06180 §4.5]. In swap mode, the group's blocks go to CPU RAM, and the engine admits no new requests until all preempted sequences finish; CPU swap space never exceeds the GPU KV-cache size [arxiv:2309.06180 §4.5].

### Preemption by recomputation

Evicted blocks are dropped. On rescheduling, the prompt and the tokens generated so far are concatenated and processed as one prompt-phase iteration, which is cheaper than the original decode [arxiv:2309.06180 §4.5].

### Distributed (tensor-parallel) execution

Megatron-LM tensor parallelism with attention split across heads. One KV cache manager serves all workers: all use the same block IDs but store only their own heads' KV. The scheduler broadcasts token IDs and block tables each step; workers all-reduce among themselves and return sampled tokens [arxiv:2309.06180 §4.6].

### Speculative decoding inside the serving loop

A drafter proposes a block of tokens in one parallel pass, and the target verifies the whole block in one forward pass, keeping the accepted prefix and its own next token (the bonus token) [arxiv:2602.06036 §3.1, §4.1]. The drafter reads hidden states from a fixed set of target layers. These enter every draft layer as extra keys and values, not as inputs [arxiv:2602.06036 §4.1, app. A.3]. The paper calls the output lossless [arxiv:2602.06036 §1]. Results are author-reported. With the Transformers backend on an H200, the seven-task greedy average for Qwen3-4B is 4.91x, with an average acceptance length of 6.54 [arxiv:2602.06036 tab. 1]. These are not serving-engine numbers. Inside SGLang on one B200, the Qwen3-4B speedup on MATH-500 falls from 4.8x at one request to 2.9x at 32 [arxiv:2602.06036 tab. 3]. Inside vLLM with Qwen3.5-9B on Math500, it falls from 4.0x to 1.9x over the same range [arxiv:2602.06036 tab. 12]. Card: [DFlash](/papers/2602.06036). Method-level detail is on [speculative decoding](/concepts/speculative-decoding).

### Tree-based speculation in a serving system (SpecInfer)

Small draft models propose a token tree for each request rather than one sequence. The paper's evaluation uses one draft model, expanded with configuration `<1, 1, 3, 1, 1, 1, 1, 1>` [arxiv:2305.09781 §6.1]. The LLM then scores every tree node in one decoding step, using a causal mask that follows the tree, so each node sees only its ancestors and itself [arxiv:2305.09781 §3, §4.2]. Under greedy verification, the engine follows the child whose token matches the LLM's prediction at each node, and appends the LLM's own next token when no child matches [arxiv:2305.09781 §4.3, alg. 2]. Wider trees help the match rate: for LLaMA-7B with LLaMA-68M on the CIP dataset, the success rate of verifying a token rises from 70% with the top-1 draft token to 89% with the top-5 under greedy decoding [arxiv:2305.09781 tab. 1]. The paper also describes multi-step speculative sampling for stochastic decoding and merging trees from several draft models [arxiv:2305.09781 §3, §4.3]. The released `inference` branch implements only greedy verification and one draft model, and it caps tree width at 3 where the paper's tables use up to 5 [code: src/runtime/request_manager.cc#L2581] [code: src/runtime/request_manager.cc#L2820-L2822] [code: include/flexflow/batch_config.h#L196]. Results are author-reported. Against incremental-decoding systems (vLLM, HuggingFace TGI, FasterTransformer), per-token latency is 1.5-2.5x lower on one node and 2.4-2.8x lower across two nodes; with OPT-13B and OPT-30B offloaded to CPU memory on one GPU, it is 2.6-3.5x lower than FlexGen [arxiv:2305.09781 §6.2, §6.3]. Card: [SpecInfer](/papers/2305.09781).

### Hot and cold neurons on one consumer GPU (PowerInfer)

Feed-forward neurons are split by how often they fire on profiling text. Frequently active ("hot") neurons stay in GPU memory and the rest ("cold") are computed on the CPU [arxiv:2312.12456 §3.1, §4]. A small per-layer MLP predictor picks the active neurons for each input [arxiv:2312.12456 §5.1], and an offline integer linear program decides which neurons go to the GPU [arxiv:2312.12456 §6.3]. The design depends on sparse activation, and the authors report the largest gains for ReLU-family models [arxiv:2312.12456 §4, §8.4]. Author-reported at batch size 1: up to 11.69x over llama.cpp for Falcon-40B on an RTX 4090, and an average of 7.23x in FP16 across models [arxiv:2312.12456 §8.2, fig. 10]; an average of 4.71x on an RTX 2080 Ti [arxiv:2312.12456 §8.2, fig. 11]. Against vLLM on an A100 at input length 1, the paper reports OPT-30B at 18% slower [arxiv:2312.12456 §8.3.5, fig. 18]. The released code accepts only ReLU, ReGLU and squared ReLU, and its predictor training code is not released [code: README.md#L280] [code: README.md#L298]. Card: [PowerInfer](/papers/2312.12456). This is an offloading design; the cache and placement questions it raises are on [on-device inference](/concepts/on-device-inference).

### Mixed-precision weight storage (LayerCast)

Weights and biases of every linear layer are stored in BF16. Each layer's weight is cast back to FP32 just before its matmul, so the arithmetic runs in FP32 [arxiv:2506.09501 §4]. Across 12 runtime configurations on DeepSeek-R1-Distill-Qwen-7B, its greedy accuracy spread is close to FP32's on all five benchmarks, and it matches FP32 exactly on AIME'24 [arxiv:2506.09501 app. F]. The paper reports 34% less memory than FP32, with no table or method given [arxiv:2506.09501 §4]. The released code is a vLLM patch. Its evaluation script imports a module the repository does not contain [code: evaluation/eval_layercast.py#L23]. The patch keeps the KV cache in FP32 [code: evaluation/patch_layercast.py#L132], so it does not shrink KV memory, and it changes the attention backend relative to the baseline [code: evaluation/patch_layercast.py#L179]. Card: [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501). Results are author-reported.

### INT8 weights and activations (SmoothQuant)

Post-training W8A8 quantization of every linear layer and of the two attention matrix products, with no retraining. A per-input-channel factor divides the activation channel and multiplies the matching weight row, so the product is unchanged but the activation outliers are easier to store in 8 bits [arxiv:2211.10438 §4] [eq. 3]. The migration strength α sets how the difficulty is split; the authors use 0.5 for OPT and BLOOM, 0.75 for GLM-130B and 0.8 for LLaMA [arxiv:2211.10438 §4, §5.1, tab. 6]. Three granularity levels trade accuracy for speed: O1 uses per-token dynamic activation scales, O2 per-tensor dynamic and O3 per-tensor static, all with per-tensor weights [arxiv:2211.10438 tab. 2]. On OPT-175B the three levels stay within 0.5 points of FP16 average accuracy, while naive W8A8 falls to 35.5% from 66.9% [arxiv:2211.10438 tab. 3]. Speed and memory come from the authors' FasterTransformer integration: up to 1.56x over FP16 and about half the memory [arxiv:2211.10438 fig. 9, §5.3]. On MT-NLG 530B the INT8 build runs on 8 GPUs where FP16 needed 16, with 527 GB against 1040 GB at sequence length 128, but its latency there is slightly higher, 253 ms against 232 ms [arxiv:2211.10438 tab. 10]. Card: [SmoothQuant](/papers/2211.10438). Method-level detail is on [post-training quantization](/concepts/post-training-quantization).

### Power-capped data-centre GPUs (measured, not an engine)

Lowering the GPU power limit is a setting, not an engine change. On A100 80 GB GPUs running LLaMA 65B on GSM8K at batch 64, a 175 W cap against the 250 W default gave about 23% less energy for about 7% more time, averaged over three output lengths [arxiv:2310.03003 §4.5, tab. III]. A 150 W cap gave 33% to 35% less energy for 15% to 22% more time, depending on output length [arxiv:2310.03003 tab. III]. The study covers one model, one dataset, one batch size and one GPU type, with one run per cell and no error bars, and the paper says a single static cap may not suit every workload [arxiv:2310.03003 §4.5]. Card: [From Words to Watts](/papers/2310.03003). Method-level detail is on [GPU power capping](/concepts/gpu-power-capping).

## Measuring a serving engine

Report tail latency (p50, p95 and p99), time to first token (TTFT) and mean inter-token latency (ITL) together. TTFT covers queueing, prefill and the first decode step, and is an approximation; ITL is decode time divided by output tokens [arxiv:2512.23914 §5.2] [eq. 2] [eq. 3] [eq. 4]. The survey does not say whether the first generated token counts toward the ITL denominator, so state which convention you use [arxiv:2512.23914 §5.2]. Goodput counts output tokens only from requests that meet a latency objective, divided by a fixed window [arxiv:2512.23914 §5.2] [eq. 6]. The survey gives the objective as an example (TTFT, p99 or both) with no value, and leaves the window and batch sizes unstated [arxiv:2512.23914 §5.2], so each team has to fix and report them.

The PagedAttention paper reports a different metric: normalized latency (end-to-end latency divided by output length), plotted against request rate, with a sustainable request rate read off the curve [arxiv:2309.06180 §6.1]. Its figures are not interchangeable with TTFT, ITL or goodput figures. The SpecInfer paper reports per-token latency against incremental-decoding baselines [arxiv:2305.09781 §6.2]; check how each paper defines its metric before comparing it with the others.

For energy, the "From Words to Watts" study reads GPU energy with NVIDIA DCGM on the rank-0 node, multiplies it by the node count, and divides by run time, output tokens or responses [arxiv:2310.03003 §3.3]. The node scaling assumes every node draws what rank 0 draws, which the paper does not check [arxiv:2310.03003 §3.3]. Use the same definition on both sides before comparing energy figures.

For KV memory planning, the survey gives a dense-cache estimate. Per token, the cache holds keys and values for every layer, KV head and head dimension, at 2 bytes per element for FP16 or BF16. The total is linear in context length and in the number of sequences [arxiv:2512.23914 §5.2] [eq. 15]. It omits paging overhead, fragmentation and quantised caches, so use it only for planning [arxiv:2512.23914 §5.2]. For OPT-13B's shape (40 layers, 40 heads of size 128) this gives 819,200 bytes, about 800 KB, per token, the figure the PagedAttention paper states [arxiv:2309.06180 §3]. The arithmetic is ours, applied to the survey's formula.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Memory-bound serving (long or variable outputs, moderate model size) | vLLM-style engine | 1.7x-2.7x higher sustainable request rate than Orca (Oracle), 2.7x-8x vs Orca (Max), up to 22x vs FasterTransformer on ShareGPT [arxiv:2309.06180 §6.2, fig. 12]; 2.2x/4.3x more batched requests than Orca Oracle/Max [arxiv:2309.06180 §6.2, fig. 13] |
| Compute-bound serving (e.g. OPT-175B on short Alpaca requests) | Gain is smaller | The advantage over Orca (Oracle/Pow2) shrinks [arxiv:2309.06180 §6.2, fig. 12f] |
| Workloads mixing sampling/beam search | Shared-batch engine with block sharing | 1.3x to 2.3x over Orca (Oracle) going from basic sampling to beam width 6 [arxiv:2309.06180 §6.3, fig. 14] |
| Chatbot with long prompts (1024-token cap) | vLLM-style engine | 2x sustainable rate over all three Orca variants; note no KV cache is kept between turns [arxiv:2309.06180 §6.5, fig. 17] |
| Small KV block size | Recompute preemption | Swapping suffers from many small PCIe transfers; recompute cost does not depend on block size [arxiv:2309.06180 §7.3, fig. 19] |
| Large KV block size | Swap preemption | Swapping becomes more efficient than recompute [arxiv:2309.06180 §7.3] |
| Block size 16-64 | Either | Comparable end-to-end performance [arxiv:2309.06180 §7.3, fig. 19] |
| Low batch size, one token per step (memory-bound decoding) | Raise the batch where requests allow it; otherwise cut the bytes moved per token | Decoding runs matrix-vector products with low arithmetic intensity [arxiv:2403.14123 §2.1]; the penalty shrinks for batched decoding and long prompts [arxiv:2403.14123 §2.2, fn.]. Quantization and pruning are discussed as directions, with figures from cited work [arxiv:2403.14123 §3.2] |
| Latency-bound, few concurrent requests, a trained drafter for the target model | Speculative decoding in the engine | About 4x-5x at one request, e.g. 4.8x (Qwen3-4B, MATH-500, SGLang) and 4.0x (Qwen3.5-9B, Math500, vLLM) [arxiv:2602.06036 tab. 3, tab. 12]. The drafter is trained per target; the training recipe is described in [arxiv:2602.06036 §4.2] and [arxiv:2602.06036 app. A.1] |
| Model sharded across GPUs or weights offloaded to CPU memory, few concurrent requests, an off-the-shelf small draft model | Tree-based speculation (SpecInfer-style), if the released greedy-only code is acceptable | Author-reported 1.5-2.5x (one node), 2.4-2.8x (two nodes) and 2.6-3.5x over FlexGen with offloading [arxiv:2305.09781 §6.2, §6.3]. The drafts were LLaMA-68M and OPT-125M from HuggingFace; the paper does not state which pair each experiment used [arxiv:2305.09781 §6.1]. Gains shrink as concurrency grows, and at batch 4 or more width 2 or 3 is best [arxiv:2305.09781 §6.2, §6.4] |
| High concurrency (16 to 32 requests) | Plain batching; speculation gains are smaller | At 32 requests the speedup is 2.9x on SGLang (Qwen3-4B, MATH-500) and 1.9x on vLLM (Qwen3.5-9B, Math500) [arxiv:2602.06036 tab. 3, tab. 12]; the paper says large blocks cost more to verify when compute-bound [arxiv:2602.06036 §5.5.4] |
| Many narrow, repetitive agent calls (e.g. intent recognition, data extraction) | One fine-tuned small model per task, instead of one large model for all calls | Reports 10-30x lower latency, energy and FLOPs for a 7B model than for 70-175B models [arxiv:2506.02153 §3.2]. The paper's conversion procedure suggests 10k-100k examples per task, a rule of thumb cited to outside work [arxiv:2506.02153 §6, S2]. Its replaceable-call shares for three agents (40-70%) are author estimates with no stated method [arxiv:2506.02153 app. B] |
| Estimating energy or carbon of serving | Per-token estimate at batch size 1 | About 26 J per generated token on one A100, with PUE 1.1 and 72,000 tokens per hour [arxiv:2610.00223 §5.1.5]. A prospective estimate, not a measurement. Assumed constant only up to about 512 prompt tokens; batched and quantized serving are not estimated [arxiv:2610.00223 §5.1.5] |
| Measuring energy of batched generation for a large model sharded over several GPUs | Measure with the rank-0 method and read the figures as approximate | Energy per output token for 65B at maximum length 512 is about 3 to 4 J across 8 to 32 V100 shards, read from plots [arxiv:2310.03003 §4.3, fig. 6]. Energy per second rises with shard count [arxiv:2310.03003 §4.2, fig. 4]. Batch 64, with no continuous batching [arxiv:2310.03003 §3.3] |
| Consumer GPU with a model larger than its memory, batch size 1, ReLU-family model | Hot and cold neuron split (PowerInfer-style) | Author-reported 4.71x to 7.23x average over llama.cpp on two desktops and up to 11.69x at peak [arxiv:2312.12456 §8.2, fig. 10, fig. 11]. The gain is smaller for SiLU-family models [arxiv:2312.12456 §4, §8.4]. The predictor training code is not released, so the predictors must be used as published [code: README.md#L298] |
| Data-centre GPUs, energy matters more than latency, and some slack in time | Lower the power cap, then measure on your own workload | 175 W on A100 gave about 23% less energy for about 7% more time on one workload [arxiv:2310.03003 §4.5, tab. III]. One model and one dataset only, with no error bars [arxiv:2310.03003 §4.5] |
| Comparing energy across task types, measured rather than estimated | Per-task figures as a relative guide; scale energy to your own grid | 0.002 kWh per 1,000 text-classification inferences, against 0.047 for text generation and 2.907 for image generation (task means) [arxiv:2311.16863 tab. 2]. Inputs run one at a time, without batching [arxiv:2311.16863 §3.2]. The paper states 297.6 g CO2eq/kWh, but its data imply about 139 g/kWh [arxiv:2311.16863 §3.2, tab. 3] |
| Greedy outputs must match across batch sizes or GPU counts (regression tests, reproducible evaluation) | FP32 for weights and compute | Under BF16 the accuracy std across 12 greedy configurations reaches 9.15% on AIME'24 for DeepSeek-R1-Distill-Qwen-7B; under FP32 it is 0 [arxiv:2506.09501 tab. 3]. The paper says FP32 doubles memory and inference time versus BF16, without a table [arxiv:2506.09501 §4] |
| Same goal, memory is tight | LayerCast (BF16 storage, FP32 compute) | Reported FP32-level greedy spread at 34% less memory [arxiv:2506.09501 app. F, §4]. Requires the authors' patch, and the released script does not run as shipped [code: evaluation/eval_layercast.py#L23] |
| Large model, memory-bound, INT8 matmul kernels available, accuracy must stay near FP16 | W8A8 with smoothing (SmoothQuant); start at O3 and step to a finer level if accuracy drops | Author-reported: up to 1.56x speed-up and about half the memory on FasterTransformer [arxiv:2211.10438 fig. 9]. The coarsest level lost about 0.8 points on BLOOM-176B and 1 point on GLM-130B [arxiv:2211.10438 §5.2]. The paper advises the coarsest level that keeps accuracy and testing on your own model [arxiv:2211.10438 §5.5, §5.2] |
| Expecting a fixed seed to make greedy runs repeatable | Do not rely on it | The paper argues fixed seeds do not ensure determinism under greedy decoding [arxiv:2506.09501 §2.1] |
| Sizing KV memory before choosing a batch size or context length | Dense-cache estimate from eq. 15, then measure on your engine | A rough planning figure: it counts a dense cache only, so paging overhead and fragmentation come on top [arxiv:2512.23914 §5.2] [eq. 15]. Grouped-query attention lowers the KV-head count and with it the cache size [arxiv:2512.23914 §4.4] |
| Reporting a serving result | p50, p95 and p99 latency with TTFT, ITL and goodput, and the goodput objective stated | The survey recommends reporting these together [arxiv:2512.23914 §5.2]; it gives no goodput threshold, so one must be chosen [arxiv:2512.23914 §5.2] |
| Choosing hardware for LLM decoding | Compare memory bandwidth and memory capacity, not peak FLOPS alone | Peak compute grew 3.0x per two years against 1.6x for DRAM bandwidth [arxiv:2403.14123 §1]; the survey argues decode is often limited by memory traffic [arxiv:2512.23914 §1, §4.4] |

## Implementation sketch

```text
waiting = FCFS queue; running = []; swapped = []

loop each iteration:
    # 1. schedule
    if swapped: swap preempted groups back in when blocks free up; admit no new requests
    else: admit from waiting while blocks are available
    for group in running (oldest first):
        while group needs a block and none is free:
            victim = latest-arrived running group          # whole sequence group
            mode == SWAP ? move victim's blocks to CPU, push to swapped
                         : free victim's blocks, push back to waiting (recompute later)
    # 2. build batch
    tokens = concat(full prompt for prefill/recomputed groups, last token for decode groups)
    send tokens + block tables to all tensor-parallel workers
    # 3. run + sample
    workers run model, all-reduce, return sampled tokens
    apply fork/append/free per decoding algorithm; free blocks of finished requests
```

The paper fixes FCFS with latest-first preemption and offers both recovery modes; which mode is the default is not stated [arxiv:2309.06180 §4.5].

## Contradictions

The two core papers point the same way and do not conflict. The PagedAttention paper says throughput follows batch size [arxiv:2309.06180 §3]; the analysis says decoding at batch size 1 is memory-bound [arxiv:2403.14123 §2.2]. Their scopes differ. The analysis profiles one CPU at batch size 1 with 8-bit values and reports no GPU or accelerator profile [arxiv:2403.14123 §2.2, fn.]. The serving results come from A100 GPUs under request traces [arxiv:2309.06180 §6.1]. The analysis gives no ratio at which a device becomes memory-bound [arxiv:2403.14123 §2.1], so measure that on your own hardware. Paper vs code, per the card: at the checked vLLM HEAD (V1 engine), preemption works by recomputation only with no swap-space setting found, beam search is driven from the entrypoint rather than by in-engine block forking, and a PRIORITY scheduling policy exists alongside FCFS [arxiv:2309.06180 §4.5, §5.2]. The headline throughput numbers are against the authors' own Orca reimplementation and a FasterTransformer with a custom scheduler [arxiv:2309.06180 §6.1].

Batch size differs between the energy estimates and the serving results. The Noor estimate assumes batch size 1 [arxiv:2610.00223 §5.1.5], and the measured task-level figures run one input at a time without batching [arxiv:2311.16863 §3.2]. The Words to Watts runs use fixed batches of 64 with no continuous batching, on V100 shards for 65B, and its per-token figures are plotted and approximate [arxiv:2310.03003 §3.3, §4.3, fig. 6]. Those settings differ from the Noor estimate in batch size and hardware, so the energy figures are not comparable. The PagedAttention gains come from larger batches [arxiv:2309.06180 §3]. None of these papers measures continuous batching under arriving requests, so the energy of a request-driven engine remains unmeasured here.

The agent position paper and its counter-view disagree on serving cost. It argues that small models are cheaper to serve per call [arxiv:2506.02153 §3.2]. It also grants that, at scale, centralised LLM serving is cheaper per token than running many small models, and that balancing load across many small endpoints is harder [arxiv:2506.02153 §4.2]. It calls the counter-view valid but case-specific and gives no measurement for either side. Its cost figures are not serving-engine measurements, so they cannot be checked against the PagedAttention or DFlash tables above.

The survey and the PagedAttention paper measure serving differently, as described under Measuring a serving engine, so their numbers cannot be compared directly [arxiv:2512.23914 §5.2] [arxiv:2309.06180 §6.1]. The survey also says that in a naive LLM serving system compute can sit mostly idle during a long decode, about 90% of the time [arxiv:2512.23914 §2.5]. It gives no source or setup for that figure, and the PagedAttention paper measures wasted KV memory, not idle compute [arxiv:2309.06180 §1, fig. 2]. Treat the 90% as unverified.

The DFlash speedups are author-reported and their serving tables use different backends and hardware from the main table, so do not compare them across tables [arxiv:2602.06036 §5]. The repository contains inference code only. Its training recipe, which the speedups depend on, cannot be checked against code [code: dflash/model.py#L179-L324] [arxiv:2602.06036 §4.2]. The PagedAttention and DFlash results are not directly comparable: one measures a memory manager and scheduler, the other a decoding method. The PagedAttention baselines are the authors' own reimplementations [arxiv:2309.06180 §6.1].

The DFlash paper calls its output lossless, meaning it keeps the target model's output distribution [arxiv:2602.06036 §1]. The numerics paper finds that the target's own greedy output under BF16 varies with batch size and GPU count [arxiv:2506.09501 §3.2, tab. 3]. The two claims are not in conflict as stated, because lossless is defined against the target's output. But a speedup or equality check is only meaningful if the baseline and drafted runs share a batch configuration and numerics. The DFlash card does not record whether they did [arxiv:2602.06036 §5]. Check this on your own engine before comparing outputs across runs.

The SpecInfer paper says its greedy verification gives the same output as incremental decoding for all prompts [arxiv:2305.09781 §6.2]. Tree-parallel decoding runs a different batch shape from incremental decoding, and the numerics paper shows that batch shape can change greedy outputs under BF16 [arxiv:2506.09501 §3.2, tab. 3]. The SpecInfer evaluation uses half precision for LLaMA-7B and OPT-30B only, and does not state precision for LLaMA-65B or the offloading runs [arxiv:2305.09781 §6.2]. Its equality claim therefore cannot be checked from the paper alone. The two speculative serving papers also measure different things: SpecInfer compares per-token latency with incremental-decoding systems on A10 GPUs [arxiv:2305.09781 §6.1], and DFlash reports speedups on an H200 with Transformers and on a B200 inside SGLang [arxiv:2602.06036 tab. 1, tab. 3]. Their speedups are not comparable.

The SmoothQuant speed and memory gains come from FasterTransformer and PyTorch integrations that its repository does not include. The repository simulates quantization for perplexity and has no latency benchmark [arxiv:2211.10438 §5.1] [code: smoothquant/fake_quant.py#L103-L108]. Neither that paper nor the PagedAttention paper measures INT8 weights together with a paged KV cache in a serving engine. SmoothQuant says activation quantization could reduce KV-cache memory but does not measure it [arxiv:2211.10438 app. A]. Treat its 1.56x as a kernel-level figure, not an engine-level one.

## Papers

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](/papers/2309.06180): vLLM, an iteration-level serving engine with paged KV cache, FCFS scheduling and swap/recompute preemption.
- [SmoothQuant](/papers/2211.10438): post-training W8A8 quantization with per-channel smoothing of activation outliers; author-reported speed-ups from FasterTransformer, no serving engine.
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](/papers/2305.09781): tree-shaped speculation with small draft models, verified in one LLM pass inside a serving system; author-reported speedups over incremental decoding; released code is greedy-only.
- [AI and Memory Wall](/papers/2403.14123): analysis of the widening gap between compute and memory bandwidth, and why low-batch decoding is memory-bound; no serving engine or algorithm.
- [Small Language Models are the Future of Agentic AI](/papers/2506.02153): position paper arguing small models should handle most narrow agent calls, with a five-step conversion procedure; no experiments of its own, so its figures are cited or estimated.
- [Hardware Acceleration for Neural Networks: A Comprehensive Survey](/papers/2512.23914): survey of accelerators and LLM-serving chips; gives serving metrics, the roofline bound and a dense KV-cache size formula; runs no experiments.
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501): measures how BF16, FP16 and FP32 change greedy outputs across GPU count, GPU type and batch size, and proposes LayerCast, BF16 storage with FP32 compute.
- [DFlash: Block Diffusion for Flash Speculative Decoding](/papers/2602.06036): a small block-diffusion drafter that proposes a block in one pass, with target features as keys and values; author-reported speedups in Transformers and in SGLang and vLLM.
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](/papers/2311.16863): measures energy and CO2eq of 1,000 inferences for 88 models on ten tasks, one input at a time; no serving engine.
- [A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model](/papers/2610.00223): carbon accounting of a model family, including a prospective per-token inference energy estimate at batch size 1; no serving engine.
- [From Words to Watts](/papers/2310.03003): measures tokens per second and GPU energy for LLaMA 7B to 65B on V100 and A100 nodes at batch 64, plus a power-cap test; no serving engine, and no quantisation or continuous batching.
- [PowerInfer](/papers/2312.12456): hybrid CPU-GPU engine that keeps hot FFN neurons on one consumer GPU and computes cold ones on the CPU, with MLP predictors; author-reported speedups over llama.cpp at batch size 1.