KV-cache management

Overview

During autoregressive decoding every past token keeps its keys and values for every layer, and this KV cache is what limits how many sequences fit on a GPU. For OPT-13B one token's KV cache is 800 KB, so a 2048-token sequence can need up to 1.6 GB 2309.06180 §3. Systems that store each request's cache as one contiguous tensor sized for its maximum possible length waste memory three ways: slots reserved for tokens not yet generated, internal fragmentation from over-provisioning, and external fragmentation from the allocator. The PagedAttention paper measures that only 20.4%-38.2% of KV memory in such systems holds real token states 2309.06180 §1, fig. 2. Contiguous storage also rules out sharing cache between sequences 2309.06180 §3.

The main idea is to treat the cache like OS virtual memory: logical blocks per sequence, mapped to physical blocks anywhere in a preallocated pool 2309.06180 §4.1, §4.2. This page covers the memory layout, allocation and sharing of the cache. Scheduling, batching and preemption policy live on LLM serving.

A second lever shrinks the cache per token by storing fewer key and value heads. That is set in the model design, not at serving time 2305.13245 §2.2.

A third option is to bound the cache for streams that never end. Options include a sink token plus a rolling window, a plain rolling buffer that overwrites its oldest entries, or eviction with a summary of the dropped tokens. All of them drop context, so they suit work where old detail matters less than a fixed memory budget.

This page draws on ten papers: PagedAttention / vLLM, which changes the cache layout, SpecInfer, which keeps every node of a speculative token tree in one shared cache, GQA, which changes the number of K/V heads, a causal video diffusion model, which keeps a chunk-level cache for streaming generation, Hunyuan-GameCraft-2, which bounds its cache with a sink token and a rolling window, Mistral 7B, which bounds each layer's cache with a fixed-size rolling buffer, HERMES, which evicts tokens from a streaming video model's cache layer by layer, VAR, which uses a plain KV cache across coarse-to-fine image scales, Doc-to-LoRA, which moves a document out of the prompt and into a LoRA adapter written by a hypernetwork, and a hardware acceleration survey, which runs no experiments but gives a dense KV-cache size formula and serving metrics. No paper here tests these levers together. All score-based eviction evidence comes from video models, and the only text-model result for a fixed window, Mistral 7B, has no cache-quality ablation. The page will grow as more KV-cache papers are compiled (e.g. quantized-cache methods).

Variants

Three levers. The first is the runtime layout, sharing and retention of the cache: the paged design and its contiguous baseline, a shared cache for a speculative token tree, a chunk-level cache, a sink-plus-window cache, a plain rolling buffer, layer-aware eviction, and a plain cache for next-scale steps. The second is to keep a document out of the cache altogether and write it into adapter weights. The third is the number of key and value heads each token stores, which is a model choice: grouped-query attention, with multi-query attention as its one-group case. The cache variants come first, then the adapter variant, then GQA.

Contiguous per-request allocation (baseline)

Each request gets one contiguous chunk sized for the longest output it might produce (the paper's Orca/FasterTransformer setting, with buddy allocation assumed for the Orca reimplementation) 2309.06180 §3.1, §6.1. Simple, and attention kernels read memory directly, but it suffers all three kinds of waste and cannot share blocks. Compaction would remove fragmentation but is too costly at KV-cache sizes 2309.06180 §3.1.

Paged KV cache (PagedAttention)

Split each sequence's cache into blocks of B tokens (default 16) 2309.06180 §4.1, §7.2. A block table per sequence maps logical blocks to physical blocks and records how many slots are filled 2309.06180 §4.2. Each GPU worker allocates one contiguous region and carves it into physical blocks; the same is done in CPU RAM for swapped blocks 2309.06180 §4.2. Allocation is on demand: prefill gets just enough blocks for the prompt, and a new block is allocated only when the last one fills, so each sequence wastes at most one partially filled block 2309.06180 §4.3. The paper keeps separate blocks and tables per layer and per head, reporting this performs the same as one block holding all layers and heads 2309.06180 §4.1. Attention is rewritten as a sum over blocks fetched through the table (eq. 4) 2309.06180 §4.1. Card: Efficient Memory Management for Large Language Model Serving with PagedAttention.

Block sharing with reference counts and copy-on-write

Physical blocks carry reference counts. Several logical blocks (from different sequences) can point to one physical block; a writer that finds count > 1 allocates a new block, copies the data, decrements the original's count and writes into the copy 2309.06180 §4.4, fig. 8. Three sharing patterns use this 2309.06180 §4.4:

• Parallel sampling: all samples share the prompt's blocks. Reported saving 6.1%-9.8% of KV memory on Alpaca and 16.2%-30.5% on ShareGPT (OPT-13B) 2309.06180 §6.3, fig. 15.
• Beam search: sharing extends past the prompt and changes as beams are dropped; blocks are freed at count 0, and copy-on-write copies at most one block. Reported saving 37.6%-55.2% (Alpaca) and 44.3%-66.3% (ShareGPT) 2309.06180 §6.3, fig. 15.
• Shared prefix: the provider keeps physical blocks for predefined prefixes (e.g. few-shot instructions); requests map onto them with the last block marked copy-on-write, so prefill covers only the user's own input 2309.06180 §4.4, fig. 10. Reported throughput gains over Orca (Oracle) of 1.67x (1-shot, 80-token prefix) and 3.58x (5-shot, 341-token prefix) on WMT16 en-de with LLaMA-13B 2309.06180 §6.4, fig. 16.

Shared cache for a speculative token tree (SpecInfer)

A small draft model proposes a token tree per request, and the LLM checks all of its nodes in one decoding step 2305.09781 §2, §3. To do that, a depth-first traversal of the tree writes the keys and values of every node into one shared cache, so the branches do not each need their own cache 2305.09781 §4.2 fig. 4. A topology-aware causal mask then lets each node see the verified prefix and its own ancestors, not sibling branches. The paper says this matches incremental decoding exactly 2305.09781 §4.2. The token-tree memory is estimated to be small next to the key-value cache, and one draft model adds under 1% memory per GPU. Both are estimates, not measurements 2305.09781 §5.3. The reported speed-ups are per-token latency against serving systems, 1.5-2.5x on one node and 2.4-2.8x on two nodes 2305.09781 §6.2, and 2.6-3.5x with offloading against FlexGen 2305.09781 §6.3. The authors report them themselves. The released inference branch implements only greedy verification and caps the tree width at 3, while the paper's tables go to width 5 batch_config.h 2305.09781 tab. 2. Card: SpecInfer. The decoding side is on Speculative decoding.

Block size choice

Block size trades kernel efficiency against fragmentation and sharing. Small blocks underuse GPU parallelism and make swap transfers small and slow; large blocks increase internal fragmentation and reduce sharing 2309.06180 §7.2, §7.3. The best range reported was 16-128 on ShareGPT and 16-32 on Alpaca, where sequences are short 2309.06180 §7.2, fig. 18b.

Chunk-level cache for streaming generation

A causal video diffusion model denoises one chunk of latent frames at a time. Each chunk attends to the cached keys and values of the clean chunks before it, and after the chunk is denoised one extra pass at timestep 0 writes its keys and values to the cache 2412.07772 §4.4 alg. 2. Attention inside a chunk is bidirectional, and a block-causal mask stops a chunk from seeing later chunks during training 2412.07772 §4.1. The card describes no block table, sharing or eviction for this cache. The paper's reported result is 1.3 s latency and 9.4 FPS for a 10 s, 120-frame video on one H100, against 208.6 s for CogVideoX-5B 2412.07772 tab. 3. These numbers are author-reported and compare whole video systems, not cache allocators, and the tables give no memory figure for the cache. Card: From Slow Bidirectional to Fast Autoregressive Video Diffusion Models.

Sink token and rolling window (interactive generation)

The interactive game world model in Hunyuan-GameCraft-2 is distilled into a few-step causal generator. Its initial frame is kept for the whole run as a sink token in the KV cache, and local attention runs block-sparse over the preceding blocks 2511.23429 §4.2.3 fig. 6. At inference the cache is a fixed-length rolling window: the sink tokens at the front plus a local window over recent blocks 2511.23429 §4.3. When the user gives a new prompt, the last generated block is recomputed with the new interaction embedding, which updates both the self-attention and cross-attention caches. The paper calls this ReCache 2511.23429 §4.3 fig. 7. Long-video tuning rolls the model out on its own predictions, so the student learns to recover from states it produced itself 2511.23429 §4.2.4. The sink size, window size and block size of the final model are not stated. The ablation compares sink 1 with local 6 against sink 3 with local 9, without giving the unit of "local" 2511.23429 fig. 16. The authors report 16 FPS after FP8 quantization, a different attention kernel, parallel VAE decoding and sequence parallelism. The GPU model and count are not stated 2511.23429 §4.4. They also report semantic drift beyond about 500 frames 2511.23429 §6. Card: Hunyuan-GameCraft-2.

Rolling buffer with a fixed sliding window (Mistral 7B)

Each layer lets a position attend only to the previous W positions, and its cache holds W entries. The key and value for position i go to slot i mod W, so once the text is longer than W the newest entry overwrites the oldest one 2310.06825 §2 fig. 2. Unlike the sink variant, nothing is pinned: a token is dropped as soon as it leaves the window. The paper gives W = 4096 with 32 layers 2310.06825 tab. 1. It says the cache shrinks 8× at 32K tokens with no loss of quality 2310.06825 §2, but no table or ablation backs the quality claim. Information still moves past the window through the stacked layers, which the paper puts at about 131K tokens in theory 2310.06825 §2 fig. 1. Each layer reads only the last W tokens directly, and no accuracy is measured at that length. Prefill can be split into chunks of size W, with attention covering the cache through the window and the chunk through a causal mask 2310.06825 §2 fig. 3. The only speed figure is about 2× faster than vanilla attention for a 16K sequence, with FlashAttention and xFormers kernels 2310.06825 §2. The model also uses grouped-query attention, with 8 key/value heads for 32 query heads 2310.06825 tab. 1. The reference code allocates the full window per layer up front, and its own docstring calls this wasteful. The 8× saving is therefore a property of the paper's analysis, not of the released code cache.py. Card: Mistral 7B. The sketch is in the implementation section below.

Layer-aware eviction with a summary token (streaming understanding)

HERMES is a training-free method for a streaming video MLLM. The authors inspected LLaVA-OV-7B and found three roles by depth: shallow layers attend mostly to the newest tokens, deep layers to about one anchor per 196-token frame, and middle layers to a blend of the two 2601.14724 §2 fig. 2. Each video token gets a score in each layer. Shallow layers use an exponential recency curve, deep layers use attention to a generic guidance query, and middle layers use a depth-weighted blend 2601.14724 §3.1 eq. 1-4. Each layer's score is then blended with the next deeper layer's score eq. 5. The top tokens within a fixed per-layer budget are kept eq. 6. In deep layers the dropped tokens are merged into one summary token whose value is the mean of the pruned values 2601.14724 §3.2 alg. 1. Positions are then renumbered to stay within the trained range 2601.14724 §3.3. The budget is 4K or 6K tokens per layer 2601.14724 §4.2 tab. 1. Author-reported: Qwen2.5-VL-7B on StreamingBench real-time goes from 73.31 to 79.44 at 4K tokens tab. 1. Offline the same model's VideoMME average falls from 64.52 to 60.63 at 4K tab. 4, so the method is not a uniform win. In the LLaVA-OV-7B ablation, cross-layer smoothing matters: without it the VideoMME average is 54.74, against 58.44 with the default weights tab. 6. The forgetting rate and the summary token's target position are not stated eq. 1 alg. 1. Card: HERMES.

Plain KV cache over next-scale steps (VAR)

VAR generates an image as a series of token maps, from 1×1 up to 16×16, each predicted in one parallel step from the coarser maps before it 2404.02905 §3.2. Training uses a block-wise causal mask so that each scale attends only to its prefix. At inference the paper says KV caching can be used and no mask is needed 2404.02905 §3.2. The card records no block table, sharing or eviction for this cache. The paper reports no cache-size or cache-only speed figure. Its time column in tab. 1 is for whole models, relative to VAR-d30 2404.02905 tab. 1. Card: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. Context on the image-side method is on Autoregressive image generation.

Internalize the document into adapter weights (Doc-to-LoRA)

Instead of keeping a document's keys and values in the cache, a hypernetwork reads it once through the frozen model and writes a rank-8 LoRA adapter for the model's down-projections. Later queries need no document in the prompt 2602.15902 §1, §3.2, §5. The paper's memory comparison is for needle-in-a-haystack: a base gemma-2-2b-it needs more than 12 GB of extra memory at 128K tokens, while the internalized model stays below 50 MB 2602.15902 §4. No table backs that figure, and it is a single experiment.

On 2WikiMultihopQA the paper reports an update of 0.209 s batched and 0.551 s iterative, against 40.171 s for oracle context distillation, at normalized performance 0.857 and 0.844 against 0.901 2602.15902 tab. 1. These are the costs of building the adapter, not of serving queries against it. Batched update peak memory is 11.522 GB, above the oracle's 7.820 GB, so the fast update is not free in memory. Iterative mode uses 3.791 GB 2602.15902 tab. 1. The results are author-reported on one base model (gemma-2-2b-it, single H200 GPU) 2602.15902 app. B.2; other base models are in app. E. The method also trails the oracle on MultiFieldQA and QASPER in some settings 2602.15902 tab. 9, tab. 10. Card: Doc-to-LoRA: Learning to Instantly Internalize Contexts. Background on the training target is on Context distillation.

Fewer key/value heads per token (GQA)

This changes the model, not the allocator. Query heads are split into G groups, and each group shares one key head and one value head. The KV cache then shrinks by H/G relative to multi-head attention 2305.13245 §2.2. GQA-1 is multi-query attention and GQA-H is multi-head attention 2305.13245 §2.2. The paged layout still applies unchanged. Each block holds fewer bytes, so the same pool fits more tokens (arithmetic from the shapes, not stated in the paper).

The paper's low-cost route is conversion: mean-pool the K and V projections within each group of an existing multi-head checkpoint, then uptrain for 5% of the original steps 2305.13245 §2.1, §3.1. On T5-XXL, uptrained GQA-8 averages 47.1, against 47.2 for multi-head and 46.6 for multi-query. Inference time per sample is 0.28 s against 1.51 s for multi-head 2305.13245 tab. 1. These are author-reported dev-set numbers, and they come from encoder-decoder models only. Card: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Group choice and the full recipe are on Grouped-query attention.

Sizing the dense cache (planning formula)

The survey estimates the dense cache as layers × 2 × tokens × KV heads × head dimension × bytes per element, with 2 bytes for FP16 or BF16 2512.23914 §5.2 eq. 15. The estimate is linear in the number of concurrent sequences. It counts keys and values only, so it leaves out block-table overhead, fragmentation, quantised caches and other buffers 2512.23914 §5.2. Only the KV-head count changes with grouped-query attention, which is why the formula uses KV heads rather than query heads 2512.23914 §4.4 eq. 15. The survey gives no measured cache size, so use the formula to size a budget and measure the real footprint on your own stack. Card: Hardware Acceleration for Neural Networks: A Comprehensive Survey.

Measuring a cache design

The survey recommends reporting p50, p95 and p99 latency, time to first token, inter-token latency, system throughput and goodput together, with energy and KV footprint alongside 2512.23914 §5 §5.2. Goodput counts only the output tokens of requests that meet a latency objective, over a fixed window 2512.23914 eq. 6. The paper gives no value for that objective or the window, so a comparison of cache designs must state both 2512.23914 §5.2.

When to use which

Implementation sketch

pool        = preallocate N physical blocks of B tokens (per layer, per head) on GPU
cpu_pool    = same on CPU, for swapped-out blocks (size <= GPU pool)
block_table[seq] = list of (physical_block_id, n_filled)
ref_count[block] = 0

on_prefill(seq, prompt):
    block_table[seq] = alloc(ceil(len(prompt) / B))      # only what the prompt needs
    write prompt K/V into those blocks                    # fused reshape+block-write kernel

before_append(seq):                                      # must run BEFORE writing
    last = block_table[seq].last
    if last.n_filled == B:  append alloc(1) to block_table[seq]
    elif ref_count[last.id] > 1:                          # copy-on-write
        new = alloc(1); queue_copy(last.id -> new)       # batch copies into one kernel
        ref_count[last.id] -= 1; replace last with new

fork(parent) -> child:  child.table = copy(parent.table); ref_count[b] += 1 for each b
free(seq):  for b in block_table[seq]: ref_count[b] -= 1; if 0: return b to pool

attention(q, seq):  iterate blocks via block_table[seq], softmax over all blocks (eq. 4)

The paged variants differ only in who shares which blocks: parallel samples fork after the prompt, beams fork and free every step, shared prefixes are pre-filled blocks mapped by many sequences 2309.06180 §4.4, §5.2. A speculative token tree is the one case where several candidate continuations are cached at once. They write to a shared, depth-first-ordered cache, and the mask keeps each branch to its own ancestors 2305.09781 §4.2. The chunk-level cache has no block table. It appends the clean keys and values of each finished chunk, and the next chunk's denoising steps read them 2412.07772 alg. 2. The bounded caches do not share blocks. They keep a fixed set of tokens and drop the rest, which is the source of their memory cap and of the context they lose.

# Shared cache for a speculative token tree  [arxiv:2305.09781 §4.2] [fig. 4]
# Input: verified prefix already cached; tree of candidate tokens
for node in depth_first_order(tree):
    write the node's key and value into the shared cache   # all branches, one cache
# mask: a node attends to the prefix and to its ancestors and itself, not siblings

# Bounded cache with a sink token  [arxiv:2511.23429 §4.2.3, §4.3]
sink = KV of the initial frame             # kept for the whole run; size not stated
window = KV of the most recent blocks      # rolling; block size not stated
on each new block:
    append its KV; drop the oldest non-sink blocks
on a new prompt:
    recompute the last block's KV with the new interaction embedding   # ReCache

# Rolling-buffer cache for one layer  [arxiv:2310.06825 §2] [fig. 2]
cache_k, cache_v = zeros(W, n_kv_heads, head_dim)   # fixed size, allocated up front
on each new token at absolute position i:
    slot = i mod W                 # overwrites the entry from position i - W
    cache_k[slot], cache_v[slot] = key_i, value_i
    attend to the entries in the buffer   # W or W + 1 keys: boundary not settled [fig. 1]

# Layer-aware eviction after each chunk  [arxiv:2601.14724 §3, alg. 1]
for each layer l:
    score each video token                  # recency, attention, or a blend by depth
    blend its score with layer l+1's        # smoothing weight by layer type  [eq. 5]
    keep the top-|M| tokens plus the prefix # |M| is the per-layer budget  [eq. 6]
    if l is a deep layer:
        merge the dropped tokens into one summary token   # mean of pruned values
renumber positions to stay within the trained range       # [§3.3]

# Dense-cache size for planning  [arxiv:2512.23914 §5.2] [eq. 15]
# Estimate only: no paging overhead, fragmentation or quantised cache
bytes_per_element = 2                       # FP16 or BF16
per_token = 2 * kv_heads * head_dim * bytes_per_element   # keys and values, one layer
bytes     = layers * per_token * context_len * sequences  # linear in sequences

# Internalize a document into adapter weights  [arxiv:2602.15902 §3.2]
# The base model is frozen; only the hypernetwork was trained beforehand
for each chunk of the document (at most 8192 tokens [arxiv:2602.15902 §5]):
    hidden = frozen_llm.hidden_states(chunk)        # every layer, one pass
    A, B   = hypernet(hidden)                       # rank r per chunk, per layer
concatenate the chunks' A and B along the rank dimension   # rank r*K  [§3.2]
apply W' = W + alpha_l * B_l A_l to each down-projection   # alpha_l learned [§3.2]
answer queries with no document tokens in the prompt
# The released code adds one more rank block, so the rank is (K+1)*r
#   [code: src/ctx_to_lora/modeling/lora_merger.py#L55-L66]

GQA is not a runtime variant. The allocator sees only a smaller per-token entry. Its conversion step is below.

# Convert a multi-head checkpoint to GQA   [arxiv:2305.13245 §2.1, §3.1]
Input: MHA checkpoint with H heads, group count G, uptraining fraction alpha
for each decoder self-attention and cross-attention layer:  # encoder: unchanged
    partition the H heads into G groups       # assignment not stated
    for each group g:
        W_K[g] = mean of W_K[h] over heads h in group g
        W_V[g] = mean of W_V[h] over heads h in group g
    keep all H query projections and the output projection
continue pre-training for alpha * original steps, same recipe and data
# Cache per token per layer: 2 * G * head_size values, was 2 * H * head_size
#   (arithmetic from the shapes, not stated in the paper)

Contradictions

None known. The sources measure different things. The hardware survey runs no experiments, so its KV-cache formula is a planning estimate and not a measured size; it does not conflict with the measured figures above 2512.23914 §5.2. PagedAttention reports memory and throughput on OPT models 2309.06180 §3, §6.1. GQA reports quality and timing on T5 encoder-decoder models 2305.13245 §3.1. The video diffusion paper reports quality and latency for a causal video model, with no memory measurement for its cache 2412.07772 §5, tab. 3. Hunyuan-GameCraft-2 reports 16 FPS for a rolling-window cache on hardware not stated 2511.23429 §4.4. HERMES reports GPU memory and time-to-first-token for a video MLLM on a single A800 2601.14724 tab. 3. Their speeds and memory figures use different models, tasks and hardware, so they should not be compared. HERMES's offline results are mixed: its gains on LLaVA-OV-7B do not carry over to Qwen2.5-VL-7B on VideoMME 2601.14724 tab. 4. That is a result within one paper, not a conflict with the others. SpecInfer's memory statements are estimates 2305.09781 §5.3, and its speed-ups are measured against vLLM, TGI, FasterTransformer and FlexGen 2305.09781 §6.1, not against a paged cache run on the same tree. It does not contradict PagedAttention's figures, which measure KV memory use 2309.06180 §1, fig. 2. Mistral 7B reports a 2× speed-up at 16K and an 8× cache saving at 32K, both from the authors' own analysis, and its reference cache is preallocated for the full window 2310.06825 §2 cache.py. That gap is between the paper and its code, not between papers. The Doc-to-LoRA memory figures are peak memory for building an adapter, while the PagedAttention figures are KV memory during serving 2602.15902 tab. 1 2309.06180 §1, fig. 2, so the two should not be compared. Doc-to-LoRA's own paper-and-code differences are internal to that paper 2602.15902 Paper vs code. No source tests the levers together, so their combined effect is not shown by any of them.

Papers

• Efficient Memory Management for Large Language Model Serving with PagedAttention: introduces the paged, block-table KV cache with reference-counted copy-on-write sharing.
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification: verifies a speculative token tree in one LLM pass, with all nodes in one shared cache and a topology-aware mask.
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints: grouped-query attention, and converting multi-head checkpoints into it by mean-pooling K/V heads and uptraining.
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models: a chunk-level KV cache for streaming causal video generation, trained by distilling a bidirectional teacher.
• Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model: a distilled causal generator with a sink-token rolling-window KV cache and ReCache on prompt changes.
• Mistral 7B: a 7B language model with grouped-query attention and a fixed-size rolling-buffer KV cache over a sliding window.
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding: training-free, layer-aware eviction with a summary token for streaming video MLLMs.
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction: next-scale image generation whose inference uses KV caching with no attention mask.
• Doc-to-LoRA: Learning to Instantly Internalize Contexts: a hypernetwork that writes a LoRA adapter from a document in one pass, so queries need no context in the prompt.
• Hardware Acceleration for Neural Networks: A Comprehensive Survey: a survey with no experiments; gives the dense KV-cache size formula and the serving metrics to report.