{"hits":[{"concepts":["kv-cache-management","token-pruning","positional-encoding","multimodal-language-model"],"id":"2601.14724","kind":"paper","path":"papers/2601.14724.md","published":"2026-01-21","score":46.71929168701172,"status":"draft","summary":"Training-free streaming video method for MLLMs. It splits the KV cache into three memory roles by layer depth: shallow layers favour recency, deep layers attention to frame anchors, middle layers blend both. Deep layers fold evicted tokens into one summary token. Author-reported: Qwen2.5-VL-7B at 4K tokens reaches 79.44% on StreamingBench real-time (base 73.31%).","tags":["streaming-video","kv-cache","token-eviction","training-free","video-llm"],"title":"HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding"},{"concepts":["llm-serving","grouped-query-attention","multi-query-attention","diffusion-models","token-pruning","multimodal-language-model","autoregressive-image-generation","context-distillation","hypernetwork","low-rank-adaptation","sliding-window-attention","arithmetic-intensity","low-precision-arithmetic","speculative-decoding","decoding-strategies"],"id":"kv-cache-management","kind":"concept","path":"concepts/kv-cache-management.md","score":46.71929168701172,"status":"draft","summary":"How to lay out, allocate, share and bound the attention key/value cache during generation. Paging it into fixed-size blocks behind per-sequence block tables removes most fragmentation and enables copy-on-write sharing, at the cost of slower kernels. Sink-plus-window and eviction schemes cap stream caches by dropping context. Grouped-query attention shrinks it per token but changes the model.","tags":["inference","memory-management","attention","serving","streaming"],"title":"KV-cache management"},{"concepts":["speculative-decoding","llm-serving","kv-cache-management","decoding-strategies"],"id":"2305.09781","kind":"paper","path":"papers/2305.09781.md","published":"2023-05-16","score":20.764129638671875,"status":"draft","summary":"Small draft models propose a token tree per request; the LLM verifies every node in one decoding step using tree attention with a topology-aware causal mask, then keeps the accepted tokens. Author-reported speedups over existing serving systems: 1.5-2.8x distributed, 2.6-3.5x with offloading.","tags":["inference","serving","speculative-decoding","tree-attention","kv-cache"],"title":"SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification"},{"concepts":["grouped-query-attention","multi-query-attention","kv-cache-management"],"id":"2305.13245","kind":"paper","path":"papers/2305.13245.md","published":"2023-05-22","score":20.764129638671875,"status":"draft","summary":"Grouped-query attention (GQA) splits query heads into G groups that each share one key and one value head, sitting between multi-head (G=H) and multi-query (G=1) attention. Multi-head T5 checkpoints are converted by mean-pooling K/V heads, then uptrained for 5% of pre-training steps. GQA-8 T5-XXL averages 47.1 vs 47.2 for MHA at 0.28 s vs 1.51 s per sample.","tags":["attention","inference","kv-cache","uptraining"],"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"},{"concepts":["kv-cache-management","llm-serving"],"id":"2309.06180","kind":"paper","path":"papers/2309.06180.md","published":"2023-09-12","score":20.764129638671875,"status":"draft","summary":"PagedAttention stores each sequence's KV cache in fixed-size blocks (default 16 tokens) that need not be contiguous, mapped through per-sequence block tables with reference counts and copy-on-write, like OS virtual memory. vLLM, built on it, reports 2-4x higher serving throughput than FasterTransformer and Orca at the same latency.","tags":["inference","serving","memory-management","attention","systems"],"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention"},{"concepts":["sliding-window-attention","grouped-query-attention","kv-cache-management","instruction-tuning"],"id":"2310.06825","kind":"paper","path":"papers/2310.06825.md","published":"2023-10-10","score":20.764129638671875,"status":"draft","summary":"Mistral 7B is a 7-billion-parameter Transformer language model that uses grouped-query attention, sliding-window attention (window 4096) and a rolling-buffer KV cache for cheaper inference, plus an instruction-tuned variant. Author-reported: it beats Llama 2 13B on all but one benchmark row and Llama 1 34B on reasoning, math and code.","tags":["llm","attention","inference","kv-cache","instruction-tuning"],"title":"Mistral 7B"},{"concepts":["autoregressive-image-generation","vector-quantization","classifier-free-guidance","kv-cache-management"],"id":"2404.02905","kind":"paper","path":"papers/2404.02905.md","published":"2024-04-03","score":20.764129638671875,"status":"draft","summary":"VAR replaces raster-order next-token prediction with next-scale prediction: a multi-scale VQ tokenizer turns an image into token maps from 1x1 to 16x16, and a GPT-style transformer predicts each map from the coarser ones in one pass. Author-reported FID 1.73 on ImageNet 256 for the 2.0B model with rejection sampling.","tags":["image-generation","autoregressive-models","scaling-laws"],"title":"Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction"},{"concepts":["diffusion-models","knowledge-distillation","kv-cache-management","latent-diffusion"],"id":"2412.07772","kind":"paper","path":"papers/2412.07772.md","published":"2024-12-10","score":20.764129638671875,"status":"draft","summary":"Turns a bidirectional video diffusion transformer (about 50 denoising steps) into a 4-step causal generator that produces frames chunk by chunk with KV caching. Distribution matching distillation with a bidirectional teacher supervises the causal student, after ODE-pair initialization. Author-reported VBench-Long total 84.27 and 9.4 FPS streaming on one GPU.","tags":["video-generation","autoregressive","distillation","diffusion"],"title":"From Slow Bidirectional to Fast Autoregressive Video Diffusion Models"},{"concepts":["diffusion-models","knowledge-distillation","kv-cache-management","mixture-of-experts"],"id":"2511.23429","kind":"paper","path":"papers/2511.23429.md","published":"2025-11-28","score":20.764129638671875,"status":"draft","summary":"Hunyuan-GameCraft-2 is a 14B image-to-video mixture-of-experts model that turns free-form text, keyboard and mouse inputs into interactive game video. It is distilled into a causal, autoregressive generator with a sink-token KV cache, trained with randomized long-video tuning, and reaches 16 FPS. Author-reported: top InterBench scores against three video baselines.","tags":["video-generation","world-models","interactive","autoregressive","distillation","kv-cache"],"title":"Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model"},{"concepts":["llm-serving","kv-cache-management","arithmetic-intensity","low-precision-arithmetic"],"id":"2512.23914","kind":"paper","path":"papers/2512.23914.md","published":"2025-12-30","score":20.764129638671875,"status":"draft","summary":"Survey of hardware acceleration for neural networks: GPUs and tensor cores, TPUs and NPUs, ASIC and FPGA inference engines, LPU-style LLM-serving chips, in-memory, analog and neuromorphic designs. It sorts the field by workload, execution setting and optimisation lever, and gives the serving metrics and KV-cache size formula. It runs no experiments, so it reports no new numbers.","tags":["survey","hardware-acceleration","gpu","tpu","fpga","asic","llm-serving","kv-cache","benchmarking"],"title":"Hardware Acceleration for Neural Networks: A Comprehensive Survey"}],"query":"kv cache"}