HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

In plain words

A video model that answers questions while the video is still arriving has a memory problem. Each new frame adds many visual tokens to the model's working memory (its KV cache), so memory and response time grow with the length of the stream. Earlier fixes either drop tokens in a simple first-in, first-out order, or look things up in an external store when a question arrives, which adds delay.

HERMES keeps that memory small without any retraining. It treats the layers of the model as different kinds of memory. The early layers mostly look at the most recent frames, the late layers keep returning to one anchor per frame, and the middle layers mix the two. HERMES keeps a fixed number of tokens in each layer and chooses which ones to keep according to that role. In the late layers, the tokens that get dropped are merged into one summary token. The position numbers are then compacted so the model does not run past the range it was trained on.

All of this happens as new frames arrive, so a question can be answered straight from the cache.

TL;DR

• A training-free, plug-in method for streaming video MLLMs: the KV cache is managed as a hierarchical memory with layer-dependent eviction rules §3.1.
• Three components: a per-role eviction rule, smoothing across depth, and renumbering of positions §3.
• Author-reported: up to 68% fewer video tokens than uniform sampling, with competitive or better accuracy, and about 10× lower time-to-first-token than a prior training-free SOTA at 256 frames §1 §4.3 fig. 4.
• Headline numbers: Qwen2.5-VL-7B with 4K tokens reaches 79.44% on StreamingBench real-time (base 73.31%) tab. 1. LLaVA-OV-7B reaches 60.3% on RVS-Ego with 6K tokens (base 56.2%) tab. 2.
• Offline results are mixed. The method helps LLaVA-OV-7B on VideoMME but lowers Qwen2.5-VL-7B's VideoMME average tab. 4.

Problem

Streaming video QA needs stable accuracy, real-time answers and low GPU memory at the same time. The paper says existing MLLMs do not meet all three §1. Three problems are named:

• Training-free cache methods such as ReKV and LiveVLM keep earlier segments in CPU or disk and retrieve them when a question arrives. That adds latency and depends on external resources §1.
• Many streaming methods need model-specific training §1.
• Sliding-window methods evict tokens with a coarse rule such as FIFO, applied the same way in every layer, which ignores how layers differ §2. Offline compression methods are judged unsuitable because future frames and queries are unknown §1.

The authors also argue that KV-cache eviction is a good place for this, because the cache is already a model-internal memory that the video stream keeps interacting with §1.

Method

Layer roles (investigation) §2 fig. 2. The authors examined LLaVA-OV-7B on VideoMME subsets, with 0.5 fps sampling, 8-frame chunks, a FIFO cache and a 6K-token budget per layer §2. Each frame is 196 tokens §2. They describe three roles:

• Shallow layers act as sensory memory: attention concentrates on the newest tokens and decays with age §2 fig. 2.
• Deep layers act as long-term memory: recency bias largely disappears and attention peaks at regular intervals of 196 tokens, one anchor per frame §2 fig. 2.
• Middle layers act as working memory: recency bias fades with depth and the pattern moves toward the deep-layer pattern §2 fig. 2.

Hierarchical KV cache management §3.1. For each visual token i in layer l, an importance score decides whether it is kept. The score is the recency curve in shallow layers, the attention weight in deep layers, and a depth-dependent blend of the two in middle layers. Recency is an exponential forgetting curve with a forgetting rate k > 0; the paper does not give its value eq. 1. Attention is taken with respect to a generic guidance prompt, used as a pseudo-query, because future queries are unknown §3.1 app. B.

S_i^l = \alpha_i^l \, e^{-k\,\Delta t_i}, \quad \Delta t_i = T - 1 - i eq. 1

S_i^l = \alpha_i^l \, W_i^l eq. 2

\omega^l = \omega_0 - \gamma \, \frac{l - l_{\text{short}}}{l_{\text{long}} - l_{\text{short}}}, \quad \omega_0 = 0.75, \ \gamma = 0.6 eq. 3

S_i^l = (1 - \omega^l) \, A_i^l + \omega^l \, R_i^l eq. 4

Here \alpha is a normalisation factor, W an attention weight, A and R the normalised attention and recency scores, and T the number of video tokens in the cache §3.1 eq. 1 eq. 2.

Cross-layer memory smoothing §3.2. Eviction in one layer can disagree with eviction at the same cache index in a deeper layer. The paper therefore blends each layer's raw score with the next deeper layer's score eq. 5. The top-|M| tokens by smoothed score are kept per layer eq. 6. Evicted tokens of each deep layer are merged into one summary token, which stays in the cache §3.2 alg. 1.

\tilde S_i^l = (1 - \lambda_l) \, S_i^l + \lambda_l \, S_i^{l+1} eq. 5

\mathcal{I}_l = \mathrm{TopK}(\tilde S_l, |M|), \quad K_l \leftarrow K_l[\mathcal{I}_l], \quad V_l \leftarrow V_l[\mathcal{I}_l] eq. 6

Position re-indexing §3.3 app. E. Positions keep growing as the stream grows, and past the model's maximum range generation quality degrades §3.3. Two strategies are given. Lazy re-indexing runs only when positions approach the limit, and is used for streaming benchmarks. Eager re-indexing runs at every compression step, and is used for offline benchmarks §4.1 §4.4. After compaction, cached keys are corrected by a rotary delta so they do not need recomputation app. E.

Summary token alg. 1 §3.2. The value of the summary is the mean of the pruned values. Each pruned key is first rotated from its old position to a target position, and the rotated keys are then averaged. The target position is not stated in the paper alg. 1.

# One compression step after a new chunk  [§3] [§4.1]
# Input:  kv_cache (per layer), chunk of frames, budget |M| per layer
# Output: kv_cache with at most |M| video tokens per layer, plus summaries

def on_chunk(kv_cache, chunk, budget):
    # Prefill the new frames' visual tokens into every layer  [§4.1]
    kv_cache = append_visual_tokens(kv_cache, chunk)
    # Two guidance queries, built from the last answer if one exists  [app. B]
    local_query, global_query = guidance_queries(history)
    # Attention of each layer to the queries, pooled over heads
    # [code: inference/llavaov_hermes.py#L531]
    attn = attention_to(kv_cache, local_query, global_query)
    # Score every video token per layer  [§3.1] [eq. 1] [eq. 2] [eq. 4]
    scores = [raw_score(attn[l], l) for l in layers]
    # Blend each layer with the next deeper one  [§3.2] [eq. 5]
    scores = smooth_across_layers(scores)
    # Keep the top-|M| tokens per layer, plus the fixed system prefix  [eq. 6]
    for l in layers:
        keep = topk(scores[l], budget) + prefix_indices
        if l is a deep layer:
            # Merge the dropped tokens into one summary token  [alg. 1]
            keep_summary = summarise(kv_cache[l], dropped_indices(keep))
        kv_cache[l] = gather(kv_cache[l], keep)
    # Re-number positions so they stay in the model's range  [§3.3] [app. E]
    kv_cache = reindex_positions(kv_cache)
    return kv_cache

The pseudocode condenses the paper's description. Concrete rules, such as the exact scores per role and the summary target position, are in the hyperparameter and pitfall sections below.

Hyperparameters

Guidance prompts app. B. The deep layers use the global prompt. The middle layers use both prompts joined into one string. The paper calls the shallow query "local", but app. B does not state which query the shallow layers use app. B. With conversation history, the prompts are:

• Local: "Find recent details related to: {last_conv}. Describe the current scene in detail, focusing on specific objects, fine-grained actions, and spatial relationships."
• Global: "Context summary: {last_conv}. Summarize the video narrative, identifying main characters, key events, timeline changes, and the overall theme."

Without history, the prompts are "Describe the current scene in detail, focusing on specific objects, fine-grained actions, and spatial relationships." and "Summarize the video narrative, identifying main characters, key events, timeline changes, and the overall theme." app. B. The code uses the same strings llavaov_hermes.py.

Datasets app. D §4.1: StreamingBench real-time subset (2,500 questions over 500 videos); OVO-Bench real-time perception and backward tracing subsets (644 videos, about 2,800 multiple-choice questions in total); RVS-Ego and RVS-Movie (open-ended, 21+ hours); MVBench; Egoschema (official dev split); VideoMME without subtitles.

Results

Author-reported. Setup: LLaVA-OV-7B and Qwen2.5-VL-7B/32B as base models, streaming benchmarks with chunked encoding, 4K and 6K budgets §4.1 tab. 1. Accuracy on OVO-Bench "Avg." is the mean of its real-time and backward subsets tab. 1. Per-task breakdowns are in app. G and are not reproduced here.

Streaming benchmarks tab. 1 (accuracy %, higher is better):

Open-ended streaming, RVS tab. 2 (accuracy % and score on a 1–5 scale; judged by GPT-3.5-turbo-0125 §4.2). The ReKV row marked "†" is treated as an upper bound because it caches all previous frames tab. 2:

Efficiency, LLaVA-OV-7B with a 4K budget, single A800 §4.3 tab. 3:

The paper reports, from its comparison figure fig. 4, about 1.04× lower peak memory than LiveVLM and about 10× lower TTFT than StreamingTOM at 256 frames. The figure has no values in the text, so these ratios are the paper's statements only §4.3.

Offline benchmarks tab. 4 (accuracy %):

Note: the first VideoMME "Avg." column for LLaVA-Video-7B, Qwen2-VL-7B and InternVL-V2-8B is copied as the paper prints it. The paper's text cites 60.29% (Egoschema) and 58.85% (VideoMME) for LLaVA-OV with 4K tokens §4.2. The 58.85 value is the VideoMME average, not the long-video subset.

Ablations (LLaVA-OV-7B, VideoMME unless stated; these use the 6K budget).

Layer-dependent budgets tab. 5. Total budget fixed at 6000 × L. Weights are shallow / middle / deep:

Cross-layer smoothing \lambda tab. 6 (columns are \lambda_{deep}, \lambda_{mid}, \lambda_{shallow}):

The last row is the paper's default tab. 6 app. C.

Summary token in deep layers tab. 7:

Re-indexing tab. 8 (streaming, StreamingBench RT and OVO):

Re-indexing tab. 9 (offline, VideoMME):

Guidance prompts tab. 10. Three alternative wordings were tried; the paper reports results are close to the generic prompt (generic prompt, LLaVA-OV-7B, VideoMME average 58.44; alternatives 57.19 to 57.63; Qwen2.5-VL-7B, generic 62.00; alternatives 60.89 to 61.48) tab. 10.

Total memory budget fig. 5: LLaVA-OV-7B and Qwen2.5-VL-7B were run with budgets from 1K to 10K. The paper reports streaming scores stabilise by 4K and long offline (VideoMME long) scores fall below 4K §4.4 fig. 5.

Limitations (paper's own section): the 72B model was not tested because of compute; combinations with other training-free methods were not studied.

Paper vs code

Code: github.com/haowei-freesky/HERMES at commit 8d699b16a6bedb9086c1b39ec4253c6a1d1ce789 (MIT, Copyright 2026 Haowei Zhang). The paper links this repository in its front matter. Files compared: inference/llavaov_hermes.py, inference/qwenvl_hermes.py, inference/reindex_1d.py, inference/abstract_hermes.py, video_qa/*.py, scripts/run_infer.sh, eval/eval_open_ended.py.

Matches:

• Scoring roles for shallow, middle and deep layers, with the middle-layer blend §3.1 qwenvl_hermes.py.
• Smoothing \lambda 0.1 / 0.3 / 0.4 by layer type app. C llavaov_hermes.py.
• Guidance prompt wording, and the mixed prompt for middle layers app. B llavaov_hermes.py llavaov_hermes.py.
• Summary token: mean of pruned values; keys rotated to one target position then averaged alg. 1 llavaov_hermes.py llavaov_hermes.py.
• Lazy re-indexing for streaming benchmarks, eager for offline §4.4 run_infer.py.
• Compression after every encoded chunk, not at question time §1 hermes_vqa.py.
• Greedy decoding through temperature=0 llavaov_hermes.py.

Differences and gaps:

1. Layer split. The paper uses 10% / 60% / 30% §4.1. The code sets the shallow boundary as int(0.1 * L) and the deep boundary as int(0.7 * L) llavaov_hermes.py. For LLaVA-OV-7B, which has 28 decoder layers §2, this gives layers 0–1 shallow, 2–18 middle and 19–27 deep, which is 2 / 17 / 9 layers (about 7% / 61% / 32%). The Qwen path sets the same ratios qwenvl_hermes.py.
2. Middle-layer recency weight. The paper gives \omega = 0.75 - 0.6\,p §3.1 eq. 3. The Qwen2.5-VL code matches it qwenvl_hermes.py. The LLaVA-OV code uses \omega = 1 - 0.9\,p llavaov_hermes.py. The repository does not say which of the two paths produced the LLaVA-OV results in the paper.
3. Forgetting rate and distance. The paper leaves k unstated eq. 1. The LLaVA-OV code uses k = 20 in shallow layers, declines to 0.5 across middle layers, and measures distance as a fraction of the visual-token span rather than in tokens llavaov_hermes.py llavaov_hermes.py llavaov_hermes.py. The Qwen code declines from 20 to 8 qwenvl_hermes.py.
4. Smoothing reads the refined layer, not the raw one. Eq. 5 uses the raw score S^{l+1} §3.2 eq. 5. The code loops from the deepest layer upward and blends with the already smoothed \tilde S^{l+1}, so smoothing cascades through the stack llavaov_hermes.py. The deepest layer has no deeper layer and is not smoothed llavaov_hermes.py.
5. Interpolation not in the paper. When two layers' score vectors differ in length, which happens once summary tokens are present in deep layers, the code linearly interpolates the deeper scores to the current length llavaov_hermes.py.
6. Normalisation. The paper's \alpha is a normalisation factor with no formula eq. 1 eq. 2. The code min-max normalises attention and recency separately in each layer, then blends them llavaov_hermes.py.
7. Attention pooling. The code takes the attention of each layer, averages over heads, then averages over the query tokens of the guidance prompt, and excludes the system prefix and question tokens llavaov_hermes.py. The paper does not state this.
8. Budget allocation. The paper's layer-dependent budget study uses weights (tab. 5), but the code allocates the same budget to every layer llavaov_hermes.py llavaov_hermes.py. Deep layers give up one slot to make room for the summary token llavaov_hermes.py. The system prefix is always kept on top of the budget llavaov_hermes.py.
9. Lazy compaction threshold. The paper says lazy re-indexing triggers "when positional indices approach the model limit" §3.3. The code compacts only when the largest position exceeds the model's max_position_embeddings minus 1024 llavaov_hermes.py qwenvl_hermes.py. The margin is not in the paper.
10. Summary target position. Algorithm 1 takes a target position t but does not define it alg. 1. The LLaVA-OV code uses the number of kept tokens under eager compaction and the last kept position plus one under lazy compaction llavaov_hermes.py. The Qwen code uses the largest kept position plus one qwenvl_hermes.py.
11. Repetition penalty. The paper says greedy decoding §4.1. The LLaVA-OV question_answering defaults to repetition_penalty=1.1 llavaov_hermes.py, and the VQA driver does not override it base.py base.py. Results from the repository's defaults are therefore not pure greedy decoding.
12. Models in the tables. Qwen3-VL-8B and Qwen3-VL-4B appear in the tables with 2 fps sampling tab. 1 tab. 4. The repository's supported list is LLaVA-OV (0.5B, 7B, 72B) and Qwen2.5-VL (3B, 7B, 32B), and the Qwen wrapper is for Qwen2.5-VL run_infer.sh. The Qwen3-VL numbers cannot be reproduced from this repository.
13. Budgets in the repository. The shell script runs with --kv_size 6000 and 0.5 fps run_infer.sh. The 4K rows and the 1 fps Qwen rows need other arguments; the repository does not record them.
14. Efficiency. The repository has no efficiency benchmark script. The question-answer path prints TTFT llavaov_hermes.py, so the numbers can be checked only by re-running it. The paper's A800 setup is not recorded in the repository.
15. Rotary delta notation. The equations in app. E write the correction as a product with a complex exponential. The code applies the cosine and sine form of the same rotation reindex_1d.py reindex_1d.py. These are equivalent rotations; the difference is notation only.

No other discrepancies were found in the eviction thresholds, the smoothing values for each layer type, the summary-token algorithm, the 16-frame chunking or the guidance prompts.

Pitfalls

• "Up to 11.4%" is absolute. It is the RVS-Movie accuracy gain of HERMES (6K) over LLaVA-OV-7B at 64 uniform frames, 43.0 to 54.4 points tab. 2. It is not a relative gain.
• "Up to 68% fewer tokens" is relative to uniform sampling. 4K tokens against 64 frames × 196 tokens = 12,544 gives a reduction of about 68% §2 §4.1. This is an arithmetic check on the paper's numbers, not a separate result.
• Qwen2.5-VL offline results are lower. VideoMME average drops from 64.52 to 62.00 (6K) and 60.63 (4K) tab. 4. The paper's text does not mention this. Only LLaVA-OV gains on VideoMME average.
• Ablations use 6K, headline tables mostly use 4K. Tables 5–10 report the 6K setting (VideoMME average 58.44). Tables 1 and 4 report the 4K rows as best (for example 58.85 VideoMME average for LLaVA-OV-7B, and 59.21 OVO average for Qwen2.5-VL-7B). Do not compare an ablation row with a headline row without checking the budget.
• Efficiency numbers are in a figure and a table. The teaser caption says TTFT < 30 ms on an A800 80 GB with LLaVA-OV fig. 1. The ratio claims (1.04×, 10×) come from the comparison figure fig. 4 and are not given as numbers in the text.
• Forgetting rate is undefined. k has no value in the paper eq. 1. The code's 20 in shallow layers is a very sharp decay, applied to distances normalised by the visual-token span. Re-implementing from the paper alone would give a different recency curve.
• Middle-layer weight depends on which code you follow. Eq. 3 with \omega_0 = 0.75, \gamma = 0.6 matches the Qwen code. The LLaVA-OV code uses different constants. Pick one and say which in your implementation.
• Summary token and the budget. The paper does not say whether the summary counts toward |M|. The code gives the deep layers |M| - 1 visual slots plus one summary slot, so the total stays at |M| llavaov_hermes.py.
• Smoothing order. Eq. 5 uses raw scores, while the code cascades through already smoothed scores. The paper's ablations do not test this choice tab. 6.
• Guidance prompt wording. Alternative wordings change the LLaVA-OV-7B VideoMME average by up to 1.25 points (58.44 generic, 57.19 lowest alternative) tab. 10.
• Position margin. The code starts lazy compaction 1024 positions below the model's maximum llavaov_hermes.py. A larger margin starts compaction earlier; a smaller one lets positions drift further before compaction. The paper does not state a margin.
• Open-ended scoring is model-judged. RVS accuracy and score come from GPT-3.5-turbo-0125 §4.2 eval_open_ended.py. Results depend on the judge version.
• Benchmark coverage. Only the real-time subset of StreamingBench and the real-time and backward subsets of OVO-Bench are used app. D. The per-task numbers are in the appendix app. G and are not reproduced here.
• Cost. The efficiency table reports TTFT, TPOT and peak memory, not the time spent in compression after each chunk. Compression runs once per encoded chunk hermes_vqa.py, so its cost scales with the stream length, not with the number of questions.

Lineage

• Builds on: the Ebbinghaus forgetting curve for the recency term §3.1; the sensory, working and long-term memory framing cited in §2; StreamingVLM's positional stability strategy for re-indexing app. E.
• Compared against: ReKV, LiveVLM, StreamMem, StreamingTOM, InfiniPot-V, Flash-VStream, Dispider, TimeChat-Online, StreamForest, StreamKV, VideoLLM-online tab. 1 tab. 2 tab. 4.
• Superseded by: not stated.