---
kind: concept
slug: token-pruning
title: Token pruning
summary: >-
  Removing tokens from the sequence so later computation runs over fewer of them. FastV prunes vision-language image tokens after an early layer, ranked by attention, with no training. ToMe merges similar vision-transformer tokens in every block, a close relative. HERMES evicts KV-cache video tokens by layer role while a stream arrives. Results are author-reported and not comparable across methods.
tags: [inference, efficiency, multimodal, vision-language-model, vision-transformer, streaming-video, kv-cache]
papers: ["2210.09461", "2403.06764", "2601.14724"]
related: [multimodal-language-model, self-attention, kv-cache-management, llm-serving, efficient-vision-backbone, vision-transformer, positional-encoding, network-pruning]
status: draft
updated_at: 2026-10-19
---

## Overview

A transformer spends most of its compute on the tokens it keeps in the
sequence. In a vision-language model, one image can add hundreds of them, and
the paper argues that most of this cost is wasted in the deeper layers, where
the model attends to very few image tokens [arxiv:2403.06764 §1] [arxiv:2403.06764 §3.3].
Token pruning removes some of those tokens, so the computation that follows
runs on a shorter input.

Three carded methods apply this idea at different points. FastV removes image
tokens inside one forward pass, once per prompt [arxiv:2403.06764 §4.1]. HERMES
keeps a streaming video model's KV cache at a fixed size, so tokens are evicted
each time a new chunk of frames arrives [arxiv:2601.14724 §1] [arxiv:2601.14724 §3].
ToMe does not remove tokens at all. It merges the most similar pairs in every
block, so the count still falls by a fixed amount per layer, and the merged
tokens keep their information [arxiv:2210.09461 §1] [arxiv:2210.09461 §3].

FastV and HERMES decide what to keep with the model's own attention. ToMe
decides with the similarity of attention keys, not attention weights
[arxiv:2210.09461 §4.1]. The FastV invariant is that the text tokens (system
prompt, instruction and output) are never dropped [arxiv:2403.06764 §4.1].
HERMES does not state that invariant, and it merges evicted deep-layer tokens
into a summary token instead of discarding them [arxiv:2601.14724 §3.2]. Other
pruning methods exist and are not compiled here, so this page covers only the
three carded papers.

## Variants

The carded methods differ in when tokens are removed or merged, and what they
keep.
FastV's design space has three knobs. The filter layer K sets how many layers
run on the full input. The filter ratio R sets the share of image tokens
removed. The ranking score decides which tokens survive. The paper's headline
and main ablations vary K and R, and it ablates the ranking
[arxiv:2403.06764 §4.1] [arxiv:2403.06764 tab. 1] [arxiv:2403.06764 tab. 7].

### Attention-ranked pruning after an early layer (FastV)

Run the first K layers unchanged. At layer K, rank each image token by the
attention it receives, keep the top (1 - R) share and drop the rest. The
deeper layers then run on the reduced sequence [arxiv:2403.06764 §4.1]. The
method needs no training and can be added to an existing model
[arxiv:2403.06764 §4]. At K = 2 and R = 50%, the average score on four
benchmarks is 69.7 for LLaVA-1.5-7B against 69.8 unpruned, and 73.6 against
73.6 for LLaVA-1.5-13B, at about 55% of theoretical FLOPs
[arxiv:2403.06764 tab. 1]. Scores are author-reported. Card: [FastV](/papers/2403.06764).

### Pruning before the first layer (K = 0)

With K = 0 the image tokens are removed before any layer runs, and the paper
ranks them uniformly at random [arxiv:2403.06764 §4.1]. The FLOPs saving is
slightly larger than at K = 2 (52% against 55% for LLaVA-1.5-7B at R = 50%),
and the average score is lower (69.0 against 69.7) [arxiv:2403.06764 tab. 1].
The paper's latency table uses this setting, so it is the one with measured
speed [arxiv:2403.06764 tab. 4]. Card: [FastV](/papers/2403.06764).

### Merging similar tokens in every block (ToMe)

Token Merging applies to plain vision transformers. In each block, between
attention and the MLP, it links every token in one half of the sequence to its
most similar partner in the other half, keeps the r strongest links, and
averages each linked pair [arxiv:2210.09461 §3]. The count falls by r per layer
and the schedule is set by r, not by a ratio, so every image gets the same
number of merges [arxiv:2210.09461 §3]. Off the shelf, ViT-L/16 MAE with r = 8
runs at 183 im/s against 93 im/s for the baseline, at 83.92% against 85.66%
top-1 on ImageNet-1k [arxiv:2210.09461 app. A]. The paper's ablation table
reports 84.25% at 182.9 im/s for the same model and r with its default
settings [arxiv:2210.09461 tab. 1]. Training with the method cuts video
fine-tuning time to about half [arxiv:2210.09461 tab. 6]. Card:
[Token Merging](/papers/2210.09461). All numbers are author-reported.

### Layer-role eviction in a streaming KV cache (HERMES)

HERMES targets streaming video question answering. It compresses the KV cache
after every encoded chunk, so a question can be answered from the cache without
retrieval from CPU or disk [arxiv:2601.14724 §1] [arxiv:2601.14724 §3]. The
authors measured attention in LLaVA-OV-7B and describe three roles: shallow
layers favour recent tokens, deep layers attend to one anchor per frame, and
middle layers mix the two [arxiv:2601.14724 §2] [arxiv:2601.14724 fig. 2].

The score of each cached visual token depends on its role. Shallow layers use
an exponential recency curve, deep layers use attention, and middle layers blend
the two with a depth-dependent weight [arxiv:2601.14724 eq. 1] [arxiv:2601.14724 eq. 2]
[arxiv:2601.14724 eq. 4]. Each layer's score is then smoothed with the next
deeper layer's score, and the top tokens within a per-layer budget |M| are kept
[arxiv:2601.14724 §3.2] [arxiv:2601.14724 eq. 5] [arxiv:2601.14724 eq. 6].
Deep layers merge their evicted tokens into one summary token
[arxiv:2601.14724 alg. 1]. Positions are renumbered so they stay in the model's
range, either lazily near the limit or at every step [arxiv:2601.14724 §3.3].

The authors report, on LLaVA-OV-7B and Qwen2.5-VL-7B with a 4K token budget per
layer, StreamingBench real-time accuracy of 73.23 against 71.34 for LLaVA-OV-7B
at 0.5 fps, and 79.44 against 73.31 for Qwen2.5-VL-7B at 1 fps
[arxiv:2601.14724 tab. 1]. The 6K budget gives slightly lower streaming numbers
for both models [arxiv:2601.14724 tab. 1]. Card: [HERMES](/papers/2601.14724).

The paper leaves the forgetting rate k of the recency curve unstated
[arxiv:2601.14724 eq. 1]. The other HERMES constants, such as the middle-layer
weight and the layer split, are given in the text [arxiv:2601.14724 eq. 3]
[arxiv:2601.14724 §4.1]. An implementation from the paper alone will therefore
need to choose k.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Vision-language model, cannot retrain, prefill cost dominates, short answers | FastV at K = 2, R = 50% | Average scores held on the tested models at about 55% of theoretical FLOPs [arxiv:2403.06764 tab. 1]. The speed test used K = 0 with one-token answers [arxiv:2403.06764 tab. 4] |
| Need the largest cut and can accept lower scores | FastV at K = 2, R = 75% | About 33% of FLOPs. Average on LLaVA-1.5-7B falls to 67.1 [arxiv:2403.06764 tab. 1]. OCR-VQA Rouge-L on 13B stays at 0.54 against 0.55 [arxiv:2403.06764 tab. 5] |
| Benchmarks other than the paper's, or a model other than LLaVA and Qwen-VL | Re-test at your K and R | Results vary by model: InstructBLIP-Vicuna-13B drops on several benchmarks at K = 2, R = 50% and recovers at K = 5 [arxiv:2403.06764 tab. 2] [arxiv:2403.06764 §5.3] |
| Need measured speed for long generation | Measure it yourself | The paper's speed test covers one-token answers on one GPU [arxiv:2403.06764 tab. 4]. Its FLOPs are analytic [arxiv:2403.06764 §4.2] |
| Non-vision inputs (text-only) | Not covered | The paper's experiments are on image and video inputs of vision-language models [arxiv:2403.06764 §5] |
| Streaming video, questions arrive while the stream continues, GPU memory must stay bounded | HERMES at a 4K budget per layer | Author-reported gains over the base models on streaming benchmarks [arxiv:2601.14724 tab. 1]. On one A800 with LLaVA-OV-7B, time to first token stayed between 27 and 28 ms from 16 to 512 frames [arxiv:2601.14724 tab. 3] |
| Streaming video, but retrieval from CPU or disk is acceptable | Compare with retrieval methods first | The paper names retrieval-based streaming methods as adding latency and depending on external storage [arxiv:2601.14724 §1]. Its tables compare them with HERMES on the same benchmarks [arxiv:2601.14724 tab. 1] |
| Offline video QA, where the whole video is available | Test before use | Gain only on LLaVA-OV-7B: VideoMME average 57.67 to 58.85 at 4K. Qwen2.5-VL-7B falls from 64.52 to 60.63 [arxiv:2601.14724 tab. 4] |
| Qwen3-VL or LLaVA-OV-72B | Not reproducible from the authors' repository | The paper's tables include Qwen3-VL [arxiv:2601.14724 tab. 1], but the released code covers LLaVA-OV and Qwen2.5-VL, and the 72B model was not tested [arxiv:2601.14724 §4.1] |
| Plain vision transformer, every image needs the same token count, batched inference | ToMe with a constant r | ToMe removes a fixed r per layer, while the paper says pruning methods that remove a different count per image make batching hard [arxiv:2210.09461 §1] [arxiv:2210.09461 §3] |
| Vision transformer, can accept a small drop and wants a plug-in with no training | ToMe, off the shelf, r chosen by a sweep | Off the shelf, r = 8 on ViT-L/16 MAE cost 1.74 points of top-1 for about 2x throughput [arxiv:2210.09461 app. A]. Its schedule search was run on one model [arxiv:2210.09461 §4.1] |
| Vision transformer, pruning methods already in use | Compare with pruning first | The ToMe paper's throughput comparison is against A-ViT, DynamicViT and SP-ViT, not FastV or HERMES. The paper says its pruning rows were timed in different settings, so their throughput is not comparable [arxiv:2210.09461 tab. 4] |

Pick K and R together. On OCR-VQA, a small K is sensitive to R, while a large
K makes R matter less [arxiv:2403.06764 fig. 7]. Test the
ranking on your own task before you commit: the paper's ranking criterion
differs from the one in the released code, see Contradictions below. For ToMe,
the paper's constant schedule was close to the best of its random schedules on
one model [arxiv:2210.09461 fig. 2], so start from a constant r and test.

## Implementation sketch

Pseudocode of the FastV core. Positions of the image tokens must be known,
which the paper does not state for any given prompt template.

```pseudocode
# Drop low-attention image tokens after layer K  [arxiv:2403.06764 §4.1]
# Input:  hidden (B, N, d); K filter layer; R filter ratio in [0, 1]
#         image tokens sit at positions [start, start + M)
# Output: hidden (B, N_kept, d), kept positions in original order

def fastv_forward(layers, hidden, K, R, start, M):
    # Layers before K run on the full sequence  [arxiv:2403.06764 §4.1]
    attention = None
    keep_count = round(M * (1 - R))
    for index, layer in enumerate(layers):
        if index == K:
            # Score each image token by the attention it receives
            # Paper: average over all other tokens  [arxiv:2403.06764 §4.1]
            scores = attention_received(attention, start, M)
            # Random ranking when K == 0  [arxiv:2403.06764 §4.1]
            if K == 0:
                scores = random_scores(M)
            # Keep the top image tokens and every non-image token
            kept_image = top_k(scores, keep_count) + start
            kept = concat(range(0, start), sorted(kept_image),
                          range(start + M, N))
            hidden = hidden[:, kept]
        hidden, attention = layer(hidden)
    return hidden
```

Where the variants differ: the ranking step (attention criterion, or random
at K = 0) and whether pruned tokens are removed from the sequence or only
masked. The released code has both modes, and only the removal mode shortens
the sequence [arxiv:2403.06764 §4.1]. The [FastV card](/papers/2403.06764)
lists the code-level details.

ToMe differs in kind: it averages pairs instead of dropping rows. Its core is a
bipartite match on key similarity, then a size-weighted average of each linked
pair [arxiv:2210.09461 §3]. Its pseudocode is on the
[Token Merging card](/papers/2210.09461), so it is not repeated here.

HERMES works on a cache rather than a single pass. Its core is a per-layer
score by role, then a top-|M| selection after every chunk, with the deep-layer
evictions folded into a summary token [arxiv:2601.14724 §3]. Its pseudocode is
on the [HERMES card](/papers/2601.14724), so it is not repeated here.

## Contradictions

- **Ranking criterion (paper against released code).** The paper ranks tokens
  by the average attention each token receives from all other tokens
  [arxiv:2403.06764 §4.1]. The released code uses the attention that the last
  query token gives to each key [arxiv:2403.06764 §4.1]. The two are different
  quantities, and the repository does not say which one produced the tables.
- **"No performance loss" against the tables.** The paper says FastV costs no
  performance at the headline setting [arxiv:2403.06764 §5.3]. The tables show
  smaller drops on some benchmarks, such as InstructBLIP on MME at K = 2
  (1129.8 against 1143.5) [arxiv:2403.06764 tab. 2]. The paper says InstructBLIP
  needs K = 5 to close that gap [arxiv:2403.06764 §5.3].
- **Offline video against streaming video (HERMES).** HERMES improves
  streaming scores for both models it tests, but lowers Qwen2.5-VL-7B's offline
  VideoMME average at 4K [arxiv:2601.14724 tab. 1] [arxiv:2601.14724 tab. 4].
  The paper's text does not discuss the offline drop.
- **Ranking against random choice.** Both papers show a small gap. FastV's
  ablation at K = 2, R = 50% gives an average of 99.5 on Nocaps with random
  ranking, against 100.1 with its attention ranking [arxiv:2403.06764 tab. 7].
  ToMe's matching ablation gives 79.22% top-1 for random pruning against 79.48%
  for attention-based pruning, in its ViT-L/16 MAE setting [arxiv:2210.09461 tab. 2].
  The settings differ, so the two gaps are not directly comparable, but neither
  paper shows a large advantage for attention.

## Papers

- [Token Merging: Your ViT But Faster](/papers/2210.09461): similarity-based merging of tokens in every ViT block, parameter-free, with off-the-shelf and trained results for image, video and audio.
- [FastV: An Image is Worth 1/2 Tokens After Layer 2](/papers/2403.06764): attention-ranked removal of image tokens after an early layer, no training, FLOPs and latency measured for LLaVA-1.5 and others.
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](/papers/2601.14724): training-free KV-cache eviction for streaming video models, with layer-dependent scoring, a summary token and position re-indexing.