Token pruning

Overview

A transformer spends most of its compute on the tokens it keeps in the sequence. In a vision-language model, one image can add hundreds of them, and the paper argues that most of this cost is wasted in the deeper layers, where the model attends to very few image tokens 2403.06764 §1 2403.06764 §3.3. Token pruning removes some of those tokens, so the computation that follows runs on a shorter input.

Three carded methods apply this idea at different points. FastV removes image tokens inside one forward pass, once per prompt 2403.06764 §4.1. HERMES keeps a streaming video model's KV cache at a fixed size, so tokens are evicted each time a new chunk of frames arrives 2601.14724 §1 2601.14724 §3. ToMe does not remove tokens at all. It merges the most similar pairs in every block, so the count still falls by a fixed amount per layer, and the merged tokens keep their information 2210.09461 §1 2210.09461 §3.

FastV and HERMES decide what to keep with the model's own attention. ToMe decides with the similarity of attention keys, not attention weights 2210.09461 §4.1. The FastV invariant is that the text tokens (system prompt, instruction and output) are never dropped 2403.06764 §4.1. HERMES does not state that invariant, and it merges evicted deep-layer tokens into a summary token instead of discarding them 2601.14724 §3.2. Other pruning methods exist and are not compiled here, so this page covers only the three carded papers.

Variants

The carded methods differ in when tokens are removed or merged, and what they keep. FastV's design space has three knobs. The filter layer K sets how many layers run on the full input. The filter ratio R sets the share of image tokens removed. The ranking score decides which tokens survive. The paper's headline and main ablations vary K and R, and it ablates the ranking 2403.06764 §4.1 2403.06764 tab. 1 2403.06764 tab. 7.

Attention-ranked pruning after an early layer (FastV)

Run the first K layers unchanged. At layer K, rank each image token by the attention it receives, keep the top (1 - R) share and drop the rest. The deeper layers then run on the reduced sequence 2403.06764 §4.1. The method needs no training and can be added to an existing model 2403.06764 §4. At K = 2 and R = 50%, the average score on four benchmarks is 69.7 for LLaVA-1.5-7B against 69.8 unpruned, and 73.6 against 73.6 for LLaVA-1.5-13B, at about 55% of theoretical FLOPs 2403.06764 tab. 1. Scores are author-reported. Card: FastV.

Pruning before the first layer (K = 0)

With K = 0 the image tokens are removed before any layer runs, and the paper ranks them uniformly at random 2403.06764 §4.1. The FLOPs saving is slightly larger than at K = 2 (52% against 55% for LLaVA-1.5-7B at R = 50%), and the average score is lower (69.0 against 69.7) 2403.06764 tab. 1. The paper's latency table uses this setting, so it is the one with measured speed 2403.06764 tab. 4. Card: FastV.

Merging similar tokens in every block (ToMe)

Token Merging applies to plain vision transformers. In each block, between attention and the MLP, it links every token in one half of the sequence to its most similar partner in the other half, keeps the r strongest links, and averages each linked pair 2210.09461 §3. The count falls by r per layer and the schedule is set by r, not by a ratio, so every image gets the same number of merges 2210.09461 §3. Off the shelf, ViT-L/16 MAE with r = 8 runs at 183 im/s against 93 im/s for the baseline, at 83.92% against 85.66% top-1 on ImageNet-1k 2210.09461 app. A. The paper's ablation table reports 84.25% at 182.9 im/s for the same model and r with its default settings 2210.09461 tab. 1. Training with the method cuts video fine-tuning time to about half 2210.09461 tab. 6. Card: Token Merging. All numbers are author-reported.

Layer-role eviction in a streaming KV cache (HERMES)

HERMES targets streaming video question answering. It compresses the KV cache after every encoded chunk, so a question can be answered from the cache without retrieval from CPU or disk 2601.14724 §1 2601.14724 §3. The authors measured attention in LLaVA-OV-7B and describe three roles: shallow layers favour recent tokens, deep layers attend to one anchor per frame, and middle layers mix the two 2601.14724 §2 2601.14724 fig. 2.

The score of each cached visual token depends on its role. Shallow layers use an exponential recency curve, deep layers use attention, and middle layers blend the two with a depth-dependent weight 2601.14724 eq. 1 2601.14724 eq. 2 2601.14724 eq. 4. Each layer's score is then smoothed with the next deeper layer's score, and the top tokens within a per-layer budget |M| are kept 2601.14724 §3.2 2601.14724 eq. 5 2601.14724 eq. 6. Deep layers merge their evicted tokens into one summary token 2601.14724 alg. 1. Positions are renumbered so they stay in the model's range, either lazily near the limit or at every step 2601.14724 §3.3.

The authors report, on LLaVA-OV-7B and Qwen2.5-VL-7B with a 4K token budget per layer, StreamingBench real-time accuracy of 73.23 against 71.34 for LLaVA-OV-7B at 0.5 fps, and 79.44 against 73.31 for Qwen2.5-VL-7B at 1 fps 2601.14724 tab. 1. The 6K budget gives slightly lower streaming numbers for both models 2601.14724 tab. 1. Card: HERMES.

The paper leaves the forgetting rate k of the recency curve unstated 2601.14724 eq. 1. The other HERMES constants, such as the middle-layer weight and the layer split, are given in the text 2601.14724 eq. 3 2601.14724 §4.1. An implementation from the paper alone will therefore need to choose k.

When to use which

Pick K and R together. On OCR-VQA, a small K is sensitive to R, while a large K makes R matter less 2403.06764 fig. 7. Test the ranking on your own task before you commit: the paper's ranking criterion differs from the one in the released code, see Contradictions below. For ToMe, the paper's constant schedule was close to the best of its random schedules on one model 2210.09461 fig. 2, so start from a constant r and test.

Implementation sketch

Pseudocode of the FastV core. Positions of the image tokens must be known, which the paper does not state for any given prompt template.

# Drop low-attention image tokens after layer K  [arxiv:2403.06764 §4.1]
# Input:  hidden (B, N, d); K filter layer; R filter ratio in [0, 1]
#         image tokens sit at positions [start, start + M)
# Output: hidden (B, N_kept, d), kept positions in original order

def fastv_forward(layers, hidden, K, R, start, M):
    # Layers before K run on the full sequence  [arxiv:2403.06764 §4.1]
    attention = None
    keep_count = round(M * (1 - R))
    for index, layer in enumerate(layers):
        if index == K:
            # Score each image token by the attention it receives
            # Paper: average over all other tokens  [arxiv:2403.06764 §4.1]
            scores = attention_received(attention, start, M)
            # Random ranking when K == 0  [arxiv:2403.06764 §4.1]
            if K == 0:
                scores = random_scores(M)
            # Keep the top image tokens and every non-image token
            kept_image = top_k(scores, keep_count) + start
            kept = concat(range(0, start), sorted(kept_image),
                          range(start + M, N))
            hidden = hidden[:, kept]
        hidden, attention = layer(hidden)
    return hidden

Where the variants differ: the ranking step (attention criterion, or random at K = 0) and whether pruned tokens are removed from the sequence or only masked. The released code has both modes, and only the removal mode shortens the sequence 2403.06764 §4.1. The FastV card lists the code-level details.

ToMe differs in kind: it averages pairs instead of dropping rows. Its core is a bipartite match on key similarity, then a size-weighted average of each linked pair 2210.09461 §3. Its pseudocode is on the Token Merging card, so it is not repeated here.

HERMES works on a cache rather than a single pass. Its core is a per-layer score by role, then a top-|M| selection after every chunk, with the deep-layer evictions folded into a summary token 2601.14724 §3. Its pseudocode is on the HERMES card, so it is not repeated here.

Contradictions

• Ranking criterion (paper against released code). The paper ranks tokens by the average attention each token receives from all other tokens 2403.06764 §4.1. The released code uses the attention that the last query token gives to each key 2403.06764 §4.1. The two are different quantities, and the repository does not say which one produced the tables.
• "No performance loss" against the tables. The paper says FastV costs no performance at the headline setting 2403.06764 §5.3. The tables show smaller drops on some benchmarks, such as InstructBLIP on MME at K = 2 (1129.8 against 1143.5) 2403.06764 tab. 2. The paper says InstructBLIP needs K = 5 to close that gap 2403.06764 §5.3.
• Offline video against streaming video (HERMES). HERMES improves streaming scores for both models it tests, but lowers Qwen2.5-VL-7B's offline VideoMME average at 4K 2601.14724 tab. 1 2601.14724 tab. 4. The paper's text does not discuss the offline drop.
• Ranking against random choice. Both papers show a small gap. FastV's ablation at K = 2, R = 50% gives an average of 99.5 on Nocaps with random ranking, against 100.1 with its attention ranking 2403.06764 tab. 7. ToMe's matching ablation gives 79.22% top-1 for random pruning against 79.48% for attention-based pruning, in its ViT-L/16 MAE setting 2210.09461 tab. 2. The settings differ, so the two gaps are not directly comparable, but neither paper shows a large advantage for attention.

Papers

• Token Merging: Your ViT But Faster: similarity-based merging of tokens in every ViT block, parameter-free, with off-the-shelf and trained results for image, video and audio.
• FastV: An Image is Worth 1/2 Tokens After Layer 2: attention-ranked removal of image tokens after an early layer, no training, FLOPs and latency measured for LLaVA-1.5 and others.
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding: training-free KV-cache eviction for streaming video models, with layer-dependent scoring, a summary token and position re-indexing.