An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

In plain words

A vision-language model turns each image into hundreds of tokens and feeds them to a language model together with the text. Most of the compute goes to these image tokens. The authors looked at how the model attends to them and found that, in the later layers, the model pays very little attention to the image tokens. It mostly looks at a few anchor tokens, most of them in the system prompt.

FastV uses that finding. The model runs its first few layers as usual. At a chosen layer it scores every image token by how much attention they get, keeps the best ones and throws the rest away. The deeper layers then run on a shorter input. Nothing is retrained, so the method can be added to an existing model.

The trade-off is a dial: throw away more image tokens and the compute drops further, but the answers get worse. The paper reports that cutting about half of the image tokens after the second layer costs little accuracy on the tested models. The compute savings are counted in theory, not measured on a real system, except for one short-answer test.

TL;DR

• FastV removes image tokens inside a vision-language model after an early layer, using attention scores as the ranking, with no training §4 §4.1.
• Headline setting K=2, R=50%: about 45% fewer theoretical FLOPs on LLaVA-1.5-7B, LLaVA-1.5-13B and Qwen-VL-Chat, with average scores within 0.5 points of the baseline on four tasks §5.3 tab. 1.
• Speed was measured once, on A-OKVQA with one-token answers, single A40 GPU. There, LLaVA-1.5-13B with FastV ran at about the per-example latency of the unmodified 7B model tab. 4.
• Use it when a plug-in is needed for inference-time savings. Check the Pitfalls section first: the paper's ranking description and its code differ, and the FLOPs figures are theoretical.

Problem

Current vision-language models turn each image into hundreds of tokens (576 for LLaVA at 336 by 336 pixels) and higher resolutions need more §2. On the sampled tasks, image tokens made up on average 64% of the input fig. 3. The paper argues that this cost is largely wasted in deep layers: image tokens receive far less attention than their count suggests, and the system prompt absorbs most of the attention §3.3 fig. 3 fig. 4.

The measurements behind this come from LLaVA-1.5-7B, 1000 image-text pairs sampled across Flickr30K, PCA-Bench, A-OKVQA and MMMU, with greedy decoding §3.2. The assumption is that the shallow layers (the first two) spread attention widely, while deeper layers rely on a few "anchor" tokens §3.3 §3.4.

Method

FastV is defined by three parameters: a filter layer K, a filter ratio R and a ranking function. It runs in four steps §4.1:

1. Run decoder layers 0 to K-1 on the full input, exactly as the unmodified model does §4.1.
2. At layer K, score each image token. The paper's stated criterion is the average attention a token receives from all other tokens §4.1. The repository implements a different score; see Paper vs code.
3. Keep the top (1 - R) share of image tokens and drop the rest. System prompt, instruction and output tokens are never dropped §4.1 tab. 7.
4. Run layers K onward on the shorter sequence. Pruned tokens are not read again §4.1 §4.2.

Setting K=0 removes image tokens before the first layer, and the paper then ranks them at random §4.1 tab. 1.

# FastV: drop low-attention image tokens from layer K onward  [§4.1]
# Input:  hidden (B, N, d); K filter layer; R filter ratio in [0, 1]
#         image tokens sit at positions [start, start + M)
# Output: hidden (B, N_kept, d), kept positions in original order

def fastv_forward(layers, hidden, K, R, start, M):
    # Layers before K run on the full sequence  [§4.1]
    attention = None
    keep_count = round(M * (1 - R))
    for index, layer in enumerate(layers):
        if index == K:
            # Score image tokens: last query row, averaged over heads
            # [code: src/transformers/.../modeling_llama.py#L774-L783]
            scores = mean_heads(attention)[-1, start:start + M]
            # Random ranking when K == 0  [§4.1] [code: ...#L797-L798]
            if K == 0:
                scores = random_scores(M)
            # Keep the top image tokens, every non-image token  [§4.1]
            kept_image = top_k(scores, keep_count) + start
            kept = concat(range(0, start), sorted(kept_image),
                          range(start + M, N))
            hidden = hidden[:, kept]
        hidden, attention = layer(hidden)
    return hidden

Computing cost. The paper gives the per-layer cost as 4nd² + 2n²d + 2ndm for n tokens, hidden size d and FFN size m §4.2. Summed over T layers with pruning after layer K and n̂ = (1 - R)·n tokens afterwards, the FLOPs ratio is given in eq. 5 fig. 6. The paper's FLOPs tables come from this formula, not from timing §4.2 tab. 1.

Two variants are described. A reduced-resolution baseline (retraining on pooled tokens) is compared in §4.3 and in the ablation tab. 7. The repository also contains a masking variant that keeps the sequence length and only masks tokens; see Paper vs code.

Hyperparameters

Results

Author-reported. Setup: benchmark scores with greedy decoding, FLOPs from the formula in §4.2. Table numbers follow the order of the LaTeX source.

Main results, average over Nocaps, Flickr30k, A-OKVQA and MMMU tab. 1

More models and fine-grained benchmarks tab. 2

MME sub-scores, LLaVA-1.5-13B vs FastV K=2, R=50% tab. 3

Real latency, A-OKVQA, single A40 GPU tab. 4

PCA-Bench and OCR-VQA tab. 5. P, C, A: perception, cognition, action; G-PCA: genuine PCA score.

Video question answering (Video-LLaVA; GPT-based scoring) tab. 6

Ablations, LLaVA-1.5-7B retrained baseline tab. 7

Figure results, described in words: the K/R trade-off curve for average performance against FLOPs reduction shows near-flat performance up to about 45% reduction across models fig. 1. The K/R heat map on OCR-VQA shows that at small K, lowering R helps accuracy at a smaller FLOPs saving, while at large K, R matters little fig. 7 fig. 6.

Paper vs code

The official repository is https://github.com/pkunlp-icler/FastV, linked from the abstract, at commit d1659729b5bf1be225e99ee15783deeea80f63b1. It contains a fork of HF transformers with the FastV logic in the Llama decoder, a LLaVA fork, and eval scripts for A-OKVQA and OCR-VQA. The repository has no top-level LICENSE file; the bundled transformers and LLaVA forks carry Apache-2.0 licenses.

• Ranking score. Paper: "average attention-score one token received from all other tokens" §4.1. Code: average attention over heads, then the row of the last query token, so it is attention that the last token gives to each key modeling_llama.py modeling_llama.py. These are different quantities. The repo does not say which of the two produced the tables.
• Layer index. Paper: rank "at layer K" §4.1. Code: layers with index below K run unchanged, and the scores come from the attention output of layer K-1 (zero-based) modeling_llama.py modeling_llama.py. This matches the paper's "after layer 2" when K=2 means two unchanged layers.
• Demo defaults. The demo uses K=3 and R=0.75 demo-hf.py, not the headline K=2, R=50%. Copying the demo without changing it gives a different setting from the paper's table.
• Latency setting. Table 4 labels the latency rows K=0, R=50% tab. 4. The shipped latency script runs K=2 with in-place pruning and rank 288, which is R=50% eval_aokvqa_latency_fastv_inplace.sh. The repo does not show which run produced the table rows.
• Pruning mechanism. The transformers fork offers an in-place mode, which drops tokens and changes sequence length, and a mask mode, which keeps all tokens and masks them modeling_llama.py ...modeling_llama.py. The in-place branch comments say KV-cache is not supported. The mask mode does not shorten the computation, so its speed claims would not be expected to follow.
• Image token count. The llava-hf fork keeps round(M·(1 - R)) image tokens modeling_llama.py. The paper states R as a percentage of image tokens §4.1 tab. 1, which matches.
• Random ranking at K=0. The code uses random ranking only when K=0 modeling_llama.py. The paper says the same §4.1.

Pitfalls

• Ranking criterion. Implement the code's version (last query row, averaged over heads) if you want to match the reported numbers. The paper's words describe attention a token receives, and the code does not match them. Test both on one benchmark before choosing.
• Offsets are not in the paper. The code needs the position of the first image token and the length of the prefix before it. The paper does not state these values. The two repo settings differ (36 and 5), so they depend on the prompt template. A wrong offset silently prunes the wrong tokens.
• FLOPs are theoretical. §4.2 gives an analytic count. Real speed-up was measured once, on A-OKVQA with one-token answers, on a single A40 tab. 4. For long generation, the paper does not measure speed, and the KV-cache is not covered.
• Decoding with cache. The paper's KV-cache discussion is commented out in the source (in the implementation details of §4, not in the rendered paper). The llava-hf path prunes only when the input has more than one token, so it acts on the prefill step. How decode steps treat pruned positions is not stated in the paper.
• Cost formula. eq. 5 shrinks all n tokens to (1 - R)·n after K, though the paper introduces it as a reduction "related to image tokens" §4.2. The paper does not say which n produced the FLOPs in tab. 1. Compute your own numbers from your model's shapes.
• "No performance loss" is a summary. The abstract and §5.3 say FastV costs no performance at the headline setting. Small drops are visible: InstructBLIP at K=2, R=50% scores 1129.8 on MME vs 1143.5 tab. 2, and Qwen-VL drops from 94.9 to 94.4 on Nocaps tab. 1. The paper also reports that InstructBLIP needs K=5 to close the gap §5.3.
• Low-attention claims. The introduction says image tokens get 0.21% of the score given to system prompts, and "50%" in the first two layers §1. The comparison in those figures is not spelled out, so use the tables fig. 3 instead.
• Ablation baseline. The ablation uses a retrained LLaVA-1.5-7B tab. 7. Its baseline row is not the released model's row in tab. 1.
• Section and figure numbering. The text says task details are in "section A" §5.1. The LaTeX places them in appendix B, with appendix A holding the full attention maps app. A app. B. The text also calls the all-layer attention figure "figure-7" §3.3; the LaTeX order gives it number 8. Figure and table numbers in this card follow LaTeX order, not a compiled PDF, which was not checked.
• Author and title in source. The LaTeX author line says "Shuang Bai"; arXiv metadata says "Shuai Bai". The LaTeX title says "VLLM" where arXiv metadata says "Large Vision-Language Models". The card uses arXiv metadata (v3). The LaTeX source is the COLM 2024 submission template.
• Video evaluation. Video QA scores use GPT-based grading through a commercial API and only the first 1K examples per benchmark app. B. Re-running them needs the same grader and access.
• Output-length setting. The paper says the A-OKVQA model outputs one option tab. 4. The eval script sets max_new_tokens=1 for that task inference_aokvqa.py. Longer generation, such as captioning, is not measured for latency.

Lineage

• Builds on: not stated as such. The paper relates its finding to StreamingLLM's observation that the first few tokens dominate §5.4 and to information-flow analysis of LLMs §3.4.
• Compared against: a model trained with pooled image tokens §4.3 tab. 7; random ranking tab. 7 (c); system-prompt and instruction pruning tab. 7 (d), (f); StreamingLLM's attention pattern tab. 7 (g).
• Superseded by: not stated in the content.