---
kind: concept
slug: token-merging
title: Token merging
summary: >-
  Cutting the token count of a transformer by averaging the most similar tokens inside each block, instead of deleting them. Token Merging (ToMe) matches tokens bipartitely on attention keys and merges a fixed number r per layer, with no added parameters. Off the shelf, ViT-L/16 MAE runs about 2x faster for 1.7 points less top-1 (author-reported).
tags: [inference, efficiency, vision-transformer, token-reduction]
papers: ["2210.09461"]
related: [token-pruning, vision-transformer, efficient-vision-backbone]
status: draft
updated_at: 2026-10-19
---

## Overview

A transformer's cost grows with its token count, and in a vision transformer
the count is set by the image resolution and patch size. Token merging reduces
that count inside the network. In each block, it finds tokens that are similar
and replaces each matched pair with one token, so the next block runs on fewer
tokens [arxiv:2210.09461 §1] [arxiv:2210.09461 §3].

The shared idea is that merging keeps information that pruning throws away.
Averaging a pair of tokens keeps a share of both, while deleting a token
removes it entirely. The paper argues that this is why merging can reach a
higher token reduction at the same accuracy [arxiv:2210.09461 §1]. Its
comparison at the same setting supports this, with merging ahead of random and
attention-based pruning [arxiv:2210.09461 tab. 2].

Token merging is one family of token reduction. The paper's comparison set is
pruning methods, which remove tokens, and clustering methods, which merge
them more slowly [arxiv:2210.09461 §2] [arxiv:2210.09461 tab. 2]. See the
[token pruning](/concepts/token-pruning) page for removal-based methods. Only
one carded merging method exists so far, so this page covers its design space.

## Variants

The carded method, ToMe, has four design choices that the paper ablates: how
many tokens to merge per layer, how to match them, how to combine merged tokens,
and whether to change attention. The paper also contrasts its matching with
other matching rules and with a second schedule.

### Bipartite soft matching with a fixed count (ToMe)

Split the tokens into two sets of about equal size by alternating positions.
Each token in the first set links to its most similar token in the second set,
using cosine similarity on the attention keys averaged over heads. The r
strongest links merge, and the rest of the tokens pass through unchanged
[arxiv:2210.09461 §3] [arxiv:2210.09461 §4.1] [arxiv:2210.09461 tab. 1]. The
count r is a fixed number per layer, not a ratio, so every image in a batch
loses the same number of tokens [arxiv:2210.09461 §3].

The ablation uses ViT-L/16 MAE with r = 8 and reports 84.25% top-1 at 182.9
im/s off the shelf, against 85.96% at 93.3 im/s for the baseline without
merging [arxiv:2210.09461 tab. 1]. The matching is the authors' main choice, so
the other matching rules below are compared against it. Card:
[Token Merging: Your ViT But Faster](/papers/2210.09461).

### Other matching rules

The paper compares two clustering-style rules with the same r = 8 setting.
K-means with 2 iterations reaches 80.19% at 169.7 im/s, and with 5 iterations
80.29% at 147.5 im/s. Greedy matching reaches 84.36% at 179.4 im/s, which is
slightly more accurate than ToMe and slower [arxiv:2210.09461 tab. 2]. Greedy
matching is therefore the nearest alternative in the paper, and the choice
between the two is a trade-off between about 0.1 points of accuracy and about
3.5 im/s of throughput at this setting [arxiv:2210.09461 tab. 2]. The paper
does not give a cost model for either rule beyond these measurements.

### Constant and decreasing schedules

A constant schedule removes r tokens in every layer. A decreasing schedule
removes 2r in the first layer and falls linearly to 0 in the last. Both remove
about rL tokens in total over L layers [arxiv:2210.09461 §4.1] [arxiv:2210.09461 eq. 2]
[arxiv:2210.09461 eq. 3]. On a 15,000-schedule search, the best schedules
below about 1600 to 1800 im/s are close to constant, and above that range the
decreasing schedule is best [arxiv:2210.09461 fig. 2] [arxiv:2210.09461 app. C].

Trained with merging, the decreasing schedule at r = 8 reaches 84.2% at 241
im/s, and the constant schedule at r = 8 reaches 85.1% at 183 im/s
[arxiv:2210.09461 tab. 3]. The choice is therefore a trade-off between
throughput and accuracy at the same r.

### Off the shelf or trained with merging

Merging can be added to a trained model with no weight change. Off the shelf,
ViT-L/16 MAE at r = 8 loses 1.74 points against its baseline, and at r = 4 it
loses 0.27 points [arxiv:2210.09461 app. A]. The paper's model sweep reports up
to 2x throughput with a constant schedule across several models, with losses
that vary by model size [arxiv:2210.09461 §4.2] [arxiv:2210.09461 fig. 3].

Training with merging recovers part of the loss. ViT-L/16 MAE trained with
merging at r = 8 loses 0.61 points at 183 im/s [arxiv:2210.09461 app. A]. The
paper's training recipes are the MAE defaults, which the authors did not change
for merging [arxiv:2210.09461 app. B]. Merging is also used to speed up video
and audio fine-tuning [arxiv:2210.09461 §5] [arxiv:2210.09461 §6].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Vision transformer, no retraining allowed, small accuracy budget | Off-the-shelf ToMe with a small constant r, such as r = 4 | At r = 4, ViT-L/16 MAE loses 0.27 points at 123 im/s against 93 for the baseline [arxiv:2210.09461 app. A] |
| Vision transformer, no retraining, about 2x throughput needed | Off-the-shelf ToMe with constant r = 8 | Loses 1.74 points at 183 im/s [arxiv:2210.09461 app. A]. The paper reports the wider 2x sweep across models [arxiv:2210.09461 §4.2] |
| Retraining is affordable and accuracy matters most | ToMe in training, constant schedule | At r = 8 trained, ViT-L/16 MAE reaches 85.05% against 85.66% for the baseline, at 183 im/s [arxiv:2210.09461 app. A] |
| Throughput above about 1600 im/s is the target | Decreasing schedule | The schedule search finds decreasing schedules best in that range [arxiv:2210.09461 app. C] |
| Batched inference with equal-size inputs | ToMe, because the count per layer is fixed | Counts do not depend on the image, so batching stays simple [arxiv:2210.09461 §3] |
| Video fine-tuning with a long training budget | ToMe in training, r = 65 | Trained at r = 65 constant, fine-tuning took 136 h against 263 h estimated for the baseline on 8 GPUs [arxiv:2210.09461 tab. 6] |
| Audio transformer | Test the r you need | Off the shelf at r = 20, mAP falls from 47.3 to 46.2, and at r = 40 to 43.1 [arxiv:2210.09461 tab. 7] |
| Language model, dense prediction, or a high-resolution input without a test | Not covered | The paper's results cover classification, video and audio. Its dense-prediction results are not reported [arxiv:2210.09461 §5] |

Compare with pruning at the same r before committing. The paper's comparison
is at one setting [arxiv:2210.09461 tab. 2], and the throughput of pruning
methods was measured in other settings in its tab. 4, so do not compare those
numbers across methods.

## Implementation sketch

Pseudocode of the common core, one block at a time. Tokens and sizes are the
per-token inputs, and r is the number of merges in this layer.

```pseudocode
# Merge r similar token pairs inside one transformer block  [arxiv:2210.09461 §3]
# tokens (N, D), sizes (N,) with the number of input patches per token
# keys (N, C): attention keys, averaged over heads  [arxiv:2210.09461 §4.1]

def merge_block(tokens, sizes, keys, r):
    # Normalize keys so a dot product is a cosine similarity  [arxiv:2210.09461 §4.1]
    keys = keys / norm(keys)
    # Alternate positions into two sets A and B  [arxiv:2210.09461 tab. 1]
    key_a, key_b = keys[0::2], keys[1::2]
    # Link each A token to its most similar B token  [arxiv:2210.09461 §3]
    score, link = max(key_a @ key_b.T, axis=-1)
    # Keep the r strongest links  [arxiv:2210.09461 §3]
    order = argsort(score, descending=True)
    merged, kept = order[:r], order[r:]
    # Combine by size-weighted average  [arxiv:2210.09461 tab. 1]
    weighted = tokens * sizes
    sum_b = scatter_add(weighted[1::2], link[merged], weighted[0::2][merged])
    size_b = scatter_add(sizes[1::2], link[merged], sizes[0::2][merged])
    # Concatenate kept A tokens with updated B tokens  [arxiv:2210.09461 §3]
    return concat(tokens[0::2][kept], sum_b / size_b)
```

Proportional attention adds log(size) to the attention logits, so a merged
token counts as the patches it represents [arxiv:2210.09461 §3] [arxiv:2210.09461 eq. 1].
The paper uses it for trained models but not for off-the-shelf MAE, see the
contradictions below. The class token is protected from merging
[arxiv:2210.09461 app. A]. Schedules change only the value of r per layer
[arxiv:2210.09461 eq. 2] [arxiv:2210.09461 eq. 3].

## Contradictions

- **Two baselines for ViT-L/16 MAE.** The caption of the ablation table gives
  85.96% at 93.3 im/s [arxiv:2210.09461 tab. 1]. The appendix sweep gives
  85.66% at 93 im/s, and its drops are relative to that row
  [arxiv:2210.09461 app. A]. The paper does not reconcile the two.
- **Same setting, different numbers.** The ablation reports 84.25% at 182.9
  im/s for off-the-shelf ViT-L/16 MAE at r = 8 with default settings
  [arxiv:2210.09461 tab. 1]. The appendix reports 83.92% at 183 im/s for what
  looks like the same setting, with proportional attention off
  [arxiv:2210.09461 app. A]. The paper gives no reason for the gap.
- **Proportional attention for trained MAE.** The appendix says trained MAE
  models use it [arxiv:2210.09461 app. A]. The main text says it is used except
  for off-the-shelf MAE [arxiv:2210.09461 §4.1]. The two readings give different
  settings for the trained rows.
- **Video baseline.** The video comparison re-evaluates the baseline at one
  crop and ten clips, not the 3x7 protocol of the original paper, so its 84.7%
  is not the published figure [arxiv:2210.09461 app. A] [arxiv:2210.09461 tab. 5].

## Papers

- [Token Merging: Your ViT But Faster](/papers/2210.09461): merges r similar token pairs per block with bipartite soft matching and size-weighted averaging, with off-the-shelf and trained results on images, video and audio.