Vision Transformers Need More Than Registers

In plain words

Vision Transformers summarise a whole image in one special token, the one used for classification. Those models also turn every small image patch into its own token, and the paper finds that the summary token often gets its meaning from patches of background such as sky, grass or wall, rather than from the object. The patches that look most like the summary are background, so the model's maps point at the wrong place.

The fix is simple to describe. Instead of building the summary from all patches, or from a single special token, the model looks at each feature channel and keeps only the patches whose values in that channel stay steady when smoothed. Steady patches are more likely to be the object, so the summary is built from them.

The change is a small readout step that needs no extra labels, and the authors apply it to to models trained with class labels, with image-text pairs or with self-supervision. The paper reports that the object-focused maps improve several dense-prediction tasks. The results are the authors' own and come from specific models and benchmarks.

TL;DR

Under global attention, ViT patch tokens that match the image-level [CLS] vector are often background, a pattern the paper calls lazy aggregation §4. LaSt-ViT replaces the [CLS] readout with a channel-wise average of the most stable patch tokens, where stability comes from a frequency low-pass across channels §5.1. It adds no loss term; the paper reports gains in Points-in-Box, object discovery, zero-shot segmentation and open-vocabulary detection, all author-reported §6. The released code covers only ImageNet classification training and the Point-in-Box check, and it does not match the paper's stability formula exactly [Paper vs code].

Problem

Dense features from ViTs, meaning per-patch features used for segmentation or detection, show artifacts: the patch most aligned with the image summary often lies on background. The authors measure this with Patch Score (a cosine match of each patch to the global vector) and Point-in-Box (whether the top-scoring patch falls inside the annotated object box) §4.1, eq. 3. On ImageNet, the ViT row reaches 42.7 Point-in-Box against 68.4 for ResNet [§1, tab:pib]. Removing the highest-scoring half of patches barely changes classification accuracy, which the authors read as evidence that the summary relies on background §4.2. The paper argues the cause is coarse image-level supervision combined with global attention §4.3, §4.4. Those two causal claims rest on intervention experiments (larger patches, window attention), which also lower accuracy §4.3, §4.4.

Method

Preliminary. A ViT splits the image into P×P patches, embeds them into N tokens of width D, runs the encoder, and produces a global token either by global average pooling over patch tokens eq. 1 or with a learned [CLS] query eq. 2 §3.1.

Analysis probes. The first probe scores each patch by its cosine match to the global token eq. 3 §4.1. For ConvNets the global vector is the pooled feature. Point-in-Box uses ImageNet validation images with one annotated object; the score is the fraction of images whose highest-scoring patch lies in that box §4.1.

Stability score (LaSt-ViT, step 1). Let x_patch be the N×D patch outputs with [CLS] removed, and g a vector of D Gaussian weights scaled to [0, 1]. Take a 1-D FFT of each patch along the channel axis, multiply by g, apply the inverse FFT and keep the real part, giving a low-pass version x̂ eq. 4. The stability of patch i in channel j is x̂[i,j] divided by |x̂[i,j] − x_patch[i,j]| + ε eq. 5 §5.1. The paper does not state σ of g or the value of ε.

Channel-wise top-K pooling (step 2). For each channel j, take the K most stable patches eq. 6 and average their original values in that channel eq. 7. The result replaces the [CLS] output as the global representation.

Vote count (visualisation). The number of channels that select patch i is its vote count eq. 8. The paper draws patches with votes above 50%, 30% and 20% of the image's maximum [§5.1, fig:cls_after_lazystrike].

Transfer. Object discovery thresholds the patch-score map at its mean plus one standard deviation §5.2. For text-supervised ViTs, zero-shot segmentation compares patch features with text embeddings §5.2. For open-vocabulary detection, the table applies the method to F-ViT's frozen CLIP backbone and compares with F-VLM, which uses a ConvNet backbone §6.4.

# LaSt-ViT readout: [CLS] output replaced by top-K pooling  [§5.1]
# Input:  patch_tokens (B, N, D), encoder outputs with [CLS] dropped
# Output: cls_token (B, D)

def lazy_strike_readout(patch_tokens, k):
    # Low-pass every patch along its channel axis: FFT, Gaussian
    # weights in [0, 1]^D, inverse FFT, keep the real part  [eq. 4]
    low_pass = channel_low_pass(patch_tokens)  # (B, N, D)

    # Stability of each (patch, channel) pair  [eq. 5]
    stability = low_pass / (abs(low_pass - patch_tokens) + eps)

    # Per channel, indices of the k most stable patches  [eq. 6]
    top_indices = topk(stability, k, patch_axis)  # (B, k, D)

    # Average the selected original values, channel by channel  [eq. 7]
    selected = gather(patch_tokens, top_indices)  # (B, k, D)
    cls_token = mean(selected, patch_axis)  # (B, D)
    return cls_token

# Vote count per patch, for the visualisation in §5.1  [eq. 8]
# Input:  top_indices (B, k, D); Output: votes (B, N)

def vote_counts(top_indices, num_patches):
    # Count the channels in which each patch is selected  [eq. 8]
    votes = count_selections(top_indices, num_patches)  # (B, N)
    return votes

Hyperparameters

Only values the paper or code state are listed. Everything else is "not stated".

Results

Author-reported, with setups as stated in §6. Setups differ between rows; see the Source column (for example [§6.2 tab:pibv2]).

Points-in-Box (ImageNet validation, single-object images; [§6.2 tab:pibv2]):

Coarse segmentation on VOC12 (patch-score maps, label-supervised, and a self-supervised DINO reference; [§6.3 tab:full]):

Zero-shot semantic segmentation (mIoU %; [§6.4 tab:semantic]); selected columns:

The table also has Cityscapes and COCO-Object columns, not repeated here.

Open-vocabulary detection and segmentation (LVIS and COCO; frozen CLIP backbone; [§6.4 tab:ovd]):

Unsupervised object discovery (CorLoc %, ViT-S backbone; [§6.5 tab:corloc_lost]):

Ablation, label-supervised ViT (ImageNet top-1 and object-location results on VOC07 and VOC12; [§6.6 tab:ablations2]):

The text says the ablation pool is the label-supervised ViT-B/32 model. The caption calls the VOC columns downstream object-location results; the source gives no further detail on that metric.

Ablation, text-supervised ViT-B/16 (OpenCLIP; [§6.6 tab:ablations]): IN1K top-1 is 55.8 (attention-pool), 53.1 (max-pool), 55.8 (K = 49), 56.2 (K = 98) and 55.3 (K = 196, all). Segmentation columns peak at K = 98 on VOC and at K = 49 on COCO.

Paper vs code

The released repository is ChengShiest/LAST-ViT (MIT, Copyright 2026 ChengShiest) at commit cdeb884af65e7774f2da80f666d95cf09a76b717. It contains a classification training config, a Point-in-Box evaluation, visualisation scripts and README snippets. It does not contain DINO or CLIP training integration (the README points to other repositories) README.md, object discovery, open-vocabulary detection, zero-shot segmentation or PCA code.

• Stability numerator. Paper eq. 5 puts the low-pass value x̂ in the numerator §5.1. The training model divides the original patch value by |x̂ − x| conf.py. The visualiser's default repo formula does the same, using a clamp instead of +ε visualize_patch_score_distribution.py. Its paper branch puts x̂ on top but drops the absolute value in the denominator, so it does not match eq. 5 either visualize_patch_score_distribution.py.
• ε. Paper eq. 5 has ε; the training model has none conf.py.
• Kernel width and centring. Paper: σ not stated. Code: σ = √768, kernel length 768, and the kernel is hard-coded to 768 conf.py. The kernel peak sits at index 383, while fftshift on a length-768 vector puts zero frequency at index 384 conf.py. The effect of this one-bin offset is not measured here.
• Top-K. Paper: general K §5.1. Code: k = 1 hard-coded conf.py. README reports k = 1 and k = 7 results README.md.
• [CLS] output. The training loss reads the logits from the pooled token conf.py. The original [CLS] output is computed but not used in the loss conf.py.
• Patch-hit evaluation. evaluate_patch_hit.py imports DenseViT from patch_score.py evaluate_patch_hit.py, whose patch scores use the output at position 0 as CLS patch_score.py. That class has no LaSt-ViT readout, and the model load uses strict=False evaluate_patch_hit.py. The only evaluation path that builds the LaSt readout is visualize_patch_score_distribution.py (aggregation="last") visualize_patch_score_distribution.py. Applying the hit script to a LaSt checkpoint therefore scores patches against an output the loss never reads.
• Single-object filter. Paper: images with one annotated object §4.1. Code also keeps only boxes covering less than 25% of the image imagenet_dataloader.py.
• Point-in-Box geometry. The script counts every patch that the box touches, using inclusive integer bounds evaluate_patch_hit.py. The paper says the top patch must fall within the box §4.1; the difference is small but not stated.
• Training recipe. The paper's main text does not state an optimiser or schedule. Code: AdamW, lr 1e-3, weight decay 0.3, 100 epochs, cosine schedule conf.py.
• Architecture naming. The paper's ablation text names ViT-B/32 §6.6; the released training config is ViT-B/16 conf.py.
• Reported accuracy. README gives about 67.4% (k = 1) and 67.6% (k = 7) top-1 for ViT_190k, citing "Table 11" of arXiv v1 README.md. The paper's label-supervised ablation reports 64.6 and 64.8 [§6.6 tab:ablations2]. The two may be different setups; the source package does not say.
• Appendix. The paper refers to an appendix for patch-size details, dataset settings and training recipes §4, §4.3. The source package does not include it, so those details are not stated here.

No discrepancies found for the Patch Score formula eq. 3: the code and paper both use cosine similarity patch_score.py.

Pitfalls

• Stability formula is ambiguous across sources. The paper's numerator is the low-pass value; the training code uses the original value. Pick one, and say which one your results use.
• Signed scores. The code takes largest=True on signed stability values conf.py. The paper says "highest stability" without saying whether sign matters. A channel where all patches are negative will still select its least negative patches.
• Hard-coded width. The kernel is built for D = 768 conf.py. The paper's ViT-S experiments (§6.3, §6.5) use a narrower embedding (384 in the standard ViT-S definition; the paper does not state D), so the code does not run unchanged there, and the paper does not say how the kernel was set for ViT-S.
• No ε in the training path. Division by |x̂ − x| with no ε is a division by zero whenever a patch equals its low-pass value in a channel; the code does not guard it. The visualiser clamps at 1e-6, but the training path does not conf.py.
• Misleading name. The variable x_detach is never detached, so gradients reach the patch tokens through the gathered values; top-K indices carry no gradient conf.py.
• Cost. Each forward pass runs an FFT over channels for every patch and a top-K over patches for every channel. The paper does not report this overhead; the throughput figure in §6.5 is not tied to hardware in the text.
• Mechanism claims are inferred. "Lazy aggregation" and the causal links to coarse supervision and global attention rest on intervention experiments that also lower accuracy §4.3, §4.4. Treat them as the authors' hypothesis.
• Arithmetic in the tables. Printed gains do not match the values in four places:
  • tab:ovd, F-ViT ViT-B/16, COCO base AP50: 41.0 → 50.1 is +9.1, printed +11.1.
  • tab:ovd, F-ViT ViT-L/14, LVIS novel: 24.2 → 32.1 is +7.9, printed +6.6; the text repeats the 6.6.
  • tab:corloc_lost: 64.4 − 61.9 = +2.5; the text says +2.7 for VOC07.
  • tab:pib in §1 gives DINO-ViT 45.3, while tab:pibv2 in §6.2 gives DINO-v1 44.5 for the same family; the +25.2 gain uses 44.5.
• Column naming. The text calls the Context59 column "Pascal"; the table's VOC20 column is the Pascal-VOC benchmark §6.4.
• Citation key. The intro cites dino for emergent segmentation in self-supervised ViTs, but bib key dino resolves to a DETR-based paper (arXiv:2203.03605), not the self-supervised DINO [main.bib]. Check which work the authors mean.
• Version drift. The source is arXiv v2 (updated 2026-04-14). The repository README cites v1 for its table.
• Missing appendix. Patch-size details, dataset and evaluation settings, and training recipes are said to be in an appendix §4, §4.3 that the source package does not include.

Lineage

• Builds on: DINO [arXiv:2104.14294, card exists] for the self-supervised setting; ViT (arXiv:2010.11929) for the architecture; LOST (arXiv:2109.14279) for unsupervised object discovery §6.5.
• Related: Vision Transformers Need Registers (arXiv:2309.16588, bib key registers) reports high-norm tokens in ViTs and is the paper's main point of comparison §1, §6.2.
• Compared against: CLIP, MetaCLIP and EVA-CLIP for zero-shot segmentation §6.4; F-VLM and F-ViT for open-vocabulary detection §6.4; LOST and DINO-seg for object discovery §6.5; attention-pool, mean-pool and max-pool readouts §6.6.
• Superseded by: not known from content in this wiki.