---
kind: concept
slug: attention-sink
title: Attention sink
summary: >-
  Attention heads that put much of their weight on a few early tokens, often the first one, whatever those tokens mean.
  In pre-norm decoder LLMs they co-occur with massive activations, and the paper ties both to the architecture.
  Conditional gating and training-context changes reduce sinks in its ablations, with different costs.
  All numbers are single-run and author-reported.
tags: [attention, interpretability, massive-activations, normalization, inference, kv-cache]
papers: ["2603.05498"]
related: [attention-mechanism, attention-interpretability, mechanistic-interpretability, kv-cache-management, network-pruning, post-training-quantization]
status: draft
updated_at: 2026-11-05
---

## Overview

An attention sink is a token that receives a large share of attention from many
query positions, even when the token carries little relevant content. The first
token is the usual case [arxiv:2603.05498 §3.1]. A head is called a sink head when
its attention is concentrated this way [arxiv:2603.05498 §3.2].

Sinks usually appear together with massive activations: a few tokens carry
values in a few channels that are far larger than elsewhere in the model. The
paper's main claim is that this co-occurrence comes from the architecture, not
from one phenomenon needing the other [arxiv:2603.05498 §1, §4.5]. It describes
a chain of steps for pre-norm decoder-only Transformers:

1. Early feed-forward blocks, which use SwiGLU, create the spikes in the
   residual stream. One or two "step-up" blocks inject them and a late
   "step-down" block cancels them [arxiv:2603.05498 §3.1, tab. 1].
2. For spike tokens, the SwiGLU block behaves roughly like a quadratic form in
   the normalized input, so one direction in the input switches many spike
   channels on together [arxiv:2603.05498 §3.1, eq. 15–20].
3. The first token, and delimiters such as periods and newlines, are the tokens
   that reach that direction. The first token's attention output is a fixed
   linear map of its own state under causal attention [arxiv:2603.05498 §3.1].
4. RMSNorm turns a spike token into a nearly sparse, nearly constant vector.
   The keys of those tokens collapse to one or two dimensions, and a head whose
   queries lie near those keys attends to the sink [arxiv:2603.05498 §3.2, fig. 6].

Because the sink is a near-fixed key, it can be a parking place for attention
the head does not need. Changing the normalization can remove the spikes while
the sinks remain [arxiv:2603.05498 §4.2.2]. The paper names the sink's
relevance to quantization, pruning, KV-cache management and long-context
inference, but it does not evaluate methods for those tasks [arxiv:2603.05498 §1].

The paper studies Llama-family and Qwen2.5/Qwen3 models for the mechanism, and
a Llama-style 7B model trained from scratch for the ablations
[arxiv:2603.05498 §1, app. A, app. C].

## Variants

"Variants" here are levers that change how many sinks a model develops. None
is a sink-removal method with a published implementation. Each lever was tested
in one training setup.

### Feed-forward block

Replacing SwiGLU with GeLU keeps perplexity and raises the sink ratio. A linear
block or an attention-only model lowers the spike magnitude but raises the sink
ratio, and the linear block also raises perplexity [arxiv:2603.05498 §4.2.1, tab. 4].
Card: [The Spike, the Sparse and the Sink](/papers/2603.05498).

### Normalization

Sandwich normalization lowers the spike magnitude, keeps the sink ratio close to
the baseline and gives slightly lower perplexity than pre-norm. Sandwich
normalization on queries and keys lowers the spikes further. DynamicTanh lowers
spikes but raises the sink ratio [arxiv:2603.05498 §4.2.2, tab. 5].

### Gated attention

A per-channel or per-head gate that depends on the input (a conditional gate)
removes almost all sinks in the paper's runs. An unconditional per-head gate
does not [arxiv:2603.05498 §4.3.2, tab. 7]. The paper's gate details are
incomplete, so reproduce from the cited gated-attention work
[arxiv:2603.05498 §4.3.2].

### Head dimension

Head dimension is the main architectural driver of sinks in the paper's
ablations. Across the runs, the sink ratio rises with head dimension: 4.1% at
dimension 8, 37.7% at 64 and 46.0% at the 128 baseline. Dimension 8 also has
much higher perplexity, while dimension 64 stays close to the baseline
[arxiv:2603.05498 §4.3.1, tab. 6].

### Training context and budget

Sinks are largely induced by short-context training. Excluding short positions
from the loss makes the sink ratio collapse [arxiv:2603.05498 §4.3.3, tab. 8].
Training on more tokens gave a higher sink ratio in the paper's single run
[arxiv:2603.05498 tab. 3]. Weight decay of 0.0 gave a much larger spike
magnitude and a lower sink ratio than the baseline [arxiv:2603.05498 tab. 3].

### Reported numbers

Author-reported, one run per row, Llama-style 7B baseline with pre-norm and
SwiGLU [arxiv:2603.05498 app. A]. The "Spike" column is the paper's maximal
activation magnitude, with no tensor or layer stated [arxiv:2603.05498 §4].
Rows with a restricted loss range use a different set of positions, so their
perplexities are not directly comparable with the other rows
[arxiv:2603.05498 §4.3.3].

| Setting | Perplexity | Sink ratio | Spike | Source |
|---|---|---|---|---|
| Baseline (SwiGLU, pre-norm) | 10.1 | 46.0% | 3818 | [tab. 3] [tab. 4] |
| FFN: GeLU | 10.1 | 69.3% | 3369 | [tab. 4] |
| FFN: linear | 12.5 | 58.9% | 688 | [tab. 4] |
| FFN: attention-only | 10.8 | 73.9% | 637 | [tab. 4] |
| Norm: sandwich | 9.8 | 44.7% | 520 | [tab. 5] |
| Norm: sandwich (QK) | 10.0 | 42.0% | 92 | [tab. 5] |
| Norm: DynamicTanh | 10.0 | 61.0% | 153 | [tab. 5] |
| Gate: per-channel, conditional | 10.0 | 4.5% | 202 | [tab. 7] |
| Gate: per-head, conditional | 10.1 | 6.4% | 186 | [tab. 7] |
| Gate: per-head, unconditional | 10.1 | 41.3% | 1884 | [tab. 7] |
| Head dimension 8 | 11.3 | 4.1% | 291 | [tab. 6] |
| Head dimension 64 | 10.3 | 37.7% | 2112 | [tab. 6] |
| Weight decay 0.0 | 10.4 | 33.8% | 12275 | [tab. 3] |
| 200B training tokens | 9.5 | 63.3% | 1848 | [tab. 3] |
| Loss on positions 2048 to 4096 only | 10.6 | 1.2% | 7193 | [tab. 8] |

## When to use which

The paper gives no method for sink removal in deployed models. The table below
maps constraints to the levers the paper measured, with the trade-off it reports.

| Situation | Use | Why |
|---|---|---|
| Debugging or auditing a released model | Sink ratio (eq. 27–28, eps = 0.3, T = 64) | Measures how many heads are sinks; the paper's tables use these settings [arxiv:2603.05498 app. A]. |
| Released model, architecture fixed | Treat sinks as part of the model; the levers above are not tested on released weights | The mechanism was checked on open models, but every ablation trains from scratch [arxiv:2603.05498 §4, app. C]. The card records no post-hoc test. |
| Training a pre-norm model from scratch, want fewer spikes at similar quality | Sandwich normalization | Spikes fall from 3818 to 520 with perplexity 9.8 vs 10.1 and a sink ratio of 44.7% vs 46.0% [arxiv:2603.05498 tab. 3, tab. 5]. Single run. |
| Training from scratch, want sinks gone | Conditional gate on attention | Sink ratio falls to 4.5–6.4% with perplexity about unchanged [arxiv:2603.05498 tab. 7]. Unconditional gating did not remove sinks. |
| Choosing head dimension | Trade-off: smaller heads for fewer sinks, larger for perplexity | Dimension 8 gives a 4.1% sink ratio but perplexity 11.3; dimension 64 gives 37.7% and perplexity 10.3 [arxiv:2603.05498 tab. 6]. |
| Long-context training | Keep short sequences in the training mix, or check the loss range | Short-context training induces sinks; long-only loss removes them and raises perplexity in the paper's comparison [arxiv:2603.05498 §4.3.3, tab. 8]. |
| Planning compression or KV-cache work | Check which heads are sinks before choosing a method | The paper says sinks matter for these tasks but tests none of them [arxiv:2603.05498 §1]. |

## Implementation sketch

Sink detection with the paper's measure. It is not a sink-removal method. The
threshold and the sequence length are the paper's [arxiv:2603.05498 app. A,
eq. 27–28].

```pseudocode
# Fraction of heads that are sinks in one causal attention pass
# [arxiv:2603.05498 app. A] [eq. 27] [eq. 28]
# attn[l][h]: causal (T, T) attention weights; T = 64; eps = 0.3

def sink_fraction(attn, eps=0.3):
    # Keep a count of heads whose first-half key draws enough attention
    hits = 0
    for l in range(num_layers):
        for h in range(num_heads):
            # Mean attention each key receives over query rows  [eq. 27]
            received = attn[l][h].mean(axis=0)  # (T,)
            # The head is a sink if some key in the first half exceeds eps  [eq. 28]
            if received[: T // 2].max() > eps:
                hits += 1
    return hits / (num_layers * num_heads)
```

Note that the paper's measure uses the mean over query rows of a causal matrix,
so late keys receive low values by construction. Do not compare the output of
this sketch across lengths without rechecking the paper's setup
[arxiv:2603.05498 app. A].

## Contradictions

Only one paper with a card covers this concept so far, so no cross-paper
disagreement is recorded. Within that paper, several statements conflict or
cannot be checked, and the card lists them:

- **Training budget.** The appendix gives 5,000 steps at 2M tokens per batch,
  about 10B tokens. The text and tables say 100B [arxiv:2603.05498 app. A, tab. 9]
  [§4] [tab. 3]. The paper does not resolve this.
- **Model count.** The appendix caption says 12 open models were evaluated, but
  the model table lists seven [arxiv:2603.05498 app. C, fig. 7, tab. 10].
- **Learning-rate rows.** The "minimal learning rate" rows in Table 3 are not
  labelled. The 3e-4 row is read as no decay, which the paper does not state
  [arxiv:2603.05498 tab. 3].
- **Mechanism arithmetic.** Under the paper's own definitions, the step from
  eq. 20 to eq. 21 does not follow as printed, by our reading of the card. Check
  it before relying on that step [arxiv:2603.05498 eq. 19–21, eq. 23].
- **Single runs.** No ablation reports seeds, error bars or repeats, so small
  differences such as perplexity 10.0 vs 10.1 are not evidence of an effect
  [arxiv:2603.05498 §4].

## Papers

- [Massive activations and attention sinks in pre-norm LLMs (arXiv:2603.05498)](/papers/2603.05498): explains the co-occurrence of spikes and sinks in pre-norm LLMs as architectural, and tests normalization, feed-forward, gating, head-dimension and training-context changes in 7B ablations.