Attention sink

Overview

An attention sink is a token that receives a large share of attention from many query positions, even when the token carries little relevant content. The first token is the usual case 2603.05498 §3.1. A head is called a sink head when its attention is concentrated this way 2603.05498 §3.2.

Sinks usually appear together with massive activations: a few tokens carry values in a few channels that are far larger than elsewhere in the model. The paper's main claim is that this co-occurrence comes from the architecture, not from one phenomenon needing the other 2603.05498 §1, §4.5. It describes a chain of steps for pre-norm decoder-only Transformers:

1. Early feed-forward blocks, which use SwiGLU, create the spikes in the residual stream. One or two "step-up" blocks inject them and a late "step-down" block cancels them 2603.05498 §3.1, tab. 1.
2. For spike tokens, the SwiGLU block behaves roughly like a quadratic form in the normalized input, so one direction in the input switches many spike channels on together 2603.05498 §3.1, eq. 15–20.
3. The first token, and delimiters such as periods and newlines, are the tokens that reach that direction. The first token's attention output is a fixed linear map of its own state under causal attention 2603.05498 §3.1.
4. RMSNorm turns a spike token into a nearly sparse, nearly constant vector. The keys of those tokens collapse to one or two dimensions, and a head whose queries lie near those keys attends to the sink 2603.05498 §3.2, fig. 6.

Because the sink is a near-fixed key, it can be a parking place for attention the head does not need. Changing the normalization can remove the spikes while the sinks remain 2603.05498 §4.2.2. The paper names the sink's relevance to quantization, pruning, KV-cache management and long-context inference, but it does not evaluate methods for those tasks 2603.05498 §1.

The paper studies Llama-family and Qwen2.5/Qwen3 models for the mechanism, and a Llama-style 7B model trained from scratch for the ablations 2603.05498 §1, app. A, app. C.

Variants

"Variants" here are levers that change how many sinks a model develops. None is a sink-removal method with a published implementation. Each lever was tested in one training setup.

Feed-forward block

Replacing SwiGLU with GeLU keeps perplexity and raises the sink ratio. A linear block or an attention-only model lowers the spike magnitude but raises the sink ratio, and the linear block also raises perplexity 2603.05498 §4.2.1, tab. 4. Card: The Spike, the Sparse and the Sink.

Normalization

Sandwich normalization lowers the spike magnitude, keeps the sink ratio close to the baseline and gives slightly lower perplexity than pre-norm. Sandwich normalization on queries and keys lowers the spikes further. DynamicTanh lowers spikes but raises the sink ratio 2603.05498 §4.2.2, tab. 5.

Gated attention

A per-channel or per-head gate that depends on the input (a conditional gate) removes almost all sinks in the paper's runs. An unconditional per-head gate does not 2603.05498 §4.3.2, tab. 7. The paper's gate details are incomplete, so reproduce from the cited gated-attention work 2603.05498 §4.3.2.

Head dimension

Head dimension is the main architectural driver of sinks in the paper's ablations. Across the runs, the sink ratio rises with head dimension: 4.1% at dimension 8, 37.7% at 64 and 46.0% at the 128 baseline. Dimension 8 also has much higher perplexity, while dimension 64 stays close to the baseline 2603.05498 §4.3.1, tab. 6.

Training context and budget

Sinks are largely induced by short-context training. Excluding short positions from the loss makes the sink ratio collapse 2603.05498 §4.3.3, tab. 8. Training on more tokens gave a higher sink ratio in the paper's single run 2603.05498 tab. 3. Weight decay of 0.0 gave a much larger spike magnitude and a lower sink ratio than the baseline 2603.05498 tab. 3.

Reported numbers

Author-reported, one run per row, Llama-style 7B baseline with pre-norm and SwiGLU 2603.05498 app. A. The "Spike" column is the paper's maximal activation magnitude, with no tensor or layer stated 2603.05498 §4. Rows with a restricted loss range use a different set of positions, so their perplexities are not directly comparable with the other rows 2603.05498 §4.3.3.

When to use which

The paper gives no method for sink removal in deployed models. The table below maps constraints to the levers the paper measured, with the trade-off it reports.

Implementation sketch

Sink detection with the paper's measure. It is not a sink-removal method. The threshold and the sequence length are the paper's [arxiv:2603.05498 app. A, eq. 27–28].

# Fraction of heads that are sinks in one causal attention pass
# [arxiv:2603.05498 app. A] [eq. 27] [eq. 28]
# attn[l][h]: causal (T, T) attention weights; T = 64; eps = 0.3

def sink_fraction(attn, eps=0.3):
    # Keep a count of heads whose first-half key draws enough attention
    hits = 0
    for l in range(num_layers):
        for h in range(num_heads):
            # Mean attention each key receives over query rows  [eq. 27]
            received = attn[l][h].mean(axis=0)  # (T,)
            # The head is a sink if some key in the first half exceeds eps  [eq. 28]
            if received[: T // 2].max() > eps:
                hits += 1
    return hits / (num_layers * num_heads)

Note that the paper's measure uses the mean over query rows of a causal matrix, so late keys receive low values by construction. Do not compare the output of this sketch across lengths without rechecking the paper's setup 2603.05498 app. A.

Contradictions

Only one paper with a card covers this concept so far, so no cross-paper disagreement is recorded. Within that paper, several statements conflict or cannot be checked, and the card lists them:

• Training budget. The appendix gives 5,000 steps at 2M tokens per batch, about 10B tokens. The text and tables say 100B 2603.05498 app. A, tab. 9 §4 tab. 3. The paper does not resolve this.
• Model count. The appendix caption says 12 open models were evaluated, but the model table lists seven 2603.05498 app. C, fig. 7, tab. 10.
• Learning-rate rows. The "minimal learning rate" rows in Table 3 are not labelled. The 3e-4 row is read as no decay, which the paper does not state 2603.05498 tab. 3.
• Mechanism arithmetic. Under the paper's own definitions, the step from eq. 20 to eq. 21 does not follow as printed, by our reading of the card. Check it before relying on that step 2603.05498 eq. 19–21, eq. 23.
• Single runs. No ablation reports seeds, error bars or repeats, so small differences such as perplexity 10.0 vs 10.1 are not evidence of an effect 2603.05498 §4.

Papers

• Massive activations and attention sinks in pre-norm LLMs (arXiv:2603.05498): explains the co-occurrence of spikes and sinks in pre-norm LLMs as architectural, and tests normalization, feed-forward, gating, head-dimension and training-context changes in 7B ablations.