The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

In plain words

Large language models sometimes produce a handful of enormous numbers. A few tokens carry values in a few internal channels that are far larger than anywhere else. The same models also tend to pour a lot of attention onto a few tokens, often the first one, whatever those tokens mean. These two effects usually show up together, and this paper asks whether one causes the other.

Its answer is mostly no. The layout of the normalization step creates the conditions in which both effects appear, and the huge values act as nearly fixed settings inside the model. Attention then uses the first token as a place to park attention it does not need. When the normalization is changed, the huge values disappear and the parking spot can still work, and language modelling quality barely moves. That matters for compressing models, pruning them, managing their memory during generation and handling long inputs.

TL;DR

Massive activations and attention sinks co-occur in pre-norm decoder-only Transformers because of the architecture, not because one needs the other §1, §4.5. Early SwiGLU feed-forward blocks create the spikes §3.1. RMSNorm turns spike tokens into sparse, nearly constant vectors whose keys then sit in a low-dimensional subspace that sink heads use §3.2. Changing the normalization can remove the spikes while the sinks survive §4.2.2. All numbers are author-reported.

Problem

Both phenomena matter for quantization, pruning, KV-cache management and long-context inference §1. Earlier work described the co-occurrence and the overlap of tokens, but not the mechanism or the causal link §1. The paper studies pre-norm decoder-only Transformers in the Llama family, and checks the mechanism on Llama 2, Llama 3 and Qwen2.5/Qwen3 models §1, §2, app. C.

Method

The paper is an analysis, not a new model. It has two parts: a mechanism account of how the phenomena form in pretrained open models §3, and causal ablations on models trained from scratch §4.

Pre-norm block (background). Each block adds its output to the residual stream after normalizing its input. Odd-indexed blocks are attention, even- indexed ones are feed-forward eq. 4 §2. The normalization is RMSNorm without its learnable gain and without any epsilon in the formula eq. 5 §2.

# Pre-norm residual update  [§2] [eq. 4] [eq. 5]
# Input: hidden (T, d_model); block is attention or feed-forward

def block_update(hidden, block):
    # Row-wise RMS normalization; the paper omits gain and epsilon [eq. 5]
    normed = sqrt(d_model) * hidden / norm(hidden, axis=-1)  # (T, d_model)
    # Add the block output to the residual stream  [eq. 4]
    return hidden + block(normed)

Step 1: how spikes are born and die §3.1. Tracking the top-3 channel magnitudes across depth shows a rise, a plateau and a fall fig. 1. The spikes often exceed typical activation scales by several orders of magnitude §3.1. (1) One or two early "step-up" blocks inject extreme values; in the examined models they are feed-forward blocks tab. 1. (2) Intermediate blocks carry them through the residual sum eq. 14, where their contributions to spike channels are reported to be two to three orders of magnitude smaller than the spikes §3.1. (3) One or a few late "step-down" blocks output the opposite sign and cancel them §3.1 tab. 1.

Step 2: the feed-forward block as a quadratic amplifier §3.1. For spike tokens, SiLU is close to the identity, so the SwiGLU block is approximated by a product of two linear maps eq. 15 fig. 2. Each output coordinate then has a quadratic form, the matrix U_k built from the down, gate and up weights eq. 16 eq. 17 app. B, eq. 35–42. Its symmetric part S_k has one dominant eigenvalue in spike channels eq. 18 fig. 4. With a unit eigenvector s⋆ and eigenvalue λ⋆, the output grows with the square of the projection of the normalized input onto s⋆ eq. 19 eq. 20. Spike channels share nearly the same s⋆, so one trigger direction switches on many channels together §3.1.

Step 3: which tokens reach the trigger direction §3.1. Two cases are described.

• First token. With causal attention the first token attends only to itself, so its attention output is a fixed linear map W_VO applied to its state eq. 22. Over 98% of vocabulary items become spike tokens at position 0 in the listed models tab. 2.
• Delimiters (for example periods and newlines). Their normalized magnitude is high in early attention blocks, so heads attend to the token itself ("self-sinking"). This moves them toward the same trigger direction §3.1.

Step 4: normalization turns spikes into sink-ready inputs §3.2. After RMSNorm, every entry is bounded by sqrt(d_model) eq. 23 app. B, eq. 43–47. Because the norm is dominated by a few outlier coordinates, the normalized spike vector is approximately sparse, supported on the spike channels eq. 24. Across spike tokens, these channels keep nearly fixed ratios, so the normalized vectors are nearly identical even when the other channels differ eq. 25 fig. 5.

Step 5: keys and sink heads §3.2. The key of a sink token is W_K^T applied to that sparse vector eq. 26. Its span collapses to one or two dimensions in practice §3.2. A head becomes a sink head when its query subspace lies closer to these fixed sink keys than to ordinary keys, which gives large, consistent logit gaps §3.2 fig. 6.

Causal ablations §4. The baseline is a Llama-style 7B model trained from scratch on 100B DCLM tokens §4 app. A, tab. 9. Each ablation changes one factor. The ablations cover the optimizer, the feed-forward block, the normalization, head dimension and count, gated attention, and the training context-length distribution §4.1–§4.3. The results are in the Results section.

The sink ratio, which is the metric used throughout, is defined in two steps. A key's importance is its mean attention over query positions eq. 27. A head is a sink if some key in the first half of the sequence exceeds a threshold eq. 28.

# Sink ratio for one evaluation sequence  [app. A] [eq. 27] [eq. 28]
# Input: attn[l][h] is a causal (T, T) attention matrix; T = 64, eps = 0.3

def sink_ratio(attn, eps):
    # Count heads whose first-half key receives enough attention
    sink_heads = 0
    for each layer l and head h:
        # Importance: mean attention each key receives  [eq. 27]
        importance = mean(attn[l][h], axis=0)  # (T,)
        # Sink if a key in the first half exceeds eps  [eq. 28]
        if max(importance[: T // 2]) > eps:
            sink_heads += 1
    # Divide by the number of heads in all layers  [eq. 28]
    return sink_heads / (num_layers * num_heads)

Hyperparameters

Values for the baseline and the evaluation protocol. Where the paper is silent, the cell says so.

Results

Author-reported. Setup: Llama-style 7B trained from scratch on 100B DCLM tokens, perplexity and sink ratio on C4 samples, spike = maximal activation magnitude §4 app. A. The paper does not say which tensor or layer the spike column refers to. Each row changes one factor from the baseline (pre-norm, SwiGLU, base learning rate 3e-4, 32 heads of dimension 128, 100B tokens).

Main findings the paper reports:

• Sink ratio tracks optimization health; spike magnitude varies largely independently of perplexity and sink ratio §4.1.
• GeLU and SwiGLU feed-forward blocks give much larger spikes than linear or attention-only blocks §4.2.1 tab. 4. Sandwich norm, QKNorm and DynamicTanh each reduce spikes relative to pre-norm §4.2.2 tab. 5.
• Head dimension is the main architectural driver of sinks §4.3.1 tab. 6.
• Conditional, representation-dependent gating removes sinks, while static or unconditional gates do not §4.3.2 tab. 7.
• Sinks are largely induced by short-context training. Excluding short positions from the loss makes the sink ratio collapse §4.3.3 tab. 8.
• Open-weight checks: the step-up and step-down pattern appears in Llama 2 7B/13B, Llama 3 8B, and Qwen2.5 and Qwen3 models tab. 1 app. C, fig. 7. The paper's own count of these models is inconsistent, see Pitfalls.

No seeds, confidence intervals or repeated runs appear in the tables or text.

Paper vs code

No official code.

Searched: the paper's LaTeX source and the arXiv abstract page list no code link or project page. Three GitHub repository searches were run: "massive activations" (35 results, none owned by an author), "massive activations attention sinks" (none owned by an author), and "spike sparse sink" (one hit). That hit, savinasun/SpikeSparseSink, has a README that calls itself the official implementation of the Section 4 ablations. The owner account could not be tied to any author, so it is not cited, and nothing in this card was checked against it.

Pitfalls

• Training budget conflicts. The appendix gives 5,000 steps with a 2M-token batch app. A, tab. 9, which is about 10B tokens. Section 4 and the tables say 100B tokens §4 tab. 3. The paper does not resolve this.
• Learning-rate schedule in Table 3. The "Minimal Learning Rate" rows are not labelled. The baseline row lists 3e-5, which matches the 0.1 minimum ratio in the appendix. The 3e-4 row presumably means no decay, but the paper does not say so tab. 3 app. A, tab. 9.
• Eq. 20 to eq. 21 does not follow as printed. Eq. 21 writes lambda * sqrt(d) * cos(s, h). With ||h̃|| = sqrt(d) from RMSNorm and a unit s⋆, (s⋆ᵀ h̃)² equals d·cos², not sqrt(d)·cos eq. 19–21 eq. 23 §2. This is our arithmetic from the paper's own definitions. Check it before relying on that step.
• RMSNorm details. The formula omits the epsilon and the learnable gain, saying the gain can be absorbed into the next linear layer §2 eq. 5. The paper does not say what epsilon any implementation should use.
• Sink-ratio length. The text uses T = 64 for every reported sink ratio app. A. The sentence just above it says the chunk size matches each model's context window, which reads differently. Use T = 64 to reproduce the tables.
• Context-length ablation. The min/max values are the range of sequence positions over which the loss is computed, not the sequence lengths §4.3.3. Whether shorter sequences are truncated or packed is not stated.
• Model count mismatch. The appendix caption says 12 open models were evaluated app. C, fig. 7. The model table lists seven app. C, tab. 10, and the magnitude plots in the paper cover seven models across fig. 1 and fig. 7.
• Sandwich (QK) naming. The table row "Sandwich (QK)" tab. 5 does not match the text's description of a QKNorm variant that normalizes only queries and keys §4.2.2. The exact configuration is not stated.
• Head-count ablation. When the number of heads changes at fixed head dimension tab. 6, the total attention width changes too. The paper does not say whether hidden size or other settings were adjusted.
• Gate details. The gated-attention variants are named by granularity (channel, head, single) and by what the gate depends on tab. 7. The gate function and its placement are not given here; the cited gated-attention work is the reference §4.3.2.
• Spike measurement. "Spike" is the maximal activation magnitude in the ablations §4. The tensor, layer and token set are not stated, so the absolute values are not comparable across papers.
• Table 2 rounding. The Qwen2.5 percentages do not match their own counts: 149,587 / 152,064 is 98.37% (7B), and 149,645 / 152,064 is 98.41% (14B), while the table gives 98.40% for both tab. 2.
• Approximation, not identity. The quadratic form for the feed-forward block holds only where SiLU is near the identity, which the paper observes for spike tokens §3.1 fig. 2 app. B. It is not exact for other tokens.
• Single threshold. Only eps = 0.3 is used for the sink ratio, with no sweep app. A. Results are sensitive to that choice in ways the paper does not measure.
• No variance. Every ablation is a single value. The paper reports no seeds, error bars or repeats, so small differences (for example 10.0 vs 10.1 perplexity) should not be read as significant.

Lineage

• Builds on: Sun et al., "Massive Activations in Large Language Models" (COLM), which characterized massive activations; this paper follows it for tracking magnitudes and for the first-token and delimiter finding §1 §3.1. Xiao et al., "Efficient streaming language models with attention sinks" (ICLR), for the attention-sink term §1. Gu et al., "When Attention Sink Emerges in Language Models" (ICLR), for the sink-ratio measure app. A. Xiong et al., on pre-norm placement §1. Yu et al., "The super weight in large language models", arXiv:2411.07191, for the anomalous down-projection entries and collinear gate and up rows that the paper also reports §3.1.
• Ablation targets: Qiu et al., gated attention (NeurIPS) §4.3.2. Owen et al., "A Refined Analysis of Massive Activations in LLMs", arXiv:2503.22329 §4.2.2. Zhu et al., "Transformers without normalization" (CVPR), for DynamicTanh §4.2.2. Xiao et al., "DuoAttention", arXiv:2410.10819, for sinks and short-range attention §4.3.3. Ding et al., CogView, for sandwich normalization §4.2.2. Olmo 3 for QKNorm, arXiv:2512.13961 §4.2.2.
• Compared against: the paper compares variants against its own baseline, not against other methods. Related work is in §5.
• Superseded by: not stated.