--- kind: concept slug: self-attention title: Self-attention summary: >- Self-attention lets every token in one sequence weigh the other tokens of that sequence when building its new representation. Variants differ in who may attend to whom (full, windowed, sparse, linear, masked, shared, memory-backed or gated), and that choice sets memory cost, data needs and how easily a pretrained model can be extended. tags: [attention, transformer, efficiency, multimodal] papers: ["1804.00247", "1902.02181", "1904.02874", "1904.12304", "1910.10536", "2011.14027", "2012.06678", "2012.10638", "2101.01169", "2101.08779", "2101.12037", "2103.05247", "2103.12340", "2103.12731", "2103.14030", "2104.00680", "2104.10157", "2105.03889", "2105.13677", "2106.00908", "2106.01342", "2106.05234", "2106.09681", "2106.11539", "2107.14293", "2108.11084", "2109.02497", "2109.03201", "2109.07644", "2111.09883", "2111.13587", "2111.13680", "2201.01293", "2201.07284", "2202.06709", "2202.07925", "2202.10447", "2203.01502", "2203.08913", "2203.10638", "2203.16194", "2203.16952", "2204.02174", "2204.07756", "2204.13154", "2205.01138", "2206.10589", "2209.01206", "2211.09552", "2301.07093", "2303.00748", "2303.07399", "2305.07027", "2306.00864", "2311.17633", "2312.00752", "2312.02133", "2312.06662", "2401.00423", "2403.06764", "2405.14458", "2405.21060", "2407.08083", "2407.08608", "2412.13802", "2606.26485", "2609.16066"] related: [attention-mechanism, attention-interpretability, vision-transformer, convolutional-neural-network, state-space-model, efficient-vision-backbone, semantic-segmentation, time-series-forecasting, grouped-query-attention, multi-query-attention, kv-cache-management, language-model-pretraining, linear-attention, fused-attention-kernel, low-precision-arithmetic, neural-operator, multimodal-language-model, latent-diffusion, positional-encoding, relative-positional-encoding, nearest-neighbor-search, graph-neural-network, conditional-random-field, generative-adversarial-network, spectral-normalization, transfer-learning, parameter-efficient-fine-tuning, multi-view-representation, adaptive-instance-normalization, text-to-image-personalization, ddim-inversion, monocular-depth-estimation, instance-segmentation, self-supervised-learning, multi-layer-perceptron, masked-language-modeling, anomaly-detection, multiple-instance-learning, siamese-network, meta-learning, recurrent-neural-network, object-detection, multimodal-fusion, vision-language-pretraining, motion-prediction, collaborative-perception, bird-eye-view-perception, graph-transformer, molecular-representation-learning, sliding-window-attention, contrastive-learning, tabular-foundation-model, cost-volume, optical-flow, coordinate-classification, time-series-classification, sequence-labeling, keyphrase-extraction, image-denoising, image-deblurring, image-super-resolution, shifted-window-attention] status: draft updated_at: 2026-11-01 --- ## Overview Self-attention is attention where the queries, keys and values all come from the same set of tokens. The survey defines it as the case where queries and keys share one sequence, and notes that it suits tasks whose output is not a sequence, such as classification or recommendation [arxiv:1904.02874 §4.1]. The shared invariant of every variant is the same: each output is a weighted sum of value vectors, with weights computed from scores between tokens [arxiv:1904.02874 eq. 1] [arxiv:1904.02874 §4.1]. An NLP survey uses the same definition, where the input sequence serves as both keys and query, and notes that some architectures use no query at all [arxiv:1902.02181 §IV-A]. A 2022 survey treats the Transformer's learned projections of queries, keys and values as the main form, and names the quadratic time and memory cost as the problem most later variants try to reduce [arxiv:2204.13154 §5] [arxiv:2204.13154 §5.3]. One author group reads the effect as spatial smoothing: each output is a weighted average of other positions with learned, positive weights. They argue this lowers feature-map variance and helps optimisation [arxiv:2202.06709 app. B] [arxiv:2202.06709 §3]. That is their interpretation, not a settled view. What the cards show is that "self-attention" covers several different restrictions on who may attend to whom. It also covers attention across the rows of a batch, learned biases added to the scores, exact kernels that compute the same result faster, token pruning between layers, attention between channels instead of tokens, token mixers that replace attention, memories of past keys, attention through anchor tokens, attention inside periods found by a Fourier transform, and shared or masked attention that changes during decoding. Those choices decide the cost (full attention is quadratic in the number of tokens), the data needs, and whether a frozen model can be extended. Some variants also change the score function (cosine scores with a learned scale) or how many attention layers a block holds. The variants below are grouped by that choice rather than by paper. ## Variants ### Full self-attention Every token attends to every other token. The survey states that decoder self-attention is masked so a position cannot see later positions, and that the cost is O(n²) in time and memory [arxiv:1904.02874 §5.2] [arxiv:1904.02874 §8.8]. Card: [An Attentive Survey of Attention Models](/papers/1904.02874). The NLP tutorial gives the same causal rule for decoder self-attention, where position i sees only positions up to i [arxiv:2311.17633 §2.6]. Its card is [Introduction to Transformers: an NLP Perspective](/papers/2311.17633). C-Tran runs full attention over one joint sequence: 324 image tokens and one token per label, where each label token carries a state of unknown, negative or positive [arxiv:2011.14027 §3.1] [arxiv:2011.14027 §3.2]. The paper reports a practical limit of about 2,000 labels on four Titan X GPUs, which it ties to the memory cost of the full token set [arxiv:2011.14027 §4.4]. Card: [General Multi-label Image Classification with Transformers](/papers/2011.14027). VideoGPT's prior is a causal transformer over discrete video codes. The paper uses Sparse Transformers with local and strided attention for it [arxiv:2104.10157 §4.1]. The official code uses full attention by default and needs an extra install for the sparse option, so the two differ in practice. Card: [VideoGPT](/papers/2104.10157). Full attention also appears where the token set is small or fixed. The non-local block in BCNet gives every node of a 14×14 region feature an affinity to every other node in that region, normalised by a softmax [arxiv:2103.12340 §3.2] [arxiv:2103.12340 eq. 2]. Its ablation gives 29.63 COCO-OCC AP for a single graph layer, against 28.43 for a single fully convolutional layer, and 30.68 for the bilayer graph form [arxiv:2103.12340 tab. 3]. Card: [BCNet](/papers/2103.12340). RL-GAN-Net's latent GAN puts a self-attention block in its generator and discriminator, with spectral normalisation [arxiv:1904.12304 §3.2]. The card records that the released code instantiates two discriminator attention blocks where the paper's table lists one. Card: [RL-GAN-Net](/papers/1904.12304). STraTS runs full self-attention over the set of observation triplets in an ICU stay, with no imputation. Attention cost grows with the number of observations, which is why the paper caps them for MIMIC-III [arxiv:2107.14293 §4.4]. On MIMIC-III 24-hour mortality its PR-AUC is 0.577, against 0.559 for the best baseline [arxiv:2107.14293 tab. 4]. Card: [STraTS](/papers/2107.14293). AI Choreographer's FACT model applies full attention to music and motion in one stack. Motion tokens and then audio tokens form one sequence, and no attention mask is used, so every token attends to every input token [arxiv:2101.08779 §4.1]. The model reads a 120-frame seed and a 240-frame music window, and is trained to output the next 20 frames, not only the next one [arxiv:2101.08779 §5.2.1]. The authors' ablation changes attention and supervision together. Full attention with one-frame targets scores worse on FID_k than the causal baseline with shift-by-one targets (207.74 against 111.69), so full attention alone does not explain the gain [arxiv:2101.08779 tab. 3]. Their early fusion layout gives FID_k 35.35, against 45.76 for late fusion and 45.66 for a motion-only transformer [arxiv:2101.08779 tab. 4]. These are the authors' numbers on their AIST++ split. Card: [AI Choreographer](/papers/2101.08779). SimADFuzz uses a Transformer encoder as a learned scorer in a genetic fuzzer. It reads the per-frame states of every vehicle in one simulated run, so self-attention runs across the frames of that run [arxiv:2412.13802 §4.2.1]. The paper's table lists three layers and three heads, while the released code builds two layers [arxiv:2412.13802 tab. 4]. The paper mentions no positional encoding and the code adds none, so frame order reaches the model only through the attention pattern [arxiv:2412.13802 §4.2.1]. The card's results include no run that removes the encoder alone, so its share of the violation count is not isolated [arxiv:2412.13802 §5.2.1]. Card: [SimADFuzz](/papers/2412.13802). BENDR runs a standard transformer over a sequence of EEG vectors. The sequence has 8 layers, 8 heads, width 1536 and feed-forward width 3076 [arxiv:2101.12037 §2.3]. The encoder produces about 2.67 vectors per second of signal, and a 60-second pre-training window yields 160 steps through the transformer [arxiv:2101.12037 §2.3] [arxiv:2101.12037 §2.4.1]. Pre-training masks spans of those vectors and scores each masked output against 20 distractors with a contrastive loss [arxiv:2101.12037 eq. 2]. In the paper's best downstream setting for most tasks, the transformer is skipped and a classifier reads pooled vectors [arxiv:2101.12037 §2.4.2] [arxiv:2101.12037 tab. 2]. So the pre-trained attention layers are not always the part that does the work at test time. Card: [BENDR](/papers/2101.12037). A Transformer encoder can also read a satellite pixel's series as a sequence of dates. Each date is a vector of 13 Sentinel-2 bands, and the series is 70 dates long [arxiv:1910.10536 §4.6]. A positional encoding is added, because self-attention alone ignores order [arxiv:1910.10536 §4.2]. Every date attends to every other date. The paper's equation scales the scores by the square root of the model width, while the card records that the released code scales by the per-head key size instead [arxiv:1910.10536 eq. 4]. On raw data with 23 classes, the author-reported kappa is 0.71 for the Transformer, equal to the LSTM, and above 0.69 for MS-ResNet and 0.67 for TempCNN [arxiv:1910.10536 tab. 1]. Cleaning the data raised every model's score, and the cleaned-data models sit within a few points of each other [arxiv:1910.10536 §8.1]. Card: [Self-attention for raw optical Satellite Time Series Classification](/papers/1910.10536). ### Structural bias added to the scores Graphormer keeps full attention over the nodes of a graph and adds structural terms. A learned degree embedding is added to each node's input features [arxiv:2106.05234 eq. 5]. For each ordered pair of nodes, a learned scalar for their shortest-path distance is added to the attention score, and a further term averages learned weights over the edge features on one shortest path [arxiv:2106.05234 eq. 6] [arxiv:2106.05234 eq. 7]. A virtual graph token, connected to every node, carries the graph-level output [arxiv:2106.05234 §3.2]. In its PCQM4M-LSC ablation, validation MAE is 0.2276 with no node-relation encoding and 0.1304 with spatial distance, centrality and the attention-bias edge term together [arxiv:2106.05234 tab. 5]. The full model reaches 0.1234 against 0.1395 for GIN-VN [arxiv:2106.05234 tab. 1]. These are the authors' numbers on one benchmark. The cost stays quadratic in the number of nodes, which the paper names as a main limit [arxiv:2106.05234 §6]. Card: [Do Transformers Really Perform Bad for Graph Representation?](/papers/2106.05234). ### Separate attention maps per modality, with a shared spatial term DocFormer encodes three inputs for each page: a grid of image features, a word-piece text sequence, and per-word layout features [arxiv:2106.11539 §3.1]. Each encoder layer computes one attention map for the visual branch and one for the text branch, and adds their outputs [arxiv:2106.11539 §3.1] [arxiv:2106.11539 §3]. Each branch's score combines a content term, two relative-offset terms and a spatial term. The spatial term uses one pair of query and key matrices for both branches, so the two streams can learn cross-modal spatial interactions [arxiv:2106.11539 §3.1] [arxiv:2106.11539 supp. §6.5]. The relative-offset embedding is clipped to 8 tokens on each side, and the paper does not state its shape or the handling of longer offsets beyond that clip [arxiv:2106.11539 supp. §6.1]. In the authors' ablation on FUNSD, F1 goes from 61.56 for text-only BERT-base to 73.01 with spatial features and 76.90 with multi-modal self-attention. On CORD the same steps give 89.23, 92.28 and 93.36 [arxiv:2106.11539 tab. 9]. These are author-reported numbers. Card: [DocFormer](/papers/2106.11539). ### Intra-modal and cross-modal terms in one joint sequence IRENE turns a chest radiograph, a chief-complaint text, laboratory values, sex and age into one token sequence [arxiv:2306.00864 p. 9]. Each of two bidirectional blocks gives each modality attention over its own tokens, plus an attention term over the other modality's tokens, weighted by λ = 1.0 [arxiv:2306.00864 p. 10]. The two streams are then concatenated, and ten self-attention blocks with 12 heads run over the joint sequence [arxiv:2306.00864 p. 10]. In the authors' ablation on the eight pulmonary diseases, the full model reaches mean AUROC 0.924. Replacing the bidirectional blocks with self-attention gives 0.858, and attention from text to image only gives 0.884 [arxiv:2306.00864 tab. 3]. The method assumes every modality is present for each patient [arxiv:2306.00864 p. 7]. These are author-reported figures, on a cohort whose data the authors do not release [arxiv:2306.00864 p. 12]. Card: [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864). ### Attention across the columns of one table row TabTransformer makes each categorical column of a row one token. Each token is a column identifier concatenated with a learned class vector, and the columns get no positional encoding because they have no order [arxiv:2012.06678 §2]. Stacked Transformer layers then make each column's vector depend on the other columns of the same row [arxiv:2012.06678 eq. 2]. The reported setting is embedding size 32, six layers and eight heads [arxiv:2012.06678 §3] [arxiv:2012.06678 app. B.1]. On 15 binary datasets the authors report mean AUC 82.8 against 81.8 for an MLP, and 82.9 for GBDT [arxiv:2012.06678 tab. 2]. Those are author-reported numbers, and the paper's dataset counts disagree in places [arxiv:2012.06678 app. C]. The same layers can be pretrained on unlabelled rows, by masking column values or by detecting replaced values. In the authors' semi-supervised tables this helps on the larger datasets, but on the smaller ones an MLP with entropy regularisation is the best model at 50 labels [arxiv:2012.06678 tab. 3] [arxiv:2012.06678 tab. 4]. Card: [TabTransformer](/papers/2012.06678). ### Attention across the rows of a batch SAINT adds a second attention step to each stage. After self-attention over the columns of one row, it flattens each row's column tokens into one vector and runs attention over the rows of the batch, so every row can attend to every other row [arxiv:2106.01342 §3.2] [arxiv:2106.01342 alg. 1]. Its ablation on the binary datasets separates the two parts. Column-only attention (SAINT-s) scores mean AUROC 92.59, row-only (SAINT-i) 93.09 and both (SAINT) 93.13, against 91.06 for XGBoost [arxiv:2106.01342 tab. 2]. The mean covers 14 binary datasets, and the table shows only some of them. With 50 labelled rows, pre-training on unlabelled rows lifts SAINT from 85.78 to 86.91 [arxiv:2106.01342 tab. 3]. Since the attention runs over the batch, a row's output depends on the other rows in its batch. The paper does not say how a single row is scored, so check the batching before comparing numbers (see the [SAINT](/papers/2106.01342) card). All of these are author-reported figures. ### Windowed and segmented self-attention Each token attends only to a local neighbourhood, so cost follows the window size rather than the full token count. ESRT splits the token sequence of a super-resolution feature map into contiguous segments and runs attention inside each one, so each attention matrix is (N/s)² rather than N², with s = 4 [arxiv:2108.11084 §3.3.2]. In its ablation at 48×48 training patches and batch 16, the one-block version uses 4191M of training GPU memory against 16057M for an unspecified plain Transformer, at a similar PSNR [arxiv:2108.11084 tab. 4]. Those figures are the authors' measurements. Card: [Transformer for Single Image Super-Resolution](/papers/2108.11084). Segmenting changes what can be attended. Each token sees only the band of the feature map that falls in its segment, so results can depend on the input size and on how the image is tiled at test time [arxiv:2108.11084 §3.3.2]. The card records that inference splits large inputs into overlapping pieces, which the paper itself does not describe. nnFormer uses local attention inside small 3D volumes in its encoder, with a second layer on a grid shifted by half the volume size. Local attention cost is linear in the number of voxel tokens for a fixed volume size, which is why the encoder uses it [arxiv:2109.03201 §3.2] [arxiv:2109.03201 eq. 2]. Card: [nnFormer: Interleaved Transformer for Volumetric Segmentation](/papers/2109.03201). GMFlow restricts attention to 2×2 windows over a 1/8-resolution feature map, and shifts the windows by half a window on every second layer so that neighbouring windows connect [arxiv:2111.13680 §3.2]. The appendix computes the global correlation in blocks; in one setting the blocked version took 59.5 ms against 52.6 ms unblocked [arxiv:2111.13680 app. B]. Card: [GMFlow: Learning Optical Flow via Global Matching](/papers/2111.13680). Swin-style local windows are described by the remote-sensing survey as having a cost that is linear in image size [arxiv:2209.01206 §4.2]. Card: [Transformers in Remote Sensing: A Survey](/papers/2209.01206). Swin Transformer is the reference for shifted windows. Its stages use 7 × 7 windows, and every second block moves the window grid by half a window, so neighbouring windows exchange information [arxiv:2103.14030 §3.1] [arxiv:2103.14030 §3.2]. The shifted layer rolls the feature map instead of padding it, which keeps the number of windows fixed, and a mask keeps tokens from different regions apart [arxiv:2103.14030 §3.2]. Removing the shift from Swin-T drops ImageNet-1K top-1 from 81.3 to 80.2 and COCO box AP from 50.5 to 47.7 [arxiv:2103.14030 tab. 4]. On a V100, the cyclic shift runs at 755 images per second against 183 for naive padding [arxiv:2103.14030 tab. 5]. Those speeds are the authors' measurements. Card: [Swin Transformer](/papers/2103.14030). HaloNet builds a whole image backbone from local attention and drops the per-window masks used by SASA. The image is cut into b × b query blocks, and each block attends to one shared memory of (b + 2h) × (b + 2h) pixels, so the windows are not centred on each pixel [arxiv:2103.12731 §3.2]. The authors report 84.9% top-1 on ImageNet for their largest model trained from scratch [arxiv:2103.12731 §4.1]. Enlarging the window from 6 × 6 to 12 × 12 gives a 1.3% gain, and an unmasked window beats the masked 7 × 7 window of SASA, with no values given in the text [arxiv:2103.12731 §4.3]. Pure attention models train more slowly than the matched EfficientNets on the authors' TPUs [arxiv:2103.12731 §5]. Card: [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](/papers/2103.12731). Window size matters less than it seems at one input size. MambaVision uses windows of 14 and 7 for its two late stages at 224 input. Those windows cover each stage's whole map, so attention there is global. Its ablation with windows 7 and 7 instead gives 82.2% top-1 against 82.3% at 6318 against 6298 images per second [arxiv:2407.08083 app. A] [arxiv:2407.08083 tab. S.1]. Card: [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](/papers/2407.08083). W.A.L.T alternates two window types in its video diffusion transformer. A spatial layer attends within one latent frame, in windows of 1 × 16 × 16 tokens. A spatiotemporal layer attends across latent frames, in windows of 5 × 8 × 8 [arxiv:2312.06662 §4.2] [arxiv:2312.06662 §5.2]. The paper reports up to 2× more steps per second from local windows on UCF-101 [arxiv:2312.06662 tab. 3]. That is an author measurement on one setup. Card: [Photorealistic Video Generation with Diffusion Models](/papers/2312.06662). NeW CRFs restricts attention to non-overlapping 7×7-patch windows in its depth decoder. A second pass moves the window grid by half a window, so neighbours across a border are linked [arxiv:2203.01502 §3.2]. Its window-size ablation, retrained on 352×352 crops, gives Abs Rel 0.055 at window 7/8 in 124 ms and 0.054 at window 32 in 192 ms. Memory rises from 2.7 G to 5.1 G, and window 48 runs out of memory in training [arxiv:2203.01502 tab. 6]. The paper does not say how feature maps that are not a multiple of the window size are padded [arxiv:2203.01502 §3.2]. Card: [NeW CRFs](/papers/2203.01502). ActionFormer applies local self-attention to the one-dimensional feature sequence of a video, with a window of W steps around each position. The same W is used on every level of its pyramid [arxiv:2202.07925 §3.1]. The cost falls from O(T²D + D²T) to O(W²TD + D²T) [arxiv:2202.07925 §3.1]. The reported window is 19 steps on THUMOS14, 9 on EPIC-Kitchens and 11 on ActivityNet [arxiv:2202.07925 §4.1] [arxiv:2202.07925 §4.2] [arxiv:2202.07925 §4.3]. In its THUMOS14 ablation, windows of 9, 19, 25 and 37 steps and full attention all score between 66.4 and 66.8 average mAP. Full attention runs at 2.2× the convolutional baseline's time and window 19 at 2.0× [arxiv:2202.07925 tab. 3b]. On this benchmark, then, the window size changed accuracy very little. Card: [ActionFormer](/papers/2202.07925). ### Attention over non-empty voxels in a neighbourhood VoTr runs attention over the non-empty voxels of a 3D grid rather than over every cell. Each query attends to voxels in a local range and in dilated rings whose steps grow with distance, with at most 48 attending voxels per query on KITTI [arxiv:2109.02497 §3.2] [arxiv:2109.02497 §3.3] [arxiv:2109.02497 §4.1]. The authors state that non-empty voxels are under 0.1% of the grid on Waymo, and about 90k per frame, which rules out dense attention [arxiv:2109.02497 §1]. Neighbour lookup uses a GPU hash table, so empty cells are rejected without a scan [arxiv:2109.02497 §3.4]. On KITTI validation, moderate car AP_3D is 75.48 with local attention alone and 78.27 with the dilated rings added [arxiv:2109.02497 tab. 4]. Capping the attending voxels at 24, 32 and 48 gives 77.08, 77.72 and 78.27 [arxiv:2109.02497 tab. 6]. The paper gives the ranges only for KITTI [arxiv:2109.02497 app. B]. On Waymo LEVEL_1 3D mAP, VoTr-TSD scores 74.95, against 70.3 for PV-RCNN as reported and 74.06 for the authors' re-implementation [arxiv:2109.02497 tab. 1]. VoTr-SSD runs at 14.65 Hz against 20.73 Hz for SECOND [arxiv:2109.02497 tab. 8]. Card: [Voxel Transformer for 3D Object Detection](/papers/2109.02497). ### Cosine-normalised scores with post-norm for scaling Swin V2 keeps the Swin layout and changes how scores are formed when the model grows to 3 billion parameters and 1,536-pixel inputs [arxiv:2111.09883 §3.5] [arxiv:2111.09883 §4.2]. Queries and keys are normalised, so their product is a cosine, and the scores are divided by a learned scale τ per head and layer, kept above 0.01 [arxiv:2111.09883 §3.2] [arxiv:2111.09883 eq. 2]. The layer norm moves to the output of each residual branch [arxiv:2111.09883 §3.2]. These two changes add 0.2 top-1 on Swin-T (81.5 to 81.7) and 0.5 on Swin-B (83.6 to 84.1) in the authors' ImageNet-1K ablation [arxiv:2111.09883 tab. 6]. The plain post-norm baseline diverges at the default learning rate, so it was run at a quarter of that rate [arxiv:2111.09883 tab. 7]. The same paper replaces the learned bias table with a small network over log-spaced offsets, so a model trained at one window size can run at another [arxiv:2111.09883 §3.3] [arxiv:2111.09883 eq. 4]. Card: [Swin Transformer V2](/papers/2111.09883). ### Axial attention Axial attention runs attention along one axis at a time, such as along time or along height, instead of across the whole grid. VideoGPT's autoencoder uses it. Its BAIR ablation shows a small gain from adding it: NMSE 0.0033 against 0.0041 without attention, and R-FVD 14.9 against 15.3 [arxiv:2104.10157 tab. 3]. Card: [VideoGPT](/papers/2104.10157). ### Sparse and linear approximations These keep the same token set but cut the number of score pairs. Criss-cross attention, used inside CNNs, keeps a sparse map on a cross-shaped path through each position. A non-local block instead builds a dense map at O(N²) cost. The survey reports the criss-cross cost as O(2√N) and reports 11 times less GPU memory [arxiv:2101.01169 §3.1] [arxiv:2101.01169 §3.1.1]. These are the survey's reported figures. LogSparse uses convolutional queries and keys and attends at exponentially spaced offsets. The cited work states that memory per layer falls from O(L²) to O(L log L) [arxiv:2205.01138 §4.2]. Informer's ProbSparse attention keeps only a subset of queries, and the tutorial notes its selection rule is worded inconsistently [arxiv:2205.01138 §4.1]. Card: [Transformers in Time-series Analysis: A Tutorial](/papers/2205.01138). Efficient (linear) attention rearranges the product so that the cost falls from O(n²) to O(n). The survey writes this as ρ_q(Q)(ρ_k(K)ᵀV), with a normalisation on each factor. It also gives axial attention at O(hwm) and clustered attention at O(qc) [arxiv:2204.07756 §2.1.3]. Card: [Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756). These costs are the survey's figures and are not measured there. LoFTR uses linear attention at its coarse 1/8 scale, with the softmax replaced by φ(x) = elu(x) + 1 [arxiv:2104.00680 §3.2]. The authors call full attention impractical at that grid size. Self and cross attention alternate four times [arxiv:2104.00680 §3.6]. In the ScanNet ablation, replacing the module with a convolution of similar size drops pose AUC@10 to 32.04, against 40.8 for the full model [arxiv:2104.00680 tab. 6]. The ablation's full row does not match the main table's OT row, so compare within the ablation only [arxiv:2104.00680 tab. 2]. Card: [LoFTR](/papers/2104.00680). TransMIL approximates attention over whole-slide bags of thousands of patches with the Nyström method. It picks m landmark tokens from the queries and keys, and the paper states the cost falls from O(n²) to O(n) [arxiv:2106.00908 §3.3] [arxiv:2106.00908 eq. 9]. Its two attention layers reach AUC 0.9309 on CAMELYON16, against 0.8880 for the best compared method, MIL-RNN [arxiv:2106.00908 tab. 1]. The landmark count, pseudo-inverse iterations and head count are not stated in the paper [arxiv:2106.00908 app. B]. Card: [TransMIL](/papers/2106.00908). GRL's anchored stripe attention is a bottleneck of the same kind. Within each stripe, a few anchor tokens are average-pooled from the feature map and then projected. The anchors read the keys and collect the values, and then each query reads the anchors [arxiv:2303.00748 §4] [arxiv:2303.00748 eq. 3] [arxiv:2303.00748 eq. 4]. The cost is stated as O(NMd) time and O(NM) space, where M is the number of anchors [arxiv:2303.00748 §3.2]. The stripes alternate between horizontal and vertical from layer to layer [arxiv:2303.00748 §3.3]. The paper does not state the pooling factor [arxiv:2303.00748 §4]. On GoPro deblurring the authors report 33.93 PSNR for GRL-B against 32.92 for Restormer [arxiv:2303.00748 tab. 2]. The card lists where the released code differs from the paper, including the score function, so read it before reproducing those numbers. Card: [Efficient and Explicit Modelling of Image Hierarchies for Image Restoration](/papers/2303.00748). Reduced keys and values keep every query but shrink the key and value set. ResT pools the token grid with a strided depth-wise convolution before it computes keys and values. The reduction is 8, 4, 2 and 1 across its four stages [arxiv:2105.13677 §2.2] [arxiv:2105.13677 tab. 1]. Among the authors' reduction tests, the depth-wise convolution beats average and max pooling [arxiv:2105.13677 tab. 6]. On ResT-Lite, under the authors' simpler recipe, the reduced attention reaches 72.88 top-1 against 72.68 for standard multi-head attention. Throughput is 1246 against 512 images per second, but the batch sizes differ, so the speed gap is not a clean comparison [arxiv:2105.13677 tab. 9] [arxiv:2105.13677 app. A.3]. Card: [ResT](/papers/2105.13677). ChangeFormer's encoder shortens keys and values by a reduction ratio R, which lowers the cost from O((HW)²) to O((HW)²/R) [arxiv:2201.01293 §2.1.1] [arxiv:2201.01293 eq. 2]. The paper does not give R. Its equations also reduce the queries, which would change the output length and break the residual, so the card follows the released code, which keeps full-length queries [arxiv:2201.01293 §2.1.1]. On LEVIR-CD the authors report F1 90.40 against 89.31 for BIT, a transformer change detector with a ResNet18 encoder [arxiv:2201.01293 tab. 1]. Card: [A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293). Linear attention with a decay is the case where the decay is one scalar per step. Causal linear attention is the special case with no decay [arxiv:2405.21060 §5.2]. A decay mask of that form turns masked attention into a state-space recurrence. The chunked algorithm uses matrix products inside blocks of the sequence and a small recurrence between blocks [arxiv:2405.21060 §6]. The authors report 2–8× speed over Mamba's scan at state size 64, and faster than FlashAttention-2 from sequence length 2K [arxiv:2405.21060 fig. 10] [arxiv:2405.21060 §9.3]. Keeping one scalar decay per head excludes Mamba's per-channel decays [arxiv:2405.21060 §5]. Card: [Transformers are SSMs](/papers/2405.21060). FLASH uses exact attention inside fixed-size chunks and linear attention between them, so the cost grows linearly with length for a fixed chunk size [arxiv:2202.10447 §3.2]. Each block has one attention head with a squared-ReLU kernel, and a learned gate multiplies the attended values [arxiv:2202.10447 §2] [arxiv:2202.10447 eq. 3] [arxiv:2202.10447 eq. 5]. The authors' ablation finds that a softmax kernel or several heads each worsen perplexity [arxiv:2202.10447 tab. 1]. Across chunks, a running sum of key and value products stands in for pairwise scores. For language modelling the sum covers only earlier chunks, so the model stays causal [arxiv:2202.10447 eq. 8]. Decoding needs a constant amount of work per step [arxiv:2202.10447 §3.2.1]. The authors report 15.11 perplexity against 15.25 for Transformer++ at context 8192 on Wiki-40B, and 62.5 ms per step against 261.0 ms on 64 TPU-v4 cores [arxiv:2202.10447 tab. 13]. Training-cost speedups range from 1.2× to 4.9× over contexts 512 to 8192 [arxiv:2202.10447 §4.2]. These are author-reported. There is no official code. The appendix listings divide the scores by the length, which the equations do not, and they apply a different segment mask [arxiv:2202.10447 app. D] [arxiv:2202.10447 eq. 5]. An implementation has to choose one reading and test it. Card: [Transformer Quality in Linear Time](/papers/2202.10447). ### Cross-covariance attention over channels Cross-covariance attention (XCA) changes the axis that is compared. Instead of an N × N map over tokens, it builds a d × d map over feature channels from the channel-normalised queries and keys. The authors state that its cost is linear in the number of tokens, which is why the map does not grow with the patch count [arxiv:2106.09681 §3.2]. Each head works on d/h channels, so each head's map is (d/h) × (d/h) [arxiv:2106.09681 §3.2]. XCA alone does not let neighbouring patches exchange information, so XCiT adds a depth-wise 3 × 3 convolution step on the patch grid after it [arxiv:2106.09681 §3.3]. The paper's equation and its appendix algorithm put the softmax on different axes, so the notation alone does not fix the operation. The card follows the released code, and advises checking one layer's output against it before relying on the paper's text [arxiv:2106.09681 eq. 1] [arxiv:2106.09681 alg. 1]. On ImageNet-1k, XCiT-S12 with 16 × 16 patches reaches 82.0 top-1 without distillation [arxiv:2106.09681 tab. 1]. In the authors' ablation, removing the XCA layer gives 75.9, removing the local patch step gives 80.8, and removing ℓ2 normalisation fails to train [arxiv:2106.09681 tab. 4]. On COCO, the same 16-patch model reaches 45.3 box AP against 46.0 for Swin-T, so it does not beat Swin-T there [arxiv:2106.09681 tab. 5]. At 1024 × 1024 and batch 64 on one V100, the authors measure 7312 MB peak memory and 37 images per second for XCiT-S12/16, against 26915 MB and 29 images per second for Swin-T [arxiv:2106.09681 tab. D.5]. These are the authors' measurements. Card: [XCiT](/papers/2106.09681). EdgeNeXt applies the same channel-wise form in a few blocks near the end of a mobile CNN-transformer hybrid, in a block it calls SDTA. Queries and keys are L2-normalised, and the attention matrix is C × C over channels, multiplied by V and added back to the input [arxiv:2206.10589 eq. 3] [arxiv:2206.10589 eq. 4]. The authors state its cost is linear in the number of tokens [arxiv:2206.10589 §3]. On ImageNet-1K at 256 px, the 5.6M-parameter EdgeNeXt-S reaches 79.4% top-1 with 1.30G MAdds, against 78.4% for MobileViT-S, whose spatial attention the authors argue is the main cost on mobile hardware [arxiv:2206.10589 tab. 2] [arxiv:2206.10589 tab. 3] [arxiv:2206.10589 §2]. These are the authors' measurements. Card: [EdgeNeXt](/papers/2206.10589). ### Softmax-free operator attention The trunk of the transformer-enhanced neural operator replaces the softmax with a summary. Keys and values are summed over all query points first, and each query is then multiplied by that summary, with a point-count factor [arxiv:2609.16066 §3.1] [arxiv:2609.16066 eq. 3]. Keys and values carry trainable layer norm, and the paper writes no output projection after the attention [arxiv:2609.16066 §3.1]. On Rotor 37 the author-reported mean field error is 0.083% against 0.572% for FNO [arxiv:2609.16066 tab. 4], with 154k parameters [arxiv:2609.16066 tab. 3]. The paper's branch-trunk combination is stated differently in its equation and its text [arxiv:2609.16066 §3.1]. The result comes from one compressor and a validation split only. Card: [Transformer-enhanced neural operator](/papers/2609.16066). ### Token mixers that replace attention Some models drop the attention step and mix tokens another way. AFNO mixes tokens in the 2D Fourier domain: it transforms the grid, applies a learned block-diagonal channel mix to each frequency, shrinks weak frequencies to zero, and transforms back [arxiv:2111.13587 §4]. On ViT-B/4 inpainting its PSNR is 27.05 against 27.06 for self-attention, at 257.2 against 357.2 GFLOPs [arxiv:2111.13587 tab. 2]. On Cityscapes at 1024 × 1024 with SegFormer-B3, the authors report 80.9 mIoU, against 79.7 for efficient self-attention and 80.5 for Long-Short. Self-attention cannot run at that size in their setup because of memory [arxiv:2111.13587 tab. 4] [arxiv:2111.13587 §5.6]. A reading of the released code suggests it does not reproduce the paper's hard frequency truncation, so a paper-faithful implementation may not match it [arxiv:2111.13587 §5.3]. Card: [Adaptive Fourier Neural Operators](/papers/2111.13587). Mamba drops attention and the feed-forward block altogether. Its step size, input matrix and output matrix depend on the current token, while the state matrix stays fixed [arxiv:2312.00752 §3.2]. Because these parameters vary with the token, the layer has no convolution form, and the authors compute it with a fused parallel scan on the GPU [arxiv:2312.00752 §3.3]. On a synthetic Selective Copying task the input-dependent layer reaches 99.8% accuracy, and the fixed-parameter S4 layer in the same block reaches 56.4% [arxiv:2312.00752 §4.1]. On the zero-shot suite, Mamba-1.4B averages 59.7 against 55.2 for Pythia-1.4B [arxiv:2312.00752 §4.2.2]. The paper's advice for raw audio is to keep fixed-parameter layers in the outer stages [arxiv:2312.00752 app. E.4.1]. Card: [Mamba](/papers/2312.00752). ### Dropping tokens between layers Token pruning removes tokens from the sequence after an early layer, so later layers attend over fewer of them. FastV, for vision-language models, runs the first K layers in full, ranks the image tokens by the attention they receive, and keeps a share of them. It needs no training [arxiv:2403.06764 §4.1]. At K = 2 and R = 50%, the authors report average scores of 69.7 against 69.8 for LLaVA-1.5-7B, and 73.6 against 73.6 for LLaVA-1.5-13B, at about 55% of the theoretical FLOPs [arxiv:2403.06764 tab. 1]. In their retrained-baseline ablation, random ranking at the same setting scores lower on Nocaps and Flickr30k [arxiv:2403.06764 tab. 7]. The wall-clock timing comes from one test on a single A40 with one-token answers [arxiv:2403.06764 tab. 4]. The paper's ranking description and the released code use different attention quantities, so check which one you implement. Card: [FastV](/papers/2403.06764). ### Exact kernels An exact kernel returns the same output as standard attention and changes only how the work is scheduled on the chip. FlashAttention-3 overlaps data loads, matrix multiplies and softmax across thread groups on NVIDIA Hopper GPUs. The authors report 1.5–2.0× forward speed over FlashAttention-2 in FP16, reaching 740 TFLOPs/s [arxiv:2407.08608 §4.1]. Removing the GEMM-softmax pipelining drops its ablation from 661 to 582 TFLOPs/s [arxiv:2407.08608 tab. 2]. Its FP8 path has RMSE 9.1e-3 against 2.4e-2 for a per-tensor baseline, on synthetic inputs with rare outliers [arxiv:2407.08608 tab. 3] [arxiv:2407.08608 §4.3]. The paper's incoherent processing step for FP8 is not in the released code at the commit the card pins. Card: [FlashAttention-3](/papers/2407.08608). ### Self-attention across agents at each location OPV2V stacks the feature maps of the ego vehicle and its neighbours, then treats the vehicles as the tokens at each spatial position. Attention runs over vehicles, not over pixels, and the ego keeps only its own output row [arxiv:2109.07644 §4]. The mean number of connected vehicles per frame is 2.89, between 2 and 7 [arxiv:2109.07644 tab. 3]. With so few tokens the N × N score matrix is small. Compressing the shared features before sending them is a separate step that the authors evaluate: about 3% AP lost at a 4096× compression rate with VoxelNet, which the card notes is a figure-only result whose setup the paper does not fully state [arxiv:2109.07644 §5.6] [arxiv:2109.07644 fig. 10]. Card: [OPV2V](/papers/2109.07644). V2X-ViT applies the same per-location idea to vehicles and roadside units. Its heterogeneous multi-agent self-attention computes attention only between features at the same spatial position across agents. Projections depend on the node type (vehicle or infrastructure), and a weight matrix depends on the edge type, of which there are four [arxiv:2203.10638 §3.2.1] [arxiv:2203.10638 eq. 1] [arxiv:2203.10638 eq. 2]. On the noisy V2XSet setting, adding this module to the window and split-attention parts raises AP@0.7 from 0.548 to 0.601 [arxiv:2203.10638 tab. 2]. Against the authors' noisy-setting baselines, the full model scores 0.614 at AP@0.7, against 0.487 for OPV2V and 0.402 with no fusion [arxiv:2203.10638 tab. 1]. These are author-reported figures on a simulated dataset. Card: [V2X-ViT](/papers/2203.10638). ### Self-attention across objects in each view Multi-View Transformer for 3D grounding treats the objects of a scene as tokens. In each of N rotated views, the object features go through a shared transformer decoder: self-attention among the objects, then cross-attention to the words [arxiv:2204.02174 §3.3] [arxiv:2204.02174 eq. 10]. The per-view outputs are combined by an order-independent average [arxiv:2204.02174 §3.1] [arxiv:2204.02174 eq. 11]. On Nr3D, a model trained and tested on one view reaches 46.2% overall. Training on four views and testing on one reaches 51.6% [arxiv:2204.02174 tab. 5]. The authors credit the second gain to multi-view training, not only to extra test views [arxiv:2204.02174 §4.4.2]. Card: [Multi-View Transformer](/papers/2204.02174). ### Attention within and across cost-volume tokens FlowFormer summarises the all-pairs cost volume of each source pixel into K = 8 latent tokens of width D = 128. Then three alternate-group layers apply two kinds of self-attention. Intra-cost-map attention runs over the K tokens of one pixel [arxiv:2203.16194 eq. 2]. Inter-cost-map attention runs, for each token index, over all H·W source pixels, using spatially separable self-attention [arxiv:2203.16194 eq. 3] [arxiv:2203.16194 §3.2]. The decoder's cross-attention is a separate step [arxiv:2203.16194 eq. 4]. The ablation, run on the RAFT-style schedule, shows that each part helps in a different way. Adding the intra-cost-map part alone raises Sintel train final error from 2.72 to 2.89 but lowers KITTI train F1-all from 16.16 to 15.71. The full stack with one layer gives 1.20 and 2.85 on Sintel, and three layers give 1.10 and 2.57 [arxiv:2203.16194 tab. 2]. The authors keep the transformer form for both parts, because MLP and ConvNeXt replacements help Sintel final but hurt KITTI [arxiv:2203.16194 app. §1]. The paper leaves the spatial window and stride of the inter-cost step unstated; the card records the code's values [arxiv:2203.16194 eq. 3]. Card: [FlowFormer](/papers/2203.16194). ### Query-restricted self-attention with a CLS query The Multimodal Fusion Transformer lets only one token, the classification token built from the second sensor, act as the query. Keys and values still come from all five tokens, so each block produces a single output row [arxiv:2203.16952 §III-B3] [arxiv:2203.16952 alg. 1]. With one query row the score matrix is 1 × N, not N × N. This is a cost observation from the equations, not a measured number. Card: [Multimodal Fusion Transformer](/papers/2203.16952). UniFormerV2 uses a learnable query in its deeper blocks. The query cross-attends to every token of a clip, at a cost that is linear in the number of tokens, which the authors contrast with global self-attention [arxiv:2211.09552 §3.2]. On Kinetics-400 the global block alone gives 84.4% top-1, against 83.1% with neither block. On Something-Something V2 it gives 63.3%, against 45.1% with neither block [arxiv:2211.09552 tab. 9]. On SSV2, one query beats four or sixteen: 69.5, 69.1 and 68.6 [arxiv:2211.09552 tab. 14]. Card: [UniFormerV2](/papers/2211.09552). ### Gated self-attention over extra tokens GLIGEN freezes a pretrained latent diffusion model and inserts a new self-attention layer over the visual tokens plus grounding tokens. Only the visual outputs are kept, and they are added back through a gate β·tanh(γ), where γ is a learned scalar initialised to 0 [arxiv:2301.07093 §4.2] [arxiv:2301.07093 eq. 8]. At initialisation the model therefore equals the pretrained one. The paper reports that the layers begin to attend to the boxes only after roughly 60k to 70k iterations, so short runs may show no grounding effect [arxiv:2301.07093 app. F]. Card: [GLIGEN](/papers/2301.07093). ### Gated attention unit over a small set of tokens RTMPose uses one gated attention unit (GAU) over the K keypoint tokens of its head. The unit normalises the tokens, produces one hidden projection that is split into a gate, a value and a shared base, and derives queries and keys from that base by per-dimension scale and offset [arxiv:2303.07399 §3.3] [arxiv:2303.07399 eq. 1]. The attention kernel is a squared ReLU, and the paper's constant n in front of it is not defined in the text [arxiv:2303.07399 eq. 2]. In the authors' cumulative COCO validation chain, the step that adds the GAU moves AP from 71.4 to 71.9 [arxiv:2303.07399 §3.3]. The chain is cumulative, so this is the gain after the earlier steps, not on its own. Card: [RTMPose](/papers/2303.07399). ### One attention layer per block, between feed-forward layers EfficientViT places a single self-attention layer in each block, with feed-forward layers before and after it [arxiv:2305.07027 eq. 1] [arxiv:2305.07027 §3.1]. Its cascaded group attention gives each head its own slice of the channels, and each head adds the previous head's output to its input [arxiv:2305.07027 eq. 3] [arxiv:2305.07027 §3.1]. In the authors' 100-epoch ablation, swapping the block for a Swin block costs 3.0 points of top-1 (68.3 against 71.3), and two or three feed-forward layers per side score 70.2 and 65.7 [arxiv:2305.07027 tab. 6]. The headline model, M5, reaches 77.1% top-1 at 10,621 images per second on a V100, against 75.2% and 7,560 for MobileNetV3-Large [arxiv:2305.07027 tab. 2]. The paper's text quotes some speed ratios that do not match its table, so use the table values. Card: [EfficientViT](/papers/2305.07027). ### Hybrid backbones: attention in the late layers only Hybrids keep convolution or a state space model for most layers and put self-attention where the global mixing matters most. MambaVision has four stages. Stages 1 and 2 are convolutional. In stages 3 and 4 the first half of the layers use its Mamba-style mixer and the last half use self-attention [arxiv:2407.08083 §3.2.2]. On ImageNet-1K top-1, attention in the last half gives 82.3%, against 81.5% for attention in the first half and 81.3% for a random pattern [arxiv:2407.08083 tab. 5]. These are the authors' numbers from one backbone and one training recipe. AlterNet, from the same family of ideas, is a ResNet in which some stage-end convolutional blocks are replaced by local attention blocks. The authors' figures suggest it beats CNNs on CIFAR-100, but the text gives no values [arxiv:2202.06709 §4.2] [arxiv:2202.06709 fig. 12]. Card: [How Do Vision Transformers Work?](/papers/2202.06709). YOLOv10 adds one partial self-attention block after its lowest-resolution stage. Only half of the channels pass through attention and a feed-forward network, and the two halves are fused again by a 1 × 1 convolution [arxiv:2405.14458 §3.2]. In the authors' cumulative ablation on the small model, it adds 1.4 AP on top of the large kernel (44.9 to 46.3) and 0.15 ms (2.34 to 2.49 ms) [arxiv:2405.14458 tab. 10]. A full transformer block in the same place gives 46.0 AP at 2.54 ms [arxiv:2405.14458 tab. 13]. Card: [YOLOv10](/papers/2405.14458). ### Parallel convolution and attention branches Conformer runs a convolutional branch and a transformer branch side by side, and couples them in every block through a Feature Coupling Unit. The CNN features are pooled into patch tokens, and the transformer tokens are upsampled back into the CNN [arxiv:2105.03889 §3.1] [arxiv:2105.03889 §3.2]. The transformer branch works on 14×14 patch embeddings with a class token, using standard multi-head self-attention over those tokens. It has no positional embedding, because the convolutional branch is said to supply location cues [arxiv:2105.03889 §3.2]. On the authors' ablation, the positional embedding gives Conformer-S 83.5 against 83.4 without it, and DeiT-S 79.8 against 77.4 [arxiv:2105.03889 tab. 6]. The authors report 83.4% ImageNet-1k top-1 for Conformer-S at 10.6 G MACs, against 81.8% for DeiT-B at 17.6 G [arxiv:2105.03889 tab. 2]. Their serial hybrid, a ResNet-26d followed by DeiT-S, scores 80.2 at 3.7 G MACs, against 81.9 for Conformer-S/32 at 7.0 G [arxiv:2105.03889 tab. 5]. The compute differs between those two, so that comparison does not isolate the structure. Card: [Conformer](/papers/2105.03889). ### Frozen self-attention reused on a new modality Universal computation keeps the self-attention and feedforward blocks of a GPT-2 model pretrained on text, and freezes them. Only the linear input and output layers, the layer-norm parameters and the position embeddings are trained [arxiv:2103.05247 §2.2]. The layer norms carry most of the adaptation. On CIFAR-10, training them alone reaches 54%, against 25% for output-only training [arxiv:2103.05247 tab. 15]. Language pretraining helps in this frozen setup: random initialisation reaches 75.8% on bit memory, against 100% [arxiv:2103.05247 tab. 2]. The authors also report that random transformers already do well on some tasks [arxiv:2103.05247 §3.2]. Card: [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247). ### Masked re-embedding of visited nodes A learned routing policy embeds the nodes once, although visited nodes stop mattering for later choices. The Embedding Glimpse layer in MDAM recomputes only the top attention layer, with attention to visited nodes set to minus infinity. It runs every p steps, with p between 2 and 8 depending on instance size, rather than at every step [arxiv:2012.10638 §3.2] [arxiv:2012.10638 §4]. The encoder is three blocks of multi-head self-attention, with eight heads and width 128 [arxiv:2012.10638 §3.1.1] [arxiv:2012.10638 §4]. On 100-node TSP with greedy decoding, removing the layer raises the gap from 2.19% to 2.78% [arxiv:2012.10638 tab. 2]. These are the authors' numbers for one setting. Card: [MDAM](/papers/2012.10638). ### Shared-reference attention across a batch StyleAligned lets the self-attention of every image in a batch also attend to the keys and values of one reference image. The queries and keys of each target are first normalised to the reference's per-channel statistics. The values are not normalised [arxiv:2312.02133 §3.2]. No weights are trained. The authors report DINO set consistency of 0.51. DreamBooth-LoRA reaches 0.537, but it needs training for each style, and removing the normalisation drops the score to 0.428 [arxiv:2312.02133 tab. 2]. Sharing every image with every other image, instead of with one reference, raises DINO to 0.55 and lowers CLIP text alignment from 0.287 to 0.28 [arxiv:2312.02133 tab. 2]. Card: [StyleAligned](/papers/2312.02133). ### Memory of past keys and values Memorizing Transformers adds a non-differentiable memory of past (key, value) pairs to one attention layer. Each query takes its top-k stored pairs by approximate nearest-neighbour search. A learned per-head gate then mixes that output with local causal attention [arxiv:2203.08913 §3.1] [arxiv:2203.08913 eq. 1] [arxiv:2203.08913 eq. 2]. The stored keys come from earlier parameters, so the authors pretrain with a small memory and fine-tune with a larger one [arxiv:2203.08913 §3.2] [arxiv:2203.08913 §4.5]. On arXiv at a 2048-token context, a 65K memory reaches perplexity 2.26. The Transformer-XL baseline at the same context reaches 2.42 [arxiv:2203.08913 §4.3]. The repository also contains a dense variant that is not the paper's kNN method, so the top-k path is the one to reproduce [arxiv:2203.08913 §3.1]. Card: [Memorizing Transformers](/papers/2203.08913). ### Window encoder with a second, conditioned pass TranAD reads a short window of multivariate readings with a transformer encoder and two decoders [arxiv:2201.07284 §3.3] [arxiv:2201.07284 eq. 6]. Its window encoder applies masked self-attention to the window and then attends over the first encoder's output [arxiv:2201.07284 §3.3] [arxiv:2201.07284 eq. 5]. Inference runs two passes: the first reconstruction error becomes the focus score that conditions the second pass [arxiv:2201.07284 §3.4]. In the authors' ablation, removing the transformer lowers F1 on NAB from 0.9364 to 0.8850, and F1* on SWaT from 0.8094 to 0.6360 [arxiv:2201.07284 tab. 6]. The paper's meta-learning step is a MAML update after each pass [arxiv:2201.07284 §3.4]. The released code does not contain that update, nor the adversarial sign, so the ablations for those parts cannot be checked against it. See the [TranAD](/papers/2201.07284) card for the line-by-line comparison. ### Per-word side features in the attention input A keyphrase tagger on tweets can take per-word reading features from the ZuCo corpus, EEG and eye-tracking values averaged over each word type. In the paper's self-attention variant the features are appended after the query vector [arxiv:2606.26485 §3.2.2]. That makes the query wider than the key, and the score in eq. 7 is undefined as written [arxiv:2606.26485 eq. 7]. The card records that the released code attaches the features to the value vector instead. Author-reported F1 on Election-Trec rises from 61.67 for the Bi-LSTM baseline to 67.82 for the self-attention variant with all EEG features. On General-Twitter it rises from 70.67 to 82.46 [arxiv:2606.26485 tab. 4]. Without features, the self-attention variant (81.74) and the Bi-LSTM+CRF (81.63) are within 0.2 F1 on General-Twitter [arxiv:2606.26485 tab. 4], so that gap comes from the features. ZuCo covers only 8.6% and 4.0% of the two microblog vocabularies, so most words get imputed values [arxiv:2606.26485 tab. 6] [arxiv:2606.26485 §3.1.2]. The paper fine-tunes hyperparameters for each dataset and model [arxiv:2606.26485 §4.1], and it reports no variance test. The card notes that the released code keeps the best epoch by test score, so the reported F1 may be optimistic. Card: [Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs](/papers/2606.26485). ### Attention over the positions of each detected period MSGNet finds the k strongest periods of a multivariate input with an FFT. For each period it reshapes the series into a grid, applies a graph convolution across the variables, and then runs multi-head self-attention over the positions inside that grid [arxiv:2401.00423 §4.2] [arxiv:2401.00423 §4.3] [arxiv:2401.00423 eq. 8]. The k results are summed with softmax weights over their amplitudes [arxiv:2401.00423 §4.4] [arxiv:2401.00423 eq. 9]. The paper states no mask for the attention step. Removing the attention changes the Flight average MSE from 0.195 to 0.198, and the authors call its gain marginal [arxiv:2401.00423 tab. 2] [arxiv:2401.00423 §5.6]. The card records that the released code differs in several places, including a causal mask inside each period that the paper does not describe. Card: [MSGNet](/papers/2401.00423). ## When to use which | Situation | Use | Why | |---|---|---| | Moderate token count, with tokens of more than one kind (image plus label tokens) | Full self-attention | Every token sees every other; C-Tran reports a practical ceiling near 2,000 labels [arxiv:2011.14027 §4.4] | | Long inputs and a tight GPU memory budget | Segmented or windowed attention | Cost follows the window size. ESRT cuts the attention matrix by s² and reports about 4× less training memory against an unspecified plain Transformer [arxiv:2108.11084 tab. 4]. Lower memory is not lower latency: the card reports about 4× slower inference than CARN and IMDN [arxiv:2108.11084 tab. 2] | | Volumetric input where global context is needed only at low resolution | Local attention, with global attention only in a small bottleneck | nnFormer restricts global attention to the bottleneck after down-sampling, where its cost (hwd)² is small [arxiv:2109.03201 §3.3] [arxiv:2109.03201 eq. 4] | | Very long sequences, such as time series, and an approximation is acceptable | Sparse attention (LogSparse, ProbSparse) | Memory bound stated by the cited work as O(L log L) [arxiv:2205.01138 §4.2]. The tutorial reports no measurements | | Linear cost in sequence length is required | Linear attention | Survey-stated reduction from O(n²) to O(n) [arxiv:2204.07756 §2.1.3]. Not measured in the survey | | Hybrid CNN or state space backbone that needs global mixing | Self-attention in the last half of the late stages | Ablation on ImageNet-1K: 82.3% against 81.3% for a random pattern [arxiv:2407.08083 tab. 5] | | Small training data, no evidence yet that global mixing helps | Local windows, as one author group argues | A 5×5 local window beats global attention on CIFAR-100 in the authors' figures, with no values in the text [arxiv:2202.06709 §2] [arxiv:2202.06709 fig. 7] | | Several agents or sensors whose count changes per scene | Across-agent self-attention | Token count is the number of agents, so the N × N cost is small [arxiv:2109.07644 tab. 3]. Bandwidth is the constraint, so compress the shared features [arxiv:2109.07644 §5.6] | | One output from many tokens of another modality | Query-restricted attention | One query row per block is cheaper than a full score matrix [arxiv:2203.16952 §III-B3] | | Add a new condition to a frozen pretrained model | Gated self-attention | The zero gate keeps the original behaviour at the start [arxiv:2301.07093 §4.2]. In the paper's ablation, gated self-attention gave AP 21.7 against 16.6 for gated cross-attention on the COCO detection setting [arxiv:2301.07093 app. B] | | Exact attention on NVIDIA Hopper GPUs, with long sequences | Exact fused kernel (FlashAttention-3) | Same output as standard attention. The speed figures are the authors' H100 benchmarks [arxiv:2407.08608 §4.1]. The FP8 accuracy step is not in the pinned code [arxiv:2407.08608 §3.3] | | Vision-language model, no retraining allowed, many image tokens | Token pruning after an early layer (FastV) | At K = 2, R = 50%, average scores held on the tested models at about 55% of theoretical FLOPs [arxiv:2403.06764 tab. 1]. Measured speed is a single A40 test [arxiv:2403.06764 tab. 4] | | Fixed-cost global mixing at high resolution, where exact attention is not required | Fourier token mixer (AFNO) | 80.9 mIoU on Cityscapes at 1024 × 1024, against 79.7 for efficient self-attention [arxiv:2111.13587 tab. 4]. Self-attention is too large to run at that size in the paper's setup [arxiv:2111.13587 §5.6] | | Long recurrent decoding with a constant-size state, where one scalar decay per head is acceptable | Decay-masked linear attention (SSD) | Linear recurrence across chunks, with matrix products inside them [arxiv:2405.21060 §6]. Per-channel decays are outside its scope [arxiv:2405.21060 §5] | | Video diffusion transformer, where full attention over all frames is too costly | Alternating spatial and spatiotemporal windows | Bounded window size per layer. The authors report up to 2× steps per second on UCF-101 [arxiv:2312.06662 tab. 3] | | High-resolution vision backbone, where attention memory dominates | Reduced keys and values (EMSA) | Keys and values pooled by stride 8, 4, 2, 1 across stages [arxiv:2105.13677 §2.2]. 72.88 against 72.68 top-1 for standard attention, in the authors' simpler recipe [arxiv:2105.13677 tab. 9] | | Routing or sequencing policy that picks one item per step and reuses node embeddings | Re-run the top attention layer with visited items masked, every few steps | Removing the masked layer raises the 100-node TSP greedy gap from 2.19% to 2.78% in the authors' runs [arxiv:2012.10638 tab. 2] | | Frozen pretrained transformer for a new modality, with few trainable weights | Freeze attention and FFN; train input and output layers and layer norms | Layer norms carry most of the gain: 54% on CIFAR-10 against 25% for output-only training [arxiv:2103.05247 tab. 15] | | Irregular, sparse observation sets, where imputation is unwanted | Full attention over observation triplets, with forecasting pretraining | PR-AUC 0.577 against 0.559 for the best baseline on MIMIC-III [arxiv:2107.14293 tab. 4]. Cost grows with the number of observations [arxiv:2107.14293 §4.4] | | Style-consistent image batch, with no per-style training | Reference-shared attention with AdaIN on queries and keys | DINO 0.51 against 0.537 for DreamBooth-LoRA, which trains per style [arxiv:2312.02133 tab. 2] | | Recall of text far outside the local window, at fixed training context | kNN memory in one layer | Perplexity 2.26 with a 65K memory against 2.42 without, at 2048 context [arxiv:2203.08913 §4.3]. The memory gets no gradients [arxiv:2203.08913 §3.1] | | Video recognition from an image ViT, with limited extra compute | Local temporal module plus a learned-query global block in deep layers | Global block alone: 84.4% on Kinetics-400 against 83.1% with neither block [arxiv:2211.09552 tab. 9] | | Pixel-level affinities inside a detected region, e.g. for occlusion | Non-local graph block over the region's nodes | Single graph layer 29.63 against 28.43 for a single FCN, COCO-OCC AP [arxiv:2103.12340 tab. 3] | | Non-local attention inside a CNN feature map, where the dense N×N map is too large | Criss-cross (sparse cross-path) attention | Survey-reported cost O(2√N) against O(N²) for a non-local block, and 11 times less GPU memory [arxiv:2101.01169 §3.1.1] | | Dense prediction at high resolution, with limited memory | Shifted 7×7-window attention in a CRF decoder | Window 7/8: 124 ms and Abs Rel 0.055, against 192 ms and 0.054 at window 32, with 2.7 G against 5.1 G [arxiv:2203.01502 tab. 6] | | Operator learning on fixed field grids, as a simulation surrogate | Galerkin-type attention, with no softmax | Mean field error 0.083% against 0.572% for FNO, on one compressor and validation split only [arxiv:2609.16066 tab. 4] | | Several objects per view, combined across views | Object self-attention in each view, averaged over views | 51.6% with four training views and one test view, against 46.2% with one view for both [arxiv:2204.02174 tab. 5] | | Several modalities that must mix early, with a short context, such as music and motion | Full self-attention over the concatenated token sequence (FACT) | FID_k 35.35 against 45.76 for late fusion and 45.66 with no fusion, in the authors' AIST++ ablation [arxiv:2101.08779 tab. 4]. The one-frame full-attention variant is worse on FID_k than the causal baseline [arxiv:2101.08779 tab. 3] | | Cooperative perception with vehicles and roadside units, under pose and delay noise | Heterogeneous attention across agents at each cell (V2X-ViT) | AP@0.7 0.614 against 0.487 for OPV2V in the noisy setting, author-reported on a simulated dataset [arxiv:2203.10638 tab. 1]. Adding the attention module raises AP@0.7 from 0.548 to 0.601 [arxiv:2203.10638 tab. 2] | | Image backbone that needs local detail and global context on a fixed token grid | Parallel CNN and transformer branches with feature coupling (Conformer) | 83.4% top-1 for Conformer-S at 10.6 G MACs, against 81.8% for DeiT-B at 17.6 G, author-reported [arxiv:2105.03889 tab. 2] | | Mobile image backbone that must be small in MAdds, with a few blocks of global context | Channel-wise attention in the last blocks of a conv-attention hybrid (EdgeNeXt) | 79.4% against 78.4% for MobileViT-S at 256 px, author-reported [arxiv:2206.10589 tab. 3]. The attention cost is linear in tokens, so it does not grow quadratically with the image [arxiv:2206.10589 §3] | | Long, low-rate biosignal with few labels, where transfer to new subjects matters | Convolutional encoder plus transformer, pre-trained by masked contrastive prediction (BENDR) | Mean balanced accuracy 86.7 on MMI and accuracy 42.6% on BCI IV 2a (4 classes), author-reported, both from the pooled-vector configuration [arxiv:2101.12037 tab. 2]. In that configuration the transformer is skipped at test time [arxiv:2101.12037 §2.4.2] | | Scanned documents where words, positions and page appearance must interact in every layer | Separate visual and text attention maps with a shared spatial term (DocFormer) | FUNSD F1 76.90 with multi-modal attention against 73.01 with spatial features alone, author-reported [arxiv:2106.11539 tab. 9]. No official code was found; read the [DocFormer](/papers/2106.11539) card before implementing | | Weights must justify a decision | Not self-attention alone | Weight magnitude is weakly tied to input importance in the survey's cited studies [arxiv:1904.02874 §7] | | Tabular rows with categorical columns, and enough labels to fit a transformer | Column self-attention (TabTransformer) | Mean AUC 82.8 against 81.8 for an MLP on 15 binary datasets, author-reported [arxiv:2012.06678 tab. 2]. At 50 labels on smaller datasets an MLP with entropy regularisation scores higher [arxiv:2012.06678 tab. 4] | | Dense prediction backbone with multi-scale output, where a fixed window size is acceptable | Shifted-window attention (Swin) | Removing the shift costs 1.1 top-1 and 2.8 box AP on Swin-T [arxiv:2103.14030 tab. 4] | | Scaling a vision transformer, or training diverges at high learning rate | Cosine-normalised scores with post-norm (Swin V2) | Swin-T 81.5 to 81.7 and Swin-B 83.6 to 84.1 in the authors' ablation. The baseline needed a quarter of the learning rate [arxiv:2111.09883 tab. 6] [arxiv:2111.09883 tab. 7] | | Bags of thousands of instances with one label, and approximate attention is acceptable | Landmark (Nyström) attention in two layers (TransMIL) | AUC 0.9309 against 0.8880 for the best compared method on CAMELYON16 [arxiv:2106.00908 tab. 1]. The official code is non-commercial, so a commercial build must follow the paper (see the [TransMIL](/papers/2106.00908) card) [arxiv:2106.00908 tab. 1] | | Dense matching where a full attention grid is too large | Linear attention on a coarse grid, then local refinement (LoFTR) | Pose AUC@10 falls to 32.04 when the attention module is replaced by a convolution, in the ablation [arxiv:2104.00680 tab. 6] | | High-resolution image transformer, where token-to-token attention memory dominates | Cross-covariance attention over channels (XCiT) | 7312 MB against 26915 MB peak memory for Swin-T at 1024 × 1024, author-measured [arxiv:2106.09681 tab. D.5]. On COCO box AP the 16-patch model trails Swin-T, 45.3 against 46.0 [arxiv:2106.09681 tab. 5] | | Image classifier tuned for measured throughput on a GPU or CPU | One attention layer per block between feed-forward layers (EfficientViT) | M5 at 10,621 images/s on a V100 against 7,560 for MobileNetV3-Large, at 77.1% against 75.2% top-1 [arxiv:2305.07027 tab. 2] | | Real-time detector, with a small attention budget at the lowest resolution | One partial attention block after the last stage (YOLOv10) | Small model AP 46.3 at 2.49 ms with the block, 44.9 at 2.34 ms without it, both with the large kernel [arxiv:2405.14458 tab. 10] | | Multivariate time series with rare anomaly labels, where reconstruction error is the signal | Window transformer with a second pass conditioned on the first error (TranAD) | Removing the transformer drops NAB F1 from 0.9364 to 0.8850 in the authors' ablation [arxiv:2201.07284 tab. 6] | | Very long sequences with a constant-size state at inference, and the model must choose what to keep | Selective state-space layer (Mamba) | Selective Copying 99.8% against 56.4% for a fixed-parameter S4 layer, synthetic task [arxiv:2312.00752 §4.1]. Keep fixed-parameter layers for raw audio [arxiv:2312.00752 app. E.4.1] | | Molecular or other graph-level regression, with a Transformer as the backbone | Full attention with degree, shortest-path and edge-path biases (Graphormer) | PCQM4M-LSC validation MAE 0.1234 against 0.1395 for GIN-VN [arxiv:2106.05234 tab. 1]. Without node-relation encoding it is 0.2276 [arxiv:2106.05234 tab. 5]. Cost is quadratic in node count [arxiv:2106.05234 §6] | | Tabular rows with few labels, where other rows in the batch may carry signal | Attention across the rows of a batch, with contrastive and denoising pre-training (SAINT) | 50 labels: 86.91 with pre-training against 85.78 without [arxiv:2106.01342 tab. 3]. Predictions depend on the batch the row sits in [arxiv:2106.01342 §3.2] | | Long one-dimensional feature sequences from a single-stage detector | Local window attention on each pyramid level (ActionFormer) | THUMOS14 average mAP 66.8 with window 19 and with full attention [arxiv:2202.07925 tab. 3b]. Window 19 runs at 2.0× the conv baseline's time, full attention at 2.2× [arxiv:2202.07925 tab. 3b] | | Sparse LiDAR voxels, where a detector needs a larger receptive field than convolutions give | Attention over non-empty voxels in local and dilated ranges, at most 48 per query (VoTr) | KITTI val moderate car AP_3D 78.27 with dilated rings, against 75.48 with local attention only [arxiv:2109.02497 tab. 4]. VoTr-SSD runs at 14.65 Hz against 20.73 Hz for SECOND [arxiv:2109.02497 tab. 8] | | Image backbone built from local attention, with a parameter budget similar to a CNN's | Blocked local attention with haloing (HaloNet) | H7 reaches 84.9% top-1 from scratch, author-reported. Training is slower than EfficientNet's on the authors' TPUs [arxiv:2103.12731 §4.1] [arxiv:2103.12731 §5] | | Keypoint or other small token set in a lightweight head, with a real-time target | One gated attention unit (RTMPose) | Cumulative COCO val chain: AP 71.4 to 71.9 with the GAU step [arxiv:2303.07399 §3.3] | | Dense matching where a per-pixel cost volume must be summarised before matching | Intra- and inter-cost-map attention over latent tokens (FlowFormer) | Sintel train final 2.72 with no encoder and 2.57 with three layers, on the RAFT-style schedule [arxiv:2203.16194 tab. 2] | | Raw satellite or other sensor series of about 70 dated observations, with cloud gaps and no budget for cleaning | Full self-attention over the dates, with a positional encoding | Raw 23-class kappa 0.71, equal to the LSTM and above MS-ResNet (0.69) and TempCNN (0.67), author-reported [arxiv:1910.10536 tab. 1]. The paper does not report run time or memory [arxiv:1910.10536 §8] | | Long text in training, where full attention is too slow and a fixed chunk size is acceptable | Chunked FLASH block: exact attention inside chunks, linear between them | Wiki-40B at context 8192: 15.11 against 15.25 perplexity for Transformer++, and 62.5 against 261.0 ms per step [arxiv:2202.10447 tab. 13]. No official code, and the appendix listings disagree with the equations on scaling [arxiv:2202.10447 app. D] | | Short, noisy text with a per-word side signal that covers most of the vocabulary | Per-word feature vector at the attention input | Author-reported gains of 6.2 F1 (Election-Trec) and 11.8 F1 (General-Twitter) over the Bi-LSTM baseline [arxiv:2606.26485 tab. 4]. ZuCo covers only 8.6% and 4.0% of the vocabularies, so this row's condition is not met in the paper [arxiv:2606.26485 tab. 6] | | High-resolution restoration that needs regional detail and long-range context, under a memory budget | Window attention and anchored stripe attention in parallel, each on half the channels (GRL) | GRL-B reaches 33.93 PSNR on GoPro against 32.92 for Restormer, author-reported [arxiv:2303.00748 tab. 2]. The anchored cost is O(NMd) time [arxiv:2303.00748 §3.2]. The released code's score function differs from the paper's, so see the [GRL](/papers/2303.00748) card | | Several clinical inputs of different types, all present for every patient, that should attend to each other early | Joint token sequence with bidirectional intra- and cross-modal blocks, then self-attention (IRENE) | Mean AUROC 0.924 against 0.858 for Perceiver over eight pulmonary diseases, author-reported [arxiv:2306.00864 tab. 1]. Replacing the bidirectional blocks gives 0.858 [arxiv:2306.00864 tab. 3]. Missing modalities are not handled [arxiv:2306.00864 p. 7] | | Multivariate series with several rhythms, where a graph per rhythm may help | Self-attention over the positions of each period, after a per-period graph convolution (MSGNet) | Without the attention, Flight MSE is 0.198 against 0.195, a gain the authors call marginal [arxiv:2401.00423 tab. 2] [arxiv:2401.00423 §5.6] | ## Implementation sketch Single-head scaled dot-product self-attention. Queries, keys and values are linear maps of the same input. Multi-head versions split the width and run this in parallel, as in the cards [arxiv:2108.11084 §3.3.2] [arxiv:2203.16952 §III-B3]. ```pseudocode # Single-head scaled dot-product self-attention [arxiv:1904.02874 tab. 2] # Input: x (N, d) one sequence of N tokens # Output: y (N, d_v) def self_attention(x, w_q, w_k, w_v, d_k): # Queries, keys and values all come from the same sequence [§4.1] q = x @ w_q # (N, d_k) k = x @ w_k # (N, d_k) v = x @ w_v # (N, d_v) # Alignment: scaled dot product between every pair of tokens [tab. 2] scores = q @ transpose(k) / sqrt(d_k) # (N, N) # Distribution: softmax over the keys in each row [§3] weights = softmax(scores, axis=-1) # (N, N) # Output: weighted sum of values [eq. 1] return weights @ v # (N, d_v) ``` Where the variants differ, change only the mask or the set of rows: ```pseudocode # Restrictions that turn the sketch above into the variants [§4] [§3] # Full: scores over all N keys, joint image and label # tokens (C-Tran). [arxiv:2011.14027 §3.2] # Causal: mask keys after each position (decoder). # [arxiv:1904.02874 §5.2] # Segmented: attend only inside each of s contiguous # segments of the token axis. [arxiv:2108.11084 §3.3.2] # Windowed: attend only inside a local window, shifted # on alternate layers. [arxiv:2111.13680 §3.2] # Sparse: keep a subset of keys per query [arxiv:2205.01138 §4.1-4.2] # CLS-query: use one row of q, keep one output row. # [arxiv:2203.16952 §III-B3] # Gated: add the attended visual rows back, scaled by tanh(gamma), # with gamma = 0 at the start. [arxiv:2301.07093 eq. 8] # Reduced K/V: pool keys and values by stride s; queries kept. # [arxiv:2105.13677 §2.2] # Decay mask: replace the softmax with a decay mask, no normalisation. # [arxiv:2405.21060 §5.2] # Pruned: drop low-scoring image tokens after layer K. # [arxiv:2403.06764 §4.1] # Masked: mask visited items to -inf in the top layer, every p steps # [arxiv:2012.10638 §3.2] # Shared ref: concatenate the reference's keys and values per image # [arxiv:2312.02133 §3.2] # Memory: also attend to top-k stored (key, value) pairs, gated # [arxiv:2203.08913 §3.1] # Galerkin: sum K^T V over all points first, then multiply by Q # [arxiv:2609.16066 eq. 3] # Frozen: freeze W_q, W_k, W_v and FFN; train I/O layers and norms # [arxiv:2103.05247 §2.2] # Column: one token per categorical column, no position encoding # [arxiv:2012.06678 §2] # Cosine: normalise q and k, divide the scores by a learned scale # [arxiv:2111.09883 eq. 2] # Channel: Q^T K over channels, softmax over channels, mix V # [arxiv:2106.09681 §3.2] # Landmark: attend through m landmark rows of q and k # [arxiv:2106.00908 eq. 9] # Linear: replace the softmax kernel with elu(x) + 1 # [arxiv:2104.00680 §3.2] # Partial: attend over half of the channels, then fuse by 1 x 1 conv # [arxiv:2405.14458 §3.2] # Cascade: head j adds the output of head j-1 to its input # [arxiv:2305.07027 eq. 3] # Selective: step size, input and output matrices depend on the token # [arxiv:2312.00752 §3.2] # Two-pass: second pass takes the first pass's error as its input # [arxiv:2201.07284 §3.4] # Batch rows: flatten each row's tokens, then attend over the batch # [arxiv:2106.01342 alg. 1] # Structural: add degree embeddings; a scalar per shortest-path distance # [arxiv:2106.05234 eq. 5-6] # Sparse voxels: attend over non-empty cells in local and dilated ranges # [arxiv:2109.02497 §3.2] # Local 1D: mask keys more than W steps from the query # [arxiv:2202.07925 §3.1] # Halo: each b x b query block shares one (b+2h)^2 memory, no mask # [arxiv:2103.12731 §3.2] # Gated unit: one shared base for Q and K, a gate on the output, relu^2 kernel # [arxiv:2303.07399 eq. 1-2] # Cost tokens: intra-pixel attention over K tokens, then attention over # all pixels for each token index [arxiv:2203.16194 eq. 2-3] ``` Training notes that the cards raise: - **Warm-up and learning rate.** In the Tensor2Tensor schedule the paper uses, the decay follows the global step count. Multiplying the batch by 8 on 8 GPUs with the same `learning_rate` raises the effective rate by √8 at a given step [arxiv:1804.00247 §4.8.3]. Warm-up then has to be re-tuned: 8k steps matched 16k, 6k was slower and 2k diverged [arxiv:1804.00247 §4.8.2]. - **Pre-norm or post-norm in the code.** SAINT's equations put layer norm after each sub-layer, while its code normalises the input of each sub-layer [arxiv:2106.01342 eq. 1] [arxiv:2106.01342 eq. 2]. Graphormer's paper uses pre-LN [arxiv:2106.05234 §3.2], but several of its released scripts train post-LN unless a flag is set (see the [Graphormer](/papers/2106.05234) card). The cards do not test which placement reproduces the reported numbers. - **Layer-norm placement.** The tutorial describes the original ordering, with layer normalisation after the residual add. It says pre-normalisation performs worse empirically [arxiv:2205.01138 §5.3]. Treat that as the tutorial's claim, not a measured result. - **Post-norm at the default rate.** With the layer norm after each residual branch, the authors' baseline diverged at the default learning rate, so it was run at a quarter of that rate. Compare runs only at matched rates [arxiv:2111.09883 tab. 7]. - **Per-head scale.** The NLP tutorial's per-head formula divides the scores by the square root of the full model width, not the head width. It does not say which the implementer should use [arxiv:2311.17633 §2.3]. ## Contradictions - **Why do transformers need so much data?** The survey says transformers need large training sets because they carry a weak inductive bias [arxiv:1904.02874 §5.2]. ESRT's authors make the same point about vision transformers [arxiv:2108.11084 §1]. The authors of the vision-transformer analysis attribute the weak small-data results to optimisation instead. They report a loss with many negative Hessian eigenvalues early in training, and they state that the model does not overfit even at 2% of CIFAR-100 [arxiv:2202.06709 §2] [arxiv:2202.06709 fig. 4] [arxiv:2202.06709 fig. 5]. The Multimodal Fusion Transformer reports the best overall accuracy in each of its settings on scarce hyperspectral training samples [arxiv:2203.16952 tab. I], but its numbers depend on how checkpoints were selected (see its card). These are different explanations of related observations, and the cards do not settle which is right. - **Global or local attention.** The vision-transformer analysis argues that data specificity, not long-range dependency, is the key property, and that a local 5×5 window beats global attention on CIFAR-100 [arxiv:2202.06709 §2] [arxiv:2202.06709 fig. 7]. nnFormer keeps global attention in its bottleneck, and its cumulative ablation adds 0.2 points of average DSC on ACDC from one bottleneck global layer, from 91.26 to 91.46 [arxiv:2109.03201 tab. 6]. The two papers differ in task, data and what they vary, so they do not directly test the same question. - **Does a wider reach help?** ActionFormer finds almost no change in average mAP across window sizes from 9 to full attention on THUMOS14 [arxiv:2202.07925 tab. 3b]. VoTr finds that adding dilated rings to local attention raises KITTI moderate car AP_3D from 75.48 to 78.27 [arxiv:2109.02497 tab. 4]. The tasks, inputs and ablations differ, so the two results do not test the same question. HaloNet's window ablation, on ImageNet, finds a 1.3% gain from enlarging the window from 6 × 6 to 12 × 12 [arxiv:2103.12731 §4.3]. Its data and task differ from both of the above. - **Is pre-norm worse than post-norm?** A time-series tutorial says that pre-normalisation performs worse empirically [arxiv:2205.01138 §5.3]. The NLP tutorial says pre-norm eases optimisation in deep models, and that neither placement is always better [arxiv:2311.17633 §4.2]. Neither source reports a measurement of its own on this point, so treat both as claims and test the placement. - **Are attention weights a usable importance signal?** FastV ranks image tokens by the attention they receive and reports near-baseline scores after dropping half of them [arxiv:2403.06764 §4.1] [arxiv:2403.06764 tab. 1]. In its retrained-baseline ablation, random ranking scores lower at the same setting [arxiv:2403.06764 tab. 7]. The survey, citing studies of its own, says weight magnitude is weakly tied to input importance [arxiv:1904.02874 §7]. The FastV card notes that the paper's ranking and the code's ranking differ, so the ablation does not show which one produced the reported numbers. This page does not resolve the conflict. A satellite time-series study reads its attention maps as focusing on a few observations rather than on cloudy dates [arxiv:1910.10536 §8.4]. It shows only the first of three layers, from one trained model, so it is weak evidence either way. The NLP survey says the same debate is open: attention may be a plausible explanation without being a faithful one [arxiv:1902.02181 §VI-A]. ## Papers - [An Attentive Survey of Attention Models](/papers/1904.02874): taxonomy of attention with self-attention as the same-sequence case, and the O(n²) cost. - [Training Tips for the Transformer Model](/papers/1804.00247): learning-rate and warm-up behaviour when the batch grows across GPUs. - [General Multi-label Image Classification with Transformers](/papers/2011.14027): full attention over image and label tokens, with a label-count ceiling. - [Transformer for Single Image Super-Resolution](/papers/2108.11084): self-attention over segments of unfolded patches to cut memory in a lightweight model. - [nnFormer: Interleaved Transformer for Volumetric Segmentation](/papers/2109.03201): local 3D volume attention in the encoder and global attention in the bottleneck. - [OPV2V](/papers/2109.07644): self-attention across connected vehicles at each spatial location. - [GMFlow: Learning Optical Flow via Global Matching](/papers/2111.13680): 2×2 shifted-window self-attention and cross-attention in a flow network. - [How Do Vision Transformers Work?](/papers/2202.06709): attention as spatial smoothing, local windows, and a hybrid ResNet with local attention blocks. - [Multimodal Fusion Transformer for Remote Sensing Image Classification](/papers/2203.16952): a CLS token from a second sensor as the only query in each attention block. - [Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756): taxonomy of self-attention variants, including linear and axial attention. - [Transformers in Time-series Analysis: A Tutorial](/papers/2205.01138): the original self-attention block and its sparse variants for time series. - [Transformers in Remote Sensing: A Survey](/papers/2209.01206): Swin-style local windows and hybrid transformer use in remote sensing. - [Transformers in Vision: A Survey](/papers/2101.01169): taxonomy of vision designs by their self-attention block, with the quadratic cost of global attention and criss-cross attention in CNNs. - [GLIGEN: Open-Set Grounded Text-to-Image Generation](/papers/2301.07093): gated self-attention added to a frozen diffusion model, with the gate starting at zero. - [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](/papers/2407.08083): self-attention only in the late layers of stages 3 and 4, with a placement ablation. - [VideoGPT: Video Generation using VQ-VAE and Transformers](/papers/2104.10157): axial attention in the autoencoder and a causal transformer prior over codes. - [ResT: An Efficient Transformer for Visual Recognition](/papers/2105.13677): keys and values pooled by a strided depth-wise convolution in each stage. - [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](/papers/2111.13587): a Fourier-domain token mixer that replaces self-attention. - [Photorealistic Video Generation with Diffusion Models](/papers/2312.06662): alternating spatial and spatiotemporal window attention in a video diffusion transformer. - [FastV: token dropping for vision-language model inference](/papers/2403.06764): FastV drops image tokens after an early layer, ranked by attention. - [Transformers are SSMs](/papers/2405.21060): masked linear attention with one scalar decay per head, computed in chunks. - [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](/papers/2407.08608): an exact attention kernel for Hopper GPUs, with an FP8 path. - [RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion](/papers/1904.12304): self-attention blocks inside a latent GAN that an RL agent samples from. - [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638): a self-attention encoder with masked re-embedding of visited nodes. - [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247): frozen self-attention and feedforward blocks, with only I/O layers and norms trained. - [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](/papers/2103.12340): a non-local graph block over the nodes of each region in the mask head. - [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](/papers/2107.14293): full self-attention over observation triplets, with forecasting pretraining. - [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](/papers/2203.01502): shifted-window attention inside a CRF depth decoder. - [Memorizing Transformers](/papers/2203.08913): a kNN memory of past keys and values in one attention layer. - [Multi-View Transformer for 3D Visual Grounding](/papers/2204.02174): self-attention over objects in each rotated view, averaged across views. - [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](/papers/2211.09552): a learned-query global block on a pretrained image ViT. - [Style Aligned Image Generation via Shared Attention](/papers/2312.02133): reference-shared self-attention with AdaIN, with no training. - [Transformer-enhanced neural operator for compressor flow fields](/papers/2609.16066): softmax-free Galerkin-type attention in a neural operator. - [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678): one token per categorical column, with self-attention across the columns of a row. - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](/papers/2103.14030): shifted 7 × 7 windows, with relative position bias and patch merging. - [LoFTR: Detector-Free Local Feature Matching with Transformers](/papers/2104.00680): linear self and cross attention on a coarse grid, then local refinement. - [TransMIL](/papers/2106.00908): correlated multiple-instance learning for whole-slide images, with Nyström-approximated attention over patch bags and a convolutional position module. - [Swin Transformer V2: Scaling Up Capacity and Resolution](/papers/2111.09883): cosine-normalised scores, post-norm and a log-spaced position bias for scaling. - [A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293): sequence-reduced attention in a Siamese encoder for change detection. - [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](/papers/2201.07284): a window transformer with a second pass conditioned on the first pass's error. - [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](/papers/2106.01342): attention across the columns of each row and across the rows of a batch. - [Do Transformers Really Perform Bad for Graph Representation?](/papers/2106.05234): Graphormer, full attention with degree, shortest-path and edge-path biases. - [SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems](/papers/2412.13802): a Transformer encoder over per-frame vehicle states, used as a violation score in a genetic fuzzer. - [Voxel Transformer for 3D Object Detection](/papers/2109.02497): attention over non-empty voxels in local and dilated ranges. - [ActionFormer: Localizing Moments of Actions with Transformers](/papers/2202.07925): local window self-attention on a feature pyramid for video action localisation. - [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](/papers/2305.07027): one attention layer per block between feed-forward layers, with cascaded heads. - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](/papers/2312.00752): an attention-free selective state-space layer with input-dependent parameters. - [YOLOv10: Real-Time End-to-End Object Detection](/papers/2405.14458): one partial self-attention block after the lowest-resolution stage. - [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](/papers/2101.08779): full self-attention over motion and music tokens, with multi-frame future targets. - [Conformer: Local Features Coupling Global Representations for Visual Recognition](/papers/2105.03889): a CNN branch and a transformer branch coupled in every block. - [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](/papers/2203.10638): self-attention across vehicles and roadside units at each spatial cell, plus window attention. - [XCiT: Cross-Covariance Image Transformers](/papers/2106.09681): cross-covariance attention over feature channels, linear in the token count, with a depth-wise patch step. - [Attention in Natural Language Processing](/papers/1902.02181): a survey whose definition of self-attention and whose section on attention as explanation are cited above; it has no experiments. - [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](/papers/2103.12731): HaloNet, blocked local attention with haloing and strided attention downsampling. - [FlowFormer: A Transformer Architecture for Optical Flow](/papers/2203.16194): intra- and inter-cost-map self-attention over latent cost tokens, then a recurrent cross-attention decoder. - [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](/papers/2303.07399): one gated attention unit over keypoint tokens in a SimCC head. - [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](/papers/2101.12037): a convolutional encoder and an 8-layer transformer pre-trained on unlabelled EEG. - [DocFormer: End-to-End Transformer for Document Understanding](/papers/2106.11539): separate visual and text attention maps with a shared spatial term. - [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](/papers/2204.13154): a survey of attention from early visual models to Transformer variants; no experiments. - [EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications](/papers/2206.10589): channel-wise self-attention in the SDTA block of a mobile hybrid. - [Self-attention for raw optical Satellite Time Series Classification](/papers/1910.10536): a Transformer encoder over raw and cleaned Sentinel-2 date series, compared with LSTM and CNN models. - [Transformer Quality in Linear Time](/papers/2202.10447): FLASH, a gated single-head unit with exact attention in chunks and linear attention between them. - [Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs](/papers/2606.26485): per-word EEG and eye-tracking features added at the input of a self-attention tagger. - [Efficient and Explicit Modelling of Image Hierarchies for Image Restoration](/papers/2303.00748): window attention and anchored stripe attention in parallel, with a channel-attention convolution branch. - [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864): bidirectional intra- and cross-modal attention blocks, then self-attention over one joint token sequence. - [Introduction to Transformers: an NLP Perspective](/papers/2311.17633): a tutorial on attention, positions, normalisation and warm-up choices; it has no experiments. - [MSGNet](/papers/2401.00423): self-attention over the positions of each FFT-detected period, after a per-period graph convolution.