Self-attention

Overview

Self-attention is attention where the queries, keys and values all come from the same set of tokens. The survey defines it as the case where queries and keys share one sequence, and notes that it suits tasks whose output is not a sequence, such as classification or recommendation 1904.02874 §4.1. The shared invariant of every variant is the same: each output is a weighted sum of value vectors, with weights computed from scores between tokens 1904.02874 eq. 1 1904.02874 §4.1. An NLP survey uses the same definition, where the input sequence serves as both keys and query, and notes that some architectures use no query at all 1902.02181 §IV-A. A 2022 survey treats the Transformer's learned projections of queries, keys and values as the main form, and names the quadratic time and memory cost as the problem most later variants try to reduce 2204.13154 §5 2204.13154 §5.3.

One author group reads the effect as spatial smoothing: each output is a weighted average of other positions with learned, positive weights. They argue this lowers feature-map variance and helps optimisation 2202.06709 app. B 2202.06709 §3. That is their interpretation, not a settled view.

What the cards show is that "self-attention" covers several different restrictions on who may attend to whom. It also covers attention across the rows of a batch, learned biases added to the scores, exact kernels that compute the same result faster, token pruning between layers, attention between channels instead of tokens, token mixers that replace attention, memories of past keys, attention through anchor tokens, attention inside periods found by a Fourier transform, and shared or masked attention that changes during decoding. Those choices decide the cost (full attention is quadratic in the number of tokens), the data needs, and whether a frozen model can be extended. Some variants also change the score function (cosine scores with a learned scale) or how many attention layers a block holds. The variants below are grouped by that choice rather than by paper.

Variants

Full self-attention

Every token attends to every other token. The survey states that decoder self-attention is masked so a position cannot see later positions, and that the cost is O(n²) in time and memory 1904.02874 §5.2 1904.02874 §8.8. Card: An Attentive Survey of Attention Models. The NLP tutorial gives the same causal rule for decoder self-attention, where position i sees only positions up to i 2311.17633 §2.6. Its card is Introduction to Transformers: an NLP Perspective.

C-Tran runs full attention over one joint sequence: 324 image tokens and one token per label, where each label token carries a state of unknown, negative or positive 2011.14027 §3.1 2011.14027 §3.2. The paper reports a practical limit of about 2,000 labels on four Titan X GPUs, which it ties to the memory cost of the full token set 2011.14027 §4.4. Card: General Multi-label Image Classification with Transformers.

VideoGPT's prior is a causal transformer over discrete video codes. The paper uses Sparse Transformers with local and strided attention for it 2104.10157 §4.1. The official code uses full attention by default and needs an extra install for the sparse option, so the two differ in practice. Card: VideoGPT.

Full attention also appears where the token set is small or fixed. The non-local block in BCNet gives every node of a 14×14 region feature an affinity to every other node in that region, normalised by a softmax 2103.12340 §3.2 2103.12340 eq. 2. Its ablation gives 29.63 COCO-OCC AP for a single graph layer, against 28.43 for a single fully convolutional layer, and 30.68 for the bilayer graph form 2103.12340 tab. 3. Card: BCNet.

RL-GAN-Net's latent GAN puts a self-attention block in its generator and discriminator, with spectral normalisation 1904.12304 §3.2. The card records that the released code instantiates two discriminator attention blocks where the paper's table lists one. Card: RL-GAN-Net.

STraTS runs full self-attention over the set of observation triplets in an ICU stay, with no imputation. Attention cost grows with the number of observations, which is why the paper caps them for MIMIC-III [arxiv:2107.14293 §4.4]. On MIMIC-III 24-hour mortality its PR-AUC is 0.577, against 0.559 for the best baseline 2107.14293 tab. 4. Card: STraTS.

AI Choreographer's FACT model applies full attention to music and motion in one stack. Motion tokens and then audio tokens form one sequence, and no attention mask is used, so every token attends to every input token 2101.08779 §4.1. The model reads a 120-frame seed and a 240-frame music window, and is trained to output the next 20 frames, not only the next one 2101.08779 §5.2.1. The authors' ablation changes attention and supervision together. Full attention with one-frame targets scores worse on FID_k than the causal baseline with shift-by-one targets (207.74 against 111.69), so full attention alone does not explain the gain [arxiv:2101.08779 tab. 3]. Their early fusion layout gives FID_k 35.35, against 45.76 for late fusion and 45.66 for a motion-only transformer 2101.08779 tab. 4. These are the authors' numbers on their AIST++ split. Card: AI Choreographer.

SimADFuzz uses a Transformer encoder as a learned scorer in a genetic fuzzer. It reads the per-frame states of every vehicle in one simulated run, so self-attention runs across the frames of that run 2412.13802 §4.2.1. The paper's table lists three layers and three heads, while the released code builds two layers 2412.13802 tab. 4. The paper mentions no positional encoding and the code adds none, so frame order reaches the model only through the attention pattern 2412.13802 §4.2.1. The card's results include no run that removes the encoder alone, so its share of the violation count is not isolated 2412.13802 §5.2.1. Card: SimADFuzz.

BENDR runs a standard transformer over a sequence of EEG vectors. The sequence has 8 layers, 8 heads, width 1536 and feed-forward width 3076 2101.12037 §2.3. The encoder produces about 2.67 vectors per second of signal, and a 60-second pre-training window yields 160 steps through the transformer 2101.12037 §2.3 2101.12037 §2.4.1. Pre-training masks spans of those vectors and scores each masked output against 20 distractors with a contrastive loss 2101.12037 eq. 2. In the paper's best downstream setting for most tasks, the transformer is skipped and a classifier reads pooled vectors 2101.12037 §2.4.2 [arxiv:2101.12037 tab. 2]. So the pre-trained attention layers are not always the part that does the work at test time. Card: BENDR.

A Transformer encoder can also read a satellite pixel's series as a sequence of dates. Each date is a vector of 13 Sentinel-2 bands, and the series is 70 dates long 1910.10536 §4.6. A positional encoding is added, because self-attention alone ignores order 1910.10536 §4.2. Every date attends to every other date. The paper's equation scales the scores by the square root of the model width, while the card records that the released code scales by the per-head key size instead 1910.10536 eq. 4. On raw data with 23 classes, the author-reported kappa is 0.71 for the Transformer, equal to the LSTM, and above 0.69 for MS-ResNet and 0.67 for TempCNN [arxiv:1910.10536 tab. 1]. Cleaning the data raised every model's score, and the cleaned-data models sit within a few points of each other 1910.10536 §8.1. Card: Self-attention for raw optical Satellite Time Series Classification.

Structural bias added to the scores

Graphormer keeps full attention over the nodes of a graph and adds structural terms. A learned degree embedding is added to each node's input features 2106.05234 eq. 5. For each ordered pair of nodes, a learned scalar for their shortest-path distance is added to the attention score, and a further term averages learned weights over the edge features on one shortest path 2106.05234 eq. 6 2106.05234 eq. 7. A virtual graph token, connected to every node, carries the graph-level output [arxiv:2106.05234 §3.2]. In its PCQM4M-LSC ablation, validation MAE is 0.2276 with no node-relation encoding and 0.1304 with spatial distance, centrality and the attention-bias edge term together 2106.05234 tab. 5. The full model reaches 0.1234 against 0.1395 for GIN-VN 2106.05234 tab. 1. These are the authors' numbers on one benchmark. The cost stays quadratic in the number of nodes, which the paper names as a main limit 2106.05234 §6. Card: Do Transformers Really Perform Bad for Graph Representation?.

Separate attention maps per modality, with a shared spatial term

DocFormer encodes three inputs for each page: a grid of image features, a word-piece text sequence, and per-word layout features 2106.11539 §3.1. Each encoder layer computes one attention map for the visual branch and one for the text branch, and adds their outputs 2106.11539 §3.1 2106.11539 §3. Each branch's score combines a content term, two relative-offset terms and a spatial term. The spatial term uses one pair of query and key matrices for both branches, so the two streams can learn cross-modal spatial interactions 2106.11539 §3.1 2106.11539 supp. §6.5. The relative-offset embedding is clipped to 8 tokens on each side, and the paper does not state its shape or the handling of longer offsets beyond that clip 2106.11539 supp. §6.1. In the authors' ablation on FUNSD, F1 goes from 61.56 for text-only BERT-base to 73.01 with spatial features and 76.90 with multi-modal self-attention. On CORD the same steps give 89.23, 92.28 and 93.36 2106.11539 tab. 9. These are author-reported numbers. Card: DocFormer.

Intra-modal and cross-modal terms in one joint sequence

IRENE turns a chest radiograph, a chief-complaint text, laboratory values, sex and age into one token sequence 2306.00864 p. 9. Each of two bidirectional blocks gives each modality attention over its own tokens, plus an attention term over the other modality's tokens, weighted by λ = 1.0 2306.00864 p. 10. The two streams are then concatenated, and ten self-attention blocks with 12 heads run over the joint sequence 2306.00864 p. 10. In the authors' ablation on the eight pulmonary diseases, the full model reaches mean AUROC 0.924. Replacing the bidirectional blocks with self-attention gives 0.858, and attention from text to image only gives 0.884 2306.00864 tab. 3. The method assumes every modality is present for each patient 2306.00864 p. 7. These are author-reported figures, on a cohort whose data the authors do not release 2306.00864 p. 12. Card: A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics.

Attention across the columns of one table row

TabTransformer makes each categorical column of a row one token. Each token is a column identifier concatenated with a learned class vector, and the columns get no positional encoding because they have no order 2012.06678 §2. Stacked Transformer layers then make each column's vector depend on the other columns of the same row 2012.06678 eq. 2. The reported setting is embedding size 32, six layers and eight heads 2012.06678 §3 2012.06678 app. B.1. On 15 binary datasets the authors report mean AUC 82.8 against 81.8 for an MLP, and 82.9 for GBDT 2012.06678 tab. 2. Those are author-reported numbers, and the paper's dataset counts disagree in places 2012.06678 app. C.

The same layers can be pretrained on unlabelled rows, by masking column values or by detecting replaced values. In the authors' semi-supervised tables this helps on the larger datasets, but on the smaller ones an MLP with entropy regularisation is the best model at 50 labels 2012.06678 tab. 3 2012.06678 tab. 4. Card: TabTransformer.

Attention across the rows of a batch

SAINT adds a second attention step to each stage. After self-attention over the columns of one row, it flattens each row's column tokens into one vector and runs attention over the rows of the batch, so every row can attend to every other row 2106.01342 §3.2 2106.01342 alg. 1. Its ablation on the binary datasets separates the two parts. Column-only attention (SAINT-s) scores mean AUROC 92.59, row-only (SAINT-i) 93.09 and both (SAINT) 93.13, against 91.06 for XGBoost 2106.01342 tab. 2. The mean covers 14 binary datasets, and the table shows only some of them. With 50 labelled rows, pre-training on unlabelled rows lifts SAINT from 85.78 to 86.91 2106.01342 tab. 3. Since the attention runs over the batch, a row's output depends on the other rows in its batch. The paper does not say how a single row is scored, so check the batching before comparing numbers (see the SAINT card). All of these are author-reported figures.

Windowed and segmented self-attention

Each token attends only to a local neighbourhood, so cost follows the window size rather than the full token count.

ESRT splits the token sequence of a super-resolution feature map into contiguous segments and runs attention inside each one, so each attention matrix is (N/s)² rather than N², with s = 4 2108.11084 §3.3.2. In its ablation at 48×48 training patches and batch 16, the one-block version uses 4191M of training GPU memory against 16057M for an unspecified plain Transformer, at a similar PSNR 2108.11084 tab. 4. Those figures are the authors' measurements. Card: Transformer for Single Image Super-Resolution.

Segmenting changes what can be attended. Each token sees only the band of the feature map that falls in its segment, so results can depend on the input size and on how the image is tiled at test time 2108.11084 §3.3.2. The card records that inference splits large inputs into overlapping pieces, which the paper itself does not describe.

nnFormer uses local attention inside small 3D volumes in its encoder, with a second layer on a grid shifted by half the volume size. Local attention cost is linear in the number of voxel tokens for a fixed volume size, which is why the encoder uses it 2109.03201 §3.2 2109.03201 eq. 2. Card: nnFormer: Interleaved Transformer for Volumetric Segmentation.

GMFlow restricts attention to 2×2 windows over a 1/8-resolution feature map, and shifts the windows by half a window on every second layer so that neighbouring windows connect 2111.13680 §3.2. The appendix computes the global correlation in blocks; in one setting the blocked version took 59.5 ms against 52.6 ms unblocked 2111.13680 app. B. Card: GMFlow: Learning Optical Flow via Global Matching.

Swin-style local windows are described by the remote-sensing survey as having a cost that is linear in image size 2209.01206 §4.2. Card: Transformers in Remote Sensing: A Survey.

Swin Transformer is the reference for shifted windows. Its stages use 7 × 7 windows, and every second block moves the window grid by half a window, so neighbouring windows exchange information 2103.14030 §3.1 2103.14030 §3.2. The shifted layer rolls the feature map instead of padding it, which keeps the number of windows fixed, and a mask keeps tokens from different regions apart 2103.14030 §3.2. Removing the shift from Swin-T drops ImageNet-1K top-1 from 81.3 to 80.2 and COCO box AP from 50.5 to 47.7 2103.14030 tab. 4. On a V100, the cyclic shift runs at 755 images per second against 183 for naive padding 2103.14030 tab. 5. Those speeds are the authors' measurements. Card: Swin Transformer.

HaloNet builds a whole image backbone from local attention and drops the per-window masks used by SASA. The image is cut into b × b query blocks, and each block attends to one shared memory of (b + 2h) × (b + 2h) pixels, so the windows are not centred on each pixel 2103.12731 §3.2. The authors report 84.9% top-1 on ImageNet for their largest model trained from scratch 2103.12731 §4.1. Enlarging the window from 6 × 6 to 12 × 12 gives a 1.3% gain, and an unmasked window beats the masked 7 × 7 window of SASA, with no values given in the text 2103.12731 §4.3. Pure attention models train more slowly than the matched EfficientNets on the authors' TPUs 2103.12731 §5. Card: Scaling Local Self-Attention for Parameter Efficient Visual Backbones.

Window size matters less than it seems at one input size. MambaVision uses windows of 14 and 7 for its two late stages at 224 input. Those windows cover each stage's whole map, so attention there is global. Its ablation with windows 7 and 7 instead gives 82.2% top-1 against 82.3% at 6318 against 6298 images per second 2407.08083 app. A 2407.08083 tab. S.1. Card: MambaVision: A Hybrid Mamba-Transformer Vision Backbone.

W.A.L.T alternates two window types in its video diffusion transformer. A spatial layer attends within one latent frame, in windows of 1 × 16 × 16 tokens. A spatiotemporal layer attends across latent frames, in windows of 5 × 8 × 8 2312.06662 §4.2 2312.06662 §5.2. The paper reports up to 2× more steps per second from local windows on UCF-101 2312.06662 tab. 3. That is an author measurement on one setup. Card: Photorealistic Video Generation with Diffusion Models.

NeW CRFs restricts attention to non-overlapping 7×7-patch windows in its depth decoder. A second pass moves the window grid by half a window, so neighbours across a border are linked 2203.01502 §3.2. Its window-size ablation, retrained on 352×352 crops, gives Abs Rel 0.055 at window 7/8 in 124 ms and 0.054 at window 32 in 192 ms. Memory rises from 2.7 G to 5.1 G, and window 48 runs out of memory in training 2203.01502 tab. 6. The paper does not say how feature maps that are not a multiple of the window size are padded 2203.01502 §3.2. Card: NeW CRFs.

ActionFormer applies local self-attention to the one-dimensional feature sequence of a video, with a window of W steps around each position. The same W is used on every level of its pyramid 2202.07925 §3.1. The cost falls from O(T²D + D²T) to O(W²TD + D²T) 2202.07925 §3.1. The reported window is 19 steps on THUMOS14, 9 on EPIC-Kitchens and 11 on ActivityNet 2202.07925 §4.1 2202.07925 §4.2 2202.07925 §4.3. In its THUMOS14 ablation, windows of 9, 19, 25 and 37 steps and full attention all score between 66.4 and 66.8 average mAP. Full attention runs at 2.2× the convolutional baseline's time and window 19 at 2.0× 2202.07925 tab. 3b. On this benchmark, then, the window size changed accuracy very little. Card: ActionFormer.

Attention over non-empty voxels in a neighbourhood

VoTr runs attention over the non-empty voxels of a 3D grid rather than over every cell. Each query attends to voxels in a local range and in dilated rings whose steps grow with distance, with at most 48 attending voxels per query on KITTI 2109.02497 §3.2 2109.02497 §3.3 2109.02497 §4.1. The authors state that non-empty voxels are under 0.1% of the grid on Waymo, and about 90k per frame, which rules out dense attention 2109.02497 §1. Neighbour lookup uses a GPU hash table, so empty cells are rejected without a scan 2109.02497 §3.4. On KITTI validation, moderate car AP_3D is 75.48 with local attention alone and 78.27 with the dilated rings added 2109.02497 tab. 4. Capping the attending voxels at 24, 32 and 48 gives 77.08, 77.72 and 78.27 2109.02497 tab. 6. The paper gives the ranges only for KITTI 2109.02497 app. B. On Waymo LEVEL_1 3D mAP, VoTr-TSD scores 74.95, against 70.3 for PV-RCNN as reported and 74.06 for the authors' re-implementation 2109.02497 tab. 1. VoTr-SSD runs at 14.65 Hz against 20.73 Hz for SECOND 2109.02497 tab. 8. Card: Voxel Transformer for 3D Object Detection.

Cosine-normalised scores with post-norm for scaling

Swin V2 keeps the Swin layout and changes how scores are formed when the model grows to 3 billion parameters and 1,536-pixel inputs 2111.09883 §3.5 2111.09883 §4.2. Queries and keys are normalised, so their product is a cosine, and the scores are divided by a learned scale τ per head and layer, kept above 0.01 2111.09883 §3.2 2111.09883 eq. 2. The layer norm moves to the output of each residual branch 2111.09883 §3.2. These two changes add 0.2 top-1 on Swin-T (81.5 to 81.7) and 0.5 on Swin-B (83.6 to 84.1) in the authors' ImageNet-1K ablation 2111.09883 tab. 6. The plain post-norm baseline diverges at the default learning rate, so it was run at a quarter of that rate 2111.09883 tab. 7. The same paper replaces the learned bias table with a small network over log-spaced offsets, so a model trained at one window size can run at another 2111.09883 §3.3 2111.09883 eq. 4. Card: Swin Transformer V2.

Axial attention

Axial attention runs attention along one axis at a time, such as along time or along height, instead of across the whole grid. VideoGPT's autoencoder uses it. Its BAIR ablation shows a small gain from adding it: NMSE 0.0033 against 0.0041 without attention, and R-FVD 14.9 against 15.3 2104.10157 tab. 3. Card: VideoGPT.

Sparse and linear approximations

These keep the same token set but cut the number of score pairs.

Criss-cross attention, used inside CNNs, keeps a sparse map on a cross-shaped path through each position. A non-local block instead builds a dense map at O(N²) cost. The survey reports the criss-cross cost as O(2√N) and reports 11 times less GPU memory 2101.01169 §3.1 2101.01169 §3.1.1. These are the survey's reported figures.

LogSparse uses convolutional queries and keys and attends at exponentially spaced offsets. The cited work states that memory per layer falls from O(L²) to O(L log L) 2205.01138 §4.2. Informer's ProbSparse attention keeps only a subset of queries, and the tutorial notes its selection rule is worded inconsistently 2205.01138 §4.1. Card: Transformers in Time-series Analysis: A Tutorial.

Efficient (linear) attention rearranges the product so that the cost falls from O(n²) to O(n). The survey writes this as ρ_q(Q)(ρ_k(K)ᵀV), with a normalisation on each factor. It also gives axial attention at O(hwm) and clustered attention at O(qc) 2204.07756 §2.1.3. Card: Visual Attention Methods in Deep Learning: An In-Depth Survey. These costs are the survey's figures and are not measured there.

LoFTR uses linear attention at its coarse 1/8 scale, with the softmax replaced by φ(x) = elu(x) + 1 2104.00680 §3.2. The authors call full attention impractical at that grid size. Self and cross attention alternate four times 2104.00680 §3.6. In the ScanNet ablation, replacing the module with a convolution of similar size drops pose AUC@10 to 32.04, against 40.8 for the full model 2104.00680 tab. 6. The ablation's full row does not match the main table's OT row, so compare within the ablation only 2104.00680 tab. 2. Card: LoFTR.

TransMIL approximates attention over whole-slide bags of thousands of patches with the Nyström method. It picks m landmark tokens from the queries and keys, and the paper states the cost falls from O(n²) to O(n) 2106.00908 §3.3 2106.00908 eq. 9. Its two attention layers reach AUC 0.9309 on CAMELYON16, against 0.8880 for the best compared method, MIL-RNN 2106.00908 tab. 1. The landmark count, pseudo-inverse iterations and head count are not stated in the paper 2106.00908 app. B. Card: TransMIL.

GRL's anchored stripe attention is a bottleneck of the same kind. Within each stripe, a few anchor tokens are average-pooled from the feature map and then projected. The anchors read the keys and collect the values, and then each query reads the anchors 2303.00748 §4 2303.00748 eq. 3 2303.00748 eq. 4. The cost is stated as O(NMd) time and O(NM) space, where M is the number of anchors 2303.00748 §3.2. The stripes alternate between horizontal and vertical from layer to layer 2303.00748 §3.3. The paper does not state the pooling factor 2303.00748 §4. On GoPro deblurring the authors report 33.93 PSNR for GRL-B against 32.92 for Restormer 2303.00748 tab. 2. The card lists where the released code differs from the paper, including the score function, so read it before reproducing those numbers. Card: Efficient and Explicit Modelling of Image Hierarchies for Image Restoration.

Reduced keys and values keep every query but shrink the key and value set. ResT pools the token grid with a strided depth-wise convolution before it computes keys and values. The reduction is 8, 4, 2 and 1 across its four stages 2105.13677 §2.2 2105.13677 tab. 1. Among the authors' reduction tests, the depth-wise convolution beats average and max pooling 2105.13677 tab. 6. On ResT-Lite, under the authors' simpler recipe, the reduced attention reaches 72.88 top-1 against 72.68 for standard multi-head attention. Throughput is 1246 against 512 images per second, but the batch sizes differ, so the speed gap is not a clean comparison 2105.13677 tab. 9 2105.13677 app. A.3. Card: ResT.

ChangeFormer's encoder shortens keys and values by a reduction ratio R, which lowers the cost from O((HW)²) to O((HW)²/R) 2201.01293 §2.1.1 2201.01293 eq. 2. The paper does not give R. Its equations also reduce the queries, which would change the output length and break the residual, so the card follows the released code, which keeps full-length queries 2201.01293 §2.1.1. On LEVIR-CD the authors report F1 90.40 against 89.31 for BIT, a transformer change detector with a ResNet18 encoder 2201.01293 tab. 1. Card: A Transformer-Based Siamese Network for Change Detection.

Linear attention with a decay is the case where the decay is one scalar per step. Causal linear attention is the special case with no decay 2405.21060 §5.2. A decay mask of that form turns masked attention into a state-space recurrence. The chunked algorithm uses matrix products inside blocks of the sequence and a small recurrence between blocks 2405.21060 §6. The authors report 2–8× speed over Mamba's scan at state size 64, and faster than FlashAttention-2 from sequence length 2K 2405.21060 fig. 10 2405.21060 §9.3. Keeping one scalar decay per head excludes Mamba's per-channel decays 2405.21060 §5. Card: Transformers are SSMs.

FLASH uses exact attention inside fixed-size chunks and linear attention between them, so the cost grows linearly with length for a fixed chunk size 2202.10447 §3.2. Each block has one attention head with a squared-ReLU kernel, and a learned gate multiplies the attended values 2202.10447 §2 2202.10447 eq. 3 2202.10447 eq. 5. The authors' ablation finds that a softmax kernel or several heads each worsen perplexity 2202.10447 tab. 1. Across chunks, a running sum of key and value products stands in for pairwise scores. For language modelling the sum covers only earlier chunks, so the model stays causal 2202.10447 eq. 8. Decoding needs a constant amount of work per step 2202.10447 §3.2.1. The authors report 15.11 perplexity against 15.25 for Transformer++ at context 8192 on Wiki-40B, and 62.5 ms per step against 261.0 ms on 64 TPU-v4 cores 2202.10447 tab. 13. Training-cost speedups range from 1.2× to 4.9× over contexts 512 to 8192 2202.10447 §4.2. These are author-reported. There is no official code. The appendix listings divide the scores by the length, which the equations do not, and they apply a different segment mask 2202.10447 app. D 2202.10447 eq. 5. An implementation has to choose one reading and test it. Card: Transformer Quality in Linear Time.

Cross-covariance attention over channels

Cross-covariance attention (XCA) changes the axis that is compared. Instead of an N × N map over tokens, it builds a d × d map over feature channels from the channel-normalised queries and keys. The authors state that its cost is linear in the number of tokens, which is why the map does not grow with the patch count 2106.09681 §3.2. Each head works on d/h channels, so each head's map is (d/h) × (d/h) 2106.09681 §3.2. XCA alone does not let neighbouring patches exchange information, so XCiT adds a depth-wise 3 × 3 convolution step on the patch grid after it 2106.09681 §3.3.

The paper's equation and its appendix algorithm put the softmax on different axes, so the notation alone does not fix the operation. The card follows the released code, and advises checking one layer's output against it before relying on the paper's text 2106.09681 eq. 1 2106.09681 alg. 1.

On ImageNet-1k, XCiT-S12 with 16 × 16 patches reaches 82.0 top-1 without distillation 2106.09681 tab. 1. In the authors' ablation, removing the XCA layer gives 75.9, removing the local patch step gives 80.8, and removing ℓ2 normalisation fails to train 2106.09681 tab. 4. On COCO, the same 16-patch model reaches 45.3 box AP against 46.0 for Swin-T, so it does not beat Swin-T there 2106.09681 tab. 5. At 1024 × 1024 and batch 64 on one V100, the authors measure 7312 MB peak memory and 37 images per second for XCiT-S12/16, against 26915 MB and 29 images per second for Swin-T 2106.09681 tab. D.5. These are the authors' measurements. Card: XCiT.

EdgeNeXt applies the same channel-wise form in a few blocks near the end of a mobile CNN-transformer hybrid, in a block it calls SDTA. Queries and keys are L2-normalised, and the attention matrix is C × C over channels, multiplied by V and added back to the input 2206.10589 eq. 3 2206.10589 eq. 4. The authors state its cost is linear in the number of tokens 2206.10589 §3. On ImageNet-1K at 256 px, the 5.6M-parameter EdgeNeXt-S reaches 79.4% top-1 with 1.30G MAdds, against 78.4% for MobileViT-S, whose spatial attention the authors argue is the main cost on mobile hardware 2206.10589 tab. 2 2206.10589 tab. 3 2206.10589 §2. These are the authors' measurements. Card: EdgeNeXt.

Softmax-free operator attention

The trunk of the transformer-enhanced neural operator replaces the softmax with a summary. Keys and values are summed over all query points first, and each query is then multiplied by that summary, with a point-count factor 2609.16066 §3.1 2609.16066 eq. 3. Keys and values carry trainable layer norm, and the paper writes no output projection after the attention 2609.16066 §3.1. On Rotor 37 the author-reported mean field error is 0.083% against 0.572% for FNO 2609.16066 tab. 4, with 154k parameters 2609.16066 tab. 3. The paper's branch-trunk combination is stated differently in its equation and its text 2609.16066 §3.1. The result comes from one compressor and a validation split only. Card: Transformer-enhanced neural operator.

Token mixers that replace attention

Some models drop the attention step and mix tokens another way. AFNO mixes tokens in the 2D Fourier domain: it transforms the grid, applies a learned block-diagonal channel mix to each frequency, shrinks weak frequencies to zero, and transforms back 2111.13587 §4. On ViT-B/4 inpainting its PSNR is 27.05 against 27.06 for self-attention, at 257.2 against 357.2 GFLOPs 2111.13587 tab. 2. On Cityscapes at 1024 × 1024 with SegFormer-B3, the authors report 80.9 mIoU, against 79.7 for efficient self-attention and 80.5 for Long-Short. Self-attention cannot run at that size in their setup because of memory 2111.13587 tab. 4 2111.13587 §5.6. A reading of the released code suggests it does not reproduce the paper's hard frequency truncation, so a paper-faithful implementation may not match it 2111.13587 §5.3. Card: Adaptive Fourier Neural Operators.

Mamba drops attention and the feed-forward block altogether. Its step size, input matrix and output matrix depend on the current token, while the state matrix stays fixed 2312.00752 §3.2. Because these parameters vary with the token, the layer has no convolution form, and the authors compute it with a fused parallel scan on the GPU 2312.00752 §3.3. On a synthetic Selective Copying task the input-dependent layer reaches 99.8% accuracy, and the fixed-parameter S4 layer in the same block reaches 56.4% 2312.00752 §4.1. On the zero-shot suite, Mamba-1.4B averages 59.7 against 55.2 for Pythia-1.4B 2312.00752 §4.2.2. The paper's advice for raw audio is to keep fixed-parameter layers in the outer stages 2312.00752 app. E.4.1. Card: Mamba.

Dropping tokens between layers

Token pruning removes tokens from the sequence after an early layer, so later layers attend over fewer of them. FastV, for vision-language models, runs the first K layers in full, ranks the image tokens by the attention they receive, and keeps a share of them. It needs no training 2403.06764 §4.1. At K = 2 and R = 50%, the authors report average scores of 69.7 against 69.8 for LLaVA-1.5-7B, and 73.6 against 73.6 for LLaVA-1.5-13B, at about 55% of the theoretical FLOPs 2403.06764 tab. 1. In their retrained-baseline ablation, random ranking at the same setting scores lower on Nocaps and Flickr30k 2403.06764 tab. 7. The wall-clock timing comes from one test on a single A40 with one-token answers 2403.06764 tab. 4. The paper's ranking description and the released code use different attention quantities, so check which one you implement. Card: FastV.

Exact kernels

An exact kernel returns the same output as standard attention and changes only how the work is scheduled on the chip. FlashAttention-3 overlaps data loads, matrix multiplies and softmax across thread groups on NVIDIA Hopper GPUs. The authors report 1.5–2.0× forward speed over FlashAttention-2 in FP16, reaching 740 TFLOPs/s 2407.08608 §4.1. Removing the GEMM-softmax pipelining drops its ablation from 661 to 582 TFLOPs/s 2407.08608 tab. 2. Its FP8 path has RMSE 9.1e-3 against 2.4e-2 for a per-tensor baseline, on synthetic inputs with rare outliers 2407.08608 tab. 3 2407.08608 §4.3. The paper's incoherent processing step for FP8 is not in the released code at the commit the card pins. Card: FlashAttention-3.

Self-attention across agents at each location

OPV2V stacks the feature maps of the ego vehicle and its neighbours, then treats the vehicles as the tokens at each spatial position. Attention runs over vehicles, not over pixels, and the ego keeps only its own output row 2109.07644 §4. The mean number of connected vehicles per frame is 2.89, between 2 and 7 2109.07644 tab. 3. With so few tokens the N × N score matrix is small. Compressing the shared features before sending them is a separate step that the authors evaluate: about 3% AP lost at a 4096× compression rate with VoxelNet, which the card notes is a figure-only result whose setup the paper does not fully state 2109.07644 §5.6 2109.07644 fig. 10. Card: OPV2V.

V2X-ViT applies the same per-location idea to vehicles and roadside units. Its heterogeneous multi-agent self-attention computes attention only between features at the same spatial position across agents. Projections depend on the node type (vehicle or infrastructure), and a weight matrix depends on the edge type, of which there are four 2203.10638 §3.2.1 [arxiv:2203.10638 eq. 1] 2203.10638 eq. 2. On the noisy V2XSet setting, adding this module to the window and split-attention parts raises AP@0.7 from 0.548 to 0.601 2203.10638 tab. 2. Against the authors' noisy-setting baselines, the full model scores 0.614 at AP@0.7, against 0.487 for OPV2V and 0.402 with no fusion 2203.10638 tab. 1. These are author-reported figures on a simulated dataset. Card: V2X-ViT.

Self-attention across objects in each view

Multi-View Transformer for 3D grounding treats the objects of a scene as tokens. In each of N rotated views, the object features go through a shared transformer decoder: self-attention among the objects, then cross-attention to the words 2204.02174 §3.3 2204.02174 eq. 10. The per-view outputs are combined by an order-independent average 2204.02174 §3.1 2204.02174 eq. 11. On Nr3D, a model trained and tested on one view reaches 46.2% overall. Training on four views and testing on one reaches 51.6% 2204.02174 tab. 5. The authors credit the second gain to multi-view training, not only to extra test views 2204.02174 §4.4.2. Card: Multi-View Transformer.

Attention within and across cost-volume tokens

FlowFormer summarises the all-pairs cost volume of each source pixel into K = 8 latent tokens of width D = 128. Then three alternate-group layers apply two kinds of self-attention. Intra-cost-map attention runs over the K tokens of one pixel 2203.16194 eq. 2. Inter-cost-map attention runs, for each token index, over all H·W source pixels, using spatially separable self-attention 2203.16194 eq. 3 2203.16194 §3.2. The decoder's cross-attention is a separate step 2203.16194 eq. 4.

The ablation, run on the RAFT-style schedule, shows that each part helps in a different way. Adding the intra-cost-map part alone raises Sintel train final error from 2.72 to 2.89 but lowers KITTI train F1-all from 16.16 to 15.71. The full stack with one layer gives 1.20 and 2.85 on Sintel, and three layers give 1.10 and 2.57 2203.16194 tab. 2. The authors keep the transformer form for both parts, because MLP and ConvNeXt replacements help Sintel final but hurt KITTI 2203.16194 app. §1. The paper leaves the spatial window and stride of the inter-cost step unstated; the card records the code's values 2203.16194 eq. 3. Card: FlowFormer.

Query-restricted self-attention with a CLS query

The Multimodal Fusion Transformer lets only one token, the classification token built from the second sensor, act as the query. Keys and values still come from all five tokens, so each block produces a single output row 2203.16952 §III-B3 2203.16952 alg. 1. With one query row the score matrix is 1 × N, not N × N. This is a cost observation from the equations, not a measured number. Card: Multimodal Fusion Transformer.

UniFormerV2 uses a learnable query in its deeper blocks. The query cross-attends to every token of a clip, at a cost that is linear in the number of tokens, which the authors contrast with global self-attention 2211.09552 §3.2. On Kinetics-400 the global block alone gives 84.4% top-1, against 83.1% with neither block. On Something-Something V2 it gives 63.3%, against 45.1% with neither block 2211.09552 tab. 9. On SSV2, one query beats four or sixteen: 69.5, 69.1 and 68.6 [arxiv:2211.09552 tab. 14]. Card: UniFormerV2.

Gated self-attention over extra tokens

GLIGEN freezes a pretrained latent diffusion model and inserts a new self-attention layer over the visual tokens plus grounding tokens. Only the visual outputs are kept, and they are added back through a gate β·tanh(γ), where γ is a learned scalar initialised to 0 [arxiv:2301.07093 §4.2] 2301.07093 eq. 8. At initialisation the model therefore equals the pretrained one. The paper reports that the layers begin to attend to the boxes only after roughly 60k to 70k iterations, so short runs may show no grounding effect 2301.07093 app. F. Card: GLIGEN.

Gated attention unit over a small set of tokens

RTMPose uses one gated attention unit (GAU) over the K keypoint tokens of its head. The unit normalises the tokens, produces one hidden projection that is split into a gate, a value and a shared base, and derives queries and keys from that base by per-dimension scale and offset 2303.07399 §3.3 2303.07399 eq. 1. The attention kernel is a squared ReLU, and the paper's constant n in front of it is not defined in the text [arxiv:2303.07399 eq. 2]. In the authors' cumulative COCO validation chain, the step that adds the GAU moves AP from 71.4 to 71.9 2303.07399 §3.3. The chain is cumulative, so this is the gain after the earlier steps, not on its own. Card: RTMPose.

One attention layer per block, between feed-forward layers

EfficientViT places a single self-attention layer in each block, with feed-forward layers before and after it 2305.07027 eq. 1 2305.07027 §3.1. Its cascaded group attention gives each head its own slice of the channels, and each head adds the previous head's output to its input 2305.07027 eq. 3 2305.07027 §3.1. In the authors' 100-epoch ablation, swapping the block for a Swin block costs 3.0 points of top-1 (68.3 against 71.3), and two or three feed-forward layers per side score 70.2 and 65.7 2305.07027 tab. 6. The headline model, M5, reaches 77.1% top-1 at 10,621 images per second on a V100, against 75.2% and 7,560 for MobileNetV3-Large 2305.07027 tab. 2. The paper's text quotes some speed ratios that do not match its table, so use the table values. Card: EfficientViT.

Hybrid backbones: attention in the late layers only

Hybrids keep convolution or a state space model for most layers and put self-attention where the global mixing matters most. MambaVision has four stages. Stages 1 and 2 are convolutional. In stages 3 and 4 the first half of the layers use its Mamba-style mixer and the last half use self-attention 2407.08083 §3.2.2. On ImageNet-1K top-1, attention in the last half gives 82.3%, against 81.5% for attention in the first half and 81.3% for a random pattern 2407.08083 tab. 5. These are the authors' numbers from one backbone and one training recipe.

AlterNet, from the same family of ideas, is a ResNet in which some stage-end convolutional blocks are replaced by local attention blocks. The authors' figures suggest it beats CNNs on CIFAR-100, but the text gives no values 2202.06709 §4.2 2202.06709 fig. 12. Card: How Do Vision Transformers Work?.

YOLOv10 adds one partial self-attention block after its lowest-resolution stage. Only half of the channels pass through attention and a feed-forward network, and the two halves are fused again by a 1 × 1 convolution 2405.14458 §3.2. In the authors' cumulative ablation on the small model, it adds 1.4 AP on top of the large kernel (44.9 to 46.3) and 0.15 ms (2.34 to 2.49 ms) 2405.14458 tab. 10. A full transformer block in the same place gives 46.0 AP at 2.54 ms 2405.14458 tab. 13. Card: YOLOv10.

Parallel convolution and attention branches

Conformer runs a convolutional branch and a transformer branch side by side, and couples them in every block through a Feature Coupling Unit. The CNN features are pooled into patch tokens, and the transformer tokens are upsampled back into the CNN 2105.03889 §3.1 2105.03889 §3.2. The transformer branch works on 14×14 patch embeddings with a class token, using standard multi-head self-attention over those tokens. It has no positional embedding, because the convolutional branch is said to supply location cues 2105.03889 §3.2. On the authors' ablation, the positional embedding gives Conformer-S 83.5 against 83.4 without it, and DeiT-S 79.8 against 77.4 2105.03889 tab. 6. The authors report 83.4% ImageNet-1k top-1 for Conformer-S at 10.6 G MACs, against 81.8% for DeiT-B at 17.6 G 2105.03889 tab. 2. Their serial hybrid, a ResNet-26d followed by DeiT-S, scores 80.2 at 3.7 G MACs, against 81.9 for Conformer-S/32 at 7.0 G 2105.03889 tab. 5. The compute differs between those two, so that comparison does not isolate the structure. Card: Conformer.

Frozen self-attention reused on a new modality

Universal computation keeps the self-attention and feedforward blocks of a GPT-2 model pretrained on text, and freezes them. Only the linear input and output layers, the layer-norm parameters and the position embeddings are trained 2103.05247 §2.2. The layer norms carry most of the adaptation. On CIFAR-10, training them alone reaches 54%, against 25% for output-only training 2103.05247 tab. 15. Language pretraining helps in this frozen setup: random initialisation reaches 75.8% on bit memory, against 100% 2103.05247 tab. 2. The authors also report that random transformers already do well on some tasks 2103.05247 §3.2. Card: Pretrained Transformers as Universal Computation Engines.

Masked re-embedding of visited nodes

A learned routing policy embeds the nodes once, although visited nodes stop mattering for later choices. The Embedding Glimpse layer in MDAM recomputes only the top attention layer, with attention to visited nodes set to minus infinity. It runs every p steps, with p between 2 and 8 depending on instance size, rather than at every step 2012.10638 §3.2 2012.10638 §4. The encoder is three blocks of multi-head self-attention, with eight heads and width 128 2012.10638 §3.1.1 2012.10638 §4. On 100-node TSP with greedy decoding, removing the layer raises the gap from 2.19% to 2.78% 2012.10638 tab. 2. These are the authors' numbers for one setting. Card: MDAM.

Shared-reference attention across a batch

StyleAligned lets the self-attention of every image in a batch also attend to the keys and values of one reference image. The queries and keys of each target are first normalised to the reference's per-channel statistics. The values are not normalised 2312.02133 §3.2. No weights are trained. The authors report DINO set consistency of 0.51. DreamBooth-LoRA reaches 0.537, but it needs training for each style, and removing the normalisation drops the score to 0.428 2312.02133 tab. 2. Sharing every image with every other image, instead of with one reference, raises DINO to 0.55 and lowers CLIP text alignment from 0.287 to 0.28 2312.02133 tab. 2. Card: StyleAligned.

Memory of past keys and values

Memorizing Transformers adds a non-differentiable memory of past (key, value) pairs to one attention layer. Each query takes its top-k stored pairs by approximate nearest-neighbour search. A learned per-head gate then mixes that output with local causal attention 2203.08913 §3.1 [arxiv:2203.08913 eq. 1] 2203.08913 eq. 2. The stored keys come from earlier parameters, so the authors pretrain with a small memory and fine-tune with a larger one 2203.08913 §3.2 2203.08913 §4.5. On arXiv at a 2048-token context, a 65K memory reaches perplexity 2.26. The Transformer-XL baseline at the same context reaches 2.42 2203.08913 §4.3. The repository also contains a dense variant that is not the paper's kNN method, so the top-k path is the one to reproduce 2203.08913 §3.1. Card: Memorizing Transformers.

Window encoder with a second, conditioned pass

TranAD reads a short window of multivariate readings with a transformer encoder and two decoders 2201.07284 §3.3 2201.07284 eq. 6. Its window encoder applies masked self-attention to the window and then attends over the first encoder's output 2201.07284 §3.3 2201.07284 eq. 5. Inference runs two passes: the first reconstruction error becomes the focus score that conditions the second pass 2201.07284 §3.4. In the authors' ablation, removing the transformer lowers F1 on NAB from 0.9364 to 0.8850, and F1* on SWaT from 0.8094 to 0.6360 2201.07284 tab. 6. The paper's meta-learning step is a MAML update after each pass 2201.07284 §3.4. The released code does not contain that update, nor the adversarial sign, so the ablations for those parts cannot be checked against it. See the TranAD card for the line-by-line comparison.

Per-word side features in the attention input

A keyphrase tagger on tweets can take per-word reading features from the ZuCo corpus, EEG and eye-tracking values averaged over each word type. In the paper's self-attention variant the features are appended after the query vector 2606.26485 §3.2.2. That makes the query wider than the key, and the score in eq. 7 is undefined as written 2606.26485 eq. 7. The card records that the released code attaches the features to the value vector instead. Author-reported F1 on Election-Trec rises from 61.67 for the Bi-LSTM baseline to 67.82 for the self-attention variant with all EEG features. On General-Twitter it rises from 70.67 to 82.46 2606.26485 tab. 4. Without features, the self-attention variant (81.74) and the Bi-LSTM+CRF (81.63) are within 0.2 F1 on General-Twitter 2606.26485 tab. 4, so that gap comes from the features. ZuCo covers only 8.6% and 4.0% of the two microblog vocabularies, so most words get imputed values 2606.26485 tab. 6 2606.26485 §3.1.2. The paper fine-tunes hyperparameters for each dataset and model 2606.26485 §4.1, and it reports no variance test. The card notes that the released code keeps the best epoch by test score, so the reported F1 may be optimistic. Card: Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs.

Attention over the positions of each detected period

MSGNet finds the k strongest periods of a multivariate input with an FFT. For each period it reshapes the series into a grid, applies a graph convolution across the variables, and then runs multi-head self-attention over the positions inside that grid 2401.00423 §4.2 2401.00423 §4.3 2401.00423 eq. 8. The k results are summed with softmax weights over their amplitudes 2401.00423 §4.4 2401.00423 eq. 9. The paper states no mask for the attention step. Removing the attention changes the Flight average MSE from 0.195 to 0.198, and the authors call its gain marginal 2401.00423 tab. 2 2401.00423 §5.6. The card records that the released code differs in several places, including a causal mask inside each period that the paper does not describe. Card: MSGNet.

When to use which

Implementation sketch

Single-head scaled dot-product self-attention. Queries, keys and values are linear maps of the same input. Multi-head versions split the width and run this in parallel, as in the cards 2108.11084 §3.3.2 2203.16952 §III-B3.

# Single-head scaled dot-product self-attention  [arxiv:1904.02874 tab. 2]
# Input:  x (N, d)  one sequence of N tokens
# Output: y (N, d_v)

def self_attention(x, w_q, w_k, w_v, d_k):
    # Queries, keys and values all come from the same sequence [§4.1]
    q = x @ w_q                              # (N, d_k)
    k = x @ w_k                              # (N, d_k)
    v = x @ w_v                              # (N, d_v)
    # Alignment: scaled dot product between every pair of tokens [tab. 2]
    scores = q @ transpose(k) / sqrt(d_k)    # (N, N)
    # Distribution: softmax over the keys in each row [§3]
    weights = softmax(scores, axis=-1)       # (N, N)
    # Output: weighted sum of values [eq. 1]
    return weights @ v                       # (N, d_v)

Where the variants differ, change only the mask or the set of rows:

# Restrictions that turn the sketch above into the variants  [§4] [§3]
# Full:       scores over all N keys, joint image and label
#             tokens (C-Tran).                 [arxiv:2011.14027 §3.2]
# Causal:     mask keys after each position (decoder).
#                                              [arxiv:1904.02874 §5.2]
# Segmented:  attend only inside each of s contiguous
#             segments of the token axis.      [arxiv:2108.11084 §3.3.2]
# Windowed:   attend only inside a local window, shifted
#             on alternate layers.             [arxiv:2111.13680 §3.2]
# Sparse:     keep a subset of keys per query  [arxiv:2205.01138 §4.1-4.2]
# CLS-query:  use one row of q, keep one output row.
#                                              [arxiv:2203.16952 §III-B3]
# Gated:      add the attended visual rows back, scaled by tanh(gamma),
#             with gamma = 0 at the start.     [arxiv:2301.07093 eq. 8]
# Reduced K/V: pool keys and values by stride s; queries kept.
#                                              [arxiv:2105.13677 §2.2]
# Decay mask: replace the softmax with a decay mask, no normalisation.
#                                              [arxiv:2405.21060 §5.2]
# Pruned:     drop low-scoring image tokens after layer K.
#                                              [arxiv:2403.06764 §4.1]
# Masked:     mask visited items to -inf in the top layer, every p steps
#                                              [arxiv:2012.10638 §3.2]
# Shared ref: concatenate the reference's keys and values per image
#                                              [arxiv:2312.02133 §3.2]
# Memory:     also attend to top-k stored (key, value) pairs, gated
#                                              [arxiv:2203.08913 §3.1]
# Galerkin:   sum K^T V over all points first, then multiply by Q
#                                              [arxiv:2609.16066 eq. 3]
# Frozen:     freeze W_q, W_k, W_v and FFN; train I/O layers and norms
#                                              [arxiv:2103.05247 §2.2]
# Column:     one token per categorical column, no position encoding
#                                              [arxiv:2012.06678 §2]
# Cosine:     normalise q and k, divide the scores by a learned scale
#                                              [arxiv:2111.09883 eq. 2]
# Channel:    Q^T K over channels, softmax over channels, mix V
#                                              [arxiv:2106.09681 §3.2]
# Landmark:   attend through m landmark rows of q and k
#                                              [arxiv:2106.00908 eq. 9]
# Linear:     replace the softmax kernel with elu(x) + 1
#                                              [arxiv:2104.00680 §3.2]
# Partial:    attend over half of the channels, then fuse by 1 x 1 conv
#                                              [arxiv:2405.14458 §3.2]
# Cascade:    head j adds the output of head j-1 to its input
#                                              [arxiv:2305.07027 eq. 3]
# Selective:  step size, input and output matrices depend on the token
#                                              [arxiv:2312.00752 §3.2]
# Two-pass:   second pass takes the first pass's error as its input
#                                              [arxiv:2201.07284 §3.4]
# Batch rows: flatten each row's tokens, then attend over the batch
#                                              [arxiv:2106.01342 alg. 1]
# Structural: add degree embeddings; a scalar per shortest-path distance
#                                              [arxiv:2106.05234 eq. 5-6]
# Sparse voxels: attend over non-empty cells in local and dilated ranges
#                                              [arxiv:2109.02497 §3.2]
# Local 1D:   mask keys more than W steps from the query
#                                              [arxiv:2202.07925 §3.1]
# Halo:       each b x b query block shares one (b+2h)^2 memory, no mask
#                                              [arxiv:2103.12731 §3.2]
# Gated unit: one shared base for Q and K, a gate on the output, relu^2 kernel
#                                              [arxiv:2303.07399 eq. 1-2]
# Cost tokens: intra-pixel attention over K tokens, then attention over
#             all pixels for each token index  [arxiv:2203.16194 eq. 2-3]

Training notes that the cards raise:

• Warm-up and learning rate. In the Tensor2Tensor schedule the paper uses, the decay follows the global step count. Multiplying the batch by 8 on 8 GPUs with the same learning_rate raises the effective rate by √8 at a given step 1804.00247 §4.8.3. Warm-up then has to be re-tuned: 8k steps matched 16k, 6k was slower and 2k diverged 1804.00247 §4.8.2.
• Pre-norm or post-norm in the code. SAINT's equations put layer norm after each sub-layer, while its code normalises the input of each sub-layer 2106.01342 eq. 1 2106.01342 eq. 2. Graphormer's paper uses pre-LN 2106.05234 §3.2, but several of its released scripts train post-LN unless a flag is set (see the Graphormer card). The cards do not test which placement reproduces the reported numbers.
• Layer-norm placement. The tutorial describes the original ordering, with layer normalisation after the residual add. It says pre-normalisation performs worse empirically 2205.01138 §5.3. Treat that as the tutorial's claim, not a measured result.
• Post-norm at the default rate. With the layer norm after each residual branch, the authors' baseline diverged at the default learning rate, so it was run at a quarter of that rate. Compare runs only at matched rates 2111.09883 tab. 7.
• Per-head scale. The NLP tutorial's per-head formula divides the scores by the square root of the full model width, not the head width. It does not say which the implementer should use 2311.17633 §2.3.

Contradictions

• Why do transformers need so much data? The survey says transformers need large training sets because they carry a weak inductive bias 1904.02874 §5.2. ESRT's authors make the same point about vision transformers 2108.11084 §1. The authors of the vision-transformer analysis attribute the weak small-data results to optimisation instead. They report a loss with many negative Hessian eigenvalues early in training, and they state that the model does not overfit even at 2% of CIFAR-100 2202.06709 §2 2202.06709 fig. 4 2202.06709 fig. 5. The Multimodal Fusion Transformer reports the best overall accuracy in each of its settings on scarce hyperspectral training samples [arxiv:2203.16952 tab. I], but its numbers depend on how checkpoints were selected (see its card). These are different explanations of related observations, and the cards do not settle which is right.

• Global or local attention. The vision-transformer analysis argues that data specificity, not long-range dependency, is the key property, and that a local 5×5 window beats global attention on CIFAR-100 2202.06709 §2 2202.06709 fig. 7. nnFormer keeps global attention in its bottleneck, and its cumulative ablation adds 0.2 points of average DSC on ACDC from one bottleneck global layer, from 91.26 to 91.46 2109.03201 tab. 6. The two papers differ in task, data and what they vary, so they do not directly test the same question.

• Does a wider reach help? ActionFormer finds almost no change in average mAP across window sizes from 9 to full attention on THUMOS14 2202.07925 tab. 3b. VoTr finds that adding dilated rings to local attention raises KITTI moderate car AP_3D from 75.48 to 78.27 2109.02497 tab. 4. The tasks, inputs and ablations differ, so the two results do not test the same question. HaloNet's window ablation, on ImageNet, finds a 1.3% gain from enlarging the window from 6 × 6 to 12 × 12 2103.12731 §4.3. Its data and task differ from both of the above.

• Is pre-norm worse than post-norm? A time-series tutorial says that pre-normalisation performs worse empirically 2205.01138 §5.3. The NLP tutorial says pre-norm eases optimisation in deep models, and that neither placement is always better 2311.17633 §4.2. Neither source reports a measurement of its own on this point, so treat both as claims and test the placement.

• Are attention weights a usable importance signal? FastV ranks image tokens by the attention they receive and reports near-baseline scores after dropping half of them 2403.06764 §4.1 2403.06764 tab. 1. In its retrained-baseline ablation, random ranking scores lower at the same setting 2403.06764 tab. 7. The survey, citing studies of its own, says weight magnitude is weakly tied to input importance 1904.02874 §7. The FastV card notes that the paper's ranking and the code's ranking differ, so the ablation does not show which one produced the reported numbers. This page does not resolve the conflict. A satellite time-series study reads its attention maps as focusing on a few observations rather than on cloudy dates 1910.10536 §8.4. It shows only the first of three layers, from one trained model, so it is weak evidence either way. The NLP survey says the same debate is open: attention may be a plausible explanation without being a faithful one 1902.02181 §VI-A.

Papers

• An Attentive Survey of Attention Models: taxonomy of attention with self-attention as the same-sequence case, and the O(n²) cost.
• Training Tips for the Transformer Model: learning-rate and warm-up behaviour when the batch grows across GPUs.
• General Multi-label Image Classification with Transformers: full attention over image and label tokens, with a label-count ceiling.
• Transformer for Single Image Super-Resolution: self-attention over segments of unfolded patches to cut memory in a lightweight model.
• nnFormer: Interleaved Transformer for Volumetric Segmentation: local 3D volume attention in the encoder and global attention in the bottleneck.
• OPV2V: self-attention across connected vehicles at each spatial location.
• GMFlow: Learning Optical Flow via Global Matching: 2×2 shifted-window self-attention and cross-attention in a flow network.
• How Do Vision Transformers Work?: attention as spatial smoothing, local windows, and a hybrid ResNet with local attention blocks.
• Multimodal Fusion Transformer for Remote Sensing Image Classification: a CLS token from a second sensor as the only query in each attention block.
• Visual Attention Methods in Deep Learning: An In-Depth Survey: taxonomy of self-attention variants, including linear and axial attention.
• Transformers in Time-series Analysis: A Tutorial: the original self-attention block and its sparse variants for time series.
• Transformers in Remote Sensing: A Survey: Swin-style local windows and hybrid transformer use in remote sensing.
• Transformers in Vision: A Survey: taxonomy of vision designs by their self-attention block, with the quadratic cost of global attention and criss-cross attention in CNNs.
• GLIGEN: Open-Set Grounded Text-to-Image Generation: gated self-attention added to a frozen diffusion model, with the gate starting at zero.
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone: self-attention only in the late layers of stages 3 and 4, with a placement ablation.
• VideoGPT: Video Generation using VQ-VAE and Transformers: axial attention in the autoencoder and a causal transformer prior over codes.
• ResT: An Efficient Transformer for Visual Recognition: keys and values pooled by a strided depth-wise convolution in each stage.
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers: a Fourier-domain token mixer that replaces self-attention.
• Photorealistic Video Generation with Diffusion Models: alternating spatial and spatiotemporal window attention in a video diffusion transformer.
• FastV: token dropping for vision-language model inference: FastV drops image tokens after an early layer, ranked by attention.
• Transformers are SSMs: masked linear attention with one scalar decay per head, computed in chunks.
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision: an exact attention kernel for Hopper GPUs, with an FP8 path.
• RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion: self-attention blocks inside a latent GAN that an RL agent samples from.
• Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems: a self-attention encoder with masked re-embedding of visited nodes.
• Pretrained Transformers as Universal Computation Engines: frozen self-attention and feedforward blocks, with only I/O layers and norms trained.
• Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers: a non-local graph block over the nodes of each region in the mask head.
• Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series: full self-attention over observation triplets, with forecasting pretraining.
• NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation: shifted-window attention inside a CRF depth decoder.
• Memorizing Transformers: a kNN memory of past keys and values in one attention layer.
• Multi-View Transformer for 3D Visual Grounding: self-attention over objects in each rotated view, averaged across views.
• UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer: a learned-query global block on a pretrained image ViT.
• Style Aligned Image Generation via Shared Attention: reference-shared self-attention with AdaIN, with no training.
• Transformer-enhanced neural operator for compressor flow fields: softmax-free Galerkin-type attention in a neural operator.
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings: one token per categorical column, with self-attention across the columns of a row.
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: shifted 7 × 7 windows, with relative position bias and patch merging.
• LoFTR: Detector-Free Local Feature Matching with Transformers: linear self and cross attention on a coarse grid, then local refinement.
• TransMIL: correlated multiple-instance learning for whole-slide images, with Nyström-approximated attention over patch bags and a convolutional position module.
• Swin Transformer V2: Scaling Up Capacity and Resolution: cosine-normalised scores, post-norm and a log-spaced position bias for scaling.
• A Transformer-Based Siamese Network for Change Detection: sequence-reduced attention in a Siamese encoder for change detection.
• TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data: a window transformer with a second pass conditioned on the first pass's error.
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training: attention across the columns of each row and across the rows of a batch.
• Do Transformers Really Perform Bad for Graph Representation?: Graphormer, full attention with degree, shortest-path and edge-path biases.
• SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems: a Transformer encoder over per-frame vehicle states, used as a violation score in a genetic fuzzer.
• Voxel Transformer for 3D Object Detection: attention over non-empty voxels in local and dilated ranges.
• ActionFormer: Localizing Moments of Actions with Transformers: local window self-attention on a feature pyramid for video action localisation.
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention: one attention layer per block between feed-forward layers, with cascaded heads.
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces: an attention-free selective state-space layer with input-dependent parameters.
• YOLOv10: Real-Time End-to-End Object Detection: one partial self-attention block after the lowest-resolution stage.
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++: full self-attention over motion and music tokens, with multi-frame future targets.
• Conformer: Local Features Coupling Global Representations for Visual Recognition: a CNN branch and a transformer branch coupled in every block.
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer: self-attention across vehicles and roadside units at each spatial cell, plus window attention.
• XCiT: Cross-Covariance Image Transformers: cross-covariance attention over feature channels, linear in the token count, with a depth-wise patch step.
• Attention in Natural Language Processing: a survey whose definition of self-attention and whose section on attention as explanation are cited above; it has no experiments.
• Scaling Local Self-Attention for Parameter Efficient Visual Backbones: HaloNet, blocked local attention with haloing and strided attention downsampling.
• FlowFormer: A Transformer Architecture for Optical Flow: intra- and inter-cost-map self-attention over latent cost tokens, then a recurrent cross-attention decoder.
• RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose: one gated attention unit over keypoint tokens in a SimCC head.
• BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data: a convolutional encoder and an 8-layer transformer pre-trained on unlabelled EEG.
• DocFormer: End-to-End Transformer for Document Understanding: separate visual and text attention maps with a shared spatial term.
• Attention Mechanism in Neural Networks: Where it Comes and Where it Goes: a survey of attention from early visual models to Transformer variants; no experiments.
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications: channel-wise self-attention in the SDTA block of a mobile hybrid.
• Self-attention for raw optical Satellite Time Series Classification: a Transformer encoder over raw and cleaned Sentinel-2 date series, compared with LSTM and CNN models.
• Transformer Quality in Linear Time: FLASH, a gated single-head unit with exact attention in chunks and linear attention between them.
• Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs: per-word EEG and eye-tracking features added at the input of a self-attention tagger.
• Efficient and Explicit Modelling of Image Hierarchies for Image Restoration: window attention and anchored stripe attention in parallel, with a channel-attention convolution branch.
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics: bidirectional intra- and cross-modal attention blocks, then self-attention over one joint token sequence.
• Introduction to Transformers: an NLP Perspective: a tutorial on attention, positions, normalisation and warm-up choices; it has no experiments.
• MSGNet: self-attention over the positions of each FFT-detected period, after a per-period graph convolution.