Attention mechanism

Overview

Attention returns a weighted sum of values. Each weight comes from scoring one key against a query, then normalizing the scores. The survey writes every variant as one function, A(q,K,V)=\sum_i p(a(k_i,q))\,v_i, with a pluggable alignment function a and distribution function p 1904.02874 §3. It explains the idea as a learned version of the Nadaraya-Watson estimator, which weights known targets by their similarity to a new query 1904.02874 §2. With softmax, the weights are non-negative and sum to 1 1904.02874 §3. A second survey describes the same three parts, a compatibility function, a distribution function and a weighted sum of values 1902.02181 §II-B. The softmax variants below keep this shape, and the papers differ in what they score, how they normalize, and where the query, keys and values come from. Gated modules, such as NAM, CBAM-style blocks and the Attention U-Net gates, multiply features by a sigmoid mask instead of forming a weighted average. They are marked where they appear.

In encoder-decoder models, the NLP survey describes the decoder as weighing all encoder states at each output step, with weights that depend on how well each state matches a query. It says attention was first applied to translation and helps most on long inputs 1708.02709 §4.4.

Two cards use the same shape without a learned query. One pools a sequence into a single vector by scoring each step on its own 1512.08756 §1.2. Another scores every pixel of one frame against every pixel of the other 2111.13680 §3.1.

Variants

Alignment functions

The survey lists dot product, scaled dot product (divided by the square root of the key dimension), general (a learned matrix between query and key spaces), and additive attention, w^{T}\,\mathrm{act}(W_1 q + W_2 k + b), among others 1904.02874 §3, tab. 2. The additive form lets the key terms be computed once for all queries 1904.02874 §3. The medical report model scores image regions and tag embeddings with this additive form 1711.08195 §3.3, eq. 2–3. Card: On the Automatic Generation of Medical Imaging Reports. For neural machine translation, the NLP survey writes the weights as a softmax over alignment scores e_{ij}=a(s_{i-1},h_j), and describes the alignment model as feed-forward 2003.01200 §5.5.2.

The NLP survey by Galassi et al. lists eleven compatibility functions in one table. Besides the dot, scaled dot, general and additive forms, it includes a convolution-based form that scores windows of neighbouring keys 1902.02181 tab. 4. It gives per-query costs: the dot product is O(n_k d_k) and the additive form is O(n_w n_k d_k) 1902.02181 §IV-B. The paper gives no training method or benchmark numbers, so these costs are the only quantitative content 1902.02181 §I.

An early end-to-end speech translation model uses the additive form for its text model 1612.01744 §2, eq. 4–5. Its speech model adds a term to the score: the previous step's weights are passed through a one-dimensional convolution with a learned filter of size 25, and the result is scaled and added to each position's score 1612.01744 §2, eq. 6, §3. The authors say this discourages the model from translating the same part of the signal twice 1612.01744 §2. The model reads French speech features and writes English text, with no source transcript 1612.01744 §1. On a synthetic BTEC corpus, a 5-model ensemble reaches 48.8 BLEU on a held-out speaker's test1 set with 7 references, against 52.7 for a Google ASR plus phrase-based SMT pipeline 1612.01744 tab. 3. The authors call the results promising and limited by the synthetic data 1612.01744 §5. Card: Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation.

Soft, hard and local positions

Soft attention takes a weighted average over all positions. It is differentiable, but the cost grows quadratically with the number of positions 1904.02874 §4.3. Hard attention samples a few positions from the weights. It is cheaper, but the objective is no longer differentiable, and the survey gives no estimator details 1904.02874 §4.3. Local attention applies soft weights inside a window around one position [arxiv:1904.02874 §4.3]. A 2022 survey describes the same split. Soft attention is trained by standard backpropagation, hard attention samples one location per step and is trained with REINFORCE, and the global and local forms of Luong et al. are written as a content score over all source states or over a window around an aligned position 2204.13154 §3.

Co-attention and multi-level attention

Co-attention attends over two inputs jointly, such as image regions and question words in visual question answering 1904.02874 §4.1. In the medical report model, two attended vectors (over regions and over tags) are concatenated and passed through one fully connected layer into a context vector 1711.08195 §3.3, eq. 4. On IU X-Ray, BLEU-4 is 0.224 with no attention, 0.230 with semantic attention only, 0.238 with visual attention only and 0.247 with co-attention 1711.08195 tab. 1. On PEIR Gross the no-attention variant scores about the same as the CNN-RNN baseline, and the paper attributes its gain to the attention 1711.08195 tab. 1. Multi-level attention stacks attention over successive abstractions, for example words, then sentences, so each level must wait for the one below it [arxiv:1904.02874 §4.2, tab. 3].

One joint self-attention stack over two modalities

TransVG skips a separate co-attention step. Visual tokens, language tokens and one learned [REG] token are concatenated into a single sequence, and six post-norm encoder layers with 256 dimensions run over all of it [arxiv:2104.08541 §3.2, eq. 4]. The paper says attention lets visual tokens attend to visual tokens, language tokens to visual tokens, and so on, with no fusion rule set in advance 2104.08541 §3.2. The box is read from the output state of the [REG] token 2104.08541 §3.2. Padded image pixels and padded words are excluded from attention 2104.08541 §4.2. The ablation in the paper removes the two branch transformers, not the fusion stack, so it does not isolate the joint attention. On ReferItGame test with ResNet-50, accuracy is 64.24 with neither branch transformer, 68.48 with only the visual one, and 69.76 with both 2104.08541 tab. 4. Among the learned starts for [REG], the learned vector gives 80.32 on RefCOCO val, against 77.90 for reusing the language [CLS] output 2104.08541 tab. 3. The authors show [REG] attention maps that land on the referred object, on chosen examples 2104.08541 §4.5, fig. 3; no faithfulness test is run. Card: TransVG: End-to-End Visual Grounding with Transformers.

Bidirectional attention between image and text tokens

IRENE turns a chest radiograph, a free-text complaint, lab values, sex and age into one token sequence. Two bidirectional multimodal blocks run first. In each, every modality attends to itself and to the other modality, and the two results are summed with weight λ = 1.0 2306.00864 p. 10. Ten self-attention blocks with 12 heads then run over the joined sequence [arxiv:2306.00864 p. 10, fig. 2a]. Each lab value, age and sex is one token, and each radiograph is cut into 16×16 patches 2306.00864 p. 10. Author-reported mean AUROC is 0.924 over eight pulmonary diseases, against 0.858 for Perceiver [arxiv:2306.00864 tab. 1]. The method assumes every modality is present, and the authors name missing modalities as an open problem 2306.00864 p. 7. Card: A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics.

Self-attention and the Transformer

In self-attention, queries and keys come from the same sequence. The survey notes that recent work uses this form most often [arxiv:1904.02874 §4.1, tab. 3]. The Transformer uses scaled dot-product self-attention with several heads, six identical encoder and decoder layers and positional encodings 1904.02874 §5.2. Self-attention costs O(n^2) time and memory in sequence length, and the survey lists cheaper approximations, including Linformer, which the survey reports as linear-time 1904.02874 §8.8. The survey's efficient (linear) attention rearranges \rho(QK^T)V into \rho_q(Q)\,(\rho_k(K)^T V), which brings the cost down from O(n^2) to O(n) 2204.07756 §2.1.3.

The NLP survey describes the Transformer as multi-head self-attention with residual connections and layer normalisation, and no recurrence or convolution 1708.02709 §4.5. It gives the scaled dot-product formula but no head count, key size or layer count, so those values have to come from the original Transformer paper 1708.02709 §4.5. Card: Recent Trends in Deep Learning Based Natural Language Processing.

A review of vision transformers in medical imaging describes the same softmax, scaled dot-product, multi-head self-attention. Queries, keys and values are linear projections of the tokens 2301.03505 eq. 1–3. It surveys efficiency variants and gives their complexity expressions [arxiv:2301.03505 §10]. It runs no experiments. Its Synapse table lists 84.90 Dice for MERIT, which the authors compiled from other papers and did not re-run [arxiv:2301.03505 tab. 6]. Card: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review.

BEHRT applies a BERT-style encoder to a patient's diagnosis history, with each visit as a group of tokens. Its encoder has six layers and 12 heads, and the attention mask hides only padding, so attention is bidirectional over the whole history 1907.09538 §3, tab. 2. The paper pre-trains it by masked disease prediction, then fine-tunes it for next-visit and multi-horizon prediction 1907.09538 §2.4–2.5. Its average precision is higher than DeepR and RETAIN on all three tasks, a comparison the authors report on one UK primary-care dataset 1907.09538 tab. 1. Card: BEHRT: Transformer for Electronic Health Records.

Tranception is a decoder-only protein language model. Each layer splits its heads into four groups, and each group's query, key and value projections pass through a depthwise convolution with a different kernel. The first group has no convolution 2205.13760 §3.1. Position comes from a distance-growing term added to each head's score, with slopes set per head group rather than learned position embeddings 2205.13760 §3.2. The paper gives no kernel sizes, so an implementer must choose them or read the code [arxiv:2205.13760 §3.1]. Card: Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval.

A tutorial on the Transformer gives the same scaled dot-product form with multiple heads, and a causal mask added to the scores before softmax, with -\infty at future positions 2311.17633 §2.3, §2.6. Its per-head formula divides by the square root of the full width, not the head width, and the paper does not say which the implementer should use [arxiv:2311.17633 §2.3]. It also gives a decoder cache that stores earlier decoder states, so each generation step computes only the new one 2311.17633 §2.7. It runs no experiments. Card: Introduction to Transformers: an NLP Perspective.

Pooled keys and values at several scales

P2T keeps global attention but shortens the keys and values. The block reshapes its tokens into a map, average-pools the map to four sizes, adds a 3×3 depthwise convolution to each pooled map, and flattens and concatenates them into one shorter sequence 2106.12011 §III-B.1, eq. 2–4. Queries come from the block input, and keys and values come from this pooled sequence 2106.12011 §III-B.1, eq. 5–6. For the default ratios {12, 16, 20, 24}, the paper gives a pooled length of about N/66.3 2106.12011 §III-B.1. The attention equation omits the heads, and the output projection is not stated 2106.12011 §III-B.1. Author-reported ImageNet-1K top-1 for P2T-Small is 82.4 2106.12011 tab. 2. The paper's ablations compare pooling types and pooled sizes, and it reports that average pooling beats max pooling and a depthwise convolution 2106.12011 tab. 7. Card: P2T: Pyramid Pooling Transformer for Scene Understanding.

Local and global layers with capped logits

Gemma 2 alternates two kinds of attention layer. A local layer lets each query see only the last 4096 positions, and a global layer spans 8192 positions 2408.00118 §2 tab. 1. Attention logits are soft-capped at 50 in every layer, and the final output logits at 30 2408.00118 §2. Each pair of query heads shares one key-value head, so the 9B model has 16 query heads and 8 key-value heads 2408.00118 tab. 1. The authors chose this grouping for faster inference with no clear loss. In the 9B ablation it averages 50.8 across four benchmarks, against 50.3 for full multi-head attention [arxiv:2408.00118 §5, tab. 8]. Shrinking the local window at inference changes perplexity little, but this was measured on the 9B model and one validation set [arxiv:2408.00118 tab. 10]. The paper reports no ablation of either cap 2408.00118 §2. Card: Gemma 2: Improving Open Language Models at a Practical Size.

Attention over graph nodes, agents and feature tracks

In the lane-graph trajectory model, each lane node first attends to nearby agents, using one head with 32-dimensional projections [arxiv:2106.15004 §4.1, app. A.3]. The paper leaves the agent distance threshold unstated [arxiv:2106.15004 §4.1]. The decoder then runs 32-head scaled dot-product attention, with the motion encoding as query and the route's node features as keys and values 2106.15004 §4.3, app. A.6. Card: Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals.

OnePose matches query image features to 3D points, where each point carries a feature track of 2D descriptors. Its aggregation layer normalizes the weights over the track only, using an inner product after a shared projection 2205.12257 §3.2. Each attention group adds self-attention on the query features, self-attention on the 3D points and cross-attention between them, and the network stacks four groups 2205.12257 §3.2, §4.2. The paper says linear attention is used in every attention layer 2205.12257 §4.2. Card: OnePose: One-Shot Object Pose Estimation without CAD Models.

Graph Deviation Network learns its own graph, so no sensor list is given. Each sensor attends over its learned neighbours and itself. The score comes from a learned vector applied to the two sensors' keys, and each key joins an embedding with a projected window 2106.06947 §3.5, eq. 6–8. Each sensor's output is then gated by its own embedding before the output layer [arxiv:2106.06947 §3.5, eq. 9]. On SWaT the author-reported F1 is 0.81 with attention and 0.68 with uniform neighbour weights. On WADI it is 0.57 against 0.48 [arxiv:2106.06947 tab. 3]. The paper reports no standard deviations 2106.06947 tab. 2. Card: Graph Neural Network-Based Anomaly Detection in Multivariate Time Series.

Graph structure as attention bias

A graph transformer adds graph structure to self-attention. A survey sorts the designs into four kinds of bias 2407.09777 §3.1. Node positional bias adds a vector to each node, such as one-hot hop distances or Laplacian eigenvectors 2407.09777 §3.1.1. Edge structural bias encodes distances, edge types or substructures 2407.09777 §3.1.2. Message-passing bias combines a neighbourhood step with attention in one of three placements: before the attention stack, in every layer, or after it 2407.09777 §3.1.3, fig. 3. Attention bias changes the scores directly. A local form multiplies each score by a bias b_{ij} inside an exponential and normalizes only over neighbours. A global form adds a graph term to every pair before the softmax [arxiv:2407.09777 §3.1.4]. The local formula does not exclude a pair when b_{ij}=0. Exclusion comes only from normalizing over neighbours, so a port that masks with the bias alone will not match it 2407.09777 §3.1.4. The survey runs no experiments, and its model classification conflicts with its own text on at least one model 2407.09777 §3.2.1, §4.1. Card: Graph Transformers: A Survey.

Exponential smoothing and frequency attention for forecasting

ETSformer replaces dot-product attention in a forecaster with two blocks based on classical smoothing. Exponential smoothing attention gives each output a weighted sum of past values, with weights that decay geometrically with lag and a learned initial state 2202.01381 §4.2.1. The paper computes it with an FFT in O(L log L) 2202.01381 §4.2.1, app. A. Frequency attention takes the discrete Fourier transform of the residual, keeps the K largest-amplitude bases per variable, and rebuilds them as cosines that run past the window 2202.01381 §4.2.2, eq. 4. Each encoder stack removes a seasonal part and a growth part from the residual, and the decoder sums the damped growth and the extended season into the forecast 2202.01381 §4.1, eq. 2–3. The number of retained bases is searched over {0, 1, 2, 3} [arxiv:2202.01381 tab. 3, app. C.1]. Author-reported lowest MSE on 22 of the 24 multivariate rows of the main table, over six datasets [arxiv:2202.01381 tab. 1]. The paper's text counts 40 multivariate settings, so the table should be checked before quoting a count 2202.01381 §5.1. Card: ETSformer: Exponential Smoothing Transformers for Time-series Forecasting.

Linear attention with random features

Linear attention drops the softmax over positions and scores with an inner product of feature maps. Sums over the keys can then be computed once and reused for every query. Random Fourier features approximate the softmax kernel this way 2103.02143 §2.2, §3.1, eq. 5. Random Feature Attention (RFA) normalizes queries and keys to unit norm in its main derivation, and the paper calls this a choice rather than a requirement 2103.02143 §3.3, app. A.4. In the causal form, a running matrix state and a running vector replace the pairwise sums. The state has a fixed size, so a decoding step does not revisit the prefix 2103.02143 eq. 6, alg. 1. An optional sigmoid gate decays the old state and adds recency bias 2103.02143 §3.2, eq. 7. On WikiText-103 the gated Gaussian variant reaches 25.0 test perplexity with the big model, against 26.2 for the softmax baseline 2103.02143 tab. 1. On machine translation the gated variant matches the baseline BLEU on all three test sets, and its variants decode about 1.8 to 1.9 times faster 2103.02143 tab. 2. The paper's own text puts the gains above roughly 1,000 tokens, and shows a slowdown at short lengths 2103.02143 §3.4. Trained RFA models do not transfer to softmax attention without finetuning 2103.02143 app. C.3. All these numbers are author-reported.

Sparse and signed weights

Softmax gives dense weights. Sparsemax and entmax give non-zero weight to only a few positions, which the survey suggests for summarization and question answering 1904.02874 §3. Quasi-attention uses signed weights that do not sum to 1, so the model can subtract vectors 1904.02874 §3. The survey reports no experiments on these alternatives.

Pooling a sequence with per-step scores

Feed-forward attention drops the query. Each step gets one scalar score from its own state, a softmax over time turns the scores into weights, and the output is the weighted sum of the states 1512.08756 §1.2, eq. 1. The baseline replaces the weights with a plain mean over time [arxiv:1512.08756 eq. 2]. On the synthetic addition and multiplication tasks, the weighted version reaches perfect accuracy at every fixed length tested. The mean does not reach 100% at the two longest multiplication lengths within 100 epochs 1512.08756 tab. 1. With one model trained on lengths from 50 to 10000, the weighted version reaches 99.9% on addition and the mean 77.4% [arxiv:1512.08756 §2.3]. The pooled vector ignores step order, so the paper's two-symbol example cannot be told apart from its reverse 1512.08756 §2.4. These are author-reported results on toy tasks.

Attention over past steps in a recurrent sensor model

CAE-M runs a bidirectional LSTM over a window of multi-sensor signals. Each past step gets a score from a tanh and sigmoid layer, the scores are softmaxed over time, and the weighted sum feeds a non-linear predictor [arxiv:2107.12626 §3.4.1, eq. 14–17]. On PAMAP2, the authors report mF1 0.9639 with the attention and 0.9213 without it 2107.12626 tab. 6. Eq. 17 sums over h, while the text says h−1, so check the index range before porting [arxiv:2107.12626 §3.4.1, eq. 17]. The learning rate and training length are not stated, and the card has no code. Card: Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals.

Attention decoders over image features

A decoder LSTM writes one character per step. At each step it attends over a fused feature sequence, and the weights come from the previous decoder state 1711.04226 §3.3, eq. 1–4. The features are built by reading the image along four directions and weighting each position with learned placement clues 1711.04226 §3.2, eq. 8–10. The paper does not state the scoring form of the attention, so the implementer must choose one [arxiv:1711.04226 eq. 4]. Trained from word labels only, the model reaches 76.8% on the irregular CUTE80 set with no lexicon, against 73.7% for the authors' spatial-transformer baseline 1711.04226 tab. 1. Both are author-reported.

Supervising the attention with a focusing loss

Focusing Attention adds a second network to an attention decoder for scene text. At each output step, the attention centre is the weighted average of the receptive-field centres of the image features [arxiv:1709.02054 §3.2, eq. 9]. A patch around that centre is scored position by position against the character labels, and this loss is added to the decoder loss with weight λ 1709.02054 §3.2, eq. 13–14. The focusing loss applies only to images with character positions, which the paper says are about 30% of the training instances 1709.02054 §4.2. The paper uses λ = 0.01 [arxiv:1709.02054 §4.2]. On IIIT5k with no lexicon, accuracy is 87.4 with the focusing network and 83.7 without it. On IC15 it is 70.6 against 63.3 1709.02054 tab. 2. Labelling 1% of the words already gives 86.2 on IIIT5k [arxiv:1709.02054 tab. 8, tab. 9]. The λ and labelled-share sweeps report results on the test benchmarks, and the paper names no separate validation split [arxiv:1709.02054 §4.4]. These are author-reported numbers. Card: Focusing Attention: Towards Accurate Text Recognition in Natural Images.

Attention over earlier turns with a learned query

COSMIC scores each earlier context state with u_i^\top x_i, where u_i=\tanh(W_s c_i+b_s), and takes a softmax over the history 2010.02795 §3.4, eq. 2. The paper's x_i is ambiguous, and the released code uses other query forms: one built from the current utterance for two datasets, and a learned scalar per state with no query for the other two. Check the card's Paper vs code section before reimplementing this part 2010.02795 §3.4.

Bilinear attention over the dialogue history

DialogueRNN scores each earlier global state against the current utterance with a learned matrix, then takes a softmax over the history. The weighted sum is the context for the speaker's party GRU 1811.00405 §3.3.3, eq. 2–4. The paper does not say what context the first utterance gets [arxiv:1811.00405 §3.3.3]. A second attention runs over all emotion states of the dialogue. It needs future turns, so it is used only by the attention variants and the bidirectional model 1811.00405 §3.4.2, §3.4.3, §3.4.4, eq. 13–14. On IEMOCAP text, the weighted accuracy is 59.33 for DialogueRNN, 61.80 with the emotion attention added, and 63.40 for the bidirectional version with it, against 56.56 for CMN 1811.00405 tab. 2. The ablation removes the party state and the emotion GRU, and does not isolate attention 1811.00405 §5.6, tab. 4. The card's Paper vs code section reports that the released scripts do not all use the score form of eq. 2. Card: DialogueRNN: An Attentive RNN for Emotion Detection in Conversations.

Attention on spiking membrane potentials

In a spiking network, attention acts on the membrane potential before the threshold step, so that gated input is less likely to cross the threshold 2209.13929 §3.3, §6.1. Three gates are defined: temporal, channel and spatial. Each is a sigmoid over average- and max-pooled descriptors through a shared MLP, and the three-dimensional TCSA variant applies the temporal gate to the input, then the channel and spatial gates to the membrane input 2209.13929 §3.3, eq. 6–14. The paper recommends the channel and spatial gates on the integrated membrane potential, and the temporal gate on the convolution output, because the temporal gate cannot act on state that already holds spatio-temporal information 2209.13929 §6.1. On DVS128 Gesture and Gait, TCSA raises accuracy by 5.9 and 4.7 points and cuts spike counts by 84.9% and 81.6% over the re-implemented vanilla baselines 2209.13929 tab. 1. A residual variant reaches 77.08% top-1 on ImageNet-1K with four time steps, author-reported 2209.13929 tab. 2. Energy is estimated from operation counts, not measured on hardware 2209.13929 §4. The released code places attention before the spiking step, not on the membrane potential as the paper recommends, so check the card before reusing it Attention Spiking Neural Networks.

Channel and spatial attention on feature maps

A survey splits the feature-map variants into two families. Channel attention scores each channel after global pooling, as in squeeze-and-excitation: f_s = \sigma(FC(\mathrm{ReLU}(FC(f_g)))) 2204.07756 §2.1.1, eq. 1. Spatial attention scores positions. The spatial branch of CBAM pools across channels and applies a 7×7 convolution 2204.07756 §2.1.2, eq. 3.

SmaAt-UNet places a CBAM after the entry block and after each encoder stage, and feeds the attended features only to the decoder through the skip connections. The encoder path itself is unchanged 2007.04417 §3.1. The paper reports 4,111,389 parameters against 17,272,577 for U-Net, and a CBAM variant of U-Net that adds about 0.16M parameters over the plain model 2007.04417 tab. 1. On the NL-50 test set its MSE ties U-Net at 0.0122 2007.04417 tab. 4. The paper reports no seeds or repeated runs 2007.04417 §5. Card: SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture.

Row and column pooling for channel and position gates

Coordinate attention keeps the position that squeeze-and-excitation drops. It averages each channel along the width, giving one value per row, and along the height, giving one value per column 2103.02907 §3.2.1, eq. 4–5. One shared 1×1 convolution with reduction ratio r mixes the two strips, and two further 1×1 convolutions and sigmoids turn them into a row gate and a column gate. The input is multiplied by both gates [arxiv:2103.02907 §3.2.2, eq. 6–9]. The gates are sigmoid masks, not softmax weights. On ImageNet with MobileNetV2, the authors report 74.3% top-1 with coordinate attention at r = 32, against 73.5% for SE at r = 24 and 72.3% for the baseline [arxiv:2103.02907 tab. 1]. The two settings differ in r, and the paper's own r = 16 row gives 74.7% for coordinate attention against 74.1% for SE at r = 12 [arxiv:2103.02907 tab. 4]. On the transfer tasks, with MobileNetV2 as the backbone, the authors report SSDLite COCO AP rising from 22.3 to 24.5 2103.02907 tab. 6, and DeepLabV3 Pascal VOC mIoU at output stride 16 rising from 70.84 to 73.32 2103.02907 tab. 8. Card: Coordinate Attention for Efficient Mobile Network Design.

Receptive-field weights in place of a shared kernel

RFAConv replaces a k×k convolution with one whose weights differ at each window position. Each k×k window is expanded into its own block of k² values per channel, and a softmax over the k² slots of each channel at that position gives the weights 2304.03198 §3.3, eq. 3. The attention is computed from average-pooled inputs through grouped 1×1 convolutions, so no weight is shared between overlapping windows 2304.03198 §3.2, §3.3. The expanded features are produced by a learned grouped convolution, not copied from the input 2304.03198 §3.3 RFAresnet.py. On ImageNet-1k with ResNet18, top-1 rises from 69.59% to 71.23% for about 0.16M extra parameters 2304.03198 tab. 3. The paper's RFCBAMConv and RFCAConv variants, which apply the same expansion in front of CBAM and coordinate attention, reach 72.15% and 72.01% on the same setup 2304.03198 tab. 4. The authors acknowledge a larger memory cost, because the expanded tensor is k² times the input size, and they say devices with limited storage may not afford it 2304.03198 §4.6. All these numbers are author-reported. Card: RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks.

Attention over pyramid levels, positions and channels

Dynamic Head treats a detector's feature pyramid as one tensor with levels, positions and channels. It applies three attentions in sequence, one per axis. Scale-aware attention gives each level a weight from a pooled summary. Spatial- aware attention samples positions with deformable convolution, using learned offsets. Task-aware attention picks, per channel, the larger of two linear functions of that channel 2106.08322 §3.2, eq. 3–5. In a one-stage detector the head replaces the separate task branches. In a two-stage detector the first two attentions run before RoI pooling 2106.08322 §3.3. In the ResNet-50 ablation at 1x, the baseline reaches 39.0 AP, spatial attention alone reaches 41.4 and all three together reach 42.6 2106.08322 tab. 1. Six blocks give the best AP in the block sweep, and eight give 42.5 [arxiv:2106.08322 tab. 2]. The headline numbers use other backbones and test-time augmentation 2106.08322 §4.3. Card: Dynamic Head: Unifying Object Detection Heads with Attentions.

Sigmoid attention gates on U-Net skip connections

Attention U-Net puts one gate on each skip connection of a 3D U-Net, except the first, lowest-level ones. Each gate takes the skip features and a coarser gating signal, scores each voxel with an additive form, and applies a sigmoid, so every coefficient lies in [0, 1] 1804.03999 §2, eq. 1–2. The output is the skip features times these coefficients 1804.03999 §2. The gating signal is a grid over positions, so each voxel gets its own value, unlike a single global vector 1804.03999 §1, §2. The authors chose sigmoid over softmax, because sequential softmax gives sparser outputs, and they report better convergence with sigmoid 1804.03999 §2. On CT pancreas segmentation with 120 training and 30 test scans, the Dice score is 0.840 against 0.814 for U-Net. The model has 6.40 M parameters against 5.88 M, and its forward pass takes 0.179 s against 0.167 s 1804.03999 tab. 1. On the public Pancreas-CT set, the gain is smaller: 0.712 against 0.690 before fine-tuning 1804.03999 tab. 3. The paper names statistical tests only through p-values 1804.03999 §3. Card: Attention U-Net: Learning Where to Look for the Pancreas.

Batch-norm scales as channel weights

NAM reuses the scale factors γ of a batch norm as channel weights, so it adds no fully connected or convolutional layers. The channel weight is each γ divided by the sum of the γ values in the module, and a sigmoid of the weighted, normalized features gives the mask 2111.12419 §3, eq. 1–2. The module output is the input multiplied by that mask, so it is a gate and not a weighted average of values. An L1 penalty on the γ values pushes weak channels toward zero 2111.12419 §3, eq. 4. On CIFAR-100 with ResNet-50, the channel-only variant reaches 19.09% top-1 error, against 19.44% for CBAM. The gap is 0.35 points from one reported setting, with no seeds or variance [arxiv:2111.12419 tab. 1]. Its channel part has 3,840 parameters against 696,320 for CBAM's channel part, a count the paper derives by arithmetic [arxiv:2111.12419 app. A.1, tab. 3]. Card: NAM: Normalization-based Attention Module.

Attention on a convolutional feature map, added residually

In a sketch-based retrieval encoder, two 1×1 convolutions produce an attention mask from the feature map, and the block output is f + f \cdot att, so the attended features are added to the originals 1904.03451 §4.2. In the paper's ablation, adding this attention to a triplet-only model raises full-gallery mAP on Sketchy-Extended from 0.3020 to 0.3207. The row has no domain or semantic loss 1904.03451 tab. 3. The card's Paper vs code section reports that the released code uses a softmax over the spatial positions of the 7×7 map, which differs from the paper's wording. Card: Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval.

Residual-guided and cross-modality attention

Residual-guided spatial attention builds the attention map from a high-pass residual image and applies it to RGB feature maps [arxiv:2103.12376 §4.2, eq. 2]. Dual cross-modality attention is applied twice, between the RGB stream and the noise stream, through a correlation matrix over all spatial positions 2103.12376 §4.3, eq. 3. Trained on FF++ (F2F), adding residual-guided attention raises the FS AUC from 0.922 to 0.966 and the NT AUC from 0.894 to 0.927. Adding the cross-modality module raises FS and NT further but lowers DF from 0.819 to 0.801 2103.12376 tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

Viewpoint tokens reading their own view

iLRM predicts 3D Gaussians from posed images with one token per viewpoint slot. Its tokens start from Plücker ray embeddings, with no positional embedding 2507.23277 §3.2. Each of 12 update layers runs cross-attention, where a viewpoint token attends only to the image tokens of its own view, then self-attention across all viewpoint tokens of all views [arxiv:2507.23277 §3.2, eq. 3–4]. The paper frames each layer as a feedback correction, though it does not derive this from an objective 2507.23277 §3.2. Author-reported PSNR on RealEstate10K with two views at 256×256 is 28.65, against 27.47 for DepthSplat, and inference takes 0.025 s 2507.23277 tab. 1. The paper also reports 0.53 s for 32 views at 540×960 2507.23277 §1. The released code covers only the two-view, full-resolution setting Card: iLRM.

Cross-attention from map cells to camera pixels

Each cell of a bird's-eye-view grid is a query. Pixels from all six cameras are keys and values, and one softmax runs over every camera and pixel 2205.02833 §3.1. Each key carries a camera-aware embedding computed from the pixel's ray direction and the calibration [arxiv:2205.02833 §3.1, eq. 3]. The query subtracts a per-camera embedding, so a cell is compared only with keys from the camera that query is built for [arxiv:2205.02833 §3.1, eq. 3]. Card: Cross-view Transformers for real-time Map-view Semantic Segmentation.

Deformable cross-attention from map cells to cameras

BEVFormer's queries are the cells of a bird's-eye-view grid, with 200 by 200 cells in the nuScenes setting 2203.17270 §4.2. Each query is lifted to a pillar of four heights, and each 3D point is projected into every camera. The query samples camera features with deformable attention around the points that land in a camera. It then averages over the cameras that were hit 2203.17270 §3.3, eq. 2. A temporal layer lets each query attend to itself and to the previous map, after that map is moved by ego-motion 2203.17270 §3.4, eq. 5. On nuScenes val with a ResNet-101 backbone, the authors report 51.7 NDS 2203.17270 tab. 2. In a static variant with a 100 by 100 grid and single-scale features, global attention over the camera features reaches 0.404 NDS, point-only sampling 0.423 and deformable local attention 0.448. Global attention needs about 36 GB of training memory, against about 20 GB for deformable attention 2203.17270 tab. 5. Raising the number of history frames in training from one to four moves NDS from 0.448 to 0.517 2203.17270 tab. 7. The 56.9 NDS test result uses a VoVNet-99 backbone pre-trained on extra depth data, so the ResNet-101 rows are the like-for-like comparison 2203.17270 tab. 1. These are author-reported numbers. Card: BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers.

Latent bottleneck over a large input

A latent bottleneck avoids attention between every output and every input. A small set of learned latent vectors first attends to the whole input, and then the latents attend to one another. PerAct uses 2048 latents of width 512 to read 8000 patch tokens of width 128, followed by six self-attention layers over the latents 2209.05451 §3.3, app. B. The paper uses this design to keep the full voxel input workable 2209.05451 §3.3. An ablation on the open-drawer task calls the Perceiver crucial to the result [arxiv:2209.05451 §4.2, fig. 3]. In simulation, one multi-task agent beats the C2FARM-BC 3D U-Net in 25 of 36 task and demo-count cells, with average gains of 1.33x at 10 demos and 2.83x at 100 demos 2209.05451 §4.2, tab. 1. The authors report chance-level results when the language goal is removed 2209.05451 §4.2. Training took 16 days on 8 V100 GPUs 2209.05451 §3.4. All these results are author-reported. Card: Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation.

Candidates attending to all points

Group-Free 3D replaces point grouping with attention. Object candidates attend to each other, then to all seed points, through stacked decoder layers 2104.00678 §3.2, eq. 1–4. Each layer's position encoding is built from the box predicted by the previous stage 2104.00678 §3.2.

Cross-attention over objects and words, per view

Multi-View Transformer grounds a sentence to one object in a point-cloud scene. Each object's point features are shared across N rotated copies of the scene, and each copy adds a positional code built from the rotated box centre 2204.02174 §3.2, eq. 6–8. A transformer decoder, shared by all views, runs self-attention among the objects and then cross-attention to the word features, in each of its four layers 2204.02174 §3.3, eq. 10. The per-view outputs are then averaged for each object before scoring 2204.02174 §3.4, eq. 11. In the paper's Nr3D ablation, adding the decoder raises overall accuracy from 36.9 to 40.4, and the full model reaches 55.1 2204.02174 tab. 4. Averaging over views gives 55.1 and max pooling gives 52.7 2204.02174 tab. 6. The paper says the text encoder starts from the first three BERT-base layers, but the card's Paper vs code section reports that the released code builds those layers randomly 2204.02174 §4.2. Card: Multi-View Transformer for 3D Visual Grounding.

Global matching across two images

GMFlow scores every pixel of frame one against every pixel of frame two, with dot products scaled by 1/\sqrt{D}, and takes a softmax over frame two's positions 2111.13680 §3.1, eq. 1–2. The expected position under that distribution, minus the start position, is the flow [arxiv:2111.13680 §3.1, eq. 3–4]. Here the values are coordinates, not features. A self-attention step then spreads confident flow to pixels that could not be matched [arxiv:2111.13680 §3.3, eq. 6–7].

Bidirectional cross-attention between two point clouds

PREDATOR lets two scans attend to each other before any interest points are picked. Every superpoint of one cloud is linked to every superpoint of the other, and the same weights run in both directions [arxiv:2011.13005 §3.3, eq. 4–5]. The attention message, together with the query, goes through an MLP, and the result is added to the cloud's own GNN feature [arxiv:2011.13005 §3.3, eq. 6]. The block has four heads 2011.13005 §3.3. In the 3DLoMatch ablation, with overlap scores on, conditioning without the cross-overlap score lifts registration recall (RR) from 42.8 to 57.5, and adding both gives 59.8 2011.13005 tab. 3. The paper does not state the sample count for that table. Added to FCGF, the same module raises 3DLoMatch RR at 250 samples from 26.8 to 55.9 2011.13005 tab. 10. These are author-reported numbers. Memory grows with the square of the superpoint count 2011.13005 §4.3, which is why the authors train with batch size 1 2011.13005 app. A.3.

Nonlocal attention over putative correspondences

PointDSC runs nonlocal attention over a set of putative correspondences. The weight between two correspondences combines a feature similarity with a spatial length-consistency term 2103.05465 eq. 1–2. The feature similarity is computed in a separate module 2103.05465 eq. 4. In the ablation table, registration recall on 3DMatch with FCGF features is 88.17 for the nonlocal classifier with this attention and 83.30 for a plain nonlocal classifier 2103.05465 tab. 4. The full method reports 93.28, at 0.09 s per pair 2103.05465 tab. 1. Removing the L_sm term gives 92.61 2103.05465 tab. 5. The released code uses different values for the consistency scale and the loss weight than the paper, so the code is not a drop-in copy of the method 2103.05465 §4. Card: PointDSC.

Masked attention over a shrinking set of nodes

A routing decoder attends over the nodes that are still unvisited. The Embedding Glimpse layer recomputes only the top attention layer, sets the attention to visited nodes to minus infinity, and runs this every p steps, not at every step 2012.10638 §3.2, eq. 13–15. The paper's intervals are p = 2, 4, 8 for TSP20, 50, 100 and p = 2, 6, 8 for CVRP 2012.10638 §4. Several decoders share one encoder, and a KL term between their first-step distributions keeps their tours apart 2012.10638 §3.3, eq. 11. In the paper's greedy ablation on CVRP100, the full model has a 4.86% gap, 5.26% without the Embedding Glimpse layer and 5.96% without the multiple decoders 2012.10638 tab. 2. Card: Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems.

Pickup-order mask in a route builder

A pickup-and-delivery routing model adapts JAMPR, an attention-based route builder that grows several routes in parallel. The authors add a mask that blocks a delivery until its pickup is on the route, and they replace hard feasibility with a soft cost: 13 per unit of distance plus 10 per missed client 2608.14156 §3. The authors chose the two weights empirically, and the units of distance are not stated 2608.14156 §3, §4. Author-reported, the model gives better early solutions than OR-Tools at every size tested, and its final cost stays close to OR-Tools only up to 200 customers. At 400 and 1,000 customers, the final cost is about 20% higher than the reference at the longest budget 2608.14156 §5.1–§5.3. Training takes 3 to 336 hours per model on one GPU 2608.14156 tab. 1. Card: Deep RL for pickup-and-delivery routing.

Region-masked modulation of a frozen diffusion model

DenseDiffusion adds a bonus to the attention scores between a word and a pixel of the same region, and a penalty to other pairs, in the first 30% of denoising steps 2308.12964 §3.2, eq. 2–3. The raise and lower amounts move each score toward the query's maximum or minimum, so the scores stay inside their original range 2308.12964 §3.2, eq. 6. The adjustment is scaled by (1 − S), the fraction of the image covered by the segment, so small segments are changed more 2308.12964 §3.2, eq. 2. No weights are trained. On 250 MS-COCO captions, SOA-I is 77.61 against 73.08 for Stable Diffusion, and layout IoU is 34.99 against 23.76 for SD-Pww 2308.12964 tab. 2, tab. 3. The area term trades one metric for another: removing it raises IoU to 38.00 and lowers SOA-I to 73.62 2308.12964 tab. 4. Card: Dense Text-to-Image Generation with Attention Modulation.

Shared attention pooling for two related tasks

Sentiment and sarcasm are trained as one network with a shared GRU. Each task has its own ReLU projection of the GRU states, and attention pooling scores each position with tanh of a projection. The paper's normalization runs over a learned L × L matrix, not over one score vector, and the attention parameters are shared across the two tasks 1901.08014 §2, eq. 1–2, §4. The matrix ties the model to one padded sentence length 1901.08014 §2. A neural tensor network fuses the two task vectors before the sarcasm head [arxiv:1901.08014 §2]. The best variant reaches average F-score 86.66 on 994 sentences with 10-fold cross-validation, against 82.13 for a prior CNN that also uses gaze features 1901.08014 tab. 1. These are author-reported numbers. Card: Sentiment and Sarcasm Classification with Multitask Learning.

Attention over depth bins

AdaBins predicts a set of bin widths for each image with a small transformer over the decoder output. Keys come from a 3×3 convolution on the decoded features, and queries are transformer output embeddings. Each pixel gets a softmax over the bin centres, and its depth is the weighted sum of those centres 2011.14141 §3.3, eq. 1–3. Because the values are scalar bin centres, the attention output is a depth, not a feature vector. Replacing this hybrid output with standard regression lowers NYU delta1 from 0.903 to 0.881, and raises RMS from 0.364 to 0.419 2011.14141 tab. 6. Fixed log-scale bins with the same output reach 0.896 2011.14141 tab. 6. The NYU runs train on a 50K-image subset, and the attention block has 5.8 M of about 78 M parameters 2011.14141 §4.1, §4.2. Card: AdaBins: Depth Estimation using Adaptive Bins.

Learned glimpse for reinforcement-learning driving

A glimpse network chooses which part of a sensor input the convolution sees. The paper learns the glimpse motion with REINFORCE, separately from the end-to-end RL error 1704.02532 §3.4, §4.1.2. The driving pipeline trains end to end, but the paper does not say how the two learning signals combine 1704.02532 §4.3. The experiments are qualitative lane keeping in a simulator, and they give no attention metric 1704.02532 §5. Card: Deep Reinforcement Learning framework for Autonomous Driving.

Robot state as the query over image features

Robot-centric Pooling replaces the final average pooling of a ResNet18 policy with a cross-attention whose query is built from proprioception. Each image location supplies a key and a value, so one query scores all locations 2411.04331 §3.1, eq. 1–7. The policy path then uses a hard mask, the argmax over the scores, and keeps the feature of that one location [arxiv:2411.04331 §3.1, eq. 8]. A contrastive loss on the soft context vector trains the query and the projections. Two augmented views of the same pose, with the arm pasted onto other backgrounds, should give similar context vectors, and different poses should give different ones 2411.04331 §3.2, eq. 9. On simulated reaching with unseen distractors, the authors report 76.4% success for RcP, 31.9% for Spatial-Softmax and 8.3% for R3M 2411.04331 tab. 1. The two baselines also differ in pre-training, so this comparison does not isolate the pooling change 2411.04331 §4.1. Card: Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling.

Transformer over scan patches with a goal query

LiCS splits a 2D laser scan into patches, embeds each patch linearly, adds trainable position embeddings and runs three Transformer encoder layers over them. No class token is used 2406.14947 §II-B. A single goal token is the query of a three-layer decoder. The decoder has no positional encoding and no masked self-attention, but it keeps attention over the encoder output 2406.14947 §II-B. The network outputs linear and angular velocity, and it is trained by behaviour cloning with an MSE loss [arxiv:2406.14947 §II-A, §II-B]. The ablation swaps the Transformer for RNN, LSTM or MLP policies, and all three score lower on the 66 benchmark worlds 2406.14947 tab. 4. The authors report 100% success on those worlds at 1.4 m/s, against 99.49% for the FSMT expert 2406.14947 tab. 2, and 91.67% against 62.50% on 16 hard worlds 2406.14947 tab. 3. These are author-reported numbers from simulation and one hardware setup, and no code is released [arxiv:2406.14947 §IV-D, Paper vs code]. Card: LiCS: Navigation using Learned-imitation on Cluttered Space.

Attention over attention for cloze reading

The Attention-over-Attention Reader first computes, for each query word, a softmax over the document positions, as earlier readers do. It also computes a softmax over the query words for each document word. The second set of weights is averaged over the document positions to give one weight per query word, and that average weights the per-word maps 1607.04423 §3, eq. 6–10. The average has no second softmax, because an average of distributions still sums to one 1607.04423 §3, eq. 9. The paper does not define the normaliser in that average, so read it as the number of document positions. Author-reported single-model test accuracy is 74.4 on CNN, 72.0 on CBTest NE and 69.4 on CBTest CN. The AS Reader scores 69.5, 68.6 and 63.4 on the same sets [arxiv:1607.04423 tab. 2]. Re-ranking the 5-best answers with three language-model features lifts CBTest NE test to 74.0 1607.04423 tab. 4. Card: Attention-over-Attention Neural Networks for Reading Comprehension.

Knowledge-base keys as extra output logits

The KV retrieval net keeps an attention decoder and adds one logit per knowledge-base key to the vocabulary logits at each step. Each key is the sum of the subject and relation embeddings. Its score comes from a small tanh network over the key and the decoder state 1705.05414 §2.3, eq. 7. Each score is added at the position of that key's canonical token, and the softmax runs over the extended vocabulary 1705.05414 §2.3, eq. 8–9. When the decoder emits a canonical token, the system replaces it with the stored value 1705.05414 §2.3. Author-reported test results on the paper's in-car corpus are BLEU 13.2 and entity F1 48.0, against 6.6 and 43.8 for the rule-based system and 11.0 and 37.0 for Copy Net 1705.05414 tab. 3. The test split has 304 dialogues. Removing encoder attention lowers entity F1 to 40.9 1705.05414 tab. 3. The paper names no licence for its corpus 1705.05414 §1. Card: Key-Value Retrieval Networks for Task-Oriented Dialogue.

Per-pixel gates that blend two branches

UIEC^2-Net's attention block takes the raw image and the outputs of an RGB block and an HSV block, concatenated along channels. It predicts six sigmoid maps in [0, 1]: three weight the RGB output and three weight the HSV output. The final image is the sum of the two weighted outputs 2103.07138 §III-A. Because the maps are sigmoid masks, the two weights at a pixel do not have to sum to one. The paper's channel description and its combining sentence name the HSV output in different orders, so state the channel order in a port [arxiv:2103.07138 §III-A]. Author-reported real-world PSNR is 24.57 dB for the full model, against 23.82 with the RGB block alone and 24.18 without the attention block [arxiv:2103.07138 tab. IV]. The paper's loss and layer counts differ from the released code, so do not port that code as the paper's model; see UIEC^2-Net.

Per-pixel quality gate between two density maps

DecideNet counts crowds from two maps. One comes from detection boxes drawn as Gaussian bumps, and the other from a small regression network. A QualityNet of four convolutional layers reads both maps and the image, and ends in a sigmoid per pixel. That map, K, weights the detection map, and one minus K weights the regression map 1712.06679 §3.4, eq. 4. No query, key or value is involved. The gate is trained with a quality term that pulls K toward a detection-score map, weighted by a λ the paper does not state [arxiv:1712.06679 §4, eq. 8]. On Mall, the authors report MAE 3.93 for plain late fusion, 1.83 with QualityNet and 1.52 with the quality-aware loss. On ShanghaiTech PartB the same three steps give 38.63, 24.93 and 21.53 1712.06679 tab. 4. The paper's main PartB table lists CP-CNN at a lower MAE, 20.1 against 21.53 1712.06679 tab. 2. Each training iteration runs the detector on every patch, and the paper does not report what that costs 1712.06679 §5.1. Card: DecideNet.

Shallow features gated by deep features

SANet's shallow attention module upsamples a deeper feature map to the size of a shallower one. It applies a ReLU to form an attention map, and multiplies that map element-wise into the shallower features 2108.00882 §3.2, eq. 1–2. The paper says the module filters background noise from shallow features and weights each level by its contribution 2108.00882 §3.2. It does not say how the channel counts of the two maps are matched, and the element-wise product needs equal shapes 2108.00882 §3.2. In the ablation on ColonDB, adding this module to the backbone raises mDice from 0.676 to 0.728. Colour exchange and the logit correction add further gains 2108.00882 tab. 2. The backbone-only row is not described in the paper. The released code does not contain this module, so an implementation from the code is a different network; see SANet.

Parallel spatial and temporal streams with learned fusion

Each DSTformer block has two streams over the same features. One applies spatial attention across joints, then temporal attention across frames. The other applies them in the reverse order 2210.06551 §3.2, eq. 4. A learned linear map over the concatenated outputs gives a weight per frame and per joint, normalised with a softmax across the two streams 2210.06551 §3.2, eq. 5. Blocks do not share weights 2210.06551 §3.2. On Human3.6M, averaged over five runs, spatial-then-temporal attention alone gives 40.58 ± 0.31 mm MPJPE, equal averaging of the two streams 39.87 ± 0.32, and the learned fusion 39.25 ± 0.27 2210.06551 tab. 7. Card: MotionBERT: A Unified Perspective on Learning Human Motion Representations.

Attention guided by another branch's hidden state

In a driving model with a trajectory branch and a control branch, each future control step attends over the image grid. An MLP scores each grid cell from the trajectory branch's hidden state for that step and the control branch's hidden state 2206.08129 §3.2.2, eq. 2. The attended feature is merged with the control state 2206.08129 §3.2.2, eq. 3. The paper says the attention at the first step uses only the measurement feature 2206.08129 §3.2.2. The authors show attention maps for two cases, and they say the attention focuses on different regions at different future steps 2206.08129 fig. 4. In the ablation, adding this attention to the temporal module moves the driving score from 42.87 ± 4.77 to 46.08 ± 3.47 2206.08129 tab. 3. The paper does not define the plus-minus values, and the two ranges overlap. Results come from one simulator and one benchmark 2206.08129 §4.1. Card: Trajectory-guided Control Prediction for End-to-end Autonomous Driving.

Localization loss on cross-attention maps

FastComposer supervises the cross-attention of each subject's token during training only. For each token, the loss is the mean attention outside the subject's mask minus the mean attention inside it, averaged over subjects. The paper applies it to five middle U-Net blocks, with weight λ = 0.001 2305.10431 §4.2, eq. 3–4. Masks are needed only in training [arxiv:2305.10431 §4.2]. A separate choice delays subject conditioning: the subject-augmented text embedding is used only for the last part of sampling, and the paper suggests α between 0.6 and 0.8 2305.10431 §4.3, fig. 5. In the paper's ablation, which does not name its evaluation set, the localization loss raises identity preservation from 37.66% to 43.11% and lowers prompt consistency from 25.03% to 24.25% 2305.10431 tab. 3. Card: FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention.

Layout tokens as keys in a diffusion UNet

LayoutDiffusion conditions an image-space diffusion model on boxes and classes, with no text input. Object-aware cross-attention lets each image patch attend to the object tokens, with the patch position in the query and the object position in the keys 2303.17189 §3.3. A layout fusion module combines the object tokens before they reach that step 2303.17189 §3. In the ablation, the object-aware attention alone reaches FID 16.76, against 29.94 with neither component 2303.17189 tab. 2. The full model reaches 16.57, and the same layout module with plain cross-attention reaches 16.46 2303.17189 tab. 2. On COCO-Stuff at 256 pixels, the authors report FID 15.61 2303.17189 tab. 1. The conditional stage has 569M parameters and took 216.55 V100-days 2303.17189 tab. 3. Card: LayoutDiffusion.

When to use which

Implementation sketch

Generalized attention with a pluggable score function. Softmax is shown; the survey's sparse alternatives replace only that line 1904.02874 §3.

# Generalized attention: score, normalize, weighted sum  [arxiv:1904.02874 §3] [eq. 1]
# Input:  query (d_q,); keys (N, d_k); values (N, d_v)
# Output: attended vector (d_v,)

def attention(query, keys, values, score_fn):
    # Score each key against the query: dot, scaled dot or additive  [tab. 2]
    scores = [score_fn(query, key) for key in keys]    # (N,)
    # Normalize all N scores together, not one at a time  [§3]
    weights = softmax(scores)                          # (N,), sums to 1
    # Weighted sum of values  [eq. 1]
    return sum(w * v for w, v in zip(weights, values))

Notes for porting:

• Apply the normalization to the full score vector. Softmax, sparsemax and entmax need all scores at once, so writing p per term, as eq. 1 does, is not the implementation 1904.02874 §3.

• Scaled dot product divides by \sqrt{d_k}, where d_k is the dimension of q and k, not the number of positions 1904.02874 tab. 2.

• The survey's biased general row omits a transpose. Check the original source before implementing it 1904.02874 §3.

• Soft attention needs one score per (query, position) pair, so the cost is quadratic in the number of positions 1904.02874 §8.8.

• Co-attention runs this function twice, once over regions and once over tags, and fuses the two outputs 1711.08195 §3.3, eq. 4.

• Pick the logit form explicitly. The cross-view paper's score is a cosine similarity 2205.02833 §3.1, eq. 3. The card's Paper vs code section (item 10) reports that the released code uses a scaled dot product instead.

• Global matching is attention between two frames. The softmax runs over frame two's positions for each pixel of frame one, and the values are coordinates 2111.13680 §3.1, eq. 2–3. The score matrix grows with the square of the pixel count, so plan blocked computation 2111.13680 app. B.

• Linear attention computes the key sums once and applies each query to them. The causal form keeps a fixed-size running state, updated per token 2103.02143 alg. 1. The paper gives no epsilon or clamp for the denominator, so guard near-zero values in your own code [arxiv:2103.02143 alg. 1].

• Feature-track aggregation normalizes over the 2D descriptors of one 3D point only, then adds the result to that point's own descriptor [arxiv:2205.12257 §3.2]. Do not normalize over all points or all queries in this step.

• The OnePose card's Paper vs code section reports that the released code uses additive graph attention with LeakyReLU, and includes the 3D node in the softmax 2205.12257 §3.2. Choose one form and say which in your port.

• Gated modules are not softmax. NAM and CBAM-style blocks take a sigmoid of each element and multiply it into the features. Do not normalize their gates across positions or channels 2111.12419 §3, eq. 2.

• Attention-based pooling over a fixed-length input, as in the sentiment model, needs a normalizer that depends on the length. The paper's learned L × L matrix is one such choice, and it fixes the padded length 1901.08014 §2.

• For one joint sequence, concatenate the modality tokens and the learned token in the order the paper gives, mask the padding, and read the output at the learned token 2104.08541 §3.2, eq. 4. The paper's eq. 1 writes q k without a transpose as typeset, so implement the standard scaled dot product 2104.08541 eq. 1.

• Attention gates on a skip connection are sigmoid masks on the skip features, computed on a grid coarser than the skip. Resample the coefficients to the skip grid before multiplying 1804.03999 §2, fig. 2.

• Per-pixel attention over bins gives a weighted average of scalar bin centres, so the output is one depth per pixel 2011.14141 §3.3, eq. 3. The card's Paper vs code section reports that the released code's positional table holds 500 rows, which caps the token count 2011.14141 §3.3.

• Graph attention over a learned sensor graph has one open choice. The paper's candidate set excludes the sensor itself, but the card's Paper vs code section reports that the released code's top-k includes it 2106.06947 §3.4.

• Emotion attention and the bidirectional model need the whole dialogue. A causal port must drop them 1811.00405 §3.4.2.

• A learned glimpse trained with REINFORCE needs its own learning signal next to the end-to-end loss. The paper does not say how the two are combined 1704.02532 §3.4, §4.3.

• Dynamic Head's scale gate maps one pooled value per level to a weight with a hard-sigmoid. The paper's form and the released code's form differ, so state which one you port 2106.08322 §3.2, eq. 3. The card is Dynamic Head.

• In BEVFormer's spatial attention, divide by the number of cameras that were hit, not by the total number of cameras 2203.17270 §3.3, eq. 2. A camera with no projected point in view is left out of the sum.

• Gemma 2 soft-caps logits as cap times tanh(logit / cap), with cap 50 for attention and 30 for the output 2408.00118 §2. The paper does not give the order of the cap and the mask. The released code caps before masking (Gemma 2, Paper vs code section).

• Gemma 2's 27B model scales queries by the model width over the head count, not by the head size. The paper does not state this 2408.00118 tab. 1.

• Cross-attention between two clouds runs in both directions with one set of weights. Each side's query reads the other side's keys and values 2011.13005 §3.3, eq. 4–5.

• The score scale is 1/\sqrt{b}, with b the feature width [arxiv:2011.13005 eq. 5]. The card's Paper vs code section reports that the released code divides by the per-head width instead, with four heads. Pick one and say which in your port.

• Coordinate attention has two sigmoid gates, one per axis, and both multiply the input. The paper leaves the non-linearity in the shared transform open, so state which one you use 2103.02907 §3.2.2.

• RFAConv's expanded tensor holds k² values for every input value, so budget memory before replacing many layers 2304.03198 §4.6. The softmax runs over the k² slots of each channel at each position [arxiv:2304.03198 §3.3, eq. 3].

• The focusing loss in the scene-text model has no normalization stated. The paper's loss is a plain sum over steps and patch positions, with no division by a count 1709.02054 §3.2, eq. 13. With λ = 0.01 the relative scale of the two losses depends on this choice, which the paper does not state.

• The RcP text calls the alignment scores cosine similarities, but eq. 6 is a softmax of the raw dot product q k^{\top} with no normalization or temperature 2411.04331 §3.1, eq. 6. Follow the equation unless you have the code, since the two differ in scale.

• RcP's policy path takes its mask from an argmax, and the equations give no gradient path from the policy loss to the query, keys or values. The card infers this. The paper does not state it 2411.04331 §3.1, eq. 8. The contrastive term uses the soft context instead 2411.04331 eq. 7.

• LiCS's decoder takes one goal token and no positional encoding, while its encoder keeps trainable position embeddings for the patches 2406.14947 §II-B. Match that split when porting, because the decoder's query is a single goal and not a sequence.

• Attention over attention needs two softmaxes on one score matrix: one over document positions for each query word, and one over query words for each document word. The query-word weights are the mean of the second over the document positions, with no further softmax, and the passage weights are the product of the first with those means 1607.04423 §3, eq. 6–10.

• Knowledge-base logits are not normalised on their own. Add each score to the logit of its canonical token, then take one softmax over the extended vocabulary. The output layer's shape is not stated, so size it to hold the canonical tokens 1705.05414 §2.3, eq. 8–9.

• A two-stream block computes two orders of the same two attentions and mixes them with a learned map over both outputs. The mix is a softmax across the two streams for each frame and joint 2210.06551 §3.2, eq. 4–5.

• Trajectory-guided attention scores each grid cell with an MLP over two hidden states, softmaxes over the grid, then merges the attended feature with the control state. The first step uses the measurement feature only 2206.08129 §3.2.2, eq. 2–3.

• A localization loss on attention maps is a training-time term only. Compute it per subject token on the masked region, then weight it with λ. The balanced form is the mean outside the mask minus the mean inside it 2305.10431 §4.2, eq. 3–4.

Contradictions

• Are attention weights explanations? The survey reports that weights correlate weakly with feature importance, and that permuting them or changing them adversarially can leave predictions unchanged. It concludes that they are at best noisy predictors of importance 1904.02874 §7. A follow-up study argues that this depends on what "explanation" means, and proposes four replacement tests 1908.04626 §2, §5. A second survey says attention may be a plausible explanation without being a faithful one 1902.02181 §VI-A. A review of medical vision transformers doubts attention maps as explanations too 2301.03505 §10. The medical report paper shows per-sentence attention maps and traces one mis-description to semantic attention 1711.08195 §4.5.2, fig. 4. The face-forgery paper says its attention maps highlight abnormal facial boundaries [arxiv:2103.12376 §4.2, fig. 8]. Neither applied paper tests faithfulness, so this is not resolved here.
• Adversarial attention as a counterexample. On four binary LSTM tasks, Wiegreffe and Pinter find adversarial attention distributions with average JSD above 0.4 1908.04626 tab. 4. When those weights are imposed on a non-contextual MLP guide, F1 on Diabetes falls to 0.503, against 0.753 with the base LSTM attention 1908.04626 tab. 3. The paper says Anemia is the exception, with 0.932 against 0.931 1908.04626 tab. 3. The setting is binary tasks with a single-layer BiLSTM 1908.04626 §3.1, and the paper's definition of explanation is the plausibility reading 1908.04626 §5. The other side of this argument is the first bullet above.
• Raw weights versus gradient-weighted maps. On DETR segmentation, Chefer et al. report AP 5.6 for raw attention, 7.2 for Transformer attribution and 13.1 for their gradient-weighted method 2103.15679 tab. 1. This fits the survey's caution about raw weights 1904.02874 §7. The numbers are author-reported and come from one task, so they do not settle faithfulness.
• Gain from the cross-modality module. The face-forgery paper's text says performance "gradually improves" with each module 2103.12376 §5.2. Its ablation table shows the cross-modality module lowers the DF result from 0.819 to 0.801 2103.12376 tab. 3. The FS and NT results do improve.
• Attention maps as evidence. BEHRT shows attention maps linking a diagnosis to distant past diagnoses for two patients 1907.09538 §3, fig. 5. The paper gives these as illustrations and runs no faithfulness test, so they cannot settle the question raised in the first bullet above.
• Attention averages versus causal tests. The IOI paper cites attention patterns as support, for example Name Movers that attend to the indirect object 0.59 of the time 2211.00593 §3.1. It selects heads with path patching and mean ablation instead. Knocking out the three main Name Movers changes the logit difference by only 5%, because backup heads take over 2211.00593 §3.4. The authors note that faithfulness alone does not catch this redundancy 2211.00593 §1. The evidence covers GPT-2 small on one task.
• Attention for time-series forecasting. The LTSF-Linear paper argues that self-attention is permutation-invariant, so it loses temporal order unless position signals are added, and that the order is what matters most in a series 2205.13504 §1. Its linear baseline beats the published Transformer forecasters in all 36 dataset-horizon cells of its main table 2205.13504 tab. 2. The Transformer numbers are copied from the FEDformer paper, and the linear family uses a look-back of 336 against 96 for the Transformers 2205.13504 app. B.2. The paper's own look-back study is a figure without numbers 2205.13504 §5.3. Neither side is re-run at a shared look-back here, so the question stays open for this page. ETSformer, which replaces attention with smoothing-based blocks, reports the lowest MSE on 22 of 24 multivariate rows against Autoformer and Informer 2202.01381 tab. 1. Its baseline numbers come from earlier papers, and its look-back search covers {96, 192, 336, 720} 2202.01381 tab. 3. The linear paper's family uses a look-back of 336 2205.13504 app. B.2, so the two papers' settings differ, and the results cannot be ranked against each other here.
• Full model versus plain cross-attention. In the LayoutDiffusion ablation, the full model reaches FID 16.57 and the plain cross-attention variant 16.46 2303.17189 tab. 2. On FID, the table does not show the full model ahead, and the paper gives no FID spread for these rows 2303.17189 tab. 2. The gain from the layout module is therefore not shown on this metric.

Papers

• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems: feed-forward attention pooling with one scalar score per step, compared against an unweighted mean on toy memory tasks.
• Attention-over-Attention Neural Networks for Reading Comprehension: a second attention over query words weights the per-word maps of a cloze reader, with optional n-gram re-ranking.
• Deep Reinforcement Learning framework for Autonomous Driving: an attention glimpse network feeding a CNN in an end-to-end RL driving pipeline, with an LSTM and a DQN or DDAC head. Qualitative results only.
• Key-Value Retrieval Networks for Task-Oriented Dialogue: an LSTM decoder with attention over knowledge-base keys added to its output logits, for in-car dialogue.
• Recent Trends in Deep Learning Based Natural Language Processing: a survey of deep learning for NLP, covering attention and the Transformer in prose and collected benchmark tables. It runs no experiments.
• Focusing Attention: Towards Accurate Text Recognition in Natural Images: attention decoder with a focusing network that pulls each character's attention back onto the character, for scene text.
• AON: Towards Arbitrarily-Oriented Text Recognition: attention LSTM decoder over four directional feature sequences, fused with learned placement weights.
• On the Automatic Generation of Medical Imaging Reports: co-attention over image regions and predicted tags, with an attention coverage penalty, inside a hierarchical report generator.
• DecideNet: detection and regression density maps blended per pixel by a learned sigmoid gate, for crowd counting.
• Attention U-Net: Learning Where to Look for the Pancreas: additive sigmoid attention gates on the skip connections of a 3D U-Net, for CT pancreas segmentation.
• DialogueRNN: An Attentive RNN for Emotion Detection in Conversations: per-speaker GRU states with attention over earlier global states, plus an emotion attention variant for bidirectional use.
• Sentiment and Sarcasm Classification with Multitask Learning: shared GRU with task-specific attention pooling and a neural tensor network fusion for joint sentiment and sarcasm.
• Attention is not not Explanation: four tests of whether attention weights in a BiLSTM text classifier do real work, including an adversary that gives different attention.
• Attention in Natural Language Processing: a survey giving one general attention model, a four-axis taxonomy and eleven compatibility functions. No experiments.
• An Attentive Survey of Attention Models: a survey giving the generalized attention form, alignment and distribution functions, a four-axis taxonomy and the Transformer. No experiments.
• Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval: VGG-16 encoders with spatial attention for sketch and photo embeddings, trained with triplet, domain and semantic losses.
• BEHRT: Transformer for Electronic Health Records: BERT-style bidirectional self-attention over diagnosis codes, pre-trained by masked prediction and fine-tuned for next-visit prediction.
• Natural Language Processing Advancements By Deep Learning: A Survey: a survey of deep learning in NLP, with attention in neural machine translation. It proposes no method.
• SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture: CBAM attention on the skip connections of a U-Net, with depthwise-separable convolutions, for radar nowcasting.
• COSMIC: COmmonSense knowledge for eMotion Identification in Conversations: a recurrent emotion model with attention over earlier context states; the paper's and the code's attention forms differ.
• PREDATOR: Registration of 3D Point Clouds with Low Overlap: bidirectional cross-attention between two superpoint sets, predicting overlap and matchability scores that bias keypoint sampling.
• AdaBins: Depth Estimation using Adaptive Bins: a mini-transformer predicts per-image bin widths, and per-pixel attention over bin centres gives depth.
• Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems: several attention decoders with a diversity term and masked top-layer re-embedding of unvisited nodes for routing.
• Random Feature Attention: softmax attention replaced by random Fourier features, with linear cost in length and an optional recency gate.
• Coordinate Attention for Efficient Mobile Network Design: direction-aware pooling gives row and column sigmoid gates for a mobile CNN.
• PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency: nonlocal attention over putative correspondences, weighted by spatial consistency, to classify inliers.
• UIEC^2-Net: CNN-based Underwater Image Enhancement Using Two Color Space: an attention block predicts per-pixel sigmoid maps that blend RGB and HSV branch outputs.
• Generalizing Face Forgery Detection with High-frequency Features: residual-guided spatial attention and dual cross-modality attention between an RGB stream and a noise stream.
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers: gradient-weighted attention maps and relevance propagation for co-attention and DETR-style Transformers.
• Group-Free 3D Object Detection via Transformers: object candidates attend to all points through stacked decoder layers, replacing point grouping.
• TransVG: End-to-End Visual Grounding with Transformers: one joint transformer encoder over image tokens, word tokens and a learned [REG] token, regressing one box for a text query.
• Graph Neural Network-Based Anomaly Detection in Multivariate Time Series: learned sensor graph with graph attention over neighbours, scoring sensor deviations for anomaly detection.
• Dynamic Head: Unifying Object Detection Heads with Attentions: stacked scale-, spatial- and task-aware attention blocks that replace a detector's head, with deformable sampling over the feature pyramid.
• Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals: agent-node attention on a lane graph, then a route-conditioned 32-head attention decoder for driving trajectories.
• Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals: a bidirectional LSTM with attention over past steps feeding a non-linear predictor, for multi-sensor time series.
• Shallow Attention Network for Polyp Segmentation: deep features gate shallow features through a ReLU attention map, with colour exchange and a logit correction at inference.
• NAM: Normalization-based Attention Module: channel attention from batch-norm scale factors, with a sigmoid gate and an L1 penalty on the scales.
• GMFlow: Learning Optical Flow via Global Matching: global softmax matching over all pixel pairs, with Transformer-enhanced features and self-attention flow propagation.
• BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers: map queries read multi-camera features through deformable spatial cross-attention, and a temporal attention reads the previous map.
• Multi-View Transformer for 3D Visual Grounding: a transformer decoder shared across rotated views of a point-cloud scene, fusing object features with words.
• Attention Mechanism in Neural Networks: Where it Comes and Where it Goes: a survey from additive attention to the Transformer and its cost fixes, with core formulas. No experiments.
• Visual Attention Methods in Deep Learning: An In-Depth Survey: a survey of attention in vision models, grouped by how scores are computed, with formulas for channel, spatial and self-attention. No experiments.
• Cross-view Transformers for real-time Map-view Semantic Segmentation: map cells attend to the pixels of six calibrated cameras, with a camera-aware key embedding.
• OnePose: One-Shot Object Pose Estimation without CAD Models: graph attention over feature tracks of 3D points, with self and cross attention, matched to query image features.
• Are Transformers Effective for Time Series Forecasting?: one-layer linear forecasters tested against attention-based forecasters at the same look-back window.
• Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval: decoder-only protein transformer with head-grouped depthwise convolutions and grouped ALiBi, mixed with a retrieved profile at inference.
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline: control-branch attention over the image grid, guided by the trajectory branch's hidden state.
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation: latent cross-attention over voxel patches for language-conditioned robot manipulation, trained as one agent over 18 simulated tasks.
• MotionBERT: A Unified Perspective on Learning Human Motion Representations: a dual-stream spatial and temporal Transformer with learned fusion, pretrained to lift corrupted 2D skeletons to 3D.
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small: path patching and mean ablation find a 26-head attention circuit in GPT-2 small for one task.
• Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review: a review of softmax self-attention in vision transformers for medical imaging, with efficiency variants. No experiments of its own.
• LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation: object-aware cross-attention and a layout fusion module in a diffusion UNet, for layout-to-image generation.
• RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks: a convolution whose window weights come from a softmax over each window's k² slots.
• FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention: a cross-attention localization loss during training and delayed subject conditioning at sampling.
• Dense Text-to-Image Generation with Attention Modulation: training-free modulation of Stable Diffusion's attention toward region masks during the first 30% of steps.
• LiCS: Navigation using Learned-imitation on Cluttered Space: a three-layer encoder and decoder Transformer over LiDAR patches with a goal query, trained by behaviour cloning for local navigation.
• Graph Transformers: A Survey: a survey of how graph structure enters self-attention, with bias types, attention cost classes and a design guide. No experiments.
• Gemma 2: Improving Open Language Models at a Practical Size: decoder-only models with alternating local and global attention layers, logit soft-capping and grouped-query attention.
• Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling: proprioception-queried cross-attention over ResNet18 image features, with a hard argmax mask in the policy path and a contrastive loss on augmented views.
• Deep RL for pickup-and-delivery routing: an attention route builder with a pickup-order mask and a soft missed-client penalty.
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation: attention encoder-decoder from French speech to English text, with a convolutional location term for speech.
• P2T: Pyramid Pooling Transformer for Scene Understanding: pooled keys and values at four scales, with depthwise convolutions on each pooled map, in a vision backbone.
• ETSformer: Exponential Smoothing Transformers for Time-series Forecasting: exponential smoothing attention and frequency attention in a forecasting encoder-decoder.
• Attention Spiking Neural Networks: temporal, channel and spatial attention on the membrane potential of spiking neurons.
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics: bidirectional multimodal attention over radiograph, text, lab and demographic tokens, for diagnosis.
• Introduction to Transformers: an NLP Perspective: a tutorial survey of the Transformer's attention, masks, positions and warmup schedule. No experiments.
• iLRM: An Iterative Large 3D Reconstruction Model: a transformer that outputs 3D Gaussians, with viewpoint tokens updated by cross-attention to their own view's image tokens.