---
kind: concept
slug: attention-mechanism
title: Attention mechanism
summary: >-
  Attention computes a weighted average of values, with weights from scoring each key against a query. Variants differ in the scoring function, the normalization (softmax, sparse, learned or sigmoid gate), and where queries, keys and values come from. Soft softmax attention costs quadratic time in the number of positions.
tags: [attention, transformer, interpretability]
papers: ["1512.08756", "1607.04423", "1612.01744", "1704.02532", "1705.05414", "1708.02709", "1709.02054", "1711.04226", "1711.08195", "1712.06679", "1804.03999", "1811.00405", "1901.08014", "1902.02181", "1904.02874", "1904.03451", "1907.09538", "1908.04626", "2003.01200", "2007.04417", "2010.02795", "2011.13005", "2011.14141", "2012.10638", "2103.02143", "2103.02907", "2103.05465", "2103.07138", "2103.12376", "2103.15679", "2104.00678", "2104.08541", "2106.06947", "2106.08322", "2106.12011", "2106.15004", "2107.12626", "2108.00882", "2111.12419", "2111.13680", "2202.01381", "2203.17270", "2204.02174", "2204.07756", "2204.13154", "2205.02833", "2205.12257", "2205.13504", "2205.13760", "2206.08129", "2209.05451", "2209.13929", "2210.06551", "2211.00593", "2301.03505", "2303.17189", "2304.03198", "2305.10431", "2306.00864", "2308.12964", "2311.17633", "2406.14947", "2407.09777", "2408.00118", "2411.04331", "2507.23277", "2608.14156"]
related: [attention-interpretability, graph-transformer, optical-character-recognition, channel-attention, multi-query-attention, kv-cache-management, vision-transformer, self-attention, object-queries, graph-neural-network, perspective-n-point, relative-positional-encoding, retrieval-augmented-generation, linear-attention, layout-to-image-generation, emotion-recognition-in-conversation, semantic-segmentation, monocular-depth-estimation, anomaly-detection, grouped-query-attention, deformable-convolution, object-detection, bev-representation, visual-grounding, multimodal-fusion, imitation-learning, contrastive-learning, text-to-image-personalization, multi-scale-feature-fusion, time-series-forecasting, spiking-neural-network, neural-machine-translation, hierarchical-vision-transformer]
status: draft
updated_at: 2026-11-01
---

## Overview

Attention returns a weighted sum of values. Each weight comes from scoring one
key against a query, then normalizing the scores. The survey writes every
variant as one function, $A(q,K,V)=\sum_i p(a(k_i,q))\,v_i$, with a pluggable
alignment function $a$ and distribution function $p$ [arxiv:1904.02874 §3]. It
explains the idea as a learned version of the Nadaraya-Watson estimator, which
weights known targets by their similarity to a new query [arxiv:1904.02874 §2].
With softmax, the weights are non-negative and sum to 1 [arxiv:1904.02874 §3].
A second survey describes the same three parts, a compatibility function, a
distribution function and a weighted sum of values [arxiv:1902.02181 §II-B].
The softmax variants below keep this shape, and the papers differ in what they
score, how they normalize, and where the query, keys and values come from.
Gated modules, such as NAM, CBAM-style blocks and the Attention U-Net gates,
multiply features by a sigmoid mask instead of forming a weighted average. They
are marked where they appear.

In encoder-decoder models, the NLP survey describes the decoder as weighing all
encoder states at each output step, with weights that depend on how well each
state matches a query. It says attention was first applied to translation and
helps most on long inputs [arxiv:1708.02709 §4.4].

Two cards use the same shape without a learned query. One pools a sequence
into a single vector by scoring each step on its own [arxiv:1512.08756 §1.2].
Another scores every pixel of one frame against every pixel of the other
[arxiv:2111.13680 §3.1].

## Variants

### Alignment functions

The survey lists dot product, scaled dot product (divided by the square root
of the key dimension), general (a learned matrix between query and key spaces),
and additive attention, $w^{T}\,\mathrm{act}(W_1 q + W_2 k + b)$, among others
[arxiv:1904.02874 §3, tab. 2]. The additive form lets the key terms be
computed once for all queries [arxiv:1904.02874 §3]. The medical report model
scores image regions and tag embeddings with this additive form
[arxiv:1711.08195 §3.3, eq. 2–3]. Card: [On the Automatic Generation of Medical Imaging Reports](/papers/1711.08195).
For neural machine translation, the NLP survey writes the weights as a softmax
over alignment scores $e_{ij}=a(s_{i-1},h_j)$, and describes the alignment
model as feed-forward [arxiv:2003.01200 §5.5.2].

The NLP survey by Galassi et al. lists eleven compatibility functions in one
table. Besides the dot, scaled dot, general and additive forms, it includes a
convolution-based form that scores windows of neighbouring keys
[arxiv:1902.02181 tab. 4]. It gives per-query costs: the dot product is
$O(n_k d_k)$ and the additive form is $O(n_w n_k d_k)$ [arxiv:1902.02181 §IV-B].
The paper gives no training method or benchmark numbers, so these costs are the
only quantitative content [arxiv:1902.02181 §I].

An early end-to-end speech translation model uses the additive form for its
text model [arxiv:1612.01744 §2, eq. 4–5]. Its speech model adds a term to the
score: the previous step's weights are passed through a one-dimensional
convolution with a learned filter of size 25, and the result is scaled and
added to each position's score [arxiv:1612.01744 §2, eq. 6, §3]. The authors
say this discourages the model from translating the same part of the signal
twice [arxiv:1612.01744 §2]. The model reads French speech features and writes
English text, with no source transcript [arxiv:1612.01744 §1]. On a synthetic
BTEC corpus, a 5-model ensemble reaches 48.8 BLEU on a held-out speaker's test1
set with 7 references, against 52.7 for a Google ASR plus phrase-based SMT
pipeline [arxiv:1612.01744 tab. 3]. The authors call the results promising and
limited by the synthetic data [arxiv:1612.01744 §5]. Card:
[Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](/papers/1612.01744).

### Soft, hard and local positions

Soft attention takes a weighted average over all positions. It is
differentiable, but the cost grows quadratically with the number of positions
[arxiv:1904.02874 §4.3]. Hard attention samples a few positions from the
weights. It is cheaper, but the objective is no longer differentiable, and the
survey gives no estimator details [arxiv:1904.02874 §4.3]. Local attention
applies soft weights inside a window around one position [arxiv:1904.02874
§4.3]. A 2022 survey describes the same split. Soft attention is trained by
standard backpropagation, hard attention samples one location per step and is
trained with REINFORCE, and the global and local forms of Luong et al. are
written as a content score over all source states or over a window around an
aligned position [arxiv:2204.13154 §3].

### Co-attention and multi-level attention

Co-attention attends over two inputs jointly, such as image regions and
question words in visual question answering [arxiv:1904.02874 §4.1]. In the
medical report model, two attended vectors (over regions and over tags) are
concatenated and passed through one fully connected layer into a context vector
[arxiv:1711.08195 §3.3, eq. 4]. On IU X-Ray, BLEU-4 is 0.224 with no attention,
0.230 with semantic attention only, 0.238 with visual attention only and 0.247
with co-attention [arxiv:1711.08195 tab. 1]. On PEIR Gross the no-attention
variant scores about the same as the CNN-RNN baseline, and the paper attributes
its gain to the attention [arxiv:1711.08195 tab. 1]. Multi-level attention
stacks attention over successive abstractions, for example words, then
sentences, so each level must wait for the one below it [arxiv:1904.02874
§4.2, tab. 3].

### One joint self-attention stack over two modalities

TransVG skips a separate co-attention step. Visual tokens, language tokens and
one learned [REG] token are concatenated into a single sequence, and six
post-norm encoder layers with 256 dimensions run over all of it [arxiv:2104.08541
§3.2, eq. 4]. The paper says attention lets visual tokens attend to visual
tokens, language tokens to visual tokens, and so on, with no fusion rule set in
advance [arxiv:2104.08541 §3.2]. The box is read from the output state of the
[REG] token [arxiv:2104.08541 §3.2]. Padded image pixels and padded words are
excluded from attention [arxiv:2104.08541 §4.2]. The ablation in the paper
removes the two branch transformers, not the fusion stack, so it does not
isolate the joint attention. On ReferItGame test with ResNet-50, accuracy is
64.24 with neither branch transformer, 68.48 with only the visual one, and 69.76
with both [arxiv:2104.08541 tab. 4]. Among the learned starts for [REG], the
learned vector gives 80.32 on RefCOCO val, against 77.90 for reusing the
language [CLS] output [arxiv:2104.08541 tab. 3]. The authors show [REG]
attention maps that land on the referred object, on chosen examples
[arxiv:2104.08541 §4.5, fig. 3]; no faithfulness test is run. Card:
[TransVG: End-to-End Visual Grounding with Transformers](/papers/2104.08541).

### Bidirectional attention between image and text tokens

IRENE turns a chest radiograph, a free-text complaint, lab values, sex and age
into one token sequence. Two bidirectional multimodal blocks run first. In each,
every modality attends to itself and to the other modality, and the two results
are summed with weight λ = 1.0 [arxiv:2306.00864 p. 10]. Ten self-attention
blocks with 12 heads then run over the joined sequence [arxiv:2306.00864 p. 10,
fig. 2a]. Each lab value, age and sex is one token, and each radiograph is cut
into 16×16 patches [arxiv:2306.00864 p. 10]. Author-reported mean AUROC is 0.924
over eight pulmonary diseases, against 0.858 for Perceiver [arxiv:2306.00864
tab. 1]. The method assumes every modality is present, and the authors name
missing modalities as an open problem [arxiv:2306.00864 p. 7]. Card:
[A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864).

### Self-attention and the Transformer

In self-attention, queries and keys come from the same sequence. The survey
notes that recent work uses this form most often [arxiv:1904.02874 §4.1,
tab. 3]. The Transformer uses scaled dot-product self-attention with several
heads, six identical encoder and decoder layers and positional encodings
[arxiv:1904.02874 §5.2]. Self-attention costs $O(n^2)$ time and memory in
sequence length, and the survey lists cheaper approximations, including
Linformer, which the survey reports as linear-time [arxiv:1904.02874 §8.8].
The survey's efficient (linear) attention rearranges $\rho(QK^T)V$ into
$\rho_q(Q)\,(\rho_k(K)^T V)$, which brings the cost down from $O(n^2)$ to
$O(n)$ [arxiv:2204.07756 §2.1.3].

The NLP survey describes the Transformer as multi-head self-attention with
residual connections and layer normalisation, and no recurrence or convolution
[arxiv:1708.02709 §4.5]. It gives the scaled dot-product formula but no head
count, key size or layer count, so those values have to come from the original
Transformer paper [arxiv:1708.02709 §4.5]. Card:
[Recent Trends in Deep Learning Based Natural Language Processing](/papers/1708.02709).

A review of vision transformers in medical imaging describes the same softmax,
scaled dot-product, multi-head self-attention. Queries, keys and values are
linear projections of the tokens [arxiv:2301.03505 eq. 1–3]. It surveys
efficiency variants and gives their complexity expressions [arxiv:2301.03505
§10]. It runs no experiments. Its Synapse table lists 84.90 Dice for MERIT, which
the authors compiled from other papers and did not re-run [arxiv:2301.03505
tab. 6]. Card: [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](/papers/2301.03505).

BEHRT applies a BERT-style encoder to a patient's diagnosis history, with each
visit as a group of tokens. Its encoder has six layers and 12 heads, and the
attention mask hides only padding, so attention is bidirectional over the whole
history [arxiv:1907.09538 §3, tab. 2]. The paper pre-trains it by masked
disease prediction, then fine-tunes it for next-visit and multi-horizon
prediction [arxiv:1907.09538 §2.4–2.5]. Its average precision is higher than
DeepR and RETAIN on all three tasks, a comparison the authors report on one
UK primary-care dataset [arxiv:1907.09538 tab. 1]. Card:
[BEHRT: Transformer for Electronic Health Records](/papers/1907.09538).

Tranception is a decoder-only protein language model. Each layer splits its
heads into four groups, and each group's query, key and value projections pass
through a depthwise convolution with a different kernel. The first group has
no convolution [arxiv:2205.13760 §3.1]. Position comes from a distance-growing
term added to each head's score, with slopes set per head group rather than
learned position embeddings [arxiv:2205.13760 §3.2]. The paper gives no kernel
sizes, so an implementer must choose them or read the code [arxiv:2205.13760
§3.1]. Card: [Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval](/papers/2205.13760).

A tutorial on the Transformer gives the same scaled dot-product form with
multiple heads, and a causal mask added to the scores before softmax, with
$-\infty$ at future positions [arxiv:2311.17633 §2.3, §2.6]. Its per-head
formula divides by the square root of the full width, not the head width, and
the paper does not say which the implementer should use [arxiv:2311.17633
§2.3]. It also gives a decoder cache that stores earlier decoder states, so
each generation step computes only the new one [arxiv:2311.17633 §2.7]. It runs
no experiments. Card:
[Introduction to Transformers: an NLP Perspective](/papers/2311.17633).

### Pooled keys and values at several scales

P2T keeps global attention but shortens the keys and values. The block reshapes
its tokens into a map, average-pools the map to four sizes, adds a 3×3
depthwise convolution to each pooled map, and flattens and concatenates them
into one shorter sequence [arxiv:2106.12011 §III-B.1, eq. 2–4]. Queries come
from the block input, and keys and values come from this pooled sequence
[arxiv:2106.12011 §III-B.1, eq. 5–6]. For the default ratios {12, 16, 20, 24},
the paper gives a pooled length of about N/66.3 [arxiv:2106.12011 §III-B.1].
The attention equation omits the heads, and the output projection is not
stated [arxiv:2106.12011 §III-B.1]. Author-reported ImageNet-1K top-1 for
P2T-Small is 82.4 [arxiv:2106.12011 tab. 2]. The paper's ablations compare
pooling types and pooled sizes, and it reports that average pooling beats max
pooling and a depthwise convolution [arxiv:2106.12011 tab. 7]. Card:
[P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011).

### Local and global layers with capped logits

Gemma 2 alternates two kinds of attention layer. A local layer lets each query
see only the last 4096 positions, and a global layer spans 8192 positions
[arxiv:2408.00118 §2] [tab. 1]. Attention logits are soft-capped at 50 in every
layer, and the final output logits at 30 [arxiv:2408.00118 §2]. Each pair of
query heads shares one key-value head, so the 9B model has 16 query heads and 8
key-value heads [arxiv:2408.00118 tab. 1]. The authors chose this grouping for
faster inference with no clear loss. In the 9B ablation it averages 50.8 across
four benchmarks, against 50.3 for full multi-head attention [arxiv:2408.00118
§5, tab. 8]. Shrinking the local window at inference changes perplexity little,
but this was measured on the 9B model and one validation set [arxiv:2408.00118
tab. 10]. The paper reports no ablation of either cap [arxiv:2408.00118 §2].
Card: [Gemma 2: Improving Open Language Models at a Practical Size](/papers/2408.00118).

### Attention over graph nodes, agents and feature tracks

In the lane-graph trajectory model, each lane node first attends to nearby
agents, using one head with 32-dimensional projections [arxiv:2106.15004 §4.1,
app. A.3]. The paper leaves the agent distance threshold unstated [arxiv:2106.15004
§4.1]. The decoder then runs 32-head scaled dot-product attention, with the
motion encoding as query and the route's node features as keys and values
[arxiv:2106.15004 §4.3, app. A.6]. Card:
[Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](/papers/2106.15004).

OnePose matches query image features to 3D points, where each point carries a
feature track of 2D descriptors. Its aggregation layer normalizes the weights
over the track only, using an inner product after a shared projection
[arxiv:2205.12257 §3.2]. Each attention group adds self-attention on the query
features, self-attention on the 3D points and cross-attention between them, and
the network stacks four groups [arxiv:2205.12257 §3.2, §4.2]. The paper says
linear attention is used in every attention layer [arxiv:2205.12257 §4.2]. Card:
[OnePose: One-Shot Object Pose Estimation without CAD Models](/papers/2205.12257).

Graph Deviation Network learns its own graph, so no sensor list is given. Each
sensor attends over its learned neighbours and itself. The score comes from a
learned vector applied to the two sensors' keys, and each key joins an embedding
with a projected window [arxiv:2106.06947 §3.5, eq. 6–8]. Each sensor's output is
then gated by its own embedding before the output layer [arxiv:2106.06947 §3.5,
eq. 9]. On SWaT the author-reported F1 is 0.81 with attention and 0.68 with
uniform neighbour weights. On WADI it is 0.57 against 0.48 [arxiv:2106.06947 tab.
3]. The paper reports no standard deviations [arxiv:2106.06947 tab. 2]. Card:
[Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](/papers/2106.06947).

### Graph structure as attention bias

A graph transformer adds graph structure to self-attention. A survey sorts the
designs into four kinds of bias [arxiv:2407.09777 §3.1]. Node positional bias
adds a vector to each node, such as one-hot hop distances or Laplacian
eigenvectors [arxiv:2407.09777 §3.1.1]. Edge structural bias encodes distances,
edge types or substructures [arxiv:2407.09777 §3.1.2]. Message-passing bias
combines a neighbourhood step with attention in one of three placements: before
the attention stack, in every layer, or after it [arxiv:2407.09777 §3.1.3, fig. 3].
Attention bias changes the scores directly. A local form multiplies each score
by a bias $b_{ij}$ inside an exponential and normalizes only over neighbours. A
global form adds a graph term to every pair before the softmax [arxiv:2407.09777
§3.1.4]. The local formula does not exclude a pair when $b_{ij}=0$. Exclusion
comes only from normalizing over neighbours, so a port that masks with the bias
alone will not match it [arxiv:2407.09777 §3.1.4]. The survey runs no
experiments, and its model classification conflicts with its own text on at
least one model [arxiv:2407.09777 §3.2.1, §4.1]. Card: [Graph Transformers: A Survey](/papers/2407.09777).

### Exponential smoothing and frequency attention for forecasting

ETSformer replaces dot-product attention in a forecaster with two blocks based
on classical smoothing. Exponential smoothing attention gives each output a
weighted sum of past values, with weights that decay geometrically with lag and
a learned initial state [arxiv:2202.01381 §4.2.1]. The paper computes it with an
FFT in O(L log L) [arxiv:2202.01381 §4.2.1, app. A]. Frequency attention takes
the discrete Fourier transform of the residual, keeps the K largest-amplitude
bases per variable, and rebuilds them as cosines that run past the window
[arxiv:2202.01381 §4.2.2, eq. 4]. Each encoder stack removes a seasonal part
and a growth part from the residual, and the decoder sums the damped growth and
the extended season into the forecast [arxiv:2202.01381 §4.1, eq. 2–3]. The
number of retained bases is searched over {0, 1, 2, 3} [arxiv:2202.01381 tab. 3,
app. C.1]. Author-reported lowest MSE on 22 of the
24 multivariate rows of the main table, over six datasets [arxiv:2202.01381
tab. 1]. The paper's text counts 40 multivariate settings, so the table should
be checked before quoting a count [arxiv:2202.01381 §5.1]. Card:
[ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](/papers/2202.01381).

### Linear attention with random features

Linear attention drops the softmax over positions and scores with an inner
product of feature maps. Sums over the keys can then be computed once and
reused for every query. Random Fourier features approximate the softmax kernel
this way [arxiv:2103.02143 §2.2, §3.1, eq. 5]. Random Feature Attention (RFA)
normalizes queries and keys to unit norm in its main derivation, and the paper
calls this a choice rather than a requirement [arxiv:2103.02143 §3.3, app. A.4].
In the causal form, a running matrix state and a running vector replace the
pairwise sums. The state has a fixed size, so a decoding step does not revisit
the prefix [arxiv:2103.02143 eq. 6, alg. 1]. An optional sigmoid gate decays the
old state and adds recency bias [arxiv:2103.02143 §3.2, eq. 7]. On WikiText-103
the gated Gaussian variant reaches 25.0 test perplexity with the big model,
against 26.2 for the softmax baseline [arxiv:2103.02143 tab. 1]. On machine
translation the gated variant matches the baseline BLEU on all three test sets,
and its variants decode about 1.8 to 1.9 times faster [arxiv:2103.02143 tab. 2].
The paper's own text puts the gains above roughly 1,000 tokens, and shows a
slowdown at short lengths [arxiv:2103.02143 §3.4]. Trained RFA models do not
transfer to softmax attention without finetuning [arxiv:2103.02143 app. C.3].
All these numbers are author-reported.

### Sparse and signed weights

Softmax gives dense weights. Sparsemax and entmax give non-zero weight to only
a few positions, which the survey suggests for summarization and question
answering [arxiv:1904.02874 §3]. Quasi-attention uses signed weights that do
not sum to 1, so the model can subtract vectors [arxiv:1904.02874 §3]. The
survey reports no experiments on these alternatives.

### Pooling a sequence with per-step scores

Feed-forward attention drops the query. Each step gets one scalar score from
its own state, a softmax over time turns the scores into weights, and the
output is the weighted sum of the states [arxiv:1512.08756 §1.2, eq. 1]. The
baseline replaces the weights with a plain mean over time [arxiv:1512.08756
eq. 2]. On the synthetic addition and multiplication tasks, the weighted
version reaches perfect accuracy at every fixed length tested. The mean does
not reach 100% at the two longest multiplication lengths within 100 epochs
[arxiv:1512.08756 tab. 1]. With one model trained on lengths from 50 to 10000,
the weighted version reaches 99.9% on addition and the mean 77.4% [arxiv:1512.08756
§2.3]. The pooled vector ignores step order, so the paper's two-symbol example
cannot be told apart from its reverse [arxiv:1512.08756 §2.4]. These are
author-reported results on toy tasks.

### Attention over past steps in a recurrent sensor model

CAE-M runs a bidirectional LSTM over a window of multi-sensor signals. Each past
step gets a score from a tanh and sigmoid layer, the scores are softmaxed over
time, and the weighted sum feeds a non-linear predictor [arxiv:2107.12626 §3.4.1,
eq. 14–17]. On PAMAP2, the authors report mF1 0.9639 with the attention and
0.9213 without it [arxiv:2107.12626 tab. 6]. Eq. 17 sums over h, while the text
says h−1, so check the index range before porting [arxiv:2107.12626 §3.4.1, eq.
17]. The learning rate and training length are not stated, and the card has no
code. Card: [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](/papers/2107.12626).

### Attention decoders over image features

A decoder LSTM writes one character per step. At each step it attends over a
fused feature sequence, and the weights come from the previous decoder state
[arxiv:1711.04226 §3.3, eq. 1–4]. The features are built by reading the image
along four directions and weighting each position with learned placement
clues [arxiv:1711.04226 §3.2, eq. 8–10]. The paper does not state the scoring
form of the attention, so the implementer must choose one [arxiv:1711.04226
eq. 4]. Trained from word labels only, the model reaches 76.8% on the irregular
CUTE80 set with no lexicon, against 73.7% for the authors' spatial-transformer
baseline [arxiv:1711.04226 tab. 1]. Both are author-reported.

### Supervising the attention with a focusing loss

Focusing Attention adds a second network to an attention decoder for scene
text. At each output step, the attention centre is the weighted average of
the receptive-field centres of the image features [arxiv:1709.02054 §3.2,
eq. 9]. A patch around that centre is scored position by position against the
character labels, and this loss is added to the decoder loss with weight λ
[arxiv:1709.02054 §3.2, eq. 13–14]. The focusing loss applies only to images
with character positions, which the paper says are about 30% of the training
instances [arxiv:1709.02054 §4.2]. The paper uses λ = 0.01 [arxiv:1709.02054
§4.2]. On IIIT5k with no lexicon, accuracy is 87.4 with the focusing network
and 83.7 without it. On IC15 it is 70.6 against 63.3 [arxiv:1709.02054 tab. 2].
Labelling 1% of the words already gives 86.2 on IIIT5k [arxiv:1709.02054
tab. 8, tab. 9]. The λ and labelled-share sweeps report results on the test
benchmarks, and the paper names no separate validation split [arxiv:1709.02054
§4.4]. These are author-reported numbers. Card: [Focusing Attention: Towards Accurate Text Recognition in Natural Images](/papers/1709.02054).

### Attention over earlier turns with a learned query

COSMIC scores each earlier context state with $u_i^\top x_i$, where
$u_i=\tanh(W_s c_i+b_s)$, and takes a softmax over the history
[arxiv:2010.02795 §3.4, eq. 2]. The paper's $x_i$ is ambiguous, and the
released code uses other query forms: one built from the current utterance for
two datasets, and a learned scalar per state with no query for the other two.
Check the card's Paper vs code section before reimplementing this part
[arxiv:2010.02795 §3.4].

### Bilinear attention over the dialogue history

DialogueRNN scores each earlier global state against the current utterance
with a learned matrix, then takes a softmax over the history. The weighted sum
is the context for the speaker's party GRU [arxiv:1811.00405 §3.3.3, eq. 2–4].
The paper does not say what context the first utterance gets [arxiv:1811.00405
§3.3.3]. A second attention runs over all emotion states of the dialogue. It
needs future turns, so it is used only by the attention variants and the
bidirectional model [arxiv:1811.00405 §3.4.2, §3.4.3, §3.4.4, eq. 13–14]. On
IEMOCAP text, the weighted accuracy is 59.33 for DialogueRNN, 61.80 with the
emotion attention added, and 63.40 for the bidirectional version with it, against
56.56 for CMN [arxiv:1811.00405 tab. 2]. The ablation removes the party state and
the emotion GRU, and does not isolate attention [arxiv:1811.00405 §5.6, tab. 4].
The card's Paper vs code section reports that the released scripts do not all
use the score form of eq. 2. Card: [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](/papers/1811.00405).

### Attention on spiking membrane potentials

In a spiking network, attention acts on the membrane potential before the
threshold step, so that gated input is less likely to cross the threshold
[arxiv:2209.13929 §3.3, §6.1]. Three gates are defined: temporal, channel and
spatial. Each is a sigmoid over average- and max-pooled descriptors through a
shared MLP, and the three-dimensional TCSA variant applies the temporal gate to
the input, then the channel and spatial gates to the membrane input
[arxiv:2209.13929 §3.3, eq. 6–14]. The paper recommends the channel and spatial
gates on the integrated membrane potential, and the temporal gate on the
convolution output, because the temporal gate cannot act on state that already
holds spatio-temporal information [arxiv:2209.13929 §6.1]. On DVS128 Gesture and
Gait, TCSA raises accuracy by 5.9 and 4.7 points and cuts spike counts by 84.9%
and 81.6% over the re-implemented vanilla baselines [arxiv:2209.13929 tab. 1].
A residual variant reaches 77.08% top-1 on ImageNet-1K with four time steps,
author-reported [arxiv:2209.13929 tab. 2]. Energy is estimated from operation
counts, not measured on hardware [arxiv:2209.13929 §4]. The released code
places attention before the spiking step, not on the membrane potential as the
paper recommends, so check the card before reusing it [Attention Spiking Neural
Networks](/papers/2209.13929).

### Channel and spatial attention on feature maps

A survey splits the feature-map variants into two families. Channel attention
scores each channel after global pooling, as in squeeze-and-excitation:
$f_s = \sigma(FC(\mathrm{ReLU}(FC(f_g))))$ [arxiv:2204.07756 §2.1.1, eq. 1].
Spatial attention scores positions. The spatial branch of CBAM pools across
channels and applies a 7×7 convolution [arxiv:2204.07756 §2.1.2, eq. 3].

SmaAt-UNet places a CBAM after the entry block and after each encoder stage,
and feeds the attended features only to the decoder through the skip
connections. The encoder path itself is unchanged [arxiv:2007.04417 §3.1]. The
paper reports 4,111,389 parameters against 17,272,577 for U-Net, and a CBAM
variant of U-Net that adds about 0.16M parameters over the plain model
[arxiv:2007.04417 tab. 1]. On the NL-50 test set its MSE ties U-Net at 0.0122
[arxiv:2007.04417 tab. 4]. The paper reports no seeds or repeated runs
[arxiv:2007.04417 §5]. Card:
[SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture](/papers/2007.04417).

### Row and column pooling for channel and position gates

Coordinate attention keeps the position that squeeze-and-excitation drops.
It averages each channel along the width, giving one value per row, and along
the height, giving one value per column [arxiv:2103.02907 §3.2.1, eq. 4–5].
One shared 1×1 convolution with reduction ratio r mixes the two strips, and
two further 1×1 convolutions and sigmoids turn them into a row gate and a
column gate. The input is multiplied by both gates [arxiv:2103.02907 §3.2.2,
eq. 6–9]. The gates are sigmoid masks, not softmax weights. On ImageNet with
MobileNetV2, the authors report 74.3% top-1 with coordinate attention at r = 32,
against 73.5% for SE at r = 24 and 72.3% for the baseline [arxiv:2103.02907
tab. 1]. The two settings differ in r, and the paper's own r = 16 row gives
74.7% for coordinate attention against 74.1% for SE at r = 12 [arxiv:2103.02907
tab. 4]. On the transfer tasks, with MobileNetV2 as the backbone, the authors
report SSDLite COCO AP rising from 22.3 to 24.5 [arxiv:2103.02907 tab. 6], and
DeepLabV3 Pascal VOC mIoU at output stride 16 rising from 70.84 to 73.32
[arxiv:2103.02907 tab. 8]. Card: [Coordinate Attention for Efficient Mobile Network Design](/papers/2103.02907).

### Receptive-field weights in place of a shared kernel

RFAConv replaces a k×k convolution with one whose weights differ at each
window position. Each k×k window is expanded into its own block of k² values
per channel, and a softmax over the k² slots of each channel at that position
gives the weights [arxiv:2304.03198 §3.3, eq. 3]. The attention is computed
from average-pooled inputs through grouped 1×1 convolutions, so no weight is
shared between overlapping windows [arxiv:2304.03198 §3.2, §3.3]. The expanded
features are produced by a learned grouped convolution, not copied from the
input [arxiv:2304.03198 §3.3] [code: classification/RFAresnet.py#L26-L29]. On
ImageNet-1k with ResNet18, top-1 rises from 69.59% to 71.23% for about 0.16M
extra parameters [arxiv:2304.03198 tab. 3]. The paper's RFCBAMConv and RFCAConv
variants, which apply the same expansion in front of CBAM and coordinate
attention, reach 72.15% and 72.01% on the same setup [arxiv:2304.03198 tab. 4].
The
authors acknowledge a larger memory cost, because the expanded tensor is k²
times the input size, and they say devices with limited storage may not afford
it [arxiv:2304.03198 §4.6]. All these numbers are author-reported. Card:
[RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](/papers/2304.03198).

### Attention over pyramid levels, positions and channels

Dynamic Head treats a detector's feature pyramid as one tensor with levels,
positions and channels. It applies three attentions in sequence, one per axis.
Scale-aware attention gives each level a weight from a pooled summary. Spatial-
aware attention samples positions with deformable convolution, using learned
offsets. Task-aware attention picks, per channel, the larger of two linear
functions of that channel [arxiv:2106.08322 §3.2, eq. 3–5]. In a one-stage
detector the head replaces the separate task branches. In a two-stage detector
the first two attentions run before RoI pooling [arxiv:2106.08322 §3.3]. In the
ResNet-50 ablation at 1x, the baseline reaches 39.0 AP, spatial attention alone
reaches 41.4 and all three together reach 42.6 [arxiv:2106.08322 tab. 1]. Six
blocks give the best AP in the block sweep, and eight give 42.5 [arxiv:2106.08322
tab. 2]. The headline numbers use other backbones and test-time augmentation
[arxiv:2106.08322 §4.3]. Card: [Dynamic Head: Unifying Object Detection Heads with Attentions](/papers/2106.08322).

### Sigmoid attention gates on U-Net skip connections

Attention U-Net puts one gate on each skip connection of a 3D U-Net, except the
first, lowest-level ones. Each gate takes the skip features and a coarser
gating signal, scores each voxel with an additive form, and applies a sigmoid,
so every coefficient lies in [0, 1] [arxiv:1804.03999 §2, eq. 1–2]. The output
is the skip features times these coefficients [arxiv:1804.03999 §2]. The gating
signal is a grid over positions, so each voxel gets its own value, unlike a
single global vector [arxiv:1804.03999 §1, §2]. The authors chose sigmoid over
softmax, because sequential softmax gives sparser outputs, and they report
better convergence with sigmoid [arxiv:1804.03999 §2]. On CT pancreas
segmentation with 120 training and 30 test scans, the Dice score is 0.840
against 0.814 for U-Net. The model has 6.40 M parameters against 5.88 M, and its
forward pass takes 0.179 s against 0.167 s [arxiv:1804.03999 tab. 1]. On the
public Pancreas-CT set, the gain is smaller: 0.712 against 0.690 before
fine-tuning [arxiv:1804.03999 tab. 3]. The paper names statistical tests only
through p-values [arxiv:1804.03999 §3]. Card:
[Attention U-Net: Learning Where to Look for the Pancreas](/papers/1804.03999).

### Batch-norm scales as channel weights

NAM reuses the scale factors γ of a batch norm as channel weights, so it adds
no fully connected or convolutional layers. The channel weight is each γ divided
by the sum of the γ values in the module, and a sigmoid of the weighted,
normalized features gives the mask [arxiv:2111.12419 §3, eq. 1–2]. The module
output is the input multiplied by that mask, so it is a gate and not a weighted
average of values. An L1 penalty on the γ values pushes weak channels toward
zero [arxiv:2111.12419 §3, eq. 4]. On CIFAR-100 with ResNet-50, the channel-only
variant reaches 19.09% top-1 error, against 19.44% for CBAM. The gap is 0.35
points from one reported setting, with no seeds or variance [arxiv:2111.12419
tab. 1]. Its channel part has 3,840 parameters against 696,320 for CBAM's
channel part, a count the paper derives by arithmetic [arxiv:2111.12419 app.
A.1, tab. 3]. Card: [NAM: Normalization-based Attention Module](/papers/2111.12419).

### Attention on a convolutional feature map, added residually

In a sketch-based retrieval encoder, two 1×1 convolutions produce an attention
mask from the feature map, and the block output is $f + f \cdot att$, so the
attended features are added to the originals [arxiv:1904.03451 §4.2]. In the
paper's ablation, adding this attention to a triplet-only model raises
full-gallery mAP on Sketchy-Extended from 0.3020 to 0.3207. The row has no
domain or semantic loss [arxiv:1904.03451 tab. 3]. The card's Paper vs code section reports that the
released code uses a softmax over the spatial positions of the 7×7 map, which
differs from the paper's wording. Card: [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](/papers/1904.03451).

### Residual-guided and cross-modality attention

Residual-guided spatial attention builds the attention map from a high-pass
residual image and applies it to RGB feature maps [arxiv:2103.12376 §4.2,
eq. 2]. Dual cross-modality attention is applied twice, between the RGB stream
and the noise stream, through a correlation matrix over all spatial positions
[arxiv:2103.12376 §4.3, eq. 3]. Trained on FF++ (F2F), adding residual-guided
attention raises the FS AUC from 0.922 to 0.966 and the NT AUC from 0.894 to
0.927. Adding the cross-modality module raises FS and NT further but lowers DF
from 0.819 to 0.801 [arxiv:2103.12376 tab. 3]. Card: [Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376).

### Viewpoint tokens reading their own view

iLRM predicts 3D Gaussians from posed images with one token per viewpoint slot.
Its tokens start from Plücker ray embeddings, with no positional embedding
[arxiv:2507.23277 §3.2]. Each of 12 update layers runs cross-attention, where a
viewpoint token attends only to the image tokens of its own view, then
self-attention across all viewpoint tokens of all views [arxiv:2507.23277 §3.2,
eq. 3–4]. The paper frames each layer as a feedback correction, though it does
not derive this from an objective [arxiv:2507.23277 §3.2]. Author-reported
PSNR on RealEstate10K with two views at 256×256 is 28.65, against 27.47 for
DepthSplat, and inference takes 0.025 s [arxiv:2507.23277 tab. 1]. The paper
also reports 0.53 s for 32 views at 540×960 [arxiv:2507.23277 §1]. The released
code covers only the two-view, full-resolution setting [Card:
iLRM](/papers/2507.23277).

### Cross-attention from map cells to camera pixels

Each cell of a bird's-eye-view grid is a query. Pixels from all six cameras are
keys and values, and one softmax runs over every camera and pixel
[arxiv:2205.02833 §3.1]. Each key carries a camera-aware embedding computed
from the pixel's ray direction and the calibration [arxiv:2205.02833 §3.1,
eq. 3]. The query subtracts a per-camera embedding, so a cell is compared only
with keys from the camera that query is built for [arxiv:2205.02833 §3.1,
eq. 3]. Card: [Cross-view Transformers for real-time Map-view Semantic Segmentation](/papers/2205.02833).

### Deformable cross-attention from map cells to cameras

BEVFormer's queries are the cells of a bird's-eye-view grid, with 200 by 200
cells in the nuScenes setting [arxiv:2203.17270 §4.2]. Each query is lifted to
a pillar of four heights, and each 3D point is projected into every camera. The
query samples camera features with deformable attention around the points that
land in a camera. It then averages over the cameras that were hit
[arxiv:2203.17270 §3.3, eq. 2]. A temporal layer lets each query attend to
itself and to the previous map, after that map is moved by ego-motion
[arxiv:2203.17270 §3.4, eq. 5]. On nuScenes val with a ResNet-101 backbone, the
authors report 51.7 NDS [arxiv:2203.17270 tab. 2]. In a static variant with a
100 by 100 grid and single-scale features, global attention over the camera
features reaches 0.404 NDS, point-only sampling 0.423 and deformable local
attention 0.448. Global attention needs about 36 GB of training memory, against
about 20 GB for deformable attention [arxiv:2203.17270 tab. 5]. Raising the
number of history frames in training from one to four moves NDS from 0.448 to
0.517 [arxiv:2203.17270 tab. 7]. The 56.9 NDS test result uses a VoVNet-99
backbone pre-trained on extra depth data, so the ResNet-101 rows are the
like-for-like comparison [arxiv:2203.17270 tab. 1]. These are author-reported
numbers. Card: [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](/papers/2203.17270).

### Latent bottleneck over a large input

A latent bottleneck avoids attention between every output and every input. A
small set of learned latent vectors first attends to the whole input, and then
the latents attend to one another. PerAct uses 2048 latents of width 512 to
read 8000 patch tokens of width 128, followed by six self-attention layers
over the latents [arxiv:2209.05451 §3.3, app. B]. The paper uses this design
to keep the full voxel input workable [arxiv:2209.05451 §3.3]. An ablation on
the open-drawer task calls the Perceiver crucial to the result [arxiv:2209.05451
§4.2, fig. 3]. In simulation, one multi-task agent beats the C2FARM-BC 3D U-Net
in 25 of 36 task and demo-count cells, with average gains of 1.33x at 10 demos
and 2.83x at 100 demos [arxiv:2209.05451 §4.2, tab. 1]. The authors report
chance-level results when the language goal is removed [arxiv:2209.05451 §4.2].
Training took 16 days on 8 V100 GPUs [arxiv:2209.05451 §3.4]. All these results
are author-reported. Card: [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](/papers/2209.05451).

### Candidates attending to all points

Group-Free 3D replaces point grouping with attention. Object candidates attend
to each other, then to all seed points, through stacked decoder layers
[arxiv:2104.00678 §3.2, eq. 1–4]. Each layer's position encoding is built from
the box predicted by the previous stage [arxiv:2104.00678 §3.2].

### Cross-attention over objects and words, per view

Multi-View Transformer grounds a sentence to one object in a point-cloud scene.
Each object's point features are shared across N rotated copies of the scene,
and each copy adds a positional code built from the rotated box centre
[arxiv:2204.02174 §3.2, eq. 6–8]. A transformer decoder, shared by all views,
runs self-attention among the objects and then cross-attention to the word
features, in each of its four layers [arxiv:2204.02174 §3.3, eq. 10]. The
per-view outputs are then averaged for each object before scoring
[arxiv:2204.02174 §3.4, eq. 11]. In the paper's Nr3D ablation, adding the
decoder raises overall accuracy from 36.9 to 40.4, and the full model reaches
55.1 [arxiv:2204.02174 tab. 4]. Averaging over views gives 55.1 and max
pooling gives 52.7 [arxiv:2204.02174 tab. 6]. The paper says the text encoder
starts from the first three BERT-base layers, but the card's Paper vs code
section reports that the released code builds those layers randomly
[arxiv:2204.02174 §4.2]. Card: [Multi-View Transformer for 3D Visual Grounding](/papers/2204.02174).

### Global matching across two images

GMFlow scores every pixel of frame one against every pixel of frame two, with
dot products scaled by $1/\sqrt{D}$, and takes a softmax over frame two's
positions [arxiv:2111.13680 §3.1, eq. 1–2]. The expected position under that
distribution, minus the start position, is the flow [arxiv:2111.13680 §3.1,
eq. 3–4]. Here the values are coordinates, not features. A self-attention step
then spreads confident flow to pixels that could not be matched [arxiv:2111.13680
§3.3, eq. 6–7].

### Bidirectional cross-attention between two point clouds

PREDATOR lets two scans attend to each other before any interest points are
picked. Every superpoint of one cloud is linked to every superpoint of the
other, and the same weights run in both directions [arxiv:2011.13005 §3.3,
eq. 4–5]. The attention message, together with the query, goes through an MLP,
and the result is added to the cloud's own GNN feature [arxiv:2011.13005
§3.3, eq. 6]. The block has four heads [arxiv:2011.13005 §3.3]. In the
3DLoMatch ablation, with overlap scores on, conditioning without the
cross-overlap score lifts registration recall (RR) from 42.8 to 57.5, and
adding both gives 59.8 [arxiv:2011.13005 tab. 3]. The paper does not state the
sample count for that table. Added to FCGF, the same module raises 3DLoMatch
RR at 250 samples from 26.8 to 55.9 [arxiv:2011.13005 tab. 10]. These are
author-reported numbers. Memory grows with the square of the superpoint count
[arxiv:2011.13005 §4.3], which is why the authors train with batch size 1
[arxiv:2011.13005 app. A.3].

### Nonlocal attention over putative correspondences

PointDSC runs nonlocal attention over a set of putative correspondences. The
weight between two correspondences combines a feature similarity with a spatial
length-consistency term [arxiv:2103.05465 eq. 1–2]. The feature similarity is
computed in a separate module [arxiv:2103.05465 eq. 4]. In the ablation table,
registration recall on 3DMatch with FCGF features is 88.17 for the nonlocal
classifier with this attention and 83.30 for a plain nonlocal classifier
[arxiv:2103.05465 tab. 4]. The full method reports 93.28, at 0.09 s per pair
[arxiv:2103.05465 tab. 1]. Removing the L_sm term gives 92.61
[arxiv:2103.05465 tab. 5]. The released code uses different values for the
consistency scale and the loss weight than the paper, so the code is not a
drop-in copy of the method [arxiv:2103.05465 §4]. Card:
[PointDSC](/papers/2103.05465).

### Masked attention over a shrinking set of nodes

A routing decoder attends over the nodes that are still unvisited. The
Embedding Glimpse layer recomputes only the top attention layer, sets the
attention to visited nodes to minus infinity, and runs this every p steps, not
at every step [arxiv:2012.10638 §3.2, eq. 13–15]. The paper's intervals are
p = 2, 4, 8 for TSP20, 50, 100 and p = 2, 6, 8 for CVRP [arxiv:2012.10638 §4].
Several decoders share one encoder, and a KL term between their first-step
distributions keeps their tours apart [arxiv:2012.10638 §3.3, eq. 11]. In the
paper's greedy ablation on CVRP100, the full model has a 4.86% gap, 5.26% without
the Embedding Glimpse layer and 5.96% without the multiple decoders
[arxiv:2012.10638 tab. 2]. Card: [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638).

### Pickup-order mask in a route builder

A pickup-and-delivery routing model adapts JAMPR, an attention-based route
builder that grows several routes in parallel. The authors add a mask that
blocks a delivery until its pickup is on the route, and they replace hard
feasibility with a soft cost: 13 per unit of distance plus 10 per missed client
[arxiv:2608.14156 §3]. The authors chose the two weights empirically, and the
units of distance are not stated [arxiv:2608.14156 §3, §4]. Author-reported,
the model gives better early solutions than OR-Tools at every size tested, and
its final cost stays close to OR-Tools only up to 200 customers. At 400 and
1,000 customers, the final cost is about 20% higher than the reference at the
longest budget [arxiv:2608.14156 §5.1–§5.3]. Training takes 3 to 336 hours per
model on one GPU [arxiv:2608.14156 tab. 1]. Card:
[Deep RL for pickup-and-delivery routing](/papers/2608.14156).

### Region-masked modulation of a frozen diffusion model

DenseDiffusion adds a bonus to the attention scores between a word and a pixel
of the same region, and a penalty to other pairs, in the first 30% of denoising
steps [arxiv:2308.12964 §3.2, eq. 2–3]. The raise and lower amounts move each
score toward the query's maximum or minimum, so the scores stay inside their
original range [arxiv:2308.12964 §3.2, eq. 6]. The adjustment is scaled by
(1 − S), the fraction of the image covered by the segment, so small segments
are changed more [arxiv:2308.12964 §3.2, eq. 2]. No weights are trained. On 250
MS-COCO captions, SOA-I is 77.61 against 73.08 for Stable Diffusion, and layout
IoU is 34.99 against 23.76 for SD-Pww [arxiv:2308.12964 tab. 2, tab. 3]. The
area term trades one metric for another: removing it raises IoU to 38.00 and
lowers SOA-I to 73.62 [arxiv:2308.12964 tab. 4]. Card:
[Dense Text-to-Image Generation with Attention Modulation](/papers/2308.12964).

### Shared attention pooling for two related tasks

Sentiment and sarcasm are trained as one network with a shared GRU. Each task
has its own ReLU projection of the GRU states, and attention pooling scores each
position with tanh of a projection. The paper's normalization runs over a learned
L × L matrix, not over one score vector, and the attention parameters are shared
across the two tasks [arxiv:1901.08014 §2, eq. 1–2, §4]. The matrix ties the
model to one padded sentence length [arxiv:1901.08014 §2]. A neural tensor
network fuses the two task vectors before the sarcasm head [arxiv:1901.08014
§2]. The best variant reaches average F-score 86.66 on 994 sentences with
10-fold cross-validation, against 82.13 for a prior CNN that also uses gaze
features [arxiv:1901.08014 tab. 1]. These are author-reported numbers. Card:
[Sentiment and Sarcasm Classification with Multitask Learning](/papers/1901.08014).

### Attention over depth bins

AdaBins predicts a set of bin widths for each image with a small transformer
over the decoder output. Keys come from a 3×3 convolution on the decoded
features, and queries are transformer output embeddings. Each pixel gets a
softmax over the bin centres, and its depth is the weighted sum of those
centres [arxiv:2011.14141 §3.3, eq. 1–3]. Because the values are scalar bin
centres, the attention output is a depth, not a feature vector. Replacing this
hybrid output with standard regression lowers NYU delta1 from 0.903 to 0.881,
and raises RMS from 0.364 to 0.419 [arxiv:2011.14141 tab. 6]. Fixed log-scale
bins with the same output reach 0.896 [arxiv:2011.14141 tab. 6]. The NYU runs
train on a 50K-image subset, and the attention block has 5.8 M of about 78 M
parameters [arxiv:2011.14141 §4.1, §4.2]. Card:
[AdaBins: Depth Estimation using Adaptive Bins](/papers/2011.14141).

### Learned glimpse for reinforcement-learning driving

A glimpse network chooses which part of a sensor input the convolution sees.
The paper learns the glimpse motion with REINFORCE, separately from the
end-to-end RL error [arxiv:1704.02532 §3.4, §4.1.2]. The driving pipeline trains
end to end, but the paper does not say how the two learning signals combine
[arxiv:1704.02532 §4.3]. The experiments are qualitative lane keeping in a
simulator, and they give no attention metric [arxiv:1704.02532 §5]. Card:
[Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532).

### Robot state as the query over image features

Robot-centric Pooling replaces the final average pooling of a ResNet18 policy
with a cross-attention whose query is built from proprioception. Each image
location supplies a key and a value, so one query scores all locations
[arxiv:2411.04331 §3.1, eq. 1–7]. The policy path then uses a hard mask, the
argmax over the scores, and keeps the feature of that one location [arxiv:2411.04331
§3.1, eq. 8]. A contrastive loss on the soft context vector trains the query and
the projections. Two augmented views of the same pose, with the arm pasted onto
other backgrounds, should give similar context vectors, and different poses
should give different ones [arxiv:2411.04331 §3.2, eq. 9]. On simulated reaching
with unseen distractors, the authors report 76.4% success for RcP, 31.9% for
Spatial-Softmax and 8.3% for R3M [arxiv:2411.04331 tab. 1]. The two baselines
also differ in pre-training, so this comparison does not isolate the pooling
change [arxiv:2411.04331 §4.1]. Card: [Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling](/papers/2411.04331).

### Transformer over scan patches with a goal query

LiCS splits a 2D laser scan into patches, embeds each patch linearly, adds
trainable position embeddings and runs three Transformer encoder layers over
them. No class token is used [arxiv:2406.14947 §II-B]. A single goal token is
the query of a three-layer decoder. The decoder has no positional encoding and
no masked self-attention, but it keeps attention over the encoder output
[arxiv:2406.14947 §II-B]. The network outputs linear and angular velocity, and
it is trained by behaviour cloning with an MSE loss [arxiv:2406.14947 §II-A,
§II-B]. The ablation swaps the Transformer for RNN, LSTM or MLP policies, and
all three score lower on the 66 benchmark worlds [arxiv:2406.14947 tab. 4]. The
authors report 100% success on those worlds at 1.4 m/s, against 99.49% for the
FSMT expert [arxiv:2406.14947 tab. 2], and 91.67% against 62.50% on 16 hard
worlds [arxiv:2406.14947 tab. 3]. These are author-reported numbers from
simulation and one hardware setup, and no code is released [arxiv:2406.14947
§IV-D, Paper vs code]. Card: [LiCS: Navigation using Learned-imitation on Cluttered Space](/papers/2406.14947).

### Attention over attention for cloze reading

The Attention-over-Attention Reader first computes, for each query word, a
softmax over the document positions, as earlier readers do. It also computes a
softmax over the query words for each document word. The second set of weights
is averaged over the document positions to give one weight per query word, and
that average weights the per-word maps [arxiv:1607.04423 §3, eq. 6–10]. The
average has no second softmax, because an average of distributions still sums
to one [arxiv:1607.04423 §3, eq. 9]. The paper does not define the normaliser
in that average, so read it as the number of document positions. Author-reported
single-model test accuracy is 74.4 on CNN, 72.0 on CBTest NE and 69.4 on CBTest
CN. The AS Reader scores 69.5, 68.6 and 63.4 on the same sets [arxiv:1607.04423
tab. 2]. Re-ranking the 5-best answers with three language-model features lifts
CBTest NE test to 74.0 [arxiv:1607.04423 tab. 4]. Card:
[Attention-over-Attention Neural Networks for Reading Comprehension](/papers/1607.04423).

### Knowledge-base keys as extra output logits

The KV retrieval net keeps an attention decoder and adds one logit per
knowledge-base key to the vocabulary logits at each step. Each key is the sum
of the subject and relation embeddings. Its score comes from a small tanh
network over the key and the decoder state [arxiv:1705.05414 §2.3, eq. 7].
Each score is added at the position of that key's canonical token, and the
softmax runs over the extended vocabulary [arxiv:1705.05414 §2.3, eq. 8–9]. When
the decoder emits a canonical token, the system replaces it with the stored
value [arxiv:1705.05414 §2.3]. Author-reported test results on the paper's
in-car corpus are BLEU 13.2 and entity F1 48.0, against 6.6 and 43.8 for the
rule-based system and 11.0 and 37.0 for Copy Net [arxiv:1705.05414 tab. 3]. The
test split has 304 dialogues. Removing encoder attention lowers entity F1 to
40.9 [arxiv:1705.05414 tab. 3]. The paper names no licence for its corpus
[arxiv:1705.05414 §1]. Card:
[Key-Value Retrieval Networks for Task-Oriented Dialogue](/papers/1705.05414).

### Per-pixel gates that blend two branches

UIEC^2-Net's attention block takes the raw image and the outputs of an RGB block
and an HSV block, concatenated along channels. It predicts six sigmoid maps in
[0, 1]: three weight the RGB output and three weight the HSV output. The final
image is the sum of the two weighted outputs [arxiv:2103.07138 §III-A]. Because
the maps are sigmoid masks, the two weights at a pixel do not have to sum to one.
The paper's channel description and its combining sentence name the HSV output
in different orders, so state the channel order in a port [arxiv:2103.07138
§III-A]. Author-reported real-world PSNR is 24.57 dB for the full model, against
23.82 with the RGB block alone and 24.18 without the attention block [arxiv:2103.07138
tab. IV]. The paper's loss and layer counts differ from the released code, so
do not port that code as the paper's model; see [UIEC^2-Net](/papers/2103.07138).

### Per-pixel quality gate between two density maps

DecideNet counts crowds from two maps. One comes from detection boxes drawn as
Gaussian bumps, and the other from a small regression network. A QualityNet of
four convolutional layers reads both maps and the image, and ends in a sigmoid
per pixel. That map, K, weights the detection map, and one minus K weights the
regression map [arxiv:1712.06679 §3.4, eq. 4]. No query, key or value is
involved. The gate is trained with a quality term that pulls K toward a
detection-score map, weighted by a λ the paper does not state [arxiv:1712.06679
§4, eq. 8]. On Mall, the authors report MAE 3.93 for plain late fusion, 1.83
with QualityNet and 1.52 with the quality-aware loss. On ShanghaiTech PartB the
same three steps give 38.63, 24.93 and 21.53 [arxiv:1712.06679 tab. 4]. The
paper's main PartB table lists CP-CNN at a lower MAE, 20.1 against 21.53
[arxiv:1712.06679 tab. 2]. Each training iteration runs the detector on every
patch, and the paper does not report what that costs [arxiv:1712.06679 §5.1].
Card: [DecideNet](/papers/1712.06679).

### Shallow features gated by deep features

SANet's shallow attention module upsamples a deeper feature map to the size of a
shallower one. It applies a ReLU to form an attention map, and multiplies that
map element-wise into the shallower features [arxiv:2108.00882 §3.2, eq. 1–2].
The paper says the module filters background noise from shallow features and
weights each level by its contribution [arxiv:2108.00882 §3.2]. It does not say
how the channel counts of the two maps are matched, and the element-wise product
needs equal shapes [arxiv:2108.00882 §3.2]. In the ablation on ColonDB, adding
this module to the backbone raises mDice from 0.676 to 0.728. Colour exchange
and the logit correction add further gains [arxiv:2108.00882 tab. 2]. The
backbone-only row is not described in the paper. The released code does not
contain this module, so an implementation from the code is a different network;
see [SANet](/papers/2108.00882).

### Parallel spatial and temporal streams with learned fusion

Each DSTformer block has two streams over the same features. One applies spatial
attention across joints, then temporal attention across frames. The other applies
them in the reverse order [arxiv:2210.06551 §3.2, eq. 4]. A learned linear map
over the concatenated outputs gives a weight per frame and per joint, normalised
with a softmax across the two streams [arxiv:2210.06551 §3.2, eq. 5]. Blocks do
not share weights [arxiv:2210.06551 §3.2]. On Human3.6M, averaged over five runs,
spatial-then-temporal attention alone gives 40.58 ± 0.31 mm MPJPE, equal averaging
of the two streams 39.87 ± 0.32, and the learned fusion 39.25 ± 0.27
[arxiv:2210.06551 tab. 7]. Card:
[MotionBERT: A Unified Perspective on Learning Human Motion Representations](/papers/2210.06551).

### Attention guided by another branch's hidden state

In a driving model with a trajectory branch and a control branch, each future
control step attends over the image grid. An MLP scores each grid cell from the
trajectory branch's hidden state for that step and the control branch's hidden
state [arxiv:2206.08129 §3.2.2, eq. 2]. The attended feature is merged with the
control state [arxiv:2206.08129 §3.2.2, eq. 3]. The paper says the attention at
the first step uses only the measurement feature [arxiv:2206.08129 §3.2.2]. The
authors show attention maps for two cases, and they say the attention focuses on
different regions at different future steps [arxiv:2206.08129 fig. 4]. In the
ablation, adding this attention to the temporal module moves the driving score
from 42.87 ± 4.77 to 46.08 ± 3.47 [arxiv:2206.08129 tab. 3]. The paper does not
define the plus-minus values, and the two ranges overlap. Results come from one
simulator and one benchmark [arxiv:2206.08129 §4.1]. Card:
[Trajectory-guided Control Prediction for End-to-end Autonomous Driving](/papers/2206.08129).

### Localization loss on cross-attention maps

FastComposer supervises the cross-attention of each subject's token during
training only. For each token, the loss is the mean attention outside the
subject's mask minus the mean attention inside it, averaged over subjects. The
paper applies it to five middle U-Net blocks, with weight λ = 0.001
[arxiv:2305.10431 §4.2, eq. 3–4]. Masks are needed only in training [arxiv:2305.10431
§4.2]. A separate choice delays subject conditioning: the subject-augmented text
embedding is used only for the last part of sampling, and the paper suggests α
between 0.6 and 0.8 [arxiv:2305.10431 §4.3, fig. 5]. In the paper's ablation, which
does not name its evaluation set, the localization loss raises identity
preservation from 37.66% to 43.11% and lowers prompt consistency from 25.03%
to 24.25% [arxiv:2305.10431 tab. 3]. Card:
[FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](/papers/2305.10431).

### Layout tokens as keys in a diffusion UNet

LayoutDiffusion conditions an image-space diffusion model on boxes and classes,
with no text input. Object-aware cross-attention lets each image patch attend
to the object tokens, with the patch position in the query and the object
position in the keys [arxiv:2303.17189 §3.3]. A layout fusion module combines
the object tokens before they reach that step [arxiv:2303.17189 §3]. In the
ablation, the object-aware attention alone reaches FID 16.76, against 29.94 with
neither component [arxiv:2303.17189 tab. 2]. The full model reaches 16.57, and
the same layout module with plain cross-attention reaches 16.46
[arxiv:2303.17189 tab. 2]. On COCO-Stuff at 256 pixels, the authors report
FID 15.61 [arxiv:2303.17189 tab. 1]. The conditional stage has 569M parameters
and took 216.55 V100-days [arxiv:2303.17189 tab. 3]. Card:
[LayoutDiffusion](/papers/2303.17189).

## When to use which

| Situation | Use | Why |
|---|---|---|
| One sequence is mapped to another (translation, captioning) | Distinctive attention, soft, additive or dot-product scoring | This is the original encoder-decoder setup, and the survey classes it as distinctive, soft, single-level [arxiv:1904.02874 §4.1, §4.3, tab. 4] |
| Inputs are a single sequence and the output is not one | Self-attention with scaled dot product | The survey pairs self-attention with classification and recommendation tasks [arxiv:1904.02874 §4.1]. Cost is $O(n^2)$ in length, so long inputs need the cheaper variants in [arxiv:1904.02874 §8.8] |
| Two inputs (image and tags, image and text) should guide each other | Co-attention, with both attended vectors fused | The ablation shows gains over single-source attention on two medical datasets [arxiv:1711.08195 tab. 1]. Results are author-reported and cover report generation only |
| Most inputs are irrelevant and sparse weights help readability | Sparsemax or entmax | Only suggested by the survey [arxiv:1904.02874 §3]. No measured result in the cited cards |
| Only a few positions are needed and gradients through weights are not required | Hard attention | Cheaper than soft attention, but the survey gives no training method [arxiv:1904.02874 §4.3]. Plan extra implementation effort |
| A high-pass residual marks where manipulation traces are | Residual-guided attention on RGB features | Gains reported in one ablation on one task [arxiv:2103.12376 tab. 3] |
| A coded event sequence (diagnoses per visit) with little labelled data and a pre-training corpus | Bidirectional self-attention encoder with masked pre-training | Author-reported gains over DeepR and RETAIN on one UK dataset [arxiv:1907.09538 tab. 1] |
| A small image-to-image model where attention must add little cost | CBAM on skip connections only | About a quarter of U-Net's parameters, with MSE tied on one radar dataset [arxiv:2007.04417 tab. 1, tab. 4]. Results are author-reported on one dataset without repeated runs [arxiv:2007.04417 §5] |
| A CNN with channel attention and a tight parameter budget | Batch-norm-scale channel gate (NAM) | Author-reported 0.35 points better top-1 error than CBAM on one CIFAR-100 setting [arxiv:2111.12419 tab. 1]. The parameter saving is an arithmetic count [arxiv:2111.12419 app. A.1] |
| Order does not matter and one summary vector is needed, but a plain mean is too weak | Feed-forward attention pooling | The paper's tests are order-free toy tasks, and it does not test word order [arxiv:1512.08756 §2.4]. Weighted pooling beat the mean on variable lengths, 99.9% against 77.4% on addition [arxiv:1512.08756 §2.3] |
| Text is not horizontal and only word labels are available | Attention decoder over directional features | Author-reported 76.8 on CUTE80 against 73.7 for the spatial-transformer baseline [arxiv:1711.04226 tab. 1]. The scoring form is not stated, so choose one |
| Scene text on complicated or low-quality photos, where the attention decoder reads the wrong region, and labels for character positions exist for part of the data | Attention decoder with a focusing loss on a patch around each step's attention centre | Author-reported lexicon-free IIIT5k 87.4 against 83.7 without the focusing network [arxiv:1709.02054 tab. 2]. Labels for 1% of the words give most of that gain [arxiv:1709.02054 tab. 8, tab. 9]. The λ choice was made on the test sets [arxiv:1709.02054 §4.4] |
| Dense correspondence between two frames, with large motion | Global matching over all pixel pairs | One refinement beat RAFT with 31 refinements on Things validation [arxiv:2111.13680 tab. 3], but the KITTI test result is worse [arxiv:2111.13680 tab. 6]. The correlation matrix grows with the square of the pixel count |
| Several calibrated cameras, top-down map, no depth model | Cross-view attention with a camera-aware embedding | Removing the embedding drops vehicle IoU from 36.0 to 31.0 [arxiv:2205.02833 tab. 3, tab. 4]. Author-reported, 35 FPS on an RTX 2080 Ti [arxiv:2205.02833 tab. 1] |
| Detect objects from raw points without a grouping rule | Candidates attend to all points, with a learned candidate sampler | Grouping baselines reach 44.4 and 44.1 mAP@0.5 against 48.5 for this design [arxiv:2104.00678 tab. 8]. A DETR-style baseline reaches 21.4, and 41.0 with the sampler [arxiv:2104.00678 tab. 10] |
| Explain a Transformer with two inputs or an encoder-decoder | Gradient-weighted attention with cross-attention rules | Author-reported AP 13.1 against 5.6 for raw attention on DETR segmentation [arxiv:2103.15679 tab. 1] |
| Pick an attention family for a vision task | Survey taxonomy as a map | It groups methods by how they compute scores, with formulas for each family [arxiv:2204.07756 §2, fig. 2]. It does not benchmark the families against each other |
| You want the weights to explain the model | Do not rely on them alone | See Contradictions below |
| You need to know whether an attention module in a binary text classifier does real work | Four replacement tests: uniform weights, seed variance, a frozen-weight MLP guide, and a consistent adversary | Uniform weights match the base model on AG News and 20 Newsgroups, so the paper drops those tasks [arxiv:1908.04626 §3.2, tab. 2]. The paper's tests cover one single-layer BiLSTM on binary tasks [arxiv:1908.04626 §3.1] |
| Long inputs where the softmax cost dominates, and decoding must not revisit the whole prefix | Random-feature linear attention (RFA) | Gains reported above roughly 1,000 tokens, with decoding 1.8 to 1.9 times faster on MT [arxiv:2103.02143 §3.4, tab. 2]. Trained softmax models cannot be swapped in without finetuning [arxiv:2103.02143 app. C.3] |
| A robot must find its own arm in a camera image among distractors, and joint state is available at every step | Proprioception-queried cross-attention pooling with a hard mask, trained with a contrastive loss | Author-reported 76.4% against 31.9% for Spatial-Softmax under distractors in simulation [arxiv:2411.04331 tab. 1], and 92% against 14% with a self-distractor on a real arm [arxiv:2411.04331 §4.3]. The baselines also differ in pre-training [arxiv:2411.04331 §4.1]. The authors report no ablation of the mask or the loss weight [arxiv:2411.04331 §4] |
| Local navigation from a 2D laser scan toward a goal, with expert runs to imitate and a CPU-only robot computer | Small Transformer over scan patches with a goal query, trained by behaviour cloning | Author-reported 100% success on 66 BARN worlds at 1.4 m/s against 99.49% for the FSMT expert [arxiv:2406.14947 tab. 2], and 91.67% against 62.50% on 16 hard worlds [arxiv:2406.14947 tab. 3]. Two of three hardware tracks were completed [arxiv:2406.14947 §IV-D] |
| Many input tokens from sensors, few demonstrations per task, and a need to bound attention over the input | Latent bottleneck with learned latents | Author-reported gains over a 3D U-Net in simulation, with the Perceiver called crucial in the ablation [arxiv:2209.05451 §4.2, tab. 1]. Training cost is high: 16 days on 8 V100 GPUs [arxiv:2209.05451 §3.4] |
| Find which heads implement one behaviour in a small model | Causal interventions on heads (path patching, mean ablation) with faithfulness, completeness and minimality checks | The 26-head circuit keeps 87% of the logit difference, and the paper lists its own gaps [arxiv:2211.00593 §4, §3.4]. Each path-patching experiment needs three forward passes per sender set [arxiv:2211.00593 §3] |
| Match query features to a precomputed set of 3D points, and a per-frame time budget matters | Graph attention over each point's feature track, then a dual-softmax match | Author-reported 58 ms per frame against 618 ms for HLoc with SuperPoint and SuperGlue, with accuracy on par with or slightly above HLoc on the authors' dataset [arxiv:2205.12257 tab. 1, §4.3]. The paper's linear attention differs from the code's softmax [arxiv:2205.12257 §4.2] |
| Agents on a lane graph, and the prediction must follow legal routes | Agent-node attention in the encoder, then attention over each sampled route | Adding agent-node attention lowers MinADE5 from 1.33 to 1.30 in one nuScenes ablation [arxiv:2106.15004 tab. 2]. The agent distance threshold is not given [arxiv:2106.15004 §4.1] |
| Protein sequences where a pool of aligned homologs is available at inference | Decoder-only attention with grouped convolutions and grouped distance bias, mixed with a retrieved profile | Author-reported Spearman 0.451 with retrieval against 0.448 for EVE on ProteinGym substitutions [arxiv:2205.13760 §6.2]. The gain over EVE is small, and the paper's own table has EVE ahead on high-depth alignments [arxiv:2205.13760 §6.2] |
| Two related text tasks, such as sentiment and sarcasm, on short sentences | Shared attention pooling with a fused task vector | Author-reported average F-score 86.66 against 82.13 for a gaze-based baseline, on 994 sentences [arxiv:1901.08014 tab. 1]. The L × L normalizer fixes the sentence length [arxiv:1901.08014 §2] |
| A text query must be grounded to one box in an image, with no proposals or anchors | One joint transformer encoder over image tokens, word tokens and a learned [REG] token, with a box head | Author-reported ReferItGame test 70.73 with ResNet-101 [arxiv:2104.08541 tab. 1]. On RefCOCO testB the authors' released weights score 76.94 against 78.35 reported [arxiv:2104.08541 tab. 2]. The paper gives runtimes but no hardware [arxiv:2104.08541 tab. 4] |
| A sketch-to-photo retrieval encoder where the domain and semantic losses are also used | Spatial attention with a residual add, on top of a triplet loss | The ablation gives 0.3020 to 0.3207 mAP on Sketchy-Extended with attention alone [arxiv:1904.03451 tab. 3]. The full model reaches 0.3691 [arxiv:1904.03451 tab. 2] |
| Small structure in a U-Net segmentation, where a two-model cascade is the alternative | Additive sigmoid gates on the skip connections | Author-reported pancreas Dice 0.840 against 0.814 for U-Net, at about 9% more parameters and 7% more forward time on one split [arxiv:1804.03999 tab. 1]. The gain is smaller on the public set [arxiv:1804.03999 tab. 3] |
| Depth from one image, where the depth range varies by scene and smooth output matters | Per-image bins with attention over bin centres | Author-reported NYU delta1 0.903 against 0.881 for standard regression with the same base [arxiv:2011.14141 tab. 6]. The released training values differ from the paper's, per the card's Paper vs code section |
| Speaker-aware emotion labels, one utterance at a time, with no future turns | Causal attention over earlier global states | Author-reported 59.33 weighted accuracy on IEMOCAP text, against 56.56 for CMN [arxiv:1811.00405 tab. 2]. The emotion attention and the bidirectional model need future turns [arxiv:1811.00405 §3.4.2] |
| Whole dialogue available offline, and the best text-only score matters | Emotion attention over all states, bidirectional model | Author-reported 63.40 weighted accuracy for BiDialogueRNN+Att on IEMOCAP text [arxiv:1811.00405 tab. 2]. The paper's ablation does not isolate attention [arxiv:1811.00405 §5.6] |
| Sensors with unknown dependencies, and a score that must point to the related sensors | Learned sensor graph with neighbour attention | Author-reported F1 0.81 on SWaT and 0.57 on WADI. Uniform neighbour weights drop these to 0.68 and 0.48 [arxiv:2106.06947 tab. 3]. Check the threshold: the card's Paper vs code section reports that the default picks it on test labels |
| A graph where hop distance or edge types should shape attention, and every node pair can be scored | Attention bias from graph distances, or a positional encoding added to each node | The survey reports no numbers, and its design guide cites no source for the size thresholds [arxiv:2407.09777 §6]. Measure the effect on your graphs [arxiv:2407.09777 Results] |
| An RL policy that must choose where to look in sensor input | Learned glimpse trained by REINFORCE | The paper reports no attention metric, so treat it as a design reference only [arxiv:1704.02532 §5]. Measure the effect yourself |
| Routes over many nodes, where the decoder must ignore nodes already visited, and a time budget matters | Masked top-layer re-embedding every few steps, with several decoders | Author-reported 2.23% CVRP100 gap at 53 min, against 3.72% at 2 h for AM sampling with 1,280 samples [arxiv:2012.10638 tab. 1]. The paper gives the interval per problem size [arxiv:2012.10638 §4] |
| Multi-view 3D grounding, where left and right depend on the viewpoint | Shared decoder over rotated views, averaged across views | Author-reported 55.1 on Nr3D against 43.9 for the best non-assisted baseline [arxiv:2204.02174 tab. 1]. Max pooling is worse than averaging [arxiv:2204.02174 tab. 6] |
| Dense captions with a layout, on a frozen text-to-image model, and region masks are available | Region-masked attention modulation in the first 30% of steps | Author-reported SOA-I 77.61 and layout IoU 34.99 against 73.08 and 23.76 for the baselines [arxiv:2308.12964 tab. 2, tab. 3]. The method needs one mask per phrase [arxiv:2308.12964 §3] |
| A detection head must cover several object scales, shapes and target types with one design, and about 8.5% more GFLOPs is acceptable | Stacked scale-, spatial- and task-aware attention over the feature pyramid | Author-reported 42.6 AP against 39.0 without the three attentions, on ResNet-50 at 1x [arxiv:2106.08322 tab. 1]. Six blocks add about 21.5 GFLOPs to a 254.39 baseline [arxiv:2106.08322 tab. 2]. The paper's text calls this negligible [arxiv:2106.08322 §4.2] |
| Several calibrated cameras feed a 3D detector or a map, and past frames are available for motion cues | Deformable cross-attention from BEV queries to cameras, plus temporal attention over the previous map | Author-reported 51.7 NDS on nuScenes val with ResNet-101 [arxiv:2203.17270 tab. 2]. Deformable local attention beats global attention on the same static setup, 0.448 against 0.404 NDS, and uses about 20 GB against 36 GB of training memory [arxiv:2203.17270 tab. 5]. Inference keeps the previous map in memory [arxiv:2203.17270 §3.6] |
| Two partial scans with little shared surface, where interest points must come from the shared part | Bidirectional cross-attention between the two clouds, with overlap and matchability scores | Author-reported 59.8 RR on 3DLoMatch against 40.1 for FCGF, at 5000 samples [arxiv:2011.13005 tab. 2]. Memory is quadratic in superpoint count [arxiv:2011.13005 §4.3] |
| A decoder-only language model where the key-value cache limits inference, and long contexts are needed | Alternating local (4096-token) and global layers, with two query heads per key-value head | Gemma 2 9B averages 50.8 with grouped-query attention against 50.3 with full multi-head attention, on four benchmarks [arxiv:2408.00118 tab. 8]. The window can shrink at inference with a small perplexity change, measured on 9B only [arxiv:2408.00118 tab. 10]. The caps are not ablated [arxiv:2408.00118 §2] |
| A mobile CNN where channel attention is already used and the position of a feature matters | Coordinate attention in place of squeeze-and-excitation | Author-reported 74.3% against 73.5% for SE on ImageNet with MobileNetV2, though the two settings differ in reduction ratio [arxiv:2103.02907 tab. 1]. Pixel 4 latency rises from 14–16 ms to 17–19 ms [arxiv:2103.02907 tab. 1] |
| A CNN where each kernel position should weight its window differently, and the extra activation memory fits the device | Receptive-field attention convolution (RFAConv) | Author-reported 71.23% against 69.59% for ResNet18 on ImageNet-1k, for about 0.16M extra parameters [arxiv:2304.03198 tab. 3]. The expanded tensor is k² times the input, and the authors say limited-storage devices may not afford it [arxiv:2304.03198 §4.6] |
| Multi-step forecasting from a long look-back window, with many horizons | Start with a one-layer linear model, and test attention forecasters against it at the same look-back | Author-reported: the best linear model has lower MSE than the best Transformer baseline in all 36 dataset-horizon cells of the main table [arxiv:2205.13504 tab. 2]. On Electricity at look-back 96 and horizon 720, one linear layer has 139.7K parameters, against 13.61M for a Transformer with the same one-step decoder [arxiv:2205.13504 tab. 8] |
| Cloze-style questions over a passage, where some query words should count more than others | Attention over attention: a second softmax over query words, averaged over the document | Author-reported 74.4 on CNN test and 72.0 on CBTest NE test for one model, against 69.5 and 68.6 for the AS Reader [arxiv:1607.04423 tab. 2]. Re-ranking with n-gram features adds two points on CBTest NE [arxiv:1607.04423 tab. 4] |
| Task-oriented dialogue over a small knowledge base, with no belief tracker and dialogues from a few domains | KB-key logits added to a copy-style decoder | Author-reported entity F1 48.0 against 43.8 for the rule-based system, on 304 test dialogues [arxiv:1705.05414 tab. 3]. The paper releases no code, and its corpus has no stated licence [arxiv:1705.05414 §1] |
| Two colour or intensity branches whose strengths differ across pixels | Sigmoid gates that blend the two branch outputs per pixel | Author-reported real-world PSNR 24.57 dB against 23.82 for the RGB block alone [arxiv:2103.07138 tab. IV]. The released code differs from the paper, see [UIEC^2-Net](/papers/2103.07138) |
| Small structures in an encoder-decoder, where early features carry the detail and deep features carry the context | Deep features gate shallow features with a ReLU attention map | Author-reported ColonDB mDice 0.728 with the module, against 0.676 for the backbone [arxiv:2108.00882 tab. 2]. The released code has no such module, see [SANet](/papers/2108.00882) |
| Human motion from 2D skeletons with time and joint structure, and few 3D labels | Parallel spatial-then-temporal and temporal-then-spatial streams with learned fusion | Author-reported 39.25 mm against 40.58 mm for spatial-then-temporal alone, over five runs [arxiv:2210.06551 tab. 7] |
| A driving policy from one front camera, with a waypoint plan and direct controls | Control-branch attention guided by the trajectory branch's hidden state | Author-reported 46.08 driving score with attention against 42.87 without, with overlapping spreads [arxiv:2206.08129 tab. 3]. One simulator and one leaderboard [arxiv:2206.08129 §4.1] |
| Several people in one generated image, from one reference photo each, with no per-person training | Localization loss on subject-token attention during training, delayed subject conditioning at sampling | Author-reported identity preservation 43.11% with the loss against 37.66% without, at a cost of about 0.8 points on prompt consistency [arxiv:2305.10431 tab. 3]. The ablation's evaluation set is not named [arxiv:2305.10431 tab. 2] |
| A pickup-and-delivery route with capacity and time windows, where an answer within seconds matters more than the last few percent | Attention route builder with a pickup-order mask and a soft missed-client penalty | Author-reported better early solutions than OR-Tools at every size tested, with about 20% higher final cost at 400 and 1,000 customers [arxiv:2608.14156 §5.3]. Training takes 3 to 336 hours per model [arxiv:2608.14156 tab. 1] |
| A per-pixel choice must blend two predictors whose strengths differ by region, such as detection and density in a crowd | Sigmoid gate that blends the two maps, trained with a term pulling it toward a detection score | Author-reported Mall MAE 3.93 for late fusion, 1.83 with the gate and 1.52 with the quality loss [arxiv:1712.06679 tab. 4]. On ShanghaiTech PartB, CP-CNN reports a lower MAE [arxiv:1712.06679 tab. 2]. The loss weight λ is not stated [arxiv:1712.06679 §4] |
| Point-cloud correspondences contain many outliers, and pairwise geometry should shape the weights | Nonlocal attention over correspondences, weighted by a spatial length-consistency term | Author-reported registration recall 93.28 on 3DMatch with FCGF features, at 0.09 s per pair [arxiv:2103.05465 tab. 1], against 83.30 for a plain nonlocal classifier [arxiv:2103.05465 tab. 4]. The released code's constants differ from the paper's [arxiv:2103.05465 §4] |
| A layout or box condition must steer a diffusion model that has no text input | Object-aware cross-attention over object tokens, with a layout fusion module | Author-reported COCO-Stuff 256 FID 15.61 [arxiv:2303.17189 tab. 1]. The plain cross-attention variant scores slightly better on FID, 16.46 against 16.57 [arxiv:2303.17189 tab. 2]. The conditional stage has 569M parameters [arxiv:2303.17189 tab. 3] |
| Multi-sensor time series where the past steps of a window differ in relevance | Attention over past recurrent states, softmax over time | Author-reported mF1 0.9639 with attention against 0.9213 without, on PAMAP2 [arxiv:2107.12626 tab. 6]. The index range of eq. 17 is inconsistent with the text [arxiv:2107.12626 §3.4.1] |
| You need the history and formula families of attention before choosing a design | Survey as a map | It covers attention from the late 1980s to the Transformer and its cost fixes, with core formulas. It has no experiments [arxiv:2204.13154 §1–§5] |
| Speech input, no source transcript, and one model for the whole path | Attention encoder-decoder with convolutional location term on the previous weights | Author-reported 48.8 BLEU with a 5-model ensemble, against 52.7 for ASR plus SMT, on synthetic speech [arxiv:1612.01744 tab. 3]. The paper calls the data synthetic and the result a proof of concept [arxiv:1612.01744 §5] |
| Seasonal, trending series to forecast over a long horizon, and an interpretable decomposition is wanted | Exponential smoothing attention with frequency attention, in a stacked encoder-decoder | Author-reported lowest MSE on 22 of 24 multivariate rows against Autoformer and Informer [arxiv:2202.01381 tab. 1]. The FFT form makes the attention O(L log L) [arxiv:2202.01381 app. A] |
| A spiking network must gain accuracy and emit fewer spikes, on event-camera or static images | Temporal, channel and spatial attention on the membrane potential | Author-reported 5.9 and 4.7 point gains on DVS128 with 84.9% and 81.6% fewer spikes [arxiv:2209.13929 tab. 1]. Energy numbers are operation counts, not measurements [arxiv:2209.13929 §4] |
| A dense-prediction backbone with global attention, where the token count is too large for full attention | Pooled keys and values at several scales, with a depthwise convolution on each pooled map | Author-reported 82.4 ImageNet-1K top-1 for P2T-Small [arxiv:2106.12011 tab. 2]. The paper's ablation favours average pooling over max pooling [arxiv:2106.12011 tab. 7] |
| Several modalities (image, notes, labs, demographics), with all of them present at inference | Bidirectional multimodal attention blocks, then self-attention | Author-reported mean AUROC 0.924 against 0.858 for Perceiver over eight pulmonary diseases [arxiv:2306.00864 tab. 1]. The paper names missing modalities as an open problem [arxiv:2306.00864 p. 7] |
| Predict a 3D scene from many posed images, with a time budget per scene | Viewpoint tokens with per-view cross-attention and cross-view self-attention | Author-reported 28.65 PSNR at two views in 0.025 s against 27.47 for DepthSplat [arxiv:2507.23277 tab. 1]. The released code is limited to two views at full resolution [Card: iLRM](/papers/2507.23277) |
| A team needs one reference for post-norm, masks, positions and decoder caching before writing code | Tutorial survey as a checklist | It gives the formulas and design choices but runs no experiments [arxiv:2311.17633 §2] |

## Implementation sketch

Generalized attention with a pluggable score function. Softmax is shown; the
survey's sparse alternatives replace only that line [arxiv:1904.02874 §3].

```pseudocode
# Generalized attention: score, normalize, weighted sum  [arxiv:1904.02874 §3] [eq. 1]
# Input:  query (d_q,); keys (N, d_k); values (N, d_v)
# Output: attended vector (d_v,)

def attention(query, keys, values, score_fn):
    # Score each key against the query: dot, scaled dot or additive  [tab. 2]
    scores = [score_fn(query, key) for key in keys]    # (N,)
    # Normalize all N scores together, not one at a time  [§3]
    weights = softmax(scores)                          # (N,), sums to 1
    # Weighted sum of values  [eq. 1]
    return sum(w * v for w, v in zip(weights, values))
```

Notes for porting:

- Apply the normalization to the full score vector. Softmax, sparsemax and
  entmax need all scores at once, so writing $p$ per term, as [eq. 1] does,
  is not the implementation [arxiv:1904.02874 §3].
- Scaled dot product divides by $\sqrt{d_k}$, where $d_k$ is the dimension of
  $q$ and $k$, not the number of positions [arxiv:1904.02874 tab. 2].
- The survey's biased general row omits a transpose. Check the original source
  before implementing it [arxiv:1904.02874 §3].
- Soft attention needs one score per (query, position) pair, so the cost is
  quadratic in the number of positions [arxiv:1904.02874 §8.8].
- Co-attention runs this function twice, once over regions and once over
  tags, and fuses the two outputs [arxiv:1711.08195 §3.3, eq. 4].
- Pick the logit form explicitly. The cross-view paper's score is a cosine
  similarity [arxiv:2205.02833 §3.1, eq. 3]. The card's Paper vs code section
  (item 10) reports that the released code uses a scaled dot product instead.
- Global matching is attention between two frames. The softmax runs over frame
  two's positions for each pixel of frame one, and the values are coordinates
  [arxiv:2111.13680 §3.1, eq. 2–3]. The score matrix grows with the square of
  the pixel count, so plan blocked computation [arxiv:2111.13680 app. B].
- Linear attention computes the key sums once and applies each query to them.
  The causal form keeps a fixed-size running state, updated per token
  [arxiv:2103.02143 alg. 1]. The paper gives no epsilon or clamp for the
  denominator, so guard near-zero values in your own code [arxiv:2103.02143
  alg. 1].
- Feature-track aggregation normalizes over the 2D descriptors of one 3D point
  only, then adds the result to that point's own descriptor [arxiv:2205.12257
  §3.2]. Do not normalize over all points or all queries in this step.
- The OnePose card's Paper vs code section reports that the released code uses
  additive graph attention with LeakyReLU, and includes the 3D node in the
  softmax [arxiv:2205.12257 §3.2]. Choose one form and say which in your port.
- Gated modules are not softmax. NAM and CBAM-style blocks take a sigmoid of
  each element and multiply it into the features. Do not normalize their gates
  across positions or channels [arxiv:2111.12419 §3, eq. 2].
- Attention-based pooling over a fixed-length input, as in the sentiment model,
  needs a normalizer that depends on the length. The paper's learned L × L
  matrix is one such choice, and it fixes the padded length [arxiv:1901.08014 §2].

- For one joint sequence, concatenate the modality tokens and the learned
  token in the order the paper gives, mask the padding, and read the output at
  the learned token [arxiv:2104.08541 §3.2, eq. 4]. The paper's eq. 1 writes
  $q k$ without a transpose as typeset, so implement the standard scaled dot
  product [arxiv:2104.08541 eq. 1].

- Attention gates on a skip connection are sigmoid masks on the skip features,
  computed on a grid coarser than the skip. Resample the coefficients to the
  skip grid before multiplying [arxiv:1804.03999 §2, fig. 2].
- Per-pixel attention over bins gives a weighted average of scalar bin centres,
  so the output is one depth per pixel [arxiv:2011.14141 §3.3, eq. 3]. The card's
  Paper vs code section reports that the released code's positional table holds
  500 rows, which caps the token count [arxiv:2011.14141 §3.3].
- Graph attention over a learned sensor graph has one open choice. The paper's
  candidate set excludes the sensor itself, but the card's Paper vs code section
  reports that the released code's top-k includes it [arxiv:2106.06947 §3.4].
- Emotion attention and the bidirectional model need the whole dialogue. A
  causal port must drop them [arxiv:1811.00405 §3.4.2].
- A learned glimpse trained with REINFORCE needs its own learning signal next to
  the end-to-end loss. The paper does not say how the two are combined
  [arxiv:1704.02532 §3.4, §4.3].
- Dynamic Head's scale gate maps one pooled value per level to a weight with a
  hard-sigmoid. The paper's form and the released code's form differ, so state
  which one you port [arxiv:2106.08322 §3.2, eq. 3]. The card is
  [Dynamic Head](/papers/2106.08322).
- In BEVFormer's spatial attention, divide by the number of cameras that were
  hit, not by the total number of cameras [arxiv:2203.17270 §3.3, eq. 2]. A
  camera with no projected point in view is left out of the sum.
- Gemma 2 soft-caps logits as cap times tanh(logit / cap), with cap 50 for
  attention and 30 for the output [arxiv:2408.00118 §2]. The paper does not
  give the order of the cap and the mask. The released code caps before masking
  ([Gemma 2](/papers/2408.00118), Paper vs code section).
- Gemma 2's 27B model scales queries by the model width over the head count,
  not by the head size. The paper does not state this [arxiv:2408.00118 tab. 1].

- Cross-attention between two clouds runs in both directions with one set of
  weights. Each side's query reads the other side's keys and values
  [arxiv:2011.13005 §3.3, eq. 4–5].
- The score scale is $1/\sqrt{b}$, with $b$ the feature width [arxiv:2011.13005
  eq. 5]. The card's Paper vs code section reports that the released code
  divides by the per-head width instead, with four heads. Pick one and say
  which in your port.
- Coordinate attention has two sigmoid gates, one per axis, and both multiply
  the input. The paper leaves the non-linearity in the shared transform open,
  so state which one you use [arxiv:2103.02907 §3.2.2].
- RFAConv's expanded tensor holds k² values for every input value, so budget
  memory before replacing many layers [arxiv:2304.03198 §4.6]. The softmax
  runs over the k² slots of each channel at each position [arxiv:2304.03198
  §3.3, eq. 3].

- The focusing loss in the scene-text model has no normalization stated. The
  paper's loss is a plain sum over steps and patch positions, with no division
  by a count [arxiv:1709.02054 §3.2, eq. 13]. With λ = 0.01 the relative scale
  of the two losses depends on this choice, which the paper does not state.

- The RcP text calls the alignment scores cosine similarities, but eq. 6 is a
  softmax of the raw dot product $q k^{\top}$ with no normalization or
  temperature [arxiv:2411.04331 §3.1, eq. 6]. Follow the equation unless you
  have the code, since the two differ in scale.
- RcP's policy path takes its mask from an argmax, and the equations give no
  gradient path from the policy loss to the query, keys or values. The card
  infers this. The paper does not state it [arxiv:2411.04331 §3.1, eq. 8].
  The contrastive term uses the soft context instead [arxiv:2411.04331 eq. 7].
- LiCS's decoder takes one goal token and no positional encoding, while its
  encoder keeps trainable position embeddings for the patches
  [arxiv:2406.14947 §II-B]. Match that split when porting, because the
  decoder's query is a single goal and not a sequence.

- Attention over attention needs two softmaxes on one score matrix: one over
  document positions for each query word, and one over query words for each
  document word. The query-word weights are the mean of the second over the
  document positions, with no further softmax, and the passage weights are the
  product of the first with those means [arxiv:1607.04423 §3, eq. 6–10].
- Knowledge-base logits are not normalised on their own. Add each score to the
  logit of its canonical token, then take one softmax over the extended
  vocabulary. The output layer's shape is not stated, so size it to hold the
  canonical tokens [arxiv:1705.05414 §2.3, eq. 8–9].
- A two-stream block computes two orders of the same two attentions and mixes
  them with a learned map over both outputs. The mix is a softmax across the two
  streams for each frame and joint [arxiv:2210.06551 §3.2, eq. 4–5].
- Trajectory-guided attention scores each grid cell with an MLP over two hidden
  states, softmaxes over the grid, then merges the attended feature with the
  control state. The first step uses the measurement feature only
  [arxiv:2206.08129 §3.2.2, eq. 2–3].
- A localization loss on attention maps is a training-time term only. Compute
  it per subject token on the masked region, then weight it with λ. The
  balanced form is the mean outside the mask minus the mean inside it
  [arxiv:2305.10431 §4.2, eq. 3–4].

## Contradictions

- **Are attention weights explanations?** The survey reports that weights
  correlate weakly with feature importance, and that permuting them or changing
  them adversarially can leave predictions unchanged. It concludes that they
  are at best noisy predictors of importance [arxiv:1904.02874 §7]. A
  follow-up study argues that this depends on what "explanation" means, and
  proposes four replacement tests [arxiv:1908.04626 §2, §5]. A second survey
says attention may be a plausible explanation without being a faithful one
[arxiv:1902.02181 §VI-A]. A review of medical vision transformers doubts attention maps as explanations too [arxiv:2301.03505 §10]. The medical
  report paper shows per-sentence attention maps and traces one mis-description
  to semantic attention [arxiv:1711.08195 §4.5.2, fig. 4]. The face-forgery paper
  says its attention maps highlight abnormal facial boundaries [arxiv:2103.12376
  §4.2, fig. 8]. Neither applied paper tests faithfulness, so this is not
  resolved here.
- **Adversarial attention as a counterexample.** On four binary LSTM tasks,
  Wiegreffe and Pinter find adversarial attention distributions with average
  JSD above 0.4 [arxiv:1908.04626 tab. 4]. When those weights are imposed on a
  non-contextual MLP guide, F1 on Diabetes falls to 0.503, against 0.753 with
  the base LSTM attention [arxiv:1908.04626 tab. 3]. The paper says Anemia is
  the exception, with 0.932 against 0.931 [arxiv:1908.04626 tab. 3]. The
  setting is binary tasks with a single-layer BiLSTM [arxiv:1908.04626 §3.1],
  and the paper's definition of explanation is the plausibility reading
  [arxiv:1908.04626 §5]. The other side of this argument is the first bullet
  above.
- **Raw weights versus gradient-weighted maps.** On DETR segmentation, Chefer et
  al. report AP 5.6 for raw attention, 7.2 for Transformer attribution and 13.1
  for their gradient-weighted method [arxiv:2103.15679 tab. 1]. This fits the
  survey's caution about raw weights [arxiv:1904.02874 §7]. The numbers are
  author-reported and come from one task, so they do not settle faithfulness.
- **Gain from the cross-modality module.** The face-forgery paper's text says
  performance "gradually improves" with each module [arxiv:2103.12376 §5.2].
  Its ablation table shows the cross-modality module lowers the DF result from
  0.819 to 0.801 [arxiv:2103.12376 tab. 3]. The FS and NT results do improve.
- **Attention maps as evidence.** BEHRT shows attention maps linking a diagnosis
  to distant past diagnoses for two patients [arxiv:1907.09538 §3, fig. 5]. The
  paper gives these as illustrations and runs no faithfulness test, so they
  cannot settle the question raised in the first bullet above.
- **Attention averages versus causal tests.** The IOI paper cites attention
  patterns as support, for example Name Movers that attend to the indirect
  object 0.59 of the time [arxiv:2211.00593 §3.1]. It selects heads with path
  patching and mean ablation instead. Knocking out the three main Name Movers
  changes the logit difference by only 5%, because backup heads take over
  [arxiv:2211.00593 §3.4]. The authors note that faithfulness alone does not
  catch this redundancy [arxiv:2211.00593 §1]. The evidence covers GPT-2 small
  on one task.
- **Attention for time-series forecasting.** The LTSF-Linear paper argues that
  self-attention is permutation-invariant, so it loses temporal order unless
  position signals are added, and that the order is what matters most in a
  series [arxiv:2205.13504 §1]. Its linear baseline beats the published
  Transformer forecasters in all 36 dataset-horizon cells of its main table
  [arxiv:2205.13504 tab. 2]. The Transformer numbers are copied from the
  FEDformer paper, and the linear family uses a look-back of 336 against 96
  for the Transformers [arxiv:2205.13504 app. B.2]. The paper's own look-back
  study is a figure without numbers [arxiv:2205.13504 §5.3]. Neither side is
  re-run at a shared look-back here, so the question stays open for this page.
  ETSformer, which replaces attention with smoothing-based blocks, reports the
  lowest MSE on 22 of 24 multivariate rows against Autoformer and Informer
  [arxiv:2202.01381 tab. 1]. Its baseline numbers come from earlier papers, and
  its look-back search covers {96, 192, 336, 720} [arxiv:2202.01381 tab. 3].
  The linear paper's family uses a look-back of 336 [arxiv:2205.13504 app. B.2],
  so the two papers' settings differ, and the results cannot be ranked against
  each other here.
- **Full model versus plain cross-attention.** In the LayoutDiffusion ablation,
  the full model reaches FID 16.57 and the plain cross-attention variant 16.46
  [arxiv:2303.17189 tab. 2]. On FID, the table does not show the full model
  ahead, and the paper gives no FID spread for these rows
  [arxiv:2303.17189 tab. 2]. The gain from the layout module is therefore not
  shown on this metric.

## Papers

- [Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems](/papers/1512.08756): feed-forward attention pooling with one scalar score per step, compared against an unweighted mean on toy memory tasks.
- [Attention-over-Attention Neural Networks for Reading Comprehension](/papers/1607.04423): a second attention over query words weights the per-word maps of a cloze reader, with optional n-gram re-ranking.
- [Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532): an attention glimpse network feeding a CNN in an end-to-end RL driving pipeline, with an LSTM and a DQN or DDAC head. Qualitative results only.
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](/papers/1705.05414): an LSTM decoder with attention over knowledge-base keys added to its output logits, for in-car dialogue.
- [Recent Trends in Deep Learning Based Natural Language Processing](/papers/1708.02709): a survey of deep learning for NLP, covering attention and the Transformer in prose and collected benchmark tables. It runs no experiments.
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](/papers/1709.02054): attention decoder with a focusing network that pulls each character's attention back onto the character, for scene text.
- [AON: Towards Arbitrarily-Oriented Text Recognition](/papers/1711.04226): attention LSTM decoder over four directional feature sequences, fused with learned placement weights.
- [On the Automatic Generation of Medical Imaging Reports](/papers/1711.08195): co-attention over image regions and predicted tags, with an attention coverage penalty, inside a hierarchical report generator.
- [DecideNet](/papers/1712.06679): detection and regression density maps blended per pixel by a learned sigmoid gate, for crowd counting.
- [Attention U-Net: Learning Where to Look for the Pancreas](/papers/1804.03999): additive sigmoid attention gates on the skip connections of a 3D U-Net, for CT pancreas segmentation.
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](/papers/1811.00405): per-speaker GRU states with attention over earlier global states, plus an emotion attention variant for bidirectional use.
- [Sentiment and Sarcasm Classification with Multitask Learning](/papers/1901.08014): shared GRU with task-specific attention pooling and a neural tensor network fusion for joint sentiment and sarcasm.
- [Attention is not not Explanation](/papers/1908.04626): four tests of whether attention weights in a BiLSTM text classifier do real work, including an adversary that gives different attention.
- [Attention in Natural Language Processing](/papers/1902.02181): a survey giving one general attention model, a four-axis taxonomy and eleven compatibility functions. No experiments.
- [An Attentive Survey of Attention Models](/papers/1904.02874): a survey giving the generalized attention form, alignment and distribution functions, a four-axis taxonomy and the Transformer. No experiments.
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](/papers/1904.03451): VGG-16 encoders with spatial attention for sketch and photo embeddings, trained with triplet, domain and semantic losses.
- [BEHRT: Transformer for Electronic Health Records](/papers/1907.09538): BERT-style bidirectional self-attention over diagnosis codes, pre-trained by masked prediction and fine-tuned for next-visit prediction.
- [Natural Language Processing Advancements By Deep Learning: A Survey](/papers/2003.01200): a survey of deep learning in NLP, with attention in neural machine translation. It proposes no method.
- [SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture](/papers/2007.04417): CBAM attention on the skip connections of a U-Net, with depthwise-separable convolutions, for radar nowcasting.
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](/papers/2010.02795): a recurrent emotion model with attention over earlier context states; the paper's and the code's attention forms differ.
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](/papers/2011.13005): bidirectional cross-attention between two superpoint sets, predicting overlap and matchability scores that bias keypoint sampling.
- [AdaBins: Depth Estimation using Adaptive Bins](/papers/2011.14141): a mini-transformer predicts per-image bin widths, and per-pixel attention over bin centres gives depth.
- [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638): several attention decoders with a diversity term and masked top-layer re-embedding of unvisited nodes for routing.
- [Random Feature Attention](/papers/2103.02143): softmax attention replaced by random Fourier features, with linear cost in length and an optional recency gate.
- [Coordinate Attention for Efficient Mobile Network Design](/papers/2103.02907): direction-aware pooling gives row and column sigmoid gates for a mobile CNN.
- [PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency](/papers/2103.05465): nonlocal attention over putative correspondences, weighted by spatial consistency, to classify inliers.
- [UIEC^2-Net: CNN-based Underwater Image Enhancement Using Two Color Space](/papers/2103.07138): an attention block predicts per-pixel sigmoid maps that blend RGB and HSV branch outputs.
- [Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376): residual-guided spatial attention and dual cross-modality attention between an RGB stream and a noise stream.
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679): gradient-weighted attention maps and relevance propagation for co-attention and DETR-style Transformers.
- [Group-Free 3D Object Detection via Transformers](/papers/2104.00678): object candidates attend to all points through stacked decoder layers, replacing point grouping.
- [TransVG: End-to-End Visual Grounding with Transformers](/papers/2104.08541): one joint transformer encoder over image tokens, word tokens and a learned [REG] token, regressing one box for a text query.
- [Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](/papers/2106.06947): learned sensor graph with graph attention over neighbours, scoring sensor deviations for anomaly detection.
- [Dynamic Head: Unifying Object Detection Heads with Attentions](/papers/2106.08322): stacked scale-, spatial- and task-aware attention blocks that replace a detector's head, with deformable sampling over the feature pyramid.
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](/papers/2106.15004): agent-node attention on a lane graph, then a route-conditioned 32-head attention decoder for driving trajectories.
- [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](/papers/2107.12626): a bidirectional LSTM with attention over past steps feeding a non-linear predictor, for multi-sensor time series.
- [Shallow Attention Network for Polyp Segmentation](/papers/2108.00882): deep features gate shallow features through a ReLU attention map, with colour exchange and a logit correction at inference.
- [NAM: Normalization-based Attention Module](/papers/2111.12419): channel attention from batch-norm scale factors, with a sigmoid gate and an L1 penalty on the scales.
- [GMFlow: Learning Optical Flow via Global Matching](/papers/2111.13680): global softmax matching over all pixel pairs, with Transformer-enhanced features and self-attention flow propagation.
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](/papers/2203.17270): map queries read multi-camera features through deformable spatial cross-attention, and a temporal attention reads the previous map.
- [Multi-View Transformer for 3D Visual Grounding](/papers/2204.02174): a transformer decoder shared across rotated views of a point-cloud scene, fusing object features with words.
- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](/papers/2204.13154): a survey from additive attention to the Transformer and its cost fixes, with core formulas. No experiments.
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756): a survey of attention in vision models, grouped by how scores are computed, with formulas for channel, spatial and self-attention. No experiments.
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](/papers/2205.02833): map cells attend to the pixels of six calibrated cameras, with a camera-aware key embedding.
- [OnePose: One-Shot Object Pose Estimation without CAD Models](/papers/2205.12257): graph attention over feature tracks of 3D points, with self and cross attention, matched to query image features.
- [Are Transformers Effective for Time Series Forecasting?](/papers/2205.13504): one-layer linear forecasters tested against attention-based forecasters at the same look-back window.
- [Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval](/papers/2205.13760): decoder-only protein transformer with head-grouped depthwise convolutions and grouped ALiBi, mixed with a retrieved profile at inference.
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](/papers/2206.08129): control-branch attention over the image grid, guided by the trajectory branch's hidden state.
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](/papers/2209.05451): latent cross-attention over voxel patches for language-conditioned robot manipulation, trained as one agent over 18 simulated tasks.
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](/papers/2210.06551): a dual-stream spatial and temporal Transformer with learned fusion, pretrained to lift corrupted 2D skeletons to 3D.
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](/papers/2211.00593): path patching and mean ablation find a 26-head attention circuit in GPT-2 small for one task.
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](/papers/2301.03505): a review of softmax self-attention in vision transformers for medical imaging, with efficiency variants. No experiments of its own.
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](/papers/2303.17189): object-aware cross-attention and a layout fusion module in a diffusion UNet, for layout-to-image generation.
- [RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](/papers/2304.03198): a convolution whose window weights come from a softmax over each window's k² slots.
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](/papers/2305.10431): a cross-attention localization loss during training and delayed subject conditioning at sampling.
- [Dense Text-to-Image Generation with Attention Modulation](/papers/2308.12964): training-free modulation of Stable Diffusion's attention toward region masks during the first 30% of steps.
- [LiCS: Navigation using Learned-imitation on Cluttered Space](/papers/2406.14947): a three-layer encoder and decoder Transformer over LiDAR patches with a goal query, trained by behaviour cloning for local navigation.
- [Graph Transformers: A Survey](/papers/2407.09777): a survey of how graph structure enters self-attention, with bias types, attention cost classes and a design guide. No experiments.
- [Gemma 2: Improving Open Language Models at a Practical Size](/papers/2408.00118): decoder-only models with alternating local and global attention layers, logit soft-capping and grouped-query attention.
- [Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling](/papers/2411.04331): proprioception-queried cross-attention over ResNet18 image features, with a hard argmax mask in the policy path and a contrastive loss on augmented views.
- [Deep RL for pickup-and-delivery routing](/papers/2608.14156): an attention route builder with a pickup-order mask and a soft missed-client penalty.
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](/papers/1612.01744): attention encoder-decoder from French speech to English text, with a convolutional location term for speech.
- [P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011): pooled keys and values at four scales, with depthwise convolutions on each pooled map, in a vision backbone.
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](/papers/2202.01381): exponential smoothing attention and frequency attention in a forecasting encoder-decoder.
- [Attention Spiking Neural Networks](/papers/2209.13929): temporal, channel and spatial attention on the membrane potential of spiking neurons.
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](/papers/2306.00864): bidirectional multimodal attention over radiograph, text, lab and demographic tokens, for diagnosis.
- [Introduction to Transformers: an NLP Perspective](/papers/2311.17633): a tutorial survey of the Transformer's attention, masks, positions and warmup schedule. No experiments.
- [iLRM: An Iterative Large 3D Reconstruction Model](/papers/2507.23277): a transformer that outputs 3D Gaussians, with viewpoint tokens updated by cross-attention to their own view's image tokens.