Visual Attention Methods in Deep Learning: An In-Depth Survey

In plain words

Attention is a way for a neural network to put more weight on the parts of its input that matter for the task and less on the rest, much like a reader's eye settling on the important words of a page. Over the last few years many different versions of this idea have been built into image models. This paper is a guide to that zoo. It sorts the versions into families, explains how each family decides where to look, and says where each one tends to help and where it costs too much.

The sorting is by the mechanism that produces the attention weights. Some families weight every input a little, using smooth functions that can be trained by ordinary gradient descent. Others pick one input or sample one, which is harder to train. Within each family, the paper looks at what the weights are computed over: the channels of a feature map, its spatial positions, or pairs of positions in the same image (self-attention, the Transformer's core). It also covers attention across different kinds of input such as images and text, attention built from simple arithmetic, and attention that uses recurrent steps.

The paper is not a new method and does not run new experiments. Its value is the map: it tells a reader which family to look at for a given vision problem, what each family assumes, and which open problems remain. The authors also argue that current attention models are hard to compare fairly and hard to interpret.

TL;DR

This is a literature survey, not a method paper. Its main structure is a taxonomy that splits attention into soft (deterministic) and hard (stochastic) families, with sub-families for channel, spatial, self, arithmetic, multi-modal, logical and category-based attention on the soft side, and statistical, reinforcement-based, Gaussian and clustering attention on the hard side fig. 2 §2. For each family it gives the core formulas, strengths and limitations tab. 1. It has six numbered equations eq. 1–eq. 6 and a self-attention algorithm alg. 1, and it lists open problems in efficiency, generalization, training data, comparison and interpretability §4. The authors' own counts disagree (about 50, "70", and "more than 70" in different places), and several citations and symbols are wrong or undefined; see Pitfalls.

Problem

Attention modules are added to vision networks in many ways, and the literature had no single overview covering all of them. The authors note that earlier surveys covered narrower slices, such as NLP attention, transformer-based models or graph attention §1. They also argue that transformers cover only one of many self-attention categories §1 and that transformers are costly in compute and training data §1 §4.

Method

This section is the survey's taxonomy and the formulas it states. Nothing here is a new method; every item is attributed to an original paper that is cited in the source.

Taxonomy

The paper groups methods by the main function used to produce attention scores, such as softmax or sigmoid §2. A method can fit more than one family; the survey files it under the family that best describes its main mechanism and primary application fig. 2.

• Soft (deterministic), §2.1. Scores are a weighted combination of all inputs, computed with differentiable functions such as softmax or sigmoid. Training uses back-propagation. The survey notes two costs: high computational complexity, and weight given to inputs that should not be attended §2.1.
  • Channel, §2.1.1. Scores are computed per channel, typically after global pooling. Examples: SE, ECA, split attention, second-order (SOCA), high-order, harmonious, auto-learned (HOGA), double (A²-Net), dual (DAN) and frequency-channel attention (FCA) §2.1.1 fig. 3.
  • Spatial, §2.1.2. Scores are computed over spatial positions of a feature map. Examples: the spatial branch of CBAM, co-attention and co-excitation, SPAN, spatial-spectral attention, PiCANet, pyramid feature attention, spatial attention pyramid and RANet §2.1.2 fig. 4.
  • Self-attention, §2.1.3. Each input attends to all others by comparing queries with keys. Examples: the Transformer, Swin, deformable attention, standalone self-attention, clustered attention, slot attention, SMYRF, random feature attention (RFA), non-local networks, NLSA, X-Linear, axial attention and efficient (linear) attention §2.1.3 fig. 5 alg. 1.
  • Arithmetic, §2.1.4. Scores come from simple operations on other scores or features, such as dropout masks, mirroring, reversing or reciprocals. Examples: attention-based dropout (ADL), mirror attention and reverse attention §2.1.4 fig. 6.
  • Multi-modal, §2.1.5. Scores use more than one input type, or relate two sets of inputs. Examples: cross attention (CAN), criss-cross attention, Perceiver, stacked cross attention and boosted attention §2.1.5 fig. 7.
  • Logical, §2.1.6. Scores come from recurrent or sequential networks, or are built to respect set structure. Examples: S3TA, permutation-invariant set transformer, show-attend-tell, Kalman-filter attention and prophet attention §2.1.6 fig. 8.
  • Category-based, §2.1.7. Attention is supervised or guided by class information. Examples: GAIN, curriculum-enhanced supervised attention and the attentional class feature network (ACFNet) §2.1.7 fig. 9.
• Hard (stochastic), §2.2. The model selects or samples one state, or uses latent random variables. Training needs variance-reduced estimators or reparameterization, and the survey notes gradient problems §2.2 fig. 10.
  • Statistical, §2.2.1. Bayesian attention modules, Bayesian attention belief networks, repulsive multi-head attention and variational attention §2.2.1.
  • Reinforcement-based, §2.2.2. Self-critic attention trained with REINFORCE, reinforced self-attention and reinforcement attention for video §2.2.2.
  • Gaussian, §2.2.3. Attention maps from a 2D Gaussian kernel instead of softmax; uncertainty-aware attention §2.2.3.
  • Clustering, §2.2.4. Expectation-maximization attention §2.2.4.

Formulas stated in the survey

Only the formulas below are typeset with equation numbers in the source. Others in the survey are written inline; they are listed in the sub-sections above.

Squeeze-and-excitation (SE) gate, as typeset in the survey §2.1.1 eq. 1. Here f_g is the globally pooled channel vector and FC are fully connected layers:

f_s = \sigma\big(FC(\mathrm{ReLU}(FC(f_g)))\big) eq. 1

CBAM channel attention §2.1.1 eq. 2. f is the input feature map, and the same MLP is applied to the max-pooled and average-pooled descriptors:

f_{ch} = \sigma\big(MLP(\mathrm{MaxPool}(f)) + MLP(\mathrm{AvgPool}(f))\big) eq. 2

CBAM spatial attention §2.1.2 eq. 3. Max and average pooling are taken across channels, concatenated, and convolved with a 7×7 kernel:

f_{sp} = \sigma\big(Conv_{7\times7}([\mathrm{MaxPool}(f); \mathrm{AvgPool}(f)])\big) eq. 3

Scaled dot-product attention and multi-head attention, as in the survey's algorithm alg. 1. The survey writes the value projection as \mathbf{V} = \mathbf{X}\mathbf{V}_Q, which is a typo for \mathbf{W_V} (see Pitfalls):

\mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V} alg. 1

Slot attention §2.1.3 eq. 4 eq. 5 eq. 6. The survey gives an attention step with a softmax, followed by a weighted mean, and an \ell_2 reconstruction loss. Symbols n(\cdot), q(\cdot) and v(\cdot) are the learned maps, and c is not defined in the survey:

a = \mathrm{Softmax}\left(\frac{1}{\sqrt{D}}\, n(h)\, q(c)^T\right) eq. 4

r = \mathrm{WeightedMean}\big(a, v(h)\big) eq. 5

\mathbb{L} = \Big\lVert \Big(\sum_{k=1}^{K} m_k x_k\Big) - \mathbb{I} \Big\rVert_2^2 eq. 6

Other formulas the survey states inline, each cited to its sub-section:

• Second-order channel statistics (SOCA): the covariance \Sigma = U\Lambda U^T is raised to the power \alpha, giving \Sigma = U\Lambda^\alpha U^T, with \alpha < 1/2 taken from earlier work §2.1.1.
• Efficient (linear) attention: \rho(QK^T)V is rearranged to \rho_q(Q)\,(\rho_k(K)^T V), with normalization applied to each factor, which drops the cost from O(n^2) to O(n) §2.1.3.
• Axial attention: cost O(hwm) §2.1.3. Clustered attention: cost O(qc) §2.1.3. Perceiver: cost O(MN) §2.1.5. Induced set attention: cost O(mn) §2.1.6.
• Boosted attention: I' = W I \circ \log(W_{sal} I + \epsilon) §2.1.5.
• Reverse attention, as the survey writes it for convergence: \mathrm{sigmoid}\big(\frac{1}{\mathrm{ReLU}(F_{conv}) + 0.125} - 4\big) §2.1.4.
• Gaussian attention: K = \exp\big(-\frac{1}{\alpha}(u-\mu)^T\Sigma^{-1}(u-\mu)\big) with u = [x, y]^T §2.2.3.

Pseudocode: scaled dot-product self-attention

The survey's steps for multi-head self-attention alg. 1 §2.1.3, written in our own words.

# Multi-head self-attention  [alg. 1] [§2.1.3]
# Input:  sequence x of shape (n, d_model)
# Output: attended sequence of shape (n, d_model)

def self_attention(x):
    # Project the same input into queries, keys and values  [alg. 1]
    queries = x @ W_Q                       # (n, d_k)
    keys = x @ W_K                          # (n, d_k)
    values = x @ W_V                        # (n, d_v)
    # Scores: similarity of every query with every key  [alg. 1]
    scores = queries @ keys.T / sqrt(d_k)   # (n, n)
    # Normalize the scores and mix the values  [alg. 1]
    weights = softmax(scores, axis=-1)
    return weights @ values
    # The survey says to repeat this for all heads [alg. 1].
    # Concatenation and output projection are not stated.

Hyperparameters

The survey is not an experiment, so it has no training configuration of its own. The values it quotes from original papers are listed here.

Results

No experiments of the authors' own. The survey quotes numbers from the original papers; they are not re-measured or checked here, and the comparison conditions are not restated.

Paper vs code

The abstract links https://github.com/saeed-anwar/VisualAttention as the place where "all the information" is provided. At commit c7dda96e2feaf867167e9ccf834980dd198eb578 (shallow clone of the default branch), that repository contains only a README.md with the title "VisualAttention", and no LICENSE file and no code. There is nothing to compare against the paper. Other branches or later commits were not checked. The owner saeed-anwar matches the name of author Saeed Anwar, and the paper itself links the repository; the owner was not checked further on GitHub.

Pitfalls

• Not a method. There is no algorithm to implement beyond the self-attention steps alg. 1 and the formulas eq. 1–eq. 6 and inline formulas above. Agents should not treat the taxonomy as a specification.
• Counts disagree. The abstract says 50 techniques; the introduction says "70" §1; the conclusion says "more than 70" §5. Use the category list above rather than a number.
• Citation mismatches.
  • Table 1 cites ding2020high for A²-Net, but the text gives A²-Net as chen20182 and ding2020high as High-order attention tab. 1 §2.1.1.
  • Sparse Transformers are cited to the Swin paper §2.1.3. Check the original before using that attribution.
  • The Criss-cross caption in the multi-modal figure cites hou2019cross (the cross-attention paper) for criss-cross attention fig. 7.
  • GatCluster is cited to two different sources (gaussian_attention and niu2020gatcluster) §2.2.3 tab. 1.
• Algorithm typo. The value projection is written as \mathbf{V} = \mathbf{X}\mathbf{V}_Q alg. 1. It should be the value weight \mathbf{W_V}. The algorithm also omits head concatenation and the output projection.
• Slot attention is under-specified. The symbol c and the maps n(\cdot), q(\cdot), v(\cdot) are not defined. The text says the softmax is "followed by normalization according to slot dimension" without saying along which axis §2.1.3. The loss mixes m_k, x_k and \mathbb{I} without defining them eq. 6. Read the original slot-attention paper before implementing.
• Garbled explanations. The SOCA paragraph's argument about eigenvalues is hard to follow §2.1.1. The reverse-attention constants are given "for convergence" with no further reason §2.1.4.
• Uncited claims. The application list in §3 gives examples such as non-local attention in video processing and graph attention in segmentation without citations §3. Treat those as the authors' assertions.
• Efficiency claims are not measured here. The FLOP and memory figures in Results are quoted from original papers under their own conditions.
• Dated. The source is v3 of the arXiv paper (2024-05-05 on arXiv), and the survey's literature ends around 2022. Methods after that are not covered. The fetched LaTeX source (Manuscript.tex) was used for section, equation, table and figure numbers. The Response_v1.tex file in the source is a reviewer response, not part of the paper, and was not used.
• Overlapping categories. Perceiver and criss-cross are transformer-type methods placed under multi-modal §2.1.5. Harmonious attention mixes soft and hard attention but is filed under channel attention §2.1.1. Expect a method to fit in more than one place.

Lineage

• Surveys cited by this paper: earlier transformer surveys §1; NLP-focused and graph-focused attention surveys §1. None is compiled on this wiki.
• Compared against: other attention surveys are discussed only in the introduction, with no comparison of their content §1.
• Superseded by: not stated.