Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

In plain words

Neural networks that read a long input, such as a sentence or an image, used to squeeze all of it into one fixed-size summary before answering. Attention lets the network look back at the parts of the input that matter for the current step, and weigh them by how relevant they are. The idea is old and was first inspired by how the human eye focuses on a small part of a scene at a time.

This paper is not a new method. It is a road map that walks through the years of attention research: the first models, the 2015 wave that made attention standard for translation and image captioning, the Transformer that uses only attention, and the many later variants that try to make it cheaper. Read it to see which mechanism exists and where it came from, then read the original papers before you implement one.

TL;DR

A survey of attention mechanisms in neural networks, from the late 1980s to the Transformer and its variants. It gives the core formulas of additive (Bahdanau), soft and hard, global and local, and scaled dot-product (multi-head) attention. It reports no experiments and no code, so use it as a map of mechanisms and variants, not as an implementation guide.

Problem

Early work on attention in neural networks tried to copy visual attention in the human eye, which sees only a small patch in high resolution and moves to other parts of a scene §1. For sequence models, the paper names one failure that attention fixes: an encoder-decoder for neural machine translation compresses the whole source sentence into a fixed-length vector, which makes it hard to capture every detail of a very long sentence §3.

The paper also names the cost of the cure. Self-attention needs to store the attention matrix to compute gradients, so time and memory grow quadratically with sequence length, which makes training slow on very long sequences §5.3. The variants in §5.1 and §5.3 are mostly attempts to reduce that cost.

Method

The paper covers attention in five stages: early visual attention models §2; the 2015 additive, soft/hard and global/local attention for sequence and image tasks §3; memory networks and self-attention models of 2015–2016 §4; the Transformer, which replaces recurrence and convolution with attention §5; and self-attention and Transformer variants §5.1–§5.3. It has no algorithm block of its own. The pseudocode below is our condensation of the formulas in §2 and §5.

Additive (Bahdanau) attention §2. A bidirectional RNN encodes the source into annotations, one per word. For each output step, a small feedforward network scores each annotation against the previous decoder state. A softmax turns the scores into weights, and the context vector is the weighted sum of the annotations. The paper gives the energy, softmax and context equations §2. Its alignment model is a multilayer perceptron with a single hidden layer, whose size is not stated §2.

Soft and hard attention §3. The image captioning model of Xu et al. applies the same idea to CNN feature vectors at different locations. The soft variant takes the expected context vector and is trained by standard backpropagation. The hard variant samples one location per step from a multinoulli distribution over the weights and is trained with REINFORCE. The paper also describes a deterministic variant with an extra training constraint that spreads attention over all image parts §3.

Global and local attention §3. Luong et al. compare two forms for neural machine translation. Global attention scores the current target state against every source state with a content-based score function. Local attention first picks an aligned source position and attends only to a window around it. The paper names three score alternatives without writing them out §3.

Scaled dot-product attention §5. Queries, keys and values are compared by dot product, divided by the square root of the key dimension, normalized with softmax, and used to weight the values. The Transformer applies this in multiple heads, with learned projections of queries, keys and values, then concatenates the heads and projects them once more §5. In the decoder, a mask prevents a position from using later words §5.

Variants §5.1–§5.2. Relation-aware, directional, reinforced, outer product, block, sparse Sinkhorn and adaptive-span self-attention change how tokens are paired or how many are compared. The Universal Transformer applies the same multi-head block iteratively to all positions in parallel §5.2.

Equations, as written in the paper:

e_{ij} = f(h_{i-1}, a_j) §2

\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x}\exp(e_{ik})} §2

c_i = \sum_{j=1}^{T_x} \alpha_{ij} a_j §2

Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}})V §5

MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O §5, with head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)

Pseudocode for the two core computations:

# Additive (Bahdanau) attention for one decoder step  [§2]
# Input:  annotations (T_x, d_a); previous decoder state h_prev (d_h,)
# Output: context (d_a,); weights (T_x,)

def additive_context(annotations, h_prev):
    # Score each annotation against the previous state  [§2]
    energies = [f(h_prev, a_j) for a_j in annotations]
    # Softmax over source positions gives the weights  [§2]
    weights = softmax(energies)                          # (T_x,)
    # Weighted sum of annotations is the context vector  [§2]
    context = sum(w_j * a_j for w_j, a_j in zip(weights, annotations))
    return context, weights

# Scaled dot-product attention  [§5]
# Input:  queries (n, d_k); keys (m, d_k); values (m, d_v)
# Output: outputs (n, d_v)

def scaled_dot_product_attention(queries, keys, values):
    # Dimension of the keys, used for scaling  [§5]
    d_k = keys.shape[-1]
    # Compare every query with every key  [§5]
    scores = queries @ keys.T / sqrt(d_k)                # (n, m)
    # Softmax over keys gives the weights of each query  [§5]
    weights = softmax(scores, axis=-1)                   # (n, m)
    # Each output is a weighted sum of values  [§5]
    return weights @ values                              # (n, d_v)

Figures and tables: fig. 1 shows the Bahdanau encoder-decoder extension §3. fig. 2 shows two attention examples: image captioning and visual question answering with word-, phrase- and question-level co-attention maps §3. fig. 3 shows the Transformer, with multi-head and scaled dot-product attention in detail §5. tab. 1 is a notation summary §5.

Hyperparameters

The paper is a survey and states no training setup. The only numbers below are the ones it gives in text.

Results

No experiments. The paper reports no benchmark numbers of its own. It mentions results only in passing, for example that the Transformer performed well on two machine translation tasks and on English constituency parsing §5. There is no baseline and no hardware setup. Take any performance claim from the original paper.

Paper vs code

No official code. The abstract and the LaTeX source contain no repository link, and a literature review has no implementation to link. Discrepancies do not apply.

Pitfalls

• Survey, not a method. Nothing here can be implemented as stated. For a working system you need the original papers for each mechanism you pick.
• Equation numbers are not cited. The source uses labelled equations, and I could not confirm the numbers in the compiled PDF, so equations are cited by section.
• Unstated details in the Bahdanau alignment model. The hidden size of the scoring MLP is not stated §2. The formula also writes the energy as a function of the previous state and the annotation only §2, so any other inputs are the implementer's choice.
• Hard attention is a sampling problem. Training hard attention needs REINFORCE, which is not described in the paper beyond naming it §3.
• Unspecified constraint. The doubly stochastic constraint of the deterministic captioning variant is described in words only; its form is not given §3.
• Unnamed score functions. Luong's three score alternatives are referred to but not written out §3. How local attention chooses its aligned position, and the window size, are not stated §3.
• Masking is described loosely. The paper says masked embeddings are "multiplied with zero" §5. Where the mask enters the score computation is not stated, so check the original before implementing decoder masking.
• Notation mismatch. The multi-head text says projections go to dimensions d_q, d_k and d_v §5, but the attention equation uses d_k for queries and keys and d_q appears nowhere else. Treat d_q as an undefined symbol.
• Cost is not hidden in the paper. Self-attention's time and memory grow quadratically with sequence length, because the attention matrix must be stored for gradients §5.3. Most variants in §5.1 and §5.3 trade this for other limits, but the paper does not compare them with numbers.
• Coverage ends at its own cut-off. The paper states no date cut-off for the works it covers. Recent work (for example ViT §5.2) appears only as a one-line mention.

Lineage

• Builds on: the visual attention models of the late 1980s and 1990s §2, then the additive attention of Bahdanau et al. §2 and the Transformer of Vaswani et al. §5. The source gives no arXiv ids for these works.
• Compared against: none; the paper has no experiments.
• Superseded by: not stated in the paper.