Recent Trends in Deep Learning Based Natural Language Processing

In plain words

Computers learned to read text by first being given lists of hand-written rules and features. Over the last few years, the field moved to neural networks that learn their own internal description of words and sentences from large amounts of text. This paper is a long tour of that change. It goes through the main kinds of network used for language, explains what each one is good at, and lists how well they did on standard tests.

The first step is to turn each word into a list of numbers, so that words used in similar ways get similar lists. Then come three families of network that read those lists. Convolutional networks look for short patterns, like a phrase that signals a negative review. Recurrent networks read one word at a time and carry a memory forward. Recursive networks follow the grammatical tree of a sentence. Later sections cover attention, which lets a model look back at the most relevant words, and the Transformer, which uses attention alone. The paper also covers models that generate text, models that learn sentence representations without labels, and models with an external memory.

If you want one takeaway: the paper is a map and a scoreboard, not a recipe. It does not give you settings you can reuse, and most of its numbers are copied from other papers. The pitfalls below say which parts are reliable.

TL;DR

A survey of deep learning for natural language processing. It covers word vectors, convolutional, recurrent and recursive networks, attention and the Transformer, reinforcement and generative models, and memory networks. It gathers the benchmark results that the cited papers reported for tagging, parsing, entity recognition, role labelling, sentiment, translation, question answering, dialogue and contextual embeddings. It runs no experiments of its own. Use it to pick a model family and to check reported baselines. Do not use it as a source of hyperparameters, because it states almost none.

Problem

The authors observe that, before deep learning, NLP systems depended heavily on hand-crafted features, which are slow to build and often incomplete §1. Dense vector representations learned from data, and deep models built on them, have since given better results on many tasks §1, §2. The paper sets out to cover the most used deep learning methods in NLP in one place, and to show their evolution and their reported results §1, §8. It claims to be the first survey of this kind to do so, and says it intends to update the text as the field changes §1, footnote.

Method

This is a survey, so it has no method of its own. Below is the structure of the paper, with the mechanisms the survey explains. The equations and pseudocode are the survey's, rewritten here for implementers. Citations point to sections, not equation numbers, because the equation numbering was not checked against the compiled PDF.

Distributed representations §2

1. Word embeddings §2.1. Words are mapped to dense vectors using the distributional hypothesis: words in similar contexts get similar vectors. Vectors are usually pre-trained on a large unlabelled corpus with an auxiliary prediction task, and then used as the first layer of a model. The survey credits an early neural model with the first use of pre-trained embeddings §2.1.
2. word2vec §2.2. Two models: CBOW predicts a target word from the words around it, and skip-gram predicts the surrounding words from the target. The survey describes a simplified CBOW with one context word, which is a bigram model §2.2. Negative sampling draws the negative words by frequency-based sampling §2.2.
3. GloVe §2.1. A count-based method: co-occurrence counts are normalised and log-smoothed, then factorised by minimising a reconstruction loss §2.1.
4. Other embedding variants §2.2, §2.3. Multi-word phrase vectors, sentiment-specific vectors that are trained with a polarity loss, task- specific re-training, and multi-sense vectors learned from parallel text §2.2. Character-level and subword methods: FastText represents a word as a bag of character n-grams inside a skip-gram model, which helps with unknown words §2.3.
5. Contextual embeddings §2.4. ELMo gives each token a vector that depends on its sentence. It takes the layers of a bidirectional language model and combines them with task-specific weights. The survey gives the weighted-sum form below. OpenAI-GPT pre-trains a Transformer language model. BERT is pre-trained with masked-word prediction and next-sentence prediction §2.4.

# One-context CBOW: a bigram-style model  [§2.2]
# Input:  context word index c; W_in (V, N), W_out (N, V)
# Output: probability of each vocabulary word given c, shape (V,)

def cbow_probabilities(c, W_in, W_out):
    # Context vector: row c of the input matrix  [§2.2]
    v_c = W_in[c, :]                       # (N,)
    # Score each word by its output vector: u_i = v_wi . v_c  [§2.2]
    scores = W_out.T @ v_c                 # (V,)
    # Softmax over the whole vocabulary  [§2.2]
    return softmax(scores)

# ELMo vector for one token: task-specific mix of biLM layers  [§2.4]
# layers[0] is the token embedding; layers[j], j >= 1, concatenates the
# forward and backward LSTM states of layer j  [§2.4]

def elmo_vector(layers, layer_logits, gamma):
    # Softmax-normalise one learned weight per layer  [§2.4]
    weights = softmax(layer_logits)
    # Weighted sum over all L + 1 layers  [§2.4]
    mixed = sum(w * h for w, h in zip(weights, layers))
    # gamma is a task-specific scale and a hyperparameter  [§2.4]
    return gamma * mixed

Convolutional networks §3

1. Sentence model §3.1.1. Each word has an embedding of size d. A filter of width h is applied to every window of h consecutive words. Each window gives one value, which is passed through a non-linearity (the survey uses tanh as the example). Max pooling over positions gives one value per filter, so the output length does not depend on the sentence length. The survey notes that typical models use hundreds of filters of different widths §3.1.1.
2. Window approach for tagging §3.1.2. For word-level labels (tagging, entity recognition, role labelling), a CNN classifies the word in the middle of each window. A conditional random field can be added to score whole label sequences §3.1.2. A time-delay network applies the convolution across all windows of the sentence §3.1.2.
3. Applications §3.2. Sentence classification with pre-trained or random embeddings §3.2. The dynamic CNN (DCNN) uses dynamic k-max pooling, which keeps the k strongest features in their original order, so that narrow filters can relate words far apart §3.2. The survey notes two weaknesses: CNNs need a lot of training data, and they model long-range order poorly §3.2.

# Basic sentence CNN with max pooling over positions  [§3.1.1]
# Input:  word embeddings W, shape (n, d); h = filter width
# Output: one pooled feature per filter

def sentence_cnn(W, filters, biases, h):
    # Concatenate each window of h consecutive words  [§3.1.1]
    windows = [W[i : i + h].reshape(-1)
               for i in range(len(W) - h + 1)]
    pooled = []
    for kernel, bias in zip(filters, biases):
        # Feature per window: c_i = f(w_{i:i+h-1} . k + b), f = tanh  [§3.1.1]
        feature_map = [tanh(win @ kernel + bias) for win in windows]
        # Max pooling keeps the strongest response of each filter  [§3.1.1]
        pooled.append(max(feature_map))
    # Fixed length, whatever the sentence length  [§3.1.1]
    return pooled

Recurrent networks §4

1. Simple RNN §4.2. The hidden state is s_t = f(U x_t + W s_{t-1}), where f is a non-linearity such as tanh or ReLU, and U and W are shared across time steps. The survey says plain RNNs suffer from vanishing gradients §4.2.
2. LSTM §4.2. Adds input, forget and output gates and a cell state, which lets errors flow back over many steps. Equations and code are in the pseudocode below.
3. GRU §4.2. Two gates (reset and update) and no separate memory cell. It exposes its whole hidden state. The survey reports that LSTM and GRU both beat the simple RNN on music and speech tasks, and that neither is clearly better than the other §4.2. These comparisons are not on NLP tasks §4.2.
4. Applications §4.3. Bidirectional LSTM with a CRF for entity recognition §4.3. LSTM encoding of tweets for sentiment §4.3. Encoder decoder (sequence-to-sequence) models for translation, captioning and dialogue, where a decoder generates tokens one by one from a fixed summary vector, often with beam search at inference §4.3.
5. Attention §4.4. The decoder looks at all encoder states for each output token, using a weighted sum whose weights depend on compatibility with a query. Attention was first applied to translation, and it helps most for long inputs §4.4.
6. Transformer §4.5. Drops recurrence and convolution in favour of multi-head self-attention, with residual connections and layer normalisation. It is easier to parallelise §4.5.

# One LSTM step, with the gate equations as the survey writes them  [§4.2]
# The survey gives no shapes; the comments are ours.

def lstm_step(x_t, h_prev, c_prev, params):
    # Concatenate the previous hidden state and the current input  [§4.2]
    z = concat(h_prev, x_t)
    # Forget, input and output gates, each a sigmoid  [§4.2]
    f_t = sigmoid(params.W_f @ z + params.b_f)
    i_t = sigmoid(params.W_i @ z + params.b_i)
    o_t = sigmoid(params.W_o @ z + params.b_o)
    # Candidate cell content; the paper writes X where z is meant  [§4.2]
    candidate = tanh(params.W_c @ z + params.b_c)
    # Keep part of the old cell and add part of the candidate  [§4.2]
    c_t = f_t * c_prev + i_t * candidate
    # Hidden state read from the cell  [§4.2]
    h_t = o_t * tanh(c_t)
    return h_t, c_t

# Multi-head scaled dot-product attention  [§4.5]
# queries (n_q, d); keys, values (n_k, d); heads = one (W_q, W_k, W_v) per
# head. The paper does not state d_k, the number of heads, or projection sizes.

def multi_head_attention(queries, keys, values, heads, d_k, W_o):
    outputs = []
    for W_q, W_k, W_v in heads:
        # Project the inputs for this head  [§4.5]
        q = queries @ W_q
        k = keys @ W_k
        v = values @ W_v
        # Scores are dot products divided by sqrt(d_k)  [§4.5]
        scores = q @ k.T / sqrt(d_k)
        # Softmax over the keys, then weight the values  [§4.5]
        outputs.append(softmax(scores) @ v)
    # Concatenate the heads and project back  [§4.5]
    return concat(outputs) @ W_o

Recursive networks §5

1. Basic model §5.1. Each node of a parse tree is a function of its two children. In the simplest form, a parent vector p = tanh(W [left; right]), with W of size D by 2D. The survey shows this applied bottom-up to phrases §5.1.
2. Variants §5.1. MV-RNN gives every word and phrase both a vector and a matrix; combining two constituents multiplies one's matrix by the other's vector. The recursive neural tensor network adds a tensor of bilinear forms between the two child vectors, with more interaction and fewer parameters than MV-RNN §5.1.
3. Applications §5.2. Parsing with a plausibility score per phrase, beam search for the best tree, and a max-margin training objective §5.2. Phrase-level sentiment, where each node gets a label §5.2. Tree-structured LSTMs, which the survey says help with sentiment and sentence relatedness §5.2.

Reinforcement learning, unsupervised and generative models §6

1. Sequence-level training §6.1. Teacher forcing trains the model on true prefixes, which leaves a gap at test time called exposure bias. Reinforcement learning (REINFORCE) optimises a task reward directly, for example a metric that is not differentiable. Adversarial training uses a discriminator in place of a hand-written reward §6.1. The survey lists two drawbacks: the state and action spaces must be kept small, and reward functions are hard to design and measure §6.1.
2. Sentence encoders §6.2. Skip-thought predicts the sentences before and after a given sentence §6.2. A later model reconstructs the input sentence with an LSTM autoencoder, and the survey reports that reconstruction worked better than next-token prediction as the auxiliary task §6.2.
3. Generative models §6.3. A recurrent variational autoencoder gives a global latent code for each sentence, so that samples from a prior produce sentences. GAN-based text models either feed the latent code at each step and use soft word outputs, or treat the generator as a stochastic policy and train it with Monte Carlo search and the discriminator as reward §6.3. Evaluation is hard, so the survey mentions oracle grammars and BLEU against held-out data §6.3.

Memory-augmented networks §7

Memory networks store entries that the model reads through attention. In the end-to-end version the reading is soft, so training is differentiable, and several reading hops are needed for multi-fact questions §7. The dynamic memory network runs the same idea over text, with input, attention and answer modules, and the same design was applied to visual question answering §7.

Hyperparameters

The survey has no hyperparameter section and gives almost no training settings. The table lists the few concrete numbers it does give, plus the symbols it uses without values. Most rows are "not stated", as the rules require.

Results

Author-reported. The survey runs no experiments. Each number is the one the cited paper reported, as the survey copies it, on the dataset and metric in each table caption. The survey does not state the training setup of each cited model, so rows are not normalised for data size, model size or compute. Scores are as of the survey's version and may not be the best known today.

POS tagging, WSJ-PTB, per-token accuracy (%) tab. 2

Parsing, WSJ tab. 3. Dependency parsing (UAS / LAS) and constituency parsing (F1).

Named entity recognition, CoNLL 2003, F1 (%) tab. 4

Semantic role labelling, F1 (%) tab. 5

Sentiment classification, accuracy (%) tab. 6

Machine translation, BLEU tab. 7

Question answering tab. 8

Dialogue tab. 9

Contextual embeddings: ELMo over previous state of the art tab. 10. The caption says this is adapted from the ELMo paper, and that the previous-SOTA numbers come from it. Source column entries are the bibliography keys used by the survey (see Pitfalls).

BERT against a BiLSTM + ELMo + attention model, GLUE-style tasks tab. 11

The frameworks named for building embeddings are listed in tab. 1: S-Space (Java), Semanticvectors (Java), Gensim (Python), Pydsm (Python), Dissect (Python), FastText (Python) and ELMo (Python). The table gives a URL for each.

Paper vs code

No official code. The survey has no method or experiments of its own, so there is no implementation to compare with the text. The only URLs in the source are third-party toolkits and resources in tab. 1 and figure credits, none by the authors. The arXiv metadata has no comment field. No GitHub search was run, because a survey has no official repository to find.

Pitfalls

• Not a method. No experiment is run by the authors. Every number in the results tables is copied from a cited work, and the survey does not say how each was measured. Rows in one table may differ in data, model size, training budget and preprocessing, so do not rank rows across papers from the table alone tab. 2–11.
• The text and the table disagree on translation. The text says the Transformer reaches state-of-the-art results on English-German and English-French translation §4.5. In the table, its English-French score (41.0) is below two rows in the same table (41.16 and 41.29) tab. 7.
• Transformer details are missing. Positional encodings and dropout are named but not specified. The number of heads, the key size d_k and the number of layers are symbols with no values §4.5. The scaled dot-product formula is given, so use it, but fill the rest from the original Transformer paper.
• LSTM candidate term. The candidate equation writes the capital letter X, which is never defined. Read it as the concatenated vector [h_{t-1}; x_t] that the gate equations use §4.2.
• GRU candidate term. The candidate-state equation uses U_z, the same weight as the update gate, where a separate candidate weight is expected. This looks like a typo. Check the GRU paper the survey cites before you implement it §4.2.
• CBOW hidden size. The text gives the hidden layer size as N in one place and H in another. Treat them as the same value §2.2.
• CBOW objective and gradient. The log-likelihood is written as a sum over the whole vocabulary. The usual form is the log-probability of the observed target word only. The gradient the survey shows covers only the target-word term; the general softmax gradient also has a term for every other word, which is minus the probability times the context vector. This is our reading of the maths, not something the survey states §2.2.
• Embedding size. The survey says the dimension is chosen by prediction accuracy, but it gives neither a value nor a procedure §2.2.
• BERT masking. The masking rate is "a percentage", with no value §2.4.
• ELMo training. The survey does not say whether the biLM is frozen or fine-tuned while the downstream model is trained §2.4.
• Question answering table. The second column header reads "Farbes" [sic]. Its values (0.54, 0.73, 0.83) are fractions under an accuracy-percent header, and the survey does not explain them tab. 8.
• Bibliography keys in the ELMo and BERT tables. For liu, chen, he, lee, peters and mccann, the bibliography entries list given names as surnames (for example chen is labelled "Qian et al. (2017)"). Look up the original paper by title before you cite these rows tab. 10.
• Survey version. The source is arXiv version 8, last updated 2018-11-25. The survey says it will be updated, so later versions may change the text and tables §1, footnote.
• Scores are dated. The "state-of-the-art" labels describe the cited papers at the time of writing. Newer results are not covered.

Lineage

• Builds on: the survey positions itself against an earlier tutorial on neural networks for NLP by Goldberg (2016), which covers the basics §1. That paper is not compiled in this wiki.
• Compared against: the survey does not compare methods with one another beyond the reported benchmark rows §8.
• Superseded by: not stated in this card. Later survey versions are not checked here.