Natural Language Processing Advancements By Deep Learning: A Survey

In plain words

Natural language processing (NLP) is the part of computing that tries to get machines to read, write and understand human language. This survey looks at how deep learning, which trains large neural networks on big datasets instead of relying on hand-written rules, has changed that field. It is a guide to the ideas and the literature, not a new technique you can build.

It starts with the basics: how words and letters become numbers a model can use, how one model can turn one sequence of text into another, and why training such models is tricky. Then it goes task by task: tagging the grammatical role of each word, parsing sentences, sorting text into categories, finding names and relations, translating between languages, answering questions, summarising documents and holding conversations.

For each task it names the models that were prominent and the public datasets they are tested on. Its score tables are copied from earlier papers that used different setups, so they are not a like-for-like ranking.

TL;DR

A survey, not a method. It maps deep learning across NLP: representation learning §3.1, sequence-to-sequence models and their training problems §3.2–3.3, benchmark datasets §4, and models per task §5. It gives a few equations and no hyperparameters or code, so use it to orient yourself and find the original papers. Its score tables mix metric labels and are not from the survey's own experiments.

Problem

The authors say earlier reviews mostly cover architectures with little on applications, or a single sub-field §1. This survey aims to cover tasks and applications together, with the challenges and how the impact is evaluated §1. Traditional NLP methods hand-craft features §2.2. Most deployed dialogue systems still depend on hand-made features §5.8. Sequence models trained with teacher forcing and cross-entropy face exposure bias, and their training objective differs from the non-differentiable metrics used at test time §3.3.

Method

The survey has no algorithm of its own. It organises the literature in six sections §1 and states the threads below. Equation and table numbers follow the compiled paper.

1. Background. Deep learning is described as applying deep neural networks to large data to learn a procedure for a task §2.1.1. Architectures summarised: multilayer perceptrons, CNNs, RNNs with LSTM cells, autoencoders and GANs §2.1.2 fig. 1–5. The motivation is that simple bag-of-words models already work reasonably well, and that neural language models address the curse of dimensionality §2.2.
2. Representations §3.1. One-hot vectors are very sparse and carry no similarity between words §3.1.1. CBOW predicts a word from its context, and a weighted variant uses TF-IDF weights §3.1.2. Learned word embeddings generalise better than sparse vectors; pre-trained vectors are preferred for rarer words §3.1.3. Character-level CNN embeddings handle out-of-vocabulary words §3.1.4. Paragraph vectors and skip-thought extend this to sentences and documents §3.1 fig. 6.
3. Sequence-to-sequence §3.2. An encoder reads the input and a decoder writes the output, usually with RNN or LSTM cells. Training uses teacher forcing and the cross-entropy loss eq. 1. At inference the model picks the most probable token at each step eq. 2, or uses beam search §3.2. Evaluation uses BLEU, ROUGE and METEOR §3.2.
4. Reinforcement learning §3.3. Exposure bias: training feeds gold tokens, testing feeds the model's own outputs. Scheduled sampling replaces gold inputs with model samples over time. Policy-gradient and actor-critic methods can optimise a non-differentiable metric. The survey argues this is hard for NLP because the action space is the vocabulary (50K to 150K tokens), against fewer than 20 actions on average in Atari §3.3.
5. Datasets §4 tab. 1. Benchmarks are real, synthetic or toy data. The survey lists common datasets per task and notes that creating a dataset requires defining the problem, splitting train and test, preparing data, and settling access rights §4.
6. Tasks §5. Summaries of model families for each task are in the Results table below, with the task-specific details in the Pitfalls. Key points:
   • POS and parsing §5.1: CharWNN combines word and character CNNs; BiLSTM-CRF sequence taggers; a globally normalised transition-based network; RNN grammars and self-attentive encoders for constituency; biaffine and stack-LSTM parsers for dependencies.
   • Semantic role labeling §5.1.3: syntax-aware models (LISA), joint predicate-argument prediction, contextual representations.
   • Text classification §5.2: CNN on pre-trained word vectors, dynamic k-max pooling CNN, LSTM with sentence embeddings, hierarchical attention network, combined RNN/CNN; ULMFiT is the best result in its table tab. 5.
   • Information extraction §5.3: NER with CNN, RNN and BiLSTM-CNN; relation extraction with recursive and tree-structured LSTMs; coreference with RL and span prediction; event extraction with graph CNNs.
   • Sentiment §5.4: document, sentence and aspect levels, with aspect extraction and aspect classification handled as separate sub-tasks.
   • Machine translation §5.5: an RNN translation model that maps a source sentence to a distribution over target sentences with convolutional layers eq. 3–7; the LSTM encoder-decoder that reverses source word order eq. 8; additive attention §5.5.2; Transformer and back-translation results tab. 7.
   • Question answering §5.6: rule-based systems, then a CNN question-answer matcher with a tensor layer eq. 9, a dynamic memory network with episodic memory, a dynamic coattention network, and visual QA.
   • Summarisation §5.7: extractive ranking models; abstractive seq2seq with attention and a pointer-generator; a Fast-RL hybrid that trains an extractor and an abstractor with actor-critic Q-learning. The survey says ROUGE misses good paraphrases.
   • Dialogue §5.8: task-based systems as an NLU, dialogue manager and NLG pipeline, or as end-to-end models; non-task chatbots as retrieval or generative models.

Key equations

Cross-entropy loss for teacher-forced training §3.2 eq. 1:

\mathcal{L}_{CE} = -\sum_{t=1}^{L} \log p_{\theta}(y_{t} \mid y_{t-1}, s_t, X)

Greedy next-token rule at inference §3.2 eq. 2:

\hat{y}_{t} = \arg\max_{y} \, p_{\theta}(y \mid \hat{y}_{t-1}, s_{t})

Recurrent continuous translation model: sentence probability as a product of per-word factors eq. 3; the sentence encoder eq. 4; the recurrent update eq. 5, 6; the output layer eq. 7:

P(f \mid e) = \prod_{i=1}^{m} P(f_i \mid f_{1:i-1}, e)

s = \mathbf{S} \cdot \mathrm{csm}(e)

h_1 = \sigma(\mathbf{I} \cdot v(f_1) + s)

h_{i+1} = \sigma(\mathbf{R} \cdot h_i + \mathbf{I} \cdot v(f_{i+1}) + s)

o_{i+1} = \mathbf{O} \cdot h_i

LSTM sequence-to-sequence objective, to be maximised eq. 8:

\frac{1}{|\mathcal{D}|} \sum_{(\eta, \zeta) \in \mathcal{D}} \log P(\eta \mid \zeta)

Attention in NMT, not numbered in the paper §5.5.2. Each context vector is a weighted sum of encoder states, with weights from a softmax over alignment scores:

c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j, \qquad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}, \qquad e_{ij} = a(s_{i-1}, h_j)

Matching score for a question-answer pair in the CNN matcher eq. 9. The paper says f is an element-wise non-linearity, M is n_s × n_s × r, V is r × 2n_s, and b, u are in R^r:

s(q,a) = \mathbf{u}^{T} f\left(\mathbf{v}_q^{T} \mathbf{M}^{[1:r]} \mathbf{v}_a + \mathbf{V} \begin{bmatrix} \mathbf{v}_q \\ \mathbf{v}_a \end{bmatrix} + \mathbf{b}\right)

Pseudocode

Teacher-forced training loss, from the description in §3.2 and eq. 1:

# Seq2seq training with teacher forcing  [§3.2] [eq. 1]
# Input:  source tokens X; gold target tokens y_1..y_L
# Output: scalar cross-entropy loss

def training_loss(source_tokens, target_tokens):
    # Encode the whole source sequence into hidden states  [§3.2]
    encoder_states = encoder(source_tokens)
    loss = 0.0
    # Start symbol: not stated in the paper  [§3.2]
    previous_token = START_SYMBOL
    for step in range(len(target_tokens)):
        # Condition on the gold previous token, not the model output  [§3.2]
        probabilities = decoder(previous_token, encoder_states)
        # Add the negative log-probability of the gold token  [eq. 1]
        loss = loss - log(probabilities[target_tokens[step]])
        previous_token = target_tokens[step]
    return loss

Greedy generation at test time, from §3.2, eq. 2 and §5.5.2:

# Greedy generation at test time  [§3.2] [eq. 2] [§5.5.2]
# Input:  source tokens X; maximum output length (not stated)
# Output: predicted token sequence

def greedy_decode(source_tokens, max_length):
    # Encode once; the decoder reuses these states  [§3.2]
    encoder_states = encoder(source_tokens)
    outputs = []
    # Start symbol: not stated in the paper  [§3.2]
    previous_token = START_SYMBOL
    for step in range(max_length):
        probabilities = decoder(previous_token, encoder_states)
        # Keep the most likely token; beam search keeps K  [eq. 2] [§3.2]
        previous_token = argmax(probabilities)
        # Stop at the end-of-sentence token  [§5.5.2]
        if previous_token == END_SYMBOL:
            break
        outputs.append(previous_token)
    return outputs

Additive attention for one decoder step, as the survey describes it §5.5.2:

# Additive attention for one decoder step  [§5.5.2]
# Input:  previous decoder state s_prev; encoder states h_1..h_T
# Output: context vector for this step

def attention_context(s_prev, encoder_states):
    # Each h_j joins forward and backward RNN states  [§5.5.2]
    # Score each source position with a feed-forward alignment model  [§5.5.2]
    scores = [alignment(s_prev, h_j) for h_j in encoder_states]
    # Normalise scores into weights that sum to one  [§5.5.2]
    weights = softmax(scores)
    # Weighted sum of the encoder states  [§5.5.2]
    return sum(w_j * h_j for w_j, h_j in zip(weights, encoder_states))

Hyperparameters

The survey defines no model and trains nothing, so it states no training hyperparameters. The only numeric settings in the text are these:

Results

Author-reported, but not by this paper: the survey copies numbers from the works it cites. It runs no experiments of its own. The setup is the dataset and metric named in each table caption; model sizes, training budgets and seeds are not stated. The table lists the top entries of each comparison, in the order the survey gives them. Scores are from the cited papers, not reproduced here.

Other reported figures, with no table: the recurrent continuous translation model reports perplexity 43% lower than the alignment-based baseline, with the dataset not named §5.5.2. QA and dialogue have no numeric comparison §5.6, §5.8.

Paper vs code

No official code. The paper has no code link in its text or footnotes. The arXiv metadata in the repository has no comments field, and no project page is linked. A GitHub repository search for the exact title returned zero results. The survey names a few third-party code releases (for example for BLOCK §5.6.3); those belong to other papers and are not used here. There is no method name to search for. raw/src/2003.01200 holds the LaTeX source; no repository was cloned.

Pitfalls

• Survey numbers are not controlled comparisons. The tables collect results from different papers, datasets and training budgets, and the survey does not state splits, seeds or hardware. Do not rank methods from these tables alone.
• Metric labels are inconsistent. The column headers of the POS, constituency, SRL, classification, NER and MT tables all say "Accuracy" tab. 2–7, but the NER caption says F1 tab. 6, the SRL header says F1 tab. 4, and the MT caption says BLEU tab. 7. The constituency table names no metric tab. 3. Check the original paper before quoting a score.
• NER text and table disagree. §5.3.1 gives an F-score of 93.5 on CoNLL-2003 for Baevski et al. §5.3.1. That score is not in the NER table, which lists 93.09 as the top entry tab. 6.
• A row without a paper. The DeepL row in the MT table has a footnote to a company press page, not a paper, so its setup is unknown tab. 7.
• Figure caption does not match the text. Figure 2 is captioned as a CNN for object detection, though the surrounding section discusses NLP CNNs fig. 2. Rely on the text, not the figure caption.
• Teacher-forced objective and inference are not fully specified. The start symbol, the end-of-sentence id and the beam width K are not given §3.2, §5.5.2.
• RCTM equations leave symbols undefined. The paper does not expand csm eq. 4, does not give the sizes of S, I, R and O eq. 4–7, and writes the output layer as o_{i+1} = O·h_i, with mismatched indices eq. 7. Take the exact form from the RCTM paper it cites §5.5.2 only after checking it there.
• The MT objective is a maximum-likelihood average over training pairs, written without a minus sign eq. 8. Maximise it, or minimise its negative, and state which you do.
• The RL section lumps methods together. Policy gradient, REINFORCE, actor-critic, "value-based iteration" and Q-learning appear in one list §3.3. Look up each method's own definition rather than this list.
• Dates and links. Many cited results are from 2017 to 2020, and the paper says its references may be dated §6. The dataset links in the table are not checked here tab. 1.
• Several claims are one line each. Some model and dataset details are a single sentence with no equations, so there is not enough to reimplement them from this paper.

Lineage

• Builds on earlier surveys, which the paper cites: Young et al. 2018 (architectures, few applications), Kang 2020 and Wu 2020 (application-specific), and Li et al. 2020 (NER) §1 §5.3.1.
• Compared against: none. The survey compares only by putting together scores from other papers §5.
• Superseded by: not known from content in this wiki.