Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

In plain words

Language models that are first trained on huge amounts of raw text, such as BERT, GPT and T5, now sit under most NLP systems. This survey explains the three main ways people use them. The first is to train the model on raw text and then adjust it briefly on a specific task. The second is to phrase the task as a sentence with a blank or a question, so the model can answer it the way it was trained to continue text. The third is to turn every task, including labelling and classification, into producing text.

The survey also covers a side use: asking these models to write extra training examples or explanations for other models. It ends with open questions, such as how much data these models really need and whether the prompts that work so well carry real instructions or just patterns the model has already learned.

TL;DR

• A survey of 2021 that covers how large pre-trained transformer language models (BERT, GPT, T5 and similar) are used for NLP tasks §1.
• It organizes the work into three paradigms: pre-train then fine-tune §2, prompt-based learning §3, and NLP as text generation §4, plus using these models to generate data §5.
• It runs no experiments and has no code. Use it as a map of the literature and for its discussion of open questions §6.

Problem

Before large pre-trained models, each NLP task needed its own feature design or its own model trained from scratch on labelled data §1. The survey argues that pre-training one model on a generic language task and reusing it across tasks changed this. It asks how such models are best applied to non-generative tasks such as classification, sequence labelling and structure prediction, and it explicitly scopes out generative tasks unless noted §1.

Method

This is a survey, so the "method" is the taxonomy it proposes and the techniques it describes in each paradigm.

Model classes. Three classes of pre-trained language models: autoregressive (next-token prediction, e.g. GPT), masked (predict hidden tokens from both sides, e.g. BERT), and encoder-decoder (e.g. BART, T5). Each class is defined by its training objective and architecture part §2.2 fig. 1. The objectives are stated in prose, not as numbered equations:

\sum_i \log P(x_i \mid x_1, \dots, x_{i-1}; \theta_T) §2.2, autoregressive

\sum_i m_i \log P(x_i \mid x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n; \theta_T) §2.2, masked; m_i = 1 if token i is masked

\log P(y_1, \dots, y_n \mid x_1, \dots, x_m; \theta_T) §2.2, encoder-decoder

Pre-training corpora. Data size and source differ greatly between models; see the table under Hyperparameters tab. 1. The survey notes a trend toward larger and more varied corpora, and that cleaning crawled web text helps §2.3.

Paradigm 1: pre-train then fine-tune §2. Four options, in order of how much of the model is changed §2.4 fig. 2:

1. Contextual embeddings: keep the model frozen and use its outputs as features for a new model trained from scratch §2.4.1.
2. Fine-tune the PLM with one or two output (prediction) layers on top, trained end to end. For classification the survey describes using the representation of the special first token §2.4.2.
3. Fine-tune the PLM inside a larger custom architecture, for example a CRF or a parsing algorithm on top §2.4.3.
4. Efficient fine-tuning: adapters (small new layers inside each Transformer block, with the PLM frozen) or tuning only a few selected weights §2.4.4.

The survey describes it as common practice, citing prior work, to keep the PLM layers frozen at first while the new layers train, then unfreeze the PLM layers gradually §2.4.2.

# Staged fine-tuning of a pre-trained language model  [§2.4.2]
# Input:  plm (pre-trained layers), head (new, randomly initialised)
# Output: plm and head adapted to the task
def staged_fine_tune(plm, head, train_data):
    # Stage 1: keep the PLM frozen while the new head learns  [§2.4.2]
    train(head, frozen=plm, data=train_data)
    # Stage 2: unfreeze PLM layers gradually with a low learning rate;
    # the survey gives 1e-3 to 1e-5 as the range  [§2.4.2]
    for layer in plm.layers:  # order of unfreezing not stated  [§2.4.2]
        unfreeze(layer)
        train(head, plm, data=train_data)
    return plm, head

Paradigm 2: prompt-based learning §3. A prompt is natural-language text added to the input or output to get a pre-trained model to perform a task §3. The survey lists three sub-approaches fig. 3:

1. Learning from instructions and demonstrations: the model gets a task instruction and possibly a few examples in its input, usually with no parameter update §3.1 tab. 2.
2. Template-based learning §3.2 tab. 3. Templates can be written by hand, found automatically as discrete text (mining, paraphrasing, gradient-guided search), or learned as continuous vectors. Prefix tuning prepends task-specific continuous vectors and keeps the model frozen; it tunes about 0.1% of the total parameters §3.2. The survey also covers answer generation and task-specific tuning for templates §3.2 tab. 4.
3. Learning from proxy tasks: recast the task as a related one the model already handles, such as question answering or textual entailment §3.3.

Paradigm 3: NLP as text generation §4. Reformulate a non-generative task so a generative model (GPT, BART, T5) writes the answer as text §4 fig. 4. The survey groups this work into: generating label-augmented text §4.1, generating word indices §4.2, generating answers §4.3, filling templates §4.4, generating structure-linearized text §4.5, and ranking input-output pairs §4.6.

Data generation §5. PLMs generate extra data for training in two ways: (a) generate labelled or silver-labelled examples that are added to the training set, with filtering of noise §5.1; (b) generate auxiliary data such as counterexamples, clarifying questions, contexts or inference rules, which explain or support a model §5.2.

Cross-paradigm view §6. The paradigms are not exclusive. Fine-tuning is often combined with prompting, and generation models often use prompts §6. One cited system combines all three in a single text-generation QA setup §6.

Hyperparameters

This survey runs no training of its own. The values below are the ones it states in the text or in its tables; none is a recommendation from the authors.

Pre-training sources and sizes, as listed in the survey tab. 1:

Source for all rows: tab. 1. The survey notes that data sources differ and are described in each model's own citation tab. 1.

Results

Author-reported, from cited works; the survey itself runs no experiments. Setup is given per row. "This paper" is not applicable.

Paper vs code

No official code. The LaTeX source links no repository, project page or code (its only URLs are two tutorials on the Transformer §2.2). A GitHub repository search for the exact title returned no results. As a survey it has no method to implement; any awesome-style paper list linked from a survey is a paper index, not code.

Pitfalls

• The survey covers non-generative tasks; generative tasks are included only where the text notes them §1. Do not read it as a complete review of generation.
• The corpus table compares data in different units: tokens, gigabytes and documents. Rows are not directly comparable tab. 1.
• The survey states that prompts may work without meaningful instructions: irrelevant templates performed about as well in one cited few-shot entailment study §6. Do not assume a prompt's wording is what makes it work.
• Staged fine-tuning (freeze, then unfreeze) is presented as common practice, not as a fixed recipe. The order in which layers are unfrozen and the number of stages are not stated §2.4.2.
• Fine-tuning with a new head is most suitable for sequence classification, tagging and span extraction §2.4.2. The survey notes that for complex tasks (for example parsing), frozen embeddings are often used because of compute and data limits §2.4.1.
• Efficient methods (adapters, prefixes) change few weights but are not free of cost: the survey cites that adapters add weights at every layer §2.4.4.
• Results and numbers are quoted from the cited works and were not reproduced here.

Lineage

• Builds on: the Transformer architecture §2.2, which the models in this survey are based on. Not compiled here.
• Compared against: not applicable; the survey compares paradigms qualitatively.
• Superseded by: not stated.