BERT: A Review of Applications in Natural Language Processing and Understanding

In plain words

BERT is a language model from 2018 that reads a sentence in both directions at once. It learns by hiding some words and guessing them from the words around them, using a large pile of unlabeled text. To use it for a task, you add one small output layer and train the whole model on labeled examples, and it often beats older methods. This review collects what happened after BERT appeared: how to adapt it to specialised text such as science or medicine, how it helps pick key sentences from a text, how to score generated text with it, and how to attack classifiers built on it. It also points out where the approach is still weak.

TL;DR

This is a 2021 survey of BERT p. 1. It describes the architecture and the fine-tuning recipe, retraining for specific domains, extractive annotation, the BERTScore metric, adversarial attacks on BERT classifiers, and interlingual and domain-specific variants. Its tables and numbers are taken from the cited papers tab. 1 p. 8 p. 12. Use it as a map of the literature, and check every number against the original paper.

Problem

• Static word vectors such as word2vec [52] and GloVe [64] give each word one vector whatever its context p. 4.
• Recurrent and LSTM models handle long-range context poorly p. 2.
• Left-to-right language models such as GPT [66] cannot use the tokens after the current one p. 2.
• Models built per task need labeled data. For specialised texts that data is expensive, since it needs expert annotators p. 10.
• Text annotation has no accepted automated quality metric, so quality can only be judged by hand p. 5.
• Classic text metrics such as BLEU count n-gram overlap and miss much of the meaning p. 5.
• Text classifiers can be fooled by inputs that look the same to a human but get a wrong label p. 7.

Assumption: the base model reads at most 512 tokens per input p. 3.

Method

BERT is a stack of bidirectional Transformer encoder blocks, pretrained on unlabeled text and then adapted to a task p. 2–3.

• Architecture. The base model has 12 blocks, 12 attention heads and hidden size 768, about 110 million parameters. The large model has 24 blocks, hidden size 1024 and about 340 million parameters p. 2–3.
• Input. One or two text segments. Each sequence begins with a [CLS] token, and a [SEP] token separates segments. A sequence is at most 512 tokens p. 3.
• Pretraining objectives. Masked language modeling predicts randomly masked tokens from context on both sides p. 2 p. 9. The review also describes next sentence prediction (NSP) as a second training task p. 12.
• Adaptation. Add one output layer to the pretrained model. The review distinguishes two uses: feature extraction, where the pretrained weights stay fixed and their output feeds a separate model, and fine-tuning, where all weights are updated on the task p. 2.

Domain retraining. Before fine-tuning, the model can be trained further on text from the target domain. The review names three data options: an intra-task dataset, an intra-subject corpus and a cross-subject corpus p. 3. It reports that retraining helps, and that intra-subject data helps more than cross-subject data p. 3 (cited from [74]). It also notes that further training can cause catastrophic forgetting, which is reduced by a low learning rate p. 3.

Multitask training. The review suggests training on several related tasks at once when they share a subject area p. 3.

Extractive annotation. The review describes using BERT vectors of each sentence, clustering them into K groups, and selecting representative sentences. It cites [53] for this and notes that the number of clusters can be set freely p. 5.

BERTScore. The review describes BERTScore [93] as follows. Each token is embedded in context by BERT, every reference–candidate token pair gets a cosine similarity, and each side keeps its best match p. 6.

# BERTScore, as the review describes it  [p. 6]
# Input:  reference tokens x, candidate tokens x_hat
# Output: F1 score

def bert_score(reference, candidate):
    # Contextual vector for every token in both sentences  [p. 6]
    ref_vectors = bert_embeddings(reference)       # (|x|, d)
    cand_vectors = bert_embeddings(candidate)      # (|x_hat|, d)

    # Cosine similarity for every reference-candidate token pair  [p. 6]
    similarity = cosine_matrix(ref_vectors, cand_vectors)  # (|x|, |x_hat|)

    # Recall: each reference token keeps its best candidate match  [p. 6]
    # Optional IDF weights, smoothed by +1, enter the average  [p. 6]
    best_for_ref = similarity.max(axis=1)          # (|x|,)
    recall = mean(best_for_ref)

    # Precision: each candidate token keeps its best reference match  [p. 6]
    best_for_cand = similarity.max(axis=0)         # (|x_hat|,)
    precision = mean(best_for_cand)

    # F1 is the harmonic mean of precision and recall  [p. 6]
    return 2 * precision * recall / (precision + recall)

The review gives the same scores as equations, with the recall and precision averages written over their own sentences p. 6:

R = \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat x_j \in \hat x} x_i^{\top} \hat x_j p. 6

P = \frac{1}{|\hat x|} \sum_{\hat x_j \in \hat x} \max_{x_i \in x} x_i^{\top} \hat x_j p. 6

F = \frac{2PR}{P + R} p. 6

The review's own figure labels the IDF weighting as optional p. 6.

TextFooler-style attack on text classifiers. The review summarises the black-box attack of [25] (card: /papers/1907.11932) in four stages p. 7–8:

• Rank the words by importance to the model. The review notes that for a white-box model this reduces to a gradient calculation p. 7.
• Replace important words with synonyms. Candidate synonyms come from cosine distance between word embeddings [54], are filtered by part of speech, and must keep the sentence's semantic similarity above a threshold ε, measured with the Universal Sentence Encoder [9] p. 8.
• Keep a candidate only if the model's prediction changes p. 8.
• Repeat with the next most important word when no candidate works p. 8.

An adversarial example must satisfy two conditions: the model's label changes, and the similarity to the original stays at or above ε p. 7.

Interlingual training. The review describes cross-lingual pretraining (XLM [34]) with three objectives: conditional language modeling, masked language modeling, and translation language modeling (TLM). TLM masks words in a pair of parallel sentences and lets the model use either sentence to fill the gap p. 9. TLM needs parallel text. Without it, masked language modeling alone is the fallback p. 10.

Domain models. BioBERT continues BERT's training on PubMed and PMC text, then fine-tunes on biomedical NER, relation extraction and question answering p. 10. SciBERT follows the original BERT training method on a scientific corpus, with a vocabulary built for that corpus p. 10–11. The review gives the reason for the second: the general and scientific vocabularies overlap by only 42% p. 10.

Robustness of training. The review reports that RoBERTa changes four things: much more text (about 160 GB, from four corpora), dynamic instead of static masking, removing NSP, and larger batches p. 11–12.

Hyperparameters

Results

Author-reported. Each number is quoted from the paper the review cites; the review ran no new experiments of its own in these tables. Setup: as given in the review.

Table 1 of the review, from SciBERT [6]. Macro F1 for NER (span), REL and CLS (sentence), PICO (token); micro F1 for ChemProt; LAS and UAS for DEP. Averages over several random seeds tab. 1.

Retraining ablation of RoBERTa, in the review's table with its header labels p. 12. The review's source line for this table is [43].

Paper vs code

No official code. The paper links no repository. A GitHub repository search for "koroteev bert" returned no results. The paper is a survey and describes no method of its own to implement.

Pitfalls

• Second-hand descriptions. Each method is the review's paraphrase of another paper. Implement from the original: BERT (arXiv:1810.04805), BERTScore (arXiv:1904.09675), TextFooler (arXiv:1907.11932, card at /papers/1907.11932), SciBERT (arXiv:1903.10676), RoBERTa (arXiv:1907.11692).
• Scanned tables. The TextFooler table p. 8 is a low-resolution scan, so the BERT values above were read from it. The adversarial-training table on the same page is clearer, and its MR "original" after-attack value (11.5) matches the BERT MR row. Check the BERT row against [25] before relying on it.
• Numbering. Only Table 1 p. 11 is numbered. The other tables and figures are unnumbered, so they are cited by page.
• Table 1 details. The BC5CDR SOTA entry carries footnote 7, which the review does not reproduce. The caption says bold marks SOTA where differences fall within a 95% bootstrap interval; bold marks are not reproduced in the table above p. 11.
• RoBERTa table labels. Row headers say "Our reimplementation", but the source line says [43], the RoBERTa paper. The review does not say whether the first number of each SQuAD pair is EM or F1, and it does not define K for the XLNet rows p. 12.
• BERTScore equations. The review writes the recall sum with index x_i and the maximum over x̂_j p. 6. The equations above use one consistent index. Check the original paper for the exact formulas.
• Attack threshold. The similarity threshold ε is not given a value anywhere in the review p. 7–8. An implementer needs it from [25].
• Long inputs. The review says the base model reads 512 tokens and that longer texts need a special retraining method p. 3–4, but it does not describe the method. It cites [74].
• Hidden costs. BERTScore is about three times slower than SacreBLEU p. 7. Pretraining costs days on specialist hardware p. 10–11.
• Dated results. The review was written in 2021, with references retrieved in 2020. Its comparisons use BERT-Base and BERT-Large, and it does not cover later models.
• Human baselines. On SQuAD the review's BERT score (83.1) is below its human score (89.5) p. 2. On SWAG the BERT score (86.3) is above the human-expert score (85.0) p. 3. These are single benchmarks, so do not read them as a general ranking against humans.

Lineage

• Builds on: BERT [16], arXiv:1810.04805 (no card yet). The Transformer encoder [80] is the base architecture.
• Compared against: ELMo [65], OpenAI GPT [66], CoVe [50], ULMFiT [24], and the earlier models in tab. 1 p. 11.
• Related cards: XLNet [87], card at /papers/1906.08237, which the review presents as combining autoregressive and autoencoder training and beating BERT on several tasks p. 3. TextFooler [25], card at /papers/1907.11932.
• Related, no card yet: RoBERTa [43] (arXiv:1907.11692), SciBERT [6] (arXiv:1903.10676), BERTScore [93] (arXiv:1904.09675), XLM [34] (arXiv:1901.07291).
• Superseded by: not stated in the review.