Data Augmentation Approaches in Natural Language Processing: A Survey

In plain words

Data augmentation means making more training examples from the ones you already have, by changing copies of them or by inventing new ones. It works well for pictures, where simple operations such as flipping or rotating an image leave its content recognisable. Text is harder, because words are discrete, and a change to a sentence can change its meaning or its label.

This survey sorts the many text methods into three groups by how different the new text is from the original. The first group makes small rewordings, using synonyms, translations or a language model. The second adds disturbance such as swapped, deleted or inserted words. The third creates new text from rules or models that have learned what the data looks like, which gives the most variety but needs more care to keep the label right. The survey also covers how to combine methods, how to set their knobs, how to filter bad output, and which kinds of task use which kind of method. It is a review, so it does not rank the methods on one benchmark, and it says results are hard to compare across tasks.

TL;DR

A survey of data augmentation (DA) for natural language processing. The authors group methods into paraphrasing, noising and sampling, ordered by how much new diversity the augmented data adds §2. The survey describes 15 method families with their advantages and limits, a table of their properties tab. 1, tips on stacking, optimisation and filtering §3, and how DA is used across text classification, text generation and structured prediction §4. It reports no experiments of its own, so no benchmark numbers come from it.

Problem

Deep learning can fail when labelled data is scarce, and DA is one remedy §1. DA is widely applied in computer vision, for example with flipping and rotation, and was then introduced to NLP §1. Natural language is discrete, which makes DA harder and less explored there §1. The authors say earlier surveys divided methods by method type, which they find too limited or general, and that one covered only text classification §1. The survey's stated aim is to explain why DA works, and to organise methods by the diversity of the augmented data §1.

Augmented data must stay valid for the task. In the authors' words, it should belong to the same distribution as the original, for example keeping the same meaning in translation or the same label in classification §2. Diversity is then the second requirement, since it is what helps a model generalise §2.

Method

This is a survey, so the "method" is the taxonomy and the per-method descriptions. The three categories are defined in §2 and illustrated in fig. 2. The full taxonomy is fig. 1.

• Paraphrasing: limited semantic change, based on restrained edits §2. Levels are word, phrase and sentence fig. 3.
• Noising: adds weak discrete or continuous noise that should not change the meaning for a human reader. It aims at robustness §2 §2.2.
• Sampling: learns or encodes the data distribution and draws new data from it. Methods are task-specific and need labels and format §2.3.

Paraphrasing §2.1

• Thesauruses §2.1.1 fig. 4: replace words by synonyms or hypernyms from a resource such as WordNet. Zhang et al. pick r words with a geometric distribution and a synonym index from a second one. EDA replaces n non-stop words with random synonyms, where n grows with sentence length. Coulombe et al. also use hypernyms and order parts of speech by difficulty.
• Semantic embeddings §2.1.2 fig. 5: replace a word with a neighbour in a pretrained embedding space (GloVe, Word2Vec, FastText). Wang et al. use k-nearest neighbours by cosine similarity and also substitute semantic frames.
• Language models §2.1.3 fig. 6: a masked LM (BERT, RoBERTa) predicts words in masked positions from context. Jiao et al. replace each word piece with probability 0.4 and fill whole-word masks with BERT predictions.
• Rules §2.1.4 fig. 7: regular expressions for contractions such as "is not" to "isn't", word-pair dictionaries, and dependency-tree rewrites (for example active to passive).
• Machine translation §2.1.5 fig. 8: back-translation translates to another language and back; unidirectional translation translates once into several languages for multilingual data. Cloud translation services are common. Variants add temperature settings (Nugent et al.), adversarial training (Qu et al.) or a discriminator filter (Zhang et al.) §3.2.4 §3.3.
• Model generation §2.1.6 fig. 9: Seq2Seq, Transformer and GAN models generate paraphrases. Hou et al. feed a diversity rank k into a Seq2Seq model. Li et al. mask half of the words that carry a non-O sequence label and reconstruct them §2.1.6, footnote.

Noising §2.2 fig. 10

• Swapping §2.2.1: swap random word pairs, repeated n times with n proportional to sentence length (EDA). Sentence-level and half-tweet swaps are also used.
• Deletion §2.2.2: delete each word with probability p (EDA). Sentence deletion removes statements from legal documents.
• Insertion §2.2.3: insert synonyms of random non-stop words at random positions (EDA). Sentence insertion borrows sentences with the same label.
• Substitution §2.2.4: replace words with common misspellings, a placeholder, a vocabulary word chosen by TF-IDF or unigram frequency, or a word from another language (code-switching).

Sampling §2.3 fig. 11

• Rules §2.3.1: build new examples with heuristics. Examples include subject-object swaps with a rule-assigned label (Min et al.), equation reuse for math word problems (Liu et al.), and entailment chains for NLI (Kang et al.).
• Non-pretrained models §2.3.2: train a target-to-source Seq2Seq model and use it to make pseudo-parallel source text from target text. This is the sampling form of back-translation, which the survey distinguishes from the paraphrasing form §2.3.2, footnote.
• Pretrained models §2.3.3: generate labelled sentences with GPT-2 (LAMBDA, Tavor et al., with a classifier filter; Kumar et al., without one), corrupt-then-reconstruct with a masked LM (Ng et al.), or condition GPT-2 on the label (Quteineh et al.).
• Self-training §2.3.4: label unlabelled text with a fine-tuned model. SentAugment (Du et al.) retrieves web sentences that are close to task queries built from labelled data.
• Mixup §2.3.5: interpolate in a vector space instead of producing text. Three equations give the word-level, sentence-level and label mixtures eq. 1 eq. 2 eq. 3. The labels can then differ from the originals.

# Mixup for sentence classification  [§2.3.5] [eq. 1] [eq. 2] [eq. 3]
# Inputs: word matrices (N, d) or sentence states (d,); labels; lam in [0, 1]
# Output: mixed input and mixed label

def mix_pair(x_i, x_j, y_i, y_j, lam):
    # Word-level variant: interpolate the word vectors  [eq. 1]
    mixed_words = lam * x_i + (1 - lam) * x_j
    # Sentence-level variant: same rule on encoder states f(x)  [eq. 2]
    # Label mixture for either variant  [eq. 3]
    mixed_label = lam * y_i + (1 - lam) * y_j
    return mixed_words, mixed_label

# EDA-style random swap  [§2.2.1]
# Input: tokens (list of words), n (swap count, grows with length)
# Output: tokens with n random pairs exchanged

def random_swap(tokens, n):
    # Repeat n times: pick two positions and exchange their words  [§2.2.1]
    for _ in range(n):
        i, j = two_random_positions(len(tokens))
        tokens[i], tokens[j] = tokens[j], tokens[i]
    return tokens

Properties of the methods §2.4 tab. 1

tab. 1 classifies each family by whether it is learnable, whether the augmentation runs online or offline, whether it needs external knowledge or a pretrained model, whether it uses the task, and at what level and granularity it acts. Points the survey draws from it §2.4:

• Nearly all paraphrasing and noising methods are not learnable, except model generation and Mixup. Sampling methods are mostly learnable, except rules.
• Mixup is the only online method. It is the only one whose output is an embedding with a mixed label.
• Most non-learnable methods depend on external resources such as WordNet or PPDB.
• Paraphrasing and noising, except model generation, ignore labels and task definition. All sampling methods use them.
• Learnable methods only work at sentence level. The survey says document augmentation still relies on simple non-learnable methods.

Hyperparameters

The survey lists the knobs for each family §3.2.2 fig. 12. It states no default values, so most cells below are "not stated".

Results

Not applicable. This is a survey with no experiments of its own. tab. 2 counts the papers that apply each DA family to each NLP task, and tab. 3 gives a timeline of such applications. Those tables are counts, not measured performance, and this card does not transcribe them.

Paper vs code

No official code. The paper links a resource list of DA papers (BohanLi0110/NLP-DA-Papers) in its introduction §1, footnote, and its appendix lists third-party libraries and repositories such as EDA, UDA and nlpaug app. A. These are a paper collection and other authors' tools, not the code of this survey, so the card has no code field.

Pitfalls

• Categories follow diversity, not method family. One technique can sit in two places. Back-translation is a paraphrasing method §2.1.5, but non-pretrained back-translation, which trains a target-to-source model, is a sampling method §2.3.2, footnote. Check which one a source means.
• Noise can change the label. Random insertion and substitution may flip the label. The survey's tips are label-independent resources at word level, and same-label samples at sentence level §2.2.3 §2.2.4.
• Sampling needs the task. Sampling methods use labels and task format §2.3 tab. 1. A label-free pipeline can only use paraphrasing or noising.
• Most knobs are unstated. Probabilities, counts and the Mixup weight are not given numeric defaults in the survey fig. 12 §2.3.5. Any value you pick is your choice, not the authors'.
• Mixup is online and mixes labels. Its output is an embedding, not text, and the survey names poor interpretability as a drawback §2.3.5 tab. 1.
• Filters are a separate step. Some pipelines filter output with a classifier or discriminator §3.3. The survey gives no thresholds.
• Learnable methods and documents. Learnable methods work only at sentence level, and document augmentation still relies on simple methods §2.4.
• Task comparisons are not direct. The survey says different tasks, metrics, datasets and models make direct comparisons hard §4.
• Table classifications are judgements. tab. 1 is the authors' assignment; for example, model generation is marked learnable and offline.
• Version. The card covers arXiv v3, updated 2022-06-27. Later work is not covered.

Lineage

• Builds on: none. It is an overview of earlier DA methods.
• Compared against: the surveys of Liu et al. and Feng et al. (grouped by method), and Bayer et al. (text classification only) §1. Computer-vision DA is cited as background §1.
• Superseded by: not stated.