A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models

In plain words

A computer can only work with numbers, so before a machine can sort a piece of text into a category (spam or not, positive or negative, and so on), the text has to become a list of numbers. This survey walks through the ways people have done that. It starts with cleaning the text: removing noise, fixing spelling and standardising words. Then it covers the simple ways of counting words, such as bag of words and TF-IDF, which ignore word order and meaning. Next come word embeddings, which place each word at a point in a space so that words used in similar contexts end up close together. After that come models that give a word a different vector depending on its sentence, and the large pre-trained transformer models built on that idea. The survey also summarises the classifiers that sit on top of these representations and the usual scores for judging them. It runs no experiments of its own: it compares methods by describing them and by summary tables.

TL;DR

• A survey, not a new method: it covers text pre-processing, classical count features, static word embeddings, contextual and transformer language models, classifiers and evaluation metrics §1.
• It reports no experiments and no numbers of its own. Use it as a map of options and a list of known limitations, not as a source of benchmark results §1.
• Several tables and passages disagree with each other or with the paper's own equations. Check every value against the original source before copying it (see Pitfalls).

Problem

Social-media and other low-quality text is noisy and informal, and most classifiers need fixed-length numeric input §1. The paper states its scope as representing text for low-quality text classification, and uses "content", "data" and "text" interchangeably §1. Count-based representations drop word order and semantics, and their vector length grows with the vocabulary §3.1. Static word vectors give one vector per word whatever its context, and they handle unseen words poorly §3.2.1, §3.3. The authors also state that the current state-of-the-art does not perform well on low-quality text, but the paper presents no experiment that shows this §3.2.1.

Method

The survey follows the standard text-classification pipeline: pre-process the text §2, extract features from it, from count-based to learned representations §3, pick a classifier §4 and evaluate it §5. fig. 1 shows this pipeline.

1. Pre-processing §2.1. The paper lists these steps, each with the purpose it gives:

• Tokenisation: split text into sentences and tokens §2.1.
• Noise removal: strip leftover strings and Unicode; replace URLs with a tag; remove user mentions §2.1.
• Word segmentation of hashtags into their words §2.1.
• Replace emoticons and emojis with their meanings §2.1.
• Replace abbreviations and slang with full words §2.1.
• Normalise elongated words (repeated letters) to their base form §2.1.
• Correct spelling mistakes; TextBlob and Norvig's corrector are named §2.1.
• Expand contractions, e.g. "can't" to "cannot" §2.1.
• Remove punctuation, or map "!" and "?" to tags §2.1.
• Remove numbers, after converting numeral slang such as "gr8" to words §2.1.
• Lower-case all words, with the caveat that "US" can be a pronoun §2.1.
• Remove stop-words; NLTK, scikit-learn and spaCy lists are named §2.1.
• Stemming (Porter, Lancaster, Snowball) and lemmatisation §2.1.
• Part-of-speech tagging §2.1.
• Negation handling by a "NEG_" prefix or by antonyms §2.1.

The cited studies disagree on several of these steps §2.2. Bao et al. find that stemming and lemmatisation hurt and that keeping URLs and transforming negations help. Haddi et al. find that stemming and negation prefixing help. Jianqiang and Xiaolin find that removing stop-words, special characters and URLs hurts. Symeonidis et al. find that removing punctuation performs poorly. The survey concludes that no work yet recommends a pre-processing combination §2.2.

2. Count-based features §3.1.

• One-hot encoding: one dimension per vocabulary term, set to 1 for the term present §3.1.
• Bag of words (BoW): the sum of the one-hot vectors of the words in a document; order and grammar are lost §3.1, fig. 2.
• Term frequency (TF): occurrences of a term divided by the number of words in the document §3.1.
• TF-IDF: TF multiplied by the log of the document count over the number of documents containing the term §3.1.

# TF-IDF weight per term and document  [§3.1]
# Input:  docs, a list of token lists; D = len(docs)
# Output: weights[d][t]
def tf_idf(docs):
    # df_t: number of documents that contain term t  [§3.1]
    doc_freq = count_docs_containing_each_term(docs)
    for d, doc in enumerate(docs):
        for term in set(doc):
            # TF: occurrences over document length  [§3.1]
            tf = count(term, doc) / len(doc)
            # IDF as written in the paper: log(D / df_t); log base not stated
            weights[d][term] = tf * log(D / doc_freq[term])
    return weights

3. Learned word vectors §3.2.1. The paper builds on the distributional hypothesis: words that appear in similar contexts have similar meanings.

• Neural network language model (Bengio et al.): predict the next word from the previous n−1 words; the word embeddings are a by-product of training §3.2.1.
• Word2Vec (Mikolov et al.): a shallow network with two training variants. CBOW predicts the centre word from its context; skip-gram predicts the context from the centre word §3.2.1, fig. 3. Negative sampling and hierarchical softmax are the two speed-ups named. The paper says skip-gram suits less data and rarer words, while CBOW is faster and does better on frequent words §3.2.1.
• GloVe: factorises a global word co-occurrence matrix. The paper gives its objective, with a weighting function f that it never defines §3.2.1.
• FastText: based on CBOW but built from character n-grams, so it can give vectors to out-of-vocabulary words §3.2.1.

4. Context-dependent models §3.2.1.

• Context2Vec: a CBOW-like model whose context encoder is a bidirectional LSTM §3.2.1, fig. 4.
• CoVe: a two-layer BiLSTM encoder trained for machine translation, started from GloVe vectors. Its output is combined with GloVe vectors for the downstream task §3.2.1.
• ELMo: a token's vector is a task-specific, softmax-weighted, scaled sum of the layer representations of a bidirectional LM §3.2.1, eq. 1, fig. 5.
• ULMFiT: three steps, general LM pre-training, target LM fine-tuning, then target classifier fine-tuning. It uses discriminative fine-tuning and slanted triangular learning rates, and the pre-trained LM is trained on WikiText-103 §3.2.1, fig. 6.
• Transformer language models §3.2.1, tab. 3, fig. 7, fig. 8: GPT (decoder, left-to-right); BERT (encoder, masked LM and next-sentence prediction); GPT-2; XLNet (permutation LM); RoBERTa (no next-sentence prediction, dynamic masking); ALBERT (factorised embeddings, cross-layer sharing, sentence-order loss); XLM, StructBERT, DistilBERT, MegatronLM, CTRL, ERNIE and ERNIE 2.0, SpanBERT, UniLM, ELECTRA, MASS, T5 and BART. Domain models named are SciBERT, BioBERT, BioALBERT, BERTweet and CT-BERT §3.2.1.
• Sentiment-aware embeddings: hybrid models that change skip-gram or refine pre-trained vectors with sentiment lexicons, e.g. HyRank and Re(*) §3.3.

5. Classifiers §4. Naive Bayes, SVM, logistic regression, decision tree, random forest and deep learning models are compared qualitatively in tab. 4. The deep models covered are RNN, LSTM, GRU and CNN §4.2. The paper gives no training recipe for any of them.

6. Evaluation §5. Accuracy, precision, recall and F1 from the confusion matrix tab. 5. The equations are in the Key equations section.

Key equations

TF-IDF, as written §3.1:

\mathrm{TF\text{-}IDF}(t,d,D) = \mathrm{TF}(t,d)\times\log\left(\frac{D}{df_t}\right)

GloVe objective, as written inline in the text §3.2.1. The weighting function f is not defined in the paper:

J = \sum_{k,j=1}^{V} f(X_{kj})\left(w_k^{\top} w'_j + b_k + b_j - \log X_{kj}\right)

ELMo bidirectional LM log-likelihood, as printed §3.2.1:

\sum_{n=1}^{k}\left(\log p(t_n \mid t_1,\dots,t_{n-1};\Theta_x,\overrightarrow{\Theta}_{LSTM},\Theta_s) + \log p(t_n \mid t_{n+1},\dots,t_n;\Theta_x,\overleftarrow{\Theta}_{LSTM},\Theta_s)\right)

ELMo token representations, as printed: the layer set R_n is the token embedding plus the forward and backward states for layers j=1,\dots,L §3.2.1.

ELMo task vector eq. 1. The TeX source labels this equation eq:2; it is the only numbered equation in the compiled paper, so it is cited as eq. 1:

\mathrm{ELMo}_n^{\text{task}} = \gamma^{\text{task}} \sum_{j=0}^{L} s_j^{\text{task}}\, h_{n,j}^{LM}

Here s^{\text{task}} are softmax-normalised layer weights and \gamma^{\text{task}} is a scaling hyperparameter §3.2.1, eq. 1. The printed text writes the layer index as h_{h,j}; h_{n,j} is used here, as the surrounding definitions imply.

LSTM cell, as printed §4.2:

# LSTM cell, equations as printed in the paper  [§4.2]
# x_t: input at time t; h_{t-1}, C_{t-1}: previous state
def lstm_step(x_t, h_prev, C_prev):
    # input gate  [§4.2]
    i_t = sigmoid(W_i @ [x_t, h_prev] + b_i)
    # candidate memory  [§4.2]
    C_hat = tanh(W_c @ [x_t, h_prev] + b_c)
    # forget gate  [§4.2]
    f_t = sigmoid(W_f @ [x_t, h_prev] + b_f)
    # new memory cell  [§4.2]
    C_t = i_t * C_hat + f_t * C_prev
    # output gate; printed as sigmoid(W_o)[x_t, h_prev] + b_o  [§4.2]
    o_t = sigmoid(W_o @ [x_t, h_prev] + b_o)
    # hidden state  [§4.2]
    h_t = o_t * tanh(C_t)
    return h_t, C_t

GRU, as printed §4.2. The paper gives the update as h_t = z_t h_{t-1} + (1+z_t)\sigma_h(\cdot); see Pitfalls:

z_t = \sigma_g(W_z x_t + U_z h_{t-1} + b_z)

\hat r_t = \sigma_g(W_r x_t + U_r h_{t-1} + b_r)

h_t = z_t h_{t-1} + (1+z_t)\,\sigma_h(W_h x_t + U_h(r_t h_{t-1}) + b_h)

Metrics §5:

\mathrm{Accuracy} = \frac{TP+TN}{TP+FP+FN+TN}

\mathrm{Precision} = \frac{TP}{TP+FP} \qquad \mathrm{Recall} = \frac{TP}{TP+FN}

F_1 = 2\cdot\frac{\mathrm{Recall}\cdot\mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}

Hyperparameters

The survey trains nothing, so these are the values it states for the models it describes. Anything not listed is "not stated" anywhere in the paper.

Results

The paper reports no experiments and no benchmark numbers of its own. Its comparisons are qualitative: they are summary tables and prose that cite other work §1, §2.2, §3.3.

Paper vs code

No official code. The paper has no project page. Its footnotes point to Norvig's spelling-corrector page, a noisy-text workshop site and Google's BERT repository. None of these is the authors' code. A GitHub repository search for "Naseem word representation survey" returned no results. Only one search query was tried, so a repository could still exist that was not found.

Pitfalls

• Survey, no experiments. Every figure of merit in the paper is quoted from other work or stated qualitatively. Do not treat any table here as a benchmark.
• Scope versus content. The stated scope is representation for low-quality text classification §1, but most of the material covers general NLP, and the paper shows no results on low-quality text.
• Architecture disagrees between tables. tab. 1 lists Word2Vec, GloVe and FastText as "Log Bilinear". tab. 3 lists them as "FCNN" (a fully connected network). Decide from the original papers, not from either table.
• GPT-2 release date. tab. 3 gives Jun-18, the same as GPT. The text says GPT-2 was released in 2019 §3.2.1.
• OOV handling is described two ways. §3.2.1 says Word2Vec and GloVe give an out-of-vocabulary word a random vector. §3.3 says they map unknown words to a UNK token with one shared vector. The two descriptions differ, and neither is sourced.
• ELMo combination. The text says ELMo uses the linear concatenation of the biLM layers §3.2.1. eq. 1 is a softmax-weighted sum, scaled by γ. Implement the equation; the sum is what is written.
• Confusion matrix swap. In tab. 5, the false-negative and false-positive cells sit in the wrong places for its own axis labels. The cell for predicted positive and actual negative holds FN, and the cell for predicted negative and actual positive holds FP. The bullet definitions in §5 are correct: FP is a wrongly predicted positive and FN a wrongly predicted negative.
• Typesetting errors in equations. The GRU update is printed as z_t h_{t-1} + (1+z_t)\sigma_h(\cdot), a form that does not match the standard GRU §4.2. Take GRU and LSTM equations from the original papers, which the survey cites but does not reproduce correctly. The LSTM output gate has a misplaced parenthesis in print §4.2. The BiLM sum's backward term ends at t_n, and its upper limit is k §3.2.1. The ELMo index h_{h,j} is a typo for h_{n,j} eq. 1. The GRU reset gate is written \hat r_t but used as r_t §4.2.
• GloVe objective is incomplete. The weighting function f is never defined, and the text defines X_k and P_{kj} without using them in the objective §3.2.1. Use the weighting from the original GloVe paper.
• TF-IDF is one variant. The paper gives \log(D/df_t) with no smoothing and no log base §3.1. Other TF-IDF variants exist; use the one your library implements and say which.
• BERT's masking policy is partly stated. The 15% rate is given §3.2.1, but the paper does not describe how the selected tokens are replaced, so the replacement policy is not stated.
• Unsourced figures. The RoBERTa claim of "fewer parameters" than BERT §3.2.1 has no parameter count in the paper. The ALBERT, DistilBERT and MegatronLM ratios have no configuration or table. Table 3's corpus column, e.g. ELMo on WikiText-103, is not sourced in the text. Check each against its original paper.
• Qualitative pros and cons. tab. 1 and tab. 4 state pros and cons, for example that Naive Bayes "can easily be trained less data" tab. 4. These are not backed by experiments in the paper.

Lineage

• Builds on: the classical count-based methods (one-hot, BoW, TF, TF-IDF) §3.1; the distributional-semantics line from Bengio et al. §3.2.1.
• Surveys (no implementation): Word2Vec, GloVe, FastText, Context2Vec, CoVe, ELMo, ULMFiT, GPT, BERT and its variants, XLNet, RoBERTa, ALBERT, T5, BART and others listed in Method §3.2.1.
• Compared against: none; the paper is a comparison of methods, not a benchmark.
• Superseded by: not stated.