Word embedding

Overview

A word embedding is a vector of real numbers for each word, learned from text without labels. The working idea is that words used in similar contexts get similar vectors 1708.02709 §2.1. The vectors are usually pre-trained on a large corpus and then used as the first layer of a model for a specific task. The survey credits an early neural model with the first use of pre-trained embeddings 1708.02709 §2.1.

The variants below differ in how the vector is produced. Some predict a word from its neighbours, some factorise co-occurrence counts, some build vectors from character pieces, and some compute a new vector for each token from a language model. Each choice decides what the model can handle, for example words it never saw in training, or the sentence a token sits in. One variant, Sent2Vec, skips the per-word output and gives each sentence a single vector (see below).

Variants

Predictive static vectors: word2vec

CBOW predicts a word from the words around it. Skip-gram predicts the surrounding words from one word 1708.02709 §2.2. The survey describes CBOW with a single context word, which it calls a bigram-style model, and notes that negative sampling draws its negative words by frequency 1708.02709 §2.2. The 2026 paper trains both with 300 dimensions, a minimum count of 5 and negative sampling, as baselines 2609.34138 §2. The method's own paper has no card. Card: Recent Trends in Deep Learning Based Natural Language Processing.

Count-based static vectors: GloVe

GloVe normalises and log-smooths word co-occurrence counts, then factorises them by minimising a reconstruction loss 1708.02709 §2.1. The survey describes it but does not compare it with word2vec on any task. Card: Recent Trends in Deep Learning Based Natural Language Processing.

Subword vectors: FastText

FastText represents a word as a bag of character n-grams inside a skip-gram model. The survey says this helps with words that are not in the vocabulary 1708.02709 §2.3. The 2026 paper reports an author-built variant, "FastText w/ OOV", which builds vectors for missing words with a character-based fallback 2609.34138 §6. Its scores are higher than plain FastText in Urdu, Marathi and Telugu, as the authors report 2609.34138 tab. 2 2609.34138 tab. 3 2609.34138 tab. 4. Card: Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems.

Other static variants

The survey also lists phrase vectors, sentiment-specific vectors trained with a polarity loss, re-training for a single task, and multi-sense vectors learned from parallel text 1708.02709 §2.2. Card: Recent Trends in Deep Learning Based Natural Language Processing.

Sentence vectors from averaged n-grams: Sent2Vec

Sent2Vec gives a whole sentence one vector: the mean of learned vectors for its unigrams and bigrams 1703.02507 §2.1 1703.02507 eq. 2. Each word is predicted from the rest of its sentence with negative sampling. The objective is close to CBOW's, but the context is the whole sentence, not a window 1703.02507 §2.1 1703.02507 §2.3. Bigrams are hashed into a fixed number of buckets 1703.02507 §2.2. Ten negatives are drawn per target 1703.02507 tab. 5. It is listed here because its training objective is close to CBOW's. Its output is one vector per sentence, not per word, so it belongs with text embeddings as much as with this page. Card: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features.

Author-reported results: on six supervised tasks, trained on Toronto books, the unigram+bigram model averages 82.0 and the unigram model 81.4 1703.02507 tab. 1. On STS 2014 and SICK 2014 the unigram+bigram model averages 0.65 Spearman and 0.67 Pearson, and the authors say it matches or beats C-PHRASE 1703.02507 tab. 2. The paper's averages count the MSRP task by accuracy only 1703.02507 tab. 1. Word order is used only through adjacent word pairs 1703.02507 §1.

Contextual vectors: ELMo, OpenAI-GPT, BERT

ELMo gives each token a vector that depends on its sentence. It combines the layers of a bidirectional language model with softmax-normalised, task-specific weights and a task-specific scale 1708.02709 §2.4. OpenAI-GPT pre-trains a Transformer language model. BERT is pre-trained with masked-word prediction and next-sentence prediction 1708.02709 §2.4. The survey does not say whether the language model stays frozen while the task model trains, nor what the BERT masking rate is 1708.02709 §2.4. Card: Recent Trends in Deep Learning Based Natural Language Processing.

Vectors as the input layer of a task model

A task model can take pre-trained static vectors as its first layer. In a short-text classifier, the authors used word2vec vectors (Google News, 300 dimensions) for one dialog dataset and GloVe Twitter vectors (200 dimensions) for two others. They chose each set from the public vectors they tried 1603.03827 §4. The vectors were updated during training with the rest of the model, using SGD with the Adadelta rule 1603.03827 §3.2. The paper treats the vector set as a per-dataset choice, so its scores do not isolate the vectors' effect 1603.03827 §4. Card: Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks.

A sarcasm-detection CNN takes the same kind of input layer. It uses 300-dimensional word2vec vectors (CBOW, Google News) and gives random vectors to words the set lacks. The vectors are trained with the rest of the network, not frozen 1610.08815 §4.1. The authors' stated reasons are that tweet vocabulary is often missing from word2vec and that sarcasm shifts polarity in ways word2vec was not trained for 1610.08815 §4.1. On the first dataset, with all features, the authors report 86.23% F1 with random vectors in place of pre-trained ones, against 97.71% with pre-trained vectors 1610.08815 §5.3 1610.08815 tab. 2. This is one dataset and one comparison. Card: A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks.

Checking vectors: word-similarity test sets

Each test pair gets a human score, and the vectors give a cosine score. The result is Spearman's rho between the two, times 100 2609.34138 §2. The 2026 paper builds six sets of 97 to 163 pairs for Indian languages by translating RG-65 and WordSim-353, then rating each pair on a 0 to 10 scale. Each score is the mean of eight native-speaker ratings 2609.34138 §4.1 2609.34138 §4.2. Three caveats from that paper apply to any such set. The procedure for choosing pairs from the base lists is not stated. A word missing from the vocabulary gets a zero vector, and the cosine with a zero vector is undefined 2609.34138 §2. Small sets give no confidence intervals, and a few pairs can move rho noticeably 2609.34138 §6.

When to use which

Results from the 2026 paper are author-reported on test sets of 97 to 163 pairs, so use them to compare methods on one set, not to rank languages.

Implementation sketch

The lookup, the fallback for missing words and the similarity check are the same for every static variant. Contextual variants replace the lookup with a forward pass through a language model. Where a detail is not stated in the cards, the code marks it.

# Static lookup: one row of a V x d matrix per word  [arxiv:1708.02709 §2.1]
def static_vector(word, table, subword_model=None):
    if word in table:
        return table[word]
    # Subword models (FastText style) build a vector from character
    # n-grams for unseen words  [arxiv:1708.02709 §2.3] [arxiv:2609.34138 §6]
    if subword_model is not None:
        return subword_model(word)
    # Missing word: zero vector. Cosine with it is undefined and the paper
    # does not say what score it gets  [arxiv:2609.34138 §2]
    return zeros(d)

# Contextual: one vector per token from a pre-trained bidirectional LM
# [arxiv:1708.02709 §2.4]
def contextual_vector(tokens, lm_layers, layer_logits, gamma):
    layers = lm_layers(tokens)  # L + 1 layer outputs, one per token
    # Softmax the learned layer weights, take a weighted sum, then scale
    # by gamma, a task-specific hyperparameter  [arxiv:1708.02709 §2.4]
    s = softmax(layer_logits)
    return gamma * sum(s[j] * layers[j] for j in range(len(layers)))

# Word-similarity check: rank correlation of cosines with human scores
# [arxiv:2609.34138 §2]
def similarity_score(pairs, vector_of):
    human, model = [], []
    for word_a, word_b, rating in pairs:
        model.append(cosine(vector_of(word_a), vector_of(word_b)))
        human.append(rating)
    return 100 * spearman(human, model)

# Sent2Vec sentence vector: mean over unigram and bigram vectors
# [arxiv:1703.02507 eq. 2]
def sentence_vector(tokens, vocab, V, bigram_id):
    # The paper does not say how unknown words are handled; this drops them
    # [arxiv:1703.02507 §2.2]
    ids = [vocab[t] for t in tokens if t in vocab]
    # Unigrams plus hashed bigrams; repeats count  [arxiv:1703.02507 §2.2]
    features = ids + [bigram_id(a, b) for a, b in zip(ids, ids[1:])]
    return mean(V[features], axis=0)  # (h,)

Not stated in the cards: the context window size, the number of negative samples, the learning rate, the number of epochs, and the FastText n-gram range 1708.02709 §2.2 2609.34138 §2. For Sent2Vec, the bucket count is not stated 1703.02507 §2.2. Set them from the original method papers, and record your choice. The Sent2Vec paper card also records that the released code's defaults differ from the paper's settings, so pass the paper's values explicitly (Sent2Vec card).

Contradictions

None known between the cards. The survey reports no word-similarity scores, the 2026 paper tests no contextual model, and the short-text classifier reports task accuracy, not word-similarity scores 1603.03827 tab. 4. Sent2Vec is scored on sentence tasks and sentence similarity 1703.02507 tab. 1 tab. 2, so its numbers are not comparable with the word-similarity scores above. One point to read carefully: the survey credits FastText with handling unseen words 1708.02709 §2.3, while in the 2026 paper plain FastText is the lowest of the three static methods in Marathi 2609.34138 tab. 3. These are different languages and tasks, so they do not conflict, but FastText should not be assumed to win on every language.

Papers

• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features: Sent2Vec, a sentence vector as the mean of learned unigram and bigram vectors, trained with a C-BOW-style objective.
• Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks: uses word2vec or GloVe vectors as the input layer of an LSTM or CNN text encoder for dialog-act classification.
• A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks: word2vec vectors, updated during training, as the input layer of sarcasm and sentiment CNNs; reports the effect of random versus pre-trained vectors on one dataset.
• Recent Trends in Deep Learning Based Natural Language Processing: survey of word vectors (word2vec, GloVe, FastText, ELMo, BERT) and other NLP models; runs no experiments of its own.
• Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems: word-similarity test sets for six Indian languages, with CBOW, skip-gram and FastText baselines.