Diachronic word embeddings and semantic shifts: a survey

In plain words

Words change meaning over time, and the change often shows up in the words that appear around them. This survey looks at how researchers use computers to find such changes automatically. The basic trick is to collect text from different time periods, turn each word into a vector that summarises the words around it, and see how much a word's vector moves between periods.

Doing this well is harder than it sounds. Each period's vectors come out of a separate, random training run, so they cannot be compared directly until they are lined up. Researchers also disagree on how to measure change, what counts as a real shift, and whether some proposed "laws" of language change hold up. The paper maps these choices, names the open problems, and lists where the work is applied. It does not propose a new method and does not run experiments.

TL;DR

This is a 2018 survey of methods that trace semantic shifts with diachronic word embeddings. It organises the field by data source, model family, alignment method and comparison measure, and it reviews the "laws" of semantic change. It reports no new experiments. The paper's own list of open challenges covers languages, small data, gold test sets, formal models and shift analysis §7.

Problem

The paper asks how to find words whose meaning differs between time periods, or the words that changed most, given corpora C_1 to C_n from periods 1 to n §3. Shifts are often reflected in a word's co-occurrence context, and they are not always accompanied by a change in frequency, so models of meaning are preferred to frequency counts §3.2. The data is scarce: gold-standard lists of shifted words are hard to build, so evaluation often relies on self-made test sets or manual inspection §3.1.2.

Method

The survey has no single method. It compares existing approaches along four axes, and this card follows that structure.

1. Diachronic data. Choose the source corpus and the time granularity. Earlier work used long spans; newer work uses yearly or finer slices to catch subtle cultural shifts §3.1.1.
2. Model family. Options range from frequency counts §3.2 to sparse count matrices weighted by mutual information, and to Random Indexing, LSA, dense SVD of PPMI matrices and prediction-based models such as skip-gram with negative sampling (SGNS) and CBOW §3.2. The survey says that SVD-factorised PPMI and SGNS are equivalent from a mathematical point of view §3.2.
3. Making periods comparable. Either align independently trained period models with linear transformations §3.3, or avoid alignment: update one model incrementally from period to period §3.3, or train all periods jointly so they share one space §3.3.
4. Comparison measure. Global measures use a word's similarity to the whole vocabulary (second-order embeddings). Local measures compare nearest-neighbour lists §3.3.

The survey also reviews what the results mean: laws of semantic change §4, diachronic semantic relations such as temporal analogies §5, and applications §6.

The overview below is a generic pipeline that combines the steps the survey describes. It is an illustration, not a method given in the paper.

# Generic pipeline assembled from the survey  [§3] [§3.3]
# Input:  corpora, a list of (period, texts) ordered oldest first
#         words, the target words to score
# Output: score per word; higher means a larger change

def trace_semantic_shifts(corpora, words):
    # 1. Learn one vector space per period; with incremental or joint
    #    training, the spaces come out related  [§3.3]
    period_spaces = learn_period_spaces(corpora)
    # 2. Align independently trained spaces into one coordinate system,
    #    e.g. by orthogonal Procrustes  [§3.3]; skip after joint training
    aligned = align_spaces(period_spaces)
    # 3. Score each word by comparing its vectors across periods  [§3.3]
    #    global measures follow regular linguistic drift; local ones
    #    follow cultural shifts  [§3.3]
    scores = {}
    for word in words:
        scores[word] = compare_across_periods(aligned, word)
    return scores

Hyperparameters

The survey gives no training settings, since it reports no experiments §8. The only per-study setting it reports is the time granularity of the cited corpora §3.1.1.

Results

No new results. The paper summarises findings from earlier studies, as reported by the authors of the survey (not re-measured here).

Paper vs code

No official code. The paper is a survey and links no repository. Its only links are to the Google Books Ngrams and COHA home pages, both as data sources §3.1.1. I ran one GitHub repository search for "diachronic word embeddings semantic shift" and it returned only third-party projects, none owned by the four authors, so none was used. No code was read, cited or compared.

Pitfalls

• No runnable method. The survey gives no algorithm or settings. An implementer has to choose the data, model, alignment and measure, and the paper gives no default choice §3.
• Raw vectors are not comparable across models. Embeddings are stochastic and invariant under rotation, so cosine similarity between the same word in two independently trained models is meaningless without alignment §3.3.
• Frequency confounds. Shifts do not always come with frequency change, and the claimed laws may follow from count vectors and frequency effects, not from meaning change §3.2 §4.
• Test sets are scarce. Gold-standard shift lists are rare, even for English. Options named in the survey are word epoch disambiguation, cross-time alignment, event prediction and synthetic merged-word tasks §3.1.2.
• Polysemy. Most surveyed algorithms are not sense-aware, though one method (Biemann et al. 2014) handles senses and classifies the type of shift §3.2 §7.
• Languages. Almost all work is on English. Whether the best methods transfer to other languages has not been shown empirically §7.
• Small data. Historical and digital-humanities corpora are often small, and methods for them are an open need §7.
• Time window. The timeline runs to early 2018, and later work is not covered fig. 1 §8.

Lineage

• Builds on: the earlier diachronic studies the survey cites, from Google Ngrams frequency work §3.1.1 §3.2 to embedding alignment and laws §3.3 §4.
• Compared against: the frequency-based methods in §3.2.
• Superseded by: not known from content on this wiki.