Semantic change detection

Overview

The problem is to find words whose meaning differs between periods, or to rank words by how much they changed, given one corpus per period 1806.03537 §3. The shared core is to learn a vector space per period, then compare each word's vectors across periods. The invariant every variant must handle is that vectors from independent training runs live in different coordinate systems, so a raw cosine between them means nothing until they are aligned or trained together 1806.03537 §3.3.

Variants

The survey organises the field along four axes. A method is a choice on each.

Model family

Options run from frequency counts, through sparse count matrices weighted by mutual information and dense SVD of PPMI matrices, to prediction-based skip-gram with negative sampling (SGNS) and CBOW 1806.03537 §3.2. The survey reports that frequency methods detect shifts less precisely than distributional models 1806.03537 §3.2. Card: Diachronic word embeddings and semantic shifts: a survey.

Making periods comparable

• Alignment: train each period separately, then map them into one space with linear transformations 1806.03537 §3.3.
• Incremental training: update one model from one period to the next, so no alignment step is needed 1806.03537 §3.3.
• Joint training: train all periods together so they share one space 1806.03537 §3.3.

The survey lists these three options but does not rank them. Card: Diachronic word embeddings and semantic shifts: a survey.

Comparison measure

• Global measures compare a word's similarity to the whole vocabulary, using second-order embeddings 1806.03537 §3.3.
• Local measures compare nearest-neighbour lists 1806.03537 §3.3.

Card: Diachronic word embeddings and semantic shifts: a survey.

When to use which

The survey reports no runnable method and gives no default settings 1806.03537 §3 1806.03537 §8. The time granularity of the cited corpora ranges from one month to ten years 1806.03537 §3.1.1. Test sets are scarce, so validate on your own labelled examples, or on the test-set types the survey names: word epoch disambiguation, cross-time alignment, event prediction and synthetic merged-word tasks 1806.03537 §3.1.2.

Implementation sketch

A generic pipeline that combines the survey's steps. It is an illustration, not a method given in the paper.

# Generic pipeline assembled from the survey  [§3] [§3.3]
# Input:  corpora, a list of (period, texts) ordered oldest first
#         words, the target words to score
# Output: score per word; higher means a larger change

def detect_semantic_change(corpora, words):
    # 1. Learn one vector space per period; incremental or joint training
    #    yields spaces that are already related  [§3.3]
    period_spaces = learn_period_spaces(corpora)
    # 2. Map independently trained spaces into one coordinate system with
    #    linear transformations; skip this after joint or incremental training
    #    [§3.3]
    aligned = align_spaces(period_spaces)
    # 3. Score each word by comparing its vectors across periods  [§3.3]
    #    a global measure suits regular drift; a local one suits cultural shifts
    scores = {}
    for word in words:
        scores[word] = compare_across_periods(aligned, word)
    return scores

Contradictions

• Laws of semantic change. The survey lists three proposed laws: frequent words change more slowly (conformity), polysemous words change faster at matched frequency (innovation), and change correlates with distance to the cluster centroid (prototypicality) 1806.03537 §4. It also reports Dubossarsky et al. (2017), who found the same patterns in shuffled and synchronous control data, which suggests the laws may be artefacts 1806.03537 §4. Both sides are reported; the survey does not resolve this.
• SGNS and PPMI. The survey says SVD-factorised PPMI and SGNS are equivalent from a mathematical point of view 1806.03537 §3.2. It also reports an advantage for SGNS in shift analysis, which low-rank SVD can match, especially on smaller data 1806.03537 §3.2.

Papers

• Diachronic word embeddings and semantic shifts: a survey: surveys data, model families, alignment and comparison measures for tracing word meaning change, and reviews the proposed laws. Reports no new experiments.