wikillm

Paper · 2018-01-21 · cs.CL

A Survey of Word Embeddings Evaluation Methods

Survey of how word embeddings are evaluated. Sorts about 16 intrinsic methods (human similarity and analogy judgments, neuroimaging and eye-tracking data, thesauri, language-driven measures) and 12 extrinsic downstream tasks, lists the datasets for each, and argues that task scores do not correlate, so no single global score exists. Contains no experiments.

Word embeddingWord embedding evaluationevaluationsurveyword-embeddings

arXivPDFMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Word embeddings turn each word into a list of numbers, so that words used in similar ways end up with similar lists. Researchers still disagree about how to check whether a set of such lists is good. This survey collects the checks that have been used and sorts them into two families. In the first family, the word lists are plugged into a real language task, such as tagging parts of speech or spotting paraphrases, and judged by how well that task goes. In the second family, the word lists are compared directly with something else: ratings from people, brain recordings, hand-built dictionaries or thesauri, or patterns in text.

The author argues that the task-based scores do not agree with each other, so no single number can rank word lists for every purpose. The card lists the test sets used for each check, so that a reader can pick one that fits their use.

TL;DR

A survey of word-embedding evaluation methods. It describes about 16 intrinsic methods (the author's count) and 12 extrinsic downstream tasks, with a typology of conscious, subconscious, thesaurus-based and language-driven intrinsic checks. It lists datasets for each check and states that extrinsic scores do not correlate across tasks. Useful for choosing an evaluation and its datasets. It contains no experiments and no code.

Problem

There is no agreed way to evaluate word embeddings. NLP engineers usually test them on downstream tasks, while linguists studying meaning tend to use experiments borrowed from cognitive science §1. The paper recalls a 2016 workshop list of problems in the field: the notion of semantics is unclear, evaluation datasets are often not split into training and test sets, intrinsic and extrinsic scores do not correlate, significance tests are often missing, and "hub" vectors (for very frequent words) distort cosine distances §2. It says that many of these questions remain open §2.

Method

The paper is a literature survey. It proposes a typology and does not define an algorithm, so there are no equations or pseudocode of its own.

  1. Extrinsic evaluation §3. Use the word embeddings as input features for a supervised model trained on a downstream task, and use that model's score as the measure of embedding quality. The paper lists 12 tasks (see the datasets table). The author's view is that scores on different tasks do not correlate, so extrinsic scores can show strengths on one task but cannot serve as an absolute quality measure §3.
  2. Intrinsic evaluation §4. Compare embeddings with human judgments or other reference data. Two ways of collecting human data are named: absolute evaluation (rate each item, then compare the two data sets with an aggregate such as Spearman's correlation) and comparative evaluation (raters pick the better of two models). Human data can come from in-house experiments or from crowd-sourcing platforms §4. The author sorts intrinsic methods into four classes §4:
    1. Conscious (judgments made with time to think) §4.1: word similarity §4.1.1, word analogy §4.1.2, thematic fit §4.1.3, concept categorization §4.1.4, synonym detection §4.1.5 and outlier detection §4.1.6.
    2. Subconscious (implicit responses, such as reading time or brain signals) §4.2: semantic priming §4.2.1, neural activation patterns from fMRI and EEG §4.2.2, and eye-movement data §4.2.3.
    3. Thesaurus-based (reference data built by linguists or ontology engineers) §4.3: thesaurus vectors, as in QVEC §4.3.1; dictionary definition graphs §4.3.2; the cross-match test §4.3.3; semantic difference based on attributes §4.3.4; and semantic networks such as WordNet and DBpedia §4.3.5.
    4. Language-driven (data from the language itself, such as spelling, sound or corpus counts) §4.4: phonosemantic analysis, measuring how much word form and meaning correlate §4.4.1, and bigram co-occurrence frequency §4.4.2.
  3. Author's definition §4. The paper proposes that any measure of agreement between embeddings and data that could hypothetically carry lexical-semantic information counts as an evaluation method. The paper itself raises the question of where "exploration" ends and "evaluation" begins §4. Treat the typology as the author's proposal.

For the word-similarity check, the paper's description reduces to the following. It states no formula for the model-side score, and it does not say how to handle words missing from the vocabulary §4.1.1 [not stated].

Pseudocode
# Word-similarity check  [§4.1.1] [§4]
# Input:  pairs: list of (word_a, word_b, human_score)
#         vectors: map from word to embedding vector
# Output: one correlation value (higher is better)

def word_similarity_score(pairs, vectors):
    # 1. Model score per pair, e.g. cosine (paper: cosine is the usual
    #    choice for similarity [§4.1.2]); missing words not stated [not stated]
    model = [cosine(vectors[a], vectors[b]) for a, b, _ in pairs]
    # 2. Compare model scores with human scores by rank correlation  [§4]
    human = [h for _, _, h in pairs]
    return spearman(model, human)

Word analogy §4.1.2: given a pair a:a* and a word b, find b* so that b:b* has the same relation as a:a*. The paper names three scoring metrics (3CosAdd, with 3CosMul as a similar variant; PairDir; and Analogy Space Evaluation) but gives no formulas for them §4.1.2.

Hyperparameters

Not applicable. The paper trains no model and states no hyperparameters §1. Dataset scales and sizes are given in the datasets table below.

Datasets and test sets

Sizes are as stated in the paper §4.1.1–§4.4.2. Rating scales are given as the paper states them. The paper lists these in its enumerated lists; it contains no tables of its own.

FamilyDatasetSize (as stated)NotesSource
Word similaritySimVerb-35003 500 verb pairssimilarity, scale 0–4§4.1.1
Word similarityMEN3 000 pairsrelatedness, scale 0–50§4.1.1
Word similarityRW (Rare Word)2 034 pairsrare words; similarity, 0–10§4.1.1
Word similaritySimLex-999999 pairssimilarity, 0–10§4.1.1
Word similaritySemEval-2017500 pairssimilarity, 0–4; includes collocations§4.1.1
Word similarityMTurk-771771 pairsrelatedness, 0–5§4.1.1
Word similarityWordSim-353353 pairssimilarity, 0–10; instructions called ambiguous§4.1.1
Word similarityMTurk-287287 pairsrelatedness, 0–5§4.1.1
Word similarityWordSim-353-REL252 pairssubset with no similar-concept pairs§4.1.1
Word similarityWordSim-353-SIM203 pairssubset of similar or unassociated pairs§4.1.1
Word similarityVerb-143143 verb pairssimilarity, 0–4§4.1.1
Word similarityYP-130130 verb pairssimilarity, 0–4§4.1.1
Word similarityRG-6565 pairssimilarity, 0–4§4.1.1
Word similarityMC-3030 pairssubset of RG-65; MC-28 drops 2 pairs§4.1.1
Word analogyWordRep118 292 623 questions26 semantic classes; superset of Google Analogy plus WordNet data§4.1.2
Word analogyBATS99 200 questions4 classes, 10 subclasses§4.1.2
Word analogyGoogle Analogy19 544 questions8 869 semantic, 10 675 morphological§4.1.2
Word analogySemEval-201210 014 questions10 classes, 79 subclasses; year inconsistent in text§4.1.2
Word analogyMSR8 000 questions16 morphological classes§4.1.2
Word analogySAT5 610 questions374 semantic classes§4.1.2
Word analogyJAIR430 questions20 semantic classes; includes collocations§4.1.2
Thematic fitMSTNN1 444 verb-object-subject triples§4.1.3
Thematic fitGDS720 verb-object pairssubsets GDS-poly and GDS-mono§4.1.3
Thematic fitF-Inst & F-Loc522 verb pairs248 instrument verbs, 274 location verbs§4.1.3
Thematic fitP07414 verb-object-subject triples§4.1.3
Thematic fitUP211 verb-noun pairsrole set not limited§4.1.3
Thematic fitMT98200 verbssubset of MSTNN§4.1.3
CategorizationBM5 321 words56 categories§4.1.4
CategorizationAP402 words21 categories§4.1.4
CategorizationBLESS200 words27 semantic classes§4.1.4
CategorizationESSLLI-200845 words9 classes, or 5 in coarser grouping§4.1.4
Synonym detectionRDWP300 questions5-word tuples§4.1.5
Synonym detectionTOEFL80 questions§4.1.5
Synonym detectionESL50 questions§4.1.5
Outlier detection8-8-8 Dataset8 clusters8 words plus 8 outliers per cluster§4.1.6
Outlier detectionWordSim-500500 clusters8 words plus 5–7 outliers per cluster§4.1.6
Semantic primingSemantic Priming Project6 337 pairs768 subjects, 1 661 target words; SOA 70 and 200 ms; naming or lexical decision§4.2.1
fMRIStudyForrestGerman audio of "Forrest Gump"suggested by the paper§4.2.2
fMRIWord Processing datasetreadings of English wordssuggested by the paper§4.2.2
EEGnone known—author knows no public EEG dataset§4.2.2
Eye movementProvo Corpus55 paragraphs, 84 readers; 1 185 words, 26 features§4.2.3
Eye movementGECO5 000 sentences, 33 participants; 987 words, 48 features§4.2.3
ThesaurusQVEC (thesaurus vectors)WordNet super-sensesauthor says any knowledge-category document set could be used§4.3.1
ThesaurusDictionary definition graphany dictionaryno specific dataset named§4.3.2
ThesaurusCross-match testnone named§4.3.3
ThesaurusBLESS attributes200 word pairsattribute sets per pair§4.3.4
ThesaurusFeature Norms Dataset24 963 word pairsdistinctive features§4.3.4
ThesaurusWordNet155 287 words, 117 659 synsetscounts as at writing§4.3.5
ThesaurusDBpediaabout 4.22 million wordsgraph built from Wikipedia§4.3.5
Language-drivenPhonosemantic analysisnone openauthors used their own data§4.4.1
Language-drivenBigram co-occurrenceGoogle 1T frequency data or any n-gram corpus§4.4.2

Extrinsic tasks §3, with the datasets the paper names:

  1. Noun phrase chunking: CoNLL-2000.
  2. Named entity recognition: CoNLL-2002, CoNLL-2003, MUC-7.
  3. Sentiment analysis: movie reviews (Maas et al. 2011, as cited).
  4. Shallow syntax parsing: parsing datasets, e.g. the one cited as Marcus et al. 1993.
  5. Semantic role labeling: Proposition Bank.
  6. Negation scope: the paper says no dataset exists.
  7. Part-of-speech tagging: the Stanford Treebank, as named in the paper.
  8. Text classification: 20 Newsgroups.
  9. Metaphor detection: the Trope Finder dataset and a dataset from Tsvetkov et al. 2014.
  10. Paraphrase detection: Microsoft Research Paraphrase Corpus; Quora Question Pairs.
  11. Textual entailment (NLI): SICK; SNLI; Story Cloze Dataset.
  12. Input for neural networks (embeddings as initial weights): no dataset named; the paper cites Kocmi et al. 2017 as a possible framework §3.

Results

Author-reported counts. The paper runs no experiments; these numbers describe its survey scope §5.

QuantityAuthor's countSource
Intrinsic methods16§5
Extrinsic methods12§5
Total evaluation methods28§5

Setup: literature survey §1. The paper gives no baselines and no measured scores.

Paper vs code

No official code. The paper has no code link and no project page; its footnotes point only to a state-of-the-art wiki, dataset pages and a dataset release §3 §4.1.1 §4.4.2. A GitHub repository search for "word embeddings evaluation" with the author's name returned no repositories. Nothing in the paper is implemented to compare.

Pitfalls

  • Method count disagrees with the section headings. The abstract and §5 say 16 intrinsic methods. The §4 headings list 17 entries: 6 conscious, 4 subconscious (fMRI and EEG counted as two), 5 thesaurus-based and 2 language-driven. The paper does not say which entry it leaves out.
  • SemEval-2012 entry is inconsistent. The dataset is named SemEval-2012, but its text says it was prepared for SemEval-2017 Task 2, using the task name "Measuring Degrees of Relational Similarity" §4.1.2. The year is not resolved in the paper.
  • WordSim-353 subsets are not explained. REL (252 pairs) and SIM (203 pairs) add up to more than the 353 pairs of the parent set. The paper does not say whether they overlap §4.1.1.
  • Ranking claim not followed. §4.1.1 says the similarity datasets are ranked by size, but SemEval-2017 (500 pairs) is listed before MTurk-771 (771 pairs).
  • Analogy metrics are named, not defined. 3CosAdd, 3CosMul, PairDir and Analogy Space Evaluation appear with no formulas §4.1.2. An implementer needs the original papers the author cites for them, and this card does not reproduce those formulas.
  • Missing-word handling is not stated for any intrinsic check. Coverage of the test vocabulary changes the score, so report the number of pairs scored §4.1.1 [not stated].
  • Several datasets are not usable as stated. No public EEG dataset is known to the author §4.2.2. No dataset exists for negation scope §3. No open dataset exists for phonosemantic analysis §4.4.1. Availability is judged as of the paper's writing (2018).
  • The typology and the definition of evaluation are the author's view. The paper argues for a broad definition of evaluation and says that the last three intrinsic classes are its own proposal §4. Other authors may draw the line differently.
  • Author's correlation claim is a cited position. The statement that extrinsic scores do not correlate rests on the cited work of Schnabel et al. and on the author's reading §2 §3. This card did not verify the underlying experiments.
  • Human-judgment pitfalls the paper raises: rater subjectivity, rater fatigue on large pair sets, low inter-rater agreement and connotation effects §4.1.1. The paper reports no numbers for these.
  • Reliability of sizes. Sizes come from the paper's own text, not from the datasets. Check dataset releases before relying on the numbers.

Lineage

  • Builds on: the paper's history section names earlier work without using its content: Baroni et al. 2014 (overview of evaluation approaches), Schnabel et al. 2015 (split into extrinsic and intrinsic), Faruqui et al. 2016 (problem list) and the RepEval 2016 papers by Avraham et al. and Gladkova et al. §2. None has a card here.
  • Compared against: none; the paper is a survey.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “A Survey of Word Embeddings Evaluation Methods” by Amir Bakarov (arXiv:1801.09536), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.