Paper · 2018-01-21 · cs.CL
A Survey of Word Embeddings Evaluation Methods
Survey of how word embeddings are evaluated. Sorts about 16 intrinsic methods (human similarity and analogy judgments, neuroimaging and eye-tracking data, thesauri, language-driven measures) and 12 extrinsic downstream tasks, lists the datasets for each, and argues that task scores do not correlate, so no single global score exists. Contains no experiments.
Word embeddingWord embedding evaluationevaluationsurveyword-embeddings
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Word embeddings turn each word into a list of numbers, so that words used in similar ways end up with similar lists. Researchers still disagree about how to check whether a set of such lists is good. This survey collects the checks that have been used and sorts them into two families. In the first family, the word lists are plugged into a real language task, such as tagging parts of speech or spotting paraphrases, and judged by how well that task goes. In the second family, the word lists are compared directly with something else: ratings from people, brain recordings, hand-built dictionaries or thesauri, or patterns in text.
The author argues that the task-based scores do not agree with each other, so no single number can rank word lists for every purpose. The card lists the test sets used for each check, so that a reader can pick one that fits their use.
TL;DR
A survey of word-embedding evaluation methods. It describes about 16 intrinsic methods (the author's count) and 12 extrinsic downstream tasks, with a typology of conscious, subconscious, thesaurus-based and language-driven intrinsic checks. It lists datasets for each check and states that extrinsic scores do not correlate across tasks. Useful for choosing an evaluation and its datasets. It contains no experiments and no code.
Problem
There is no agreed way to evaluate word embeddings. NLP engineers usually test them on downstream tasks, while linguists studying meaning tend to use experiments borrowed from cognitive science §1. The paper recalls a 2016 workshop list of problems in the field: the notion of semantics is unclear, evaluation datasets are often not split into training and test sets, intrinsic and extrinsic scores do not correlate, significance tests are often missing, and "hub" vectors (for very frequent words) distort cosine distances §2. It says that many of these questions remain open §2.
Method
The paper is a literature survey. It proposes a typology and does not define an algorithm, so there are no equations or pseudocode of its own.
- Extrinsic evaluation §3. Use the word embeddings as input features for a supervised model trained on a downstream task, and use that model's score as the measure of embedding quality. The paper lists 12 tasks (see the datasets table). The author's view is that scores on different tasks do not correlate, so extrinsic scores can show strengths on one task but cannot serve as an absolute quality measure §3.
- Intrinsic evaluation §4. Compare embeddings with human judgments or
other reference data. Two ways of collecting human data are named: absolute
evaluation (rate each item, then compare the two data sets with an
aggregate such as Spearman's correlation) and comparative evaluation (raters
pick the better of two models). Human data can come from in-house
experiments or from crowd-sourcing platforms §4.
The author sorts intrinsic methods into four classes §4:
- Conscious (judgments made with time to think) §4.1: word similarity §4.1.1, word analogy §4.1.2, thematic fit §4.1.3, concept categorization §4.1.4, synonym detection §4.1.5 and outlier detection §4.1.6.
- Subconscious (implicit responses, such as reading time or brain signals) §4.2: semantic priming §4.2.1, neural activation patterns from fMRI and EEG §4.2.2, and eye-movement data §4.2.3.
- Thesaurus-based (reference data built by linguists or ontology engineers) §4.3: thesaurus vectors, as in QVEC §4.3.1; dictionary definition graphs §4.3.2; the cross-match test §4.3.3; semantic difference based on attributes §4.3.4; and semantic networks such as WordNet and DBpedia §4.3.5.
- Language-driven (data from the language itself, such as spelling, sound or corpus counts) §4.4: phonosemantic analysis, measuring how much word form and meaning correlate §4.4.1, and bigram co-occurrence frequency §4.4.2.
- Author's definition §4. The paper proposes that any measure of agreement between embeddings and data that could hypothetically carry lexical-semantic information counts as an evaluation method. The paper itself raises the question of where "exploration" ends and "evaluation" begins §4. Treat the typology as the author's proposal.
For the word-similarity check, the paper's description reduces to the following. It states no formula for the model-side score, and it does not say how to handle words missing from the vocabulary §4.1.1 [not stated].
# Word-similarity check [§4.1.1] [§4]
# Input: pairs: list of (word_a, word_b, human_score)
# vectors: map from word to embedding vector
# Output: one correlation value (higher is better)
def word_similarity_score(pairs, vectors):
# 1. Model score per pair, e.g. cosine (paper: cosine is the usual
# choice for similarity [§4.1.2]); missing words not stated [not stated]
model = [cosine(vectors[a], vectors[b]) for a, b, _ in pairs]
# 2. Compare model scores with human scores by rank correlation [§4]
human = [h for _, _, h in pairs]
return spearman(model, human)Word analogy §4.1.2: given a pair a:a* and a word b, find b* so that b:b* has the same relation as a:a*. The paper names three scoring metrics (3CosAdd, with 3CosMul as a similar variant; PairDir; and Analogy Space Evaluation) but gives no formulas for them §4.1.2.
Hyperparameters
Not applicable. The paper trains no model and states no hyperparameters §1. Dataset scales and sizes are given in the datasets table below.
Datasets and test sets
Sizes are as stated in the paper §4.1.1–§4.4.2. Rating scales are given as the paper states them. The paper lists these in its enumerated lists; it contains no tables of its own.
| Family | Dataset | Size (as stated) | Notes | Source |
|---|---|---|---|---|
| Word similarity | SimVerb-3500 | 3 500 verb pairs | similarity, scale 0–4 | §4.1.1 |
| Word similarity | MEN | 3 000 pairs | relatedness, scale 0–50 | §4.1.1 |
| Word similarity | RW (Rare Word) | 2 034 pairs | rare words; similarity, 0–10 | §4.1.1 |
| Word similarity | SimLex-999 | 999 pairs | similarity, 0–10 | §4.1.1 |
| Word similarity | SemEval-2017 | 500 pairs | similarity, 0–4; includes collocations | §4.1.1 |
| Word similarity | MTurk-771 | 771 pairs | relatedness, 0–5 | §4.1.1 |
| Word similarity | WordSim-353 | 353 pairs | similarity, 0–10; instructions called ambiguous | §4.1.1 |
| Word similarity | MTurk-287 | 287 pairs | relatedness, 0–5 | §4.1.1 |
| Word similarity | WordSim-353-REL | 252 pairs | subset with no similar-concept pairs | §4.1.1 |
| Word similarity | WordSim-353-SIM | 203 pairs | subset of similar or unassociated pairs | §4.1.1 |
| Word similarity | Verb-143 | 143 verb pairs | similarity, 0–4 | §4.1.1 |
| Word similarity | YP-130 | 130 verb pairs | similarity, 0–4 | §4.1.1 |
| Word similarity | RG-65 | 65 pairs | similarity, 0–4 | §4.1.1 |
| Word similarity | MC-30 | 30 pairs | subset of RG-65; MC-28 drops 2 pairs | §4.1.1 |
| Word analogy | WordRep | 118 292 623 questions | 26 semantic classes; superset of Google Analogy plus WordNet data | §4.1.2 |
| Word analogy | BATS | 99 200 questions | 4 classes, 10 subclasses | §4.1.2 |
| Word analogy | Google Analogy | 19 544 questions | 8 869 semantic, 10 675 morphological | §4.1.2 |
| Word analogy | SemEval-2012 | 10 014 questions | 10 classes, 79 subclasses; year inconsistent in text | §4.1.2 |
| Word analogy | MSR | 8 000 questions | 16 morphological classes | §4.1.2 |
| Word analogy | SAT | 5 610 questions | 374 semantic classes | §4.1.2 |
| Word analogy | JAIR | 430 questions | 20 semantic classes; includes collocations | §4.1.2 |
| Thematic fit | MSTNN | 1 444 verb-object-subject triples | §4.1.3 | |
| Thematic fit | GDS | 720 verb-object pairs | subsets GDS-poly and GDS-mono | §4.1.3 |
| Thematic fit | F-Inst & F-Loc | 522 verb pairs | 248 instrument verbs, 274 location verbs | §4.1.3 |
| Thematic fit | P07 | 414 verb-object-subject triples | §4.1.3 | |
| Thematic fit | UP | 211 verb-noun pairs | role set not limited | §4.1.3 |
| Thematic fit | MT98 | 200 verbs | subset of MSTNN | §4.1.3 |
| Categorization | BM | 5 321 words | 56 categories | §4.1.4 |
| Categorization | AP | 402 words | 21 categories | §4.1.4 |
| Categorization | BLESS | 200 words | 27 semantic classes | §4.1.4 |
| Categorization | ESSLLI-2008 | 45 words | 9 classes, or 5 in coarser grouping | §4.1.4 |
| Synonym detection | RDWP | 300 questions | 5-word tuples | §4.1.5 |
| Synonym detection | TOEFL | 80 questions | §4.1.5 | |
| Synonym detection | ESL | 50 questions | §4.1.5 | |
| Outlier detection | 8-8-8 Dataset | 8 clusters | 8 words plus 8 outliers per cluster | §4.1.6 |
| Outlier detection | WordSim-500 | 500 clusters | 8 words plus 5–7 outliers per cluster | §4.1.6 |
| Semantic priming | Semantic Priming Project | 6 337 pairs | 768 subjects, 1 661 target words; SOA 70 and 200 ms; naming or lexical decision | §4.2.1 |
| fMRI | StudyForrest | German audio of "Forrest Gump" | suggested by the paper | §4.2.2 |
| fMRI | Word Processing dataset | readings of English words | suggested by the paper | §4.2.2 |
| EEG | none known | — | author knows no public EEG dataset | §4.2.2 |
| Eye movement | Provo Corpus | 55 paragraphs, 84 readers; 1 185 words, 26 features | §4.2.3 | |
| Eye movement | GECO | 5 000 sentences, 33 participants; 987 words, 48 features | §4.2.3 | |
| Thesaurus | QVEC (thesaurus vectors) | WordNet super-senses | author says any knowledge-category document set could be used | §4.3.1 |
| Thesaurus | Dictionary definition graph | any dictionary | no specific dataset named | §4.3.2 |
| Thesaurus | Cross-match test | none named | §4.3.3 | |
| Thesaurus | BLESS attributes | 200 word pairs | attribute sets per pair | §4.3.4 |
| Thesaurus | Feature Norms Dataset | 24 963 word pairs | distinctive features | §4.3.4 |
| Thesaurus | WordNet | 155 287 words, 117 659 synsets | counts as at writing | §4.3.5 |
| Thesaurus | DBpedia | about 4.22 million words | graph built from Wikipedia | §4.3.5 |
| Language-driven | Phonosemantic analysis | none open | authors used their own data | §4.4.1 |
| Language-driven | Bigram co-occurrence | Google 1T frequency data or any n-gram corpus | §4.4.2 |
Extrinsic tasks §3, with the datasets the paper names:
- Noun phrase chunking: CoNLL-2000.
- Named entity recognition: CoNLL-2002, CoNLL-2003, MUC-7.
- Sentiment analysis: movie reviews (Maas et al. 2011, as cited).
- Shallow syntax parsing: parsing datasets, e.g. the one cited as Marcus et al. 1993.
- Semantic role labeling: Proposition Bank.
- Negation scope: the paper says no dataset exists.
- Part-of-speech tagging: the Stanford Treebank, as named in the paper.
- Text classification: 20 Newsgroups.
- Metaphor detection: the Trope Finder dataset and a dataset from Tsvetkov et al. 2014.
- Paraphrase detection: Microsoft Research Paraphrase Corpus; Quora Question Pairs.
- Textual entailment (NLI): SICK; SNLI; Story Cloze Dataset.
- Input for neural networks (embeddings as initial weights): no dataset named; the paper cites Kocmi et al. 2017 as a possible framework §3.
Results
Author-reported counts. The paper runs no experiments; these numbers describe its survey scope §5.
| Quantity | Author's count | Source |
|---|---|---|
| Intrinsic methods | 16 | §5 |
| Extrinsic methods | 12 | §5 |
| Total evaluation methods | 28 | §5 |
Setup: literature survey §1. The paper gives no baselines and no measured scores.
Paper vs code
No official code. The paper has no code link and no project page; its footnotes point only to a state-of-the-art wiki, dataset pages and a dataset release §3 §4.1.1 §4.4.2. A GitHub repository search for "word embeddings evaluation" with the author's name returned no repositories. Nothing in the paper is implemented to compare.
Pitfalls
- Method count disagrees with the section headings. The abstract and §5 say 16 intrinsic methods. The §4 headings list 17 entries: 6 conscious, 4 subconscious (fMRI and EEG counted as two), 5 thesaurus-based and 2 language-driven. The paper does not say which entry it leaves out.
- SemEval-2012 entry is inconsistent. The dataset is named SemEval-2012, but its text says it was prepared for SemEval-2017 Task 2, using the task name "Measuring Degrees of Relational Similarity" §4.1.2. The year is not resolved in the paper.
- WordSim-353 subsets are not explained. REL (252 pairs) and SIM (203 pairs) add up to more than the 353 pairs of the parent set. The paper does not say whether they overlap §4.1.1.
- Ranking claim not followed. §4.1.1 says the similarity datasets are ranked by size, but SemEval-2017 (500 pairs) is listed before MTurk-771 (771 pairs).
- Analogy metrics are named, not defined. 3CosAdd, 3CosMul, PairDir and Analogy Space Evaluation appear with no formulas §4.1.2. An implementer needs the original papers the author cites for them, and this card does not reproduce those formulas.
- Missing-word handling is not stated for any intrinsic check. Coverage of the test vocabulary changes the score, so report the number of pairs scored §4.1.1 [not stated].
- Several datasets are not usable as stated. No public EEG dataset is known to the author §4.2.2. No dataset exists for negation scope §3. No open dataset exists for phonosemantic analysis §4.4.1. Availability is judged as of the paper's writing (2018).
- The typology and the definition of evaluation are the author's view. The paper argues for a broad definition of evaluation and says that the last three intrinsic classes are its own proposal §4. Other authors may draw the line differently.
- Author's correlation claim is a cited position. The statement that extrinsic scores do not correlate rests on the cited work of Schnabel et al. and on the author's reading §2 §3. This card did not verify the underlying experiments.
- Human-judgment pitfalls the paper raises: rater subjectivity, rater fatigue on large pair sets, low inter-rater agreement and connotation effects §4.1.1. The paper reports no numbers for these.
- Reliability of sizes. Sizes come from the paper's own text, not from the datasets. Check dataset releases before relying on the numbers.
Lineage
- Builds on: the paper's history section names earlier work without using its content: Baroni et al. 2014 (overview of evaluation approaches), Schnabel et al. 2015 (split into extrinsic and intrinsic), Faruqui et al. 2016 (problem list) and the RepEval 2016 papers by Avraham et al. and Gladkova et al. §2. None has a card here.
- Compared against: none; the paper is a survey.
- Superseded by: not known from this wiki.
Source and license
Adapted from “A Survey of Word Embeddings Evaluation Methods” by Amir Bakarov (arXiv:1801.09536), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.