Concept · 1 paper · updated 2026-11-12
Word embedding evaluation
How to check whether word embeddings are good. Two families: extrinsic checks score a downstream task that uses the vectors, and intrinsic checks compare the vectors with human ratings, brain or eye-tracking data, thesauri or corpus counts. The survey finds that task scores do not correlate with each other, so no single global score exists. Only one survey card so far, with no measured results.
nlpevaluationrepresentation-learning
Markdown4 min read
AI-compiled overview, updated 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
A word embedding is judged by how well it agrees with some reference. The field has no agreed reference, so the choice comes down to two families 1801.09536 §3 1801.09536 §4:
- Extrinsic checks use the vectors as input features for a supervised model on a downstream task, and treat that model's score as the quality measure 1801.09536 §3.
- Intrinsic checks compare the vectors with reference data, such as human judgements, neuroimaging, thesauri or language statistics 1801.09536 §4.
The survey lists 16 intrinsic and 12 extrinsic methods 1801.09536 §5. It names four problems that remain open: the notion of semantics is unclear, evaluation data is often not split into training and test sets, intrinsic and extrinsic scores do not correlate, and significance tests are often missing. It also flags "hub" vectors for very frequent words, which distort cosine distances 1801.09536 §2.
The page covers the survey's typology and dataset lists. It does not report measured scores, because the survey runs no experiments 1801.09536 §1.
Variants
Extrinsic: downstream task scores
Train a supervised model on a downstream task with the embeddings as features, and report that model's score on the task's test data 1801.09536 §3. The survey lists 12 tasks and names datasets for most of them, among them chunking, named-entity recognition, text classification, paraphrase detection and entailment 1801.09536 §3. Card: A Survey of Word Embeddings Evaluation Methods.
Intrinsic, conscious: human judgements
Humans rate items with time to think about them. Ratings can be collected absolutely (each item rated, then compared with an aggregate such as Spearman's correlation) or comparatively (raters pick the better of two models), in-house or through crowd-sourcing 1801.09536 §4. Six methods fall here:
- Word similarity: the distances between word pairs in the embedding space are compared with human similarity ratings 1801.09536 §4.1.1. Fourteen datasets are listed, from 30 to 3 500 pairs 1801.09536 §4.1.1.
- Word analogy: given a pair a:a* and a word b, find b* with the same relation. Three scoring metrics are named (3CosAdd, with 3CosMul as a variant; PairDir; Analogy Space Evaluation), but the survey gives no formulas for them 1801.09536 §4.1.2.
- Thematic fit, concept categorization, synonym detection and outlier detection use human judgements on word relations, pairs or groups 1801.09536 §4.1.3–1801.09536 §4.1.6.
Intrinsic, subconscious: implicit responses
Measures taken without deliberate judgement, such as reading times or brain signals 1801.09536 §4.2.
- Semantic priming uses reaction data on word pairs 1801.09536 §4.2.1.
- Neural activation from fMRI or EEG. The survey suggests two fMRI datasets, and says it knows no public EEG dataset for this task 1801.09536 §4.2.2.
- Eye movement from eye-tracking features per word, compared with the embeddings by predicting each word's nearest neighbours 1801.09536 §4.2.3.
Intrinsic, thesaurus-based: reference data built by experts
Reference data from linguists or ontology engineers: thesaurus vectors 1801.09536 §4.3.1, dictionary definition graphs 1801.09536 §4.3.2, a cross-match test 1801.09536 §4.3.3, attribute-based semantic difference 1801.09536 §4.3.4, and semantic networks such as WordNet and DBpedia 1801.09536 §4.3.5.
Intrinsic, language-driven: statistics of the language itself
- Phonosemantic analysis measures how far word form and meaning correlate. The survey knows no open dataset for it 1801.09536 §4.4.1.
- Bigram co-occurrence frequency, from Google 1T data or any n-gram corpus 1801.09536 §4.4.2.
When to use which
| Situation | Use | Why |
|---|---|---|
| A downstream system with labelled data | Extrinsic check on that task 1801.09536 §3 | The survey argues that scores on one task do not carry over to others, so a task score says little about the rest 1801.09536 §3. |
| No task data, but you need a quick comparison | Word similarity or analogy sets 1801.09536 §4.1.1 1801.09536 §4.1.2 | Check each set's size and licence before use. The survey names rater bias and low inter-rater agreement as problems 1801.09536 §4.1.1. |
| Analogy scoring | Any of the three named metrics 1801.09536 §4.1.2 | The survey names the metrics without formulas, so take the formulas from the original papers the survey cites. |
| You want to avoid new human ratings | Thesaurus or corpus-count checks 1801.09536 §4.3 1801.09536 §4.4.2 | No new annotation is needed. Results depend on the coverage and design of the reference resource (our reading). |
| Brain or eye-tracking data | Only where an open dataset exists 1801.09536 §4.2.2 1801.09536 §4.2.3 | No public EEG dataset is known to the survey. |
Because the survey argues that no single score exists 1801.09536 §3, report more than one check, and state which task, dataset and version were used. Missing-word handling is not stated for any intrinsic check, so report how many pairs or items were scored 1801.09536 §4.1.1.
Implementation sketch
The survey gives no code. This sketch is the word-similarity check it describes: model distances between word pairs, then a rank correlation with human scores 1801.09536 §4 1801.09536 §4.1.1. The survey does not name the model-side measure. The survey calls cosine distance "intuitively adequate" for similarity 1801.09536 §4.1.2.
# Word-similarity check [arxiv:1801.09536 §4.1.1]
# Input: pairs: list of (word_a, word_b, human_score)
# vectors: map from word to embedding vector
# Output: one correlation value (higher is better)
def word_similarity_score(pairs, vectors):
# 1. Model-side distance per pair, e.g. cosine [arxiv:1801.09536 §4.1.2]
# Missing-word handling: not stated
model = [cosine(vectors[a], vectors[b]) for a, b, _ in pairs]
# 2. Compare with human ratings by rank correlation [arxiv:1801.09536 §4]
human = [h for _, _, h in pairs]
return spearman(model, human)Contradictions
- Correlation of intrinsic and extrinsic scores. The survey lists the lack of correlation between the two families as an open problem 1801.09536 §2. Some researchers, it notes, assume that results on different tasks do correlate and so define a global score 1801.09536 §3. For similarity ratings, the survey reports that other researchers attribute the lack of correlation to unrepresentative human assessments 1801.09536 §4.1.1. The paper does not resolve these positions.
- Number of intrinsic methods. The survey states 16 intrinsic methods 1801.09536 §5. Its §4 section headings list more entries than that, counting fMRI and EEG separately, and the survey does not say which entry is left out 1801.09536 §4.
Papers
- A Survey of Word Embeddings Evaluation Methods: a typology of 16 intrinsic and 12 extrinsic evaluation methods, with dataset lists and open problems. No experiments.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.