---
kind: concept
slug: word-embedding-evaluation
title: Word embedding evaluation
summary: >-
  How to check whether word embeddings are good. Two families: extrinsic checks score a downstream task that uses the vectors, and intrinsic checks compare the vectors with human ratings, brain or eye-tracking data, thesauri or corpus counts. The survey finds that task scores do not correlate with each other, so no single global score exists. Only one survey card so far, with no measured results.
tags: [nlp, evaluation, representation-learning]
papers: ["1801.09536"]
related: [word-embedding, word-embedding-association-test, debiasing-word-embeddings, benchmark-validity]
status: draft
updated_at: 2026-11-12
---

## Overview

A word embedding is judged by how well it agrees with some reference. The field
has no agreed reference, so the choice comes down to two families [arxiv:1801.09536 §3]
[arxiv:1801.09536 §4]:

- **Extrinsic** checks use the vectors as input features for a supervised model
  on a downstream task, and treat that model's score as the quality measure
  [arxiv:1801.09536 §3].
- **Intrinsic** checks compare the vectors with reference data, such as human
  judgements, neuroimaging, thesauri or language statistics [arxiv:1801.09536 §4].

The survey lists 16 intrinsic and 12 extrinsic methods [arxiv:1801.09536 §5].
It names four problems that remain open: the notion of semantics is unclear,
evaluation data is often not split into training and test sets, intrinsic and
extrinsic scores do not correlate, and significance tests are often missing.
It also flags "hub" vectors for very frequent words, which distort cosine
distances [arxiv:1801.09536 §2].

The page covers the survey's typology and dataset lists. It does not report
measured scores, because the survey runs no experiments [arxiv:1801.09536 §1].

## Variants

### Extrinsic: downstream task scores

Train a supervised model on a downstream task with the embeddings as features,
and report that model's score on the task's test data [arxiv:1801.09536 §3].
The survey lists 12 tasks and names datasets for most of them, among them
chunking, named-entity recognition, text classification, paraphrase detection
and entailment [arxiv:1801.09536 §3]. Card: [A Survey of Word Embeddings Evaluation Methods](/papers/1801.09536).

### Intrinsic, conscious: human judgements

Humans rate items with time to think about them. Ratings can be collected
absolutely (each item rated, then compared with an aggregate such as Spearman's
correlation) or comparatively (raters pick the better of two models), in-house
or through crowd-sourcing [arxiv:1801.09536 §4]. Six methods fall here:

- **Word similarity**: the distances between word pairs in the embedding space
  are compared with human similarity ratings [arxiv:1801.09536 §4.1.1]. Fourteen
  datasets are listed, from 30 to 3 500 pairs [arxiv:1801.09536 §4.1.1].
- **Word analogy**: given a pair a:a* and a word b, find b* with the same
  relation. Three scoring metrics are named (3CosAdd, with 3CosMul as a variant;
  PairDir; Analogy Space Evaluation), but the survey gives no formulas for them
  [arxiv:1801.09536 §4.1.2].
- **Thematic fit**, **concept categorization**, **synonym detection** and
  **outlier detection** use human judgements on word relations, pairs or groups
  [arxiv:1801.09536 §4.1.3]–[arxiv:1801.09536 §4.1.6].

### Intrinsic, subconscious: implicit responses

Measures taken without deliberate judgement, such as reading times or brain
signals [arxiv:1801.09536 §4.2].

- **Semantic priming** uses reaction data on word pairs [arxiv:1801.09536 §4.2.1].
- **Neural activation** from fMRI or EEG. The survey suggests two fMRI
  datasets, and says it knows no public EEG dataset for this task
  [arxiv:1801.09536 §4.2.2].
- **Eye movement** from eye-tracking features per word, compared with the
  embeddings by predicting each word's nearest neighbours [arxiv:1801.09536 §4.2.3].

### Intrinsic, thesaurus-based: reference data built by experts

Reference data from linguists or ontology engineers: thesaurus vectors
[arxiv:1801.09536 §4.3.1], dictionary definition graphs [arxiv:1801.09536 §4.3.2],
a cross-match test [arxiv:1801.09536 §4.3.3], attribute-based semantic
difference [arxiv:1801.09536 §4.3.4], and semantic networks such as WordNet
and DBpedia [arxiv:1801.09536 §4.3.5].

### Intrinsic, language-driven: statistics of the language itself

- **Phonosemantic analysis** measures how far word form and meaning correlate.
  The survey knows no open dataset for it [arxiv:1801.09536 §4.4.1].
- **Bigram co-occurrence** frequency, from Google 1T data or any n-gram corpus
  [arxiv:1801.09536 §4.4.2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| A downstream system with labelled data | Extrinsic check on that task [arxiv:1801.09536 §3] | The survey argues that scores on one task do not carry over to others, so a task score says little about the rest [arxiv:1801.09536 §3]. |
| No task data, but you need a quick comparison | Word similarity or analogy sets [arxiv:1801.09536 §4.1.1] [arxiv:1801.09536 §4.1.2] | Check each set's size and licence before use. The survey names rater bias and low inter-rater agreement as problems [arxiv:1801.09536 §4.1.1]. |
| Analogy scoring | Any of the three named metrics [arxiv:1801.09536 §4.1.2] | The survey names the metrics without formulas, so take the formulas from the original papers the survey cites. |
| You want to avoid new human ratings | Thesaurus or corpus-count checks [arxiv:1801.09536 §4.3] [arxiv:1801.09536 §4.4.2] | No new annotation is needed. Results depend on the coverage and design of the reference resource (our reading). |
| Brain or eye-tracking data | Only where an open dataset exists [arxiv:1801.09536 §4.2.2] [arxiv:1801.09536 §4.2.3] | No public EEG dataset is known to the survey. |

Because the survey argues that no single score exists [arxiv:1801.09536 §3],
report more than one check, and state which task, dataset and version were
used. Missing-word handling is not stated for any intrinsic check, so report
how many pairs or items were scored [arxiv:1801.09536 §4.1.1].

## Implementation sketch

The survey gives no code. This sketch is the word-similarity check it
describes: model distances between word pairs, then a rank correlation with
human scores [arxiv:1801.09536 §4] [arxiv:1801.09536 §4.1.1]. The survey does
not name the model-side measure. The survey calls cosine distance
"intuitively adequate" for similarity [arxiv:1801.09536 §4.1.2].

```pseudocode
# Word-similarity check [arxiv:1801.09536 §4.1.1]
# Input:  pairs: list of (word_a, word_b, human_score)
#         vectors: map from word to embedding vector
# Output: one correlation value (higher is better)

def word_similarity_score(pairs, vectors):
    # 1. Model-side distance per pair, e.g. cosine [arxiv:1801.09536 §4.1.2]
    #    Missing-word handling: not stated
    model = [cosine(vectors[a], vectors[b]) for a, b, _ in pairs]
    # 2. Compare with human ratings by rank correlation [arxiv:1801.09536 §4]
    human = [h for _, _, h in pairs]
    return spearman(model, human)
```

## Contradictions

- **Correlation of intrinsic and extrinsic scores.** The survey lists the
  lack of correlation between the two families as an open problem
  [arxiv:1801.09536 §2]. Some researchers, it notes, assume that results on
  different tasks do correlate and so define a global score [arxiv:1801.09536 §3].
  For similarity ratings, the survey reports that other researchers attribute
  the lack of correlation to unrepresentative human assessments
  [arxiv:1801.09536 §4.1.1]. The paper does not resolve these positions.
- **Number of intrinsic methods.** The survey states 16 intrinsic methods
  [arxiv:1801.09536 §5]. Its §4 section headings list more entries than that,
  counting fMRI and EEG separately, and the survey does not say which entry is
  left out [arxiv:1801.09536 §4].

## Papers

- [A Survey of Word Embeddings Evaluation Methods](/papers/1801.09536): a typology of 16 intrinsic and 12 extrinsic evaluation methods, with dataset lists and open problems. No experiments.
