---
kind: concept
slug: word-embedding-association-test
title: Word embedding association test
summary: >-
  Permutation test of whether two target word sets lean toward two attribute word sets, giving a p-value. Gonen and Goldberg run it on two debiased embeddings and report all six p-values below 0.05. Wan et al. reuse the WEAT score on words from LLM-generated letters, with no p-values. Neither states every detail an implementer needs.
tags: [word-embeddings, bias, fairness, hypothesis-testing]
papers: ["1903.03862", "2310.09219"]
related: [debiasing-word-embeddings, algorithmic-fairness, text-embedding, word-embedding]
status: draft
updated_at: 2026-11-01
---

## Overview

The test asks whether two sets of target words sit closer to one attribute
set than to the other. Each word gets an association score with the two
attribute sets. A split of the target words scores the summed association of
the first set minus the same sum for the second [arxiv:1903.03862 §4]. The
p-value comes from a permutation test: reshuffle the pooled target words into
two equal halves and count how often a reshuffled split scores as high as the
observed one [arxiv:1903.03862 §3].

The two papers below use the same kind of word-set test in different ways.
Gonen and Goldberg run it as a permutation test on debiased embeddings, with
word sets of 8 words per side [arxiv:1903.03862 §4]. No before-and-after
comparison is reported for this test, unlike the other checks in that paper.
Wan et al. report the score, called WEAT, as an effect size on word sets
drawn from LLM-generated recommendation letters, and report no p-value for it
[arxiv:2310.09219 §4.3.5] [arxiv:2310.09219 tab. 3].

## Variants

The variants differ in how the p-value is computed, or whether one is computed
at all. The first two come from the same paper and share one score function.

### Exact enumeration (the paper's setting)

Enumerate every way to split the pooled target words into two equal halves.
With 8 + 8 words that is 12,870 splits [arxiv:1903.03862 §4]. The p-value is
the share of splits that score at least as high as the observed one. Reported
p-values for the debiased spaces are 0 and 7.7e-5 (names against family and
career words), 0.00016 and 0.00031 (names against arts and maths words), and
0.0467 and 0.0064 (names against arts and science words), for Hard-Debiased
and GN-GloVe respectively [arxiv:1903.03862 §4].

### Sampled permutations (authors' code only)

The authors' notebook refuses word sets larger than 10 and points to a
sampled version, which caps the number of samples at 1,000,000. The paper
text does not describe this version, so treat it as an implementation choice
rather than a tested method [arxiv:1903.03862 §4].

### Effect size on generated text (Wan et al.)

Wan et al. report WEAT(MF), with male and female names, and WEAT(CF), with
career and family words, for ChatGPT and Alpaca letters, split into nouns and
adjectives [arxiv:2310.09219 §4.3.5] [arxiv:2310.09219 tab. 3]. Each word
list has 8 entries [arxiv:2310.09219 app. F.4]. The values for
ChatGPT are 0.393 and 0.901 for nouns, and 0.493 and 0.535 for adjectives.
The values for Alpaca are 0.579 and 0.419 for nouns, and 1.009 and 0.419 for
adjectives [arxiv:2310.09219 tab. 3]. The card flags that both Alpaca CF values
are 0.419, so check the source before reusing them. Card:
[Wan et al., 2023](/papers/2310.09219).

The paper says WEAT measures embedding distance but does not name the
embedding model [arxiv:2310.09219 §4.3.5]. It gives no p-value, no permutation
count and no formula for these values. Do not assume they equal the split
score above.

The sign conventions differ in wording. Wan et al. say a positive WEAT(MF)
means female words lean toward female-stereotyped traits [arxiv:2310.09219
§4.3.5]. Gonen and Goldberg define the score as the first set minus the second
[arxiv:1903.03862 §4]. Check which set is "first" before comparing signs
across papers.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Small word sets, such as 8 + 8, and you need a p-value | Exact enumeration | It is what the paper reports, and 12,870 splits is cheap to enumerate [arxiv:1903.03862 §4] |
| Sets too large to enumerate | Sampled permutations | Enumeration grows fast with set size; the authors' code refuses sets above 10 words and points to the sampled version [arxiv:1903.03862 §4] |
| You want to compare with the paper's numbers | Exact enumeration on the same word lists | The authors' notebook, which reproduces the reported values, enumerates 8 + 8 sets exactly; the paper text does not state the method [arxiv:1903.03862 §4] |
| You need to know whether debiasing removed the association | Run the test on the original and the debiased space | The paper reports the test on debiased spaces only, so this comparison is not shown there [arxiv:1903.03862 §4] |
| You want a direction and rough size for word-level gender lean in generated text, without a significance claim | Effect size on the generated words | Wan et al. report WEAT values in this setting and give no p-value [arxiv:2310.09219 tab. 3] |
| You need a significance claim on generated text | Exact enumeration on an embedding you name and state | The card has no p-value for the generated-text values, so the only p-values come from the debiasing study [arxiv:1903.03862 §4] |

## Implementation sketch

The score and the exact p-value share one core. Only the enumeration changes
between the two variants. The effect-size variant needs only the observed
score.

```pseudocode
# Score of one split of target words  [arxiv:1903.03862 §4]
def split_score(first, second, attrs_a, attrs_b, space):
    # Summed association for the first set...
    total = 0
    for w in first:
        total += association(w, attrs_a, attrs_b, space)
    # ...minus the same sum for the second set
    for w in second:
        total -= association(w, attrs_a, attrs_b, space)
    return total
```

```pseudocode
# Exact permutation p-value  [arxiv:1903.03862 §3] [arxiv:1903.03862 §4]
def exact_p_value(xs, ys, attrs_a, attrs_b, space):
    observed = split_score(xs, ys, attrs_a, attrs_b, space)
    pooled = xs + ys
    hits = 0
    total = 0
    # Each choice of first half is one split; 8 + 8 gives 12,870
    for half in combinations(pooled, len(xs)):
        total += 1
        rest = [w for w in pooled if w not in half]
        # The authors' code counts strictly greater; the text says at least
        if split_score(half, rest, attrs_a, attrs_b, space) > observed:
            hits += 1
    return hits / total
```

## Contradictions

The paper's text and its code disagree on the permutation count. The text
describes counting splits that score equal or greater than the observed one
[arxiv:1903.03862 §3]. The authors' code counts only splits scoring strictly
greater, which the paper text does not state. The two rules differ only when
a split ties with the observed score, so the gap is small in practice, but an
implementer should pick one and state it.

## Papers

- [Gonen and Goldberg, 2019](/papers/1903.03862): defines the test over Caliskan et al.'s word sets and runs three experiments on debiased spaces, with exact p-values reported.
- [Wan et al., 2023](/papers/2310.09219): applies WEAT(MF) and WEAT(CF) to ChatGPT and Alpaca recommendation letters, reporting effect sizes only.