Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them

In plain words

Word embeddings are lists of numbers that stand for words, learned from text. Earlier methods found one direction in that space that runs from "he" to "she", measured each word's position along it, and pushed the gender-related words back to neutral. Measured along that one direction, the bias looks gone.

This paper argues that the direction is only a measuring stick, and that shrinking the measurement hides the bias instead of removing it. Words that were gendered still group together, a word's neighbours still lean the way they did before, and a simple classifier can still tell the gender-associated words apart. A debiased embedding should not be trusted just because the usual score drops.

TL;DR

Two debiasing methods, Bolukbasi et al.'s Hard-Debiased and Zhao et al.'s GN-GloVe, reduce the projection bias on the he-she direction §2. The authors show that gender information survives: clustering, neighbour-based bias, professions, association tests and a classifier all still reveal it §4. All numbers are author-reported. Use this card to judge whether a debiasing step removes bias or only hides it on the measure you care about.

Problem

Debiasing methods that target the gender direction are judged by how much that one projection shrinks §2. The authors question whether this measures bias at all, because stereotyped words keep their relations to each other after the projection is removed §2. A downstream model can then learn gender from neighbourhoods, not from the removed direction §5.

Method

The paper is an analysis, not a new method. It takes two released debiased embeddings, compares each to its own biased starting point, and runs five tests §3, §4.

Gender bias of a word, as defined by earlier work, is its projection on the he-she direction, with all vectors normalised §2:

\mathrm{bias}(w) = \vec{w} \cdot (\vec{he} - \vec{she}) §2

1. Models. Hard-Debiased (Bolukbasi et al.) is compared with the original word2vec embedding trained on Google News §1 §3. GN-GloVe (Zhao et al.) is compared with a standard GloVe model trained on the same corpus §3.
2. Vocabulary. Take the most frequent 50,000 entries, drop words with uppercase letters, digits or punctuation, and drop words that are too long; then remove the gendered words that each original work lists. This gives 26,189 words for Hard-Debiased and 47,698 for GN-GloVe §3.
3. Clustering. Take the 500 most male-biased and 500 most female-biased words by the original bias. Split them into two k-means clusters, once in the original space and once in the debiased space, and score each split against the original gender labels. t-SNE plots of the same words are shown for both models §4 fig. 1.
4. Bias by neighbours. For each word, count how many of its nearest neighbours are male-biased. A neighbour counts as male-biased when its bias in the original embedding is positive, even in the debiased space. Correlate each word's original bias with this count, in both spaces §4. The footnote in §4 explains the labels are a proxy, since social bias cannot be read directly from the debiased vectors.
5. Professions. Run step 4 on the list of occupations used by the two earlier papers, and plot count against original bias §4 fig. 2.
6. Association tests. Replicate three WEAT-style experiments of Caliskan et al.: names against family and career words, names against arts and maths words, and names against arts and science words. Only the debiased spaces are tested §4. The word lists are in the §4 footnote.
7. Classifier. Take the 2,500 most male- and 2,500 most female-biased words. Train an RBF-kernel SVM on 500 of each gender, using the original or debiased vectors, and test on the remaining 2,000 of each gender §4.

# Projection bias of one word on unit vectors  [§2]
# [code: source/remaining_bias_2016.ipynb#L259-L265]
def projection_bias(word, vectors, he, she):
    # Positive values lean male, negative values lean female
    return dot(vectors[word], he) - dot(vectors[word], she)

# Male-biased neighbour count of one word  [§4]
# [code: source/remaining_bias_2016.ipynb#L324-L341]
def male_neighbour_count(word, space, bias_before, k=100):
    # k nearest words by cosine, inside the reduced vocabulary, excluding word
    neighbours = nearest_by_cosine(word, space, k)
    # Labels always come from the original embedding's bias
    count = 0
    for other in neighbours:
        if bias_before[other] > 0:
            count += 1
    return count

# Agreement between original bias and neighbour counts  [§4]
# [code: source/remaining_bias_2016.ipynb#L366-L380]
def neighbour_correlation(words, bias_before, count_by_word):
    # One pair per word: its original bias and its male neighbour count
    bias = [bias_before[w] for w in words]
    counts = [count_by_word[w] for w in words]
    return pearson_r(bias, counts)

# Clustering of the most biased words  [§4] [fig. 1]
# [code: source/remaining_bias_2016.ipynb#L441-L455]
def clustering_accuracy(vectors, true_labels):
    # Two-means on raw vectors; t-SNE is used only for the plot
    predicted = kmeans(vectors, clusters=2, seed=1)
    # Cluster ids are arbitrary; see Pitfalls before using this number
    return mean(predicted == true_labels)

# Group score in the association test  [§4]
# [code: source/remaining_bias_2016.ipynb#L835-L843]
def group_score(xs, ys, attrs_a, attrs_b, space):
    # Sum of association for X, minus sum of association for Y
    total = 0
    for x in xs:
        total += association(x, attrs_a, attrs_b, space)
    for y in ys:
        total -= association(y, attrs_a, attrs_b, space)
    return total

# Permutation p-value over all equal-size splits  [§4]
# [code: source/remaining_bias_2016.ipynb#L846-L869]
def weat_p_value(targets_x, targets_y, attrs_a, attrs_b, space):
    # Observed score for the real split
    observed = group_score(targets_x, targets_y, attrs_a, attrs_b, space)
    pooled = targets_x + targets_y
    larger = 0
    total = 0
    # Enumerate every way to split the pooled words into two equal halves
    for half in all_combinations(pooled, len(pooled) // 2):
        total += 1
        rest = [w for w in pooled if w not in half]
        if group_score(half, rest, attrs_a, attrs_b, space) > observed:
            larger += 1
    return larger / total

Hyperparameters

Results

Author-reported. Setup: Hard-Debiased and GN-GloVe on their own vocabularies, each compared with its own biased starting point §3. The corpus behind GN-GloVe is not named in the paper body. The numbers below are the paper's text; the notebook outputs match them.

Association tests report p-values for the debiased spaces only §4:

The paper's Pearson p-values are reported as 0 or below 1e-30 in the text §4.

Paper vs code

Official code found through the link in the introduction §1: github.com/gonenhila/gender_bias_lipstick, commit d1cd879f9825d96cd8501450e5ab79051c767a5b, Apache-2.0. It contains two notebooks: source/remaining_bias_2016.ipynb (Hard-Debiased) and source/remaining_bias_2018.ipynb (GN-GloVe). Embeddings are not in the repository and must be downloaded as the README describes.

Discrepancies found:

• Word length. The paper drops words "longer than 20 characters" §3. The code drops words of length 20 or more remaining_bias_2016.ipynb.
• Neighbour count k. The paper says "k nearest neighbors" without a value §4. The code uses 100 remaining_bias_2016.ipynb and 100 for professions remaining_bias_2016.ipynb.
• GN-GloVe dimension. The paper does not mention dropping a coordinate. The code removes the last coordinate of the 300-dimensional GN-GloVe vectors and renormalises before any "after" measure remaining_bias_2018.ipynb.
• Cluster labels. The paper does not say how k-means labels are matched to genders. The code compares raw labels to the true labels with no alignment remaining_bias_2016.ipynb.
• Permutation test. The paper describes counting permutations that give an equal or greater score §3. The code counts scores strictly greater than the observed one remaining_bias_2016.ipynb.
• Classifier settings. The paper says "RBF-kernel SVM" §4. The code calls the library default SVC() remaining_bias_2016.ipynb. The notebook does not set C or gamma.
• Zero projection. Words with bias exactly 0 are counted as female-labelled neighbours remaining_bias_2016.ipynb. The paper does not say how ties are handled.
• Reproduced numbers. The notebook outputs match the paper's text for clustering precision (0.999, 0.925 for 2016; 1.0, 0.856 for 2018), correlations, the SVM accuracies and the three WEAT values remaining_bias_2016.ipynb remaining_bias_2018.ipynb. No discrepancy found there.
• Python version. The notebooks use Python 2 print statements. The README says Python 2.7 [README].

Pitfalls

• Vocabulary cut-offs. The 50,000-entry cut is applied to the first rows of each file, so it depends on the file's row order. The length filter follows the code, not the text (see Paper vs code).
• Neighbour search is expensive. Each word's neighbours come from a full sort over the reduced vocabulary. The notebook output shows about 72 seconds per space for 26,189 words remaining_bias_2016.ipynb. Plan for an O(V²) cosine pass per space.
• k is an unstated choice. The bias-by-neighbours correlation depends on k and the paper does not give it. Use 100 as the reference value and report it.
• Cluster accuracy is label-sensitive. The code does not align cluster ids with genders. An implementer who skips this step may get a different number. Compute the best of the two alignments and say which one you report.
• Same split, two spaces. The classifier split is shuffled once with seed 10 and reused for the original and debiased runs remaining_bias_2016.ipynb. Re-shuffling between runs changes the comparison.
• Association tests are exact but slow. The 8 + 8 case enumerates 12,870 splits. The code refuses sets larger than 10 and points to a sampled version that caps samples at 1,000,000 remaining_bias_2016.ipynb.
• Debiasing targets only the direction. The paper's own argument is that reducing the he-she projection leaves the neighbourhood structure intact. A debiasing step that uses this measure as its only check will look fine here and still fail the tests above §2 §5.
• Scope. Results are for two embedding families and English word lists. The paper does not test other debiasing methods, although §2 mentions one adversarial method it did not verify §2 footnote.

Lineage

• Builds on: Bolukbasi et al. (Hard-Debiased, post-processing, he-she direction) §2; Zhao et al. (GN-GloVe, debiasing during training) §2; Caliskan et al. (association tests) §3.
• Compared against: Hard-Debiased and GN-GloVe, each against its own biased starting point §3.
• Also cited: Zhang et al. (adversarial debiasing) mentioned, not tested §2 footnote.
• Superseded by: not stated in this card.