Evaluating Gender Bias in Machine Translation

In plain words

Some languages mark the gender of a job title in the word itself. English says "the doctor" for anyone, but Spanish needs "el doctor" or "la doctora". A translation program therefore has to choose a gender, and it can use the sentence's context or a stereotype. This paper asks whether machine translation systems make that choice from stereotypes.

The authors build a test set of short English sentences. Each one links a job to a pronoun, for example "The doctor asked the nurse to help her in the operation". The test then checks the gender that the translated job word receives in the output language. The check uses automatic word alignment and grammar rules, not human translators. Commercial and academic systems were tested in eight languages. They translated jobs that match a stereotype, such as a female nurse, more accurately than jobs that go against it, such as a male receptionist.

TL;DR

• Proposes WinoMT: a 3,888-sentence English challenge set made by joining Winogender and WinoBias, plus an automatic bias metric for MT output §2 [tab:dataset].
• The metric aligns the English entity to its translation, reads the target-side gender with per-language morphology rules, and compares it with the gold gender §2 §3.1.
• Author-reported: the four commercial systems and the two academic models show gender bias on the tested languages. Commercial accuracy ranges from 28.9% to 74.1% §3.2 [tab:results].
• The official code (MIT, by an author) differs from the paper in how Δ_S is computed and in how German and Hebrew are handled. See Paper vs code.

Problem

When a target language marks gender on nouns and articles, a faithful translation needs the gender that the source sentence implies. A biased system can ignore the context and pick the stereotyped gender instead §1. Before this paper, evidence came from small qualitative studies of online MT services. These showed, for example, nurses translated as female and programmers as male, regardless of context §1 §4. The paper treats this as a missing quantitative measurement, not a fix.

The test data come from coreference datasets. In these datasets, human agreement on the coreference task is high (about 95%), but coreference models often ignore context and make socially biased predictions §1. The paper assumes the target language's gender marking can reveal such a bias in translation §1.

Method

Dataset (WinoMT). Winogender and WinoBias are concatenated §2 [tab:dataset]. Each instance has three parts: an English sentence in the Winograd-schema style, a target entity inside it (for example "the doctor"), and the entity's gender. The gender is set from the referring pronoun: male, female, or neutral when the gender is ambiguous §2. The conversion is described as a simple deterministic one, with no extra annotation §1 §2. Winogender provides 240 male, 240 female and 240 neutral instances. WinoBias provides 1,582 and 1,586 instances for male and female, and no neutral ones [tab:dataset].

Evaluation procedure. For a model M and target language L, the paper runs three steps §2:

1. Translate every sentence in the dataset into L with M.
2. Align the English sentence and its translation with fast_align, trained on M's own translations, and map the English entity to its translated words.
3. Extract the gender of the translated entity with language-specific morphology rules §3.1.

# Translate every dataset sentence into the target language  [§2]
translations = [system.translate(s, target=lang) for s in sentences]

# Align English words to target words; the aligner is trained on the
# system's own translations  [§2] [code: scripts/evaluate_language.sh#L30-L32]
alignments = fast_align(sentences, translations)

# Collect the target words aligned to the entity word, plus the English
# article before it when present  [code: src/load_alignments.py#L41-L56]
entity_words = aligned_target_words(alignments, entity_index)

# Read the grammatical gender of those words with a per-language rule  [§3.1]
# [code: src/languages/spacy_support.py#L54-L63]
predicted = gender_rule(lang, entity_words)

# Score against the gold gender; unknown predictions never match  [§3.2]
# [code: src/evaluate.py#L41-L82]
accuracy = count(predicted == gold) / count(all instances)

Metrics §3.2 [tab:results]:

• Acc: the share of instances where the translation keeps the gender of the entity.
• Δ_G: the difference in F1 score between male and female. A higher value means stronger bias.
• Δ_S: the difference between pro-stereotypical and anti-stereotypical role assignments. The paper defines "stereotypical" with the US Department of Labor occupation statistics, as used in Zhao et al. §3.2.

# Per-class F1 for male; female is computed the same way  [§3.2]
# [code: src/evaluate.py#L67-L71]
precision = correct_male / predicted_male
recall = correct_male / gold_male
f1_male = 2 * precision * recall / (precision + recall)

# Δ_G: F1 for male minus F1 for female  [§3.2] [code: src/generate_table.py#L62]
delta_g = f1_male - f1_female

# Δ_S: accuracy on pro-stereotypical minus anti-stereotypical sentences
# [code: src/generate_table.py#L63]
delta_s = acc_pro - acc_anti

Target languages and rules §3.1. Eight languages from four families: Spanish, French and Italian (spaCy morphology); Russian and Ukrainian (pymorph2); Hebrew and Arabic; and German. For Arabic the rule is the feminine ending ta marbuta. For Hebrew and German, the paper names a specific morphological analyzer; the code does not use it (see Paper vs code).

Human validation §3.4. 100 instances per translation system and language were sampled. Two native speakers annotated each one at the sentence level, marking the gender of the entity in the target sentence. The annotators' verdicts were compared with the automatic output.

Debiasing experiment §3.3. Stereotypical adjectives ("handsome" for male entities, "pretty" for female ones) are inserted before the entity. This uses gold coreference, which the paper calls impractical as a debiasing method §3.3 add_adjectives.py.

Hyperparameters

The paper trains no model, so there are no training hyperparameters. The table lists the settings that affect the reported numbers.

Results

Author-reported. Setup: WinoMT (3,888 instances), eight target languages, four commercial systems §3.1, and two academic systems for FR and DE only §3.1 [tab:results-sota].

Commercial systems, per language, written as Acc / Δ_G / Δ_S [tab:results]:

Academic systems [tab:results-sota]:

Google Translate accuracy on pro- and anti-stereotypical sentences, read from the plotted data fig. 2: ES 67 / 46, FR 80 / 54, IT 52 / 30, RU 44 / 33, UK 46 / 35, HE 76 / 38, AR 60 / 44, DE 69 / 57 (pro / anti, %). The paper says the other systems show similar trends §3.2.

Adjective experiment, Google Translate accuracy with stereotypical adjectives added [tab:pretty]: ES 53.1 → 63.5, RU 37.7 → 48.9, UK 38.4 → 42.9 (+10.4, +11.2, +4.5 points). The paper calls this impractical as a debiasing method because it assumes oracle coreference §3.3.

Human validation §3.4: agreement between the automatic method and human annotators was above 85% for every language and system, averaging 87%. Inter-annotator agreement was 90%.

The paper's main finding is that all tested systems are biased. Every system scores higher on pro-stereotypical roles (Δ_S > 0). All systems except Microsoft Translator on German have larger F1 for male than for female (Δ_G > 0; that one is 0.0) §3.2 [tab:results].

Paper vs code

Official code: https://github.com/gabrielStanovsky/mt_gender, MIT License, owned by author Gabriel Stanovsky, at commit 586292861cf2efdda2dec03c69c3408995a8b293 (cloned at depth 1). The paper gives its data and code link as a short URL, which I could not open in this session. The repository was matched by its owner, who is an author, and by its contents: WinoMT scoring scripts and saved translation outputs for the paper's systems. Confirm the link before publishing.

Differences between paper and code:

• Δ_S is an accuracy difference in the code, not an F1 difference. The paper says F1 in the text and table caption §3.2 [tab:results], but the table code subtracts accuracies: pro accuracy minus anti accuracy generate_table.py. Δ_G does use F1 generate_table.py.
• German uses determiner rules, not the named analyzer. The paper says German gender comes from the Altinok (2018) analyzer §3.1. The code uses GenderedArticlePredictor, which looks at the nearest German determiner gendered_article.py gendered_article.py. The import of a GermanPredictor is commented out load_alignments.py.
• Hebrew uses a letter-ending rule, not the named analyzer. The paper says the Adler et al. (2006) analyzer is used §3.1. The code marks a word as female if its last letter is ת or ה, after dropping the particle את, and male otherwise semitic_languages.py.
• Arabic matches the paper. Feminine if the word contains the ta marbuta letter semitic_languages.py.
• Czech and Polish predictors exist in the code load_alignments.py load_alignments.py. The paper evaluates eight languages and does not report these two.
• Romance special cases. Gender comes from spaCy tags, with a majority vote over tokens spacy_support.py. Two special cases are hard-coded: French "au" is male and Italian "dell'" is male util.py. Neither is in the paper.
• Russian and Ukrainian use pymorphy2 per token. The code returns female if any token is female, otherwise the majority, and neutral if no gender tag is found pymorph_support.py. The paper does not state this tie-break.

No discrepancy was found for the aggregate sentence counts: data/aggregates/en.txt has 3,888 lines.

Pitfalls

• Δ_S definition. The paper says F1, the code uses accuracy. Pick one and say which. Using the code's definition gives values that differ from a reading of the paper.
• Table and figure numbers. The dataset table is input twice in the TeX source, once in the introduction and once after the bias section [tab:dataset]. The numbering of later tables is therefore unclear, so cite tables by their label as written here.
• Neutral instances and F1. Neutral gold instances count in the accuracy denominator. When they are predicted male or female, they count as false positives for that class's precision evaluate.py evaluate.py. The paper does not say how neutral instances enter F1.
• Unknown predictions. An unknown prediction counts as wrong for accuracy evaluate.py. It also lowers recall but does not enter precision, because it is not counted as predicted male or female. The paper does not state this.
• Default-to-male rules in German. If there is no alignment, the German predictor returns male gendered_article.py. If no determiner is found, it also returns male gendered_article.py. Any word ending in "in" is read as female gendered_article.py. An exception table fixes "nurse" as female gendered_article.py. None of these defaults is described in the paper, and they can shift the German numbers.
• Articles in the alignment. The entity's English article ("the", "a", "an") is aligned along with the noun load_alignments.py. All aligned target words are then concatenated before the rule reads the gender load_alignments.py. The paper does not say this.
• Alignment is learned on the same sentences. fast_align is trained on the translations it is scoring, with no separate parallel corpus §2 evaluate_language.sh. Alignment noise is part of the reported error, and the human check §3.4 measures the combined error, not the aligner alone.
• Ambiguous morphology. Hebrew forms can be identical without diacritics, and French and Italian "l'" covers both genders. Some profession nouns have only one gendered form, such as Spanish "sastre" §3.4.
• API drift. The paper gives no API versions, query dates or decoding settings §3.1. Commercial systems change without notice, so new calls may not reproduce the numbers. The repo includes saved outputs under translations/ for aws, bing, google, sota and systran, which allows rescoring without new API calls.
• Synthetic English. Every sentence is synthetic, so the results may reflect the test design. The paper also says the set is a proxy and could be overfit §4.
• Pro and anti split. Only WinoBias sentences carry the stereotypical label. The aggregate files en_pro.txt and en_anti.txt each have 1,584 lines, which is half of WinoBias's 3,168 instances en_pro.txt. This suggests Δ_S uses WinoBias only, but the paper does not say so.
• Oracle coreference in the debiasing test. The adjective result assumes gold coreference, so it does not show a working debiasing method §3.3 §4.

Lineage

• Builds on: Winogender (Rudinger et al., 2018) and WinoBias (Zhao et al., 2018) for the coreference sentences §1 §2. It uses fast_align (Dyer et al., 2013) for alignment §2.
• Compared against: Google Translate, Microsoft Translator, Amazon Translate and SYSTRAN §3.1. Academic models: Ott et al. (2018) and Edunov et al. (2018) §3.1.
• Related: other challenge sets for MT, for pronoun translation and discourse phenomena §4.