Getting Gender Right in Neural Machine Translation

In plain words

Some languages mark the gender of a speaker in ordinary words. A French speaker who says "I am happy" uses one word for a man and another for a woman, while English does not change the sentence at all. A translation program has to know the speaker's gender to pick the right form, and a standard translator usually cannot know it from one sentence.

The paper's fix is simple. Each English sentence in the training data gets a short label at its start that says whether the speaker is a man or a woman. The translator learns to use that label when it chooses word forms. The authors also collected speaker labels for a large set of parliamentary speeches, in 20 language pairs, so that this kind of training is possible. Gains are real for some language pairs and small. For a few pairs the labelled model was even worse. For sentences spoken by women, the gain was largest.

TL;DR

• Adds a speaker-gender tag to the English source side of each sentence, then trains the same LSTM encoder-decoder with and without it §4.1 §4.2.
• Compiles Europarl sentences with speaker gender for 20 language pairs and uses 10 of them for experiments §3 tab. 1 §4.1.
• Author-reported: the tagged system is significantly better for French, Italian and Danish and significantly worse for Finnish, and gains about 1.2 BLEU on EN-FR test sets of female speakers §5 tab. 2 tab. 3.
• The speaker labels are gender predictions from an earlier MEP metadata set, not verified labels, and the paper does not measure how accurate they are §3. See Pitfalls.

Problem

In languages with grammatical gender, adjectives and verb forms can agree with the speaker's gender, so a faithful translation depends on information the English source often leaves out §1 §4.1. A standard NMT system with no speaker information falls back on the form that is most frequent in its training data §1. Those data are skewed: in the EN-FR corpus 67.39% of sentences are spoken by men, and in 10 datasets 32–33% of sentences are by women §3.1. The paper's claim is that the gender of the author or speaker is useful information that is lost at translation time §1.

Earlier work on this problem was limited. The authors state that their work is, to their knowledge, the first attempt at a speaker-informed NMT system §2. The main obstacle they name is the lack of large parallel corpora with speaker annotations for most language pairs §3.

Method

Dataset construction §3. The authors follow Rabinovich et al. §3. Parallel sentences from Europarl are tagged with speaker attributes. Speaker names come from the Europarl source files; the file names give the session date. Speakers are then matched to metadata on members of the European Parliament (MEPs) from Rabinovich et al., which includes name, country, birth date and gender predictions for each MEP §3. The attributes stored per sentence are name, gender, age, date of birth, euroID and session date §3. Gender is therefore the MEP's predicted gender, not a verified one §3.

Tagging §4.1. Every source sentence in the experiments is augmented with one tag at its start, on the English source side only. The tag names the gender of the speaker. The paper's example is the English sentence "Madam President, as a..." with the tag FEMALE in front §4.1. The paper does not show the male tag string. The method reuses an idea from earlier work that adds a tag to the input to steer the output, as in politeness control §2 §4.1.

Systems §4.2. For each of the 10 language pairs the authors train two systems. The baseline is trained on untagged data; the tagged system is trained on the tagged source §4.1. Both use OpenNMT-py §4.2. The architecture is a sequence-to-sequence encoder-decoder with an LSTM recurrent unit, "trained with the default parameters" of that toolkit §4.2. Words are split with byte-pair encoding (BPE) using 89,500 operations §4.2. Training runs for 13 epochs; the reported system is then picked from the run, with no stated rule §4.2.

Evaluation sets §4.1. Each system is tested on a randomly selected general test set of 2K sentences. For EN-FR the authors also test on a 2K male-only and a 2K female-only test set. They add two more sets, one of male and one of female speakers, in which the first person singular pronoun appears §4.1.

The paper contains no equations and no algorithm box. The steps are as follows.

# Add the speaker's gender to the English side of one sentence  [§4.1]
def tag_source(source, speaker_gender):
    # Tag on the English source only; male token is not stated  [§4.1]
    tag = gender_token[speaker_gender]
    return tag + " " + source

# Train one system for one language pair  [§4.1] [§4.2]
# condition is "baseline" (untagged source) or "tagged"
def train_system(pairs, condition):
    # Learn 89,500 BPE operations; whether shared across sides is
    # not stated  [§4.2]
    merges = learn_bpe(pairs, operations=89500)
    data = apply_bpe(pairs, merges)
    # LSTM encoder-decoder with OpenNMT-py default settings  [§4.2]
    model = seq2seq_lstm(default_opennmt_settings)
    # 13 epochs over the training data  [§4.2]
    for epoch in range(13):
        train_one_epoch(model, data)
    # Checkpoint choice criterion is not stated  [§4.2]
    return select_best(model)

# Score one system with BLEU on every test set  [§4.1] [§5]
def evaluate(model, test_sets):
    scores = {}
    for name, (sources, references) in test_sets.items():
        # Decoding settings are not stated  [§4.2]
        hypotheses = translate(model, sources)
        # BLEU tool and tokenisation are not stated  [§5]
        scores[name] = bleu(hypotheses, references)
    return scores

# Compare baseline and tagged systems at p < 0.05  [tab. 2] [§5]
# MultEval is the tool named for significance  [§5] [tab. 2]
significant = multeval(baseline_scores, tagged_scores, alpha=0.05)

Analysis of the data §3.1 fig. 1. The paper analyses the EN-FR corpus by speaker age and gender. Figure 1 plots the percentage of female and male speakers per age group. In the text: the youngest group, aged 20–30, is 0.71% of the sentences, and it is the only group where women outnumber men. The 50–60 group holds 43.76% of all sentences §3.1 fig. 1.

Hyperparameters

Europarl sentences per language pair, annotated with speaker information tab. 1:

Results

Author-reported. Setup: BLEU on randomly selected 2K test sets, OpenNMT-py LSTM systems, one baseline and one tagged system per language pair §4.1 §5.

BLEU on the general test set, baseline (EN) vs. tagged (EN-TAG). The Δ column is the difference of the two cited numbers, computed here. Starred rows are significant at p < 0.05 per the authors tab. 2:

EN-FR on four test sets, baseline (EN) vs. tagged (EN-TAG). The Δ column is derived. All four tagged results are starred as significant tab. 3:

The authors report the largest BLEU gain on the female test sets, and the largest on F1 §5 tab. 3. The paper also says the gains come partly from word choice, not only from morphological agreement §5. The example in which "penser" is preferred to "croire" §5 is a qualitative case, not a measured effect.

Paper vs code

No official code. The paper links one repository, https://github.com/evavnmssnhv/Europarl-Speaker-Information, owned by an author §3. It holds the speaker-annotated corpora as one .tar.gz archive per language pair, a README, and no scripts. Its GitHub licence field is empty, while the paper says the corpora are under CC BY 4.0 §3. I did not open the archives. The NMT systems were built with OpenNMT-py, a third-party toolkit §4.2. Searches for the method and its experiments found nothing: "gender neural machine translation speaker tag europarl", "gender nmt tag", "Europarl speaker gender tags OpenNMT", and the paper's dataset name within the author's GitHub account. Because no code exists, there are no code-level values to compare with the paper.

Pitfalls

• Speaker gender labels are predictions. The gender of each speaker comes from the MEP metadata of Rabinovich et al., which the paper describes as gender predictions per MEP §3. The paper does not measure how many labels are wrong, and an error would tag a sentence with the wrong gender.
• Unlabelled speakers are not covered. Table 1 counts only annotated sentences tab. 1. The paper does not say how a sentence without a speaker gender label is tagged at training or test time.
• The male tag is not shown. The paper shows only the FEMALE tag §4.1. Use the same token format for MALE in an implementation, and test that the tag is the only change between the two systems.
• Test-time tags are implied, not stated. The paper says every sentence is augmented §4.1. It does not say that the test sources carry the gold tag, but the gender test sets only make sense if they do.
• Text and table disagree on significance. The text says French, Italian, Portuguese and Greek improve significantly §5, but Table 2 stars only French, Italian and Danish as significant gains tab. 2. Portuguese and Greek are not starred.
• Significance threshold is inconsistent. Table 2's caption says p < 0.05 tab. 2. Table 3's caption says p < 0.5 tab. 3. The text gives no threshold in §5. The 0.05 value is the one used in Table 2 and is the one assumed here.
• Finnish is a significant loss. Finnish drops by 0.47 BLEU with a significant result tab. 2. The text says only that the Finnish system does not improve §5.
• One run per system. No seeds, repeated runs or confidence intervals are reported. The effect sizes are below 1.5 BLEU for every pair. MultEval is named for the tests §5 tab. 2, but how many runs enter it is not stated.
• Data imbalance affects the result. The EN-FR data are 67.39% male §3.1, and the authors expected the largest gains on female-speaker sentences for this reason §5. A tag does not change the training mix, so the gain on female sets may depend on how much female data there is.
• Training details are missing. The paper gives only the toolkit and "default parameters" §4.2. Learning rate, batch size, model size, dropout, beam size and the checkpoint rule are not stated. An implementer must choose these, and results may not match.
• BPE scope is unclear. The 89,500 operations §4.2 are not described as joint or per language. The vocabulary can change the result, so do not assume one.
• Small pronoun sets. The sizes and selection rules of the M1 and F1 test sets are not stated §4.1.
• Only one baseline. The only comparison is the same architecture without the tag §4.1. The paper does not compare the tag with other ways of supplying speaker information. The authors say they plan to try them §6.
• Source-side tag only. The tag is put on the English source side, and the paper does not test the reverse direction §4.1.

Lineage

• Builds on: Rabinovich et al. (2016) for the speaker annotations and MEP gender metadata §3; Sennrich et al. (2016) for tagging sentences to control politeness, which the paper calls a similar approach §2. Sennrich et al. (2015) for BPE §4.2. Klein et al. (2017) for OpenNMT-py §4.2. Koehn (2005) for Europarl §3. None of these has a card on this wiki.
• Related: Mirkin et al. (2015), which motivates personalised MT without experiments §2; zero-shot multilingual NMT, which uses similar source tags §4.1; domain adaptation by source tags §4.1.
• Compared against: the same architecture trained without the tag, per language pair §4.1.
• Superseded by: not stated in this wiki.