Neural Machine Translation of Rare Words with Subword Units

In plain words

Translation models usually work with a fixed list of words. Anything outside that list, such as a rare compound or a name, is either lost or copied across unchanged, and that breaks down quickly when languages build words differently.

This paper takes a different route: it cuts rare words into smaller pieces that the model already knows, much as a reader can guess a long German compound from its parts. The pieces come from a compression trick, byte pair encoding, which repeatedly glues the most common pair of neighbouring characters into a new symbol. Because the pieces are learned from the text, the model can also handle words it never saw in training.

The authors report that this beats the dictionary fallback they compare against on two English translation tasks from the 2015 WMT shared task.

TL;DR

Sennrich, Haddow and Birch adapt byte pair encoding (BPE) to split words into learned subword units, so a neural translation model works on a fixed set of pieces and needs no back-off dictionary for rare words. Character bigrams are tested as a second segmentation. On WMT 2015 English-German and English-Russian, both segmentations beat a back-off dictionary baseline by up to 1.1 and 1.3 BLEU in ensembles (author-reported). Read this if you build an NMT or tokenizer pipeline that must handle open vocabularies, and check the Pitfalls first.

Problem

• Word-level NMT models keep a vocabulary of typically 30,000 to 50,000 words, but translation is an open-vocabulary problem §1.
• Compounds such as the German Abwasserbehandlungsanlange (sewage water treatment plant) are rare and get no useful fixed-length representation §1.
• The standard fix is to back off to a dictionary look-up for unknown words, or to copy them across §1. The authors argue that this assumes a 1-to-1 correspondence between source and target words, and that copying fails when morphology changes or the alphabets differ, as in transliteration §1.
• Word-level models cannot produce words they never saw in training §1.
• The main claim is that word-level NMT translates rare and unknown words poorly, and that subword models improve them §5.1. Rare and unseen words make up about 9 to 11% of the test sets §4.2.

Method

The paper's changes are only to the units the model reads and writes. The architecture is an attentional encoder-decoder NMT system after Bahdanau et al., with a bidirectional GRU encoder, a single-layer alignment model and a recurrent decoder, trained with stochastic gradient descent and decoded with beam search §2. The authors state that their approach is not specific to this architecture §2. Section numbers below follow the arXiv v5 source, which is the ACL 2016 layout.

Candidate segmentations §3 §4.1

1. Character n-grams: split each word into character n-grams that do not cross word boundaries, marking word-final units. A shortlist can keep the k most frequent words unsegmented. The authors test bigrams with a 50,000-word shortlist (C2-50k) §4.1.
2. Compound splitting, Morfessor, hyphenation: reported in statistics only. The authors find they reduce vocabulary only moderately and do not solve the unknown-word problem, so they reject them for this goal §4.1.
3. Byte pair encoding §3.2: the method this paper adopts. Learn merge operations on the training text, apply them to new text, and restore the original tokenisation after translation with the word-final marker §3.2. BPE needs no shortlist, and its only hyperparameter is the number of merge operations. The final symbol vocabulary size equals the initial character vocabulary plus the number of merges §3.2.

BPE in use §3.2

• Start from characters, with an end-of-word symbol on each word. Count adjacent symbol pairs, weighted by word frequency, never across word boundaries. Merge the most frequent pair into a new symbol, and repeat §3.2 alg. 1.
• At test time, split each word into characters and apply the learned merges in the order they were learned §3.2 fig. 1. Any word can be segmented, so the network sees only known symbols, except unknown characters §3.2 footnote.
• Two ways to apply BPE: learn separate encodings for source and target, or learn one joint encoding over the concatenated source and target training text. Joint BPE keeps segmentations consistent across languages, which helps names §3.2 §5.2.
• For English-Russian, the authors transliterate the Russian vocabulary into Latin letters with ISO-9 to learn the joint encoding, then transliterate the merges back into Cyrillic for the Russian text §3.2.
• Infrequent symbols are kept out of the network vocabulary because of noise, for example characters from foreign alphabets. So the network vocabulary is slightly smaller than the number of symbol types §4.1.

Learn BPE merges alg. 1 §3.2 (our pseudocode; the paper's listing is Python)

# Learn BPE merge operations  [alg. 1] [§3.2]
# Input:  word_counts: dict, word -> frequency
# Output: merges: ordered list of symbol pairs

def learn_bpe(word_counts, num_merges, min_frequency):
    # Start from characters, plus an end-of-word symbol  [§3.2]
    # Code output uses a different marker placement; see Pitfalls
    vocab = {split_into_chars_plus_end(word): count
             for word, count in word_counts.items()}
    merges = []
    for step in range(num_merges):
        # Count adjacent pairs, weighted by word frequency; pairs never
        # cross word boundaries  [§3.2]
        pair_counts = count_pairs(vocab)
        if not pair_counts:
            break
        best_pair = most_frequent(pair_counts)   # ties: see Pitfalls
        # Stop when even the best pair is too rare (code only, not in
        # the paper)  [code: subword_nmt/learn_bpe.py#L346-L348]
        if pair_counts[best_pair] < min_frequency:
            break
        # Replace whole-symbol occurrences of the pair by the new
        # symbol, left to right  [§3.2] [alg. 1]
        vocab = merge_pair(vocab, best_pair)
        merges.append(best_pair)
    return merges

Segment one word §3.2 apply_bpe.py

# Segment a word with learned merges  [§3.2]
# Input:  word (str); merges: list of pairs in learned order
# Output: list of subword units

def segment(word, merges):
    # Split into characters, marking the word end  [§3.2]
    symbols = split_into_chars_plus_end(word)
    merge_rank = {pair: rank for rank, pair in enumerate(merges)}
    while len(symbols) > 1:
        # Keep only adjacent pairs that have a learned merge  [§3.2]
        candidates = [pair for pair in adjacent_pairs(symbols)
                      if pair in merge_rank]
        if not candidates:
            break
        # Apply the merge learned earliest
        # [code: subword_nmt/apply_bpe.py#L310]
        best_pair = min(candidates, key=lambda pair: merge_rank[pair])
        symbols = merge_occurrences(symbols, best_pair)
    # Drop the end-of-word marker from the final unit  [§3.2]
    return strip_end_marker(symbols)

Hyperparameters

Hyperparameters are author-stated, since the NMT training code is not in the released repository (see Paper vs code).

Corpus statistics (German training data; tab. 1, numbers as printed)

Results

Author-reported. Setup: WMT 2015 English-German and English-Russian, Groundhog NMT implementation, one or eight models, newstest2015 as test set §4 §4.2. Baselines are a word-level model without back-off (WUnk) and with a bilingual dictionary back-off (WDict), plus the authors' published syntax-based (EN→DE) and phrase-based (EN→RU) statistical systems tab. 2 tab. 3.

English→German tab. 2. Test-set sizes: all words n = 44,085; rare words n = 2,900; OOV n = 1,168. Single = best of 8 on dev; ens-8 = ensemble of 8.

English→Russian tab. 3. Test-set sizes: all words n = 55,654; rare words n = 5,442; OOV n = 851.

Headline gains, as computed from the tables. The abstract's "1.1 and 1.3 BLEU" are ensemble differences against WDict: EN→DE C2-50k 25.3 vs 24.2 tab. 2, and EN→RU BPE-J90k 24.1 vs 22.8 tab. 3. For EN→DE BPE, the gains are smaller: BPE-J90k 24.7 vs 24.2 (ens-8) and 22.8 vs 22.0 (single) tab. 2.

Other reported results (author-reported):

• Rare-word unigram F1 rises from 36.8% to 41.8% (EN→DE) and from 26.5% to 29.7% (EN→RU) §4.2 tab. 2 tab. 3.
• OOV precision and recall, EN→DE: WDict 60.6% precision and 26.5% recall; BPE-J90k 38.6% precision and 29.8% recall §5.1. The paper's C2-50k precision is 29.1% and its recall 33.0% §5.1.
• OOV precision and recall, EN→RU: WDict 9.2% precision and 5.2% recall; BPE-J90k 21.9% precision and 15.6% recall §5.2.
• Oversplit or non-morphemic segmentations still translated correctly in the examples given §5.2 tab. 4 tab. 5.
• Figures fig. 2 and fig. 3 plot unigram F1 by training-frequency rank for each system. The baseline spikes on OOVs, mostly copied names. The word system with a 500k target vocabulary (C2-3/500k, an analysis-only system with the WDict vocabulary size) degrades between rank 50k and 500k, while C2-50k stays more stable §5.1 fig. 2.

Paper vs code

The paper links github.com/rsennrich/subword-nmt in its conclusion §6. The repository is by the first author and is MIT-licensed LICENSE. It holds the segmentation scripts only. The NMT system, Groundhog §4, is a separate project that was not fetched. So every NMT hyperparameter above is paper-only.

• Merge count. Default --symbols is 10,000 learn_bpe.py. The paper's 59,500 and 89,500 merges §4.1 are not defaults, and the repository has no config for the paper's runs. Use the paper's values explicitly.
• Stopping rule. The paper gives none. The code stops when the best pair falls below --min-frequency, default 2 learn_bpe.py learn_bpe.py.
• Tie-breaking. Algorithm 1 uses max(pairs, key=pairs.get) with no tie rule alg. 1. The code picks the highest count and breaks ties by the pair's string order learn_bpe.py.
• Speed-only pruning. The code keeps a pruned candidate set and rebuilds it when the best pair drops below a threshold, to save time. Its comment says this should affect only speed learn_bpe.py. The paper's listing recounts all pairs each step; the code updates counts incrementally learn_bpe.py.
• End-of-word marker. The prose uses a dot-like symbol and the listing uses a separate </w> symbol §3.2 alg. 1. The released code's default format (version 0.2, written in the file header) attaches </w> to the word's last character instead learn_bpe.py [code: README.md#L168-L182]. Word-final and word-internal units therefore differ between the two formats.
• Test-time merge order. The code applies the lowest-ranked applicable merge first apply_bpe.py. This matches "the learned operations" in the paper §3.2.
• Output separator. The paper does not name one. The code appends @@ to every non-final unit apply_bpe.py and defaults to that separator apply_bpe.py.
• Vocabulary-based fallback. --vocabulary and --vocabulary-threshold revert merges that produce out-of-vocabulary units [code: subword_nmt/apply_bpe.py#L252-L255] apply_bpe.py. The paper only sketches this as a possible fix §3.2 footnote and does not evaluate it. The README shows a threshold of 50 as an example [code: README.md#L74].
• Features not in the paper. BPE-dropout (--dropout), glossaries and byte-level BPE (--byte) exist in the code learn_bpe.py apply_bpe.py. The paper describes none of them.
• Joint learning. The joint script learns on a list of texts such as the source and target sides learn_joint_bpe_and_vocab.py, which matches the footnote's concatenation §3.2.
• Russian transliteration. The paper's ISO-9 step for joint BPE on Russian §3.2 has no implementation in the repository. A grep for transliteration and ISO-9 finds no code; the only match is a README sentence about names README.md. Reproducing the EN→RU joint results would need preprocessing that is not released.
• Not checked. The BLEU and chrF evaluation, and the NMT training and decoding, are outside this repository.

Pitfalls

• Headline gains come from different systems. The 1.1 BLEU (EN→DE) comes from the character-bigram C2-50k system, not BPE. BPE-J90k gains 0.5 BLEU in the ensemble and 0.8 BLEU as a single model tab. 2. Check which system a claimed gain belongs to.
• Small gaps are inside the reported noise. The authors report differences of up to 1 BLEU between dev models §4.2. Single systems are the best of 8 on dev, a choice the authors say has a stabilising effect §4.2. Gaps of 0.3 to 0.5 BLEU are within that range.
• Rare words are a small share of the score. Rare and unseen words make up 9 to 11% of the test sets, so BLEU and chrF3 may understate the effect §4.2.
• Merge tie-breaking. Equal-count pairs can be ordered differently by different implementations, which changes the merge list. The code's rule is the highest count, then pair string order learn_bpe.py.
• Marker format. Choose one marker convention and keep it consistent between learning, applying and detokenising. The code default is version 0.2 learn_bpe.py.
• Joint BPE is not closed. Joint BPE can produce source segments unseen in training, because they only occur in the English text. The authors report this for 0.05% of test tokens §4.1 footnote. The paper saw no unknown symbols at test time, and it suggests reversing merges as a fix [§3.2 footnote].
• Vocabulary size is not justified. The authors call their choice of vocabulary size "somewhat arbitrary" and motivated by earlier work §6. Treat the merge count as a tuning parameter §3.2.
• Length and cost. BPE raises token counts modestly, from about 100M to about 112M German tokens tab. 1. Character-level units raise them much more, to about 550M tab. 1. The authors note that time grows at least linearly with sequence length, so oversplitting costs speed §4.1 footnote.
• Segmentation need not match morphemes. Oversplit or non-morphemic units still translated well in the examples §3 footnote §5.2.
• Cross-language consistency. Separate BPE for source and target can split the same name differently, and that produced transliteration errors (BPE-60k, EN→RU) §3.2 §5.2 tab. 5.
• Unstated details. Learning rate, step count, Adadelta epsilon and decay, maximum sentence length and hardware are not stated §4. Use the paper's design choices only where the card gives them.
• Plots and tables are the evidence. Figures are described in words and cited fig. 2 fig. 3. Use the tables for exact numbers.

Lineage

• Builds on: the attentional encoder-decoder of Bahdanau, Cho and Bengio (2014) §2; byte pair encoding, from Gage (1994), adapted here to merge characters for word segmentation §3.2; the dictionary back-off and shortlist techniques of Jean et al. (2015) and Luong et al. (2014) §1 §4.
• Compared against: word-level WUnk and WDict tab. 2 tab. 3; the authors' syntax-based SMT system for EN→DE and phrase-based SMT for EN→RU tab. 2 tab. 3 §4.2; a neural result of Luong, Pham and Manning (2015), quoted without re-running §4.2.
• Superseded by: none in this wiki yet.